Erik: [00:00:00] I think a lot of these proofs and things like that are also based on known failure modes, right? And it’s, it’s the unknown unknowns in production are what get you every time.
Bridget: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout, and today we have a very special episode. This week, I was in Chicago to curate the distributed systems track at GOTO Chicago, and after the track, I sat down with all the speakers for a live panel discussion. The show notes for this episode can be found at arresteddevops.com/gotochicago2018. But first, a word from our sponsors. ChefConf will be held May 23rd through 26th in Chicago. Chef has been a longtime supporter of the DevOps movement and of this podcast. ChefConf will have talks on infrastructure automation with Chef, compliance automation with Inspect, application automation with Habitat, and a ton of other relevant content. Register with discount code ADO2018 to save 10%. Visit chefconf.com for all the details. And remember, code ADO2018 gets you 10% off the ticket price at chefconf.com.
[00:01:28] Your application sits on layers of dynamic infrastructure and supporting services. Datadog brings you visibility into every part of your infrastructure, plus APM for monitoring your application’s performance. Dashboarding, collaboration tools, and alerts let you develop your own workflow for observability and incident response. Datadog integrates seamlessly with all of your apps and systems, from Slack to Amazon Web Services, so you can get visibility in minutes. Go to arresteddevops.com/datadog to get started with Datadog and get a free t-shirt. With full observability, distributed tracing, and customizable visualizations, Datadog is loved and trusted by thousands of enterprises, including Salesforce, PagerDuty, and Zendesk. If you haven’t tried Datadog at your company or on your side project, go to arresteddevops.com/datadog to get a free t-shirt and support Arrested DevOps. So, without further ado, let’s kind of go to our— in order of our track today, let’s talk about our distributed systems track at GOTO Chicago. Please introduce yourself, guest.
Jeff: [00:02:34] Hi, I’m Jeff Hodges. I’m a consultant at this company called Darkish Green. I run the company. I did a talk on productionizing distributed systems and what it means to actually go from not having one to having some and turning it into real code and migration.
Jordan: My name is Jordan Hendricks. I work for Joyent. I’ve been there about 2 years, and today I talked about Manta, which is Joyent’s object storage product, and adding a new feature to Manta and what that means in the context of Manta’s design and the trade-offs that were made for that system when it was created and how we work within those.
Erik: I’m Erik St. Martin. I work for Microsoft as a developer advocate. Today I talked about using Kubernetes as kind of building blocks for your own distributed systems, and kind of how the control plane works and its extensibility.
Alena: Hi everyone, I am Lena Hall. I have talked about running distributed data stores on Kubernetes, the concepts behind it, like StatefulSets, Persistent Volumes, but also about Kubernetes jobs, and a little bit about running Spark on Kubernetes as a scheduler.
Kyle: [00:03:45] My name is Kyle Kingsbury, and I talk today about testing distributed systems for safety with Jepsen and some recent findings in different databases about concurrency bugs.
Bridget: Awesome. So, welcome to our panel. So, the first thing that I want to ask, and you can ask each other questions too, but the first thing I want to ask is, when you’re thinking about state, when you’re thinking about storing things, and a lot of you touched on storing things in distributed systems and Spoiler alert, that doesn’t always go well. For those in the audience who think to themselves, but no one is going to let me insert really exciting thing here, I’m wondering if all of you with your vast experience in this area can give us a few tips, just a few hints about how can people be a little safer in their distributed systems? And I’m gonna, you know, make this a softball and lob it right to Kyle since he spends a lot of time testing distributed systems.
Kyle: Okay, so specifically with respect to distributed state, I would say do as much immutable as you possibly can. It solves consensus trivially if you’re familiar with the Lamport proof that says it takes 2 rounds only if you have conflicting proposals. If your data is mutable, no conflicts.
Bridget: [00:04:58] Absolutely. All right, and you touched on this too. Can you talk about this a little, Alena?
Alena: Yes, I also like what Efraim— I mean, Kyle mentioned in his talk that you should not really trust the documentation that much. You should always try to learn how the system works and learn some distributed algorithms behind it and try to understand it and test it out for your own use case before actually believing everything and implementing it in your own system. But also, if it works, maybe you should just keep running as it works. But if you need more, I don’t know, flexibility in timing, you can try and experiment with new things.
Bridget: Yeah, absolutely. And I think that I also wanna hear from Jordan because she spoke about when you are adding to an existing system, and I don’t know about you, but not everyone I’ve ever met gets to work in a greenfield. Some of us, spoiler alert, already have legacy microservices. And so if you’re dealing with a system that already exists, people kind of care about the stuff that’s in it. Maybe they don’t want that state to go away. So can you address that a little bit from your perspective?
Jordan: [00:06:10] Yeah, I think it’s really useful to think very clearly about if you’re working with microservices in a distributed system, what services you want to be stateless and what things you want to be stateful and really keep that second list very short, which makes it a lot easier to verify. And also keep your design as simple as possible. One of the things I really like about Manta is that it’s actually pretty simple to understand just looking at a design diagram. And that makes it much easier to reason about because Turns out we’re not very good at reasoning about these things on our own.
Bridget: And that makes me think too, Jeff mentioned that in his time at Twitter, they were dealing with state that was also things that were being sent to Twitter that they didn’t want, like spam things. So there’s, you mentioned verification, but it’s like some things are rights, but they aren’t the rights you want. I’m wondering if you can talk a little bit about how you distinguish between This is the content we want to keep. This is the content we don’t want to keep. And how do we do that in a timely manner with a really complex system?
Erik: [00:07:10] Uh, wow.
Jeff: Okay, we should write a dissertation together. Um, but the initial feelings are, uh, the first thing you do is you actually can accept writes and take them into your system. You can take those writes in. But you don’t actually have to distribute them back out. You actually can take this and like, instead of fanning out a tweet, you take it in, analyze it. If it looks slightly suspicious, using whatever heuristics you’ve already got prebuilt, you can actually just choose to not deliver it. And that’s actually very common for most systems. The acceptance of writing doesn’t mean that you have to accept all the reads that are gonna come out of it. And there are far, far, far more reads for any given system than there are writes. And blocking at that point usually works out better.
Bridget: That’s actually a really interesting point. And that makes me think of the systems that Eric was talking about when you’re when you’re building on Kubernetes and you’re building complex systems that might be kind of different than what people expected, or if you’re— the reason that you might be doing custom operators or whatever is that you’re trying to build things that no one else thought of yet. Can you talk a little bit about that anticipating and then creating? This is kind of a— you’re trying to build the systems of the future, like we all are, but How do you even know what that system should be?
Erik: [00:08:29] So I think a lot of it— a lot of us here have backgrounds in building distributed systems kind of from the ground up. I think the important thing is, is many of these things we would have created anyway, right? We’re talking about schedulers and things of that nature. So kind of what you’re looking at is these areas where we can build on top of what’s already been built rather than kind of creating our own. Because that’s where mistakes are made, right? Like, we can see a history of this with most distributed systems. Like, we run into things where failure modes we didn’t anticipate. And if we can leverage kind of the guarantees that exist for things like Kubernetes already and etcd, and we can build on top of those things, then it prevents us from making some of the same mistakes, and it prevents us from introducing unneeded complexity into our applications.
Bridget: And that’s making me think, okay, Kyle has spent a bunch of time Uh, talking about and looking at, and we looked at in Kyle’s presentation, um, the, like, over a really long period of time, and we’re talking years, it seems like a lot of organizations keep making similar mistakes. Can you talk a little bit, Kyle, about the way people are or aren’t learning from other people’s failure in distributed systems?
Kyle: [00:09:48] I think there’s been a rapid adoption of additional testing and of testing under failure conditions like partitions and clock skews, people still insist these things don’t happen. Sometimes, but it’s rarer, right? It used to be people would just laugh it off and now they’re like, oh yeah, I guess we need to do that. And often that’s driven by like a customer comes to you and says, we had this happen in production, please database vendor, fix this for us. But to your point, I think that there’s like a settling-in period for a distributed system where you sort of have to run at a sufficient scale and at sufficient with sufficient weird inputs, and over time you solve all those weird corner cases. And when you start something like a greenfield project, you don’t have the advantage of all of that accrued corner case experience.
Bridget: It’s almost like— if you want to give the mic to Jeff, because I know he has a thought on this— it’s almost like there’s no reality like production. Like, there’s every Every test that you want to do, every mock that you create, that’s great, but there’s nothing quite like production.
Jeff: [00:10:56] Yeah.
Erik: Wow. Damn.
Jeff: You just gave me a whole other thing I want to talk about, but I’m not going to talk about that. I want to first talk about one of the reasons why— so I came in here and I got one of the things I got to update for this talk, which was it was a talk from 5 years ago. It’s been lost to the ether because it got pulled for acquisition reasons. Strange. But one of the things I didn’t get to talk about back then that I talked about as being a future idea was talking about Docker, Mesos, and data center scheduling systems. And, like, now I get to come to a conference 5 years later and listen to these folks, like, talking about Kubernetes and giving us, like, the actual work that I’ve been hoping we would see happen. The reusable pieces that are coming out. Where it’s not just like, okay, etcd is a nice piece of software, but I actually don’t have to build all these things again. Really neat. It’s been— writing this talk and having to update it was actually pretty stressful. Fortunately, we still screw up the same exact things. Like the majority of it, it’s the same mistakes. But we have so much more to do.
Alena: Yeah, so also on the topic of errors that we just discussed and corner cases, I think there are like 2 types of errors in distributed systems. One is engineering errors and another one is algorithmic errors. So We probably talk mostly about engineering errors, but there are also ways to prevent algorithmic errors when you’re building new distributed systems, like specification languages like TLA+ for describing the algorithm, which don’t have, like, a real one-to-one connection with the code itself. They’re more like mathematics that describes the algorithm to prevent and catch those logic errors and to test out all of the possible scenarios. First before you actually try to make some engineering errors.
Kyle: [00:12:45] And I just— I see this sort of a part of a continuum of safety. Like, you have your proof and your algorithm, and there’s an implementation of it which maybe is based on a proven algorithm, but some of the inputs are malformed, or the underlying storage guarantees weren’t there, like it didn’t fsync. And then you discover that, you know, because of that subtle violation, the correct proof wasn’t actually okay. It’s like you need both specification language and proof and also implementation tests, and it’s a whole sweeping spectrum.
Bridget: Absolutely. But, yeah, go ahead.
Erik: I was just gonna say, like, I think a lot of these proofs and things like that are also based on known failure modes, right? And it’s the unknown unknowns in production are what get you every time.
Bridget: Absolutely. That brings us back to, like, Hey, now that we’ve figured out a bunch of this stuff is gonna fail, and we can break it and make it fail, I wanna hear about production, the unknown unknowns, when you’re diving into the practicalities. Like, we started the day with Jeff telling us about some of the practical things we have to worry about with distributed systems. And that’s basically, What I want to talk about now is just how do we figure out which unknown unknowns? I mean, do we wait for them to page us in the middle of the night? Or like, how do we try to keep ourselves as, you know, asleep and happy and not paged as possible? And let’s start with Jeff, but I know a few of you have some thoughts on this.
Jeff: [00:14:24] Okay, I’m super curious what everybody else is going to say because I gave a talk on it. But I will say that One thing that I will continue to insist on, I think, is actually— it’s difficult to ship. I still every once in a while will do a Google search every 6 months to see if there’s a better version of the software already written for me, which is feature flags and like an actual robust implementation of them in whatever environment I need to be deploying in. That’s actually still kind of buggy and less done. And part of the trick, of course, is that feature flags starts getting really integrated into your own deploy processes and your own user models and like how you see access patterns. And that’s one of the reasons why we’ve had less systems that are good at it. But we’ve also had really bad metric systems, and we’re starting to get decent ones now. Like Prometheus is not the worst, and like Stackdriver isn’t the worst, and like these things are like reasonable. And I’m hoping that we get to build out more feature-flaggy runtime, a way for us to, you know, darkly roll new software out without actually having to use its results, and that stuff requires stuff like feature flags. I mean, even Let’s Encrypt had to build their own.
Jordan: [00:15:29] I have something. So, um, this is the, the thing you brought up about how to deal with production, uh, finding bugs or errors or whatever you see in production versus testing is something that, uh, my team at Joyent is really, uh, going through right now, and I don’t have a great answer for it, but we’ve really seen Manta, um, like I mentioned in my talk, uh, Samsung is using this, and it’s a really big deployment, like very big, and we see the limits of our software in ways that I never thought were possible. So like, an instance of this is we’re worrying about, at this point, can we fit all of the services we need to in DNS packets, like, for every single instance of it? Like, not— I’m not saying we’re there yet, but that’s a question we’ve had to think about because it’s just at that scale. And I think being able to assess the practical realities of the system you have while also Making sure that you have robust testing is a real challenge. And I think just thinking about what is this thing at infinite scale from the very beginning is something that I’ve had to really internalize and not something that I had a good education in before this job.
Jeff: [00:16:31] I have one thing. You’re not alone in having to deal with Samsung problems. There’s actually one very clear patch I remember us having to add to a web server that’s very large and does a couple million requests specifically for an old mobile platform from Samsung. That they were not doing HTTP in the way that you would expect. And we, like, held on to that thing for, like, 4 years.
Erik: Yeah. So I would agree with the feature flag aspect. And one of the things I’m really excited about that people are starting to do more is chaos engineering. And by leveraging things that are shared, you can learn from other people’s problems too. Right? Because— and I ran into an example one time. When we think about, like, working on large-scale systems, one of the things most of us are used to is exhausting file descriptors, right? Local ports, right? Like, and this happens. So, but other people are not necessarily used to this, and it’s something simple, and they’re like, oh, I’m getting this weird, you know, error from the database or something, and the local ports are—
Jeff: [00:17:33] I have a Mac, though, and my Mac runs out of file descriptors constantly.
Erik: Ah, yeah, yeah, yeah.
Jeff: We’ll move on. Sorry, sorry.
Erik: So, but I use it as an example, right? Like, you don’t know it’s an issue until you know it’s an issue, and it’s a common one. So, by having these shared kind of chaos engineering suites and things like that, we can help learn from each other when we run into these weird failure modes.
Jordan: Also running out of TCP connections. That happens a lot too.
Bridget: Yeah.
Kyle: What up, time wait?
Jordan: Yeah.
Erik: Yeah. And that was— so that was the exact scenario I was thinking of. Most people don’t realize that there’s a period of time between you closing a connection and handing it back to the operating system and then the operating system allowing you to reuse it. Which means if you do something very batchy and say you fan out and you open 50 connections and then you close them and you open, you can potentially be handing them back to the operating system faster than it’s handing it back and then you run out.
Kyle: I, I, I’ve got another spitballing here but completely delicious failure mode for production. I don’t know how much I can say, but a major cloud provider at one point took it upon itself to occasionally swap pages of your RAM with other VMs’ RAM. Defending against that is a whole open question.
Bridget: [00:18:51] Wow.
Erik: I really hope it’s not, not us. Did anybody catch a side-eye there?
Kyle: It’s fixed, it’s fixed.
Jordan: All right.
Jeff: There is at some point, by the way, to get back to time wait really fast, there will be at some point if you do anything that’s like slightly non-trivial, which can be like 20 machines, there will be at some point in which you are staring at a diagram of TCP and how the protocol state machine, and you will be wondering, what the fuck, what is this time? Like looking at time wait in your LSOF or whatever and just crying because you have no idea why your MySQL database or whatever just isn’t working anymore. And there’s no, I have— we’ve got a whole thing about World Cup that I can’t do on this room, but we should talk about it afterwards. What?
Bridget: Oh, it’s so true. So, so, Lena, you were talking about Cassandra, and you were talking about, you know, Spark and Kubernetes and using all these things together. You had some cautions up there. Uh, you were saying, you know, is this right for you?
Erik: Maybe.
Bridget: And I think that a lot of times when people— and that, that made me laugh, and I had to tweet about it— because a lot of times When people are looking at the aspirational distsys stuff that sounds awesome, and clearly it doesn’t have any of the caveats because it hasn’t been tested yet. So it doesn’t have any of the caveats of things you should be terrified of. But when people are looking at all of the things that sound exciting, you know, conference-driven development, they want to go home and do all of the things you just all talked about. I’m wondering what kind of advice can you give them, like, from the example that you talked about?
Alena: [00:20:21] Yeah, for sure. When you listen to a talk and you see that, wow, all of those things work and all the demos work, but if you try something complicated that you would try in production, you might be the first person who would actually run it with Spark on Kubernetes. So, like, you might be very— you might be a very brave person to try it in production and then report bugs and submit pull requests, and that is a good thing to do, but some companies don’t want to skip this phase. So, I would say if you look at the code for Spark and see how the specification of drivers and executors is implemented in Kubernetes, they’re implemented as pods. And there are some corner cases right now that are being worked on when there is like a single point of failure where if a pod, like a driver pod dies, something might go wrong. As you might have seen from Kyle’s talk, there was a really great image about process dying. So, yeah. And there are many other corner cases that you will only find out when you actually start to browse through the project issues. And trying out the thing yourself on something that involves many systems integrated together and not in particularly just one project. Like, for example, right now there is an issue with Cassandra to Spark connector where, because it supports Spark, the older version of Spark, it doesn’t support the 2.3 version of Spark. So, you need to build a JAR and attach it to a cluster separately to make it work. It’s really simple. It’s just simple fixes to simple workaround issues.
Erik: [00:22:22] Uh, so one thing I’ll say about, uh, coming to conferences and stuff, I take it away from, um, his name’s Lost, he makes the badges at DEF CON, and he made a comment once that I really loved about watching talks, is that these aren’t exhaustive, you know, training material, right? Like, we, we’re, we come up here and we give you enough hooks to send you home with things that you might want to research further on your own. So, you know, we don’t have all the disclaimers and exactly how to use these things and exactly what their failure modes are. It’s just enough to kind of catch your attention and send you off. And it’s also a meaning of if it’s not necessarily understandable to you, don’t feel bad because it gets really hard to, in 30 minutes, give people enough information about a topic.
Bridget: Yeah, absolutely. So I’m going to ask you all to do something even harder than giving all the information about topic X in an hour or 30 minutes. I’m going to ask you to, because as it turns out, we only have about 10 minutes, maybe a little bit less for this podcast left. And so, I want to make sure that each of you has a chance to tell our audience here and our rest of DevOps listeners and the world your best advice. It can be a multi-part. That’s okay. We’ll do it in order of the talks, and we’ll say, what’s your best advice for people who are trying to do distributed systems in production today on services that they care about, with data they care about? Like, what’s your best advice for what they might want to consider doing? We’ll start with you, Jeff.
Jeff: [00:24:01] Oh, no, no, no.
Bridget: You’re not starting with me. Oh, I’m not starting with you?
Jordan: What?
Bridget: Okay. All right. I’ll start with someone who’s holding a mic, and go.
Erik: All right, I’ll volunteer to go first. Um, yeah, so this is actually a really interesting question because it turns out it doesn’t matter how many distributed systems you build or how long you do it for, it’s still hard. Like, it’s always hard.
Bridget: Spoiler alert, distributed systems are hard. I was promised hugs and puppies.
Erik: So yeah, I mean, I think it’s really important to understand, um, your actual use case what consistency guarantees you need. You know? Because there’s no panacea, right? There’s no, like, I get all of it. And it’s just not gonna happen. So you kind of have to think about your use case. There’s a lot of things Jeff talked about, you know, using backpressure, circuit breakers, things like this, so you can eliminate things like cascading failure. Item potency. A lot of these things are kind of, like, core to building distributed systems. Like, these things should be thought of from the beginning. And I think based on everybody’s history, we’re gonna hear a lot of other horror stories that will lead to best practices.
Jordan: [00:25:17] I can go next. So, the really low-hanging fruit, if you haven’t done much distributed systems, is that everything fails all the time. Don’t pretend otherwise. It’s just everywhere. So that’s step 1. Step 0. Another— a couple other things I think I’ve learned observing production systems that I didn’t have a chance to really grok before: bounding request time in some way, like as a function of the type of work that you’re doing, is really useful to make sure that services don’t run off into la-la land doing— you don’t know whether it’s slow or whether it’s just a variable-length request, something like that. So avoiding compound operations in as many places as possible I think is really useful for being able to reason about the system. And I had one more. Oh, so one thing that was not obvious to me, I think, at first was that when we talk about horizontal scalability and deploying more instances of a thing to solve a bottleneck, that’s not necessarily going to solve your problem all the time. So if you have an instance that has a memory leak, deploying more instances of that thing is just going to be more memory leaks, and you’re not going to increase capacity that way. So I think those are my really practical pieces of advice that And backpressure is another one that’s not always at the forefront of my mind too, but you already mentioned that one.
Alena: [00:26:30] Yep. I agree with Eric and with Jordan that failure is everywhere and that the understanding of distributed systems is really hard. And that actually reminds me, I attended a workshop on TLA+ and the instructor was actually Leslie Lamport. And he was like, it’s okay if you don’t understand distributed system after 3 days of my super, super smart workshop. It took me 20 years to get all of the pieces. So, it’s okay to not understand it. But I found that reading the same paper for 20 times actually helps. And about failures, I found that in production especially, building in observability in as many pieces of infrastructure and software as you can actually really helps. Because you start noticing patterns if something goes wrong, or you can even build in analysis on those metrics, and that will help you discover some new unknown unknowns.
Kyle: [00:27:34] These are really good ideas. I’m completely with all of you on all of this. And for more, if readers want to go deeper, Jeff Hodges has a really good blog post called Distributed Systems for Youngbloods. And I have a whole class which partly builds on that with open notes. Look for A4 distributed systems class.
Bridget: We’ll have a link in the show notes.
Kyle: Cool, there’ll be a link in the show notes. And to build on top of this, I think we have a common theme, which is assume the worst, right? And in mathematics, you say, let’s make a weak hypothesis. Let’s assume we have very little to go on. What’s the strongest thing we can claim? So assume your clocks are garbage. Assume your runtimes will pause. Assume your nodes will occasionally just, you know, wander off for a while. And if you make these assumptions, even if they’re unreasonable, even if they don’t happen much of the time, it will free you from unknown unknowns down the road when these things really do happen.
Jeff: This is great because y’all have covered so much technical ground I can steal the social ground, which is distributed systems, since we tend to work with more machines, it also means we work with more capital, with more teams, with more organizations, with other organizations outside of our own. And that means that as distributed systems engineers, we have to be as comfortable talking about machines as we are talking about humans and how they interact. If you’ve got— if you don’t have a migration plan to get people onto your new distributed system, it’s not gonna go anywhere. Like, you actually have to think about how other people are gonna use this, what their desires and incentives are, and then work with them as they ask for change. And, like, we have, like, a technical set of problems embedded inside of a social space of humanity, capitalism, economics, and all the rest of it. Like, this is— there are other constraints outside of just our values as engineers.
Bridget: [00:29:22] All right. And that’s our time. So, I think what we’ve taken away from this is, spoiler alert, like Soylent Green, distributed systems are made of people and their efforts. And we have to pay attention to that. All right. Thank you so much. Let’s have a round of applause for our panel. So, that was what we recorded live at GOTO Chicago. Just to wrap up, we have some community and event stuff. OpenCFPs, lots of DevOps Days, devopsdays.org/speaking. The Velocity Europe CFP closes May 8th, and we’ll have a link in the show notes. And the discount code ADO2018 will get you 20% off lots of DevOps Days. 10% off ChefConf, 5% off GopherCon, some amount off all the Velocities. If you have an upcoming conference you’d like to see promoted on Arrested DevOps, you can fill out the handy form at arresteddevops.com/conf. Many thanks to our panelists, Jeff, Jordan, Eric, Lena, and Kyle. Also, thanks to the organizers of GOTO Chicago. Shout out to Adam for letting us host such a fun panel to close out the distributed systems track. Head over to arresteddevops.com/gotochicago2018 for this episode’s show notes. The site also has our newsletter, Patreon, all the Arrested DevOps stuff you could ever want. Visit arresteddevops.com/itunes and leave us a review in the iTunes store if you want to help other people find the podcast. I’m Bridget, @bridgetkromhout. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.
