Jeff: [00:00:00] Don’t worry, it’s fine. You’re fired.
Matty: Right, right.
Bridget: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout, and with me today, I’m Matt Stratton.
Matty: Today, we’re going to be talking about communication in the time of disaster. The show notes for this episode can be found at arresteddevops.com/disastercommunication. But before we get started, a word from the people who pay our bills. Arrested DevOps is brought to you by 10th Magnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. 10th Magnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/10thmagnitude.
Bridget: [00:01:04] This episode is brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 120 common infrastructure tools such as Chef, Docker, and AWS. So that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog.
Matty: This episode is sponsored by VictorOps. Built for modern incident management, VictorOps provides a unified platform for real-time alerting, collaboration, and documentation. Driven by your IT and DevOps system data, VictorOps helps you to respond to incidents more effectively so you can minimize downtime and make being on call suck less. Visit arresteddevops.com/victorops to schedule a demo or start your trial. Mention you heard about VictorOps on Arrested DevOps and you’ll be eligible for some sweet discounts too. Uh, yeah, so joining us today we’ve got, uh, Jeff Smith and Mark Umbriaco. So, uh, Jeff, can you tell us a little bit about yourself, a little quick background?
Jeff: [00:02:15] Sure, yeah.
Matty: Why we care to have you here?
Jeff: I’m not sure why you care, but I can tell you what I know. My name, Jeff Smith. I’m a production operations manager at Centro, a media services company here in Chicago. Prior to that, I was with Grubhub where I helped deliver the DevOps gospel and build out our SRE team. I’ve been in IT operations for about 15 years now and just generally love all things tech. Love new technology, love learning about how other people solve problems, and I’m hoping that’s why you brought me here, but I’m not sure.
Bridget: It is absolutely why we brought you here. That, and you had your amazing talk yesterday in my DevOps track. And you may have noticed that I’ve schemed to have every single person, with the exception of one who’s running a track today, every single person minus one from my track doing these podcasts.
Jeff: Yeah, that worked out pretty good, actually.
Bridget: Absolutely.
Matty: So, Mark, Besides the fact that you had the DevOps license plate before I did, what can you tell us a little bit about some of your background?
Mark: [00:03:16] Yeah, so I’m Mark Imbriaco. I work at Pivotal as of yesterday. My orientation was this conference. I’ve been in, much like Jeff, in the kind of IT and ops world for about 20 years. I’ve been lucky enough to work at a bunch of places doing interesting things and fixing things that they break all the time. GitHub, Heroku, DigitalOcean, places like that. So, I guess you want me here because I’m usually opinionated.
Bridget: And I think that you have a lot of opinions around how we communicate in the face of disaster, which is what we want to talk about here today. So, let’s start with a little background of Jeff’s talk. Can you give us a little bit of a picture for those who are not here, you know, at GOTO Chicago, which of course, it’s very sad. They should have been. Not everyone can be in Chicago at all times, for sure.
Jeff: They should be though.
Bridget: They should be. But for those who can’t be, give us a little bit of a picture of what your talk was about.
Jeff: Sure. So, I did a talk on an actual postmortem of an incident that we had at one of my unnamed companies and walked us through the incident process, exactly how alerts fired, what we learned when those alerts fired, and then how we sort of triage and troubleshoot incidents in distributed systems and in complex microservices architecture because you always hear about how great great microservices is gonna be and how it’s gonna be so easy to scale out, but then sometimes the problems don’t necessarily involve scaling out, and then suddenly you’ve got a whole host of problems that you need to figure out. So it was really a walkthrough and sort of a live postmortem of that particular incident.
Matty: [00:04:53] So I was gonna say, for, again, given that our listeners did not see your talk, what can you tell us a couple of the lessons learned? Like, what were some of the adaptations that you made to how you worked after this?
Jeff: Sure. Some of the big lessons learned were about context, right? We always talk about alerting on everything and monitoring and, you know, putting metrics behind everything, but sometimes that’s information overload. So you have to figure out how you’re going to frame the information that you have to tell a particular story specifically in the context or in the face of a disaster, because you don’t want to have to sort of put together these breadcrumbs while you’re going through an actual outage and an incident. One of the other things I think we learned is just about like how that information gets filtered in terms of when the state of the incident has changed. So I think Brian talked about this a little bit in his keynote today too. You have all of these alerts firing and that’s great. So then what do you immediately do? You silence them all, right? Because they’re annoying and they’re bugging you constantly. But what ends up happening is the context or the landscape of the incident could change. Your alerting might be in place, but then you don’t get it because you’ve shut it off assuming that everything is related to the initial incident, and then you end up fighting the wrong fire.
Bridget: [00:06:10] I think Brian also mentioned you find out to your dismay that getting some of your alerts to fire is dependent upon the very systems that are down.
Mark: Right. Yeah.
Jeff: We had that incident once where there was a billing error and they shut down our Azure account. Of course, the only alert was like, I can’t reach our website. I’m like, that’s funny.
Matty: The anecdotal story of years and years ago in one of my first IT jobs when I was annoyed when people would come by my desk. I basically was running tech for this small company of 20, 30 employees. And people would just come over and ask me to do a thing. And I said, whatever it is, I will only take requests via email. And they said, what if the problem is that I can’t get into my email? And I was like, yeah.
Bridget: Corner case. OK, so this is, I think, a really good framing for the discussion that I— full disclosure— dragged Imbriaco, a brand new coworker of mine, all the way to Chicago specifically because I really wanted to have his perspective on this. Because you’ve written some of the postmortem reports for a few of the places that everybody here has used and heard of. And I’m wondering if you can kind of respond to what you’re hearing from Jeff here.
Mark: [00:07:25] What do you think? I mean, there’s a lot there. I think the context thing is super important. And I think context kind of across the team. One of the things— so Brian talked about— The long outage we had at Heroku at one point, 67 hours. Now that was the kind of the last thing came back up after 67 hours, but still it’s a long time. So you start to think about adaptations for how you handle incidents. You’re not built for that most of the time. Usually companies you’re— most companies you’re at, you don’t even think about what happens if I’m in an outage that goes, you know, more than 12 hours. 12 hours, okay, fine, you, you know, everybody goes into superhero mode and you sort of plow through. But what happens if this goes 3 days, right? So we had to think about those things. We thought about things like making sure that we had shifts so that people were on call to follow the incident and to work on the incident throughout the lifecycle of it. Make sure that there was overlap so that the next person, we always had 2 people that were working the incident and their shifts were offset by 50%, right? So if it was an 8-hour shift, 4 hours into it somebody else would come in so that they could build up the context they need over that next 4 hours while the previous person was still there. Just so we’d have continuity through the context so that everybody sort of knew where we were in the incident and could communicate the current status with anybody else who needed to work on it or know where we were at.
Jeff: [00:08:36] And that’s usually something that I find that happens a lot in regards to what you’re saying about sort of the sleep management processes. If you have the same people or the same person that’s sort of like driving this incident through the entire lifecycle, chances are it’s because they’re storing all of the context, right? And that is a very dangerous place to be both for the organization but also for the individual because you’ve already got a tremendous amount of stress on you And then you don’t want to have to go through the stress of like, OK, I’ve got 7 hours of data that I now need to transfer to someone before I pass out.
Bridget: I mean, that’s actually a really interesting point too because I like to joke that you probably shouldn’t keep all of your documentation in the active working set, as it were. You perhaps need to write it to disk from time to time. And at Pivotal, we’re real big on pair programming, but you can’t pass all context orally at all times. But you’re mentioning that in the context of these incidents, the oral transfer of information, the having another person bring you up to speed does seem to be key. Like, kind of, can you kind of address, like, what is the distinction between the person on the job with you bringing you up to speed on the incident in progress versus any kind of written thing?
Mark: [00:09:49] Yeah, that’s great. I think, so for me, I think the— I think about context versus state, right? So if I’m joining an incident that’s in progress and I want to know, what’s up or down, I can look at the monitoring page. I can look for all the little red flaky dots. But if I wanna know, like, what’s the story that led us to this point, that’s what I need somebody else to tell me. And that somebody else probably hasn’t had time to write it down. That somebody else has been living it. They’ve been too busy trying to fight these fires and to leave a trail of breadcrumbs behind, but not necessarily in kind of a fully formed, fully kind of complete thought, right? They didn’t write in complete sentences. They left notes behind. They left sort of, you know, marks on trees as they went through the forest solving the problem so that somebody could come back behind them and follow the path and figure out what happened.
Jeff: And I think a common scenario happens is even when you’re trying to be diligent and take good notes, your notes are typically when you’re jotting down facts, right? And context a lot of times is also the theory that you’re tracking down at that particular moment because that theory or hypothesis that you did is part of the debugging process. And if you don’t have that state written somewhere, when you finally go away, who knows? You might have some tests that you were running or some backup validation script that executed. Now it finishes, you’re asleep, and I’m like, I don’t know what he was gonna do with with this, right? What, what were we trying to solve or prove here? And that’s often what gets lost, uh, in the written translation because you’re trying to keep a factual record as opposed to this is what I think might happen and then find out like, oh, this didn’t pan out, so now I’ve just wasted a page and a half.
Mark: [00:11:15] Even worse, you don’t write down the negative results, right? So the next person might follow exactly the same trail that you went down.
Matty: And I, I think one of the things that, that is, is somewhat challenging, like when we think about how we manage incident there’s the top-down part, there’s how as an organization we do that. But then I also— there’s going to be people, people probably in the audience, people listening, who are individual contributors who don’t get to make these decisions about how we do this, right? So I want to think a little bit about like maybe some suggestions from managing up, because I, I think we’ve, you know, all been in this scenario where you are on an incident and you have some management or someone who’s like What are you doing? What are you doing? What are you doing? What’s the action? What’s the action? And you’re like, hey, I need a minute to think, right? Like, you know, I specifically remember that like I had a manager before where it was a company owner, but I used to like have to have people run interference on them to say like, I need you to like just go pretend to do something so that Dan’s paying attention to you so I have 5 minutes to like hypothesize and think it out. So what are some things that you could think of, again, when from a managing up perspective, when you’re in a scenario where you’re on a bridge, you’ve got someone that’s 7 layers above you that’s just like, all I know is shit’s down, Jeff, what’s going on, what’s going on? You’re, you’re so busy having to provide updates that you can’t think.
Jeff: [00:12:40] Like, that was a common problem that we had at Grubhub during, during some major outages, because one of the big issues that we would have is we would have this common bridge And it would be business people and technical people. Business people need to get out the messaging because it’s a consumer platform. So we need to tell people what the status of their orders are, what we’re doing to work on it. And they want to know, where are we? And the question that always kills me is like, everything’s broke. I don’t know what’s going on. How long is it going to take for you to fix it?
Matty: Right?
Jeff: And you have to give a number, right? You’re just constantly pressured.
Matty: I’ll use the Scotty principle.
Jeff: 10.
Bridget: It will take 10.
Jeff: Right, 10.
Bridget: Yep. Do they have to be units?
Jeff: Honestly, they’ll probably take that answer and run with it. All right, we’re sending out a notification right now.
Matty: But they’ll take it as seconds.
Jeff: Right. But I think the key thing, though, is to sort of separate these sort of functions so that someone that is troubleshooting can be troubleshooting without necessarily having to give updates. So that sort of forces that person to give at least a periodic update to one particular person. And then that person becomes the parrot of the same message. Because how many times you’ve been on a conference call, it’s like, boop, boop, boop, Fred has joined. Hey, where are we at? Boop, boop, boop, Bob has joined. Hey, where we at? And you’re giving the same update over and over and over again.
Bridget: [00:13:52] Hashtag chat ops.
Jeff: Right. The other thing that helped really well with that is a running Google Doc. I think we stole that from the Google SRE handbook. But we would literally just update this document. When I first suggested it, people thought it was crazy. And then when people said, what’s the status? We said, well, if you look in the chat room, there’s a link to the doc that will get you up to speed. And it’s like, oh, OK, cool. Questions answered. So yeah, so separating those functions, I think, is big. And I think the other thing that I just lost my train of thought off, so I’m going to pass it to him.
Matty: The other thing is whatever Mark says.
Mark: I love what you said about separating the responsibility of communicating from the responsibility of doing in the moment. We did this at Heroku and GitHub and everywhere else I’ve worked as well, where we had sort of an incident command role. And their job isn’t to say, Jeff, go look at that thing. Their job is to— kind of capture the context for the global team and to help people. If Jeff says, I’m going to look at this, your job is to remember, OK, Jeff is looking at that. And somebody else says, I want to look at this. You’re like, no, Jeff’s looking at that. Maybe you can go look at something else. Or maybe you talk to Jeff. Maybe you can help him. I don’t know, whatever the context may be. And their role is to distill that context at whatever the right interval for your organization is to whoever’s responsible for doing the outward communication. And plot twist, that person shouldn’t be the one who’s communicating externally to your customers either. Because they have other things to do. Their job is just to pass the context on in a way that it can be externally communicated so that someone else can make the decision. If I’m the incident commander for an incident and GitHub is down, the last thing I want to think about is how I need to word the fact that we’re bringing a database back up to make people understand that we didn’t actually catastrophically lose data. And I’ve done that. I’ve been that person that had to make that message before. And it’s like, we’re recovering— no, we don’t want to say recovering. That makes them think about backups and shit.
Jeff: [00:15:36] People discount how difficult it is to craft a message, right? So when you say, like, we want periodic updates every 15 minutes, right? It’s gonna take me 10 to write the update, right? So there’s not— because you’re like, oh, write an update, and then you send the update, and it’s like, well, you didn’t give the proper context, or you’re talking to business users, you can’t say that. So that takes some cognitive energy to do that. I think the other thing that I wanna hit on that actually you reminded me of was that you, like, jumping in to help isn’t always helpful, right? If you haven’t been engaged specifically, if you’re going to look at something, make sure you check in with the incident commander or whoever’s running something, because you could be doing something that’s detrimental to a theory that they’re— or a debugging practice that they’re trying to track down.
Matty: Now, contrary to that, so one of the things we just talked about was, again, used a lot of like, well, our processes, we have an incident commander, we have all the That’s awesome. If you don’t, like, let’s say your organization is fucked, right? You don’t actually know how to do this right. So I’m thinking again about— because this happens a lot of times, we talk about how things would be great, and I’m like, I’m the sysadmin, right? Like, I— this is what’s happening. So that could be a way, like, again, to Jeff’s point, don’t just jump— again, the thing we naturally want to do is jump in and help with the problem solving. But one of the things is, I would think, if you have a peer that’s struggling, like, I know this is happening, I know Jeff is underwater, what I can do is maybe jump in and say, hey, can I firewall you? Can I help with the communication? And maybe we’ll pass that back and forth. But I think the thing that happens a lot is we hear these structural ways to do it. And I think it’s really important to talk about that because there’s ways, I mean, this doesn’t always completely work bottom up, but we also can’t just throw our hands up and say, our organization doesn’t know how to do incidents right. So I’m just screwed, right? I think there are things we can do. And then when those show success, like let’s say that happens, let’s say we did that, I saw Jeff was struggling, I jumped in to help be that firewall, then what I can do is go to my management and say, hey, you know what we did? Because of this, it made us more effective. Jeff was able to focus on the problem. Let’s see if we can make this something that we do more formally rather than I heard Marc Imbriaco talking on a podcast and he said it was a smart thing to do. Right. Exactly.
Jeff: [00:17:57] Oh, go ahead.
Bridget: Oh, I was going to say that the idea of, especially if you’re trying to start some sort of decent, non-terrible incident response in your organization, and maybe you don’t have a strong history or culture or practice of that, just the shared Google Doc where you’re updating the— your chat logs from the incident are not what you’re going to publish. Because even though they have timestamps in them, they also have a lot of rabbit holes that you’ve gone down that aren’t don’t turn out to be as germane to what you end up actually solving. And so, especially if you’re helping, but you’re not the person driving this, use what’s already in the chat logs and just start building something in the Google Doc of like, this is what happened when. Because if you start building that even before everything is done, as you’re building it, you might reflect upon it and go, wait a minute, did we miss this other thing with the database?
Jeff: Right. And it’s interesting because we talk about these roles like incident commander and all that stuff. And people say, oh, we don’t have a formalized structure. But in reality, you do, right? Because even in this process, the person that grabs the incident initially is all of those roles, right? They’re the communicator. They’re the scribe. They’re the incident commander. So you sort of intrinsically have these roles. And what Matt said is spot on, right? If you’re someone and you don’t know exactly what the process is, Go to that person as the incident commander and say, what do you need? What can I do to help? And it’s amazing how sometimes this person could be at wit’s end and they’re like, I don’t even know, right? Like, I have no idea. Are you testing a hypothesis at the moment? No, I’m completely stuck. And it’s like, okay, now we can work together and figure out what we need to do. Or maybe they say, you know, hey, I need you to track down this, or I need you to notify this, or I need you to page someone. You’d be amazing. It’s amazing how helpful that can actually be just to say, I need more resources, and I don’t have time to look at some stupid wiki doc to figure out who the dev on call is because they wouldn’t convince themselves to take a pager. So now I’m going through my own stuff.
Bridget: [00:19:57] I’m sorry.
Matty: This is a safe space. Yeah. Only 6,000 people are going to listen.
Mark: I think back to the managing up question, I think I can sympathize with the system admin who gets woken up in the middle of the night solving a problem and is getting harassed by people. What’s the status? Status? And there’s some simple things you can do here. You can, when people ask you that question, say, I don’t know. I’ll tell you something new in 20 minutes. And if I have something new before then, I’ll tell you. But if not, I’ll give you an update in 20 minutes about where we’re at.
Matty: Even if the update is nothing’s changed.
Mark: Even if the update is, I’ll tell you again in 20 minutes. And that works internally and externally. I mean, it’s frustrating being on the end of it, getting harassed with this question. But at the same time, think about yourself, right? Your home internet is down. You call the cable company and you say, How, how long? And they say it’s going to be 10. Yep. Yeah, like it’s about managing expectations. If they have at least the expectation that I’m going to get some more information— they don’t have— like, I don’t have anything to tell you right now. I don’t know. The answer right now is I don’t know. I’ll know more in 20 minutes, maybe. And if I don’t, I’ll tell you. And but you’ll at least hear from me in the next 20 minutes. So setting a cadence so that people know what to expect, even if you don’t have the answers, but they know when they’re going to get more information. They know that you’re still working on the problem. They know that you haven’t just given up and gone to bed. I think that just something as simple as that can buy you some slack to actually work on the problem.
Matty: [00:21:17] Acknowledgment is huge for people feeling reassured, right? Like, that’s the thing. Like you said, I have had that experience, just like you said, with customer service, which is the, hey, you know, I’m having a problem with whatever thing. You know what? We don’t know how long it’s going to take, but I will call you back in half an hour with an update. And they can call me back in half an hour and say, just like you said, still broke, call you another half an hour and I will say, thank you, I love you, I will be your customer for years, right? And they didn’t fix anything, but I feel good about it.
Mark: But I do want to know, it’s still broke and we’re still working on it.
Matty: And we’re still working on it.
Mark: That’s the part that, like, if you don’t hear anything, you wonder, right? Are they still working on this? You get this from vendors when you’re in IT. You get this from, you know, from your own vendors when you’re at home. Not knowing is worse than getting bad news.
Jeff: So I have a question for you guys, actually. So there’s sort of like multi-phases to the incident, right? And there’s a part where you’ve got an outage, right? And then there’s that part where either you recognize or you concede that you actually have a disaster, right? How do you— what sort of structures do you have in place to sort of denote that mark, right? Because if you tell me, you know, I have an outage, we’ll give you an update in 20 minutes, right? In 20 minutes, I’m expecting to hear progress, right? But if you tell me, We’ve got a disaster. We don’t know when it’s going to be back. I may stop trying Hulu for the next 8 hours versus every 15 minutes.
Matty: [00:22:42] I believe Mark was on our episode about this. If you go to arrestodevops.com/disasters, we had an episode about disasters. But let’s keep talking. But yeah, how do you say this is now— this has gone beyond an outage and now it’s a disaster?
Mark: Yeah, I mean, I think for me, it’s a case of I don’t know, but I know it when I see it.
Bridget: Right.
Mark: It’s like you give the 20-minute update, and you give another one. You’re like, I’m getting ready to tell them that I’m giving them another update in 20 minutes, but I don’t honestly expect to know anything in 20 minutes. So maybe I’m going to say, you know what? We don’t have anything new. We’ll tell you in an hour, or we’ll tell you in 2 hours. And if we have something before that, we’ll tell you immediately. But at the worst case, you can expect to hear from us in 2 hours. And if you don’t hear from us in 2 hours, we’re still working on it. I promise. But we just don’t have anything to tell you.
Matty: So was it in your Heroku example?
Mark: I mean, that’s— we totally did that.
Matty: Yeah, we spread out. How did you, like, was there a point when you felt, like you said, you didn’t, you can’t describe, but you know when you see it, but like, when did you feel that in your gut?
Mark: So the Heroku outage was complex. The first part of it, so EBS went completely dark in US East in 2011. The control plane had a cascading failure across multiple AZs in the region. It just basically blew up. So we lost everything, right? The biggest problem that we had, the extended— so the first part of it, you couldn’t run Heroku apps. So you couldn’t launch new apps. You couldn’t— if a dyno idled out, you couldn’t start it back up again. So we had a bunch of issues there. And that got resolved in something like 8 hours. And that was just kind of a normal outage. The longer part was we also hosted 200,000 Postgres databases. And some of those EBS volumes took up to 67 hours to come back or not. Or they either took 67 hours to come back or for AWS to give up on trying to bring them back. And that’s the part where you’re like, we’ve restored the main functionality, and here’s the long tail, and we don’t know how long it is. And we’ll provide you updates as we have them, and we’ll be working directly with the affected customers. But, you know, it’s, again, it’s so case by case. You sort of have to look at the situation and say, when do I reasonably think I’ll have an update? And maybe massage that a little bit and say, how long can I reasonably expect them to believe that I’m still working on it if I haven’t given them an update.
Jeff: [00:24:48] I was fortunate to mostly work in consumer-facing applications. And even at the media services agency, it’s not on demand. But we always had a scenario where it was like each particular visitor had a specific lifecycle. So we could sort of time that lifecycle and know that if we haven’t corrected it by this point, then we might as well just consider it a disaster, let everyone know that pizza you ordered is not coming. It’s just not going to show up. So then from that point on, we call it a disaster and work from there. But I was just curious how— from organizations?
Bridget: I mean, I would say like with some outages that I’ve been involved in, you know immediately. Like when I got the 3:00 AM call from the developer who was on call, and I was not on call, but I got the 3:00 AM call from him that was like, the HBase cluster is gone. Like Amazon says it’s terminated. I was like, that’s not good. That’s where a whole bunch of important data that keeps our startup going is. And like that, Took days to resolve and lots of fun with the backups that were broken and sadness and tears. And I have a SysAdvent blog post from, I think, like 2013 ranting about that. But you can see the details in the show notes. But the TLDR on that is like, I knew the second I got that phone call that we were fucked. And I was like, our startup may or may not be done at this point. I guess we’ll find out. Spoiler alert, we did not in fact go out of business for that. And we eventually later got acquired for some amount of money. But sometimes you know immediately.
Jeff: [00:26:22] And I think the key takeaway with these stories too is the cloud is just someone else’s computer. So it’s funny because I constantly have these conversations about, well, Amazon’s going to do this so much better than we are. And it’s like, OK, but that still doesn’t absolve you of your responsibilities of making sure you recover from an Amazon failure. And what is the cost of being able to survive a long extended Amazon failure? Which is not funny.
Matty: We did an episode called Who Owns Your Availability? You may want to check it out.
Bridget: We had Charity Majors and Pete Cheslock on talking about that. And that’s— Jeff has a really good point, which is all of us are probably, if we’re at all ops inclined, and I’m guessing everyone who’s sitting in the room who decided to come to an episode all about communicating with disasters in the face of disaster is somewhat ops inclined. But All of us probably are professional, you know, paranoids. Like, we think about the— we go down the rabbit holes of the worst-case scenario all the time, possibly driving our loved ones completely batty. Yes, Joe, I see you smiling. And I think that it’s important to, you know, rein in our paranoia a little, but also do a little bit of planning for how— not only how are we going to make sure our backups are actually working. Spoiler alert, if you haven’t checked them, you have Heisenberg’s, like, Backups, it’s like they may or may not work. But sorry, not Heisenberg, Schrödinger’s. If you have not checked them, you have Schrödinger’s backups.
Matty: [00:27:48] Heisenberg was actually in the last episode when we talked about measuring.
Bridget: Exactly. We talked about changing things when you’re observing them. Right. But not only do you need to make sure your, say, backups work, but I would love to hear what you two think about making sure that you have some kind of plan in place for how you’re going to communicate, since we’re talking about the communications here, both internally and externally? Not just like we have a person who’s assigned to communicate, but maybe start with Mark. How do you make a plan for how are you going to communicate around this sort of thing?
Mark: For me, it’s really simple. Every time I— whenever I join a new company, it’s one of the first things I look at is how do we communicate with people when there’s problems? Because it’s all about managing expectations. And I think there’s a handful of things you want to keep in mind. So frequency of updates, The tone that you’re going to use. So think about who your audience is. If you’re Heroku and your audience is very technical, they want technical details. If you work at Basecamp and your audience is small businesses that may or may not be very technical, you have a different messaging style. So you need to be explicit about what your style is and maybe even have some canned responses. So frequency, the communication mechanisms you use, The TL;DR is you want to build a plan that’s prescriptive enough so that you don’t have to think about it when you’re working the problem. You shouldn’t have to make value judgments about when I should be updating or what the tone of the message should be or how I should think about communicating this or how transparent can I be. Plot twist, you should be completely transparent because the last thing you want to do is mislead people and then find out later that you undersold the problem. So just be prescriptive, and it can be as lightweight or as complex as necessary, but usually less than a page.
Matty: [00:29:35] And it has to be, because the thing is, with some things we develop muscle memory where we’re just like, I do this enough. Hopefully your disaster and incident issue is not so frequent that you have muscle memory of knowing exactly what to do every time. But like, yeah, during an incident you don’t want to be spending any of your brain power thinking about crafting the message. You want to use all of your cycles to solve the problem. Now, and I know, again, this might sound to some folks like this is, we talk about this all the time or whatever, but I think it, I’d like to think about, like, to Bridget’s point, she’s like, your backups are not useful if you didn’t test them. So like we do DR tests, that’s a thing that people do, whether you do them well or not. But I would like, what are some things people can do to like, ’cause you don’t wanna have the first time you test your incident process be when shit’s on fire, yo, right? Like, so what are some examples? I know like a lot of people probably have heard this.
Bridget: I think Jeff talked about that a little bit too.
Jeff: [00:30:36] Um, yeah, so the best way to do it, in my opinion, is, is to sort of like simulate these sort of incidents internally. And it’s— this is actually a talk that I’ve been sort of ideating on myself, is like, you know, when we’re talking about a system, whether it be a computer system or a, you know, incident management system or whatever, we always have these like the environment, and then we have the things that are within the system. And we tend to discount everything that’s in the environment because it might happen. We need to start pulling those things into the system. So incident creation, right? Like incident creation is something that happens out here, but if you pull it into the system and you’re sort of forcing it to happen on a simulated basis, it forces you to sort of test these muscles and sort of get into that mode. So, you know, Chaos Monkey, right? Exactly. Chaos Monkey is the canonical example, right? It’s like something just blew up now. With Chaos Monkey for incident management, hopefully it’s not that bad because you’ve already designed a system for that. But if you just have someone come in and just say, I just socked Cassandra in the face, right? Who’s going to do what? Where’s the communication plan? They would do Hot Shot. Yeah. Yeah. We would do these things, Failure Fridays, War Room Wednesdays, where we would basically simulate these incidents and see what happens. Do we get alerting? If we didn’t get alerting, what’s the plan to put it in place? What was the communication path? How did we determine what the impact was? It’s even more fun for bonus money if you have someone that’s not part of the exercise killing stuff, or at the very least saying what’s dying, because then it’s very easy to map out the roadmap when you’re like, all right, War Room Wednesday is next week, so let’s just pre-generate our templates now.
Bridget: [00:32:10] I should, by the way, for our listeners who are like, this sounds kind of exciting, I should do this to add spice to my work life, make sure you clear this with people before you just start taking production down for funsies.
Matty: And think about, like, from a test perspective, War Room Wednesdays, Failure Fridays, those are— think about tests that can be mocked or stubbed. You don’t actually have to really break a thing, right? Think about, there’s a great Parks and Rec episode where they had to simulate bird flu or something like that. And they’re like, okay, boom, you’re dead. Boom, you have the flu. Boom, the phones are down. You can do that, right? And do that first before— like, maybe you can get to the place where you’re like, we’re really going to actually take down the main database cluster and respond to it quickly and whatever. Yeah, not yet. You’re not— don’t start with that. Make your database cluster a sticky note on the board and somebody gets to go and pull it off the board and throw it on the floor. And that is just as good as actually breaking the database.
Bridget: Well, it’s a good start.
Mark: [00:33:10] Well, for better than nothing, right?
Jeff: I think the interesting thing with microservices too that we found that was sort of a side effect of this is that You can isolate a node out of 5 nodes or whatever, and you can basically simulate the failure just by introducing network problems, right? So there was a great tool called NetImpair that we extended to sort of fit into our environment, but it allows you to add jitter, inject latency, cut off communications altogether, and that’s just as good as the system disappearing from that node’s perspective, right? So you can isolate a single node and inject a bunch of failure and just just sort of see what happens from there. So, uh, net impair was a great thing that worked for us.
Matty: Again, do this on paper first. You know how like we tell you before you go and like build your crazy rally scrum board online thing or whatever, just put some freaking stickies on a wall? Do that first, right? Do all this stuff, like get good at it on paper when it’s safe and you don’t— and, and you’re focused just on what you’re trying to do. Not only is it safe, but you can iterate faster ’Cause you’re just trying it and then you can—
Jeff: [00:34:13] Right, ’cause everyone has restarted a service that wasn’t supposed to do anything and ended the world.
Bridget: Yeah. I was about to say, like, sometimes if you accidentally, you know, step down the wrong, you know, leader node or you, you know, terminate the wrong instance, not the one that you intended to, just the other one, you have that oh shit moment where you’re like, Ctrl+C, Ctrl+C, and it doesn’t Ctrl+C fast enough.
Matty: I know there are a couple folks folks from Allstate in the room. So this is one of my favorite, uh, anecdotal stories about my oh shit moment, which happened when I was at Allstate. So back in the olden days of hardware, um, my main job in that was I was just going in— they’d keep rebuilding, you know, 1U servers for, you know, dev environments. And so what Matt would do is go in there, turn the box off, drop in the Smart Start CD, sit in the freezing data center and read a book, right? Good money if you can get it. So the other thing though is like the way servers were named was in this completely esoteric, you know, you need a Rosetta Stone to figure out what’s what, right? Because, you know, naming standards are fun. And so I went in and I’m like, oh, this is the server they want me to rebuild. Boom, powered off, you know, hit the KVM. I’m like, box still up.
Bridget: [00:35:27] Like, has your phone started going off yet?
Matty: Do you have signal in the data center? Turn it on. Walk out of the data center, stand in the cafeteria next to the door. Nobody comes running down in the next 15 minutes. I guess everything’s fine.
Bridget: Oh my God. Yeah. So what you’re saying is /etc/rc.d on that server was like set up correctly and everything came back up?
Matty: Or it was a dev box and it was just whatevs, right? Like, but that’s the thing. It’s like in these racks, you know, I don’t know what I turned off because it was called X52962Z and I’m like, There is nothing—
Jeff: there’s no way to explain that feeling when you hit enter and you immediately get that feedback and you’re like, it’s gonna be me. Yes, yes, exactly. Yeah, you’re like, did I just do that?
Bridget: And that actually— that takes— you’re quoting from Brian Cantrell this morning, you know, please don’t be me, please don’t be me. And I actually think that there were so many rabbit holes that we went down in our discussion with him earlier that we didn’t get to this one. But there is something important, not necessarily about the external-facing communication, though sure, redact to protect the names of the innocent and guilty alike. But there’s something important about making sure that your reaction inside the company is not going to be, oh shit, was it me? Are people— am I fired? Like, is everything done for me because I did the wrong thing? Yeah, right.
Jeff: [00:36:48] Well, I actually just experienced that a little bit just this week. You know, we’re going through doing our AWS cost savings. Oh, look at this environment. Nobody knows what this is. No one’s using this. 67 terabytes of EBS volumes with logs, just logs. Let’s ask the business. Do we need this stuff? No, we don’t need it. Everyone says we don’t need it. Delete. Oh, and we’re shutting down all these instances too. Hey, there’s a bunch of shit that’s broken. Really? Yeah.
Matty: No, that’s one way to find out who’s using it.
Jeff: Yeah. So the guy that did it is like, you know, he’s paranoid. He’s like, oh God, I’m just waiting to figure out how soon they’re going to walk me out. And it’s like, no, that’s not the deal. You know, you did exactly what we asked you to do. So it’s just— you can never assume that people know that they’re in the clear, that they did the right thing, right? You sort of have to reinforce it and let them know, like, look, this was not your fault. Don’t worry about it.
Matty: Well, first, you have to have that be actually true. Absolutely. Right, right. So there’s that.
Jeff: [00:37:50] Don’t worry, it’s fine, you’re fired. Right, right.
Matty: And that’s harder than you think, right? I mean, so the first thing is, first of all, even if it’s true, nobody believes it till it happens. And I remember, so there’s an old episode of the Food Fight show when they had some folks from GitHub on, and I don’t remember who it was, but someone was talking about, they were talking about blamelessness and again, saying making the mistake. And whoever it was, might have been you, Mark, I don’t know. Maybe. I don’t know. But whoever was on was basically like, I kept hearing him say, you won’t get fired, you can break shit, it’s okay. And until they— he’s like, until I broke something and didn’t get fired, I didn’t believe it. And so that’s— so you have to reinforce that. And like a couple of things that I like to keep in mind. So when we had John Cowie from Etsy was on our show, a long time ago, and he had a great comment where he said, how amazing is it when the only thing that happens when you make a mistake is you learn something new? And I’m fond of saying too that, like, right, and this again, people have to not have that don’t let it be me, is if people know that they’re going to be punished for making a mistake, it does not make them make fewer mistakes. It makes them hide the mistakes. It makes them become subject matter experts in hiding mistakes, and now you’re well and truly fucked. Out because you have no idea what’s going on in your environment.
Bridget: [00:39:13] And so the people who would have the most insight into it have the most motivation to not tell you, right?
Matty: And so now we’re gonna— I mean, because we gotta talk about Etsy some more. It’s been a while. It’s been at least 30 seconds. But the 3-armed sweater, right? They celebrate failure. So what that does is it just says it’s totally okay, right? It doesn’t reinforce people doing stupid stuff. It just says you’re okay. But I still do believe You can be at a company that’s like, we’ll give you an award, we’ll do all that, and you still won’t believe it in your gut till you actually break something and don’t get fired.
Mark: You’re always going to have that emotional response, even if you know, right? So you make a mistake, you’re like, oh my God, what did I just do? Yeah. Did I just break everything? And when you can move beyond that to— when you can move from— and I say the please don’t be me thing because we can all feel it, and it’s funny. But when you can get past that from a please be me, because if it’s me, I know what the hell’s wrong, Like, if it’s me, did I just break that? Please let that be me. Please let that be the thing that I just did, because I just— I know exactly what the problem is and I can fix it. I don’t have to go look now. I know. And that— if you can get to that place, it’s really empowering. But it’s not all companies are there, and it takes a long time to get there sometimes. And people are kind of motivated for the wrong reasons, and they are incented for the wrong things far too often, and it makes it really difficult.
Jeff: [00:40:27] And if you’re a technical leader in your organization— and I don’t even mean like through the hierarchy, but like you’re one of the senior people or one of the strong resources, you can give people a lot of air cover by publicly owning your own mistakes, right? When you come forward and say, yep, I screwed this up, right? I know you ain’t gonna fire me though, right? You ain’t gonna fire me. That gives air cover for the people that maybe aren’t so senior so that when they make a mistake, they’re like, oh, well, if Jeff can delete production by accident, surely I’m not gonna get in trouble for accidentally deleting 67 terabytes worth of logs, right?
Mark: Well, at GitHub, it’s easy, right? Because one of One of the co-founders, there’s like, if you look back in GitHub ancient blog history, there’s a post from one of the co-founders about deleting the production database because they thought it was a test database. And it’s like, if he can talk publicly about it, what makes you think you have a problem breaking something?
Bridget: And even with believing that I worked someplace that understood all of these things, I still vividly remember that sinking feeling when I walked into work in 2013 at that startup, the day after the 3 AM call from the developer where we figured out that there was an AWS bug that terminated all of our Hadoop instances on EMR, which is a thing that happens.
Mark: [00:41:33] I think you’re always going to have that worry. And all my customer databases were on it.
Bridget: And I was like, I didn’t know this could happen. I know it can happen now. But I remember walking into work that morning and thinking, I’m probably fired because I probably should have anticipated this and made this not happen.
Mark: I think we’re always going to have that sort of fear, right? When something bad happens, you’re going to have that fear. But if the organization— if you have the comfort that the fear is driven by, did I let my customers down versus am I going to lose my job? You’re in a good place.
Bridget: Or, well, I mean, honestly, like, my fear was perhaps I’ve just tanked our startup.
Mark: Exactly. That’s not a great plan. Absolutely. No, I’ve lived in that. It’s, it’s, was this a company-killing event? And you’re— and that’s terrifying. And there’s no way to get around that. You just have to live through it.
Matty: Your, your, your, your fear is that you did something that was impactful, not that you’re afraid of being punished. Exactly right. Because that’s now you’re actually thinking about the organization. You’re thinking about your customers. It’s, it’s versus— so that, that is a shift in mindset, right? You’re not going to stop being upset at yourself for making mistakes. That’s never going to happen. In fact, to be quite honest, if you’re in a place where you’re like, well, made a mistake, who cares, then maybe I don’t really want you running my systems because I want you to like not, like, not care if you make mistakes.
Bridget: [00:42:49] Let’s not be cavalier about our jobs.
Matty: Yeah, but we should care because we we’re impactful to the larger thing, not because I want to protect myself from being spanked, grounded, or punished, or whatever.
Jeff: I think Bridget touched on something too that’s really important, especially as things get more and more complex, is that idea that you are on the hook for not knowing something about this extremely complicated system that led to an incident or an outage. And that can really kill you. Like, oh, man. How did I not know that there was a separate thread that could die that was responsible for closing socket connections buried within this, you know, 5 million line job?
Matty: How many times have you been somewhere where you have an outage, right? And it’s because, oh, XYZ thing happened. And then your CTO comes to you and says, well, why weren’t you monitoring for that? And you’re like, well, I will now.
Bridget: Well, and then you end up with a whole bunch of alerts for the thing that broke last time, which may or may not ever happen again or be actionable. Or they could be actionable.
Matty: [00:43:50] But the thing was, we’re like, we didn’t know that could happen. That’s why.
Bridget: I will use my time travel powers to go and set up that Nagios alert.
Matty: I say the first thing we have to do is invent a time machine.
Jeff: You’re always fighting yesterday’s war, right? And it’s never prevalent than when you’ve got to go to the airport and take your shoes off.
Mark: The good news there is there’s lots of research and science to back us up on that one, right? There’s a ton of research in this area. AllSpa has written about it a ton. Richard Cook and David Woods have written about this a lot. The way that complex systems fail is fundamentally unknowable. So don’t kick yourself for not knowing that these 2 things could interact, or these 15 things could interact in this weird way, because you couldn’t have known. It wasn’t possible for you to know. And if you think it was, that’s just you Monday morning quarterbacking yourself. That’s not a— don’t give yourself a pass to be lazy, but just realize that complex failures are complex. Give yourself a pass to be human.
Bridget: Exactly. And I think that’s a really good point. And I know we’re almost out of time, but remember, we set the timer to 45 minutes this time, not 50. So we’re actually okay.
Matty: You did now just use up 30 seconds of time explaining that.
Bridget: [00:44:54] It’s probably fine.
Mark: We don’t blame here, Matt.
Matty: The fact that— This is a blameful podcast. Do not listen to this show.
Bridget: And that’s exactly where I was going is having— when you’re doing this communication outwards about incidents, Having the culture of blamelessness inside your organization. Since you brought up Etsy, I will mention that Tim Gross, who is in the room and also was on our podcast earlier, sent out an email when he was a director of operations at Drama Fever quoting AllSpa, quoting Etsy’s principles of blamelessness when talking about one of our outages. And I feel like it’s really important to make sure that anywhere in your organization that you can influence that, putting that communication out makes it so much safer for everyone who wants to tell the truth about something that went wrong. Like, if assuming it’s true, like, make sure that’s communicated really well in your org.
Jeff: So one thing I kind of wanted to circle back on too in regards to like communicating incident management, something that worked for us, it’s a little people power heavy, but when you need to communicate an outage, one thing that we had done was like we’ve got the one person sort of scribing everything, and they give it out to key contacts throughout the organization that are then going to disseminate it. And those contacts are responsible for essentially translating it into their audience’s dialect so that you don’t have to write one large generic message and you’re not over-inundating people with super technical jargon. So for the devs, they get something from a developer dialect. Business gets something from a business dialect. Marketing has something that they are going to put out publicly. And that worked out pretty well for us.
Bridget: [00:46:31] And I think that’s another really good point. I feel like we could talk about this forever. But since we are just about out of time, I would love to have our guests give a final summing, final sum up of shit’s broke, yo. How should you focus and shape your communications? Start with Mark.
Mark: Yeah, I guess the thing I’ll leave you with is the thing that I always tell people about when writing postmortems. I kind of have a formula for writing a public postmortem. These are specifically for outward-facing audiences. And there’s 3 things that if you do, you’re going to get a good result. The first one is apologize and mean it. The second is demonstrate a thorough understanding of what happened. And the third is describe what you’re going to do to— and don’t say that we’re going to take these steps so this problem never happens again. Say, we’re going to do this and we think it’ll reduce the likelihood of this kind of thing happening again. But if you do those 3 things, you can rebuild confidence to a level that was higher than what you had before the outage. And I’ve seen this over and over again. Just by being transparent, being vulnerable, being honest, and taking responsibility and demonstrating that you understand what happened and that you’re doing the right things, it’s amazing how powerful that is.
Jeff: [00:47:40] Yeah, I was going to say something along the same lines. Like, always try to get better. Always bring in other stakeholders, right? The people that you’re communicating to, they’re part of the incident process too. Bring them in and ask them, you know, what did you guys need from us that you didn’t get this time? Every incident is an opportunity to learn. I think the other thing is like when you’re, when you’re coming up with these action items to sort of resolve this incident, sometimes there are difficult problems. Remember, there’s really 3 axes that you can sort of try to address it, right? How severe is the incident when it actually occurs? Can we make this less impactful if we can’t actually solve it? What is the likelihood that it’s going to occur? Can we eliminate it from occurring altogether? And if you can’t do those 2, what is the likely— or in addition to, can we get better at detecting this when it does occur so that we can action on it faster?
Bridget: Yeah, absolutely. 100% true on all accounts.
Matty: Great. Well, if you head over to arresteddevops.com/disastercommunication, we’ll have the show notes from this episode. Episode, which will include links to some of the episodes we’ve talked about and some other things that were mentioned that we’ll figure out when we write the show notes.
Bridget: [00:48:48] Woods and Cook. And this is going to be one full of homework for us. Yes.
Matty: Or possibly for Joe. We’re going to start writing better show notes, I promise. Or at least better descriptions of the show. You need a scribe. We do. Don’t get me started. But also, if you go to our website at arresteddevops.com, You can sign up for our newsletter, The Banana Stand, so you can keep up to date with when we have new episodes. You can support us on Patreon. You can follow us on Twitter. We’re on Instagram, which is a thing, I guess. Find us in the iTunes Store and leave us a review. That helps people find our podcast so they can also listen to it, and that would be amazing. And we love our listeners. Absolutely.
Bridget: And thank you so much, Jeff and Mark, for joining us. A lot of fun, yeah. This is great. So I’m Bridget at Bridget Cromwell. And I’m Matt at Matt Stratton. We’re Arrested DevOps.
Matty: And remember, there’s always DevOps in the banana stand.
