Disasters!

Matty: [00:00:08] Welcome to Arrested DevOps, episode 37, Disasters. I’m your co-host, Matt Stratton, @MattStratton on Twitter.

Trevor: I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.

Bridget: And I’m your co-host, Bridget Kromhout, @bridgetkromhout on Twitter.

Trevor: Arrested DevOps is brought to you by 10th Magnitude, A cloud services company that figures if you’re listening to this podcast, you must be pretty cool. You can find out about joining our cloud services team at arresteddevops.com/10tmagnitude.

Matty: This episode is also sponsored by VictorOps. From initial alarm to final retrospective, the mission at VictorOps is to make on-calls suck less. Easily integrate with your existing monitoring systems and manage on-call schedules with rules for intelligent routing. In the live infrastructure timeline, get real-time context and see annotated alarms with resolution documentation. And when you’re in the firefight, collaboratively troubleshoot using native chat or bidirectional integrations with your favorite chat clients. Visit arresteddevops.com/victorops and sign up for a 14-day free trial to see how they’re making on-call suck less.

Trevor: [00:01:19] Today we’re going to be talking about disasters, how we communicate, how we recover. We’ve got 2 great guests with expertise in this area. First, Stephanie Van Dyk, who is an SRE at this Google company that some of you might have heard of that company. And also has worked on that— was it healthcare.gov? Another thing some of you may have heard of that was the most successful website in the history of the US government. Wait, no, that’s not right.

Stephanie: I think that’s an accurate summary. It’s true, I’ve worked on a broken website that’s a little famous. These days I work on stuff that works pretty well, I think. So that’s not quite as relevant to this conversation. But today I’ll be talking about disasters from my perspective, from my work on healthcare.gov.

Bridget: All right, thanks, Stephanie. Joining us also is Mark Imbriaco. Lifted entirely from his Twitter bio, it says co-founder and CEO at operable.inc, and previously, of course, DigitalOcean, GitHub, LivingSocial, Heroku, 37signals. If you’ve ever read a really good incident report, there’s a decent chance Mark wrote it. So Mark, I’m going to put you on the spot and ask you what this Operable thing is all about.

Mark: [00:02:37] So Operable is trying to help people that are kind of on the pointy end to deal with incidents, right? We want to build some tools to make it easier to collaborate around fixing problems when you have problems. There’s a lot of tools today to do things like wake you up and alert you when there’s a problem, but we think there’s a lot of room to help people actually solve problems, so that’s what we’re working on.

Matty: Those of you who are just listening to the audio and would have missed it, but Mark very nicely— just for me, he said— put his old DevOps license plate in frame. Behind him, probably mostly to point out that he did it before it was hip and cool. He’s moved on now. Now he has the license plate, what, microservices? I don’t know how you would fit that in the plate. An emoji of a whale.

Mark: That one’s not very micro.

Bridget: I’m really excited to talk to both of you because I know I’ve definitely followed along with the healthcare.gov saga. I think that was the first Thanksgiving that my family ever wanted to talk about web performance and operations. Really exciting. I mean, they wanted to talk about it in the context of politics, which was maybe a little less exciting, but that was pretty cool. Can you give us a little bit of just perspective? And, you know, everybody has heard the stories that were on CNN, but can you give us a little bit of perspective of what it was like to be dealing with that actual problem, Stephanie?

Stephanie: [00:03:58] So I remember the weeks leading up to that pretty good because I had a lot of conversations sort of similar. I listened to a couple podcasts where they’d bring on like a token software engineer who would like explain to the like political podcasters what exactly went wrong. And I was listening, I was like, that seems reasonable, not really understanding that I’d get to find out. And in fact, like maybe 2 nights before I left, I was in a bar with some friends and we were, I don’t know, just whining about stuff. And we’re like, well, our lives could be a lot worse. We could be working on healthcare.gov right now. And we all laughed. And so a few days later, I flew out to do that. And I think my experience on healthcare.gov was perhaps pretty unique as far as disasters go, but I think there’s a lot to learn there. We worked very hard and very long hours, and I wouldn’t necessarily recommend that for anybody else. We are also in the fortunate position of having a lot of authority, which is a thing that I think is somewhat important if you’re hoping to fix a disaster. There’s a lot of problems to solve and you don’t want any of your additional problems to be like, but who gave you permission to do that? You want everyone to be on board if you’re going in to fix something.

Bridget: [00:05:18] No, that makes a ton of sense. And I think that’s— it’s also a really good point when you say that not all disasters are created equal. Like maybe we should take a step back and think, okay, classic disaster, everyone can point to healthcare.gov, but I know, Imbriaco, you’ve been involved in a lot of different kinds of disasters. Like what would you say, what would both of you say, what are the ingredients that make something a disaster?

Mark: Yeah, I mean, I think healthcare.gov is something that I’m really excited to talk to Stephanie about it because it’s not something that I have experience with, something that’s that long-running, right? I’m used to I’m used to catastrophic problems that last for a few hours at most, or in the really bad case, you know, maybe it lasts for 2 or 3 or 4 days, not something that goes on for weeks or months. So that’s a really different perspective than what I have. So I’m super interested to hear about that. You know, it’s really interesting, the sort of the highs and lows that happen throughout a disaster. I’m sure they’re a little bit different in the shorter-term things than they are in the long ones. There’s still highs and lows, but I imagine they’re a little sharper and they come on more quickly or go away more quickly when, you know, you’ve got a problem that you can fix in an hour versus a problem that you can fix— you’re like, okay, well, I’m gonna make this change and it might help in 3 or 4 days.

Matty: [00:06:34] Yeah, I was thinking, and I don’t know why this didn’t pop into my head until just now, but this episode— there’s been something about when we’ve been planning for this episode that’s been in the back of my head that I’m like, this reminds me of something. And I finally nailed it. And it’s— there was an episode of This American Life that was about fiascos. And it’s an actual— it’s a very funny episode. We’ll put it in the show notes. And it’s about this community theater putting on a play of Peter Pan. And it’s just, you know, again, from one thing to another. And I, you know, I’m not going to spoil it. You should listen to it. But there’s a— I think there’s the disasters like, Mark, like you talked about, which is sort of like when it’s a thing that happens that’s maybe localized to one type of scenario. Then there is— and again, this will, you know, kind of thinking about the unraveling that can happen. And that’s where, you know, what happens in this fiasco of this play of Peter Pan. It’s one thing after another. At one point, you know, Captain Hook’s hook goes flying off into the audience and they’re trying to figure it out. And someone falls from the rafters and the fire, you know, like the local bomb shelter alarms are going off and all these things. And There’s a quote that’s actually from the episode that I’d like to kind of think about when we think about these longer, these bigger disasters that are more than just kind of an outage that may be far-reaching. And they said, you know, one ingredient of many fiascos is that great, massive, heart-wrenching chaos and failure are more likely to occur when great ambition has come into play, when plans are big, expectations are great, and hopes are at their highest.

Bridget: [00:08:08] Stephanie, does that sound like any websites that you’ve ever worked on?

Stephanie: That’s really interesting. Before this, I was thinking, like, in my mind that a disaster was maybe like a collective noun for outages, like a disaster of outages. But I think that maybe this quote—

Matty: Is that the plural of outages?

Stephanie: Yeah, yeah, exactly.

Matty: There’s a disaster of outages. I love it.

Stephanie: But I think that you’re probably right, and I love that episode of This American Life, by the way. It’s super funny. But I think that you’re sort of right in the sense that in order for something to be a disaster, the stakes have to be quite high, I think. You can think of like everyone has done something like— you can think of like the first like little web server you ever ran and the first time it went down and you didn’t notice for a couple of days because like whatever you were just starting out. Like, that’s not a disaster even though it was like a long-term prolonged outage because nobody noticed. And so I think there’s something sort of true about that, that there has to be something where the stakes are really high. Because an outage that you find and you fix and you write a postmortem and everyone learns something and the users all like get over their hurt, that’s not a disaster, that’s just life.

Bridget: [00:09:25] So, Mark, you said that you’ve obviously— I’ve read some of your postmortem reports. You know, I’ve used various services that you’ve worked on and written postmortem reports for that have been really great from an end-user point of view or the customer point of view. But would you say that you’ve experienced some of the disasters like Stephanie is describing? Like, what are some of the things that you would consider to be in that category of like heart-wrenching chaos and failure?

Mark: Yeah, I definitely have. I would say one that comes to mind is sort of the first postmortem that really got a lot of attention that I wrote was the one at Heroku when Amazon had the first EBS apocalypse, and EBS went away for 3 days for some of our database customers. The gut-wrenching terror there is, are we going to get our customers’ data back, or is it just gone? As an ops person, there’s almost nothing more terrifying than losing data, especially when, you know, it’s like, okay, we’re losing data, the people that we use to store our data and our backups are having problems, we’re really screwed. We have backups, but gosh, we really don’t have good recovery plans for recovering 200,000 databases right now.

Bridget: [00:10:40] They say that you don’t have backups if you can’t restore from them, but sometimes if you don’t have someplace to restore to, it’s a problem.

Mark: Yeah, and for us, it was interesting because we had accounted for— the normal case for us was we’d lose a single database server, we restore 1,000 databases or 500 databases. We’re going to lose the entire fleet and have to restore hundreds of thousands of databases all at once in a reasonable amount of time scenario, it’s just not something that we had reasonably planned for. It was our bad, but we just were not well prepared for it. And we learned a lot. Again, Stephanie is right. When you come out the other side of that and everything turns out everything is okay, and you learn a lot, and you write a postmortem, and people eventually get over it. I still think it’s a disaster, but it could have been a lot worse.

Stephanie: Well, it’s interesting because if you look at some of how healthcare.gov is portrayed now, I’ve seen it portrayed as a success of the Obama administration, their successful rescue of the website. And so even that has been spun or remembered in people’s minds as a successful story now. I think, I think FGFs, if we didn’t think it was a success or something was positive about it now, like, I wouldn’t be talking to you right now. Nobody would be interested in hearing about it.

Bridget: [00:11:54] That’s, that’s actually a really good point, which is that the way— and this is one of the other reasons I thought it’d be really cool to have the two of you talk to each other, not just to us— is that the way that you communicate around the problem is a significant part of how people construct the narrative in their minds and how the myth grows and how people remember the thing that happened. You know, Mark mentioned that, you know, the EBS apocalypse, that yeah, a few of those Amazon issues left all of us with enough scar tissue that we still kind of, you know, look at EBS and go, I don’t know, do we want anything on that? And it’s been years and it really doesn’t happen anymore very often much. But anyway, I’m wondering if you want to, you want to kind of touch on that, the construction of the narrative. Around these events?

Stephanie: That’s a very important part, I think, of managing your disaster. But it’s something that I think could really get in the way of, like, you solving the problem. I felt very fortunate in my experiences to have whole teams of people who were dealing with that. So that, like, my, my hotel room wasn’t being camped by reporters. Like, I didn’t have to worry about, like, talking to people myself. I just had to, like, fix the problem. And not worry too much. But I think that that is 100% a huge part of one of the things that I think has to come after your disaster, which is the regaining of trust of whoever uses your service or whatever it’s for. Because how people think about it is going to determine sort of the fallout of it or the impact, which is sort of interesting. It’s not something engineers often like to think about very much because they’re like, oh, like you know, we fixed the problem, like, you know, marketing is just for somebody else to do or whatever. But remembering that somebody has to be the one to, like, reassure people that it’ll be okay is something to think about, I think.

Mark: [00:13:50] Yeah, I mean, outage communication both before, during, I mean, during and after the outage is something that I care a lot about and talk a lot about and complain a lot about when people do it wrong. I think Stephanie’s completely right that it’s really, really hard to be sort of in the middle of responding to a serious problem and also have to be the person that needs to communicate about that externally. It’s interesting because I think that the, the scale of the problem that you’re trying to deal with and the complexity of the problem really dictates the organizational structure of how you respond to that and who the right person is to post status updates, for example. And what the appropriate cadence is and kind of everything about how you communicate with the outage. I will say the one thing that’s constant, at least in my mind, is the more transparent you can be and the more public you can be about what’s going on without pulling punches or hiding, you just— there’s so much goodwill that you gain just by being honest, even if things are bad.

Trevor: Yeah, there is that outage with Azure during our Enterprise DevOps episode that caused all kinds of problems for Matt when it came time to cut the episode together. And the, you know, if you were watching the Azure portal, there, you know, most things were green and then some things kind of were red and then green again, but they really still weren’t working. And then it was like 3 weeks or 4 weeks before Microsoft actually came out and said, here’s, here’s what happened. And it was, I mean, it was very frustrating as, you know, trying to work through those problems for our clients. It was not a fun time.

Bridget: [00:15:28] I know that I saw the keynote that Mikey Dickerson did at Velocity last year where he talked about CNN being the alerting system. And I imagine that if you have not yet installed a monitoring system and you have CNN telling you when your site goes down, everything is very public. But how do you handle the communication aspects of the technical things that you do once you do have some actual monitoring that you do want to talk about, but you don’t— maybe you don’t want to turn CNN on your New Relic dashboards. I mean, I don’t know. Like, how do you decide exactly how and what to communicate to people?

Stephanie: You’re touching on something that’s definitely true, which is that there are sort of rings of communication. You know, you have to be able to talk to like the other engineers who are working on the problem. And those conversations are going to be very different than how you talk to your customers. Even if you’re trying to be super open and super honest, your customers like probably don’t care about like where in the logs you found that tiny little error. They want to know like when is it going to get fixed and like, like are you actually working on it and like maybe some other technical details but not, not the minutia. At healthcare.gov, um, we, for the other engineers, we brought everybody into one room. And this is something that I think is somewhat controversial. People have a lot of different opinions about how communication can work remotely. But I think in high-pressure disaster situations, if possible, especially if you’re working with people you don’t know, being in the same place physically and being able to talk to them face-to-face is, I think, very helpful because you don’t have that trust that you need to like really interact well over text communication. That’s sort of an opinion I have. I think a lot of people wouldn’t agree with me. And I am happy to live in a world where that’s true. But that’s something that I thought— I found to be true. I think one thing you can do to help facilitate that is that the person who’s in charge of solving the outage should not be the same person who’s in charge of communicating about the outage. You can have different roles for that.

Mark: [00:17:35] Yeah, I think that’s— I absolutely agree with that. When we were at Heroku, I did a talk at Monitorama about this, or at Surge about this. We would often, for more complex kinds of incidents that required a lot of focus from the person who was driving the resolution, we’d have our support team come in and manage the status communication. And their job was to sort of poke the people who were responding to the alert every 10 minutes and get an update so they knew what to tell customers, or every 20 minutes, rather than having, you know, somebody have to pull their head out of technology to think about the right wording to use. And wording is a really interesting topic as well. We had this problem at GitHub where we would compose a status message, and anyone on the team could do it. We did it through ChatOps, naturally. And we would do things like, right, we are preparing to restore service to the affected file server, and that’s when my head would explode, because I’m like, wait, you used the words file server and restore in the same sentence. People are about to freak out. We need to be very careful with the language we choose. Because we’re not talking about restoring data from a backup. Like, we just rebooted it, it’s gonna be okay. So language choice can be really difficult, especially when you’re sort of in the middle of trying to solve a technical problem.

Bridget: [00:18:47] I would also imagine for some of the incident reports that you’ve written, I’ve read and I’ve wanted to know, like, did they, do they think that they had some understanding of what happened? Do they seem to have remediation items that are, you know, commensurate with the scope of the problem. As an end user who is technical, I don’t want to hear, YOLO, something from said server. You know, it’s like, I want to hear that you think that you have some amount of assurances for me in the future, even though I understand that this sort of thing is sort of a myth.

Mark: Yeah, I mean, I think this kind of goes back to the point that Trevor made around timing for postmortems, right? There’s a tension between wanting to get information in the hands of your users or customers as quickly as you can, but also knowing that it takes time to really understand what happened. When I think about sort of public postmortems or public statements around outages, there’s sort of 3 things that I want to get across. The first one is I want to apologize to people, right? Because I— and I don’t want to use words like, we apologize for any inconvenience this may have caused you, because I know it was inconvenient. We were down. Um, so there has to be a sincere apology.

Matty: [00:20:03] You have to—

Mark: you gotta mean it. Um, the other thing I need to do is make sure that people feel confident that I understand what happened. So I need to display confidence and I need to display a really firm grasp about the problem. And the last thing I need to do is tell them what I’m gonna do to try to reduce the likelihood of something similar happening again. And if I can do those 3 things, that’s really— those are the only 3 things that I try to do in a postmortem. Everything else is secondary to that.

Bridget: That’s great. That’s probably why when you write them, people generally feel more positive about the company than they did before the outage, ideally.

Matty: So I found this, just wanted to share kind of an amusing tweet, as we tweeted out and said, how would you define a disaster versus an outage? And someone said the number of managers involved. But that being said, thinking about that, you know, I’m kind of thinking about this whole idea of disaster recovery, right? Like, we think there and we’re going to have these plans for what do we do if we have a disaster, but in a lot of ways, when I think about my history, disaster was always predicated on giant asteroid comes and hits the data center and we have to recover all the things. That’s what we tested, and so what are some of the ways that you think that you know, we can— I don’t want to say plan for disaster, but be able to be in front of this in an actual legitimate non-ceremonious way, which is what most DR tests are to me, is ceremony.

Stephanie: [00:21:39] I sort of think this is related to the idea that if you don’t test your backups, you don’t have backups. Similarly, if you don’t test your outage response, you don’t really have an outage response. And one thing that I’ve seen that I think is, uh, A, super fun, and B, works super well, is sort of like a role-playing game for outages where you have one of your experts is like, you get paged for this reason. And then someone else who’s maybe like also on the team or maybe new on the team has to like figure out by like looking at graphs and looking at logs and, uh, what is wrong. Um, and you sort of role-play it together. You can also, um, and you can do that with sort of like in an afternoon, kind of like a Dungeons and Dragons but for outage response, or sort of more in-depth where you have something like killing your jobs or something like that, like a Chaos Monkey or something.

Bridget: So is this an example of an exercise where you would actually inject failure into a system, or is it more that these are graphs that were captured from last time there was a failure and then people are trying to figure out if they could solve it.

Stephanie: [00:22:43] Or like, it’s more like you could say, I look at this graph, what do I see? And they’ll be like, you see a spike, or you see nothing. You have to have someone who really, really knows the system well to be an accurate incident creator. But especially if you’re training new people, it really helps because it really makes them feel more confident that they know what they’re doing.

Trevor: What are some, what are some kind of like I mean, you kind of gave some examples there, but like, how do you build that?

Stephanie: It’s like a— this particular example is like a social game, so you don’t have to build anything. You just sit around a table and one of you talks and the other 2 like show them where they’d look for problems and symptoms.

Trevor: So your DM just becomes a disaster master instead of a dungeon master?

Mark: Yeah, nice. Exactly.

Matty: I just made a saving throw against a blocked key.

Stephanie: Right, right.

Bridget: Is this a d20 game or a d10 game?

Stephanie: Well, sometimes it comes down to like how observant are you? And stuff like that. And you get into arguments about whether the victim is an observant person or not.

Bridget: [00:23:48] I’m not gonna lie, I want to play this at the next DevOps Days.

Trevor: That actually is a really cool idea.

Mark: I’m going to start the Kickstarter right now.

Stephanie: But yeah, I think you have to practice. You don’t want the first time you’re trying to, like, find the instructions for how to start, like, a conference call, with everybody on the team, the time you need to start a conference call with everybody on the team.

Bridget: That is a really good point. And that also, that also just the idea of who you’re going to get onto the, on that call makes me think that you’ve also dealt with a giant epic, uh, like chronic problem, whereas Mark has dealt with a lot of these, um, smaller acute problems. Can you address a little bit, um, Mark especially, like if you have a problem that you think starts out small, When do you decide that it is much, much bigger? Like, how do you handle that if your response is going to be different, you know, depending on what the scope of it is?

Mark: Yeah, it’s really, really hard. I have not yet been in a disaster that was longer than, say, a few hours that I knew less than a few hours in that it was going to take that long, right? You’re sort of— you always feel like this is going to be over in half an hour, this is going to be over in an hour. Maybe I don’t need to go to this. We don’t need to switch to shifts yet. It’s going to be fine. We’re going to have this solved before too long. You never really know until it happens. So that’s, I mean, that’s really part of the reason why it’s really important to know how you’re going to respond in those cases, right? So you can’t be 7 hours in and think, okay, now I need to switch to the shift thing. We don’t have that plan, so I need to spend 3 hours figuring out who’s going to be the next person on call. If you’re in that space, you’re already screwed. Because chances are, you know, if you’re in this gigantic outage that is suddenly going to become a 2-day ordeal, I bet 75% of your team was already pitching in and helping, and now everybody’s tired at the same time, and now you’re in a really bad spot. So it’s really hard. It’s something that you have to plan for upfront, and you have to be prepared to deal with that kind of situation, because you never know when you’re going to escalate to that point. I just wanted to throw a couple more things in on the sort of training and preparing in advance thing. I love the role-playing game thing. Another thing we did that was sort of similar to that is we had, at Heroku, we had a complete copy of the production environment in staging, and we were able to— and we also had a complete set of runbooks for common failures that came complete with simulations for how to simulate that failure. So before new people would go on call, we would run them through simulations in staging of exactly what a production failure of a particular system would look like, and try those things out. So it was a really good way for people to gain confidence that they understood how the system worked, how to move around, how to get to the monitoring systems, how the components fit together, all those kinds of things, to gain that confidence that they might not otherwise have. Another way that I think is really good and really helpful is, if you have a distributed culture and you have this ChatOps focus, new people that come on board get to see how you work and how you respond. Instead of having training materials, they get to live it. They get to watch how you respond. They get to watch more senior people or more experienced people who have been around longer sort of go through the process live. So that’s another great, great way to sort of train people how to respond.

Bridget: [00:27:01] Yeah, no, that— I really like that idea, and I think that because when I joined DramaFever, for example, I went and I read all the old incident reports, and I read as much as I, you know, could of what appeared to be relevant bits of chat from before I started, and Being able to come in and not just see the stuff that’s live, but also look at the stuff that happened before you were there. I gotta say, I mean, I prefer things that are not handled on a conference call if at all possible, just because that way new people coming in can actually read the stuff, you know, like they can, they can read the play-by-play and see what actually happened.

Trevor: Yeah, I mean, whenever somebody says we’re working on something like that, you know, can we, can we, can we just hop on Skype or on Lync or whatever, and I’m like, is— do we need to? Do we need to? Because we can— it seems like we can continue this conversation here in chat and be just as effective, and then everybody can see what happened.

Stephanie: You really have to balance the, like, problem you have now, because I think sometimes problems can get solved faster if you’re talking to people and not typing, with the value of having the record of it. Because I definitely agree, there’s a huge value in having the record of the IRC logs But also sometimes, like, that, you have to, like, balance, like, what is more important in the moment.

Bridget: [00:28:24] You make a really good point, Stephanie, and what you were saying earlier too kind of struck me as interesting about the value of the high-bandwidth face-to-face communication, especially when you don’t necessarily have high trust or know each other’s working patterns very well, that sort of thing. I work remote, and our whole ops team is actually almost all the time in a different physical place from each other. Like, every member’s in a different place. And so we use Slack extensively. We also use Google Hangouts. Sometimes we use both at the same time. The stuff that goes in Slack is, like, things that we might wanna be communicating to more people than the people who are on the Hangout. Like, we still have, you know, that written record of anything that we want to follow up on or whatever.

Mark: To me, I think, you know, I think the way that you interact on a day-to-day basis is also the way that you should interact during an outage. The last thing you want to do is change your mode of communication when everything is falling apart and you’ve got high stress. So if the way that your team works best together is through Hangouts or through chat or in person in a meeting room, you should do whatever works best for your team. And not try to kind of change things up at the last minute because all you’re going to do is introduce more stress and more confusion.

Stephanie: [00:29:40] I think that’s a really, really good point.

Trevor: Yeah, there was a— I mean, along these lines, there was a post from, I think it was from Hanselman, about being the person who works remotely and how people often forget about the person who works remotely and will, you know, either not invite them to a meeting or not tell them that a meeting has changed. And I imagine that also happens in high-pressure situations, which makes things even more frustrating.

Bridget: I think there it’s just the— you’re sitting there hoping that the person is typing something, and if they aren’t typing something to tell you that they’re typing something, you know, it’s like, have you, have you hit enter yet? Or then of course there’s the somebody pastes in what they plan to do and you start typing as quickly as you can. No, no, don’t do that! And sometimes it’s too late.

Matty: So we talked a little bit about, you know, kind of, kind of practicing for them and the role-playing. And, you know, Mark’s gonna go Kickstarter this and we’re gonna all make a ton of money. It’s gonna be like that release game. It’s gonna be awesome. So we won’t need to get sponsors on the podcast anymore. So there’s that. But now I just want to think about some more tactical things too, to think about. Like, we’ve had some big war stories, but what are some of the tactical things that maybe if you’re not in an organization that’s going to necessarily have a sweeping disaster, but a disaster is in the eye of the beholder, I think, right? Like if we talk about the difference between a disaster and an outage, you know, they made the joke, it’s how many managers, whatever, but something that makes— it’s what you think of as a disaster depends upon how you’re viewing it. And so maybe what are some tips or some ways to kind of frame it if you’re someone who is in the trenches to be able to get started with identifying when something truly is a disaster or when your boss is making it a disaster but it really isn’t?

Stephanie: [00:31:36] I think from my perspective, it’s like, are your users hurting? If they’re not hurting yet, then like you might have a disaster coming, but your manager might be upset, but I don’t think it really qualifies. But if your users are starting to notice, that’s when you really need to jump on board and get focused.

Mark: Yeah, I think that’s a great point. I think if— advice for people who haven’t been in that situation but may be, being able to answer that question, rather, being able to quantify how many users are affected and in what way they’re affected is hugely important. And that’s different than monitoring. You know, monitoring may tell you the server’s down, but it doesn’t tell you how many users that impacts. You could be in a situation— we saw this a lot at various places I’ve been, Heroku, GitHub, and so on. I’m sure you see the same thing at Google all the time, where you have partial failures all the time. At scale, everything fails. Services are down for somebody literally all of the time. What is the threshold where it becomes a disaster? When do you need to start talking about it publicly in status? Those are questions that you really need to answer upfront. Because again, it’s really easy for non-technical people to say, oh my God, the thing is down for this person. You’re like, well, that’s great. 1 out of 25,000 servers is down. Whoopie. This happens all the time.

Stephanie: [00:32:58] Yeah, and hopefully your monitoring would give you some clues as to how badly your users are bleeding, but it’s totally true. Like, sometimes it just doesn’t correlate at all, right? Because we think they’re like, oh, like, the latency of my server is this, but you’re measuring from within the same data center. And like your user is impacted by so many things that are not— your little prober isn’t impacted by at all. So yeah, I think, I think that’s a really, really good point.

Bridget: Before we get too far away from it, I do want to go back really quickly to the acute versus chronic thing because it occurs to me as you were all talking that if how bad a situation is is a question of perspective, then Maybe you have a chronic disaster brewing, even like healthcare.gov was a train wreck coming from a million miles away that nobody seemed to jump on top of or notice for a long time. Can you address— until suddenly it couldn’t be ignored. Can you address a little bit, Stephanie, and also Mark, how you can— what are some of the signs that something is chronic disaster waiting to happen that you can maybe head it off before it actually becomes— before it explodes?

Stephanie: [00:34:13] I think you have to have your goals set for your system or your project, right? And they have to be specific. Um, they can’t just be like, my website works for a lot of people, you know what I mean? You’re like, my website should be able to hold like this many concurrent users and they should be able to access it this fast and like this is the date when we’re supposed to launch. Um, and if it’s not If you have some measurable goals, it’s easier to track, like, are we meeting them? But one of the signs that you might be headed for disaster is that you don’t have those goals, or you don’t think about those goals, or you don’t know how you’re doing with respect to your goals. At healthcare.gov, when it first became apparent that it was a problem, when Mikey first went there, he discovered that they didn’t have a heck of a lot of monitoring. And there’s a story that he tells sometimes where They have a bunch of different servers and they put the monitoring bot on the first server and it’s really weird. Like, the latency is all over the place, the performance stuff is all messed up, and anyone with any experience would look at this and be like, there’s something really awry. But the people who were working on it, one person said something to the effect of like, oh, well, I bet it’s just that one. Because they sort of were thinking like, we think our system works, but they had never looked at it, so they didn’t know that it wasn’t working at all. Yeah, I don’t know if that answers your question at all, but those are some of the thoughts that came into my head when you asked it.

Bridget: [00:35:34] Yeah, that actually helps a lot because those are some real warning signs.

Mark: I think acute and chronic in disaster— I think disaster is really something that’s in the eye of the beholder, right? Is it a disaster if your site is down for 10% of your users once a day every day, or 5%, or 1% even? I don’t know. It sort of depends, right? It can be really hard to to determine whether or not this is a small problem or whether, if it’s a small problem but it happens every day, do you have a disaster suddenly? These sorts of— and pager fatigue makes it even harder to tell, right? If you’re getting paged and you’re having to support this problem every day, suddenly you have a disaster on your hands because this is going to impact your ability to respond to other problems. It’s going to impact your reputation with customers. It’s going to make you look bad. It’s going to make people lose confidence in you. The site may not be down for everybody, but you can have consequences that are just as bad.

Matty: I think, and also in addition to the scale, or like sometimes you could think about the who, and I’ll talk about that, follow up that in a second. But one thing that it made me, makes me think of is I did a career transition at one point where I worked for Bank One and then JPMorgan Chase, and I worked in treasury services. What’s Treasury? It’s all wire transfer. And I looked this number up once and it was $3.5 trillion went through those systems per day. I’m glad I discovered that like after I decided to leave the bank because it freaked me out a whole lot less to know that. And then I went to go work for Apartments.com and it’s a different way to think about it because you’re like, oh, it’s down, refresh your browser. I’m sorry that you can’t like do your apartment search right this very second versus money or, you know, the classic examples of it’s nuclear reactor, a hospital, you know. So part of it is what is actually impacted by that outage, you know, and the users. Not so much like that the company is more important, but is it involving their actual cash or their dialysis or something like that? And then the other thing is I think about from a triaging perspective of like within an organization or within your customer base, you may have certain customers that as much as we’d like to think everybody is equal in the eyes of, you know, God and Amazon or whatever, some are going to be more important than others, right? Or even within an org, you know, I think about, I worked for an executive search firm once, which is an awful lot like working for a law firm where you’ve got like, you know, 5% of the people make 95% of the revenue. And so, you know, that you may have that one user that’s having a small impact, but them not being able to work for an hour has massive impact on the organization, more than 100 people not being able to work for 10 hours. So it’s, it’s all relative, but like you said, depends upon the pain you’re causing the users, but then the pain that you’re causing your organization and its ability to fulfill its, its goal.

Stephanie: [00:38:33] I think that gets into something really interesting about chronic disasters, which is that in my experience, like, a chronic disaster is just like a, like a small outage happening like for a long period of time, but it has a huge organizational impact. And you have to really prioritize like what parts of the problem you’re going to work on because you can only get up every day and solve like so many issues, and then eventually you do have to go to sleep. So you have to prioritize like what is very important today And I think that your comment really gets to that, right? Because you can think like sometimes some users are more important or, you know, like, or it’s kind of like a bit of like an anti-egalitarian thing to say, but sometimes like certain users have like different needs or sometimes like certain problems you could solve would help a lot more people even though like there’s one person who like really needs to get something solved. At healthcare, we had to really think about that a lot because the way the insurance market works is that you go through different workflows depending on all sorts of things about your demographic, right? Like, do you have kids? Do you have— are you a veteran, et cetera, et cetera, et cetera, right? And so when you’re prioritizing your bugs, you end up in sort of tricky situations where you’re like, are we going to fix the bug about the veterans or the one where people’s spouses are being listed as their children? Right? And you have to sort of think about that, and it’s kind of brutal, but everything has to happen, like, sort of one thing at a time, or, like, you know, X things at a time, where, like, X is the number of, like, engineers you have. But prioritization, I think, is really important.

Matty: [00:40:12] I think it goes back to— I know everybody’s tired of hearing about empathy in DevOps, but we usually talk about empathy for our coworkers, but in scenarios, right, like, that’s got to be really rough because your priority— it’s really easy to abstract, just to abstract away that it’s just these, these users, but this is like someone’s life, right? You know, I mean, this is, this is making a huge deal and that’s gotta be really, really, really hard to do and not, and be empathetic about it, right? And not just be a matter of like, well, I’m just gonna move because just by fact of moving this little Post-it over here in the Kanban board, what did that do to a person? You know, out there in Paducah or whatever, you know. So that, that’s— wow, that’s all.

Stephanie: Yeah, it also, um, it could get really brutal. For example, there’s all sorts of statistics out there about how many people in the United States die because of a lack of insurance every year. And then you could also see like how many people— like how many people— how your traffic dropped off if you had an outage. So you’re like, oh, like our outage was this long, we lost this many requests, and like, if you translate it, that means like 40 people died this year that didn’t need to. Like, it could get really brutal, and you want to think about that because you don’t want to become like a cold-hearted like engineer automaton, but you also have to like be able to put that aside sometimes because otherwise you will be just like rocking back and forth in a corner.

Matty: [00:41:44] Can’t you turn that around too and flip the number, not in the the bad way, but also think about the, you know, what’s the positive of it too. I think that’s the hard part, right? Because everyone who listens to this show is tired of hearing my whole thing about that cis admins are inherently cynical and pessimistic and whatever. And that’s a topic for a different show we did. But we tend to probably, like you said, we like to focus on that, that we’re used to that. So we’re probably more likely to think about all the times that something went wrong. And like you said, the people who didn’t have their insurance and didn’t have that, and not thinking about the fact that, hey, but when the system’s working—

Stephanie: Yeah, well, I just want everything to work perfectly 100% of the time, all the time.

Bridget: But hey, Stephanie, you fixed the website and lots of people have insurance and aren’t dying.

Stephanie: No, it’s—

Bridget: thank you.

Stephanie: I mean, I get a real kick out of that. I won’t lie to you. I found a thread on Reddit recently where a bunch of women were talking about how all their birth control was free, and I was like, oh my god, I felt so good.

Bridget: [00:42:47] You’re like, I helped make that happen, yay!

Stephanie: Yeah, yeah, I know. I just was like teenagers in Nebraska being like, I can get the pill now, and I was like, you should. Good job.

Bridget: That’s great.

Matty: That’s a really good way to wrap up this episode, I think, on a pretty positive note about thinking about the impact that when things aren’t disastrous The stuff we do really changes the real world.

Bridget: It really does. I mean, even if the site you’re working on is not healthcare.gov, even if it’s one of the half a dozen sites that Ambriaco has worked on that I keep working at startups that use, the fact that the stuff that Mark and his team would do— I got to ask, how many of the seasonal migration of sysadmins followed you to Operable?

Mark: That’s a tricky question.

Bridget: You can’t tell me how many, but I’m guessing it’s a non-zero number because This is, uh, you build trust with a team and then people want to follow you.

Mark: Yeah, so there’s 2 people on the team that I’ve worked with in the past.

Bridget: [00:43:50] But yeah, that’s even if it’s, even if it’s not giving people insurance, which is obviously a super awesome thing that a good website that has been fixed does, there is a, there’s a lot to be said for these sites that a lot of people use. And when something goes wrong with them, it’s nice to have the confidence to know that you’re going to be told what happened and understand that there’s people like Mark and Stephanie actually fixing it.

Matty: So hurrah for them.

Bridget: I appreciate it.

Matty: So this episode just flew right on by. I can’t believe we’re almost done with it. So we’re gonna, we’re gonna have to figure out another reason to have y’all back on again to talk about something else too, because this was super fun. The CFPs for DevOps Days Chicago are open. If you go to cfp.devopsdaychi.org, we’ve had a bunch of really awesome proposals so far, but we’d really love to hear what you have to say. And I just want to talk real quick on that note. I’ve had a lot of conversations. For some reason, this has all happened in the last week or so with a bunch of my coworkers. Who have said the same thing to me, which is, I don’t mind speaking in front of people, but I don’t know what to say. And the advice I gave them was part of the problem that I think sometimes we have is that we have a bunch of really smart people like Allspaw and Mark and Patrick and, you know, Bridget and everybody who gets up there and they speak these like big deep thoughts, right? You know, and have these groundbreaking ideas. And so we think that that’s what a talk has to be, and we forget that it doesn’t, right? You know, so my suggestion is, again, you don’t have to think of something revolutionary to pitch a talk, because your opinion, your experiences are special to you, and people would love to hear them.

Bridget: [00:45:46] So, and even if the stuff that you think you’re going to talk about is— you feel like it’s been well covered, maybe by other people, If you think that that’s the case, go watch Stephanie’s talk from DevOps Days Vancouver last fall, because she talked at DevOps Days Vancouver about her healthcare.gov experience, and it’s not the same talk that you would see if you watched, say, Mikey Dickerson or someone else talk about the same website, because every single person has their own perceptual filters, has their own set of experiences. So—

Stephanie: No, I will admit, I ripped off some of the slides.

Bridget: Of course, the slides are not the talk. The slides are not the talk, and the slides are the least important part of your talk unless you’re pitching a slide.

Trevor: If you care about a topic, other people care about it too and want to hear about it.

Matty: And I also— I’m going to reveal my super secret new trick for getting accepted to— getting talks accepted, which is stop pitching talks to like Velocity and DevOps Days and everything. And go find the other conferences who don’t know about DevOps yet but are just starting to learn. If you look at all the places I’m speaking this year, they’re at things like developer conferences or agile conferences or the ALM Forum that I’m in Seattle for right now. So the beautiful thing there is you don’t have to— you know, you’re not talking to the same cadre who’s, like, heard all the brilliant things that all of the thought leaders have said already. You’re talking to people that are just wanting to learn it and want to know what you know. So I’ve just given away my super secret and now I’m screwed and I won’t be giving any more talks this year. Oh, but also I’m really excited. This is so dorky, but I do want to share it. There’s a conference. It is, I think it’s August 8th through the 11th or something like that. We’ll post a link. It’s called That Conference. It’s in the Wisconsin Dells, like with water parks and stuff.

Trevor: [00:47:37] That conference is super cool.

Matty: It’s super cool. And I’m speaking at it this year. I just got accepted. It’s awesome. It’s— I’m giving a talk called Making Delightful Infrastructure S’mores with Chef.

Bridget: You should bring a lot of sunscreen. The Wisconsin Dells in August sound like a good place to get sunburned.

Matty: Well, it has a big indoor water park. It’s at the Kalahari. I’m so, so excited about it. Um, so anyway, are you camping or are you staying? No, no, it’s at a resort. It’s like this super pimped out Dells resort. Oh yeah, it’s a big water park. It’s— but it’s built like, like the idea that the whole conference is structured like a camp for geeks. Yeah, right. Like, so the session, like, the speakers are called counselors and it’s all like— but I don’t know, I’m just sorry, I just got accepted for it yesterday and to be honest, this is gonna sound—

Trevor: I wanted to go to that since it started 5 years ago.

Stephanie: Yeah, so I have 2 things. The first is I want to promote the writing of a woman called Kate Hudson. I met her when I was a Google intern many years ago. And she was like a fabulous badass to like 17-year-old me. I was very, very enamored of how cool she is. And she has continued to be awesome. So I wanted to promote her website. It’s katehudson.com. That’s Kate with a C, Hudson with a T. And her blog is called Accidentally in Code. She writes about travel and tech and diversity stuff. And if you’re looking for a really good blog post to start with, she wrote one called Honey, I Left the Tech Industry, which I think a lot of people can really relate to if they’re feeling sort of feelings of burnout. The other thing I wanted to talk about, of course, the price anyone pays for hearing about healthcare.gov, is being asked to contribute to similar projects. If you are interested in doing this, they are still doing it. There are people in the Veterans Administration, in the immigration IT departments, and in the White House. So if you’re like, hey, like, I would like to not do my real job for anywhere from a few weeks to a few years, you can check out the United States Digital Service, USDS. Apply, give back to your country or a country if the US is not your country, or think about how you can use your technology skills to solve civic problems wherever you are, because there’s a lot of really important issues. I’m from Canada, and there’s all sorts of things in Canada that could be better. So think about how you can use your skills on those projects if it at all interests you.

Bridget: [00:50:26] That is great. I love it. All right, let’s hear checkouts from Mark.

Mark: So I don’t think mine are quite as good as Stephanie’s. I’m kinda jealous. But this first one is a really good podcast. So I actually got turned on to this podcast by John Allspaugh. John is really good at giving people things to make them think, like PDF files that are very long.

Matty: I was just gonna say, did he send you the podcast in PDF form?

Mark: No, but I do have some PDFs from him queued up that I still need to read. John is awesome that way. But it’s called the Pre-Accident Podcast. It’s by a guy named Todd Conklin. Todd wrote a book called Pre-Accident Investigations and is very well known in sort of the new view of system safety. And the Pre-Accident Podcast is really awesome. He has guests on every week that he has conversations with about safety. It’s not safety in terms that, that are directly applicable to the kind of work we do, but you can, you can feel the parallels, right? So he goes to places, you know, where they’re running big machinery and trying to keep people from dying. And the same sorts of techniques that they apply and the things that they look at and the way they think about safety and, and organizational performance are really applicable to what we do. So I think it’s, it’s super fascinating and everybody will— everybody that I’ve told about it that has listened to it has enjoyed it. So you guys should too. And my second thing is not nearly as serious. It’s Destiny, the video game, which I have been playing incessantly on Xbox One for quite some time. I actually took a break, but literally today the new expansion came out. So as soon as we get done with this, I’m going to start playing. I took time out to talk to you wonderful people instead of playing video games.

Bridget: [00:52:07] Great.

Mark: Awesome.

Bridget: It is my turn. All right, I have a couple of checkouts for you here today. The first one is that the videos from DevOps Days New York have gone up. That was the most recent conference I was at. I liked it quite a bit. And mine is all about remote work, and it’s kind of a love letter to my coworkers at Drama Fever. So that one is pretty good. And then the, The rest of the talks are pretty great as well. So you can check those out on, I think there’s, they’re all linked from the DevOps Days New York Twitter handle, which we’ll put in the, in the show notes. And then the other thing that I wanted to show you was Lara Hogan from Etsy wrote a really good blog post called Mean Time to Women in Tech, where she talked about the mean time it would take for people to ask her when she was talking to them for tech reasons about something else to move on to talking about being a woman in tech, and she started timing it, and she said in the last month, the mean time to the women in tech conversation was 3.5 minutes. So it’s pretty ridiculous, and her blog post goes into some good detail about all of the ways that that is really not the most constructive or helpful way to talk to people, and ways that you can do your own research, learn how to Google instead of asking someone about everything in their life that may or may not have annoyed them, because that’s a conversation they want to have every single time they talk to someone.

Stephanie: [00:53:38] That sounds hella interesting.

Trevor: Yeah.

Stephanie: I have to go read that immediately after this.

Bridget: Yeah, you should. It’s a really good blog post. So, yeah, so those are the 2 things I think you should check out. How about you, Trevor?

Trevor: So I just went camping last week, which was lots of fun, though it was very cold and windy. And I got a new toy because I happened to see that it was on the top of the camping list on Amazon. And that was a pocket chainsaw, which is basically a chainsaw, the chainsaw like band with 2 rope handles on either end, like a rope handle on either end. And it worked so well. It was like—

Matty: you just discovered this for the first time? Well, I mean, they’re awesome.

Trevor: They’re amazing. It was super helpful. It was so helpful.

Bridget: This is something that people know about, pocket chainsaws?

Matty: Oh yeah, you would know it if you saw it.

Bridget: I’m quite sure I would not.

Trevor: Well, it’s—

Matty: yeah.

Bridget: Is this something you can take on an airplane? I feel like this is not—

Trevor: [00:54:39] I doubt it. I mean, maybe in your luggage, your checked luggage.

Bridget: Oh, I don’t check luggage.

Matty: I know you don’t.

Bridget: You’ve seen me have that argument with Ducey.

Trevor: Secondly, on the kind of the game theme is Jen and I saw a talk at C2E2 with 2 of the voice actors from Dragon Ball Z, and since I’ve been on a Dragon Ball Z kick, and there’s a new game called Dragon Ball Z Xenoverse, which is so much fun. It is fun for, like, no reason. You get to— like, something’s wrong with the timeline in Dragon Ball Z, and you have to fight through all the major fights, and it’s so much fun. That’s what I’m going to play when this finishes. And finally, Microsoft Office apps for Android are— for smartphones are in preview now. So Excel, PowerPoint, and Word. We’ll put the link to the Google community in the show notes because you have to join the community to become part of the test group. Anybody who uses an Android device and participates in test programs will know what that is. Yeah, Matt?

Matty: [00:55:55] Great. Okay, we’ll wrap up. We have like a kind of a little bit of a game theme. I have a couple games for this that have come up. One is a game that I started playing a while ago on iOS, and Trevor pointed out to me it’s also available on Android, which is cool. And it’s called Does Not Commute, and it’s really kind of hard to explain, but basically it’s kind of a top-down driving game. So it’ll start where you’re like, oh, drive so-and-so, you know, Miss Smith, from her house to her office. And then you drive it there and you have X amount of time, but then it goes and it rewinds. And at the same time, then it’s like, oh, now drive Mr. Jones to the ice cream shop. But the first car that you drove is already in play, so you end up with all this overlapping and having to have them deal— you know, basically the actions you take earlier continue to happen simultaneously. It’s, it’s kind of really hard to describe, but then you play it for a— like Trevor said, like after like the third car, you’re like, oh, I get it. And then it’s— you get really upset.

Bridget: You haven’t actually just signed up as a Lyft driver and you’re not—

Matty: [00:56:57] there we go. Yeah, actually, this is—

Trevor: maybe that was the case.

Matty: I would have—

Trevor: I’m actually—

Matty: many people— like Ender’s Game meets like Google self-driving cars. That’s what this is. That’s exactly what’s happening. We’re communicating. Okay, there’s an Ansible joke in here somewhere. Okay, we’re done. Another thing is someone has been recreating highlights of Breaking Bad using the Grand Theft Auto V editor. There’s a link on the AV Club for that. It’s pretty cool. And then finally, I’ve also been a little obsessed lately with the game Hearthstone. And I know this is like old news because all things related to games for me are, but it’s this World of Warcraft kind of card game thing. So it’s kind of like Magic, I guess. I don’t know. It’s one of those— you have cards and you play against somebody and the powers do things and it’s People who, like, really know those games would probably think it’s really dumb, but for someone of my capability, it’s just a fun way to kill a bunch of time on my iPad. So I’ve been enjoying that. So, and also, I enjoy sending out the Arrested DevOps newsletter sometimes. You can sign up for it at arresteddevops.com/bananastand. It is the best way to know about upcoming podcast episodes most of the time and cool news with DevOps all of the time. We also have an iPhone app if you want one. You can download it for free at arresteddevops.com/iphone.

Trevor: [00:58:16] Thanks to our sponsors, 10th Magnitude and VictorOps. Be sure to check them out at arresteddevops.com/tenthmagnitude and arresteddevops.com/victorops. Thanks, Stephanie and Mark, for joining us, and to our loyal listeners, if you enjoy listening to Arrested DevOps, We appreciate it if you would visit arresteddevops.com/itunes and leave us a review in the iTunes Store. We would love to know what you thought of this episode. Please leave us comments at arresteddevops.com/37.

Bridget: Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter. We’re always happy to get your input, ideas, or feedback at shows@arresteddevops.com. Please let us know any ideas you have for future episodes. I’m Bridget at Bridget Kromhout.

Matty: I’m Matt at Matt Stratton.

Trevor: And I’m Trevor at Trevor G. Hess.

Bridget: We’re Arrested DevOps, and remember, remember, there’s always DevOps in the banana stand.