Into the VOID Report with Casey Rosenthal and Courtney Nash

Courtney: [00:00:00] We’ve got a bunch of data that have allowed us to bust a few myths and open a few questions and make people think more deeply about these things.

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matty Stratton, and I’m surprised I didn’t stumble over that intro after how long it’s been. I’m really excited to be coming back with a new episode for this 2023 brand new year, brand new season of Arrested DevOps. And we’re going to be talking about a topic that’s very near and dear to my heart. But before we dig into that, let’s hear a word from our sponsors. Collecting compliance evidence shouldn’t involve spreadsheets and scavenger hunts. With automated controls and over 75 integrations, Drata automates the process without needing to be an expert. Drata supports 14 frameworks including SOC 2, GDPR, HIPAA, and ISO 27001. Companies like Notion and LemonAid have shared how Drata simplifies audits through automated evidence collection. Don’t let audits slow down your dev cycle. Request a demo today and get 10% off along with waived implementation fees at drata.com/partner/arresteddevops. So if you have been listening to ADO for a while, or just listening to me, which can be both in the same, you’ll know that the area of resilience and resilience engineering and incident response and incidents and all of that is something very fascinating to me. And I’ve spent a lot of time listening and learning and even running my own mouth about it. When the opportunity came up to have 2, 2 of my favorite experts in the area who’ve done a whole bunch of research recently and want to share it, to have them come on the show, I pretty much did the equivalent of the shut up and take my money, except There’s no money part. So I’m really excited to be joined by Casey Rosenthal and Courtney Nash. Casey, you wanna introduce yourself real quick to our listeners?

Casey: [00:02:03] Sure. For those who don’t know me, I’m generally known as the chaos engineering person. I wrote the definition with my team at Netflix that I built and managed there. I started the conferences, the Chaos Community Broadcast. I wrote the book on it, stuff like that. And now I’m the CEO of a company called Verica.

Matty: Awesome. And then Courtney, I can’t imagine anyone in our audience doesn’t know who you are, but just in case, or maybe, maybe they want a refresher.

Courtney: I’m Courtney Nash. I may be known to most people as the former chair of the Velocity Conference. I worked for O’Reilly Media for a long time. I’d worked at Amazon and Microsoft before that. So I’ve been around a few other startups and Casey finally convinced me that I was crazy enough, foolish enough ’Cause really, to come work for him.

Casey: Wise enough.

Courtney: Wise enough. Very wise. But I am now in a research role at Verica. And along the way, I also fell down a rabbit hole of really becoming interested in safety, resilience, in particular, given my academic background, the role of humans and human performance in how we handle these things. And then, so, a couple years ago, we started this thing called the VOID, which is the Verica Open Incident Database. And that’s what we’re here to talk about today.

Casey: [00:03:18] Can I just interject? Courtney also has the best Oh, that’s true.

Courtney: Forget this. I am a researcher, but technically I am, I think, the world’s only internet incident librarian.

Matty: Incident librarian. I love it. It’s not to go off on that tangent, but I know there’s been a lot of conversation around how more organizations should have software librarians, you know, people who keep all that knowledge, but you’re keeping it for the incidents for the whole world. No, no problem. No small undertaking. So tell us a little bit about The Void before we talk about the report that you published, but What really is this Open Incident Database? What’s it all about?

Courtney: So, it started out of the research, some of the early product research work I was doing for Verica. We have a product that’s sort of an evolution of the chaos engineering work that Casey had done. And the sort of 2 key technologies that we were focused on initially are Kubernetes and Kafka, because as we all know, they’re perfect and not complex and never cause anyone any problems. So, I was looking for non-standard information, if you will, on what we know about that, right? So, not marketing materials or tech specs or whatever, but like, how are these things actually falling down in the wild? And so, I found various incident reports. There was, Lou had this great little GitHub repo, kubernetes.af. And so, there were a few of these things out there. And so, I started collecting incident reports because they were an interesting source of information about this. Highly variable amounts of information, which we obviously can get into. And so, I started finding these for Kafka. And then, because I’m me, I couldn’t stop, and I would end up, like, reading these things all day. And I put some feelers out to a variety of different folks, and all of a sudden, I had, like, 1,000 of these. And we started talking to people about it, and the general consensus was, holy shit. Also, thank you, and we have more of that. And so, sort of evolved into this database. And we started collecting metadata on all of that. And that’s where the void emerged from in 2020, late 2020. So we did our initial report on that in 2021. And we just released the second one more recently. And that’s, that’s so we can talk about some of those, the details of the report, but the database itself, it’s online, we’ll have the link in the show notes and everything, anybody can go and poke around. And I think the key thing, there’s a couple of key things about it. One is that it’s, when we say incident reports to this audience, probably people have a very specific idea of what that is, right? Like a postmortem review or something written by a company. And granted, there are definitely a lot of those in there, but we wanted to take sort of a broader view of incidents because I have a few chips on my shoulder that I wanted to have some data to either prove or disprove. And so, we have everything from Detailed postmortem reports, to status page updates, to tweets, to media articles, like everything that you could imagine. If somebody talked about a website or a service falling over, that’s in there. Because I really believe that there’s a lot of myths and misconceptions about software incidents out in the world, just amongst our industry. And I want to be able to start to tackle some of those things. Secondly, we collect a lot of metadata. On those incident reports. So, things like, obviously, we get the company and details around how it’s described and all of that. But we also, if it’s available, collect how long that organization said that incident lasted for, if they had any kind of severity associated with it, whether they used a particular methodology to— all that. So, we have a bunch of data that we’ve never had before about incidents. And that’s what’s really relevant here. Because we talk about incidents in a lot of specific ways, especially from a DevOps lens, right? There are a lot of us who’ve wanted to say, oh, the duration of an incident doesn’t really mean what you think it means. Or we talk a lot about, like, root cause analysis is bad, but why? We’ve never had any data before about these things. And so, that’s, I think, what’s the most powerful about this is instead of us talking in the abstract, Which annoyingly in our industry doesn’t get you very far, or sometimes gets you too far. We’ve got a bunch of data that have allowed us to bust a few myths and open a few questions, and people think more deeply about these things.

Matty: [00:07:45] And that’s a couple of things, like you said, you came in with, you’ve got, this is an opportunity to expunge some chips from your shoulder or launch those chips into the greater discourse maybe. And I know I felt that when I was at PagerDuty, I always felt like we had this opportunity, again, as for customers, what could you look at, what could you see? And it’s interesting, as I think what’s nice is as you’ve gone and done this collection, because a lot of times people feel like they don’t want to talk about this stuff because you’re bringing up this thing that is what is perceived as a failure. And I think especially around security incidents, it’s something that I’ve— and I’m curious, that’s an area that you’re talking about. I’ve always found a challenge with wanting to talk to an organization and get good practices and patterns and things from their security incidents. But that’s something that’s usually perceived as like, we absolutely cannot talk about that in public because it’s scary. And I feel like we probably felt that when we’re having that conversation on the operational side 10 years ago was the same thing. It’s, oh no, if we talk about this failure, then people are gonna wanna sue us because we had downtime, or then people will know our infrastructure and they can hack us because they know we’re running this version of Cassandra or whatnot. And it seems like we got over that. So I guess number one, are you seeing people being more transparent about security incidents And from your incident, are you seeing, like, you have this big collection of incident reports. How do security incidents play into Void?

Courtney: [00:09:08] So, the focus that we had largely for the Void was more around what we might call availability incidents, right? Because that didn’t exist. There’s a lot of security breach databases out there, right? The DBIR, you’ve got a bunch of these things that are already in MITRE. They’re tracking these things. There’s a ton of data. So, we don’t, we didn’t focus on that in terms of what we’ve been collecting. There are some, they’re mostly DDoS type security incidents, if you would say, so they still have some kind of an availability impact, and they bring it down. And so, I would— but I would say, I’m pretty— I know a bunch of folks in that industry. I still watch a lot of that. I think that the mentality that we feel like we’ve adopted somewhat in the sort of DevOps space is still a long ways away when it comes to security incidents. And I think it is— there’s so much to unpack there, right? Like, culturally, adversarial stuff, like, I can understand why people have that perspective, but the general mantra of The Void is that we need to share this kind of information because it’s the only way that we all collectively are going to get better at this. We have so much commoditized knowledge amongst individuals and in organizations that just tends to move around, and we don’t share that outwardly. And the analogy that I use often, which may be more familiar to folks also as DevOps, or if you’re a safety nerd type of person, is the airline industry in the ’90s in the United States in particular was in a really bad state. Safety rating across all the airlines was terrible. Like, faith and perception in— of airline safety was really bad. And didn’t come from regulation to change the tide of that. Like, the initial push to change that came from the people operating the planes themselves, right? So the pilots, and then eventually other folks came on board, even the air traffic controllers and ground people and all that. But pilots found each other and were like, we don’t talk to each other about what’s happening. I don’t care if you work for JetBlue and I work for American or whatever. I should actually have like a remembrance of ’90s airlines because some of them are so long gone.

Matty: [00:11:14] TWA, Pan Am. I don’t think Pan Am was still around in the ’90s, but they started sharing their incidents with each other.

Courtney: Eventually, a sort of more formalized database arose out of that. And regulation certainly came more into play over time. But that effort is sort of well known for helping to raise collective understanding of similar problems across carriers and operators that those folks were having. And I don’t even want to bring up what’s currently happening. It’s not so much a safety thing, but our airline industry is going through its own turmoils currently. But I think from a safety standpoint, that was really an impressive effort. And it was a ground— from the ground up, from the practical Commissioner’s standpoint. And I think they were all willing to set aside the competitive and other kinds of things. And I believe that is possible in our industry and also necessary. I think it will take longer on the security side of things. Even in, within liability, SRE, that type of thing, I still get asked by people all the time, like, this is great, but my lawyers will never let us share. I think it’s easy in our bubble to feel like, oh, we’re all really aware of this and we know what’s going on. I talked about the research last year at SRECon, and I had multiple people come up to me and say like, no way. I work for a bank, or I work for a— and we’re never— it’s never going to happen. And I don’t believe that. I believe that it’s entirely possible, but I also tell people that it’s possible that it will come because you are regulated to do so, which is fine.

Matty: [00:12:37] It’s interesting that regulation for good, for us instead of against us. I shouldn’t say for good, but—

Casey: Yeah, I don’t think it’s going to work for us.

Courtney: Right? That’s what’s come out of the Southwest Airlines thing, right? Is the call to regulate software and whatnot for these airlines when that was not really what was causing that problem in the first place, right? That was very much an organizational, cultural, decades-long cluster. So, I think it’s incumbent on us to work on this as hard as we can before those types of forces are imposed, because not only will they not work, but they will inevitably wreak havoc on the people at that sharp end, right? Folks who have to build, maintain, and operate and keep track of all those things in these systems, where the burden is going to still land, right?

Matty: So I think that’s really a good point about driving it towards there. And it’s a conversation that it— to make more terrible analogies, actually not more, it sounds like I said your analogy was terrible. Mine will be terrible, or maybe will be a stretch. A conversation we have about like where DevRel fits into an organization. And one of the things I’ve been telling people is say, you pick the capitalism you’re willing to do so you don’t have to do the capitalism you don’t want to do. And what I mean by that is if you are in an organization and you don’t volunteer how you’re creating the value in the way that’s understood, a way of creating value will be assigned to you. And the way that I say it’s like, I have my dogs are working dogs, they’re Australian Shepherds, right? And so the thing with a working breed is you have to give them a job because if you don’t give them a job, They will find a job and you will not like the job that they find. And similarly, I, so I think that’s an interesting, a good point, which is the more that we can drive this as practitioners and driving the industry, it will help fill that gap so that the regulation doesn’t see that as a gap that has to be filled. ’Cause we’re like, wait, we have an answer. We already fixed it. And depending on how cynical we’re feeling today, maybe.

Casey: [00:14:35] Yeah. If, uh, if the job that you gave your dogs isn’t being adorable, then that’s a crime. But yeah, I think that’s really the— and Court and I come at it from different areas of concern. The regulation area is really like my personal nightmare. I think that would ruin software engineering as a field. And you can imagine for an SRE what an availability compliance framework would look like if you want to share some nightmares. But really the theme, the biggest impact that I think The Void can have long-term in the industry is helping us redefine value of availability work, be it incident analysis, incident response, proactive stuff like chaos engineering or continuous verification. Because right now, so like the first couple reports, you mentioned myth-busting, they focus on like, okay, the things we’re trying to measure right now, are just, they’re just not good things to measure. And I’d love for Courtney to dive into that. They’re just bad. And so that part of that is like, hey, let’s prove that’s bad. Then once we do that, let’s start building up options that are good to replace those with. And that’s probably not going to look like objective metrics or statistical methods. It’s probably going to be qualitative stuff, which is good. A lot of industries operate that way, but it’s a completely different mindset from how we currently operate when we allocate resources towards things like availability or system safety. And really, availability and security are two sides of the same coin with respect to system safety.

Matty: [00:16:23] Yeah. I want to ask about some of these, start with some of the things that you’ve seen that were, again, you say came out of the report, we learned this, and then dig into the challenge of actually affecting that change when you’re talking about something that’s a radical mindset shift, especially when you’re going to be asking from that, from leadership. And I think one of the challenges, not to cut spoiler alert to it, is that let’s be honest, like what’s the average lifespan of an org chart in a company, right? 18 months. So you’re going to, maybe you do all this work to get your CIO to understand it and then he’s peace out and now you got Ms. New CIO. Who maybe came from a whole way and you’re trying to fundamentally say, this is a completely different way about thinking about the business of software. I think there’s a reason that people get exhausted because we’ve been doing this over and over again. But let’s start optimistically by saying what’s wrong that we can agree about. But I know one of the things that comes up is the big bugbear of MTTR, so mean time to resolution, which folks love this metric because I assume mostly because it makes sense. And it feels cut and dry. It’s like it’s a number and that number should go down.

Courtney: [00:17:31] Great. It feels great, right?

Matty: So why does it really suck? Okay, so we’ll start with that.

Courtney: Let’s, let’s go back in time too, because technology, or as an industry, has a really sometimes amazing capacity to borrow ideas, metaphors, tactics, techniques from other industries, right? Um, and, but we don’t always think that through enough, right? So, MTTR, mean time to resolve, mean time to remediate, what have you, comes from physical manufacturing. Like, that’s actually where it came out of, right? So, like, widgets on a conveyor belt coming out, going to get built together into a larger thing full of widgets. And they’re made in a very specific way, and they are made out of physical materials that have very specific properties, and those things degrade and wear over time. And so, you can have the mean time to remediate a wear and tear on one of those pieces should be a pretty consistent thing that has like a very normal distribution to it. And like, you should be able to like add that into your workflow and factor that in when things come in for repairs and all of that. Okay. That’s where MTTR came from. In case you haven’t noticed, modern software is nothing like that, right?

Matty: [00:18:55] It used to be.

Courtney: So, entertainingly, it kind of used to be, right? Like, we made some widgets, and they got strung together, and you could predict what the widget was supposed to do and test them, and then you shipped it on a CD.

Casey: Woo!

Courtney: Our mindset is still there, but our practices and everything have evolved so significantly. So, in the complex distributed software world, things don’t fail like that, right? They don’t fail predictably. You don’t understand even the causes or the reasons or the contributing factors to those things. But we still expect them to behave in a way that you can have a measurement like MTR. And so, we chose that as a reliability metric, even before I— it was a reliability metric for software, even before DORA and Accelerate became a thing, right? So, it’s like, I think sometimes people are like, why are you picking on DORA? And I’m like, I’m not. They just picked the thing that everybody else had already picked. Just nobody had looked at it along the way. And then you go and you look at the underlying data, which certain companies you’ve mentioned have, but don’t necessarily publish or let anyone else look at, right? So, here we have them. We finally have these distributions of duration data from incidents from software companies. And they don’t look like something that you can take an average of at all. So, everybody’s familiar with the, you know, bell curve, right? That’s where your mean is straight in the middle, and then you have standard deviations off the side. And If you took statistics classes, you learned about all the fun and fancy things you can do with that. Our incident data don’t look like that. They’re skewed, right? Which means that there’s a big, huge chunk of them that are under an hour or 2 hours. So, there’s a big bump up here, and then there’s just this long fall-off and a nice big tail. You can’t take averages of that. It’s just garbage. And So, that alone should have been the argument, in my opinion, but not all of us were like stats nerds in college, right? So, along the way, an engineer from Google, Ivanovich, he did a really cool O’Reilly report on this. And I was like, when I saw his data, I was like, those data look like my— holy crap, somebody else has actually looked at this. He looked at Google incidents and then scraped a bunch of incidents we have now in The Void, but all those incident data weren’t really published along with the report. We just saw the analysis. He ran a bunch of Monte Carlo simulations, right? So, I don’t know if people are familiar with Monte Carlo, but it’s like an A/B test. I think everybody in software knows what an A/B test is, right? You take your production data coming in, you change the buy now button to blue, and you see more people when it’s blue click than when it’s green. That’s an A/B test. We just, he just did that with all those duration data. And he made some of the durations shorter, like he made your incidents resolve quicker. And then ran a bunch of simulations of taking the average of those, and it was just a mess. We did the same thing with our data finally that we had in the last year, exact same results. So, a really nice empirical statistical demonstration that trying to take averages of incident durations gives you junk. We’re trying to tell people now with the data, that they’ve been using, like relying on, you can’t rely on this as a measure of reliability, which is super ironic. And we’re starting to see some traction with that. Like people look at that and they, I get 2 responses. One is, oh shit. And the other one is, oh shit, but I’m going to fight you on it. It’s so ingrained, right? It feels, if people say that, I can’t argue with the math, but it, I’m like, what? It feels You know, like, yeah, it’s rough how intuitively pleasant that number seems. But if you stop and look at the systems that we’re taking those data from, and then what those underlying data look like, then you’re like, oh, okay, there really is, there is no spoon, right? This particular source of metrics for our systems, we should just retire it and look at other things. But it’s very hard because people are like, they’re incentivized to report this stuff.

Matty: [00:23:15] And the time horizon is shorter than what you would think. Like, if you’re actually trying to increase reliability, you’re looking at a longer time horizon. But usually, I think this is looked at within the context of a quarter, which is to basically say, hey, does the CIO get their bonus this quarter? Or does the head of support get their bonus? And so, when you look at it in that context, it’s not a metric around increasing reliability. It’s a metric in reporting performance of the people, which it’s not good at that either, but it sounds like it makes sense. It’s the closest thing you can do to put a Nagios counter on your people because theoretically people are restoring service. And it’s— it goes back to, I think, that conversation. If you’re going to dig into this, is like, what are you using it for?

Courtney: What decision are you making based on those data? And if you can’t actually do anything differently if your MTTR goes up or your MTTR goes down? What would you do? Go and you’d look at your incidents, right? Actually, nobody looks at it if their MTTR gets better.

Matty: [00:24:20] But you might do. Yeah.

Courtney: But you can’t even trust it. But the answer is either way, you’ve got to go look at what’s happening in the system and the people operating it. So, stop running around chasing your tail, collecting data that don’t tell you anything. And go get data or information that can actually help you understand what’s happening in your systems, which will help you start to try to improve the reliability of those systems.

Casey: Yeah. Just to reiterate the, the layers there, cuz Maddie, you started to poke at, at this, like the, just flat out the statistics is wrong, right? Like we, we’ve got the simulations to show that a decrease in, in response time can still statistically move the MTTR up. So like your numbers are lying, but also that’s ignoring the fact that the data coming in is garbage. The methods for determining how long an incident actually takes are fraught, let’s say. And then there are other problems with the fact that you’re even going about this type of exercise. And yeah, the accuracy of the data, because you started to touch on this, you would need more data points than Google has in order to accurately, with any statistical certainty, even report on MTTR if it was the right model, which it isn’t. So the further you go down that path, the more just wrong you run into. So that’s why we really just kind of like, it’s just a bad statistic. It just needs to be burned.

Matty: [00:25:53] So when we think about that, agreeing, and I think again, I said, I feel like a lot of this, and Courtney, you said people either are like, oh shit, or oh shit, I’m gonna fight you. And I imagine that, oh shit, I’m gonna fight you probably does not come at the sharp end as much because I think, and tell me if I’m wrong. No, no, probably maybe some of that, but okay. Yeah.

Courtney: No, no. Oh yeah. No, it’s both. It’s both. That’s what’s so fascinating about how this belief in this metric is. Right. So Casey’s talk, Casey alluded to this at the beginning, so I’ll just, I’ll get to it now. Our entire industry is obsessed with being data-driven, right? We’re obsessed with metrics and numbers. Like you can’t, if you can’t measure it, you can’t move it. Right. Like all of this. Wharton business, then, and then infuse that with, like, technology, like, we can measure things down to the millisecond of the da-da-da-da, right? I come from a, like, psychology, cognitive neuroscience background. So, I realized there was no spoon in how people and systems interact, like, a long time ago. But I also believe and know that you can collect data about human behavior that’s still data, right? But it’s just not the kind that we lionize, or invest in, or believe in our industry. But there’s a whole set of other industries that look at human performance, and how do you measure how humans work, and how humans cooperate, and how they communicate, and what that tells you about your organization and your systems. And we need to, if we care deeply about the reliability and the impact of these things that we’re building, We need to invest in those kinds of capabilities, those kinds of data collection and analysis and qualitative kind of work that almost no one is doing as of yet.

Matty: [00:27:39] How much of this, and again, this lionization of MTTR and similar metrics, I’m wondering, comes back from the decades of siloization of IT, where if you think about it, let’s take a step back and say, I’m not even inside the organization, if I’m a solution provider, I’m providing a service to you. Like, so much of that is built in that way. Whereas if IT is the cost center, the way that you’re saying, I’m providing this service to this feature team, things like MTTR are things that are baked into my OLA, into my way of providing that, because I have to give a number to show you that I’m doing the things that we agreed to do. And then that gets— and because it, like you said, it’s so pervasive into that’s just the way that we reason about tech in an enterprise is you have to have these things because that’s the only way that people will understand it. And you can go back to how much of that do you challenge to say, do you have to do that? And it still goes back to any metric you put that has anything tied to compensation is completely even less useful. I just want to give an example. I, when I was doing, I would be giving workshops in incident command and incident response. And one of the things we talk about is say, during an incident, you don’t litigate severity. Right? It’s not the time to argue about, oh, is this a Sev 1? Whatever, let’s just solve the problem. And I would give these workshops and talk to people, and afterwards people would come and say, hey, this is the biggest problem we have in our company. Like, when we’re— instance, we spend all this time on working towards what I call mean time to innocence, right? And I say, how does your tech leadership measure reliability of your system? How many Sev 1s did we have? So then your main job in life is to prove that it wasn’t a Sev 1, because that’s how your boss’s boss knows that you’re good. And I think MTTR is a— now I want to say a proxy, but it’s an example of that, because that’s a thing that 7 layers up the food chain, they could just look at this number on a graph and understand what it means. And so I think that’s one of the things I’m interested to know, like, how do we— maybe it’s like one of those things that’s almost— uh, to not turn it political, but like we’re waiting for kind of all the old politicians to die off and retire out so that we have maybe people who understand technology making our laws. Is some of that just a matter of we need to, like, as people, like, kind of progress, folks who have been doing this work this way get into more leadership positions, do people understand it better? Because this is changing, challenging the entire way that IT has been thought about for decades.

Courtney: [00:30:13] That’s hard.

Casey: So, yeah, and there’s a great framework to think about why that’s the wrong model for the work that we do, which comes from outside of software, where people who study organizations refer to software as the bureaucratic profession. Now, so we just wholeheartedly bought this from manufacturing, going back to Scientific Method Board in the 1912s.

Courtney: Is that, Casey, that Taylorism is a corporate disease that we haven’t developed a vaccine for yet?

Casey: We haven’t developed a vaccine yet for Taylorism. And so this is no industry as in, I’m making a generality about software engineering, but no industry has done a better job of instantiating the ideal bureaucracy, right? So if you’re a software engineer, you’ve got a person manager, a project manager, a product manager, a tech lead, an architect. All of those roles exist to take responsibility of your expertise away from you, which makes complete sense if you’re building widgets on an assembly line, like Courtney was saying, in a manufacturing world, because you don’t want anybody messing up the cast of some piece that has to work perfectly with other physical pieces. It is exactly the wrong model if you believe that what you’re doing is knowledge work. So, if you view software engineering as knowledge work, then the standard organizational layout that we have for software engineering at most companies that do software engineering is just wrong. So part of that, yes, okay, those companies have to die off, but we— those cultures have to die off. But part of that is also we need to put the right models in place to pick up the slack and to be successful, to crowd them out of the market share of the marketplace of ideas. And that’s where I think Devoid is really playing a fundamental role in kind of, you know, eking out a space where we can say, yes, that level of, that type of being data-driven is not a productive path. And that’s the wrong kind of capitalism to assume, I guess. But those bureaucratic methods are at odds with other business goals like system safety, like high availability and resilience and security. Those things are better achieved in a non-bureaucratic environment. And so, if the business really wants those, the business has to change.

Matty: [00:32:43] So, I have a couple of thoughts around that, and I’m trying not to be inherently too cynical, but you caught me on that day. But also, as everyone knows, I’m the one who says I always struggle with the dissonance between believing very strongly in that we can do things so much better with DevOps and cultural change, and also my belief that people are terrible. That’s me. Anyway, but it reminds me a little bit, and I think that’s where it goes to, is there’s the Upton Sinclair quote that says it’s difficult to get someone to understand something when their salary depends on them not understanding it. And I’m wondering when, Casey, everything you just talked about and saying this bureaucracy is counter to doing this type of knowledge work. And maybe to me, one of the best examples of how this went, that this was proven out in that bureaucracy is very challenging to get rid of is SAFe, the Scaled Agile Framework. What was that? That was, we have a whole bunch of people whose existence is threatened Like, they can’t do their job. So, it was, how do we back all this stuff that we know how to do and call it agile still? That’s safe, right? And that’s an oversimplification, but I think that’s the— and I know through cultural transformation work and digital transformation work I’ve done, your challenge usually isn’t at the very high levels of executive, and it’s not at your practitioner level. It’s that frozen middle, right? Because those are the people who tend to, in a large bureaucratic enterprise are the ones whose entire existence exists because of this thing that you’re coming in and saying is of absolute no use. How many people in enterprises, their entire job, if the word MTTR disappeared from this planet, they would have no reason to exist in their current job? That’s— there are people that have that role, and that doesn’t mean they can’t do something else, but I’m just saying, if you sit there, I think that’s one of the challenges people have is You can get your CIO and your CTO to maybe buy into these are our business outcomes we’re trying to do when your chop wood, carry water folks are like, yeah, this MTTR thing is bullshit. But they have a boss who has a boss who is the support manager and they have to put together the graphs that show that they exist because they make MTTR go down. So, this is not a new challenge. This is true of any challenge in a really large organization. So, I’m wondering if we’re— how the progress is going along those lines.

Courtney: [00:35:00] Sometimes it just takes I mean, the interesting thing is that reliability and security are no longer adjacent to the core product, right? Now that literally almost everything is delivered somehow online, those things are central to business concerns. And Southwest Airlines is a perfect example of that. And it’s not just because everybody thinks it was their technology that melted down, but it became clear that their entire way of working melted down when put under the kinds of pressures they had never quite seen before.

Matty: So, you’re saying in order to have a vaccine for Taylorism, we have to have a Taylorism pandemic, like Southwest, maybe?

Casey: The pandemic’s already here, man.

Matty: Yeah.

Courtney: Yeah, it’s already here. But what’s interesting is when organizations can get leadership levels to actually crack, to get that, like Netflix got that. In its DNA eventually, right? Like reliability is a core business thing. We tracked, what was the, what was like stream, what the hell was the metric, Casey?

Matty: [00:36:08] Like, yeah, stream starts per second. Sorry. I’m just really, I talk about that one a lot, so I’m glad that it was right.

Casey: Yeah. And it’s a great example of, I think, how to win. So one thing that sets Netflix apart there is that every engineer on their, and I’m, I haven’t been there in 5 years, so this is dated knowledge, but when I was there, On your first day as an engineer, you know how to look up that metric, how to see it, and you see it on dashboards everywhere. So the entire company could align around one metric that was correlated to value, right? Like that number goes up, then we don’t care what’s happening because that’s what the business needs to happen. And if it goes down, then we need to change it. That’s a— it’s so much easier to build data-driven.

Matty: Yes, it is. It’s so funny how hard this is.

Courtney: And it’s not the same metric across different organizations, right? Like metrics for Netflix’s core business metric is going to be different than Southwest’s core metric than PagerDuty’s core metric. And that’s the other thing. Go find the thing according to the business. That’s the one thing.

Matty: [00:37:08] That’s the other thing too. You’ll go to places and you’ll say, what’s your key business metric? And they’ll give you 12. And then you’re like, they can’t all be the key one. And I think it’s one thing I find slightly interesting is in my one year of doing public sector work at Red Hat. So one of the things I always ask people in this, I say, hey, do you know how your company makes money? Which is fundamentally asking what’s your key business metric? If you don’t, go find out. I’ll wait. Because if you don’t know that, how do you make decisions? And that, that then becomes a conversation about the stream starts per second or average cart size at Amazon or notification delay at PagerDuty, knowing all these things. And what was interesting is I had to adjust a lot of my talk track when I was talking to the public sector because they’re not about revenue. But what was funny is, so the equivalent in the public sector is, what is the mission? What’s the mission of the Texas Workforce Commission? What’s— and what’s funny is, in the private sector, so many people you talk to, when you ask them how the company makes money, they think they know, but they don’t, and they’re wrong. In the public sector, they all know. They actually know it better than in the private sector. They absolutely know why that agency exists and what the key business metric would be. They’re like, our job is to Cut checks to people that are unemployed. Are we able to process that? And you go in and they’ll be like, oh, how does Netflix make money? Oh, we make money based on— do you really make money based on subscriptions?

Casey: [00:38:24] There was a bank in California not too long ago whose slogan was, our first primary priority is customer happiness. It’s like, what’s your second primary priority? What does that mean?

Courtney: Yeah. So, I want to get back. I want to do some good bureaucratic terminology. So, I get often asked, okay, what do we replace MTTR with? Right? And this is where you start to understand the mental models and like how ingrained this is. Because I’m like, okay, I have things that you can replace MTTR with, but you’re not going to like it. Because I’m not going to— the fact that we ever thought we could have one metric that would tell us about the reliability of our systems. Was the problem. And so, now I have to fight you on that plane of existence, and then convince you that you can’t encapsulate the reliability of your system in a single metric. And so, then what you have to do is go figure out what was happening in your systems. And in order to do that, you have to talk to people, because people are part of your systems, right? All software systems are sociotechnical systems. There are people who write the code, who work with the machines, who try to make the machines, you know, do what they’re supposed to do when they’re not doing, and who interact with boatloads of more people than machines on an average basis. And that’s your system, right? And so, when your system stops working, it’s just as likely that something wasn’t working properly on the socio side of that system than on the technical side of that system. Which means you need people who are skilled at spelunking those kinds of systems. And they will be able to get you information and data on what’s happening. But that’s, again, to go back to this sort of data-driven organization, if you go to the CISO and are like, you need a cultural anthropologist, they’re like, ah, come on, right? But people are hiring incident analysts. Like, that is a thing that is happening. Um, and it’s a really interesting and fascinating— or they’re training former SREs or whatever to take on those kinds of roles, right? And the things that they’re doing are fundamentally social science skills, right? It’s interviewing, it’s collecting stories, building narratives, just finding patterns across social events and those kinds of things. And so That’s what you replace MTTR with. And that’s when people start to fight me usually. But the organizations that are investing in this, I believe, will have a competitive advantage over time. That’s going to be harder to prove out. And kind of the DORA argument, right? It was like, if you focus on things like CI/CD, actually being able to release to production whenever you want, and whenever you need, velocity is not at odds with security or reliability. That’s the argument that they actually collected data to make, which is super cool. And I feel like there’s this huge opportunity to make the same argument about investing in incident analysis as a competitive advantage. But I think the data for that are a little further down the road, probably.

Matty: [00:41:36] I think it’s tough. And just so we’re clear, like, I am not even trying to play devil’s advocate. I’m just thinking out loud because I, again, have seen MTTR is junk. But if we’re going to try to make that correlation, this is the problem of not having a— this is the industry’s quote unquote standardized way of measuring reliability. Because how do I even show that Netflix did this stuff and it helped their reliability if the way that we help their reliability is all conversational? If I don’t have a, hey, look, this is— they had the same number, was better than the one that wasn’t. And I think that’s the challenge is you still have to have some way to say, We move the needle in that way, but what is the shape of that? And I think it gets harder when it’s— number one, it’s harder when it’s not standard because how do you compare? And then even when it’s very qualitative, right? It’s very editorial. So I don’t know. And good thing I don’t have to solve this problem and you do. Yeah.

Casey: Yeah. And Netflix did try it while I was there and it’s my sphere of influence was availability stuff, managing the traffic team and others. There was a big push to have a quantitative improvement in availability. And one of my favorite things working there was that we had access to the data to show that in our case, going from 4 nines to 5 nines of uptime was horseshit because we had the data to show that people who use the internet don’t have 5 nines of uptime. And most of that is because of Wi-Fi routers and ISPs. So the investment on Netflix’s part to make the jump from, at the time our measurement was 4.5 nines to 5, would’ve been completely washed out in service of Wi-Fi. So why bother? It’s just that there’s absolutely no reason to invest in availability from that point of view for that purpose. Which was great for me because then we could invest those resources in other more impactful things like regional failover and things that we know would have an impact on uptime and customer experience and availability. So part of it is, yeah, like the companies that do fixate on MTTR and trying to metric their way towards better availability are going to fail. And the ones that choose other more qualitative methods are just going to do better. So at some point, evolutionary pressure hopefully will put us in a better position, but we don’t have to do it dumbly or blindly. We, we do have the information to kind of evangelize better methods and make the world better on purpose, not just by accident.

Matty: [00:44:23] Yeah.

Courtney: My favorite example of this, Matt, is that David Lay at IBM, he’s in the office of the CIO. He gave this amazing talk. At DevOps Enterprise. We were both there earlier last year in 2022. And he has basically transformed the Office of CIO at IBM, like a 12,000-person organization, I might add, to sort of obsessed with incident analysis and what they’re getting from it. And his playbook was straight out of DevOps. So, he had a skunkworks team, that nobody in upper management knew about that was doing really good, high-quality incident analysis. And everyone who was involved in that loved it, saw the benefits, really appreciated it, and was like— So, there was this, like, little locus of sort of analysis happening. And then, they had a really bad incident, and really bad by their context, right? At which point, everyone’s looking around and staring at each other. And that was his opportunity to go in and say, hey, Can we just try something different? At which point they’re probably like, sure. Like, that he took, it’s not the, I’m not saying this is everyone’s playbook, but this was a DevOps playbook for a while too, for various different reasons, right? And he now has monthly Office of the CIO learning from incident reviews, and he’s tracking how many people come to those meetings, how many repeat people come to those meetings, how many people view the incident reports, and how many times those incident reports are used in other things like training, code commits, other kinds of code reviews. He’s getting hundreds of people at the office CIO level to come to these meetings. And so, it’s like, it’s old, like, this audience should get this though. It’s like old-school DevOps from the ground-up cultural change, demonstrating the value to the practitioners, and then finding an opportunity and striking when the iron is hot. And that’s, but this is my favorite case study for that, because if somebody can do it at IBM, Good gravy. Other people should be able to do this.

Matty: [00:46:25] And it’s interesting because I was just thinking about that and I think you’re right. And I would, it was, it made me think about how many stories I have about something transformational and it’s always, oh, then Heartbleed happened and we were able to do a thing because we did this. So you, but it’s hard to do it when everything looks great. It goes back to a lot of stuff that was in the Stella Report about like when a decision would be made to say, let’s shut this off. And then nothing happened. And then in retrospect, you’ve got all the bean counters going, oh, we lost revenue because Kasey, you turned off this service and it turned out nothing was wrong. And it’s like that hindsight bias is hard. So it’s hard to make this when everything looks great to do that. So I think, and I think to Courtney’s point, it doesn’t mean wait for something bad to happen, but you need to be put, you need to know how to find ways to win. That’s my one sports analogy I’ll ever make is you say when you’re looking at professional sports, the skill level between the best player in the league and the worst player in the league is razor thin. The reason that these teams win is they find ways to win. And I think that’s the thing for us to do is say, okay, we’re doing this, but look for those opportunities to show. And then success begets success, right? You say, okay, we did this thing. It’s a lot easier because otherwise it’s all theoretical. It’s, oh, they did this at Netflix. They did this at— but I, the one thing I just wanted to mention is you said, hey, if they could do it at IBM, I think there are places that are even harder. And this is me in my living in flyover country Midwestern who’s, hey, by the way, there’s a whole lot of IT that we in our space forget about. And I used to have this argument with Alex at PagerDuty all the time. He was the founder, he’s my boss. And I would talk to him about ways that people were doing work. He’d be like, no, nobody does that anymore. And I’m like, Alex, yes they do. And he goes, no, I talk to enterprises all the time. And I say, Alex, You talk to enterprises who, A, have even heard of PagerDuty, and then B, decided they want to talk to you. And I used to joke and say my biggest insight into how IT got done is I was a single guy and I went on dates and I would go out and I would go on a first date with a woman who was a business analyst at Abbott Labs or something and talking about, hey, what do you do? What do I do? And everything I talked about with DevOps would be, there’s no way, we don’t work that way. We actually still do have production support. We still do this. There’s all of this work that we sit and we think about the coasts and we think about either the big finance on the East Coast or whatever nonsense is happening on the West Coast. And then we’re like, what about John Deere? What about— and then we’re like, oh, but Target. Okay. But for every Target, you have how many Sears, Roebuck and Company? How many State Farm? Yeah. And it doesn’t mean that these enterprises aren’t doing interesting things, but like, it’s a very different shift. And I think we need to keep that stuff in mind and eventually we get there. But yeah, if you can do it at IBM, that’s like half the battle, right? It’s the whole— to take the DevOps thing, it’s HP LaserJet, right? It’s Gary Groover again, right? Okay. You did IBM. It’s fine. No one has an excuse.

Courtney: [00:49:16] My favorite other, if you can do it at story is one of the other things we talk about in the report that we don’t really have data for yet, but I’m going to probably keep trying until I retire, is data about how unhelpful root cause analysis is and that mentality around that. And we don’t have the data, so I’m going to tell you a story. Along the way, I read a lot of incident reports, like all the time. And I would bash on Microsoft Azure all the time because part— still do this. So if you’re listening, folks, you drive me crazy because I can’t actually point to an indexed incident report on your site because it’s just a scrolling infinite hell site. But anyways, So, I was kind of— I’ve been pretty hard on them for a while now, and I was going through their stuff to look for an example of one of their root cause analyses to put something in the report, and it wasn’t there anymore. They had changed the entire format of their incident reports from RCA, which was always like in the title of the report, RCA number whatever, Azure thing, right? This is the Microsoft Azure team. They flipped over to a PIR, because we have to have an abbreviation for everything, which is— but it’s a post-incident review. And that alone was shocking to me. But then what’s happening is the quality, the depth, the nature of those reports is, in my opinion, now one of the more industry, like, stellar examples of what a public incident report could look like. The amount of detail, the attention to things, not just technical, but organizational. And that happened in one of the biggest cloud providers, right? And so, again, this was a combination of individuals on the ground level there, other people coming in, and, of course, consulting and doing those things. But they looked around, and they realized, like, this sort of boilerplate, here’s the root cause, we swear it’ll never happen again, and then something happens again, and then it was not achieving the goals that they wanted to achieve with publishing these types of reports. And so that’s another example of even without all the data, there’s hope, right? We can walk away from old, short-sighted ways of doing things, even in a very large organization. I think it’s an example of how other companies can do this.

Matty: [00:51:39] Because like, sometimes it feels like we’re having these pedantic arguments about, and I’m sure you’ve had it as much as with everybody who’s like, no, it’s okay to call it root cause because there’s multiple roots in a tree and blah, blah, blah. And I’m like, okay, if you have to explain to everybody what you mean by it, then it’s not serving its point. And so sometimes just changing the words changes the mindset. And also, I was just thinking when you talked about that change, sometimes it is like you can force a change culturally by making a small thing without asking permission. And the example I think about was, so at PagerDuty internally, So, by default, PagerDuty’s product called it an RCA, called it— you had a field that was called root cause, but in the template, you could call it whatever you wanted. So, I remember one of the PagerDuty SREs who is still there, so I won’t shout him out, but his name rhymes, it sounds a lot like Rich Lafferty. Rich would be happy to know this. He actually just— we had a lot of— there were all these conversations about let’s stop calling it a root cause in our postmortems, let’s stop all this stuff. And then just one day he’s, I just went into the template and changed it. And nobody ever said anything. And nobody ever came back and said, you shouldn’t do that. And then how many years later, it’s not called that. And then we kept trying to push it further and trying to like get the product folks to be like, can we change the default in the tool? And that was reaching too far. But that’s a place where those of us in the vendor space You have the ability because honestly, people tend to follow the defaults. And I honestly believe that if PagerDuty changed the format of the postmortem template and took out the word root cause and made the default contributing factors, but you could still call it root cause if you wanted, that could actually potentially change a lot of how places worked.

Casey: [00:53:21] But— I fully support small acts of vandalism to change the current direction.

Matty: So there’s your, that’s everybody’s homework. If you use PagerDuty, go sneak in and change your template to say contributing factors instead of root cause. See if any, number one, see if anybody even notices and yell, and much less yells at you about it. Yeah.

Courtney: All right. We’re probably running up near your time here. I will say the other thing that we found that I just want to throw out as another little bomb is we had the data to look at whether there’s any relationship between the duration of an event and the severity of an event, right? So PagerDuty and like other people, like you can, Your status pages will say, you know, this was a sev whatever, or it was a major, minor, whatever. And we have this longstanding belief, right, that there might be something still about the duration, right? Like, maybe that how long it goes on, is it worse, or do we put everything on the bad ones and they’re shorter? And the TL;DR is there’s zero relationship between those 2 factors that we collect about incidents. That’s the last one I want people to chew on.

Matty: [00:54:24] I do. We are definitely running out of time. And so I just want to, but we could have a whole other conversation because I’m really interested because I feel like so much of our other conventional wisdom of DevOps, which I guess we can say now because it’s so old that it’s now DevOps’s conventional wisdom, is batch size stuff where you’re like, okay, instead of having— the whole point of that was like, okay, maybe you have more frequent incidents, but they’re shorter. So it’s better than having this big, like, since we release once a quarter, now we just broke shit and it was down for 3 months. Hopefully not 3 months, but I don’t know, some shit is. But I guess maybe it’s still the thing to unpack there is that the answer, the statement is not more frequent releases make for shorter incidents, but they make for lower severity. Maybe, but it’s not, but I just thought it was interesting that I, in my whole way of thinking about it, still went to this. It’s ingrained that Being down for 30 seconds is fine, but being down for 8 hours is terrible. And if I’m teasing apart what you’re saying, is you’re saying that’s not necessarily true, that you can have really high severity incidents that are very short and very low severity incidents that are very long, or very low severity incidents that are very short, and God forbid, very high severity incidents that are very long. But I think we’ve all been there too.

Courtney: [00:55:42] Yes. Yeah. So, so severity is more important than duration. No. What that means, they’re entirely unrelated. No, they’re like, okay. They don’t tell you anything about what’s happening in your systems. Right.

Matty: But is it impact to your availability though? That’s in terms of what’s wrong. I guess severity, severity is also, isn’t negotiable. I was just going to say it’s relative, right? Totally negotiable.

Courtney: Yeah, sign it differently, and different people within the same organization will use it for different purposes depending on what outcome they want.

Matty: Yeah, and I guess this is—

Courtney: or it might change over the course of an incident.

Matty: And my question about that is, I’ve been so many places where severity is defined by duration, that the severity goes up the longer the incident is running. Others go the other way.

Courtney: They’ll call a Sev 1 to get everyone all hands on deck to find out what’s happening, then they make it a SEP 3. So which one was it?

Matty: There’s, I was thinking more about like when they’re talking about impact, they’re like, wait, this has been running for this. If they’re tying revenue to the downtime, they’re going to say it’s a higher severity because it’s been running longer. But also there is the meantime to innocence, which is the, it’s been running longer, so we better downgrade it. So it doesn’t look like we had this high severity incident running for so long. So it sounds like there’s—

Casey: [00:57:00] I also don’t know of any systems at scale where it’s even possible to divvy things up like that. Oh, for sure.

Matty: So for example, this is all made up.

Casey: If, if half of, if, if the western half of Miami, because of a municipal local law, is no longer able to view Netflix at all, so that half of the city is completely unavailable for Netflix. What’s the impact on that, on availability, on Netflix’s overall availability? Is that bad? What’s that? There, there’s just so many different things to take into account to, try to factor that out. It just doesn’t make sense to assign a severity to that.

Matty: Let’s say that.

Casey: And at scale, it’s gonna have that problem. You’re gonna have some group of users, whether because of an internal issue or an external issue, that isn’t able to, uh, to access your system 100% of the time for some period of time. So the whole model’s wrong. Just throw it all out.

Matty: Throw it all out. Tune in within when we bring Casey and Courtney on a year from now and they’ll tell you what the new model is. But in the meantime, just throw it out cuz it’s gonna take you at least a year just to even get rid of it. Without something to replace it with. Clean house. There you go.

Casey: [00:58:09] Does it give you joy? If not, there you go.

Matty: Marie Kondo your Nagios complex systems. Am I continuing to date myself by talking about Nagios references? I don’t even know what you would say now. I don’t know what the default thing is because everybody uses different— is it Uptime Robot? I always used to say— so this is just kind of just a side note about this— is one of my favorite things It was my only thing I enjoyed about Progress buying Chef is that realizing that Progress Software owned What’s Up Gold. And What’s Up Gold is still one of my— I always say, no matter how many cool, big, crazy monitoring solutions you put in place, every good sysadmin still has a copy of What’s Up Gold running somewhere on their network just to say, is this thing up? Which isn’t true anymore because nobody has servers. But as we continue to date ourselves. That said, this has been awesome. If you head over to arrestadevops.com/intothevoid, You’ll have this episode’s show notes, which will have links to things we talked about. If you go to arresteddevops.com/itunes, you can leave us a review in the iTunes Store if you want to help other people find the podcast. It hasn’t been called the iTunes Store in years, but I refuse to update my spiel or the redirect. So the Apple Podcasts directory, whatever, but reviews are fun. We haven’t looked for one for a while. If it’s really great, we might even read it on the show. You can subscribe to, you can check us out on Spotify and iHeartRadio. Even on Audible, anywhere that fine and less fine podcasts are made available. Thanks so much, Casey and Courtney, for joining me today. This was a lot of fun, and I’m sure we could have talked more and more. This has been Arrested DevOps, and remember, there is always DevOps in the banana stand.