Incident Retrospectives with Amy Tobey, Alex Hidalgo, and Rein Heinrichs

Alex: [00:00:00] Oh shit moments are just about my favorite.

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton. Today we’re going to be talking about incident retrospectives, but first, A word from our sponsors.

Alex: The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener, there is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app.

Matty: [00:01:22] Tonight’s topic is something near and dear to my heart, which is how do we learn after we have an incident? How can we share information with each other? What are the things that we want to do to have really great retrospectives after everything goes horribly, horribly, horribly wrong? Joining me today are Alex Hidalgo, Amy Tobey, and Rainn Henriks. So before we get into the meat of this, let’s, let’s talk a little bit about ourselves and why— well, I think We all— listeners to this show know I love to talk about myself, but really I’d like you all to talk about yourselves and why we’re all talking about this tonight. So Alex, would you like to go first?

Alex: Sure. So, uh, yeah, I’ve been involved with the technology industry in one way or another for about 2 decades now. I’ve been all over the place from a security engineer to network engineering, spent a long time in various aspects of IT help desk support and things like that. And, uh, I’ve been a site reliability engineer for about the 10— for the last 10 years or so, and during all of that time, I’ve had to respond to incidents of some size or some sort of another, and doing so in a coordinated fashion and then afterwards learning as much as you can from those incidents has been something I’ve been focused on for a very long time. It’s part of what I really care about, and I do want to take this opportunity to point out I have a really cool book coming out in just a few weeks that actually talks about how you can use different math to help you learn about some of this a little bit better. It’s called Implementing Service Level Objectives. It’s being published by O’Reilly Media, and by the time this publishes— or this podcast publishes, it should be available in physical copy.

Matty: [00:03:01] We’ll put a link in the show notes to where you can go buy it, and it has an awesome animal on the COVID Amy, what about you? What’s up?

Rein: Yeah.

Amy: I guess I’ve been doing this a while too. I started around 1999. I wanted to make mods to MUDs, and so I had to learn C, and somebody told me to go buy the C book, the one C book at the bookstore, and I bought it. And I was complaining about the compiler being really slow in Cygwin. That’s the hell I lived in at the time. And people said, oh, go install Linux. And I went and bought Slackware Unleashed. And put it on my computer, promptly wiped all the data from it that I shouldn’t have, and then spent time getting back to a working system, and then kept going and kept going and got a job doing this and have been going ever since, largely in the SRE space even before it was called that. And today I work as a DevRel and Staff SRE at Blameless.

Matty: And Rainn, what about you? What brought you here?

Rein: [00:04:05] Yeah, hi. So in 2009, I was an engineer at Reductive Labs, which later became Puppet Labs. So I helped make Puppet, the configuration management tool, and I would like to apologize to everyone for that. I’ve been sort of straddling the fence of both sides, ops and engineering, ever since. I’m a principal software engineer now. Also, I don’t have a book, but I do co-host a podcast called Greater Than Code. It’s good. I like it.

Matty: Greater Than Code is one of my favorite shows. Fun little— without going off, this is one quick little tangent, but of Arrested DevOps history, especially with our co-host Jessica Kerr. When Jessica and I met each other for the first time at Redeploy a couple of years ago, I was absolutely being a total fan because I love Greater Than Code so much. And we were talking and something came up about, you know, about this show. And then Jessica later, she’s like, well, I want to come on your show. And I was like, what? Yes, yes, of course. And then after Jessica was on our episode, she’s like, well, if you’re ever looking for another panelist, I’m like, you’re hired. Come be on our show. So it’s amazing. I want to take one quick moment. I’ll see if I can find it to link to it. But I believe that the genesis of this particular episode came, as all great things do, from Twitter. And I think it started with Alex and Amy having a conversation about incident retrospectives, and then Alex shamelessly saying Maddie should have us on the podcast to talk about it. And Rainn said, I’m completely like attacked that I wasn’t invited as well. And I was like, you didn’t even give me a minute to do it yet. So, but we talk about this stuff a lot on the Twitters, but I want to take a second to kind of talk about like, what are we actually talking about here. We’re talking about incident retros, and this can be, right, this can be all kinds of different things. We use lots of different words, you know, if we call them postmortems, we call them after-action reviews, we call them— but really we’re talking about the thing that happens after we’ve restored service, right? We’ve had an incident, something’s gone sideways, we’ve restored service, you know, so that’s incident response, and now we want to talk about it a little bit, right? So like what Let’s take a minute to talk about, like, what’s the point of having these retrospectives? What are we trying to accomplish?

Amy: [00:06:26] I mean, we could start with— I think it’s from OzPah— the idea that an incident represents an unplanned investment in your organization, right? And so we dump a lot of resources every time we have an incident of people time and sometimes software and sometimes cloud spend. To get through that incident. And so we’ve done all this investment, and have we gotten the most out of that investment that we can? And so one view of incident retrospectives is that it’s a way of harvesting what we missed during the incident in terms of capabilities for improving our organization, people learning things that maybe they missed in the moment, things like that.

Alex: I think for me, one of the interesting aspects of incident retrospectives is really just, it’s kind of a dynamic and organic growth out of everything that operations really is. You know, you start with, at the absolute base level, you have responding to tickets, responding to interrupts, general toil work, and then you get up to things that page you. And pages are really just very high, you know, high-intensity interrupts. They’re just high-priority interrupts. They’re just a ticket that really has to be addressed likely right now. And sometimes pages result in, you know, a simple thing that only you have to respond to, and sometimes they result in what we generally refer to as incidents, which is generally when you have to involve many people, when it’s a group effort, when it is severe enough that you want to make sure you record it in some way, and then after that, you want to learn from it. And it’s really just the most sophisticated end of responding to a ticket. You know, a ticket is just fixing a problem, and incident response is just fixing a problem, but you can do so in the best possible way instead of just clicking close and moving on to the next thing. And the best possible way that you can do that is by ensuring you’re having the proper conversations with everyone possible to learn what you can.

Rein: [00:08:30] I think also in line with what Amy was saying about an incident being an unplanned investment. John Osborne and Richard Cook also like to say that an incident is an encoded message that the system is trying to deliver to you. And when you’re responding to an incident, you’re not trying to decode the whole message, you’re just trying to decode enough to fix the problem, to get things back to normal operations. But if you don’t spend time to decode the rest of the message and unpack all of the information about your system that’s in there, then you’re wasting the money that you spent on that incident. You’re now wasting by not taking advantage of this opportunity.

Matty: And I think I wanna take that to even the next thing, which is like, Rainn, you said like during the incident, you’re worried about fixing the problem. And I’ll kind of challenge that verbiage a little bit, which is actually during an incident, I don’t even really care about fixing the problem. What I’m trying to do is restore service, right? I’m just trying to get things working, which is what you were saying. But like, I, but that’s the beauty of having this incident retrospective is now, we can say, okay, let’s think. So those are the 2 things that are necessary, like to do good incident response, you have to be able to say, I’m not gonna go down a rabbit hole of trying to figure out what’s the best thing to do right now and how do we fix this and should this cron job be like this and all that. You’re just saying, let’s get things working again so that we can disband, we can do whatever, and you can do that as long as you have this kind of, sociocontract within your organization that you will then take the time shortly thereafter to decode those messages, like Rain said, to say, okay, now we can do that. And the 2 things are very, I think, symbiotic. You can’t do one without doing the other one. You can’t say that you’re going to have an attempt to learn from incidents if you’re not going to have really good incident responses focused on restoring service. You know, but you also can’t just say, well, we’re not going to worry about all that during an incident. We’ll worry about it later if you don’t actually worry about it later.

Amy: [00:10:31] Right, but a lot of people do it anyway, right? Well, I mean, you pointed out that there’s like, there’s, there’s kind of like 2 horizons of pace of engineering, right? In an incident, we are operating at the highest possible pace of engineering, right? We’re cranking out the solution and pushing it into production as quickly as we can. In software engineering, we’re kind of somewhere a little bit more to the right of that and maybe doing quick iterations and test loops and then pushing that out to CI and stuff like that. And then once you get out to that incident retrospective, we’re in a much longer timescale where we look, we think deeper, and we’re doing what a lot of people think— what most of us think belongs in the architecture phase, right, is going and doing that deep thought on this engineering problem and figuring out all of the angles. That maybe we missed in that, in that very quick engagement.

Rein: Yeah, so you’re, you’re talking— that sounds to me like Kahneman’s Thinking Fast and Slow. So like during incident response, you’re thinking fast, but thinking fast isn’t how long-term learning happens.

Amy: [00:11:34] Well, that explains a lot.

Rein: Your focus is just on performance, right? You’re just trying to do the things that will get the service back up. You’re not focused on long-term learning.

Matty: Now, that being said, so the irony, and this has been brought up before too, right, is that we talk about having the thinking slow, right, the long-term learning, but you have to do it in a 1-hour postmortem meeting, and that’s the entire time you have to talk about this problem, right? Like, you know, so I think that’s an important thing to understand is that the— that one retrospective does not encapsulate all of the learning from that incident. In fact, it should probably be a jumping point to continue that, right?

Alex: I think that’s true, but I think there’s also something to be said about starting that slow learning during an incident. I’m a huge fan of using the incident command system, even if it’s very conceptual, right? It’s way too complicated as described by FEMA, et cetera, et cetera, to totally actually adopt for our generally much smaller scale software incidents. But, you know, the basic concepts behind it of delegation and organization, I think, are incredibly important. And a thing that I often do when I’m teaching people how to use the Incident Command System is like, you shouldn’t just be thinking about mitigation. I mean, yes, that’s your goal. You want to mitigate this problem. You want to restore service for your users and bring everything back. And but you can also ask other people to start things like start the incident state document, start the incident retrospective, start thinking about the future. There’s no reason people can’t go into a side channel and start filling out these gaps, start figuring out what have we learned already. Let’s capture everything right now while the responding engineers, which could be 3, 10, 20, it depends on the incident, while they’re focused on that, have other people specifically assigned to just watch the communications and start documenting this. You don’t have to wait for your incident retrospective meeting in 3 days or within 5 business days or whatever mandated by your organization. There’s literally no reason you have to wait for that. There’s no reason you have to pigeonhole it into that either.

Amy: [00:13:41] There are reasons. Um, I like that idea a lot because there’s the scribe part that you described, which is, you know, having assigning someone the work of keeping a record. But the other part I liked was, you know, if I have like a nosy manager or something and I need to keep them busy, I can give them that task.

Matty: Well, so the only thing that, that I struggle with, and I like that idea, Alex, and to me that’s a little bit I don’t want— I never want to say it’s more advanced, but having— and you’ve done this too— but having tried to bring a lot of organizations up to even the basics of incident command, sometimes it’s a little bit of like progressive disclosure, right, as you’re learning. You know, you can’t start by having this stuff overlap when you’re still learning, like, to have that focus. So for that to work, because what you’re still saying, you’re saying as a larger organization, this stuff is happening in parallel. But individuals are not doing both things at the same time. Because to quote the great Ron Swanson, do not half-ass 2 jobs, whole-ass one job. So—

Alex: [00:14:41] 100%. I mean, in my opinion, that actually— yes, you’re absolutely correct. Getting to that point means you have a fairly sophisticated— that’s not the right word because that could come across wrong, but you have a practiced organization that has done this before. You have responding engineers that have done this before. But exactly what you just said, you don’t wanna half-ass anything. In my opinion, using the incident command system correctly means you are the person doing this one thing, and we know you are the person doing this, and everyone knows you are doing this. I only wanted to point out that once you have enough people that are kind of familiar with these roles and how to do it, that you don’t have to wait for that meeting. You can have someone go off and start some of this process.

Amy: Oh, you definitely should not wait for the meeting, right? If you show up to the meeting and most of the analysis isn’t done, the meeting is a waste of everybody’s time, in my opinion.

Matty: Well, I mean, that can go both ways depending on how you treat the meeting, ’cause I’ve seen it happen and I’ve seen people where they say the meeting is basically the place to present the document to each other. And that is when you’re not having a conversation. Like, I prefer to run my retros a lot more like we’re coming in with the beginnings of stuff that we’ve done, but I’m not coming to you with a bunch of answers. Right, if I’m putting that together. What I want to do is I actually want to share what we’ve come up with and have that generate a lot of questions with us together. So I— but I guess depends on what the meeting is for. And there’s, you know, again, we’re— I feel like we’re going to name-check John a lot. You know, you said that there are not— a lot of times what we say is a postmortem is not a postmortem. It’s a, it’s a report for stakeholders, right? So is, is the point of this meeting for us to do some learning or for us to share and output, and you actually probably end up having both of those things. So I don’t know, it’s— there’s also not one right way to do any of this. There’s several wrong ways to do it.

Rein: [00:16:38] I mean, I think the idea that there’s a 1-hour meeting where the learning happens, it’s a little silly, right? We all feel like that’s kind of silly. I mean, the reality I think is more that learning is going on Starting at the moment the incident, quote unquote, began, and then for days and weeks and months afterwards in 12, 20, 30 different brains in different ways, right? That’s where the learning is happening. I mean, what is the point of a 1-hour meeting? It’s to do stuff that you can only do when you get those brains in the same room.

Matty: And maybe put a little bit of structure around the things we want to go start doing some more investigating on. Right? So, like, one of the things, and it’s a very wrong thing in the PagerDuty postmortem guide, that says— there is a slide that says the most important outcome of a postmortem meeting is to come to consensus on action items. I strongly disagree with that, right? But it could be, but it depends on what you mean by action items. If they’re action items like these are the Jira tickets to to go and change this parameter on the load balancer? No. But if they’re action items like, these are questions that we came up with and now we have some spikes to go and Rainn is gonna go look into this and he’s gonna think about it and we’re gonna talk about it or this team’s gonna do that, those are also action items, but we don’t think about them. That’s not usually what we think of as action items. Action items usually are Jira tickets, right? And that’s, That is an outcome, you know, but it’s not the main reason to do that. But I think I really like Rainn’s point.

Amy: [00:18:18] It is like, let’s be real, right? Like, the main reason most folks are doing incident retrospectives today is to go over the follow-ups. I’ve had people that I no longer work for tell me that that was the number one thing that had to be done on every single incident was full follow-ups, followed through and chased down and Basically wanted me to be an availability cop, which I’m opposed to. But, like, I just want to start with that, right? Let’s be clear about what we’re saying. We’re saying that it should not be that, but I think that, like, out in the world, if we went and asked all of the SREs we could find, we would find a high prevalence of people who feel that those follow-ups and action items are the main point of the exercise. And because they’re the easiest to tie to business value for most folks, right?

Alex: That’s exactly what I was about to interject with. Like, that is what executives actually care about. That’s what directors actually care about. And that’s— is that correct? We can argue this forever, right? It’s because we can sit here and say, look, there’s all this academic knowledge that we can look at and all these studies and, you know, everything that’s been going on in kind of, you know, a little bit of a segment of Twitter, I think, that some of whom are probably listeners to this, some of whom probably aren’t. There’s a lot of discussions about learning from incidents and safety engineering and how we can bring resilience engineering into all this stuff. And I wanna be very clear, I’m a huge fan of all of that, but practically on the ground, does that matter on a day-to-day basis to most people, or are they just being asked by their director, we need action items, We need to make sure this doesn’t happen again. That doesn’t mean that’s the best possible way, but I do want to just kind of, yeah, jump off what Amy was saying. Realistically, for the vast, vast, vast majority of people out there doing this kind of work and conducting these kind of meetings, that is still their end goal.

Matty: [00:20:17] And that’s why I said it’s not saying it’s not a goal, but when you say that is the main point of it, because sometimes you have to over-rotate on the verbiage you use. Because the hard stuff gets lost, right? You know what I mean? So like, if you, if you reinforce that the main reason you’re doing it is that— I also like this idea that those action items aren’t necessarily follow-ups. Now here’s the big challenge that I have with this, like, okay, we all came to agreement on the follow-up items, and then we will create these SLAs upon when they must be completed by. That does not allow you to then discover a week later you actually shouldn’t do that. Right? That’s what happens a lot is you’ll create— so this is one of the— this is just a general— like, you will sometimes see people say, like, okay, there will always be some agreement that within 2 sprints of the retro, we will have completed all the action items we identified in the postmortem. That’s really dangerous because you only have the 1 hour to talk about all of this. And those ideas, so you’re going to run into one of two problems that’s going to happen. Either you’re going to not have the ability to say, I can’t do that, or more practically, what’s going to happen is people will only agree to do action items they know can be completed within that amount of time. And they will not take— well, they will not— they will only venture ones they know are going to be absolutely correct. So I think we have to have that ability to say, we believe this is the right thing to do. But as an engineer, I should be able to come back the next week and say, okay, I know we had this agreement, but we started looking at it some more and actually completely changing from Azure to GCP is not a really smart idea.

Amy: [00:22:01] I actually have a really good workaround for that. It’s been working well for me, which is—

Matty: Working around going from Azure to GCP?

Amy: No, no. As the SRE who owns the process, My goal is simply to get the follow-ups and action items into a prioritization process, and then I actually let go of them at that point. And what that does is— what I’m doing is I’m explicitly turning the choices and control over to the engineering team and the product team to decide what’s going to go into the product, because I need product to make these decisions. And sometimes I have to dump maybe decisions they’re not ready to make on them, and maybe things that require more context from, say, the rest of the SRE team. But still, it needs to go through that process. And that’s one of those things we do by like having the follow-up items out, like happen in a meeting, and then the engineers agree to go do them, and it doesn’t really go through the product cycle.

Matty: Well, that’s, that’s why I’m a big believer that the people who prioritize the work need to be in the retrospective meeting, because that’s— I mean, and I think that— I don’t think that’s in conflict with what you’ve just said, but that it’s almost more like if you don’t do that, what you just said doesn’t work out so well. Because you’ve now put that into their backlog and it’s just some annoying thing that went in the backlog from Amy that the product owner is like, I don’t even know what this is, right? But if they, you know, so—

Amy: [00:23:20] Well, that’s the magic. I never have to hear that thing. They can just drop it. That’s fine with me.

Alex: I mean, that’s part of what I was trying to get at with the whole, you know, not everyone has this sophisticated of an understanding of learning from incidents and how to conduct these things properly. And people end up with just these action items because it’s what’s being demanded by executives and by leadership and by the business aspect of things and by the product side of things. And that’s exactly the point. We need to get to a world where more people are involved in situations where all these different groups are involved with the process, where all these different groups are involved, right? I didn’t mean to say that just because that’s how it happens everywhere, we stop. Of course we can make things better. The problem is, it is non-trivial. It is not easy to get buy-in across different organizations within your company. It’s not easy to get buy-in from the business side and from the leadership side and from the product side that this is how we should be approaching things. And that was really more of my point, is we need to acknowledge that potentially a large portion of the audience listening to this right now does not live in an organization where Everyone’s on board.

Rein: [00:24:33] And—

Matty: Well, actually, I got news for you. Nobody does. They’re like, this is not happening. And so these changes have to happen in multiple directions, right? And so that’s, I would like to get to some like kind of practical things we can do about how we can do retros better. But the thing I wanted to just to put a bow on that is there is sometimes a little bit of a tone that comes from, and I feel like I hear it a little bit in the RE side, which is, Resilience engineering happens in spite of management, not because of it. But I think that that’s— there’s some of us that are tilting at more senior leadership windmills around changing that kind of transformation because it kind of has to happen in both directions. You’re right. You’re absolutely right. Like, you can’t— we’re talking about how things could be and could be better, but those directives have to change as well. But let’s talk a little bit about like how to do these better, right? We’re doing them regardless of of what the outcome— I mean, the outcome matters, but like, how can we make our retrospectives more effective?

Rein: I was thinking about action items and whether the issue is that management, you know, executives don’t understand why they’re not important. But I was just thinking about, you know, if you’re a junior SRE and you’re getting paged 5 times a week at 2 in the morning for the same thing, and you have a playbook that fixes it every time, and you’ve got an action item on someone’s backlog somewhere that if they would just work on it, you wouldn’t have to wake up at 2 in the morning next week. And then telling that junior SRE that actually we’re going to stop worrying about action items. I don’t think that’s going to work either.

Matty: [00:26:21] I think maybe a way to redirect that statement is that action items are not necessarily restricted to Jira tickets. They can be investigation and they can be— and again, a big thing to me is I still feel that there is a danger in having SLAs around the action items that come out of an incident. And the reason that I bring that up is that is a very common thing that tech leadership feels will make things better. They feel like they’re doing things better by doing that, but it has a very negative effect of coloring the action items that people will actually sign up to do.

Amy: Well, that’s like, you know, a few months ago, one of our dear leaders in the USA said they were going to manage the numbers, and I believed them, and they delivered on it, right? Because you can always count on people to manage the numbers. What— when they have control over what the numbers mean and can game the system, right? It’s almost a way to guarantee that people will game the system.

Matty: And it’s not even malicious, right? It’s— and that’s not even done consciously, right? So that’s the, you know, and that’s a whole other culture conversation. But what are—

Amy: [00:27:26] what are the Deming thing, isn’t it?

Matty: It’s— I always— Nash-Pareto equilibrium. It’s Schaeffer’s, like, kind of people work to the—

Amy: to their—

Matty: to the numbers you give them to the detriment of your organization.

Amy: That one. Yeah.

Alex: Yeah.

Matty: So So when retrospectives are done well, like what are some hallmarks of that of like, and again, like let’s, we’ve been talking a little philosophically, let’s like specifically say like, here’s some good practices, like do this, say these words, I don’t know.

Alex: My number one goal anytime I am involved with or responsible for any kind of incident retrospective is to ensure I’m telling a story in some fashion. That’s how humans actually learn. As engineers, we often like to think that data matters, that numbers matter, that we can persuade people by putting a bunch of numbers in front of them or a certain graph, but that’s not really how human psychology generally works. We’re storytellers, and that’s the best way to, I think, get things across. Make everything a narrative. In the templates that I like to help people use, I do think templates are actually very useful for producing these kinds of documents. Not that every section has to be required, but as a guide, as a prompt. And I don’t see the use in timelines. I don’t, many templates have this like at this timestamp, this happened, at this timestamp, this happened. That’s not useful, but telling the story of what happened first and what happened next, That can be very useful. So I love having a narrative timeline section, for example. Don’t give me a Google Docs table with UTC ISO 8601 timestamp. That doesn’t do anything for me, but tell me in paragraph form what happened first, what happened next, when did this start? Did this actually start weeks ago when this certain bit of code was committed? Tell me a story. That’s how I think people really walk away from things well.

Amy: [00:29:37] I 100% agree with you, but except for the part about the timeline, because the timeline is like the outline for your book before you write it that keeps you on track as you write that narrative and makes it more compelling because it’s more likely— basically, the timeline is your basis of common ground with everybody who’s going to consume this report, right? When those— when things happen out of order and like most people have a very different view of how things occurred in the actual incident, and they come to this. The timeline is how you make sure that you are lining up with the best record we have of what actually happened.

Matty: So the timeline is supporting information for the narrative. Like, that’s how I was thinking. It’s like, I don’t think you don’t want to have it, but it’s not the main narrative. It’s part of a statement of facts. It’s the thing that helps you build your story. It could be an appendix, you know, for reference, but it’s not the place people enter the narrative.

Amy: Is by the way, I put it in the appendix once. It was told.

Alex: Okay, I would say have a timeline in your incident state document, right? Have it— have a timeline you’re keeping track of while you’re managing the incident. Personally not convinced a list of timestamps and, and, and exact occurrences in your retrospective, uh, really does much for anyone reading it, but I mean, we can just— we can just disagree there.

Matty: [00:30:55] I also don’t think it has to be comprehensive either. So like when I think a little bit about, well, ’cause not every single thing that happened in the Slack channel for the 3 hours of the incident is actually that interesting to look at again. So I kind of look at how, you know, just as a reference, I’m not saying you should use it, or the way the PagerDuty postmortem tool works with its connection into the Slack channel you use is when you’re populating it, you’re going through and you’re saying like, okay, these parts. And that can be a little dangerous because you’re sort of rewriting history, but also we all know that there’s plenty of things that occur that are not necessarily germane. You don’t know they’re germane or not at the time, which is why you capture everything. But, you know, kind of where that— I don’t know, there’s probably somewhere in between where it lands. It also depends upon your organization and how they reason about things. But I really like— I think Alex is right that it’s you construct it as a narrative. That also goes a little bit if we want to buy into my bullshit about incidents as a metaphor of post-traumatic stress, right? Like the way that we process— we have to, as an organization, you need to process this incident, and we process things by telling stories, right? That actually puts it into a thing that happened, and it potentially also makes it more interesting for someone to consume because one of the biggest problems with incident retros and things like that is people don’t give a shit about ones that didn’t happen on their team, which ironically are the ones they should read the most. I’ve had many conversations where they’re like, well, I don’t look at them unless I know I need to look at them. I’m like, how do you know what you need to look at? Right? You know, so if they’re interesting, people are more likely to check them out. So there’s a little bit of marketing in that too, maybe. It’s also way harder than just copy-pasting a whole timeline of shit that happened in the Slack channel. Sorry, not sorry. You got to work a little more.

Amy: [00:32:49] I feel like we’re talking about what the— almost we’re edging up on the shallow versus deep incident report debate, right? So I am in favor of shallow incident reports for some organizations, right? Like cranking out tons of shallow incident reports. Make them cheap, make them ubiquitous, get engineers doing them, whatever it takes. Make it easy, get those narratives, right? And then maybe the difference we’re having about like the importance of the timeline is I think the more complicated and the more in-depth the incident analysis is, the more critical it is that you tie everything together with that shared common ground in the timeline.

Rein: Can I just mention, I think it’s a mistake to think that there is one timeline. Fair. I think that when you’ve got 12 people let’s say in an incident channel, you have at least 12 timelines. And one of the ways that you can get to common ground and actually tease out a lot of the information in the retro is by asking people to give you their timeline and then asking people to compare their timeline with another timeline. So you They give you their timeline and then you can say, oh, well, at such and such a time, this happened. Did you remember this happening? Oh no, I didn’t know that that happened, or whatever. And then you can actually, by taking these multiple timelines and trying to synthesize them into one narrative, I think that’s where the real richness comes from.

Amy: [00:34:21] There’s a whole talk about it.

Matty: I was interviewing people ahead of time. I may paint myself in a corner on this one and it might not work. This is an idea, but I was when we’re talking about that, this might be where the narrative structure actually makes it a little bit easier because if what we’re doing is we’re putting it together in a prose narrative and now you’re saying, Rainn, this is what happened and you’re thinking about it that way rather than you looking at a logline of a whole bunch of stuff, you’re gonna— because we don’t communicate as log entries in a timeline as humans, so if I’m asking you to trigger in your memory Your memory is not a bunch of UTC stamp stuff. It’s a bunch of memories of narrative that happened. So, trying to collaborate and/or concatenate and synthesize these 12 different experiences together in a narrative form almost might tease that out. But I still think you do want to have that supporting data because sometimes you don’t remember. Sometimes even just looking at that makes you remember Like, oh shit, that’s right. Because— and I— but I think Amy’s exactly right that the more— the longer running, the more people involved, the more systems involved, the more important it is to have that reference point because it’s too much to keep in your head. If it’s an incident that took you 10 minutes to resolve and it was quick— and I— this is a big— Amy, you made a really good point. I just wanted to echo for everybody listening. One of the biggest anti-patterns I hear when I would talk to a lot of especially enterprises, sorry, not sorry, is like, when do you do post-warms? Only on Sev 1s, only on high severity incidents. I’m like, you should be doing them on everything. And I don’t care if your retro takes 5 minutes to do, but the whole thing is just being in the habit of always doing it. But again, the reason that they don’t want to do it is because they think that their process takes 7 weeks to do and they have to write this big document and do all this stuff. So I think that’s a thing to keep in mind. Yeah.

Alex: [00:36:21] I totally agree with the concept that people should be doing these for everything. I love incident retrospectives for tiny things. I’ve been on teams where we successfully got to the point where if we got a page, there was an incident retrospective. And even if it meant that we deleted that alert because we decided it sucked, right, that was— we tried to do it for everything. And you can absolutely get there. It’s, you know, like it’s a lot of work. On the flip side, don’t make me go off about the concept of severity levels.

Amy: Let’s tap that hot button.

Matty: No, no, there’s lots of other, I was just giving that as the example, but also those things tend to go hand in hand. That belief tends to go with tying things very closely to SEV 1 means something and severity levels drive behavior in ways that people don’t think about, all kinds of ways, you know? Yeah, we will have a whole other episode about that. It’s called Twitter.

Amy: I don’t know that every incident— like, that all these things should become incidents. Again, it comes down to one of these things that some of us have the right organizations at times in our careers where we can do that and it’s successful. I keep thinking about these people I talk to out there where that will never fly and the resources aren’t available. And it doesn’t matter how many incident reports you write because the organization is not ready to hear them. And so, and this is pretty common and something I think about that like maybe we’re not going to make progress on here, but like there are enough folks in these organizations where they’re trying to make progress against the system and change it, and writing a lot of incident reports of any depth isn’t going to do anything when they already know what needs to be fixed, and it’s just that there’s there’s an organizational bottleneck somewhere that prevents that information from flowing, from the feedback from happening, right? The product people have no idea, the leadership has no idea. They keep hearing about it, but all they’re hearing is, ah, endlessly, and so we’re stuck in this state, right? And that’s one of those things I think when we say do all of them, right? Those folks, it’s again like that do giant incident reports creates a barrier to where it’s more of going back to the resilience engineering thing and figuring out those, little small insurrections that can start to create the change to fire up these feedback systems.

Matty: [00:38:37] The place I would argue that slightly— I don’t even want to say argue— is that statement, they already know what needs to be done, but they can’t communicate it. I would make the argument that writing more incident reports, even if they’re short, getting in that practice will help you start writing things in the terms of business value, because I guarantee almost every engineer that says, I’ve been telling leadership this over and over again, and they don’t listen, sometimes it’s because their leadership are dumb and stupid and evil and awful. And sometimes, quite frankly, it’s because the engineer doesn’t know how to speak their language. It’s a love language thing, right? If I am trying to tell this to somebody and they’re not hearing it, that’s on me a little bit.

Rein: I don’t know if that’s a term I would use.

Matty: I have a whole talk called The 5 Love Languages of DevOps. That’s why I refer to that one. That’s why I’m saying where the idea is, it’s like, I can’t sit there and say I need to explain this in a way that makes sense when I’m trying to explain it to Alex when Alex has a different frame of reference than I do. This again is also from the idea of small insurrections, right? Like it’s little ways, like there’s no one magic thing, do this thing and everything will be better. So I think there are things that will come from getting good at quantifying and understanding what happened in an incident, even if it’s a little one, even if all you wrote is one paragraph.

Rein: [00:39:53] Yeah, so there’s a general rule that I use, which is whenever I want to make a change, I try to think of what would have to happen to make that change easy. Because a lot of it is about the context or the environment we create for that change to happen in. And so if we want anyone to try a new retrospective style, I guess the question would be, how do we make that easy for them to try? I don’t think that we’re going to do this new thing on every incident. By the way, we had 100 incidents this week, is easy.

Matty: Well, you don’t have to start by saying you’re doing it on all, but to be honest, doing it on the small ones is easier than doing it on the big ones. That is why I tell people to start with the small ones because the stakes are low.

Rein: I think there’s a Goldilocks zone. Because the big ones are very scary, but the small ones are very boring.

Amy: I like that Goldilocks zone. I’m going to use that Goldilocks zone.

Rein: But I think you should, A, I think you have to start by trying to get the organization used to trying stuff just in general. And this can be a process that takes 6 months. And then one of the things you can try is this new way of doing a retro.

Alex: [00:41:12] A thing I’ve had success with is finding people who have kind of, let’s say, bought in, quote unquote, on some of what we’re talking here and setting up facilitator rotations. Actually have people, people who are willing to go sit down with teams that may be on the totally opposite side of the company than they are, because you don’t have to necessarily know anything about the systems to help lead people in their own discussion.

Matty: In fact, it’s sometimes better if you don’t. You’re sometimes a better facilitator when— because a good facilitator is not invested in the content. So that’s a whole, you know, and I’ve run into that a lot of times that we have that challenge when we talk about like, oh, but we can’t have— but we— and this is why you end up with things where you end up with management as facilitators of retros, and that’s a whole other problem because they’re like, well, they’re the only ones who have the, you know, so, uh, and I agree 100% with— I like the Goldilocks zone idea, and, and it goes back to again Anytime the way you try to get change to happen by making a rule is not going to work out so well. So like starting by saying we are going to do retros on every single incident is going to be terrible. But saying like, let’s start trying them in these, see how that goes. And you also always, you know, I’m a big believer in stacking the deck for change. You start this with the people who are already interested because you’re going to have rough edges and the people who are already into it will hit the rough edges and they’ll sand them off. The people that already think this thing is stupid, as soon as they hit the rough edge, they’re gonna be like, I told you this thing was awful, and you’ll never get them again. So like, that’s where that goes. And then when you have some good, you know, that anyway, that’s like I cheat. That’s my answer to any kind of organizational changes, to cheat.

Amy: [00:42:55] There’s no cheating after you get out of grade school.

Matty: Yeah, it’s only cheating if you get caught. Sometimes cheating gets you promoted. Never take career advice from Maddie. What is something— here, I’m going to ask one question and then I’m going to kind of leave this if there’s any other things we want to talk about because we’ve— not any surprise to any of us, have probably kind of, you know, gone on about this a lot. But here’s my question. What’s something you used to believe very strongly about incident retrospectives that you have since changed your mind about and why?

Alex: That timelines are important. This, this, this kind of goes back to my point earlier, right? I used to believe that it was incredibly important to have this incredibly detailed list of exactly what happened exactly when with exact timestamps, and I’ve been pretty much convinced that you don’t actually get anything out of that. It may be useful backing data, Like, not to go on too much of a tangent, but when I talk about SLOs, I’m often like, one of the things you get is you only have to page off of like one thing, right? If you really build them well, you just page off error budget burn. And then people are like, but what about all the other things we need to know about? And I’m like, I didn’t say to stop measuring them. I said that maybe they’re not the most important thing. Maybe that’s not what we should be paging off of anymore. It’s kind of a similar thought, right? When I say I don’t think timelines are that useful anymore, I think they’re not that useful for the retrospective document. I think they’re not that useful for other people to learn from. Still have them because as other people here pointed out, they’re great starting points for establishing your narrative, but the narrative is the important part, not the actual timestamps. No one’s ever gonna learn anything from those timestamps.

Rein: [00:44:38] I think it’s a shame that that’s generally all that’s done with timelines because it’s— that’s the most useless thing you could do with a timeline, put it in a Google Doc somewhere and then forget about it. When you do cognitive interviewing, one of the things you learn is to reinstantiate the context that the person was in that you’re trying to talk about. And timelines are what let you do that. Because you can tell them, okay, so it was Friday, it’s 9 PM, you’re here doing this. Tell me about that.

Alex: Yeah.

Amy: What was your least thing you learned, Ryan Green?

Matty: Oh, um, you don’t get off that easy. Just responding to Alex.

Rein: There, I think there’s— I’m trying to figure out what the most recent one was, because there are so many.

Matty: What’s the most scandalous one?

Rein: I mean, we could talk about why root cause is a lie, or we could talk about why the 5 Whys don’t work. We could talk about why correction of error doesn’t work.

Matty: [00:45:38] And these were all things you believed at one point?

Rein: I believed in root cause at one point, for sure. The most recent thing that I believed about retrospectives that I no longer believe is that the hour you spend in the meeting is the most important part of the retrospective.

Matty: So you guys, y’all listening can’t see it, but just on mute on the video, Amy went, damn! I’m pretty sure the words that came out were, goddammit, that was mine!

Amy: Pretty much exactly. Mine’s a little closer though. This is actually just close to that, which is, I’m actually not in favor of the independent meeting for incidents, especially at places that have a standing load of incidents. So, my— probably my favorite experience at GitHub was running the weekly incident review. So, you know, it was a big enough org with a big enough infrastructure that we usually had stuff. So, every week, I would review all of the incidents that were outstanding and go through and consult with everybody. And get it all ready. And then what we had was, because it was the same day every week, is we had this pace of everybody, product managers, engineers. If the engineers couldn’t make it, their managers showed up and we stepped through incidents, sometimes 4 or 5. I think the most I ever did was 10 on a day, and then I defer the rest of the next week. And, um, you know, I found that to be, uh, really effective compared to the, the trying to schedule people every, you know, after an incident. And, you know, people are freaking out, they’re trying to fix stuff, they’re busy, they’re stretched. That’s probably how we got here. And, you know, trying to just schedule it is a huge pain, but the standing meeting creates a cadence of caring about incidents. And then it’d be— and then if you’re careful with it and your facilitation works the right way, it becomes almost something that people want to go to. They want to be there for that meeting because they always learn a ton because we would cover so much content and so much ground. So I would call an individual who responded to the incident, say, what happened? Give, you know, give us a short spiel about what you saw. What are we going to do about it? More still in narrative form. And then we did all of that follow-up stuff for the most part kind of out of band was where it would happen. And so the focus in that meeting was about sharing what happened and what the context is across the organization.

Matty: [00:47:55] I really like that. And I think that goes into like the thing I want to kind of have us touch on before we wrap up, which is about learning. And I think the thing I’m taking away from that example you just gave is that we’re learning all the time. We’re not learning ad hoc, maybe on Tuesday afternoon, maybe it’s Thursday morning, maybe it’s now, whenever we manage to get our calendars aligned to think about that. It’s just, it’s a thing we do every week. We talk about what we learned, right? You know, and we’re continually doing that and it becomes encapsulated and embedded into our regular work. So how do we— I mean, again, so yeah, in 10 minutes or less, how do we completely become a learning organization? But like we talked about, you know, that yes, there may be reasons that we do retrospectives to come up with some actionable things. But if we also want to be able to learn from them, what are some of the things like we got some— we got an actionable suggestion from Amy about having this kind of cadence of review. What are some other ways that we can make this more not only about learning, but more about learning than it is today?

Alex: [00:49:06] Asking the right questions. That’s what it’s all about, in my opinion. Why? What’s that? Yeah, why? And then ask it 4 more times.

Matty: How?

Alex: You know, just, you know, again, one of the reasons I’m a fan of actually using like document templates for this kind of thing is to help prime people. To help populate them with some suggestions for things that they should maybe be thinking about that they may not have thought to think about. Things like, I love questions like, where do we get lucky? Which covers, in my opinion, things like, you know, who happened to be in that channel, right? Who happened to be up at 8:00 PM their time but happened to still be logged in? You know, who did we not have to escalate to yet was still available? What was documented well? What were we surprised to find out? Oh, we knew this even though we didn’t think we knew this. You know, just asking those kind of almost more cultural, people-oriented questions. And then also figuring out how can we make sure we don’t have to be lucky in the future? How can we ensure that next time we don’t need, you know, this particular person to happen to be watching this particular channel? You know, that kind of thing.

Matty: [00:50:22] Can our template with question prompts include the question, what questions come to mind that are not on the template right now? Because I think that’s, that’s the danger of the template is because it makes us feel like those are the only questions. And I do like the idea that, that, that a retrospective should raise more questions than it answers. Is that actually factually true? Probably not. We actually want more answers. But when we say words like that, that enables people to feel empowered to ask them. So I think that’s really good. And Alex, I really like your examples of those are the types of questions that are still relatively open-ended, but they’re still about what happened, right? They’re not about like 5 times of why. I like that.

Rein: I’ve got a couple things. The first one’s a question you can add to your template, which is How do you think our priorities should change as a result of this incident? I like that. A lot of time what happens with action items is you’re telling people to do stuff, but you haven’t told them why they should care.

Amy: [00:51:28] Yeah, I’ve run into that issue a few times.

Rein: So, I mean, generally speaking, if the priorities are in place, when people know what’s important, they will do it. So I think that if you want to get the action items done, what you have to focus on is making sure people know what has changed about what’s important.

Alex: The problem there can be, unfortunately, that, you know, again, the people expecting these reports may have a totally different concept of what is important than the responding engineers.

Rein: Wouldn’t you like to discover that though?

Matty: Yeah, just because you’re asking the question, I mean, and I guess you have to— and I do actually think the way that Rainn put it phrases it the way I want, which is not, you know, you’re saying, how do you think? This is asking their— it is definitely an opinion question. It’s not, what should we do differently? It is, how do you think they should change? And to your point, right, and Loser may look at that and go, No, because there’s context they don’t have. There’s a reason they are the way they are. Or they might go, oh shit, this is actually helpful for us to know.

Rein: [00:52:40] You don’t get a lot of the oh shits, but if you get a few, it was a worthwhile question.

Amy: Right, right.

Matty: Get one a year, you’re good, you know, or probably somewhere in between.

Alex: Oh shit moments are just about my favorite, both when talking about incident retrospectives and just in general. Anytime you can prompt someone to have a moment where they are able to take a step back and be like, wait, what? Oh, shit. That’s almost always good.

Amy: That’s what I consider my main metric for my work is how many light bulbs I get to go off in various folks around me.

Matty: So I guess to kind of bring this— we’re coming up to the top of our hour.

Rein: You want to know my second one?

Matty: Oh, you had more than one. What’s your second one?

Rein: So my second one was, I was thinking about how we were talking about, well, should you do these on— so if you assume that you don’t have the resources to do one of these deep learning teams or learning reviews on every incident, you know, if you have 5 a day, you can’t spend 40 hours on each one, right? So then the question is, how do you select which ones? And we’ve talked about maybe like Goldilocks zone incidents that aren’t too scary or too boring. I think we can do better. And going back to the first thing we said where an incident is a forced opportunity for learning, one of the questions you can ask is, how much do we know about what happened and how much more could we learn if we tried? And if you can try to find incidents that have a higher potential for learning, you can focus on those. So then the question becomes, How would you do that? Well, there are some cues you can look for, I think. So one of the cues you can look for that an incident has some potential for learning or higher than usual is when people are confused, surprised, or frustrated. They’re always confused, surprised, or frustrated, but there might be some that stand out, and those are good opportunities.

Alex: [00:54:44] Yeah, I think that is an excellent point. ’Cause I love discourse and conversation around, you know, kind of platitudes like you can learn from any incident. No, that’s actually not true, right? Like if you work for, let’s say, a major website provider, you’re gonna experience DDoSes a lot and sometimes it takes your CDN a few minutes to catch up and, you know, so you have a few minutes where things are kind of slow and, you know, is that an incident you can learn anything from or is that just how your vendor operates? On the other hand, sometimes these DDoS attacks are very complex and interesting and attacking multiple channels. And, you know, and so yes, sometimes, but sometimes it’s just a DDoS and you go down for 5 minutes. Is it really always worth the time to schedule a meeting and try to—

Matty: I would also go down this whole, like, do you have the time to do deep learning on every incident? It goes back to Amy’s thing about chat. Like, sometimes it is, I just, and we are, some of this we are actually doing, we just aren’t calling that. Sometimes you might just be writing that in the ticket. You might write a paragraph of the thing. The risk that I have with that example is, well, we always know that it’s like this, so I know I can’t learn from it. Maybe all you do is take 5 minutes, maybe less. Maybe it took maybe a minute to go, yep, this looks like the same thing it always is. But if you take the second to think about it and then go, oh, these seem to be happening more than they used to or something, I think you can always learn something. I don’t think you need to spend 10 times as long as it took to like flip the switch on the load balancer to talk about it, right?

Rein: [00:56:19] But one thing you learn in qualitative research is to try to find the surprising in the mundane and the mundane in the surprising. So it is definitely true that there might be something in those mundane, you know, those mundane DOS outages that you could learn from, but you got to start somewhere when you have limited resources. Yeah.

Matty: So let’s just close on that. Go find the mundane in the interesting and the interesting in the mundane and write good narratives and, you know, ask why 10 times, because if 5 are good, 10 must be twice as good. But that is bringing us to the end. If you go to arresteddevops.com/retrospectives, you’ll get to our show notes. Which may or may not have things in them. At the minimum, it’ll have a link to Alex’s book, so go there and buy his book, and maybe we’ll find some other stuff. If you go to arrestdevops.com/itunes and you leave us a review in the iTunes Store, ostensibly that helps people find our show, someone told me once. I don’t know. Maybe we’ll read your review. We probably won’t. I mean, we’ll read it. I just mean we might not read it on the show, but we might. When you write nice things about us, by the way, I’m the only one who reads it and it always makes my day. So that’s cool. And if you’re into Spotify and iHeartRadio, I guess we’re on those platforms too. So you can listen to us there. Alex, Amy, Rainn, thank you for spending time. This was a great conversation. I had a good time. I hope you did.

Alex: [00:57:47] I did.

Amy: Thank you.

Matty: I mean, this was basically what we usually do on Twitter. We’re just doing it with mouth words instead of typing. So it’s a change.

Alex: Look, everything, including incident retrospectives, are about having better conversations. That’s really what it all boils down to, and that’s what we’re trying to do right now.

Matty: Feel free to take those conversations up with all of us on Twitter. If you go to our show notes, you’ll find everybody’s Twitter accounts, or you probably could find us other ways and stuff. I’m Matt, @MattStratton. This is Arrested DevOps, and remember, there is always DevOps in the banana stand.