Matty: [00:00:00] I don’t care how smart you are. I don’t care how experienced you are. The real world will destroy all your plans. It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matty Stratton, and I’m talking with Arup Chakrabarty of PagerDuty today. About DevOps transformations. But first, we have a word from our sponsors. Chef is a community of professionals practicing DevOps every day. We are making, proving, learning, and shaping the future.
Arup: We are known for welcoming, encouraging, and liberating others to do the same.
Matty: We do not talk about change, we do change. Join the community and learn about our solutions at chef.io. This episode is brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 120 common infrastructure tools such as Chef, Docker, and AWS so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. The worst time to learn about incident response is during an incident. Don’t wait for an outage to strike before getting started. The PagerDuty Incident Response Training Course is now open source and free for everyone at response pagerduty.com. Based on the same training that PagerDuty employees go through, this course will show you how to streamline your incident response process, turn chaos into calm, and demonstrate the role of an incident commander. So what are you waiting for? Go to response pagerduty.com today and check it out.
[00:01:52] The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener, there is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app. So let’s say you’ve bought into this whole DevOps thing. I’m sorry to tell you that you can’t just snap your fingers and ta-da, you are the DevOps. My guest today has a lot of experience helping guide organizations along the way. So, Arup, can you tell us a bit about yourself and your experience, and then we’ll dive right into it?
Arup: [00:03:04] Yeah, so, uh, hi everyone, my name is Arup. I’m currently a Director of Engineering at PagerDuty. Uh, before PagerDuty, I’ve been at large-scale consumer companies like Amazon and Netflix that have been doing, you know, the DevOps for a long time. And, you know, one of the things that really drew me to PagerDuty originally was this idea of helping a lot of our customers achieve a similar level of amazing business results by going through these kinds of transformations. And so, you know, I feel like I’ve seen how the best and the brightest do this, and that’s something at PagerDuty that I get a lot of joy out of, is helping a lot of our customers go through these transformations and supporting them along the way.
Matty: Awesome. So let’s start with the million-dollar question, which is, where do we start when we’re trying to effect a change like this in an organization? What’s step zero?
Arup: I think the first— and I’ll probably sound like a broken record here, but it’s always start small. It’s always start with a single team or a single project or a single codebase or something where, instead of thinking about it as the the, like, well, how are we going to get this entire 100,000, 10,000-person department doing the DevOps? Instead, think, well, how can I start with my team? How can I start with an area that I have direct influence over? If you’re, let’s say, in a management position, that usually looks like 1 or 2 teams that you work with. If you’re in an engineering individual contributor position, that probably looks more like, let me start with this one project or this one codebase. Those are all great places to start. So, that’s step zero, acknowledging that the best recipe for success is you want to start small. I think after that, it’s really starting to figure out who are your champions that have kind of the same level of conviction that you have around DevOps and a different way of working. Who in the organization is like-minded like you? That usually means that you’ve probably sent out some emails in the past of ways we have to change, and those are the people replying with the, like, you know, fuck yeah, let’s do this, I’m very excited. Um, or maybe you have to build some champions. And what I find helps a lot there is giving them a lot of context around, hey, like, you know, here’s a crazy change I’m thinking about, but here’s why I think this is a change that we need. Um, so, you know, I find like starting from a— from more that, that, that starting small and then figuring out who are your allies in this journey, like, that helps so much. Because something I tell a lot of, you know, leaders that I work with, that, you know, a lot of this stuff, as you get further and further into it, you’re going to feel really lonely sometimes, where you’re going to feel like you’re the only one who’s advocating for these changes. And if you can build up some of those allies early on, it makes the rest so much easier.
Matty: [00:05:54] I think that’s, that’s a really bang-on point about that. It can, it can feel like a lonely journey. And a lot of times I will, when I talk to people who are going through this or starting the transformation, I’d say it makes me think about advice that people would give to someone who was making a movie or writing a book, which is you better love this because there are gonna be days that you hate it. And you’re gonna— there are gonna be days that are so hard and you need to be able to get yourself through it. And if you care about it and it’s important to you, you’ll be able to pull yourself through. But it can feel very lonely. And I really liked that advice of, rallying some allies, having someone who can maybe shoulder some of that for you when you’re having those tough days, when you’re having those tough times. Because sometimes it does feel like tilting at windmills, right? You’re, you’re going to be, you know, and again, it’s, I think, early, early on in the conversation, it’s a little easier. And it’s, it kind of goes from maybe an initial kind of push that’s a little rough, and then it’s, it’s not so bad for a little bit, and then it it gets worse before it gets better.
Arup: [00:07:06] Yeah. And, you know, something I’ll add there is that I find, like, you know, if you— like you said, like, if you really love something and you have a lot of conviction around it, in a really cool way, that’s actually what gets others to start having some conviction around it too. Like, it almost can become infectious. But, you know, one of the things I tell a lot of leaders is, you know, kind of counterintuitively, if you start going down the route of some of these changes, you’re likely to have some negative business impact in the short term. And, and most leaders say like, you know, how they usually manifest is like, you know, increased downtime or, or incidents or something scary like that. But in the long run, that’s where you achieve better results for your business and for your teams. But you’re right, like you have to have that conviction and that, that love for it because it’s gonna— you’re gonna have some very lonely days. You’re gonna have some days where you’re gonna feel like you’re the only sane person in the room, and that’s perfectly normal. And the more conviction and the more people you can have up front, the easier it makes those, those dark days.
Matty: [00:08:09] So with that, we’ve had a couple, but what are some actionable tips, some tips or tricks besides rallying some allies? But when you’re trying to help people understand this change, because I think that’s a big key part of it, but what are some action items that our listeners could take away from?
Arup: I think the first thing is, if you’re a leader that’s proposing these changes, really give the context around why you think this is the better way to work. One data point that I really like looking at is number of deploys over a certain period of time, and really tracking that as a metric, and over time, trying to increase that. I find for a lot of teams, they’re doing, say, yearly releases. Again, going back to start small, don’t jump to continuous deployment. Maybe that means you need to figure out ways to move from yearly to quarterly, and kind of figure out what does that look like. But the reason I harp around number of deploys, for a lot of our customers even, is that I think that’s actually a really good proxy for the operational maturity for other aspects of an organization. What does your incident response processes— what do those start to look like? Or, as an engineer, how much ownership can you really take from an end-to-end codebase operational ownership standpoint? I think number of deploys is kind of the right metric to manage around in the beginning, if you can’t come up with anything else. If you’re only trying to optimize for one thing, that, I think, is a good place to start. If you feel like you have a good handle on that afterward, I think the next best place to start is really trying to figure out, as an organization, what do your SLAs, SLOs, and SLIs look like? If you’re not familiar with that, I highly recommend reading the Google Site Reliability book that they released a few years back around how do you set appropriate ones, and then really figuring out over time how do you manage against those metrics. That would be another tactic that I’ll call out here is, as much as you can, manage against metrics and use that as context and to be able to demonstrate some of the results to the rest of the organization. One really powerful thing I saw in the previous companies I worked at, and even at PagerDuty, was as we were going through some of these transitions, there were probably like 2 or 3 metrics that we were looking at, and most of them centered around things like downtime, number of deploys. At one company, we actually had, you know, amount of revenue that we were not losing anymore as a result of increased uptime, and we were tracking that over time. So week in, week out, you know, myself along with my boss and sometimes even some of my teams, we’d look at that metric, and if it went up and to the right, in the right direction, we ask ourselves like, well, did we do anything to change that? And sometimes the answer is no, we didn’t do anything. It’s a result of a bunch of positive changes we made earlier. Or sometimes it’s yes, here’s a great step function because we fixed this one problem. Um, and then the reverse happens where, yeah, we screwed up last week and here’s what we learned and here’s the postmortem. And I find those metrics can be incredibly, uh, focusing and it drives kind of the right behaviors of, of making sure that people remain focused and have their eyes on the appropriate prize.
Matty: [00:11:26] I absolutely agree. I think it’s— you can’t know if you’re moving the needle if you don’t have a needle, right? One thing I want to stress is there’s not a magic number for deploys a day, right? There’s a number that’s appropriate for your business. So just because you hear that Amazon is releasing 10,000 times a second does not mean that that should be your goal if that’s not appropriate to your business outcomes.
Arup: And I No, I absolutely agree, Matti. Something to really point out there, something I’ve noticed for a lot of companies is that DevOps is different to every company out there, and that’s okay. I love what you just said, because I think there is a lot of DevOps FOMO out there, and I think there are a lot of these great companies doing just amazing things, and they go to conferences and they talk about it. Something I always point out at a lot of conferences when I’m presenting on behalf of PagerDuty is that you’re not PagerDuty. You’re not any of the companies that you saw at a conference, and you should have different aspirational goals. So, exactly what you said, Matty, if you’re at yearly deploys, that doesn’t mean the magic number is 5,000 times a year all of a sudden. It just means probably better than what you thought, so slightly more. And you should have those long-term targets for every single one of your metrics. And if you ever come into a meeting and say, well, this other company does it this way, so we should do it that way, you have now alienated all of your stakeholders and you’ve probably created a bunch of enemies at the same time.
Matty: [00:12:59] Yeah, and I agree. When I was doing consulting and was working with companies around transformation, there were— and granted, this was, you know, 6 years ago or whatnot, so— but I’m sure you would run into it still today— is I talked to stakeholders and they’d say, I don’t want to hear anybody else coming in and telling me how Netflix does stuff. Because guess what? I’m a bank, right? Or I’m, you know, we have a different scale. And I think the other thing to bear in mind with DevOps FOMO is also always remember that what you’re seeing people present on the stage is success. It’s the rosy picture. And so if you are listening and you are some type of someone who presents, actually much more interesting stories are your failures. But that’s a whole other conversation. But, and remember that some— the reason that you’re hearing success stories is, is for multiple reasons. One is just from a personal perspective, we want to be— we feel better talking about things we did well. The other thing is some organizations aren’t allowed to tell you about their failures. And because PR departments or whatever are afraid that if someone finds out they ever made a mistake, then they would lose reputation and trust, which is also kind of not great. You know, I mean, if you’re telling a story of a failure where, you know, you, I don’t know, checked keys into GitHub that you shouldn’t have checked into, yeah, maybe, you know, that exposed, you know, 13 million people’s ATM PIN numbers, right? Yes. I mean, again, we can talk about blamelessness. That’s great. But maybe you don’t necessarily want to— I can understand how you don’t want to advertise that.
Arup: [00:14:36] No, I, and I totally agree that like I like the example you gave, like, we’re not insert company here, we’re a bank, or especially if you’re in a more regulated industry, a lot of these things are a lot harder. But the kind of flip side of that is you might not have the same constraints as some of those larger companies. So especially, let’s say you’re a smaller company, yeah, you don’t have a 300-person engineering team focusing on, say, that one problem, but also the reality is your codebase and the complexity of your business is much smaller. In the case of being a bank, your customers are demanding that consistency of the ledger is the most important thing in the world. And so in a good way, if you go back to your customers and say, hey, we’d love to deliver you all these features, but what do you care more about, your bank balance being more accurate or a better way to access your account? Guess what? I’m going to say that number better be accurate. And if I get those other things, that’s great. But again, that’s a different— constraint that other businesses have. So, you know, I always tell people is like, as much as you can exploit those opportunities that your unique business has in the context of any transformation.
Matty: [00:15:54] All these things need to be tied back to business outcomes, which is something we’re not necessarily used to. We’re used to being incented and understanding within kind of our microcosm of the company. And it’s— I often say in my talks, I’ll say, do you know how your company makes money? If not, go find out. I’ll wait. Because you— and yes, I can understand you, maybe you’re not a for-profit organization, but you need to know what are the drivers of your business. And it’s not surprising, I guess, but maybe sometimes a little frightening to how oftentimes we don’t really know. I remember when I was working at Apartments.com, there was one day we were doing kind of an office move. And so My team got, got put on a different part of the floor a little early and we were sitting next to sales. And it was fascinating to spend a couple days listening to sales calls. And I said, wow, I understand so much more about our business because of this. Whereas before I was just looking at it as it’s a bunch of ones and zeros and there’s some computers and they should be up all the time because if they’re not, that seems bad. Right. You know, but understanding quantifiable things about what’s the value of a lead, and if we’re not sending them, what does that cost us? It helps make decisions, again, like you said, if you have to be able to sit down and say, which is more important, prioritizing a feature that helps search or prioritizing a feature that ensures we deliver leads, which is what makes us money?
Arup: [00:17:21] Yeah, and I think that’s one of the unfortunate side effects I’ve seen from a lot of technology departments being so far removed from— I hate using the term bean counters, but I think that is those are important relationships to build in any business where, you know, if you’re, if you’re an engineering leader and you can’t answer the exact question that you just asked, which is how do you make money, then you fundamentally don’t understand your customers and therefore you don’t understand what’s important to your business. And so, you know, another, you know, tactical item I always tell folks is go talk to your finance team, go talk to a product manager that can help you understand your customers a little bit better. And then as you can do that, then if you can map that back to a metric, that’s even more powerful. And, you know, I can give you 2 interesting examples. When I worked at Amazon, we had the order graph. Again, it now sounds obvious, but they didn’t just stumble on that. It took them many tries to figure out what was the one right metric, and it was orders over a period of time. At Netflix, they’ve talked about this publicly, which is they look at the number of stream starts per minute, and they know that there’s an expected range that that should be in, and if it’s outside that range, then chances are there’s an issue. At PagerDuty, for us, there’s really 3 metrics that we look at. One is the availability of our endpoints. Then it’s the time it takes us to process from an event to incident to notification. And the last is the overall web experience that we have as measured by our servers. And those are 3 metrics that we basically— I tell a lot of my engineers, if we’re looking to improve something like reliability at the company and it doesn’t map back to one of these metrics, then we’re either missing a metric, which can totally happen, or you’re working on the wrong thing, so stop. It’s so hard because I find that defining those metrics, even I can talk about the PagerDuty case, that actually took us years to get to a set of 3 metrics that we really cared about. And it’s not because we didn’t care about our customers or anything like that, but we spent so much time debating what were the simplest possible metrics and most straightforward ones that our customers cared about that we could easily instrument and measure. Because look, you know, if I could measure like the amount of love that our customers have for our products, I would love that. I just don’t know how to get it, you know. Like, am I gonna sit there asking our customers on every single, every single time they use the product if they loved it or not? No, of course not. So I have these proxies, unfortunately. And I think that’s the, the other thing that I’ll add around any metric is that, you know, any metric that you have in with regards to impact to the business, remember that they’re proxies for the customer experience and impact to the business. If we, for example, halve the number of incidents that we have at a company, does that mean we’ve doubled the positive customer experience? No, of course not, but we know we’ve made it better. That’s the idea, is that if you look for these one-to-one mappings of, well, I reduced the of downtime by X%, therefore made the business X% better. That’s not how it works. Just know that you made it better, and that’s enough.
Matty: [00:20:51] I think that’s really key is, again, it’s moving— first of all, they’re proxies, and that’s hard because we want to see these to be exact, and they’re not, which is sometimes surprising because we work with computers and we work in a world of abstractions, so abstractions shouldn’t be that hard. But they are. And also, it’s not about what the number is necessarily, unless it’s an SLA or SLI, you know, unless it’s something that you’ve contractually agreed to or informally agreed to internally with like an OLA. It’s not a magic number. And it’s— I also, I believe in questioning metrics because a lot of times we First of all, we have the ability, which is great. We’re going through a transformation. If we’re at the beginning, we get to maybe set what some of those metrics are, and that’s awesome. But we don’t always have that luxury. Sometimes we come in, the organization’s already going through a change, or we already have things. And I always recommend figuring out, asking why. Why are we measuring this? Why do we believe this is a proxy for customer value? And I will tell you, you know, 5 nines because it’s more than 4. That doesn’t count, right? Or, you know, our goal is to not have the site be slower than last month. You can’t, you know, you need some type of a line in the sand, even if it’s not a direct number, but just to say like, we know. And I’ve run into that experience before where I’ve said, you know, what are we trying to hit? Within certain things, it’s like, okay, let’s just make the needle move. And sometimes We do need to have a certain number we’re trying to hit, certain thresholds, you know, especially something like page load time. And I’ve worked in several places where when I would question and say, what is our goal? Not to be slower than last month. Okay, well, how as an engineer can I understand what’s the right thing we’re trying to do if I’m trying to make a decision about adding this extra component on the page? Not slower than last month is not something that I can test for and that I can understand. So it’s always— it’s okay to question.
Arup: [00:23:01] No, I totally agree. And like, and I think especially at the beginning of a transformation, you know, you— I think you should set some metrics up and have some goals attached to them. But, you know, probably a month or two in, question the crap out of it. You know, make sure that like those are the appropriate metrics still. And you might discover new ones that would be better proxies for what you’re trying to measure. And, you know, I had this old manager who would always tell me, be careful how you influence people. He was telling me this— I learned this lesson the hard way, and I’ll try to quickly tell the story. Basically, we were measuring availability from a purely server-side standpoint, and it was basically looking at the percentage of 200s. What a very dutiful engineer decided to do was he ended up capturing every 500 at the load balancer level and served it back up as an empty 200. 200 back to the user. So we’re looking at these metrics, we look at the week over week, and all of a sudden we see this amazing improvement and we’re all just patting ourselves on the back, shaking each other’s hands, congratulating each other. And then thankfully, my manager was the one who asked, hey, do we actually know what happened here? And then we, so later that afternoon we start digging in and there’s a couple of facepalms. We realized we had inadvertently set up the wrong incentives. And so we then reverted the change and we went back to our infinite chain and explain to them, hey, remember that great improvement we just saw? You know, about that. And, you know, we, we then had to take— as a management team, we kind of had to take a hard look at ourselves and, you know, basically ask ourselves, like, you know, are we setting the appropriate example ourselves? But also, like, how are we influencing people? And, you know, to the point around what you were saying around, like, an engineer’s day-to-day decision-making, like, just, just be careful, because I find, like, if you solely and overly only focus on the metrics, you can end up driving the exact opposite behavior of what you want.
Matty: [00:25:08] Andrew Clay Shafer calls it the Nash-Pareto equilibrium, which I usually oversimplify to that people will work to the metrics you give them to the detriment of your organization.
Arup: Yep, absolutely.
Matty: I’ve heard Jess Humble tells an anecdotal story about— he was— an organization he was helping out where they had a goal to add one test per sprint. Which meant they had a lot of tests that were assert equals true. They added some tests.
Arup: They met the goal.
Matty: They met the goal, right? And it’s not that these are bad people, right? It’s just sort of like, you’re like, okay, that’s what you’re gonna measure me by, and I think it’s dumb. I’ll do it, right? So it comes back to making sure as well that the folks who want to incent to this stuff understand the purpose of the metric, right? You know, it’s sort of getting the— when you’re trying to bring people along for the ride, There’s a difference between being committed and being compliant, right? And the compliant person is the, okay, Arup, you told me to do this and you’re my boss, so fine, I will do it. And the committed person says, okay, I understand why we have to do this, so I’m along for the ride. And the thing is, it doesn’t usually require a lot of converting people. It’s usually just people tend to fall onto the compliance side rather than the committed side, usually because they’re missing information. Because there’s this sort of belief of, I don’t, you know, that hasn’t been explained. It hasn’t been, as you talked about before, right? They haven’t understood the why behind it. And that makes— and while that seems like extra effort, it makes a huge difference because someone who’s committed versus someone who’s compliant is someone who’s going to be curious about your process. They’re going to question it, which is good. You want that because it’s continuous improvement. We want to be going back and looking at this stuff, like you said, and as we’re reviewing it maybe quarterly or whatever is the appropriate increment and saying like, okay, so we’ve been measuring this and I don’t think that’s actually turning out to be a good proxy. Yeah, that’s actually helpful. And it’s okay to make changes. It’s encouraged. Yeah.
Arup: [00:27:14] Well, one, one, you know, kind of parallel that I think about every now and then is when we think about like the legal system, we think of the intent of a law versus the letter of a law.
Matty: Right.
Arup: And, you know, we see a lot of lawyers get up and they’ll would be arguing ad nauseam about the letter of a law, and then you have an intelligent judge come in and say, look, buddy, I know exactly what you’re trying to do, but that’s not what that law was meant for, so stop trying to twist that, twist it to the letter of the law. Let’s follow the intent of that law. Again, while we might not be great at it, we do change laws. We do have amendments. We do have actual processes in place in the legal system. I think of metrics in a very similar way of like, you have to explain the intent of the metric versus just following the letter of the metric. The second you run into the— if you ask an engineer, hey, why is that metric important? They say, oh, I don’t know. I’m just shipping code to make sure the number goes in the right direction, you’ve lost. You’re so far gone at that point. I totally agree that you have to really, really take a step back and ask yourself whether— probably once, at least every quarter, do these metrics still make sense? I’ll give you one very specific example that I think went pretty well for me, and this is at PagerDuty. I track median pull request duration, and it’s a metric that I measure not because I want engineers to file a pull request and close it immediately, but I use it as a proxy to understand, hey, are our testing environments working appropriately? Are teams actually reviewing code at a reasonable pace? And answering a couple of those kinds of questions. But I specifically never set a goal against it because I actually always feared that it would set up the wrong behavior. And it’s funny because one of our engineers brought this to my attention. He knew that I was looking at it and he basically said, he’s like, you know, I could just open a bunch of pull requests and close them immediately, and that would make that metric go in the right direction, or he could do the opposite. I told him, yeah, that you absolutely could do that, but I’m trusting you not to game the metric. Really, what I was trying to get out of the metric was answering the question, does our toolchain support the behaviors that I want for engineers? If not, then what do we need to go build? What do we have to go automate? It starts prompting some of those bigger discussions. Not once did I ever pin an engineer down and say, why did you take 26 hours versus 18 hours to review someone’s code. That’s not at all the intent behind it, but interestingly enough, even without setting a goal against it, engineers were already figuring out, hey, I think I can game that.
Matty: [00:30:05] I think that’s really key to have measures that you don’t have goals against, that you’re looking at and they’re just kind of temperature. Right? You kind of want to see that because it’s going to influence something, a decision you’re going to make. And it’s tricky because we’re used to measuring things that we’re going to be measured against. And again, this is getting kind of into the ideas around incident response, but one of the things I’ve learned— so we talk a lot at PagerDuty in our response process about don’t litigate severity, right? Which means during a call, don’t argue about whether it’s a SEV 1 or a SEV 2. It’s not the place for it. And so I give a lot of workshops around this process, and that’ll come up. And quite often people in the workshop will say, oh yeah, that happens a lot to us. And almost always the correlation between people who have challenges where severity is being argued about during a call, their team or organization is measured by how many Sev 1s they have. As in, that is how you know you’re doing a good job. Their CIO or CTO or whomever has said, We have this sort of list and, you know, and probably all those managers get pulled into a meeting once a week or once a month and they look at it and they say, Roop, you had 10 Sev 1s last month. You were on the pig list, right? You better get your act together. Well, if that’s happening, all that happens is the metric that we start to work towards is mean time to innocence. We just figure out how to not classify it as a Sev 1, just like, you know, you talked about with kind of redirecting 500s to 200, right? Okay. And because people are used to being measured and evaluated, I should say, according to metrics, it can take— that’s one of the things that is one of the cultural changes. Say, okay, we’re interested in measuring this because we want to understand, but this is not part of your performance plan. Yeah, we aren’t saying you’re doing a good or a bad job because of this. And like many things in a cultural change, people won’t believe it till they see it, right? So you have to prove it. And trust is hard to earn and easy to lose. So every, you know, it’s, you can sit there and say, we’re not going to measure you against this, we’re not going to, you know, evaluate your performance against it. And over time, people will start to believe you. And then you’ll do it. And then you’re all the way, you’re not even back to square one, you’re back in the negatives. In that trust collateral. Even though it’s going to seem easy, or you’re going to want to do that, because it’s not even because you want to punish people, maybe you want to reward, and that’s the other part about that too. If you’re going to say, well, I’m not going to punish you by the number of sub-1s, you almost don’t need to celebrate the lack of them. You kind of want to celebrate the fact that they happen, if anything else. That might make, you know, so it’s— so kind of along those lines, what are some of the anti-patterns that you see? What are some mud pies people can step in?
Arup: [00:33:16] I think, you know, one expectation that a lot of folks have is that, you know, they don’t think along the lines of continuous improvement, so they think these transformations are like one-off things. They think, oh, it’s January 1st. By July 1st, I want the DevOps, and we’ll be done by then. The reality is, like anything else in good software engineering, you’re never done. There’s always this aspect of you’re constantly getting better. While I find, yes, you can actually make some pretty fast changes and rapidly change an organization over the course of months and years, Know that you’re in it for the long run. Know that this is not something that’s going to end after a year. And maybe you’re sitting in a meeting, you question the metrics, and now you have a new set of metrics. And that’s a good thing. That’s an aspect of continuous improvement. So I think, you know, an anti-pattern I see all the time is people thinking it’s going to take a matter of weeks or months. And the reality is, is like, I even look at some of the small organizations I’ve worked with, it can take years. And so, if you’re in a place where you’re year 1 into this kind of a transformation and you feel like you’re not making enough progress, that’s okay. Again, it can be frustrating, but don’t be too harsh on yourself because you’re basically— the way I think about it is if it took you a decade to get into a problem, it’s going to take at least a year to get out of it. And so, if it took you 5 decades, again, this is completely made-up ratios, but it’s probably going to take you some like, you know, 5 years to get out of it with a lot of conviction and a lot of intent. So like, that’s something to think about as well. Like, however long it took you to develop those bad habits, it’s going to take some proportionate amount of time to get out of those bad habits.
Matty: [00:35:02] I think another one that I’ve seen is this desire to plan out the entire transformation before you start.
Arup: Yeah.
Matty: And I see this a lot in It tends to happen more in regulated industries, and I think because the culture has been used to punish mistakes. So I had one customer I worked for— worked with who was a very large insurance company, and we hit a point when we’re going through trying to help them out, and this was 6 months of planning before they implemented the first thing they were doing with Chef, just the first little change. And I said, you know, at a certain point, you’re just stuck with analysis paralysis because here’s the rub, you’re never gonna find everything. Right? You know, as soon as this encounters production, you’re gonna find something you missed no matter how diligent you feel like you’re being. So you might as well control it for a small, small thing and then find out what happens as you go rather than think you’re going to be able to have the foresight to think of every single thing that might possibly go wrong because I don’t care how smart you are, I don’t care how experienced you are, the real world will destroy all your plans.
Arup: [00:36:17] Well, what is it? It’s no plan, like the second you start working on a plan, it’s already useless. Again, I think planning is helpful, but I think a rigidity around sticking to a plan can be very dangerous. And again, going back to that whole idea of questioning things every so often is super important. But like, you know, with that though, again, kind of similarly like with the anti-pattern, is they assume these— you know, folks assume that these transitions carry no risk. And I think that’s a very dangerous assumption because something will go wrong. Something, you know, that you could not have predicted in a million years, or if you had tried to predict it, it would have taken you way too much time and effort to predict it. And so when you say to, you know, let’s say you’re a leader and you’re saying, hey, we’re going to go through this transformation around the way we work, but we’re going to do it in a way where we don’t have any increased risk, you’re being naive. That’s where I go back to the start small, where at least you are accepting that you’re going to have a little bit more risk, but at least in a small— let’s say it’s one team, one project, one codebase, whatever— the increased risk isn’t that much. It’s totally worth that small amount of risk because you’re going to learn so much that then, in turn, actually de-risks the project, funnily enough, going forward.
Matty: [00:37:38] Yeah, like kind of minimizing the blast radius for the change. Yeah, exactly. And there’s— I think there’s a lot of interesting similarities when we think about chaos engineering and the principles around chaos engineering and then going through a change like this, which is what do we do in chaos? We treat it as an experiment. We have a hypothesis that, okay, if we make this particular— and so in thinking of the same thing with a transformation like this, If you think about it as an experiment, which is I have a hypothesis, which is that if we start pushing, you know, start using trunk-based development, we will have better quality. Let’s say that’s my hypothesis. Okay, so I know some measures. I’m going to do the same thing I would do with, with the chaos experiment. I’m going to limit the blast radius so I can understand. I’m going to know my measures so we can see if we need to adjust as we go. And then it’s going to be a learning experience. And treating things as experiments like that, that might not be the right word to use because it might not instill stakeholders with confidence. You might, you know, come up with some other vocabulary around it. But the thing is, if you can articulate those things, it helps build some trust because then it’s not a let’s throw it against the wall and see what sticks. It’s done somewhat methodically. Methodically.
Arup: [00:39:00] Yeah, methodically. Yeah.
Matty: And you also can understand whether or not it was successful or not.
Arup: So look, I think words are incredibly important, and that’s why I do like the word experiment quite a bit, because there’s almost this like humble acknowledgement that like, hey, we’re gonna try this out, it might blow up, you know, might not work, it might fail spectacularly. Again, limit the blast radius so it It doesn’t take out your business or anything like that, of course. But the word experiment, I think, implies that humility, which I think is super important when you’re introducing a new idea to any organization around transformation. And the other thing that, like you said, was around this idea of like, hey, in an experiment, you have hypotheses that you’re testing. And so you actually know ahead of time, like, hey, here’s what we’re looking at to know if it’s gonna work or not. And so I think the word experiment is appropriate. Like you said, maybe it does scare some of your stakeholders off, but what I find there is I always tell the stakeholders that, look, I’ll be the first to acknowledge if I screw up, and I will try to fix it as quickly as humanly possible such that nothing bad really happens and we’ll learn a ton in the meantime.
Matty: [00:40:12] Can you think of any interesting success stories that you’ve been a part of or that you’ve talked to? Because I know you talk to a lot of customers and we may— Some of them you can talk about, I’m sure, and some of them you can’t. But I’d be interested to kind of think about successful transformations that you’re aware of and maybe see if we can think of anything that they have in common.
Arup: Yeah, well, one interesting one that comes to mind just because you were talking about Chef, but ironically enough, we’ve been using Chef at PagerDuty for coming up on 7 years or so. But before that, when we first introduced it, it was an experiment. Chef was a very different product back then, of course. It wasn’t rapidly becoming the industry standard like it is now, but back then, this whole idea of infrastructure automation, all we had really was CFEngine and Puppet, and there were a bunch of different ways to do it, but it wasn’t— again, the whole idea of infrastructure as code and immutable infrastructure, these weren’t best practices like they are today. Rewinding back 7 years, That was an experiment that PagerDuty went through. And one of my predecessors, he told the teams, like, hey, we’re going to try this out. We don’t know what the learning curve is going to be like. Most of us know Ruby, so we think it won’t be too bad. But you know what? We might have to throw this all out and go with something else, or build even an in-house solution if this doesn’t work. And what we found was most of the folks that were initially a little bit hesitant about it decided, to give it a shot, again, because we called it an experiment, and there wasn’t this rigid, you have to do it edict down from above. It was happening at the lower levels of the organization. Over time, it probably took about a year and a half, but it just became the way. It just magically— people got used to it. That was the way that people knew that if they wanted to build out any infrastructure at the company, they had to use Chef. And that worked well for us. Another big one, and this is— I won’t mention their name— it was a customer of ours, and they were getting started with PagerDuty, and their biggest fear was that a bunch of their engineers were going to quit if they did the whole DevOps thing and put everyone on call. The advice that I had given them was, well, don’t put Don’t put everyone on call tomorrow. You can have a target of that, that’s fine, but just don’t do it tomorrow. Again, I told them, use the word experiment and have 1 or 2 metrics that you’re tracking to see if the experiment was successful or not. In a cool way, they were— funnily enough, one of the measurements they were using was the number of engineers who quit as a result of this, and they were correct. This is a very large organization, around 5,000 engineers. And, and so they were actually measuring this over time as they were rolling out Paytree across the organization. If the number of people quitting as a result was getting too high, they actually would slow down the rollout because they knew that they hadn’t done an effective enough job of educating people on the importance of why they want to work in this different way. Um, but that was, you know, another successful experiment. It took them almost, what, I think close to a year, maybe a little bit over that. But that was, that was the way that they were measuring along, like, you know, more engineers quitting. And the answer over time was, as they invested more and more into the context and educating their engineers, that number actually went down.
Matty: [00:43:46] And I think with any kind of a mark like that too, you have to think about, especially over a large sample set and a large amount of time, how you’re making sure that you’re not conflating correlation with causation too, because you know, kind of look, you want to correct for things like, oh, well, we had an acquisition during this time. Coincidentally, at the same time we rolled PagerDuty out to this team, they all got folded into another group. Okay, well, you know what, maybe you have to correct for some of that stuff. But I think it’s— I like the idea of getting creative with those measures, right? And it’s not always going to be the same. Again, there’s not— fortunately, we can’t tell you. We gave you some good suggestions, for some places to start, but you know your business better than we do. Yeah, I hope. If you don’t, go learn it.
Arup: Yeah, go learn it. Go talk to your product managers. Go talk to your finance team because at least, you know, one, you know, maybe this goes back to the tactics a bit, you know, one tactic that I give every single leader if they can’t answer the question of how does your business make money is go talk to your CFO or someone in your in that finance department. And, you know, I found that these are some of the folks that are just so willing to be transparent and they will educate you. That was, you know, I learned this very early on in my career that if you keep your finance team happy, everything just kind of goes a little bit more smoothly. So when the AWS bill comes in, things, you know, conversations go a little bit more smoothly. But that was super important. And, you know, it helped me as, as, as an engineer make better decisions, um, in not just in like, you know, in terms of like organizational transformations, but it made me into a better leader because I knew that if I make this decision, that here’s the impact that it’ll likely have on the rest of our business. And so again, as a tactic, go talk to your finance team if you struggle with articulating how your business makes Gonna have to talk to people, folks.
Matty: [00:45:49] Sorry to report.
Arup: Yeah, sorry. Can’t boil everything down to a single shell script, unfortunately. Not yet.
Matty: Now, what I suggest to people is that if this is not a thing you want to do, if you’re not good at selling change, you’re not good at that, find a buddy who is.
Arup: Yeah.
Matty: Find someone who can help. And because we are greater than the sum of our parts. The last little bit of collaboration to keep in mind is I’m a big believer in DevOps, as is probably no surprise to any listener to this show. It is incredibly unfortunately named. So make your tent big when you’re collaborating and you’re looking for people to come along. You know, it’s not just about software engineers and SREs. It’s your testers, it’s your product folks, it’s your FinOps people, right? It’s like Arun said, your finance folks, get them involved, get your customers, and they can be internal customers. So the more conversations you have, the more that people will understand why you’re wanting to do these things and the more allies that you will build.
Arup: [00:46:56] Yeah, and likewise, I have strong conviction that I think the DevOps way of working, however you want to articulate it, is a better way of working, but to me, I think about DevOps, it is that idea of you, as an engineer or as a leader in your organization, you’re willing to pull in whatever context, whoever stakeholders, whatever you need to solve a problem, and you own it. There’s this idea of you don’t throw things over the fence, and like you said, Maddy, that includes you are not allowed to throw problems over the fence to your product management team. You can’t just say, oh, well, Well, those darn product managers are making me do something, so I have to do it. That’s not leadership, right? To me, that goes against DevOps as well. If you don’t bring them into the conversation and ask for their input on these ideas, then, again, you end up just siloing yourself from those teams, which is very dangerous.
Matty: I think that brings us to a pretty good wrapping-up point. If you are interested in sharing your DevOps transformation story, There’s a bunch of DevOps Days that have open CFPs, so if you go to devopsdays.org/speaking, you can find out all these places that are looking for folks. DevOps Days is an event that I recommend for first-time speakers because we that organize these events, we really like first-time speakers, and we like personal stories, and we like to know what you’ve gone through. And if you’re interested in getting into this, and you got an idea, and you need some help with it, You can always find me on Twitter @MattStratton. I’m happy to help and mentor new speakers. Specifically, the CFD for DevOps Days Chicago is open right now, so I really recommend you submit to that because that’s my event and I’m in charge of picking the talks, so I want to make sure we’ve got some good ones. If you go over to arrestodevops.com/devopsmagic, that’s where this episode’s show notes will exist. Exist. You also can sign up for our newsletter, all the Arrested DevOps stuff you could want. If you go to arresteddevops.com/itunes, leave us a review in the iTunes Store. That actually does help other people find the podcast. We’re not just begging for reviews for validation. That’s just a secondary benefit of that. So please, please validate me in iTunes. So, Arup, thanks for joining today. I’m really glad we got you on the show.
Arup: [00:49:20] Yeah, my pleasure. Thank you for having me. This was a blast.
Matty: So I’m Matty at Matt Stratton. This is Arrested DevOps, and remember, there’s always DevOps in the banana stand.
