Nicole: [00:00:00] Once upon a time, kids, making software was sad.
Matty: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matty Stratton.
Jessica: And I’m Jessica Kerr. Arrested DevOps is brought to you by PagerDuty. In an always-on world, teams trust PagerDuty to help them deliver a perfect digital experience to their customers every time. With PagerDuty, teams spend less time reacting to incidents and more time building for the future. From digital disruptors to Fortune 500 companies, over 12,000 businesses rely on PagerDuty to identify issues and opportunities in real time and bring together the right people to fix problems faster and prevent them from happening again. PagerDuty Solutions before problems. DevOps shows that delivery automation is important. Our work is changing software, and software is useful after it’s delivered. So how do we develop our delivery? Is it scattered across dozens of repos, or could we use code? Is it a loose collection of YAML and Bash, or can we unit test our delivery too? Do we even need all those pipelines? There is a better way. When you’re tired of patching up pipelines, when you’re serious about safe delivery of code, check out Atomist at atomist.com.
[00:01:34] The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener. There is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained, and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app. Today, we celebrate the release of the 2019 Accelerate State of DevOps Report. We have 2 of the 4 authors with us today. Please welcome Dr. Nicole Forsgren and Jez Humble.
Jez: [00:02:43] Hello, thanks for having us on.
Nicole: Yay, I’m so excited.
Jessica: Great, great. To get started, Nicole and Jess, can you give us some historical context of this exciting report?
Nicole: I mean, we’ve been sciencing this for a while, right, Jess? I mean, it’s been 6 years now. I mean, at this point, I know we’re basically like pushing this baby out to school now. I am not crying about it though.
Jez: We get so many hours of the day back now.
Nicole: I’m, I’m so happy. So happy. Well, I mean, I think one of my favorite things is that like every year I get a little nervous that I’m gonna run out of things to ask and research, but every year I swear it’s the best one. And luckily I, I have had a few people like backchannel me or send me notes saying like, oh, you found new stuff to research and you found new things to ask about. I’m like, oh, thank goodness.
Jez: The other thing I think is interesting is every year we revalidate certain things. So for example, looking at speed and stability, every year we get new data, we throw it into a clustering algorithm, or Nicole throws it into a clustering algorithm, I should say. And, you know, that could come out any way. Every year it comes back that high performers do better at speed and stability. There’s some revalidations in this year’s report as well. You know, there was a moment when one of the other researchers working on the report, Dr. Dustin Smith, pinged me and was like, this came through. And I read it wrong. And I’m like, oh shit, that totally contradicts everything we’ve been saying for years. And then I can confirm that something was reverse coded. And it’s like, oh, thank God. But, you know, every year things like that happen where we’re like, oh shit, maybe it’s gonna, you know, come out in a way that— and then sometimes things do come out that are surprising. And we’ll talk about some of those today. Yeah.
Matty: [00:04:36] So, really quick, before we dig into it, because I want to hear a lot about what we learned from this year’s report and what was different, but some people listening, believe it or not, may not be familiar with this report or its history, so I don’t want to spend too much time on that. But first of all, if you’d like to see the report, you can check that out at cloud.google.com/devops. We’ll have a link in the show notes, but also, Nicole and Jess, could you give us maybe a quick Like, what was the impetus for starting this, lo those many 6 years ago? And what is the helpfulness about this? Not the helpfulness. Okay, Jo, take that whole bit about— let’s start that over.
Nicole: No, I like that bit.
Matty: Okay, what is the helpfulness of this? What’s the helpfulness factor?
Nicole: Well, and I, for me at least, that’s why I started doing all of this is like, make back once upon a time, kids, Making software was sad. Making software used to light people on fire, and that was awful. Exactly. Right? You would follow this very strict phase gate approach, and you would speed things. Then we did the agile, right, where you take your meetings standing up instead of sitting down. I love the way you put it, Jez, where it’s like you basically make 2 groups of people unhappy because the people who would rather be on the golf course have to spend more time with developers, and developers don’t want to spend more time talking to business people either. But then all you’re doing is you’re making software faster, but you’ve still completely ignored sysadmins. All you’re doing is throwing software over the wall faster and basically lighting them on fire. Why do we even do this DevOps thing? It’s because we want to make that software process easier and better. The end goal, the endgame, is that we want to deliver value for whatever definition of value that is. For for-profit companies, it’s often profitability, productivity, and market share, but it’s also effectiveness and efficiency and customer satisfaction, right? So, like, we found this to be true in not-for-profits, in government, right? Like, Jez was at 18F for a while, right? Like, these things still worked there, right, Jez?
Jez: [00:07:04] Yeah, absolutely. I mean, these approaches worked even in very highly regulated environments. They’ve worked everywhere, and we’ve seen that. We’ve tested against different domains and verticals over the years. And we found you can do this anywhere, in any size of organization. And I just want to add, you know, the other thing that’s very cool about this for me is, you know, a lot of these things are articles of faith. I think what’s great about this is Nicole comes from a research background, and she had this amazing insight that you could apply behavioral science to studying this. We have a real problem trying to get to do science with software. You can’t do randomized controlled experiments. This approach of taking behavioral science and applying it to test these hypotheses that we have is really powerful and, I think, industry-changing. That is very exciting for me.
Jessica: I love that. The other thing I love about this report is that it shows you, with data, some things that work and some things that get in your way and then gives you things to do about it.
Nicole: [00:08:12] Yeah, so I think that’s a really great point, right? That’s another reason we kind of got into this is instead of asking what tool should I use, because that’s— it’s fine, but also that’s such a dumb question because I hate that question. First of all, that’s not always going to work because feature sets change, right? Every time you come out with a new release of a feature, it’s going to change. And, like, there’s an infinite list of tools. You’re just never going to be able to catch up. So instead, what you can do is come up with and test a set of capabilities and practices and then test those. And so then what we can do also, by the way, then we’re no longer playing favorites. We’ve never played favorites. We’ve always been tool and vendor agnostic. So we can test capabilities and practices, and we can say, okay, this is what is predictive of excellence or high performance or whatever we want to say, like developing and delivering software with speed and stability, then anyone can go and say, okay, this is what we need to do to be doing it right, right? To be predictive of doing better. Then you can go back and you can say, oh, okay, this is what our either toolset needs to do or our practices need to do Then, if you’re a software delivery shop, you can test against that. If you’re a vendor, you can test against that. You can make your own tool better. Anyone can test against that. If you’re doing a transformation, you can say, these are the things we need to do. It comes down to tools and automation, also practices, also culture. Oh, wait, this sounds like DevOps, right? Or digital transformation, technology transformation, to find and replace, whatever it is. Takes to get the stuff done at your company, do that. It’s not just— there’s no such thing as DevOps in a box, right? You can’t just write a check and walk away, which is what we’ve known for, what, a decade now? Curses! You mean I can’t just write a check and get a gold star on my forehead and collect a big, fat bonus check?
Jez: [00:10:29] What are executives even supposed to do?
Nicole: Damn the luck.
Jessica: Yeah, is that behavioral science? It’s not like physics, which is the same everywhere at some level. The behaviors of people are different in every context. So what your report does is show that certain capabilities— are those kind of like behaviors?
Nicole: Well, so behavioral science in part, like what Jess is saying, is behavioral science is that basically we can use survey questions, but they have to be the right kind of survey questions. We go through like several steps to make sure that the survey questions are right and valid and reliable. And some people are like, oh, survey questions are like BS, you can’t use that.
Jessica: Well, they are if I make them.
Nicole: Or like, if you have ever gotten like a political survey, those are nonsense, right? But we go through several steps to make sure that they’re super rigorous. And there are some things that are great for survey questions, there are some things that are not, right? Like I can’t get millisecond precision out of survey questions. I can’t ask you, if your build time was 0.2 seconds or 0.3 seconds. People suck at that. But anyone can tell you if you’re deploying daily or weekly or monthly or quarterly. I can basically tell you that. The nice thing is people can tell you that, and many times, they can tell you that with as good or better precision than your systems. Okay, now people are like, Nicole, you’re full of shit. This is wrong. But here’s the thing. Has anyone seen Office Space? Probably. So the interesting thing with systems is that systems will do exactly what you tell them to do. Systems can tell you millisecond-level precision, but if there is an error in your system, it will collect that error and it will compound that error and it will miss small errors that will keep compounding. And so the nice thing about survey data and system data is you should collect both and triangulate, because if one is wrong, So I’ve talked to many of some of the largest companies in the world that will collect both. And if one is wrong, it’s almost always the system data because people won’t make a mistake between a day and a month. But a system collecting hundreds of thousands of metrics might combine those and one day be way off. People are never going to randomly tell you and make a mistake between a week and a month.
Jez: [00:13:02] And I think the other thing that’s interesting— Nicole wrote a paper with Dr. Mick Kirsten for ACM about the difference between system and survey data. One thing you can’t do with system data is test for the absence of something. So for example, what percentage of your systems have automated deployment? That’s not a question that you can ask of systems, because they don’t know what is not in them. If the deployment automation for a particular system isn’t in version control, version control can’t tell you, hey, that system over there isn’t— I don’t have the deployment for that automated. So there’s some things that you just can’t find out with system data.
Nicole: Yeah, only people can tell you that. So it’s nice to collect both. So the other reason surveys can be good, particularly for something like the State of DevOps report that now DORA has been doing for 6 years, is we can collect data pretty quickly. We can collect data in a month around the world. You can’t do that with systems data. First of all, skip the legal implications. Do you think we’re going to get through legal for hundreds or thousands of companies in a week, 2 weeks, 3 weeks? Not possible. Let’s pretend that that pipe dream happens. Second of all, look inside your own companies. How many people define lead time differently? Or deploy time differently. It just isn’t possible. Correlating these metrics, even within a company, is a huge effort. It can take years, honestly, for a single organization across several teams. It can take 2 years. What you can do with very carefully constructed survey measures is you can just say, this is exactly the question I’m asking. Don’t even give it a name. Continuous integration. We never call it continuous integration. Why? Because so many teams, so many organizations have redefined it to hit a goal, to hit an artificial goal. Instead, we ask specific things like, when I check in code, it automatically triggers a build of software. When I check in code, it automatically triggers automated tests. Things like that. Then people can specifically respond to how often that happens in their teams. Now we happen to know that these are the essential components of what makes up CI. That way we’re not asking about terms of art that someone else has probably redefined. So that’s the other nice thing is that we can, we can collect that really quickly. Anyone can answer that. Also, that’s not subject to legal compliance or worry that we’re collecting someone’s secret sauce, right? Because we didn’t ask you exactly how you created that or crafted it or what code you used. We’re just asking about capabilities.
Jessica: [00:15:45] So it’s kind of a low resolution.
Nicole: It’s relatively low resolution, exactly. But it’s high enough in signal that we can tell you if it’s predictive of success or not.
Jez: And this is, you know, overprecision is one of the things that is a real problem in our industry. People trying to gather very precise data and spending enormous amounts of money gathering very precise data where that level of precision is totally unnecessary to achieving the results. So estimation is the classic example of this. But this is true of the data we gather as well. We only gather at the necessary level of precision to get us what we need.
Nicole: Oh, great.
Jessica: Precision with accuracy and import.
Nicole: Yeah. I love that point. So one of my favorite books on this topic is How to Measure Anything.
Jessica: Oh, I love that book.
Matty: Yes.
Nicole: So good, right? Get the appropriate level of precision to— like, you need signal. You need to know what your problem is. So that you can move forward. Now, later on, for some teams, you do need millisecond-level precision. For some things, you just don’t. Understanding the appropriate level of precision for the appropriate types of things lets us understand what things are important predictors. As Jess said, sometimes we revalidate and we find year over year over year things are important predictors. It’s interesting, though. Sometimes we’ve been surprised. We found that Some things are not predictors. So we’re at that good level of precision where— it’s also funny, some people are like, you’re just confirming biases. Well, no, because we don’t always get confirmation. Second, maybe yes, but also that’s science. You start with a hypothesis, you state your hypothesis, you collect very carefully designed data, you test your hypothesis. That’s not confirming a bias, that’s science. That’s how we do it.
Jessica: [00:17:31] Yeah, and the part where you found some hypotheses that were not confirmed, that’s a good sign.
Nicole: Yes, exactly. So if all you ever do is confirm every single bias, you’re doing it wrong. If you never confirm any bias, you’re doing it wrong. You need some kind of middle ground.
Jessica: Yeah, and that— but then you mix this with explanations of why these things work. And that goes into your research model?
Nicole: So that goes into, I don’t know if I’d say the research model, but some of our findings in our discussion. Yeah, absolutely.
Jez: We definitely have opinions.
Jessica: Yeah.
Nicole: Well, and I think a lot of this benefits from all of the time that we spend in the community, in the background literature, so that we understand why these things are happening. And I love that you bring up this point, because if all you do is take the data that’s already existing, and you just hunt for patterns, that’s also not great because that’s just fishing for data. You’ll have spurious correlations. And the problem is that because so much of our data is interrelated, everything will make sense. One of my favorite websites is Spurious Correlations. It’s from Tyler Vegan, and he will find ridiculous correlations out in the world, like the number of backyard pool deaths in the United States is highly correlated with the year that Nicolas Cage movies come out. We know this is ridiculous. Well, I mean, of course, right?
Matty: [00:19:00] I think it’s legit.
Nicole: Right, I think strangulation by bedsheets is correlated with cheese consumption. We know this is ridiculous. However, if we take the data in our systems and just dig for correlations, it will look good because our systems are highly related. If you don’t start with hypotheses and then collect data, and then test, you will find things that exist. We’re also very careful in our report to say which things are prediction. That’s inferential prediction. As Jess mentioned, this is what’s very common in business because we can’t do randomized controlled experiments. Randomized controlled experiments is we would say, dear companies or teams over here, this is exactly what you do. You cannot change your behavior. Teams over here, you will do this, you cannot change your behavior. Companies are not going to do this because if we assign them a condition and we don’t let them change, and their condition is to do continuous delivery, and another condition is to not do continuous delivery, and let’s say the not-do-continuous-delivery condition starts losing money, they’re not going to keep losing money for the next year just to stick with my experiment.
Matty: [00:20:09] It’s for science.
Nicole: It’s for science.
Matty: Because science.
Nicole: People don’t do that in business. And they’re in such variable— there are so many conditions to control for. We have economic conditions, they have different industries, their teams work in different cultures and contexts and team dynamics. I also can’t say you’re not allowed to do a reorg. They’re going to give me a finger, and you can guess which one it is. There are implicit systems. What we do is it’s called a Sometimes they call it quasi-experiment because I can control for as much as possible. That’s what I do within design, like within org design. This is probably a little bit removed from that because I just control for as much as possible, right? So this year we controlled for— I’m using finger quotes a little bit here— we controlled for, although it’s not full finger quotes, we controlled for company size, enterprise level. We control for industry. In the productivity model, we control for years of experience. And until this year, we have never found an effect. This year we did see an effect in a few things. So I control for as much as possible. So like, if I had set up all of these conditions, we don’t see evidence. This year we do see some evidence. But there are—
Jessica: [00:21:33] Okay, what do we see evidence of?
Matty: Yeah, exactly.
Nicole: What do we see? Da da da. Okay, so for the first time, so in the past, we have never seen an effect, right? So we don’t see statistically significant effects. This year, for the first time, we see 2 effects. We see evidence that industry, one industry is significantly better. Any guesses?
Jessica: I cheated.
Matty: Video streaming.
Nicole: Is it video streaming? Okay, well, then you’re not allowed to guess then.
Jessica: I already guessed.
Matty: Rideshare. Video streaming.
Nicole: Rideshare, no. So it’s funny because most of the time people who are like, oh, but I’m in a highly regulated field, so I’m going to be worse. We do not see that effect, right? And we see significant, a significant difference where one industry is better. And so then people are like, well, it’s going to be technology. Wrong again. The industry that is better is retail. And I— so we do include a sidebar, but like, I don’t want to be the only one talking about all the stats.
Jessica: [00:22:46] But it was such a cool sidebar because it implies that retail is better is because of more competition. It’s not that retailers are moving faster, is it that the low performers are going out of business?
Nicole: It’s because they had— they have to move faster. It’s because they had to.
Jez: Yeah, I mean, government doesn’t necessarily have a lot of competition. There’s nobody else who’s going to be able to process your green card application.
Nicole: Yeah, right? I think that others, particularly, I think a great example of where I see increased competition speeding up are things like finance. If you look at retail, they’ve had the retail apocalypse over the past decade. If you haven’t been competitive, you’re basically gone. We’ve had a series of closures. They have to do things like A/B testing, which is funny because a whole bunch of people are like, A/B testing is cutting edge. Retail has been doing this for a while. Retail has had to embrace automation because if they don’t, they have to capture efficiencies, they have to be able to scale. Then what’s the big question that a bunch of people have had over the last several years? Are we going to go to the cloud? It’s the cloud. Retail is in the cloud. I mean, Black Friday, trust me, they have to be able to burst and do epic, epic bursting. There’s no question if they’re in the cloud. They’re there. Then people are like, oh, but we’re highly regulated. Retail is selling goods. They’re processing credit card transactions. Trust me, they’re regulated.
Jez: [00:24:25] Then the other thing that comes up is that enterprise organizations who have more than 5,000 employees are lower performers compared with those with fewer than 5,000 employees. The one point I think it’s important to make about this is that doesn’t mean you cannot be a high performer and a large company. We do see high performers in large companies who are highly regulated. A lot of people ask us this, like, but what about this, but what about this? The fact that we see this effect doesn’t mean you can’t be a high performer, it’s just that you’re less likely to be.
Matty: One might even say that the fact that companies over 5,000 employees are not high performers is a spurious correlation someone could draw, maybe. See, I’m paying attention.
Nicole: No, so what we’re seeing though is that there is a significant effect. So there’s— it’s statistically significant that they are lower performers. And then we did some additional analysis recently, and by we, I mostly mean Dustin, Dr. Dustin Smith, the second author in this year’s report. I love the things that he found. The additional areas where they are significantly different are— where would we guess that large enterprises are— where do they have particular strengths? Process, right? Exactly.
Jessica: [00:25:58] Committees. Yeah.
Nicole: And I, like, I want to say this with love. It’s like, this is your strength. This should not be your strength. And change approvals was the, was the strongest effect.
Matty: Strongest effect in, in what way? Can you, can you dig into that? Because I’d like to— because I knew I was kind of being a little facetious when I was like, Saying like, oh, because people will sit there and say, oh, well, the report says that if I have over 5,000 employees and I can’t DevOps, so clearly now I have an excuse, and that’s not the case. But so what are the things— so when we saw that effect, can we dig into that a little bit about like maybe what were some of those? Like you said, you saw that change approval process was impactful.
Jessica: How?
Jez: So yeah, I mean, this is something— go ahead.
Nicole: Oh, so, uh, so I’ll like chime in on what strong effect means, and then I totally want like Jez to take this. Um, so when we say like a stronger effect, it’s basically like when you compare large organizations with 5,000 employees or more to everyone else. If you want to look at the difference between them and others, there were, there were a couple things in particular where they were different. Change approvals was one, use of the cloud was another. If we look at how they’re different, in particular— oh, so it was the heavyweight change approval process. It was loosely coupled architecture, right, or tightly coupled architecture. If we think about large organizations, they’re also usually probably older, so they probably have mainframes or what are traditionally more tightly coupled architectures, and then use of the cloud, or they typically don’t use the cloud to its full potential. But then when we compare them to organizations that have less than 5,000 employees, and then we look at how far apart they are, Large organizations are using heavyweight change approval processes to the nth degree, and smaller companies just don’t use them. That’s the biggest difference that we see. Then you look at loosely coupled architectures. There’s a difference, but that difference is not nearly as distinct. Then use of cloud, we see a difference again. But they aren’t as far apart. It’s this change approval process that’s like night and day. That’s what I mean when I say the effect. What’s the degree to the effect? If you only want to talk about numbers. Now, if we want to talk about what it looks like, I’m going to let Jess talk about this, because he’s been in the government.
Jez: [00:28:50] Yeah, so I’ve kind of personally had to deal with these things. And having change control, So having a process for change control is important. There’s a regulate— there’s a control called segregation of duties, which says— I mean, it comes from accounting, this idea that the person who affects the change shouldn’t also approve that change, basically to stop people stealing or doing bad stuff. And we have the same idea in software, that there shouldn’t be one person who can take a change from writing it through to deploying it to production. In one go, because then there’s a possibility that— so, it’s the office space scenario, you know, one person could steal or make a mistake and cause an impact on production. But the question is, how do you implement that? And what we found is, if you use a very heavyweight, formal change management process, where someone external to the team, like a change advisory board, or a senior management manager, is required to approve the change, that has a significant negative impacts on software delivery performance. What you can do instead is use a much more lightweight change approval process, where someone within the team approves the change, either as part of code review, or as pair programming, something like that. And we find that that’s more effective. So what we find is, we actually looked at if a formal approval process was associated with lower change fail rates, And we found no evidence to support that hypothesis. And we also looked at whether introducing more approvals results in a slower process, and that hypothesis was supported. So what this means is, ultimately, implementing change approval process, do it in a lightweight way, don’t do it in a heavyweight way. We also found that when people know about the change approval process, when they’re confident they can get their change changes live and that they understand that process, that drives higher software delivery performance. So just having a good understanding of the process for getting changes live is important and can make a big difference.
Matty: [00:31:06] So this could be a whole other hour of episode, but having spent my fair amount of time in the enterprise and in ITIL myself and everything, I know one of the— so we’re saying we’ve got the science that says if If you have this lightweight change approval, these are the things that will make you better at software delivery. I’m thinking for some actionable things, because our listeners might be like, I totally get that, and now I have to go try to sell this, and I’m going to run into these roadblocks where people say, I don’t really care how good you are at software delivery because risk. Risk, risk, risk, risk, risk. Do you have any— and I don’t want to go too much off on a tangent, because it’s not really in the report. Maybe it is.
Nicole: No, I think this is super relevant.
Matty: Yeah. How can people frame this conversation? Especially in a large organization when people are kind of detached, the folks that you have to convince that you don’t need this heavy process may not really care about making software delivery more effective because that’s not what they’re being held to. What are some things people can do?
Jez: [00:32:14] I mean, this is the classic DevOps or the classic problem DevOps is supposed to solve, which is that 2 different groups— have different rewards. They’re evaluated based on different things. And actually, what we should all be focusing on is how can we get better as a company? So I think also, I want to say, when I’ve gone and spoken to big companies, what I find is when I actually go and talk to the auditors, a lot of them are very up for some of this stuff. I was speaking at a large bank where I was talking to the management team, and there was one of the senior auditors there. And she said, you know, we’re really interested in this continuous delivery stuff. We think it’s the way forward, and we want to work with the development teams to find out how we can achieve these control objectives with different implementations. So that kind of blew my mind a bit, but actually, as I’ve kind of spoken to different people in different organizations, I found actually, you know, a lot of the auditors are fine with this stuff. The trick is, you don’t want to be the first person that tries the new stuff. You want to know that other people have tried it and been successful. And I think there’s enough evidence from enough different domains now that actually this is not crazy. And I think, you know, the other piece of this is, you know, yes, risk. Risk is important, but risk isn’t just about downside risk. Risk is also about upside risk. If you can’t move fast at delivering software, that’s a risk to your business, because then you can’t get better. And even in a government scenario, you want to get better, you want to serve your citizens better. That’s part of your job. And I found people who were just as much, if not more, interested in doing a better job of serving their customers in the government as in other places as well. So I think on that front, it’s not the case that the auditors are just going to shut this down. I think the important point is to go and talk to them. Find those people who are going to try and— who you think are going to try and shut you down, or who the senior management or the middle management thinks is going to try and shut you down. Go and have a conversation with them. And the question to ask is, we know we have to meet these control objectives. How can we do better at meeting them in a way that’s consistent with our other objectives, our business objectives? What can we do here? And then that can be a win-win conversation. There’s lots of stuff that will allow you to implement segregation of duties using toolchains in a way that’s fully auditable, and that meets those control objectives without having to have a change approval process involving a change approval board that meets once a week. The other piece of this is, what happens to the CAB? If you’re in a change approval board, what’s your job now? And I think that is going to change. Hopefully, no one went into technology with the expectation that their job would never change. It’s the case that the CAB’s role is going to change. I think it’s actually a good change, right? You’re moving from inspecting and approving every change to thinking about the governance of the change management process. What are we going to measure to look at the effectiveness of our change management process? How are we going to implement continuous improvement for this? Those are the kind of things that the CAB is going to be moving to thinking about. It’s a more strategic role, if you like, and it kind of mirrors what we’re seeing with the PMO. Traditionally, the PMO has always been about actually how you run projects, and I think that’s also moving to a more governance-based role, a more strategic role of, you know, how do we implement program management in an agile world, in a product-centric world? How do we think about things like cost of delay and measure things like that? So in both cases, you know, the PMO and the CAB, the idea is that you’ll move to a more strategic governance-based role.
Jessica: [00:35:59] So is it kind of like instead of the project management office, we’re going to do project management for you, it’s how are we going to help you, you do project management for yourselves? And the same with the change approvals?
Jez: Yeah, exactly.
Nicole: Yeah. I also really love the point about upside risk, right? By being able to develop and deliver software, faster and more reliably and more safely. Like Jess said, it’s about taking care and delivering value to your constituents. It’s also about being able to keep up with compliance and regulatory changes. It’s about being able to keep up with and respond to security threats. It’s not just like, I’m going to deliver features so I can make money, right? It really is a holistic view. To so many things.
Matty: Is it maybe— I don’t want to say easier, that’s the wrong word— but when we think about it, I think it’s a lot more challenging in a very large organization to have that holistic view because there’s so many moving parts and people tend to be more specialized. I guess that’s been my experience. It’s a lot more about— in a smaller company, it’s like maybe I wear a bunch of hats and I’m aware and I see all the numbers and I think about those things because I’m being asked to think about them, not because I’m any way better than someone who works for a large bank, but in that it’s like we tend to be— and I don’t want to say siloed, but we’re so kind of driven into our little part that we do. I think that can be frustrating for practitioners who want to make this better when they feel like they don’t have that influence. I guess the thing is, to Jez’s point, And Adam Jacobs said one time, and I really like this, he said, nobody wants to do bad work. Like everybody wants to do good work. They’ve just been in a position maybe where they can’t. So, and we tend to make these assumptions about people and these other things where we’re like, oh, you only care about this thing. You don’t actually care about value. And then if you have these conversations, like Jess said, you find they super do care about that. They just don’t know how to frame it. But I would imagine, It’s a lot easier to have our kind of narrowed view of our one particular cog in the machine when I work for a company of 15,000 people than when I work for a company of 15.
Nicole: [00:38:26] Well, and I think like Jez said, if you change the way that you do change approvals, this represents a really important shift for the CAB, right? So, you know, there have been a few times where I’ve talked to companies and talked to organizations and the enterprise where Occasionally, there can be a reflex to say no from the CAB because they feel like you’re taking their job away from them. I understand that reflex, but it’s not that you’re taking a job away from them. It’s that, exactly like Jess said, if this is a shift to something strategic, like, who wants to do code review? First of all, that’s not exciting. That’s not fun. I understand how it can be seen as important. But it introduces delay, it introduces instability. If instead we can shift that work down or shift it left, and instead it can be seen as something strategic where they are helping to architect policies like risk-based approvals so that something that is normal goes through an automated policy, an automated approval that they help decide, they architect, if they also do things like serve as information beacons, right? Because when you’re in a huge organization, you almost necessarily have to be a little bit siloed for expertise, but you can then communicate up to someone who has that holistic view of the organization about what’s happening. They can help architect and understand policies and understand how the pieces fit instead of just reviewing 10,000 lines of code, which is not possible. I saw this amazing tweet where it’s like, 10 lines of code equals 10 comments, 10,000 lines of code equals LGTM, right? Looks good to me. There’s no way they can review 10,000 lines of code, but if instead they are deciding on what types of things get automated low-risk approvals. What types of things that are communicated up to them get what types of communication and notification across the organization and how these things stitch together. It might require some— it probably requires some type of education because now they’re strategic. If anything, it should be exciting. Like Jess said, we don’t go into technology to never learn new things and to never change. It may be scary. When we get a huge promotion, which this is, this is a huge promotion. You go from, very politely, I say this with respect, you go from a rubber stamp to someone who is now guiding and directing and strategizing for an entire organization. That’s slightly terrifying. You go from change management theater to legit strategic change management for a massive organization. It’s scary, but it is also dope. It’s cool. And I think if you can communicate to an organization that I’m not stealing your job, I’m transitioning it into something that is hugely impactful.
Jessica: [00:41:39] Yeah, to go from working within the system to actively participating changing the system.
Nicole: Yep, the impact is so much bigger. Yeah, from a change approval board to a change approval, like a scalable, repeatable, auditable mechanism, right? Once you decide a policy, instead of rubber stamping and looking over pieces of code every single week, once you set a policy so that every single line of code automatically gets checked in and done every single day, that’s huge.
Matty: I think this goes back to why this report is so important. Right? I’ve made the joke before that when I talk to my customers and talk to folks out in the community, all I’m generally doing is saying, like, go read what DORA and what Accelerate has done, because here’s your numbers, here’s your science, right? Because for so long, this has been like, we all feel like we know what’s right, and it feels right, you know, and that’ll get you somewhere, but to Jez’s point, like, you don’t want to be the first one. It’s good to have something to back it up. I think it’s really important to be able to sit there and say, it’s not just because I believe that doing these things will make us more effective. Here’s data. People will argue with math. We know that. But fewer people will argue with math than will argue with what I’m telling you I believe in because it’s in my soul and I had a DevOps license plate.
Jessica: [00:43:09] It’s on your t-shirt.
Matty: Yeah.
Nicole: Yeah, and, and like, to your point about data, really quickly, because I would like to talk about productivity stuff, um, like, for years we thought that these change approval boards— the intention was good. We wanted to introduce, like, additional eyes to make things more stable. We’ve revalidated that this, that this only makes things slower, and it introduces instability. We first looked at it in 2014.
Jessica: We’ve got—
Nicole: we revalidated this year with more rigorous research methods and better pieces, it doesn’t work. The intention was good. I understand why, like, people tried it, but it doesn’t work, right? We’ve seen it for years now.
Jessica: So as soon as you make change harder, then you get fewer changes, which means you get bigger changes.
Nicole: Bigger changes, higher likelihood of failure, like larger blast radius, higher likelihood of failure. Longer downtime, harder to debug, harder to restore.
Jessica: [00:44:11] To scale up development to involve more people and more systems, you need to scale down the changes.
Nicole: Smaller pieces.
Jez: Small is beautiful.
Matty: What about productivity? Nicole, you just teased that a minute ago. You said, I wanted to talk about productivity.
Jessica: The report has a really interesting definition.
Nicole: Yes. We decided to do extra this year. We’ve always studied performance, speed, and stability. This year, we decided to also look at productivity. But as we teased in the book Accelerate, you shouldn’t be measuring things like lines of code or story points because that leads to bad behavior. Lines of code only bloats the system. Minimal lines of code means no one can read it. Story points can lead to teams working against each other. I’m not going to help you because I don’t get my story point, even if it’s good for the organization. There’s been some work in recent years with a few research teams across the industry. There was a group out of Google Research, there was a group out of MSR, Microsoft Research, looking at productivity as defined by the ability to get complex, time-consuming tasks completed with minimal distractions and interruptions. So said another way, did I hit a good flow? Did I get in the zone? Did I feel like I had a great day? Because like, you know, when we have those days where we’re like, I got a ton of stuff done, or in contrast, do you ever have a day when it’s the end of the day and you’re like, I was busy all day and I got nothing done? That is not productivity.
Jessica: [00:45:47] So you mean maybe productivity is not about how many Tickets you close, but whether you actually got to get work done?
Nicole: Yes, and, and that may be correlated with closing tickets, but it’s not when, like, you do this weird gamification. You just close a bunch of tickets because you could be closing the tickets that are meaningless but easy to close.
Matty: I see you’ve, you’ve seen my entire career.
Nicole: I mean, I, I am sitting here with my Post-it notes, and there are a couple that I wrote on there just to cross them off for myself. Because it makes me feel good, it also doesn’t count. So the reason we like this is because productivity is important. It helps us get things done. The really interesting thing that we see in the research is that it helps reduce burnout, and it doesn’t just help reduce burnout, it helps with work-life balance. The research calls it work recovery because it’s the ability to detach. If we hit good workflow, we can go home, we can leave work at work, we can shut our brains off because we have that sense of accomplishment, right? We don’t just keep spinning on things that we could never fix. We may still have more things on our to-do list, but we felt like we got something done so we can kind of shut it off. So then we wanted to say, okay, what leads— what helps with productivity? Good culture, good tools that we aren’t fighting with all the time, Good search so that we don’t have to keep hunting for things. And what hurts is technical debt. We also looked at things that help reduce technical debt. Obviously, good practices like refactoring, but if an organization could make investments in tools, what would those investments look like? Then how does that— if there’s overlap, if we have a good Venn diagram with our performance model, what does that look like? We had some good stuff this year. I mean, of course, I’m a nerd. I get research excited all the time, but we had some good stuff this year.
Matty: [00:47:48] So, Jez, what were kind of some of the— maybe either their surprises or key findings you really want to make sure people take away as they read the report? Because you’re all going to read the report, right, people? cloud.google.com/devops.
Jez: Yeah. One of the interesting things we looked at this year is how you scale up your work. We looked at a whole bunch of different things that people have historically tried in the past, things like centres of excellence, proof of concept, communities of practice, bottoms-up, grassroots, or big bang, this kind of thing. What we look at is how commonly they’re used in low versus medium versus high versus elite performers. One thing that’s particularly interesting is communities of practice, very frequently used in elite performers, much less commonly used in low performers, versus, for example, centers of excellence, which are frequently used in low performers but less frequently used in elite performers. So that, for me, is really interesting. Centers of excellence was kind of the old-school way of doing things. Community of practice has really— we’ve seen a lot more talk about communities of practice in recent years, so it’s nice to see that that approach is actually providing dividends in high-performing companies. So there’s a whole bunch of stuff to dig into there, so again, look at the report. There’s more detail there. The other thing that I think is interesting, people always ask us, you know, what should I do first? You know, should I do the technical practices? Should I do the lean practices? Should I, you know, and the answer is, it depends. You know, I was a consultant for many years, but we use that phrase a lot, but it really does. It depends where you are, where you’re trying to go to, and what your current context is. What we— so we’ve struggled with what we can say about this that’s more helpful than that, and what we kind of came up with in the end was this idea that, well, there’s stuff you need to do at the team level, and there’s stuff you need to do at the organizational level, and there’s stuff that you need to do at both levels, if you like. So we talk about that in the report, how at the team level you can do stuff, you know, like test automation and deployment automation, for example. Those are things you can do at the team level. What you can’t do at the team level is change the way that change approval is done. That’s something that has to happen at the organizational level. But you should be doing both of those things simultaneously. You should be attacking the team-level stuff while people in different parts of the organization are attacking the organization-level stuff. So just thinking about how you can actually do some of this stuff. A lot of the time, we hear, well, you know, I can’t do anything about this. Yeah, that’s true, you have to get help to work on that stuff, but that doesn’t mean you can’t do this stuff over here. Or leaders saying, well, we can’t do anything, it’s up to the teams to get the work done. That’s not really true. There’s stuff that leaders can do at the organisational level that’s going to help those teams do better. So that’s something else that I think is interesting to point out in the report. Nicole, do you have additions to that?
Nicole: [00:51:02] I mean, I like the whole thing! But yeah, I do like the fact that we gave people a lot of places to start on their transformation. The scaling piece is definitely there in terms of what we’re seeing in terms of scaling strategies from the high and the highest performers, so you have a place to start, types of capabilities that should be addressed at what level. So, for sure, check out the report.
Matty: Amazing! So, head on over to arresteddevops.com/devopsreport2019. For this episode’s show notes, we’re going to have links to the report, to a lot of the things that were mentioned on the show, to do more reading, because you should do all the reading. You can also, if you want to do the writing, go to arrestodevops.com/itunes and write us a review on the iTunes Store. This helps people find the podcast, so look, you’re helping. We’re also— now I discovered we’re on Spotify and iHeartRadio, so if you like those things, go find us there. Listen to us everywhere.
Jessica: [00:52:03] Thank you, Nicole and Jez. Thank you for joining us today. This was super fun.
Jez: Thanks very much for having us.
Nicole: Thanks so much.
Jessica: I’m Jessica Kerr, @jessitron.
Matty: I’m Matty Stratton, @mattstratton.
Jez: This has been Arrested DevOps.
Matty: Remember, there’s always DevOps in the banana stand.
