CI, CD, Oh My! with Jez Humble

Jez: [00:00:00] So, you know, sometimes you’ll come to the temple and you’ll be feeling really down, and meditation will really help you. And then sometimes you’ll come to the temple and you’ll be feeling really great. But don’t worry, because that feeling will pass too.

Bridget: It’s time for Rested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout, and today we’ll be talking about continuous everything. The show notes for this episode can be found at arresteddevops.com/ci-cd. But first, a word from our sponsors.

Jez: Arrested DevOps is brought to you by TenthMagnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. TenthMagnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/10thmagnitude.

Bridget: [00:01:09] This episode is sponsored by VictorOps. Built for modern incident management, VictorOps provides a unified platform for real-time alerting, collaboration, and documentation driven by your IT and DevOps system data. VictorOps helps you to respond to incidents more effectively so you can minimize downtime and make being on call suck less. Visit arresteddevops.com/victorops to schedule a demo or start your trial. Mention you heard about VictorOps on Arrested DevOps and you’ll be eligible for some sweet discounts too. GoCD is the on-premise open-source continuous delivery server created by ThoughtWorks. With GoCD’s comprehensive pipeline modeling, you can model complex workflows for multiple teams with ease. And GoCD’s value stream map lets you track a change from commit to deploy at a glance. GoCD’s real power is in the visibility it provides over your end-to-end workflow, so you get complete control of and visibility into your deployments across multiple teams. Say goodbye to deployment panic and hello to consistent, predictable deliveries. To learn more about GoCD, visit gocd.org/arrested to download. It’s completely free to use. Commercial support and enterprise add-ons, including disaster recovery, are available. I’m super excited to finally sit down with the one and only Jez Humble. So, you know, Jez, you were on ADO forever ago, like before I was on the show, episode 15, about 3 years ago, back in 2014, talking about continuous delivery.

Jez: [00:02:42] That feels like a long time ago.

Bridget: Well, and we probably have some new listeners since then, and exciting things have happened for you. So, Jez, tell our listeners listeners a little about yourself.

Jez: Well, thanks very much. A pleasure to be here. Thanks for having me on. It’s always a pleasure to chat with you, Bridget. I have— well, what have I been doing since then? Well, I spent last year working at 18F, which is a team within the US federal government. We bring modern, lean, agile approaches to government agencies. I was working in infrastructure. I was on the team that managed our AWS stuff, and I helped out with cloud.gov, which is a platform as a service we built using Amazon and Pivotal Cloud Foundry open source to allow agencies to do cool modern cloud stuff. It’s all open source, you can go download it and see it for yourself, basically to prove that you could use continuous practices, DevOps, modern cloud stuff, even in the federal government. So that was great. Right now, I am working with Dr. Nicole Forsgren, and Sue Choi, and Gene Kim, at a venture called DORA, DevOps Research and Assessment. This is the team that, with Puppet Labs, brought you the State of DevOps report for the last few years. We have a ton of data from that, so we thought, well, we can use that data to help organizations get better at DevOps. We have an assessment, we can come into your organization, we can survey all your people, and we can tell you exactly where you should be investing in terms of capability development in order to get the biggest bang for your buck in terms of getting better at the DevOps.

Bridget: [00:04:24] Wait, so you can actually show people how to Dev some Ops?

Jez: Absolutely. We have like a ton of— in fact, we have 20 capabilities that we’ve demonstrated from our research actually impact IT performance, from cultural things to process things to practices, you know, like automated deployment, managing work in process, things like that. And we can tell you how you’re doing on all of those things compared to the industry. And if we have a big enough group, 50 or more people, we can actually tell you which of those things for you, if you invest in those things, is gonna give you the biggest bang for your buck.

Bridget: See, that sounds amazing, but I gotta say, having, you know, spent my last couple of years at a vendor, going out, talking to Pivotal’s customers, I’ve seen a lot of people who say, okay, these are all the things we should do, but who should do them, and they want a RACI chart. Like, how do you deal with those questions?

Jez: That’s an excellent question. And I think this comes back to actually something that is just important for any organization. We should have high-level objectives, and then it’s up to the teams to work out how to collaborate in order to achieve those. I mean, there are strategic priorities for an organization, And there should be a few of those, not very many, and then the team should work out how to achieve that. So my answer is, all of you have to do it. You all have to work out what your part is in that, and the way you work that out is by talking to the other teams and discussing, well, how, you know, what’s the next measurable step that we’re trying to achieve, say, in the next month? You agree on a measurable goal, and then You try some stuff out. I mean, there’s no way of knowing in advance who’s going to have to change, but, you know, my money is on everyone.

Bridget: [00:06:17] Wait, you can’t just have some DevOps experts do your DevOpsing?

Jez: Yeah, I mean, you know, obviously, I have the starter pack with, you know, the open-source tools and the DevOps cards, but, you know, that only gets you a certain way. So, yes, you know, I think we both share a certain skepticism of the idea that you can just follow a recipe and achieve weight loss with a simple pill.

Bridget: Right. But at the same time, though, what I think is really, really cool about DORA and the stuff that you folks have been working on and doing and publishing, and we need to talk more about that, but the stuff that you’ve been putting out there, it basically lets people put some hard data around the, well, you should change. And people, you come back from a DevOpsDays and you’re all kumbaya and, like, let’s everybody get together and try to love one another right now. And someone says, well, that’s great, but how? So I like that you can give people something that’s a little bit more tangible.

Jez: [00:07:19] Yes. I mean, we’re all about data. My boss, Dr. Nicole Forsgren, I mean, was a professor and she did accounting and stats. So you will lose any fight you pick with her about those things. I know I’ve tried. So we’re all about data. Data and measurable stuff and being very tangible. The great news is, you can be very tangible about things like culture, and you can make that measurable and tangible and talk about it. That’s really what we try and do.

Bridget: Yeah, I know, I love that. Since you’re known for continuous everything, let’s do a quick level set for those of our listeners who have not read the Continuous Delivery book. First of all, what is the difference between continuous Let’s see, we have delivery and deployment and probably other things, and there’s continuous integration somewhere at the bottom of this stack of turtles. What are all these continuous things? Can you give us the elevator pitch? Sure.

Jez: [00:08:23] So continuous integration means building and testing every change you make to your system. And that requires, for example, working off a shared master or trunk rather than on long-lived feature branches. It requires having an automated test framework, so that you can actually run tests against every check-in, a quick and comprehensive test framework. But the idea is, we keep the system working, and we find out the moment we’ve broken something, and we fix it as soon as possible, so that the default state of the system is a working one. So, that’s continuous integration, something we’ve known about for years, something that people still often think they’re doing when they actually aren’t. So, I have a very simple test I administer to determine if people are really doing it, which is, number one, Is everyone checking into a shared trunk at least once a day? Most people can’t answer yes to that question. Number 2, when the tests— when you check in, do tests run on every check-in? Again, a bunch of people can’t answer yes to that question. Then finally, the one that gets people, gets most of the rest of the people is when the build breaks, do you typically have it fixed within 10 minutes? A lot of people aren’t very disciplined about that. It turns out, Running Jenkins against your feature branches and then ignoring the build when it turns red is not continuous integration. It’s a practice and mindset, not a tool.

Bridget: [00:09:48] What? That makes so much sense, though, because it’s, I think, and you’re right, like, there is a certain amount of CI/CD theater. People want to say, we are doing the thing. And that’s probably true with a lot of the metrics that you test with DORA’s assessments, too. Like, how do you distinguish between, we would like to be doing the thing, and so we’re gonna say we’re doing the thing? How do you distinguish between that and you’re actually doing the thing?

Jez: Well, I mean, this is why we have questions with either yes/no answers or strongly agree, strongly disagree answers, so that we can actually tell. And, you know, finding the right questions to ask is the hard bit. I mean, this is what we do with the State of DevOps Report. Me and Gene and Nicole and the Puppet Labs team spent ages coming up with questions, trying get to the heart of this, and then we actually get people to take the survey and find out if these things, A, are actually valid measures, and B, if they actually predict something useful. Yeah, that’s really hard. We’ve come up with things that aren’t even valid on many, many occasions. Yeah, asking the questions is, in many ways, the hard part. I absolutely agree with what you said. People love redefining terms so they can say they’re already doing it. How many times have you heard, oh, yeah, I’m doing DevOps, or, I’m doing Agile, and you go in there, and it’s a shit show?

Bridget: [00:11:12] You’re like— You’re like, you’re devving something, but it might not be some ops. You might not be devving some ops right there.

Jez: Right, exactly. So, that’s continuous integration. Continuous delivery— well, let’s do continuous deployment first. Continuous deployment is when the CI build passes, it just deploys automatically to production. So you’re like, okay, every time we have a change and it passes CI, or maybe you add some more validations in there, and you have performance tests and other kinds of tests, but ultimately, you’ve got a pipeline which basically takes a change, runs a whole bunch of validations against it, and if it looks good, you’re going to deploy that to production. So unless there’s a good reason—

Bridget: Completely automagically?

Jez: Yeah, absolutely. Unless there’s a good reason not to, you’re going to deploy every build, is basically the thought. So, that means deploying pretty, you know, several times a day at least, and really high performers are deploying, you know, hundreds or thousands of times a day. If you look at companies like Amazon, for example, continuous delivery is basically behaving as if you were going to do continuous deployment, but not actually deploying all those builds. You might think that’s a bit weird. The reason that we have this term is because it doesn’t always make sense to do that. If you’re building, say, firmware or mobile apps, it doesn’t make sense to deploy every build, but the same practices that you would use to do continuous deployment are still very valuable. They still improve quality, They reduce costs, they reduce time to market, they increase feedback loops, they require you to do a whole bunch of stuff that increases your discipline and your hygiene. So, it turns out that even if you’re not releasing every build, you should still be doing all those practices.

Bridget: [00:13:10] I’m going to play devil’s advocate and say, well, that sounds great, but don’t you kind of have Schrödinger’s deployment? If you don’t know, like, if you say, oh, this build totally could go out, but really, there’s some sort of, you know, I don’t know, schema migration that you didn’t account for. So, if and when it does go out, it’s gonna be a shit show. Like, how do you reconcile the, we think this probably could go out, with the, this really could go out?

Jez: I think my answer to that is, If you can do continuous deployment, you should do continuous deployment. If there’s a way to change the way that you’re architected so that you can do continuous deployment, you should do continuous deployment. I think my thinking has really evolved on this, and I think I’ve seen my feelings about these things converge. I would not build any software that wasn’t web-based unless I absolutely had to. I’ve spoken to people, even people at companies like Facebook who are deploying mobile apps, and they are not happy about the fact that they can only deploy once every couple of weeks. That causes them extreme pain. I think this is something where unfortunately, the web has got to the stage now where we’re really good at it, actually. Unfortunately, now what we’ve found is new technologies emerge like mobile where we’ve basically had to reinvent the wheel, which is a really terrible problem. So, yeah, I think the answer to your question is, do continuous deployment wherever you can. Interestingly, Charity Majors just had a blog post a few days ago, which she put up, about testing in production and how no matter how good your automated tests are, you’re never going to catch everything. I think that’s true. I probably lean more on the side of doing upfront work than maybe Charity does in her blog post. But I do agree with her basic thesis, which is, you know, you’ve got instrument production, you’ve got to have good monitoring and logging and tracing in place, and you’ve also got to make it low risk to deploy things to production. A lot of what we talk about in the Continuous Delivery book is practices like blue-green deployments, and canaries, and feature flags, and techniques like this, to be able to push deployments in a way that if something goes wrong, it doesn’t cause catastrophic or cascading failures. Architectural things like, what’s that pattern that Mike mentioned? Circuit breaker, and a bunch of architectural patterns to prevent cascading failures. A lot of this is about architecting for resilience, both in terms of your application architecture, your infrastructure architecture, and the patterns and practices that you use to do deployments and make changes.

Bridget: [00:16:03] Yeah, I mean, absolutely, especially because realistically, uh, even if you take a, you know, scrub-sanitized mirror of your production traffic, you’re still not going to put that into some sort of test environment at the exact same volume. So not to mention that there could just be things about the way your instances are scattered across availability zones, and some of the availability zones do something that you don’t entirely expect, like There is no production except production.

Jez: Right. And you’re never going to mitigate all the risks. All you can do is do your best, and it’s always going to be a trade-off. There’s a certain amount of money, and time, and effort that you’re going to invest in trying to make sure that nothing bad happens. But we have to accept that failure is inevitable. That’s a lot of what DevOps is about, is about saying, failure is inevitable. How do we make it so that we can detect and respond rapidly to failures? This is why, I mean, one of the things that I got very early on from John Allspaw that made me have an aha moment was the focus on mean time to restore service over mean time between failures. So, yeah, I think a big piece of DevOps is, you know, part of it is about preventing things, part of it is architecting resilience, and a part of it is, you know, accepting that failure is inevitable. What are we going to do? And in fact, you know, as we’ve seen, actually creating stress and failures if there’s not enough so that we can actually practice that situation and get really good at it.

Bridget: [00:17:32] Yeah, like Andrew Clay Shafer likes to talk about, you know, the state of continuous partial failure that we’re all in, all of our complex distributed systems are in at all times.

Jez: Right, absolutely.

Bridget: Okay, so you’ve covered the, you know, the whole spectrum of all of the continuous things. Can you give us a little bit of insight, just a peek behind the, you know, the closed doors of all these organizations Dora goes into, give us the anonymized version, but you go into all these organizations, they all wanna improve. When people are looking to improve and they think they wanna move into some sort of CI/CD, DevOps, whatever, they would like some MacGuffin to help them improve. Like, what does that journey of improvement look like? Are there commonalities that you’re seeing?

Jez: I’m gonna give you the main one, and it probably sounds like a cop-out, but I really believe this. And the data shows it as well. The best organizations are always trying to get better. People who are really good are never satisfied. They’re always trying to get better. They’re never like, oh, this is good enough. And they always have that attitude in their daily work that, you know, we’re going to find new ways to experiment with ways to improve. So that really, I think, is the secret sauce. There’s no organization in which this stuff is easy or straightforward. Even companies like Amazon and Facebook, who are high performers in this area, they invested huge amounts of time and effort into achieving this. People forget that Amazon was a monolith, and they spent 4 years re-architecting in order to achieve that incredibly high performance that they achieved. The way they did it is Bezos sent a memo, which I’m sure you’ve read Steve Yeager’s platform rant and seen all this stuff. They made it a priority. This goes back to what we were saying earlier, about how do you achieve this? Well, you’ve got to make it an organizational priority, and then actually focus on it.

Bridget: [00:19:33] But, yeah, no, you were just actually, I watched the entire video of your Agile 2017 keynote, and you were talking about exactly this, that Amazon didn’t magically spring full-formed as, like, done, complete, perfect, you know, fait accompli. Like, this was at least a 4-year process in the making.

Jez: Yeah, and they were really single-minded about it. They focused on it. They can stay at all levels of the organization. They were consistent about it. They were clear about their objectives, and they invested a bunch of money in it. And frankly, Amazon’s not perfect. There are many pockets of Amazon that are miserable and dysfunctional, as with all companies. And so—

Bridget: Wait, wait, are you saying software is made of humans?

Jez: You know, this is a recent discovery for me. It’s very, very disappointing.

Bridget: Software like Soylent Green, made of humans. You talked about this in your Agile 2017 keynote too. I love the rant that you went off on. Do you want to give us the Cliff Notes of that?

Jez: [00:20:41] Oh, sure. My bonus material at the end. Yeah, I mean, so, uh, no doubt anyone who’s, uh, been paying attention and not on a kind of drug-induced, um, break from, from reality, which, you know, it’s quite attractive at the moment, I gotta say.

Bridget: Sounds delightful, really, waking up in 2020 when Elizabeth Warren’s president.

Jez: Oh, oh, you’re kidding me. Um, so if that hasn’t happened to you, you probably will have noticed, uh, this ex-Google employee James Damore wrote a 10-page discussion of his views on Google and what he called its ideological echo chamber and his opposition to what he perceived as positive discrimination and his discourse on biology and the biological causes of the very low levels of representation of women and people of color in the industry. So, I mean, I was, as many of us were, very angry about this, and also angry that we’re still having this discussion in 2017. And this is something that comes, you know, I’m angry that I’m talking about continuous integration in 2017, because we’ve been talking about this for 15 years. But equally, I’m, you know, just, just give me a second here, I’ll be right back.

Bridget: [00:22:08] Narrating for those who are watching without video or listening without video, for those who are listening without video, Jez has grabbed a book.

Jez: It’s a book. This is a book, the 25th anniversary printing of, uh, The Human Side of Enterprise, uh, by Douglas McGregor, where he talks about Theory X and Theory Y, uh, and, you know, management ideas. So this book came out in 1960. And here we are in 2017, still having these discussions about how to effectively motivate teams and create good working environments, which she covered very comprehensively in 1960. But even worse, even worse is that we’re still having these conversations about innate ability and its variance across gender and race in 2017. It’s nonsense. And a bunch of people have said, well, it’s not about ability, it’s about preference. But he’s actually very clear. He says the word ability, he’s talking about ability. And frankly, it’s not much better with regards to preference. And I want to be very clear about this. There are differences in biology between men and women, and there are differences in brain structure and other things. What we see in real life is a complex interaction of genetics, epigenetics, and environmental characteristics that produce these outcomes. And, you know, yes, it’s worthwhile having a discussion to try and unpick these things and discover how we make things better. But it’s absolutely certainly the case that you can’t go from those biological differences straight to differences in ability. There’s no world in which that’s the case.

Bridget: [00:23:45] Wait, so you’re saying complex systems are complex?

Jez: What?

Bridget: Why would we want software engineers to understand that?

Jez: Yeah, I spent a bunch of time debunking that and his arguments. Even the people whose work he relies on will tell you that those differences in personality only account— you can’t get from differences in personality to a less than 20% representation of non-white men in tech. The math just doesn’t work. Science doesn’t work. Even for preferences, you look at things like women coming into the tech industry, it’s very clear from early on that women, girls before they hit puberty are really into math and science, and then suddenly representation goes down basically because of gender stereotypes being reinforced at all levels of our lives. I’m preaching to the choir here. I, I know that.

Bridget: I mean, and not every single person who’s listening to this podcast may be me. It’s possible. We have 6 viewers on YouTube right now.

Jez: [00:24:53] I just feel weird saying this because I’m sure that you could express this much better than I could.

Bridget: Yeah, I mean, I’m, I’m totally with you. What I really appreciated about your keynote is that you gave a really great keynote about important topics in our industry And then you added some more important topics in our industry. And I watched the tweets go out of Agile, and there were a few people who were like, well, I didn’t come to hear this. And I’m thinking, huh, so you do care a lot about building systems that work, except you’re probably not gonna succeed because you’re not paying attention to this really important factor that affects whether or not your systems work.

Jez: Right, absolutely. And, you know, people got up and left. I mean, I had one particularly, good line that I mean, and let me be clear, I wrote this presentation on the plane the day before on the way up, and you very kindly listened to me give it and gave me some really valuable feedback. So I just want to thank you very much for that, for taking some time out to review my presentation there.

Bridget: [00:25:55] It was great.

Jez: But, you know, I said, you know, the math and science can’t explain the lack of diversity, but there is something that can, And that is having to share a work environment with people like James Damore. And a bunch of dudes got up and left at that point because, you know, people don’t want to hear that. But guess what? You know, this is a real problem. This is a real problem, A, because tech jobs are highly paid and high status, and it’s inequitable that women and people of color don’t have equal access to them. It causes us to produce crappier products. It impacts our economy. It’s terrible on all levels. We should care about this. I think you’re absolutely right. If you can care about the minutiae of tech, but you think that society and the way our teams are organized and composed is not important, sorry, you’re going to fail, or at least you’re not going to do nearly as well as you possibly could do. You just can’t treat— these are systemic problems, and they impact the system, and the system is what’s dominant.

Bridget: [00:27:01] And I think that’s a really good way to put it. Like, for people who are engaged in systems thinking, they do need to pay attention to the whole system. And that brings us, like, ratcheting right back to the, if we’re going to make continuous improvements, and we’re going to make continuous improvements in our teams, in our organizations, we are going to pay attention to all those human factors. We’re also going to pay attention to, like, the roles that people play on those teams. And this kind of, it touches a little bit on, you know, the Manifest Bro, but I think it also touches somewhat on the work that you’re doing with DORA. When you’re out there talking to people about how their organization can improve, do you get a lot of, like, people are, you know, still very highly siloed or segmented or however you want to put it? Or do you see an awful lot of, like, we have 2 pizza teams, they just don’t talk to the other 2 pizza teams? Like, what do you see? What do you see in terms of organizational dynamics?

Jez: It’s very highly variable. I mean, every organization has pockets where there’s teams who are doing really great and pockets where teams are doing really badly. Often, it’s very highly dependent on who’s in charge of that particular team. Managers and managerial approaches and philosophies and leadership approaches and dynamics have a really dominant role on what happens here, and it really depends on that. But yeah, I mean, I see a lot of dysfunction. Also, I see a lot of good stuff. One of the high points in my career was working for the US federal government last year, and people think that the US federal government is uniquely dysfunctional and terrible, but it’s not. I worked with some unbelievably smart and brilliant and crucially mission-driven people who did whatever it took to achieve the results. We work with other people in the federal government. I don’t want to pretend that 18F is this amazing thing that just the rest of the federal government is not. I work with people in all different kinds of agencies, all different positions, including InfoSec. Talking to InfoSec people was hugely important because guess what? You’ve got to develop relationships in order to achieve the outcomes you want. I found mission-driven, passionate, incredibly hardworking people everywhere in all these agencies. Mark Schwartz has a great talk that he gave at DevOps Enterprise a couple of years ago, where he talks about how actually bureaucracy is a good thing because it encodes a lot of what people think is good and the right way to do things. By understanding that, you understand a lot about the values of the organization. Bureaucracy can actually be tremendously helpful. That was another aha moment I had listening to Mark Schwartz talk about that. A big part of this is finding ways to work with the bureaucracy, finding ways to achieve the goals of the bureaucracy, but at the same time, we’ve got to accept that we’re going to have to change things as well and finding ways to do that to achieve the outcomes, but maybe different ways to do that. Again, that’s something that I see in organizations which work really well. It’s saying, listen, I’m a dev, here’s an ops person, here’s an InfoSec person. We all want the same things. We all want to produce great outcomes for our organization. We all want to make sure that our systems don’t get hacked. We all want to make sure that we can get the new features out quickly. So, how do we work together to achieve those outcomes? I often say my number one DevOps hack, find the person in the organization who everyone slags off, which in the dev world is often the InfoSec people. Go and take someone out for lunch from that team. And just spend an hour doing active listening to them, finding out what’s pissing them off and what’s getting in their way, and find ways to help them get better at what they do. You’d be amazed at how— I mean, you wouldn’t be amazed, but people in general would be amazed at how effective that is.

Bridget: [00:31:06] So, I do need the British to American translator. Is slag off makes fun of, or doesn’t listen to, or hates? What does that mean?

Jez: So, if you slag someone off, it means you’re you’re trash-talking them. Right.

Bridget: And, you know, I think that’s really, it’s really common for people to have that team that they don’t like. It’s not even necessarily individuals. Sometimes it’s just these tribal identifications inside an org. That team is the one that whenever they YOLO something out to production, I get paged, so I hate them. Or that team is the one that doesn’t care about patching their infrastructure because they’re too busy opsing so hard. And so, like, the security team hates them, or whatever. Like, if you have— for our listeners who are individual contributors and maybe can’t set, you know, policy at a really high level, do you think that that suggestion of just going out to lunch, or, you know, opening a Hangout, or whatever, with someone is enough? Or, like, tactically, what other steps would you suggest for them to try to build bridges across their org?

Jez: [00:32:10] I would say you will get a really long way by doing that. I mean, people call it networking, but really it’s about making friends and trying to understand the different perspectives. And so much of what I’ve learned in my career is by kind of randomly meeting people and talking about stuff, and they’re like, oh, I’ve had that problem, here’s what I did. And oh yeah, you know this rule? Well, this rule is in place because of this thing, and this is the person that had that problem, and this is— and, you know, you you get to understand some history and some different perspectives. Empathy and understanding these things, that’s crucial. This also brings us back to the James Damore thing. He talks about how women tend to be better at empathy than systematizing, whereas men prefer to be better at systematizing than empathy. There’s 2 pieces of this. One is, to what extent is that biologically determined versus the products of biology and society combined, which, you know, is a discussion that’s worth having. But also, the idea that our biggest obstacle is a failure to systemize is to completely misunderstand the nature of the problems we face. Most of the problems we face are due to a lack of empathy. And, you know, I’ve worked with people like this who will spend months reading documentation and processes and things like that, and come up with this amazing way to approach a problem, and then run up against a brick wall because they forgot to talk to someone who could actually help them achieve that, who probably knew all that stuff and could just, you know, actually help them do it. So, you know, we’re working in teams, we’re building products, we’re trying to achieve outcomes in complex systems of people. Empathy is important.

Bridget: [00:34:06] And I think that that’s, it’s interesting to me that you bring that up specifically because in the DORA assessments, when you go out with what you’re doing right now, when you go out and you look at what’s going on inside teams, how do you measure for that? Like, how do you get some kind of idea of what the crosstalk between teams or inside teams looks like? What does that part of the survey look like?

Jez: Well, we have a couple of things that we look at. One is the extent to which when Dev and Ops collaborate, the outcome is win-win. The other thing is we actually measure culture based on a construct created by a sociologist called Ron Westrum. This is something else that John Allspaw turned me on to. He was studying safety outcomes in healthcare and aviation, and he found a way to basically predict how effective people’s safety culture was by measuring along these 6 different axes. And, you know, we can put this in the show notes, I imagine.

Bridget: Absolutely. We’ll put a link to the Westrum model or whatever it’s called. I forget what it’s called.

Jez: [00:35:07] No, that’s right. And so, you know, we can measure where you fall on that. One of the things that we look at is how well people collaborate between different teams and departments. That one of the things that his model is telling you is information flow. It turns out making sure that the right information presented in the right way gets to the right people at the right time, that is incredibly critical in order to be able to develop resilient, agile organizations which can detect and respond rapidly to changes in the environment, that all depends on information flow. So much of information flow is dependent on cultural things. You can build the best systems in the world and use the best tools in the world, but if your cultural organizational structure is wrong, it just won’t work.

Bridget: And I think this is a really good point for, especially when you’re consulting or going into organizations and, um, or working as a vendor or partner with them. Uh, I think a lot of times people want to buy the DevOps in a box, they want the solution, and you’re like, but the solution is going to look different at your organization because you’re going to implement it in a way that works for you. And what you’re describing here, like the, you know, how healthy and collaborative the culture is, it occurs to me that that’s very related to how successful their continuous everything is going to be.

Jez: [00:36:41] Absolutely. I mean, one of the problems, there’s a lot of teams using these kind of big, big agile frameworks. There’s a bunch of them about. I’m not going to name names because I think you all know who I’m talking about. But, you know, these frameworks talk about, you know, here’s the processes that you put in place, Here’s the different roles, you know, you’ve got to create a backlog, and then you’ve got, you know, your release trains, blah, blah, blah, blah, blah. That’s all very well, but I’ve seen a lot of organizations adopt those but not change the way they do procurement and contracting, the way they think about leadership, the way they think about organizational structure. And if you don’t do that stuff, it won’t make any difference. You’re just shuffling around the chairs on the Titanic. So, the practices are useful, but the point of the practices is actually to change the behavior of the organization. And if you don’t change the behavior of the organization, you know, it doesn’t work. And this is why I have such a problem with cargo culting, which, you know, again, you know, devils in a box. If we just adopt these practices, everything will be fine. Well, those practices apply in a particular situation in the pursuit of particular goals. And if you don’t share the same initial starting state and the same goals, you shouldn’t be implementing them. So knowing where you are and where you’re going is the first and most important step. And if you don’t know where you are and where you’re going in measurable terms, you have no business implementing any practices. When you do try things out, you should be finding out if they actually move you in the direction you want to, and if they’re doing that fast enough, considering all the possible alternatives you could be investing in instead. This is what Mike Roller discovered when he was looking at Toyota and what made Toyota unique. People copied what Toyota did, but that wasn’t what was important. What was important was the ability of the people doing the work to recognize what their biggest obstacle was and experiment to remove that obstacle and then carry on doing that, not the particular things they discovered in the process of doing that. You’re copying the practices, you’re not copying the mindset and the culture of continuous improvement.

Bridget: [00:38:59] Right. So, like, so you have an andon cord, that’s cool, but if people are going to get fired when they pull it, then just having an andon cord for people to stop the production line is not going to produce the same results that it did at Toyota.

Jez: Absolutely. That’s such a great example that for me, if you go— hopefully, we’ll put this in the show notes, the story of NUMMI that’s on This American Life, that’s exactly one of the things they talked about. Other GM factories in North America tried to copy what NUMMI had done in exactly the way you described. Because the management processes weren’t in place, it didn’t make any difference. You would get fired for putting the andon cord because managers were rewarded based on how many cars came off the production line, whether or not they worked.

Bridget: Yeah, and that’s— and we, we will definitely put a link in the show notes to your, um, your Agile conference, uh, keynote too, just because we’ve talked about that and I know that you went over that somewhat in there. Um, and that, that makes me think too, like, sometimes it seems like people cannot affect organizational change as a single individual. Um, when, when I talk to people who say like, well, we don’t have buy-in from management, maybe this one you know, app dev group leader agrees that we should have improvement of some sort, but we don’t really have buy-in from above. We’re gonna, we’re gonna cowboy up and start doing this grassroots thing. And I think, well, I’ve seen organizations where that was definitely helpful, but what’s your, what’s your view on where you need buy-in for anything to work? Or conversely, if somebody comes in and says like, I am the new leader, we’re definitely going to have giant IT improvement everywhere, and the people are dragging their feet because they don’t want their jobs to change. Like, can you talk a little bit about where in an organization you need change?

Jez: [00:40:43] So, there’s a great book called Leading Change by a guy called John Kotter, and he has, you know, an 8-step plan. And step 1 is this: a majority of employees, perhaps 75% of management overall, and virtually all of the top executives, need to believe that considerable change is absolutely essential. So this is the sense of urgency. You’ve got to have a sense of urgency, and the way that exhibits itself is in ruthless prioritization and a focus on investing resources in achieving the highest priority goals and keeping that goal consistent. And it, you know, organizations don’t like doing that. I’ve worked in organizations where the leaders don’t want to expose the organization-level priorities in measurable terms because it takes away their ability to change their minds and to tell people what to do. That’s hard. But that, I think, is the number one thing. It does need to be at all levels of the organization, from the execs all the way down to the bottom. It’s much easier to change an organization where there’s a burning platform than it is to change an organization where everyone thinks that everything’s just fine.

Bridget: [00:41:54] That’s a really good point. If everything seems fine, why change?

Jez: Right.

Bridget: I don’t think most organizations that aren’t changing everything really is fine, even if they think it is.

Jez: Things are changing, right? Things are changing all the time. The question is, are they changing in a directed way through the conscious efforts of the organization, or are they, you know, is stuff just happening and I’m just doing my thing? This is what I’m working on. This is my job. I’m going to do this.

Bridget: Local optimization.

Jez: Right. Because you don’t have the context and you don’t have the direction. So, yeah, that’s a really big problem. And, you know, if you’re working in that kind of environment, I would say this: you can affect change. The impact you’re going to be able to achieve is going to be limited. That doesn’t mean you can’t do things. You can do things. The problem is doing things and making them stick and making them consistent. But, you know, conversely, organizations which have done really great things, and talked about them and written case studies and written books, people have left and then the whole thing has backslid. I mean, this is something we say right at the end of the Devil’s Handbook. This is something that I put in while we were doing our review. You know, I say this, we say this in the Devil’s Handbook, you know, the dirty secret is lots of these organizations that we’ve talked about, people have left and then the whole thing’s gone down the toilet. So, that’s life. There is no linear path from A to B. You will achieve great things, and then everything will go to shit. You’ll be in a shitty situation, and you’ll achieve great things. I mean, most of what went into this book came from me and a bunch of my colleagues working on miserable teams doing really horrible stuff with Bash and CVS. We were able to, like a team of 8 of us, was able to completely transform the way this application was deployed by using Bash and CVS. So, you can achieve great things with small teams, and most of that is actually working in horrible environments. So, just because you’re working in a horrible environment with terrible tools doesn’t mean you can’t do great things. I mean, that’s how I got where I am today. Conversely—

Bridget: [00:44:12] Let the record show it’s the Continuous Delivery book that Jez was holding up.

Jez: Sorry, yes, that’s the Continuous Delivery book. So, that was me working with Dan North and Sam Newman and a bunch of other amazing people, trying to solve these problems with basically no tools at all. It’s entirely possible to do that. I occasionally do meditation, and there’s this Zen temple down the road that I go to, which is just someone’s house, basically. They were doing the introduction to this particular meditation technique. And the guy was like, so, you know, sometimes you’ll come to the temple and you’ll be feeling really down, and meditation will really help you. And then sometimes you’ll come to the temple and you’ll be feeling really great. But don’t worry, because that feeling will pass too. I just thought, you know, that just sums up, you know, DevOps and the human condition.

Bridget: Absolutely. That’s fantastic. So, I love that. So, on that note, Where can our listeners find you, Jez? Online, in person, if they want to, you know, read some stuff? Maybe tell us about some exciting new book action that could be happening.

Jez: [00:45:25] Yes, so, um, Dr. Nicole Forster and me and Jean Kim are working on a book right now, uh, which summarizes and expands on the research in the State of DevOps report that we’ve worked on for the last few years with Puppet Labs. Um, that book is going to be coming out end of this year. If you come to DevOps Enterprise Summit in San Francisco, you will get a free copy of that book. So, working on that right now. Dora is everywhere. If I look at what I’m doing—

Bridget: Everywhere that you want to be.

Jez: So, one of my jobs, what you should do is go to devops-research.com. And sometime over the next couple of weeks, I’m going to put up a calendar of all the stuff we’re working on. So, you want to see what me, Dean, Nicole are all doing, Sue, all our stuff is going to be on there on the events page. So, head out there, subscribe to our mailing list, download our white paper. I’m also at continuousdelivery.com. If you want to find out all about continuous delivery, I stuck it all on a website. You can go to continuousdelivery.com, all my stuff’s there. I also have a— I teach a class in product management, at UC Berkeley, which is going to be starting up end of this week. All the class material for that is at a website, leanagile.pm. So if you want to download all the product management stuff that I teach, you can go there. I’m also working on a training class with my Lean Enterprise co-authors, Barry O’Reilly and Joanna Milewski. We put together a training class on the Lean Enterprise stuff that will be out later this year. So yeah.

Bridget: [00:47:06] So what you’re saying is you’re not busy at all?

Jez: No, I have nothing to do.

Bridget: It’s just relaxation and eclipse viewing.

Jez: And one of my personality traits, which probably is very annoying for everyone who I live with, is delayed gratification. So I’m like, you know, I’ll just work on these things and then everything’s going to be much better. And I’m very good at delaying the payoff. And I’m, you know, one of my life goals is to actually, you know, achieve the payoff and be like, done now, let’s go and sit on the beach.

Bridget: I’m not sure if being done is something any of us are ever going to achieve, but I like the path that you’re on.

Jez: That’s pretty cool. That’s right. And whenever I think I’m doing badly, I just go and look at Nicole because Nicole is— I mean, if you think I’m a workaholic, I look like such a lazy bastard compared to Nicole. It’s not funny.

Bridget: It seems like every time I talk to her, she’s doing something else.

Jez: I mean, really, Nicole, just watching Nicole become CEO of Dora, having worked in academia, and then, you know, she’s just like, now I’m gonna be a CEO. What do I need to learn in order to be effective at this?

Bridget: [00:48:23] Oh, Nicole is fantastic. We’ve, of course, had her on the show several times in the last 3 years. So, I’m glad that we’re getting you on the show. We just have to We’ll get Sue on next. It’s like, let’s see, probably we’ll catch her. Sue is speaking at a lot of DevOps Days lately, so we’ll probably catch her at one of those. Speaking of which, the OpenCFPs on devopsdays.org/speaking are something that a lot of our listeners might want to check out. There are a bunch of cool places that you could go and speak at, perhaps a place near you.

Jez: And be aware, like, you probably, and you being everyone in the world, you’re probably doing things that other people would be really excited to hear about. You just don’t know it. So, you know, a lot of what I kind of realized was actually really important was stuff I was like, oh, this is kind of cool, but I bet everyone is doing this. That may well not be true. And especially, I know a lot of people kind of think, well, I can’t teach the world anything. I very much doubt that. All of us have amazing things to teach the world. The trick is finding ways to express that in a way that communicates effectively and changes people’s behavior, we are far away as an industry from being as effective as we could be at that. So, if you’re not sure, if you have something to teach someone, try it out. Because I bet you do.

Bridget: [00:49:44] Absolutely. So, discount codes ADO2017 will get you 20% off a lot of different DevOps Days. It will also get you 20% 20% off Velocity New York. If you— I am on the programming committee there. I am not speaking, thank goodness. At least I don’t think I’m speaking. Nicole suckered me into giving an Ignite at Velocity Santa Clara. No, wait, San Jose. And I was just running a track. I wasn’t speaking. Suddenly I was speaking. So we’ll see. But I think I’m not speaking at Velocity New York, but I am running a track and it’s pretty exciting. And you can get 20% off that with ADO2017 as well. I don’t know if this is going to go live before I’m speaking at Uptime in Pittsburgh later this week. It probably won’t because Joe does the editing and he’s very, very busy this week, so it might not get to it until next weekend. But get in your time machine and go to Pittsburgh and see me speak at Uptime.

Jez: The other thing, just hot off the press, 18F is advertising for positions. 18F is an amazing place to work. You can work remote. I work from home. It’s a remote organization. Um, they will be opening up the cloud.gov director position really soon, um, so keep your eye out for that. That’s an amazing job being— working on an incredible team doing really important work.

Bridget: [00:51:00] Awesome, that’s very cool. Um, okay, so let’s see, is, is that the main thing that you want us to check out, Jez, or do you have anything else you’d like us to check out?

Jez: Uh, on, on the work front, I think that’s— that those are the main things that I’ve got.

Bridget: I will, I will also say to— so we’ll put the link in the show notes to cloud.gov to the job postings there. And, uh, I also am going to drop some links into my favorite food blog, Smitten Kitchen, and a new-to-me food blog, Mountain Mama Cooks, that I found the, the latter when I was Googling for heirloom tomato, um, you know, canning recipes. So tomatoes are— when you’re growing tomatoes, it’s like green, green, green, green, green. Oh my God, they’re all ripe. You got to do something with them in the next 2 days. And here are, you know, pounds and pounds and pounds or bushels or whatever it is, you know, kilos of tomatoes.

Jez: So very, very awesome.

Bridget: I canned a lot of tomatoes last night.

Jez: [00:52:01] Cool. While we’re talking about cooking, My wife and my favorite cooking show is Manjula’s Kitchen. If you like Indian food, check out Manjula’s Kitchen. That’s really good. The other thing I want to recommend, this book that I’m reading right now called Plutopia: Nuclear Families, Atomic Cities, and the Great Soviet and American Plutonium Disasters. There’s an article on salon.com recently, it’s on my Twitter feed somewhere, which talks about the Department of Energy and the really important work it does. This is the agency that is now— the person in charge of that agency was a presidential candidate who was governor of Texas who said that he wanted to abolish it. It turns out that a really big chunk of the Department of Energy’s budget is spent on cleaning up this small town called Hanford up in Washington, which was where the US produced the plutonium as part of the Manhattan Project for the first atomic bombs in the ’40s. So this story is really interesting because it talks about that and it talks about the social environment they created. It has really interesting discussions of kind of society at the time and what that meant for the plant and how the plant changed society and all the terrible things that happened there. If you’ve seen Fox Hunter, the movie. This is a movie about the guy who is in charge, the family heir, the DuPont family heir, who actually shot— he took over sponsorship of the US Olympic wrestling team and ended up shooting one of the wrestlers. It’s crazy, but DuPont was the prime contractor on Hanford. So reading about basically how DuPont made enormous amounts of money basically building this city that manufactured plutonium, fascinating stuff. Highly recommend it.

Bridget: [00:54:01] Well, that is really interesting.

Jez: Foxcatcher, not Foxhole.

Bridget: Okay.

Jez: Okay.

Bridget: That’s really interesting. I’m going to have to read that because I was actually just at a wedding. I was at Patty and Ethan’s wedding in the Tri-Cities. And it’s very close close to there. For whatever reason, the reactor— like, there are tours, but I guess the tours weren’t available over the weekend that we were in town. So they were doing some sort of, you know, important nuclear maintenance or whatever that I guess you would want them to do so that there’s not any kind of meltdown. Um, but so we couldn’t go on a tour, but it— I was actually right there in the kind of the high desert in eastern Washington State, which is a very interesting landscape.

Jez: So that was one of the reasons they picked it. Yeah. Fascinating.

Bridget: That’s the Tesla road trip that Joe and I took last month. So that was good times too. That’s, I suppose that’s another recommendation is if you get a chance to try some electric vehicle road tripping across the US, it definitely builds pauses in, but at least if you have fast charging, it’s not the kind of, you know, wait for a day before you can keep going pauses.

Jez: [00:55:08] So yeah, so my dirty secret is that I just got a Tesla as well. I guess it’s a clean secret.

Bridget: Which one did you get?

Jez: I got an S. I feel terrible for admitting this on air, um, but it’s an amazing car. I love it so much. One of the things you can do is camping in it.

Bridget: Um, wait, you got the S or the X? The S. Okay, that’s the same one we have.

Jez: Yeah, yeah. Um, so you can do camping in it. You can put the back seat down and you can keep the electricity on all night without having the headlights on. And then because it’s got a glass roof, you get this amazing view out the back. I haven’t tried this.

Bridget: Wow.

Jez: I suggested it to my wife in an email, sent her the link, and she sent me back a one-word email reply, no. But maybe you’ll have more luck than I do.

Bridget: You know, actually, Joe and I, we did consider trying to do some kind of car camping out of the car when we went last month to his family reunion, which is up in the Northwoods of Wisconsin, right on the Michigan border, where most of the family all brings their campers. But we don’t have a camper. And we also bought a Model S, which means we can’t tow anything. So a pop-up camper was not going to happen. But we did end up finding kind of a little B&B not too far away in a town that had some public charging, which was very nice because Right. As it turns out, if you don’t have charging, it’s a very limiting factor. That’s the one thing to consider.

Jez: [00:56:31] Yes, that sounds spot on.

Bridget: Um, anyways, so we should probably wrap up, and I— so I can let you go. We’re running late, as we always do, but it’s so fun to chat with you, Jez.

Jez: It’s been an unalloyed pleasure. Thank you so much for having me on the show.

Bridget: Yeah, um, awesome. So yeah, head over to arresteddevops.com/ci-cd for this episode’s show notes. I mean, not right now if you’re still on the live stream, but eventually. And the site also has our newsletter, merchandise, Patreon, all the Arrested DevOps stuff you could ever want. Uh, visit arresteddevops.com/itunes and leave us a review in the iTunes store if you want to help other people find this podcast. I’m not really sure how that works, but apparently that’s a thing. So yeah, so again, thank you so much, Jez. I’m really, really glad you were here.

Jez: This was super fun.

Bridget: I’m Bridget at Bridget Kromhout. This is Arrested DevOps, and remember, there’s always DevOps in the banana stand.