continuous delivery

Matty: [00:00:07] Welcome to Arrested DevOps, episode 15, Continuous Delivery. I am your co-host, Matt Stratton. On Twitter, I’m @MattStratton.

Trevor: I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.

Matty: Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that figures if you’re listening to this podcast, you’re pretty cool. You can find out about joining their cloud services team at 10thmagnitude.com. I also want to remind everybody really quickly that CFPs have opened for the first ever DevOps Days Chicago. You can check out all the details at devopsdays.org. If you have an idea for a talk, even if you haven’t ever spoken at a conference before, submit something. It’s super fun, and you know way more than you think you do.

Trevor: One of the commonly associated principles with DevOps is that of continuous delivery. Continuing upon our previous episode on continuous integration, we have a special guest, Jez Humble. We’ll be talking about how continuous delivery can help your organization and how Jez has seen the world of DevOps change since the first publication of the Continuous Delivery book.

Matty: This episode is sponsored by PagerDuty. PagerDuty eliminates the noise, chaos, and manual processes across the entire incident lifecycle to decrease resolution time. PagerDuty is trusted by companies like Etsy, Nike, and GitHub. To sign up for a free 30-day trial, visit arresteddevops.com/pagerduty.

Trevor: [00:01:24] This episode is also brought to you by CodeShip. CodeShip is continuous delivery made simple. Set up continuous integration in a few steps and automatically deploy when all your tests have passed. CodeShip has great support for lots of languages and test frameworks. It integrates with GitHub and Bitbucket and lets you deploy to cloud services like Heroku, AWS, Modulus, Google App Engine, or even your own servers. Setup only takes 1 minute.

Matty: Find them on arresteddevops.com/codeship and be sure to use the discount code arresteddevops to get 20% off any So, we’ve gotten some great feedback from our listeners since our last episode, which was How to F Up DevOps, which is at arresteddevops.com/14. And some of the feedback and requests we’ve gotten is for us to go a little deeper into some details. And I think that this episode is going to be a great chance to mix both theory and practice, especially with a guest that I’m super pleased to introduce, Jez Humble. Jez is one of the authors of the book Continuous Delivery and also Lean Enterprise. Welcome to the podcast, Jez. So, Jez, you want to give us the quick, quick intro for people who may not be familiar with you, whoever they might be?

Jez: [00:02:31] First of all, thanks very much for having me. It’s a pleasure to be on the show. My name is Jez Humble. I’m co-author of the book Continuous Delivery and also forthcoming book called Lean Enterprise. I’m a principal at ThoughtWorks and currently also working on a conference called Flowcon, which is this fall.

Matty: Great. So I’d like to get started with just maybe kind of the 30,000-foot overview of continuous delivery. And we talk about it a lot, But in a nutshell, how would you describe it to someone who has never heard of it before?

Jez: Okay, so I guess an easy way to do it is to contrast it to a more traditional way of doing things, which is, you know, very phase-gatey. Whenever you want to build some new product or service, you start with a kind of planning phase, and then you do development, and maybe you’re doing development in these nice kind of iterative chunks, but nothing actually gets deployed to users until right at the end. And you might have an integration phase where you try and integrate all the different components together, and then a testing phase. Phase, and then, you know, release to production. And that’s normally pretty miserable and painful. And obviously, what often happens— I’m sure many of you have experienced this— is the kind of integration and testing phase tends to telescope and be very unpredictable. And so the idea of continuous delivery is, don’t do that. Instead, have something very, very small that’s deployable from day one. Just feature number 1, which might be just having a little status page. Where you can see what build the app came from, and then keep building out the app, but make sure that every piece of work you do, every feature, every bug fix, gets as far as possible towards production, and that your software is always kept in a releasable state, and that you prioritize keeping your software releasable over doing new work. So that’s kind of the basic principle. And you can do that even if you’re not building web services. One of my favorite case studies is from the HP LaserJet firmware people, where, again, they’re building firmware. Typically, people don’t update the firmware on their printers 10 times a day, but Actually, just making sure the software is always in a releasable state changes the economics of software development. It changes people’s behavior, and it makes sure that you build quality into the product.

Matty: [00:04:33] Yeah, I love that story too. Actually, the first time that I’d ever heard of continuous delivery, you were on DevOps Cafe a few years ago. I was listening as I was driving in my car, and you were talking about how continuous delivery worked. Actually, I worked for an internet company at the time. I was sitting there in my car saying, oh, well, that wouldn’t work for us. That wouldn’t work for us. That wouldn’t work. Then you threw out the firmware story, and I went, oh! Oh, okay. I guess it could work. So, but what makes it successful? I mean, why should we do it besides the fact that it’s now just sort of the canon of that’s how you do DevOps?

Jez: I think what makes it successful is that it makes you deal with a lot of the problems that typically you don’t have to worry about until the software goes into production from early on in the project. So things like worrying about scalability, availability, actually things like logging and monitoring and all these things that typically, you know, every good project, you put these in as requirements at the beginning and you’re going to pay lip service. Oh, well, obviously you want it to scale and not to fall down and that kind of thing. But knowing that you’ve got to do something is very different from verifying that you actually did it. And so I think it forces you to deal with these issues from early on. It forces you to think about, well, you know, before I can say I’m done with my story, I need to have it working in a production-like environment and actually make sure that the thing stands up and that it runs. You know, and as your project progresses, you might do things like adding a production-like dataset in there, running production-like load against the software to make sure it works, and to make sure you’re doing that from early on. And however good you think you are at doing that, and however great the architecture is that you think you’ve put in place, we’re dealing with complex systems, and you can’t predict how the stuff you’re building is going to behave in a real environment. So forcing yourself to do that from early on forces you to actually deal with those problems when they’re relatively easy to fix. The idea that we can build in scalability or build in performance afterwards, you know, that it’s something we can just fix later, or monitorability or operability, is just false. I mean, these things come from our architectural decisions, and the point at which we want to validate our architectural decisions is early on in the project when the things are easy to fix. So I think that’s kind of the next level of, you know, what makes this actually work.

Matty: [00:06:39] So we’re not just testing our code, we’re not just testing our features, we’re testing our ability to deploy, and we’re testing our ability to operate the system that we want to put out there then.

Jez: Absolutely. And, you know, the nice benefit of that is it helps developers to learn those skills, which are really important.

Trevor: So continuous delivery sounds a little intimidating for the first time you hear it. How do you recommend getting started, and do you feel it’s possible to make a change from kind of the traditional path, as you stated, or do you have to do it with a greenfield project?

Jez: Well, it’s certainly a lot harder if you’re not working from greenfields. If you already have kind of large, complex systems in production and this kind of gnarly production environment that a lot of places have where no one knows where anything is or where the config settings are or sometimes even where the source code is, it’s going to be a lot harder than if you’re starting from scratch. But just because it’s going to be a lot harder doesn’t mean that you shouldn’t start doing it. And really, I think the idea that, okay, we’re going to do continuous delivery as a project where we kind of plan it all out and then we execute it and then we’re done, that’s as fallacious in terms of adoption and organizational change as it is in building products, right? Really, continuous delivery is just continuous improvement. It’s working always to get a little bit better in terms of your ability to ship production-ready software, make the release process really boring. How can we get better at that? How can we move towards that? And there’s tons of stuff you can do there, and the place to start is the place where you think you can get the biggest bang for your buck. I mean, a great way to start is by mapping your value stream from check-in to release, looking at all the activities you go through, and looking for the ones where you think, well, gosh, if we just did a few things here, we could probably reduce the number of times we need to kind of do this rework. So a typical problem is getting a test environment to deploy into, getting a production-like test environment. Typically, it takes a while to get those set up. It takes a while to get the software deployed to them. When you do, you have all this trouble, and you need to keep doing it again and again and again. So, you know, if that happens to be the case, you might start there just by doing something very simple. And again, don’t take the entire process and automate it all. Find ways to, you know, make that process incrementally a little bit better, maybe simplify it a bit, automate a part of it, and then just keep doing that, but across your whole value stream. I mean, if you’re fixing the part of the problem that isn’t actually the constraint, you’re not going to have any impacts on the end-to-end cycle time, which is what we really care about— how quickly can we take changes from check-in to release. And, you know, a common place to start is with development. I’m always banging on relentlessly about continuous integration, because here we are in 2014, people still say they’re doing it when they often aren’t. And actually, as a developer, making sure that you can get a build, you know, get packages built, and get feedback on any of the major bugs in a rapid, automated way, and then crucially fix them once you find them, being able to do that is really the first step. Step 0 is version control. Step 1 is being able to get automated builds and get feedback on those builds really rapidly. Rapidly, and then have the discipline to actually fix the problems straight away when you find them, rather than just leaving the build red and being like, oh, the build’s red.

Matty: [00:09:39] So, one question that I have that’s come up when I’ve worked with people and we wanted to do continuous delivery and, you know, wave the book around as my Bible and talk about, you know, happy path, one path to production, you know, everything goes through the same path. And then where things kind of get confusing sometimes is with complex systems that depend upon each other, that maybe aren’t as loosely coupled as we’d like them to be. And that don’t necessarily move through, you know, as cleanly. And so what’s your thoughts on, like, I used to be, I guess I might still be, you know, really adamant about, well, then we need to fix the architecture so that you can go through this happy path. Or is it, I hate to say, is it okay, but to kind of have multiple pipelines, like each of your product has its own pipeline, but the changes to that product go through the same path? Are you accomplishing CD, do you think, in that manner?

Jez: I think, you know, there’s many ways to do it. And actually, the kind of key patterns that I like to talk about, there’s kind of the Amazon/Netflix way where you have lots of relatively small services. You know, I know microservices is a big buzzword, but basically what happens is you just make sure architecturally those, like you say, are very loosely coupled and you can deploy them independently of each other. And then you do some level of testing before you deploy, but you also make damn sure that if you deploy a service, and something goes wrong that you can roll back. A good way to do that is just by versioning them. You make sure you deploy the new version alongside the old version, and the clients can update to the new version when they want to. That’s one way of doing it. Another way of doing it is using kind of blue-green deployments. There’s like a ton of techniques you can use to make sure that if something goes wrong when you push out a service that you don’t just break the whole world. Monitoring is really important in that world too, being able to see, you know, this is something that If people haven’t seen it, Steve Yeager’s Platform Rant is really interesting, and it talks about some of the trade-offs that you face in this architecture. But one of them is, you know, a web request comes in, it goes through about 100 different services, and comes back the other end. You have a performance problem or a latency problem, which of those 100 services is actually causing the problem, or which combination of interactions between those services is actually causing the problem? That’s a really hard question to answer, and you have to have really good monitoring in place to be able to trace those calls through the various different layers of the app and app backend to be able to diagnose those problems. The other main architecture that we see is kind of Facebook/Google, where you basically have an enormous binary that’s combined at build time that you push out in a kind of horizontally scaling way to everything. And then the call graph is much, much shallower. I mean, you do still have services in Google, so things like Bigtable and so forth. You’ve got these lower-level services, but there’s much fewer layers and kind of much more big monolithic things going out. You can do it that way, but what Google do when they have to push out these big monolithic binaries is they do continuous integration at the code level much, much more rigorously. So they have a very simple system where you can see for any code change the impact on either downstream products that consume the component that you’re working on, and they actually run the tests for all those downstream dependencies very, very rapidly to give you feedback straight away. So if you’re doing the big monolithic build thing, that’s totally fine. That’s a perfectly reasonable architectural decision. There’s trade-offs. There, but then you have to make sure your CI is really, really, really good. Where do you see it?

Matty: [00:12:53] I know another place that— a little bit putting on my devil’s advocate hat, because again, having been a consultant trying to help people along with this, I, you know, where everybody kind of throws up their hands and goes, oh, but what about this? But one place that I’ve definitely seen big challenges, right, is with data. Because we talk about like, okay, my ability to roll back. Well, you can’t really roll back a schema change, you know, because it’s always moving forward. And so Is that a place to even start thinking? I know it’s not really sexy to start thinking about your data because it’s not really features, but to me, that seems foundational to be able to have that agility is to have whatever shims around your data.

Jez: Yeah, I think that’s absolutely right. My whole career is built on talking about things that aren’t sexy, so databases don’t scare me. I think that the pattern that we see for people who’ve been able to do this very effectively is a pattern called expand contract that Michael Nygard talks about in his book Release It, which is a very powerful pattern. It’s not just for databases. It’s for all kinds of things. But basically, you never change existing objects. There’s this concept of immutability, which is very fashionable in talking about languages, but it also applies to infrastructure as well. We don’t change any existing objects in the database. What we do is we add new objects, and you can do that ahead of time. The stupid example I like to use is changing address. You have an address field. You want to turn that into address line 1 and address line 2. You don’t delete the old field and create new fields. What you do is you put the new fields alongside the old ones. And you can make that schema change well ahead of the deployment of the app. And then the app has a little abstraction layer where essentially the app looks to see if the new objects are in the database. If they aren’t there, then it just uses the old objects. If they are there, what it does is it tries to read from the new columns, address 1, address 2. If those are null, then it reads from the old column. But then it writes to both so that the application will lazily migrate the data. And then what you can do is if you need to roll back the app, you can roll back the app, and that’s fine because the old objects are still being updated with the new data. So you’ve basically decoupled the deployment of that app from the deployment of the schema changes in the database. And then once everything’s gone fine, you can then do a batch migration of the rest of the data that hasn’t been lazily migrated, and then go and delete the old columns if you need to. And if you look at organizations like Facebook, I’ve tried many times over the years to try and get someone from the Facebook DBA team to come and talk publicly at a conference. I’ve never succeeded. So if anyone’s listening that’s got deep contacts in the Facebook DBA team, try and get them to talk about what they’re doing, because I think that would be doing everyone a big favor. But from what I hear, they basically make no guarantees about what version of the database is in production at any one time. And so the developers have to defensively code against those kinds of changes. So the DBAs are like, screw you. We’re going to do what you want. We’re not going to tell you. You’re going to have to work it out. And so, you know, the important thing to bear in mind is that adds complexity to your app. You know, there’s that old saying in computer science, there’s no problem you can’t solve with an additional layer of indirection. So adding a layer of indirection to solve the problem of decoupling database changes from application changes, but that adds complexity to the application. What you get in return for that is a lot more deployment flexibility. So again, trade-off.

Matty: [00:15:59] So that being said, as you move towards more of this mature deployment flexibility, your development skills have to become more mature, right? You can’t keep developing the way that you’re used to. You can’t just say, well, this is the way I’ve always done it. You kind of have to— like you said, I kind of like the way you said it, where they have to code defensively, right? And say, OK, I can’t— you know, it’s right, it’s the Code Monkey thing, or the Chaos Monkey thing too, right, with Netflix, which is like, I can’t assume that my stuff is always going to work, so I need to think about that. And that’s great if you’re someone who thinks that way, but I think it’s going to be an interesting shift in our industry as more people have to be able to do that.

Jez: Yeah, I think that’s absolutely right. This absolutely changes the way you think about development. A lot of the other things we talk about in the Continuous Delivery book, that requires changes in the way people think about development. One of the things that we talk about is breaking large changes into small incremental chunks that keep trunk releasable. And there’s a lot of complexity in that simple phrase in terms of how you do that, and it changes the way you think about things. You know, you can’t go off on a feature branch for, you know, days on end. Developing something and then integrates it at the end. We have evidence that shows that. I’m going to have to mention the P word. The Puppet DevOps State of Practice survey that we did recently, we actually got data on that, and we showed that working in small incremental steps increases IT performance. That practice increases IT performance. We have data that shows that that works. We talk about it in the book, but it’s hard for developers to do it. I have developers— I hear developers whining about, oh, there are some things you just can’t break up. There aren’t. You can, with sufficient skill and ingenuity, you can do it. And it may take you longer to work that way, and it may be more painful to work that way, but, you know, the question is, what do you want to optimize for? People often want to optimize for how fast they can say they’re done on their feature branch rather than how quickly we can get changes out to production. So, you know, we don’t actually want to optimize for how fast I can say I’m done on my feature branch. That’s not something we actually care about in real life. So, yeah, there’s a lot of culture changes, a lot of changes in development practice, and, you know, what continues to astonish me, you know, being a bit of an old get-off-your-lawn Gulfmind and Git is developers don’t like it. They don’t like changing the way they do things. And I kind of think, well, why did you go into the technology industry if you don’t want to change the way you think about things? Because things are going to change. Technology changes. The tools change. Everyone gets into a frenzy about new language X and how exciting it’s going to be. But ask developers to change their practices, and they start getting all kind of grumpy and defensive.

Trevor: [00:18:28] So, we talk about versioning, and that’s, that’s something we were actually talking about recently, not on the show, just in a conversation I was having with some friends. It sounds like we’re talking about every commit should be versionable. What does version mean to you and to continuous delivery?

Jez: That’s a great question. I mean, the crucial thing about versioning is that you need to be able to identify all the changes in as small chunks as possible. Because— well, for a few reasons, really. One, so that you can reproduce the state of the system at any point in time for debugging purposes. And a lot of this does go back to debugging. Also, if you’re checking into CI, and you’ve got— particularly in a large team, you’ve got all these different changes coming in all the time. You’ve got to be able to quickly find out which one caused the problem, especially if there’s some longer-running test, like a performance test, that you don’t want to run. I mean, it might take hours to run a performance test or a soak test. It might take a long time to get the feedback. There might be, you know, 20 different check-ins between the last time it ran and this time it ran, and there may be some huge performance degradation. And the question is, who did it? So you need to be able to do, you know, a binary search through the versions to be able to find out which change actually caused the problem. And that, for me, is the thing about versioning, you know, working in small incremental steps, versioning everything. When something goes wrong, you need to be able to, you know, do that binary search and find out what the problem— where the problem was introduced and whose fault it was. Well, not whose fault it was, but who introduced the change, and then fix it. That, for me, is one of the biggest things. There’s a lot more to it than that, but that’s my kind of— that’s the top thing that comes to mind.

Matty: [00:20:00] Okay. So, yeah.

Jez: Okay.

Matty: So, I just want to kind of take out the thought. One of the things I really wanted to talk about with you is that— so your book, so you and David’s book were published pretty much like almost exactly 4 years ago. When I looked it up, it’s like the publication date’s like August 4th, 2010. So, happy almost 4th birthday, Continuous Delivery book. What’s changed? So 4 years, I mean, you could say that’s a long time, or you could say it’s a blink, right? What do you think has changed around software delivery in those 4 years?

Jez: Well, I mean, there are some broad trends that have changed. Obviously, you know, mobile wasn’t nearly such a big deal 4 years ago as it is now. We’ve seen the cloud continue its kind of slow juggernaut-like move towards ubiquity. The whole Internet of Things. Obviously, a ton of tools have come into the market, lots and lots of new tools. You know, this is the essential paradox of working in technology, is that all this stuff has changed, but then there’s also a whole bunch of stuff that really hasn’t changed at all, you know, around practices and around processes and the way that we think about software development. We’re actually still very conservative in the industry about the way we manage projects and the way we do product management. And the way we do software development. That is taking a really, really long time to change. I think a lot of it is we just have a terrible grasp of our history as an industry, and we’re very bad, actually, at communicating ideas around process change and around changes in techniques. We’re also really bad at experimenting with them. IT is a really curious industry because it’s really cutting edge in some ways, and it’s really backward in in other ways, and culture is another one of the ways in which it’s very backward. I think it’s still very cliquey, obviously still lots of white guys everywhere. So there’s all these ways in which it’s very, very conservative as well. And that’s reflected in my experience with continuous delivery. There’s a lot of tool support for it, there’s a lot of people who are talking about doing it, but, you know, there’s still, I think, the overwhelming majority of places who are doing software development in the world Still got a long way to go. I mean, to some extent, this is an echo chamber that we’re in right now, and it’s going to be 5, 10 years, I would say, at least before continuous delivery becomes the standard practice, as you said.

Matty: [00:22:17] Do you think that part of this— because I kind of think about where ideas like CD and everything, you know, they’re obviously very popular in kind of the startup world because you have, first of all, I guess, the advantage of it’s greenfield, right? You don’t have 20 years of technical debt. Do you think it might have something to do with how people are incentivized? I can’t even say that right. Or, you know, what their drivers are? Because if you’re in a startup, right, like you really care about, you know, making sure the company survives because you’re, you know, so that innovation is what’s going to let you, you know, get paid next week still. And then when you’re in kind of more on the enterprise side, you know, not to say that people who work in enterprises don’t care about their company because they do, but they probably care a lot more about, you know, their one-on-one weekly meeting with their boss about getting chewed out, right? Or like you said, being able to say like, check, feature done.

Jez: I mean, what—

Matty: I’m kind of at a loss as to say how to make this change, right? You know, and we’ve talked to people about it, and like John Olsbos said, he’s like, I don’t know what to tell people, I’ve always been in charge, you know. And so, you know, it’s— he could just say, this is how we’re going to do it, right? And it works great. But we talked about this a lot in our episode last week about, like you said, the echo chamber. It’s kind of like the people who get it already got it, and but there’s still so much more in the industry that have to get it. Like, how do we expand that tent, you know? And it seems to me like it’s maybe, maybe you’re not going to get there as long as we keep looking for the same things out of being a technical knowledge worker.

Trevor: [00:23:45] You know, before we get started on that, that actually reminds me, I had a message from Matt Shields this past week, who is somebody we’ve been talking to on Twitter, and one of the questions he said was It seems like the guests on your show work for smaller, mid-sized organizations. I work for a company with 35,000+ employees with 9,000 of us in IT. Even trying to work with DevOps methodologies can be challenging just from the standpoint that I don’t even have full access to my dev servers, and our infrastructure folks aren’t on a different floor, they’re 15 miles away.

Jez: Yeah, you know, and I think I understand that, but then, you know, honestly, and I say this with the greatest possible respect because, you know, ThoughtWorks consults for large companies. We’ve always worked with large companies. That’s what we do. My God, have we seen these problems close up? Absolutely. But then, again, a lot of it, and I say this with the greatest respect, is just excuses. Saying, as Matt said at the beginning, oh, that can’t possibly work here. It’s like, well, let’s assume that it can work here. How are we going to make it happen? Once you get rid of that mental obstacle, which is it can’t possibly work here, and you start instead thinking about How could I move just a little bit towards that? What little thing could I do? And you know, I think the difference is in a big enterprise what you have is friction. I’ve been doing this research for this book I’m writing, Lean Enterprise, and inevitably you always come back when you’re kind of researching this to 2 things. One is kind of the Toyota Production System, and the other one is maneuver warfare. And don’t ask me why that is, but that’s where everything seems to come back to. And so maneuver warfare It’s this whole kind of theory of war where basically Napoleon kicked the ass of all the armies in Europe, just destroyed them totally, and they were all kind of like, dude, what happened? And they were trying to work out what happened. And so the Prussian army— basically there’s these 3 generals in the Prussian army who— Clausewitz, von Moltke, and one other dude whose name escapes me. And they basically looked at what they did, and what they found is there was this whole thing about running things at scale where you have a principle of mission. Everyone knows what the mission is, and everyone is aligned on the mission, but then there is a lot of autonomy at lower levels for people to do their own thing and adapt rapidly to changing circumstances. So you tell people what it is that you want to achieve, here is the goal, and then how you achieve the goal is kind of up to you. You go and work out how to do it, and that’s basically the key to making maneuver warfare work at scale, is that idea that you tell people the outcome, but you don’t tell people how to get there, and they work out how to get there by themselves. And that’s actually kind of interestingly connected to the way the Toyota Production System works as well. So going back to your original question, how do you make this work at scale? I think big size isn’t actually the problem in many ways. I mean, Google has over 10,000 developers worldwide, and yet they’re able to do continuous delivery because they have a culture where anyone can check into trunk. And there’s certain repositories which are locked down, you can’t get to, like the crown jewels, like AdSense or whatever. But in general, everything is kind of open. And anyone can revert anyone else’s changes from trunk. So someone checks in a change to something, it breaks your stuff, you can just go and revert their change out of trunk. So there are ways to make this work at scale. Companies like Google and Amazon— I mean, Amazon has thousands of developers. Amazon in 2001 had this horrible, gnarly monolithic codebase that was miserable and painful to work with, and over the course of 4 years from 2001 to 2005, they completely re-architected it into a service-oriented architecture. Partly so they could achieve deployability, but also to enable them to do this thing that I talked about earlier from maneuver warfare, which is decentralizing authority and giving the teams the ability to do their own thing. And you’d say, well, here’s a fitness function that I want you to optimize on your team, which might be, you know, revenue for this particular thing or whatever it is. You go and work out how to do that. And so the architecture reflected the organizational structure that they put in place. Which is an example of Conway’s Law, which is the structure of the organization will reflect the architecture. So, there’s ways to do this at scale. The people who’ve succeeded are the companies who’ve worked out how to do this at scale. You can do it at scale. The problem is, it’s very hard to do it unless the leadership move away from this kind of command and control mentality to a mentality, you know, and it’s ironic that we use that word command and control, because command and control hasn’t been fashionable in military circles since Napoleon destroyed everyone else in Europe in 1806. So no one in the military is doing command and control since, like, well, over 200 years now. Moving away from that to this idea that actually, you know, we’re going to delegate authority to lower levels, tell people the high-level outcomes they want to achieve, and let them work out how to achieve it. And if the leadership isn’t like that, if it’s all political, if it’s all based on, you know, trying to get your big bonus at the end of the day, then you can’t do that. And that’s very difficult for the people who are on the ground trying to make change. You kind of have to do things under the radar somewhat, and at some point you’re going to come up against someone somewhere else in the organization who exactly as you say is incentivized in such a way to make it impossible for you to achieve your wider goal, and then you’re kind of hosed. So that’s a problem that no one’s going to be able to solve.

Matty: [00:28:46] Yeah, I think what echoes that is I remember, was it like a month or two ago, you were giving a talk in Chicago, and Torben and I were there, and I remember people were asking a lot of questions. It was probably the typical stuff you hear a lot about. This is why it won’t work, right? What about this? What about this? And I remember sitting there thinking, I was like, well, there’s like 100 people here and like 95 of them are all afraid they’ll get fired if they make the wrong move, you know? And that to me means that an organization that that’s the thing, if your first thing is when you say, I want to do change, how do I make sure that I’m protected from losing my job? Not from being automated out of a job, but because I’m, you know, getting fired for doing the wrong thing. You’re not going to take those. I even hate to say they’re taking risks because they’re really not, right? They’re just, it’s that trust, right? Like, I don’t know. I had to chuckle too with your, you know, let’s pretend it’ll work. When I was running a team of sysadmins, I would tell them that. I would say literally that all the time. I’d say, look, I know, guys, you can think of 1,000 reasons this is not going to work. Let’s just pretend it will and see what happens.

Jez: Yeah, I think that’s absolutely right. And this is what we see high-performance companies do. I was just again reading something about Toyota, and Toyota just sets these goals that are just outrageous and unachievable, and they’re like, Let’s assume that we can achieve this. Go away and work out how to do it. And that’s the way, if you do it right, that’s the way of empowering people. Now, again, there’s a doing it right and a doing it wrong, because all of us, I think, have experienced, well, we’ll ship it in a month’s time. And you’re like, no, we won’t. And they’re like, oh, yes, you will. And then you’re totally screwed and miserable, and you’re spending all your evenings eating pizza. So all of these things, all of these recommendations, there’s always a caveat, which is Modulo your culture, and I think this is one of the things that again is interesting about Toyota, is there was this thing that you wouldn’t lose your job. And one of the stories again that I really like is the story of NUMMI, the New United Motor Manufacturing Plant, that was a joint venture. It was Toyota’s first presence in the US. It was a joint venture between Toyota and GM in Fremont, just down the road from where I am. And they had this discussion where they were trying to work out whether people should be able to hire people within their team. Or whether it should be a central HR function to hire people. Now, you would think that Toyota’s all super flexible and modern, so they’d just let people hire their own teams, and they said, no, we don’t think that’s a good idea. And the reason is because if you work for your boss and your boss is the person who hired you, you’re not going to want to say no to your boss, because that’s the person who’s basically given you your job and who can fire you. And so Toyota says, no, you’re not hired by your boss, you’re hired by Toyota. And your loyalty is to Toyota, not to your boss. And so that gives people the ability to say no and to tell their boss, I don’t think that’s a good idea, or equally to automate themselves out of a job, because that’s okay, because they’re not their job, they’re an employee of Toyota, and now they can go and do something more valuable for Toyota. So even things like that around HR practice and the idea that Toyota will give you a job for however long you want, and you’re not tied to the particular role you’re doing right now, and we’ll retrain you and give you opportunities to get new skills. These are all things that are really important in creating an atmosphere of trust, and that in turn is what makes it work when you set people stretch goals, rather than an atmosphere of fear where you set someone a stretch goal and they all kill themselves trying to do something that’s impossible, rather than trying to change the game in such a way that it is possible. And again, you can only change the game if the game isn’t specified in tiny, minute details, and we have to get all of these things. You know, so it’s okay to say we’re going to release in a month if I can then decide what that release actually looks like and what goes in it. But I can’t say yes if, well, you have to do this 3 months’ worth of scope that’s specified in precise detail and do that in a month. Like, no, that’s not okay.

Matty: [00:32:31] So this is just a random question of mine. So do you have as hard a time as I do of typing the word continuous? I can never seem to spell it.

Jez: I get it wrong all the time. And I’m from Britain, I’m living in the US, I’m totally hosed because I just spell things differently all the time. This language well, so I don’t know where I am. I’ve still got the British accent, but everything else is totally screwed.

Matty: I feel better then.

Jez: OK. And I was typing continuous into Google just to see what happens. And it’s kind of cool. Google has this feature now where you can type a word, and it will give you the dictionary definition of the word. And if you’re listening along, you should try this. It’s pretty cool. So type continuous. Try and spell it right.

Matty: Oh yeah, there it is.

Jez: Right. But then there’s a little arrowy thing. You can open it up. And it tells you the origin of the word using this flowchart thing.

Matty: This is cool. I like it.

Jez: It’s got the original Latin words. And so somewhere along there, it’s like con plus tenere makes continere, which is hang together. And then that becomes continuous, which means uninterrupted. So my theory is we use continuous because it’s easier to spell than uninterrupted.

Trevor: [00:33:41] Etymology is such a cool study.

Jez: It’s amazing. And, you know, there’s someone’s job in Google who has the job of doing this, and that’s pretty cool.

Matty: It also shows, which this is just random too, I can see like it has this little graph at the bottom that’s like its use over time.

Jez: Right.

Matty: So it seems like it’s trickled down a little bit. Surprising.

Jez: Yeah. So it’s my personal mission in life to make Bring it back up again.

Trevor: It’s amazing the good things that can be done with data alongside the bad things.

Jez: Yeah. See, Google’s trying to make us feel warm and fuzzy right now. So I’ll give you another Google warm and fuzzy, which is to type— what’s that thing?

Trevor: Do a barrel roll?

Jez: When something calls itself in code.

Matty: Oh. Recursion?

Jez: Recursion. So type recursion into Google.

Matty: And it says, did you mean recursion? Yeah.

Jez: Nice.

Matty: What are some anti-patterns of continuous delivery that you see? Like you said, everybody— you made a comment earlier on about saying people say they’re doing it, but they’re not. So what are some symptoms of that or some of those anti-patterns that you see?

Jez: [00:34:47] Oh, there’s tons of them. I mean, we could just stay here all day ranting. I’m sure you’ve seen it as well, the people who screw it up really badly. And again, one doesn’t like to be mean. It’s hard to do, right? It’s totally hard to do. Every organization’s different. You can’t take a shortcut, you know, as I like to say all the time, you know, get the best practices and implement the best practices like that. That doesn’t work. It depends on where you’re starting from. It depends on your circumstance. But yeah, I mean, there’s some common patterns to people doing it badly. You know, there’s kind of like basic class, which is just totally failing to do any kind of— you know, developers failing to change the way they think about their job, and, you know, not understanding how to break stuff up into small incremental steps, not paying attention to the build being red, always deprioritizing improvement work, which I mean understood very broadly, like test automation or deployment automation stuff, always deprioritizing that in favor of features. That’s very common. That comes from the kind of management practice of focusing on utilization as a KPI. Which is a horrible KPI to have, through to things like, you know, people who get it that they have to automate stuff, but then they take these horrible, broken manual processes and automate them step for step, and you end up with these horrible, broken automated processes that are really complex to debug. There’s so many ways you can go wrong with this, and then people do, and it’s hard. And I think the thing to understand is, it’s inevitable. You’re going to screw it up. You’re going to discover a ton of anti-patterns. And that’s okay. The thing to do is to accept that and to accept that when you’re trying things out and experimenting, a bunch of stuff is going to go wrong. Hopefully a bunch of stuff doesn’t go wrong as well, and that’s cool, but a bunch of stuff is going to go wrong. 2 things. One, don’t feel bad about it. Don’t start blaming people for trying things out, because again, it goes back to that atmosphere of fear. But also, make it safe to fail. There’s a kind of new framework doing the rounds called Cynefin. Which talks about, you know, the different domains— complex domain, competence domain, and so forth— by this guy called Dave Snowden. And one of the terms that they introduce is the idea of safe-to-fail experiments, where if the experiment shows you the wrong thing, then it’s not going to bring down your whole system or, like, cause your whole project to go horribly off course. So finding ways in which you can get the feedback as quickly as possible— and this is, again, a very common theme across everything from Lean Startup to continuous delivery to the Toyota Production System— is Try and find ways to get the data from the experiment as quickly as possible by basically finding ways to cheat. If you’re going to build a product, don’t build the entire product before you find out whether it’s going to be valuable to the users. Find ways to build a minimum viable product so you can get that feedback earlier. And it’s the same with process experiments. You know, don’t automate an entire test framework. Don’t take an entire test framework and automate everything before you find out that the tool you used for automation actually probably wasn’t the best one for the job. You know, find ways. And you see this all the time in enterprises with top-down initiatives. Someone’s like, we must do continuous delivery. We must automate all the tests. And then they create a team and give it this huge budget, and they go away and automate all the tests in QTP. Oh, no, don’t do that. So, like, there’s this thing about, okay, well, let’s run an experiment. You know, let’s see what it’s like in QTP. And let’s see what it’s like if you do it in JUnit and just automate one test and put that in the deployment pipeline and have— you know, and let’s see if the test fails and actually gives us real data. You know, like 5 tests that actually, when they break, it means something’s wrong, and when they’re green, it means we have some level of confidence the product is releasable. It’s, like, way better than a completely comprehensive automated suite that everyone ignores because, like, it’s just red all the time. So, you know, finding ways to experiment in ways that are safe to fail, I think, is really important everywhere.

Matty: [00:38:40] A question that came in from someone who left a comment on the blog about this, and I read this and I said, wow, this is an optimistic question. So, and he says, so this is Aaron Lindstrom asked, he said, so once continuous delivery and DevOps are really just standard operating procedure for software development and delivery, what does Jez see as the next area of improvement for the software industry?

Jez: It’s funny, I got asked that question actually the year after the Continuous Delivery book came out, and I was like, oh shit, you mean I’ve got to think of something else already? God! And at that point, it was quite easy. I was like, well, The Lean Startup is going to be the next big thing. And oh my, was it the next big thing! So, you know, that was cool. What is it right now? I know, I mean, so much has got to change. The older I get, the more kind of freaked out I get about all the bad things that are happening in the world right now. And so I kind of think things have got to change pretty substantially. We’re kind of puttering along in our kind of, oh well, we’ve got to get better at automation, while, you know, the whole environment’s being destroyed around us. And, you know, in 20 years’ time, we’re going to have like really great automation, but there’s going to be like no crops and stuff. So, you know, I know what the next big thing is. In the world of software, I think that the pattern seems to be, you know, new technology comes along, everyone’s like, wow, let’s do this, and then we have to relearn a bunch of stuff that we learned 15, 20 years ago in a different domain. And this has happened with mobile, right? You know, people have been writing these horrible, shonky apps in a really nasty way, and we’re having to relearn things like test automation in the context of mobile and deployment automation in the context of mobile and architecture. And, you know, we moved to these rich apps that are run on the browser. Now you’ve got these huge monolithic apps built in JavaScript on the browser that are really hard to debug and maintain. So what happens is technology comes along, people hack away at it, and then we have to relearn all this stuff that we learned 10 years ago, 15 years ago in a different domain. So I think we’re going to be catching up with this for some time. My non-shocking and not very exciting prediction is that we still have a long way to go getting the basics right with the state of the art right now. And every time a new language or technology comes along, we have to relearn all this stuff again. So on the one hand, I’m very excited because I think genuinely technology does have the power to make huge positive changes in the world around us, but not if we keep screwing it up. That’s going to be bad. I genuinely don’t know how to fix that problem.

Trevor: [00:41:07] It is a problem.

Jez: I would like to say that the next big thing is going to be more women and people of color in our industry. That’s what I would actually like to see happen over the next 10 years, have our industry move away from being this pseudo-meritocratic, kind of very white male dominated And that’s something where I actually am seeing some movement, just in the form of just horrible defensiveness from a lot of people who don’t want anything to change. So, that’s already a sign that things are slowly changing. So, I think cultural change is something that I would like to see in the medium term that I think actually has a chance of happening.

Matty: So, 2 other questions before we move into our checkout. So, one thing, just to— it’s again getting more into the specifics, but, you know, like you talked about how immutable is a really popular word and all these things. I’ve been talking about CDs. Just touching on infrastructure as code, right? Because that’s another popular term, and it’s also one that I think gets bandied about and is not necessarily completely understood as to what it means. I may not have always been explaining it well, but can you do continuous delivery if you’re not treating your infrastructure as code?

Jez: [00:42:11] I think you can probably do it if your system is small enough, but it becomes much harder again once you’re faced with brownfield. You just have to put enormous effort into, you know, the opposite of infrastructure as code is manual configuration management, where you basically point and click everything. And, you know, there’s huge armies out there across the world of sysadmins who are still doing their job by pointing and clicking. And a lot of sysadmins still think that’s what it means to be a sysadmin, which is kind of even sadder. So I think it’s very hard to do continuous delivery in that way, because every time you do a release, you have to do the clicky-through-the-UI thing, and obviously there’s the possibility of error. And then it makes disaster recovery much harder as well if you have to restore systems in that way. I mean, Martin Fowler has this great experiment for your disaster recovery plan where he goes into a data center with a flamethrower and a large kind of set of light weaponry and basically blows it all up and then starts his clock and sees how long it takes you to restore service. And so, you know, in a world where you have some instructions in a big document to point and click on these different things in WebLogic to restore service, Like, that’s going to be highly variable, your restore time. So I think, you know, unless you are able to recreate— and this is my definition of it, or I guess this is an acceptance criteria for infrastructure as code— can I recreate my production system’s state purely from information stored in version control? Which means the configuration is stored in version control, the scripts to do deployment and provisioning are stored in version control, obviously the codes and the tests and all that stuff is stored in version control, database schemas stored in version control. I mean, the only thing that you can’t put in version control is your actual production data. That has to be somewhere else. But everything else apart from your production data should be in version control. It’s very hard to do that, but— and it’s also very hard to do disaster recovery as well, you know, which is one of the reasons why everyone is super freaked out about, you know, fault injection, this thing that Amazon does with disaster recovery, their game days, and Google does. So quick plug for my conference, Flowcon. We actually have Kripa Krishnan. She’s in charge of the disaster recovery testing exercises for Google, her team actually creates a whole scenario like aliens invading Silicon Valley, and they execute the scenario over the course of like 3 days, and they actually do things like disconnecting the Google campus from the rest of the internet. They actually do that stuff, and that freaks out big enterprises. You kind of think— and it’s because they would have no way of managing that. But then you kind of think, well, enterprises are risk-averse. Well, if they were really risk-averse, they would actually care about disaster recovery. The question is, what risks are you actually averse to? If you were really risk-averse, you would be averse to building, spending months or years with huge teams of people building products that nobody cared about and would pay money for. Come on, the best way to be risk-averse is to pretend there’s no problem. Right. Oh, yeah, totally.

Matty: [00:45:04] Well, it’s also like it’s the short-term risk-averse, right, which is the thing that’s at whole balance of the likelihood versus the not, but it’s the impact, right? It can be so huge.

Jez: Yeah, totally. So again, this comes back to cognitive bias. I’ve wandered horribly off tangent, haven’t I?

Matty: No, not at all. This is awesome. So I want to move into our checkouts. This went by so fast. Before we go into our checkouts, don’t forget about our awesome sponsor. What?

Trevor: Matt’s pulling his hair out right now.

Matty: Oh no, no, I’m just excited.

Jez: It does get really hot in here.

Matty: Don’t forget about our other awesome sponsor, Datadog. They are a monitoring service that lets you correlate performance changes back to commits, builds, and configuration runs. Datadog brings together system and custom application metrics, alerts, and events from over 70 infrastructure tools, such as Chef, Jenkins, and Git. Just like we talked about earlier, like Jez said, you need to be watching all of these systems when you’re making deployment changes. Datadog really helps you correlate all that data together. You can get a free 14-day trial at arresteddevops.com/datadog15. That’s the word Datadog and the number 15. So, with that, checkouts. Jez, checkout.

Jez: [00:46:12] Oh, yes. So, I have a couple of plugs. One is Flowcon. Flowcon is a conference about lean product development, continuous delivery, DevOps. It’s 3rd and 4th of September in San Francisco. Go to flowcon.org. We have some really amazing speakers: Marty Cagan, who almost never speaks, Mary Poppendieck, Dan Marsh from Netflix. We have Don Reinertsen, Kripa Krishnan, who I spoke about earlier, who does Google’s disaster recovery testing. Exercises. And in fact, also, we have over 50% female speakers, so that was a big deal for me. It’s going to be a great conference. Go online, use the code HUMB50, H-U-M-B-50, you get $50 off. And also, I’m working on a new book, Lean Enterprise, which talks about a number of things we talked about today. You can buy that on O’Reilly as an early release and get your hands on a draft of that right now. I’m working on the final draft.

Matty: Great, and we’ll put links to all that stuff in the show notes, so you can check that out at therestofdevops.com/15.

Jez: Thanks, Trevor.

Trevor: All right, well, I’m going to start with— and unfortunately this is going to be probably over before most people listen to this— but the FCC extended the deadline for comments on the net neutrality decision. So if you haven’t put your comments in already, try to do so. Apparently they’ve been having some serious server issues today, which is why they extended the deadline.

Matty: [00:47:27] They need some DevOps.

Trevor: And secondly, if you use Android and Chromecast, Google— I think if you’re above 4.4 in the version, it might be before that, but you can now cast your screen to your Chromecast and it is super low latency. I’m baffled. I’m able to play games on my phone on my TV with no issue, which is crazy cool to me.

Matty: Very cool. So speaking of Android, that actually kind of goes in a way to my checkout. So there’s a game that’s been available for Android since December called Ingress, I-N-G-R-E-S-S. It’s one of those kind of layered reality games that it depends upon your geographic location. It’s like there’s all this weird matter that’s come into the world and you go and you collect it and you are on a faction and all this stuff. And all my friends who have Android have been playing it since December and I couldn’t do it because I had iOS. And they just released the iOS version of it the other day. It’s super fun. It’s called Ingress, I-N-G-R-E-S-S, in the App Store. We’ll put a link in there. And also, Chef, we just released Chef DK, which is the Chef Developer Kit 0.2, which includes Windows support. So if you’re developing Chef cookbooks and you have a Windows workstation, you now can also use Chef DK. I want to remind everybody that we do have a newsletter. You can subscribe at arresteddevops.com/bananastand. It is the best way to know about upcoming podcast episodes and cool news with DevOps.

Trevor: [00:48:58] Jaz, you want to mention your conference one more time?

Jez: I totally have a conference. It’s called Flowcon. You should totally come.

Matty: Absolutely.

Trevor: Thanks again to our sponsors PagerDuty, CodeShip, and Datadog, and to our loyal listeners. If you enjoy Arrested DevOps, we appreciate it if you’d visit arresteddevops.com/itunes and leave us a review with the iTunes Store. We’ll read our favorite ones on upcoming episodes.

Matty: And speaking of upcoming episodes, this is a very dense month for Arrested DevOps. We did an episode last week, we did one today, and we’ll be recording before a live audience at DevOps Days Minneapolis this Friday. So the specifics will be posted at arresteddevops.com/16, including the livestream URL once I get it. So if you are going to be at DevOps Days Minneapolis and you would like to participate in that, Let me know via the Twitters.

Trevor: Thanks again to Jez for joining us.

Jez: Thanks very much for having me. It’s been a pleasure.

Matty: Absolutely. Awesome.

Trevor: Great. Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter.

Matty: [00:50:01] We are always happy to get your ideas or feedback at shows@arresteddevops.com if you like the email thing. As always, I am Matt, @MattStratton on Twitter, and I’m Trevor, @TrevorGHess on Twitter. We are Arrested DevOps, and remember, There’s always DevOps in the banana stand.