The New DevOps with Adam Jacob

Adam: [00:00:00] It didn’t matter how good you were at operations if the application didn’t run, and it didn’t matter what your application did if there was no infrastructure to run it on.

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. We have a really great show for you today, but As always, we have to go get paid, so let’s have a word from our sponsors. Let’s talk about one of the most exciting events in the DevOps community, DevOps World 2023. If you’re someone who’s passionate about learning, networking, and staying up to date on the latest trends, then attending DevOps World is an absolute must. So what can you expect from DevOps World?

Adam: The list is endless.

Matty: First off, get ready to hear from some of the most inspiring and innovative speakers in the industry. The sessions will cover everything from AI automation, cloud-native architecture, security and risk management, to continuous delivery. And the best part is that DevOps World Tour 2023 is coming to 5 cities across the globe: New York City area, Chicago, Silicon Valley, Singapore, and London. Find a city near you and register today at arresteddevops.com/devopsworld. So Uffizzi is a platform for platform teams. You can stand up your developer platform in minutes, not months. What I like about Uffizzi is that it gives platform teams control and dev teams autonomy. It’s Kubernetes native and extensible, so you can customize it with tooling that meets your team’s evolving requirements. And these clusters, they spin up fast, like super fast. Out of the box, Uffizzi combines a great dev experience, secure multi-tenancy, and cost efficiency. But try it out for yourself at uffizzi.com. Download their CLI and you can spin up your first sandbox cluster in under a minute on their free starter tier. That’s uffizzi.com, uffizzi.com. So my guest today is a longtime friend of the pod, as the kids say.

Adam: [00:02:14] Is that what the kids call it?

Matty: They do. They call it a pod. Do you know what RIZ means?

Adam: No. What’s, what’s it? No.

Matty: So rizz is apparently a Zoomer for charisma.

Adam: Oh shit.

Matty: So you need to have your rizz.

Adam: Yeah, I gotta bring my rizz today.

Matty: You do have to bring the rizz. Maybe my new goal is to talk about rizz on every episode of Arrested DevOps.

Adam: I think it should be.

Matty: Yeah, and joining us is one of the most rizztastic people in the DevOps community, Adam Jacob. So Adam, welcome back to Arrested DevOps.

Adam: Hi, thanks. Yeah, it’s good to be back. It’s been like many years. You have to be like an OG pod listener to, to remember.

Matty: Adam’s been around the space for a while though too, not just as a guest. I mean, I, I do believe that probably maybe the most important thing you’ve ever done in your career is being on Arrested DevOps, but 100%, you’ve done a couple other things.

Adam: Yeah, people come up to me in the street all the time, they’re like, oh man, you were on Arrested DevOps. I’m like, yeah, there’s, you know, always DevOps in the banana stand.

Matty: There is, there is, you know. But do you want to maybe just quickly, in case, you know, just kind of tell people a little bit about what, where you’re, where you’re from, kind of your space, you know, kind of how you got here, and then we’ll talk about how we can fix the world.

Adam: [00:03:19] Yeah, totally. Well, yeah, I’m Adam Jacob. I’m currently the CEO of System Initiative, and what we’re trying to do is sort of create a second wave of DevOps tooling. And in a previous life, same life, feels like the same life to me, I was mostly the CTO of Chef, and I wrote Chef originally. Chef was a configuration management tool, in case you’ve don’t remember, and had lots of overlap in sort of the early days of DevOps, the days before there was a DevOps, there was infrastructure automation, sort of DevOps and infrastructure automation were buddies from the jump. And so I spent a lot of those years both learning a lot about the role of CTO in the startup world, and I was on the board, and there were lots of those things, but then also just being out in the world trying to help companies figure out, like, How do they do DevOps? How do they change the way their teams work? How do the tooling work? How does all that stuff fit together? And yeah, most of that’s really been my whole career, honestly. I think there’s 2 kinds of people who started their career as systems administrators. There’s the kind who love being a systems administrator, and then there’s the kind who hated being a systems administrator and want to ensure that no one is ever a systems administrator ever again. And I definitely am on the loves being a systems administrator end. You know, like, I love being a systems administrator. I love it still. And like, I want to build things for people who like that stuff. And that’s really basically what I do.

Matty: [00:04:47] I think I fall into that same camp. You know, it’s, it’s, it’s definitely the camp of— and I think there’s, there’s most people that come from that background, I think, are the, you know, I didn’t choose system administration, system administration chose me. I used to talk about like, we didn’t— very few people like went to school and said, I want to be a sysadmin, right? And I think about folks in the space that I know that working at— I spent 20 years as a system, you know, sysadmin, sysop, you know, kind of backend, back office kind of person. And almost everybody that I worked with that was like hardcore ops, like, came into it from a different way. They were either— there are a lot of folks that came from the military, and there are a lot of folks who were like, I was a geology PhD.

Adam: Yeah, for sure. Or I was a theater major.

Matty: What’s it— theater major.

Adam: Lots of theater majors, a lot of music majors.

Matty: Yes.

Adam: Yeah.

Matty: Then we find out that that’s a way to go actually make a living at it, but now it’s a whole different thing. So when we kind of think about how we got to where we are, so we’re looking at— this is in the year 2023. DevOps is, you know, what is this, 14 years now? If we say 2009, I think, is if we look into the annals of history of Schaefer’s tweet and the first DevOps Days.

Adam: [00:05:59] Yeah, I always start counting from the— from John Allspaw and Paul Hammond’s talk at Velocity, which is 2009.

Matty: 2009 was really the inflection year. We had John and Paul’s talk, we had the Continuous Delivery book, we had the first DevOps Days, we had Gary Gruber’s book. Lots of things happened in that one year.

Adam: Yeah, it all popped roughly in that minute. Yeah, you know, it was boiling in the zeitgeist.

Matty: Yes, but so if we sort of say we work backwards from there, so we’ve been doing this for a while, for 14 years is a long time. You know, a lot of times I’ll talk to folks and they feel like everything is awful. And I actually had a little, you know, those little donut meetings you have at work where they pair you up with other people and just, we’re paired up according to geo, but most of the people in my geo were SREs at my company. So I actually got donuted up with a whole bunch of our SREs, which was actually super fun. So we just talked about, you know, ops and stuff. And we actually ended up talking through like, did DevOps work? Did it not work? What’s different? We can, you know, we’ve had lots of episodes about this, But Adam is someone who has been around for a minute and also someone who, when I’m forced to give a definition of DevOps, I often go to one that Adam has used. Like to sort of say, like, if we put our retrospective hat on a little bit, like, yeah, maybe what are a couple of things that went really well? Like what’s better, but like, where did we kind of miss? Oof.

Adam: [00:07:16] There’s so many things on both sides.

Matty: Well, good. We have a whole episode to talk about it.

Adam: Yeah. We got, we had a whole time to talk about it. So we got, we got, we got time.

Matty: That one question is everything. Yeah.

Adam: Cause it’s going to go for a while. Let’s start with things that went well. If you remember what it was like working, especially in large companies, but in any company really, in the year 2000, for example, you know, what you had was you had operations people, right? And our job was, you know, to do, to run servers and do IT. And so even the version of it where your primary objective was to run servers, was you running applications for people, was weird. You know, like, like there was plenty of places where all there was was IT really. And so like, and the internet, the idea that you ran applications on the internet was actually kind of new. For a long time, the, the early, early internet was just like, we were looking at web pages and they were like words and they were like under construction. Then we wound up in the application era where suddenly you could do stuff on the internet. And so that kind of brought out this whole new way of working and that sort of became operations separate from IT. It was very siloed. So the way that those organizations worked, Funnily enough, they look a lot like how I hear people describing platform engineering today. They’re like, well, what we would do is we would— operations people would figure out how to stitch all this shit together. And then they would try to build systems that would allow the developers to do whatever the developers needed to do without thinking about how the infrastructure worked. And never the twain shall meet. And so, if developers needed stuff, they had to request it from operations. It would go into operations budget. It would take 6 or 8 months to get gear. So you had these really long planning cycles. Capacity planning was a really big deal, kind of a lost art now, actually. People forget that they could just do capacity planning, but everybody had to do it because there was a really long lead time on whatever you wanted to do. The gear had to get racked, had to be put in, had to be operating systems and all that stuff. The best people in the world could automate it, but they didn’t do it very quickly. It took a long time. More than anything, the number one thing that the DevOps movement did was it recognized that that was a single continuum of work. That ultimately, it didn’t matter how good you were at operations if the application didn’t run. And it didn’t matter what your application did if there was no infrastructure to run it on. And if it was insecure, if it was, you know, whatever the list was, like it required our collaboration in order to do it. And the idea that we didn’t have to collaborate with each other and that that was an explicit goal, you know, and, and people forget reasonably so, because maybe they weren’t alive or they weren’t working, all of which makes perfect sense because whatever, I’m 45, so I can’t talk about these stories as if everyone experienced them anymore because they didn’t. But where we came out of was objectively worse, both from a day-to-day work experience, what the tooling was like, what you could do with the systems, how complex they could be. All that stuff was objectively worse than in a pre-DevOps world.

Matty: [00:10:24] Even maybe people don’t remember that, but even those of us that were there, our perspective changes a little bit. I had a— one of the last things when I was leaving Chef, when I was switching, I was seeing customers and so on. I had a local customer in Chicago, very large financial organization that I’d been working with for almost my entire time working at Chef. And it was sort of my like, hey everybody, this is my last getting together for you with the quarter. I’m going off to go do something else, blah, blah, blah. And one of the engineers said, You know, Matt, they’re like, I really hate that we didn’t get very much done. Like, so he’s like, he’s like, I feel like we let, let you down. Like, we didn’t do this. And I said, are you kidding?

Adam: Look how far you came.

Matty: I could see it because I went and saw them every few months. They were in the shit. So they weren’t where they wanted to be. And I think that’s the thing to look at as well is like, yeah, where we sit changes things a little bit too, because we’re in it. We— and everything’s not as good as we want it to be.

Adam: Absolutely. And objectively, the mountain that they wanted to climb, the place they really wanted to be, they did not get to. And when you said to them, look, you still did great. Look at how different you are. Look at how much changed and look at how much better your day-to-day is. That’s all super real and true. And also, you know, they wanted to be deploying 100 times a day. They wanted to have that happen with no friction. They wanted it to happen They wanted to collaborate easily with the engineers. They still weren’t really probably talking to engineers. They were probably still mostly just making pipelines and sort of hoping that it all worked out. I think that in terms of failures, that’s a big one where for a lot of folks, the promise of DevOps, which was that we would be able to work together, across all these different disciplines of experts to deliver software at this really high rate of speed, safely and securely, didn’t happen in the way that we said that it would. I think when you go look at it now, you know, if you grab like the DevOps Handbook, which is a great book, so don’t get me wrong, awesome book, but we know what happens if you just do everything that the DevOps Handbook tells you to do roughly the way they tell you to do it. And it’s mediocrity. It turns out that it’s deploying once every 6 months. It’s better than it would have been without it. It’s better, like there’s so many things there that are better and it’s all great. But also when you stitch it all together, even if you do it all, very rarely do you actually wind up with great outcomes. I think that’s where a lot of the agita I see in the industry around DevOps really comes from. You see it in the platform engineering rebranding. Brand, where they’re like, look, this DevOps thing has failed. You know it failed because we’re not getting the outcomes we wanted to see. And so the way forward is that we should be building platform engineering teams. My problem, my only— it’s not a problem. I don’t have a problem with it, but it does smell a lot like what life was like in 2001, where it’s like, okay, let ops be ops, let engineers be engineers. We’ll put some software in between them and hope that it works out.

Matty: [00:13:32] I think it’s sort of like accepting that, like, we just can’t get rid of the silos. So let’s just make sure the silos are better.

Adam: Yeah, let’s just, let’s just smooth over the silos.

Matty: Yeah.

Adam: You know, what we know about the people who actually succeeded at this transition were that they were the people who collaborated better than other people did. Like, it was actually collaboration that made the difference. The big problem is that the tooling itself, to me, never was designed for collaboration. When we looked at the problem holistically, you know, when you look at like, okay, what is it we’re trying to get these people to do? We were really busy being like, well, how do I automate all the servers? Like people forget where infrastructure automation really blossomed from, but like, you know, in the ’90s, it started out with, with like high-performance compute clusters and like college compute labs, places that had just a lot of compute they had to manage quickly. And then corporate compute, tended to be managed with people. So when I started being a systems administrator, a 5-to-1, 10-to-1 ratio wasn’t crazy. As soon as we had to run internet applications, suddenly a 10-to-1 ratio didn’t make any economic sense. And so we had to figure out how to automate it. So now we were looking for 100-to-1 ratios, 200-to-1 ratios, 500-to-1 ratios, and that required automation. But then EC2 happened, Facebook apps happened. Next thing you know, we needed 10,000 to 1 kind of ratios. And that automation loop, at each of those stages, we had to just make up an answer based on how the world looked kind of at the time and what we understood. And we had to be like, well, shit, how do I deal with the fact that if I build a Facebook app, for doing music sharing, that it could have a million users tomorrow. And what do I do? I can’t put them in a data center. You know, like it doesn’t even make sense. So suddenly cloud computing, you know, like not that that’s how that happened, but they were a little co-committent. They were really close in time. And so we just kept making all these automation choices, you know, continuous delivery as an idea predates DevOps. Because we were looking for, well, how do we deliver software quickly into these new environments? How do we actually ship? How do teams collaborate on these new internet-style platforms? What DevOps did was took all this state-of-the-art at the time of its origination and sort of ossified the shape of the world into that shape. It said, well, what did we do in 2008? And then that’s what we’ll do forever. And if you go look at John and Paul’s talk, it’s so crazy to watch it now because what were they doing at the time? They built a little portal that allowed engineers to deploy Flickr directly. They did it with feature flags and dark launching. They did it with infrastructure automation, right? Configuration management under the hood. They had capacity planning and compute and metrics. And observability, right? And they had all, literally every buzzword that we’re currently living in, roughly in the shape we’re living in it. How did a deploy start? Well, it started with code that they checked into Subversion, and then they would tag a particular piece of Subversion, and then the Subversion thing, when they hit the button, that’s what made the button appear in the web UI, and they could deploy it. So they were like, they literally built a platform roughly the way that we describe it in roughly the shape that we’re describing it now. And it hasn’t changed. And we haven’t thought any differently about any of it. I feel like we’ve maybe wandered in the woods a bit from your initial question, but—

Matty: [00:17:13] I don’t know. Is this your first time being on a podcast, Adam? This is how— this is the definition of a podcast is wandering in the woods.

Adam: But I think you really do wind up with this really interesting thing when you analyze DevOps as objectively as you can, which is that like, It unquestionably made everything better. The cultural movement that it started and the way that it thought about everyone as an important piece of a collaborative puzzle is 100% right. It’s 1,000% right. It is so correct that the idea that you wouldn’t accept it as correct bothers me in a fundamental way because the evidence to me feels so overwhelming. And then at the same time, our technical implementation of those principles has not lived up to what we hoped it would do. And to make it even weirder, I don’t think it’s like any particular thing’s fault. I don’t think you can look at it and be like, well, yeah, that’s because Chef sucks, you dummy. You know? Or like, oh, it’s because Terraform’s garbage. Or, oh, it’s because whatever, I hate Jenkins, or because Argo is better, or whatever. We’ve rebuilt every single piece of that stack 10 times over in the last 10 years, and it hasn’t made an appreciable difference in the outcomes in a way that I think is really fucking interesting.

Matty: [00:18:32] And I wonder about that a little bit because there’s some tropes and things that bounce around in DevOps, you know, common sense, if you will, or conventional wisdom of the last 14 years, if that’s what it is.

Adam: We’ll call it that.

Matty: Math is hard.

Adam: Somewhere between 10 and—

Matty: Somewhere between 10 and 20. There you go.

Adam: 20 is too long and 10 is not enough.

Matty: Yes. You know, one of the things, again, if we go back to, and we can, you know, people say, oh, you talk about culture too much. And my argument to that is always, it’s just as important as everything else. It’s just, that’s the thing we have to talk about because no one has to tell engineers to play with automation tools, but we have to be reminded of that. So I’m oversimplifying a thought, but where that comes in is people say things like containers won’t fix your broken culture, or, you know, you can’t solve the problems with tooling. Adam, you’ve said something that in the interest of good artists copy, great artists steal ends up in a lot of my decks, which is tools influence culture, culture influences tools. So you can’t expect the tool— you’re not going to just be able to throw some Kubernetes at it and solve everything, but also your tooling influences the culture. And I, I wonder how much, like, did we miss on being able to influence that? Because on one hand, we had sort of the ITIL approach, which is the very prescriptive thou shalt, and then all the ITIL tools basically implemented it that same way. And in DevOps, we were very resistant to that. There’s no manifesto, there’s no here’s how to do it, it’s all these vagaries. And then getting that way that you reinforce the, you know, sort of socio part of the socio-technical through the tooling, I wonder why we— why that hasn’t been worked, because I don’t think it’s a novel idea. I think a lot of DevOps tooling has intended to do I think it was, yeah, in the DNA of Chef. It’s in the DNA of PagerDuty in certain ways. It’s in the DNA of a lot of these things. So number one, I’d be curious to your thing on that. I, I think one of the things, and as someone who’s worked for a lot of these vendors, and you probably saw this as well, is this all works great, and then you start to sell to the enterprise and they say, no, I need you to make your tool work in my broken way. Yeah, you know, and it becomes hard to influence those changes if you’re like, hey, if you want to use this the right way you have to change and your customer who wants to give you $10 million in ARR says, but I don’t want to change, but I want to buy your tool.

Adam: [00:20:47] So yeah, and it becomes weirdly tempting. I think there’s a bunch of nested things to unpack there, all of which are related to some degree or another. The idea that it’s about culture and not about tools was wrong from the jump. And I have regrets that we ever said it. Because look, culture is what you do. That’s what culture is. So when we talk about what the culture is, the culture of a company, the culture of a scene, right? I love heavy metal. You got a Riot Fest shirt on. Like, there’s a culture in our scene. It’s overlapping. We share it, but it is what we do. It’s the thing that happens when you go to punk rock shows. It’s the thing that happens when you listen to those records and you share them with your friends. It’s a vocabulary. It’s a way of being. It’s a It’s a train of thought, but it’s expressed in our actions. It is what we do. There’s this version of culture that I think a lot of the DevOps world sort of thought that it was, that’s a little more like being a lapsed Catholic, where it’s like, well, here’s the list of things that I’m supposed to believe. If I just believe in them hard enough, then it doesn’t matter what I do, because my heart’s in the right place. You know, and it’s like, well, kinda, sort of, but it’s a little more works-oriented than divinely oriented. I think that analogy got away from me. What I, what I mean by that is that like the idea that you are going to change your culture toward a more collaborative approach, you know, without the tooling forcing a more collaborative approach was lies. Like it was never going to work because all that it meant was that sure, you could find a person across whatever the organizational gap is and be like, hey, will you talk to me about the overlap between the engineering that you’re doing for the application and the way that we deploy it in the end? And you can find a person who’s willing to have that conversation with you, maybe even go deep in it, maybe even collaborate around the margins. But organizationally, if you wanted the organization to drive its focus through collaboration, the only way to do that is for collaboration to be in the tooling. The tooling must push you together because individual people won’t do it alone. You have to coach them to it. Right. And I think there’s also some like bad mythology, like way deeper than DevOps about how we develop software. That sort of also reinforces this that we can get into. But yeah, ultimately, tooling is culture because it’s literally what we do all day.

Matty: [00:23:31] I always say that one of my most important DevOps books to read is the book Switch about changing how you work. And there’s, there’s a really key story in there about, you know, a manufacturing plant, and they had this machine where people kept cutting their hand on it. Had a blade, they kept cutting their hand. And they said, conventional way to approach this is, well, let’s train the workers, let’s put up a lot of safety signs. They redesigned the machine so for— in order to turn it on, it required you to use both hands, which meant your hands were literally out of the way of the blade. The only way to do it. Been tilting at this windmill in the DevOps tooling space for a while. I mean, this is not the first time I’ve said this story. I’ve said this story since forever. You want to make the right way the easy way. Yeah, you create those glide paths because anytime there’s a cognitive load to change how you think, first of all, when push comes to shove, you’re not going to do it. And it should just be, that’s where we go. And I run into this with other tooling. I’m fighting with Asana this way, right, where Asana wants to enable everybody to do whatever the hell they want to do inside of a project. And I’m like, no, we have a process. And the, the answer is not document it really well in your wiki and everybody has to learn, because half the time these tools these DevOps tools, you don’t use them every single day. The one time I do it, be like, oh, I know the right way to do it, because literally the only way I could do it is to do that. And how have we missed that?

Adam: [00:24:53] It was— it’s a couple of things. So one is, what is the foundational metaphor of our success? I’m gonna get really, really meta for a hot second, and, and then just promise you’ll drag me back out. Yeah, because I can connect it. It’s all connected. Yeah. Even the story you told at Switch was about a factory worker who keeps cutting their hand on a blade. We are loaded with factory metaphors because we come out of lean and agile. And those places came out of— not agile so much, but lean for sure— came out of studying factories. And it came out of studying how factories work. And it’s tempting to look at software development as a factory process where we put— product puts ideas into the hopper, they get broken down by engineering teams, and those engineering teams work those ideas. Then we track their progress, and then we try to refine the system. We come up with really complex systems, both for measuring the throughput of our engineering, but also trying to optimize its flow through that system so that we get a predictable rate of work out the other side. And when you listen to engineers talk about the experience of working in that structure, the tends— the things that tend to be frustrating: estimation, agility, being able to actually change your mind. They don’t necessarily know why they’re doing what they’re doing, right? They certainly couldn’t connect it above the level of their boss or the Scrum meeting or the thing. They can’t connect it to the higher strategy. And I think that foundational metaphor for what we do is actually really wrong. Like, I don’t think it’s a factory at all. I think it has actually much more in common with professional sports. It has much more in common with an orchestra. There’s all kinds of things it has more in common with, but the factory is actually kind of low on the list because what you need is a bunch of highly motivated individuals with different kinds of expertise unified toward how they’re going to solve a particular problem together right now, and unified in what that objective is. And, you know, so like if we’re a soccer team, like it’s get the ball in the net. And the problem we’re trying to unlock is the other team’s behavior who doesn’t want the ball to go in the net. And we’ve got, you know, strikers and defenders and midfielders, and we’re all playing slightly different roles with slightly different specialties. And we’re all working together in real time, making tons of tiny decisions, collaborating to get the ball in the net. And if you imagine trying to retrofit the DevOps process onto a game that’s best played like soccer, it’s laughably silly on its face. You’re like, okay, so here’s what we’re gonna do. We’re gonna put the whole thing in pipelines, and then we’re gonna run pipelines across the whole thing. We’re gonna put code, we’re gonna check code into source control. Then other people are gonna come and review the code somebody else wrote in source control. Sometimes we pair, mostly we don’t. Because that tends to lead to inefficiency because we tend to look at like the actual problem here is how many people can we get on the field all at once? And like, like, and oh, and by the way, we’re going to like make sure that we don’t ever coach anyone in real time. If you’re, you know, if you’re in my Scrum window, you’re not allowed to talk to me. Like wait till the next Scrum meeting to decide what we do. You know, like, like it becomes silly when you look at it. And I think a lot of what we did in DevOps took those factory metaphors and took the idea that what we’re doing is not that creative of an act. And we sucked all the creativity out of it and we sucked all of the collaboration out of it and we replaced it with process and tooling. And we said, okay, here’s your software delivery. We didn’t call them software delivery factories, for the love of God. Like we decided that the collaboration wasn’t the heart of it anymore. And that the collaboration was going to happen because, because we worked the process and the process told us where the moments of collaboration were, but they weren’t collaborative. They were review. Like, like we weren’t actually working together, right? We were, we were working near each other at best. And, and meanwhile, if you asked people, what’s the most important insight DevOps had? What’s the thing that most that most separates the great teams from the good ones or the okay ones, it’s the rate of collaboration. How much do they talk to each other? How often are they together? How do they work on those problems? That’s the delta. But we didn’t design the tooling for that at all. Not at all. No one did. It was an afterthought, right? We were like, oh yeah, of course we’re going to collaborate. We, yeah, of course you should talk to each other. Definitely do that. Also, what you’re going to do is put your code, write Terraform, check it into GitOps, then it’s going to go through the CD process, and then the blah, blah, blah, blah, blah, blah, blah, blah, blah. No collaboration involved. And that, I think, is actually the root of our trouble. I think that that’s why we’re getting the outcomes we’re getting.

Matty: [00:29:56] How much of that is outside of the domain of the people being able to do the change?

Adam: Almost all of it.

Matty: Well, because it’s, it’s sort of like one could make the argument about Agile failing, because I think it did in a lot of ways. But the reason that people have shitty implementations of Agile is because if you want to actually have your product delivered in that way, you’re— you have to change finance, you have to change sales, you have to change marketing. You— there’s, there’s more changes that have to happen because there’s inherent Taylorism in the way the rest of the organization works.

Adam: I mean, yes and no though, because like the idea, the argument there is that, well, I need a repeatable process because otherwise the sales team can’t predict the blah blahs.

Matty: Yeah, well, that’s not my argument.

Adam: Okay, great.

Matty: My argument was if you want to run experimentation, because Marty Cagan told this story about LivingSocial, and I’ve heard this story a long time ago, which was they would run experiments and then they would try to shut it off and the salespeople say, but we already sold it. What I’m saying is just sort of there’s some philosophical changes and the bigger your organization, the harder it is to—

Adam: [00:30:58] the harder those changes are, 100%. We’re completely aligned that that’s real and they’re not insurmountable.

Matty: Certainly not. I’m not giving up, just so we’re clear.

Adam: And, and if we wanted it to work a better way, the number one thing we’d have to do is actually change our own behavior. I can’t control what the shape of the sales folks are like. Well, I can. So I’m CEO now. And so I get to do a lot of things that I would have otherwise had to compromise. So, you know, the way System Initiative works internally is very different from— it’s, I mean, if you worked for me in the past, you can see echoes of what I believed, but System Initiative is like everything I believe at 12 because I don’t have to compromise really very much, right? And it really works. And I think thinking about the way that we do product and engineering as closer to team sports means that what we measure changes. So, you know, what the sales folks want to sell are outcomes for users. That’s a very Marty Cagan way to put it. But like, but ultimately they don’t want to sell a feature. They want to sell the outcome. They want to sell whatever it is that makes it better. And the resolution of that, of that process that, that you just described from LivingSocial is once again collaboration. So the reason that that happened was that the sales reps weren’t collaborating with product and engineering to understand what they could and couldn’t sell and what the outcomes were or weren’t that they were selling. And because those things were disjointed, the sales reps could look at whatever the experiment was and be like, already sold it. But like, that’s a failure of communications, a failure of collaboration, whatever process you wanted to design to, to, to, to keep that from happening, at its heart, what it needs to start with is, are we talking to the sales reps about what these opportunities are, what their status is, when we have it or when we don’t? And the answer was no, they weren’t. Or somebody did something bad. And you can’t really fix for bad actors, you know, like if there’s people who are just going to do dumb shit or mean things, like, well, okay, like maybe we fire them.

Matty: [00:33:07] Well, and that’s not endemic in your organization, you know what I mean?

Adam: Like, you hope not.

Matty: Yeah. Well, if it is, that’s a different problem. Right?

Adam: Yeah, if it is, it’s a different problem. But like, but, but when you think about what the scope of DevOps is, which is so big and broad, and you think about the outcomes that we’ve delivered, I, I just think that fundamentally, the way that we, we had this really deep insight about what makes for better software products, and what makes for us to be able to build them at a scale that we previously couldn’t really imagine. Doing together. And I think that insight was that you had to build these really incredible systems of collaboration to do that well. And then we mostly failed to build incredible systems of collaboration. And instead, we built systems that drove people into factories, you know, that drove us into— we thought about, well, I need people thinking about the product’s future. So they— your job is running the backlog. And an engineer’s job isn’t to talk about whether or not this is the right thing to do or not. Their job is to do what I told them to do because I sold the roadmap a year ago. You know, like all of that stuff, that stuff is, is the problem. And, and the way we think about building tooling needs to be targeted at saying, how do I rebuild this process? How do I rebuild this tactical thing someone does, this job someone does? In a way that makes that job collaborative with the people they need to collaborate with. And it often looks dramatically different than the way that we do it now. So in like, I’ll use chefs, so I’ll use System Initiatives as an example. Like, you know, the, I can open a board, like a single Miro board, and I can go all the way from the high-level strategy and the like pitch deck we use to raise money all the way down to an individual story being worked by an engineering team. And I can draw a line all the way from that story to the strategy. And every Monday, I read the whole strategy to everyone every Monday. And because we keep doing it that way, what happens is those engineers really understand why they’re doing the thing that they’re doing. They really understand who it’s for. They really understand like what good looks like, because we’re talking about it all the time. And we change our minds constantly, because you learn, you knew the least when you made the map. And so now you can sort of look at this map and know more. The side effect of that is that like, I can do that because I’m, I run product and engineering, but also because I’m in control of the tools that we use to run that process. Same process, but somebody tells you, Adam, you got to use Jira. It’s going to be tougher because even being— not because I couldn’t maybe express it in Jira, because Jira is kind of fabulous, but it doesn’t let me sort of zoom in and out in the way that I can in Miro. It doesn’t let me like create systems of context that like, that gets smaller or bigger, or that navigate context through frames, like the tools and the process and the culture I’m trying to create is all attached. It’s one holistic thing. And that, that is the way that we have to think about DevOps. And what we did with DevOps was we said the culture is, is one thing and we’ll describe it. And it runs at this top layer up here. Then the tooling is another thing and it runs at this lower layer down here. And you can have all kinds of choices about which tool you slot in. To this, to each piece of the process, because what you’re doing is building a little factory from Lego bricks. And, and that was a mistake. It turns out most of those solutions are wrong. Most of them lead you to bad outcomes. Most of them, and I would argue the shape of the factory is wrong. Like, it’s not just that the tools are wrong. The tools are actually kind of great and they’ve been getting better the whole time. The problem is the shape of the factory, the way we’re asking you to do it, it’s fucked up. Doesn’t actually work. Well, it does work and it works better than it did in, you know, reference the top of this podcast. It works better than it did in ’99 or 2000 or 2001 or 2007 even. But like, but it’s not actually giving us what we wanted.

Matty: [00:37:39] So I want to dig into how these tools can express it better and not continue to like sort of go and say, this is why it doesn’t work and whatnot. But, but I, but I had just one, one little dot connection I wanted to make there is when does this start to become a problem at a certain scale? So a lot of the stuff you described is amazing. How many, how big? I mean, maybe you can’t tell me. 15. Yeah, 15 people, right? Okay, let’s talk about JPMorgan Chase now, right? Exactly.

Adam: A whole nother ballgame.

Matty: We kind of have this problem in DevOps because we got away with it, so let’s do it better this time. But some of that, it just sort of happened if you’re a small— and I’m not saying that DevOps was perfect for startups, but a lot of this collision happened when we got to the— yeah, when we got to JP Morgan and Chase.

Adam: Yeah. Yeah. No, look, 100%, 100%.

Matty: But are there ways you make it smaller and it’s little pieces of it and stuff? Like, I mean, there’s, there’s ways you scale, right?

Adam: You scale horizontally, you know, but even in the world where you’re in the JP Morgans of the world, people underestimate both the, the ability for those big organizations to understand that they should change. They can understand it. The problem is that what they don’t tend to do is build their own tools. So they tend to buy tools because their primary function is being a bank.

Matty: [00:38:53] Yeah.

Adam: So with JP Morgan, we could convince them through lots of empirical data that it’s a better idea to do DevOps than not do DevOps. Great. Awesome possum. Now, how are they going to do DevOps? Well, all the people who are most successful at DevOps built all of their own tools. Google, Facebook. I mean, Facebook grabbed a few tools off the shelf, but it doesn’t look like anybody else. Do you know what I mean? Like, like their build system is different and the way that they think about code review is different and all that stuff is different, which is different than how it worked at Google, which is different from how it worked at Yahoo, which is different from how it works at Rakuten, which is different from, right? So, but all of those companies, what they had in common was that they wound up building it much more bespoke. Than people realize. Whereas the JP Morgans of the world, they’re a little less likely to build it bespoke and they’re looking for a little more off the shelf. And so it kind of leads you to that problem you alluded to earlier where, okay, so I sell— now I sell my DevOps solutions to JP Morgan. JP Morgan tells me, well, I’ll buy it, but if you could just adjust it here to fit my culture that I currently have, that’d be great for me because it’d be way easier to adopt and the check will get a lot bigger. Then you’re like, well, okay. Yeah, let’s. And the fact that it doesn’t in the end meet their needs, that’s a tomorrow problem for tomorrow people. I had this conversation with a different large organization that was all in on Cloud Foundry. And it was incredible spend that they figured out. It was nuts what they were going to do with Cloud Foundry. And they put a bunch of that money down upfront. And It just went nowhere. They just got completely stymied. And it wasn’t because Cloud Foundry is bad or whatever. It’s because all of the ways they had to manipulate it and all the stuff they had to integrate with and all that stuff, it just fell apart under its own weight. But it was, and it was kind of obvious if you knew what was going to happen when you started from the way it was framed, but you couldn’t really tell anybody about it. And if you did, they would just be like, well, you know, already spent the money, you know? And so you kind of get, you wind up stuck. And I think, And the way you can get past that is by saying, okay, if, if we know that folks like JP Morgan are willing to acquire tools to solve their problems, then if we design a tool that in order to use it effectively requires an organizational change, they will change to use that tool to get the effectiveness out of it. That’s the one thing we learned they could do. They will 100% Buy tools that promise them better outcomes. We know they’ll do that. That’s the industry. So build one that actually teaches them how to do the right thing and they’ll do it. As opposed to coming in from the top and saying, well, hey, JP Morgan, you got to change your whole culture in order to adopt this thing. Instead, just be like, look, here’s a better way of doing this work and watch how much easier it is. And try using it. And they’ll be like, whoa, that was better. At which point their culture will start to change and like let the tool change them, not the other way around.

Matty: [00:42:05] You’re not telling me anything that I haven’t wanted to be true for at least the last 8 years. So I’m curious to know how it’s going to be different this time. And what I want to know, and I’m not meaning that in a big challenge thing, I also want to know because clearly this is the DNA you’re building in System Initiative and I’d love to sort of know how that looks and understand it, I feel like— and we don’t need to, you know, we’re kind of going off into so much naysaying, but just to sort of where I’m looking at it and the cynic in me, which exists because I’m an old school—

Adam: yeah, you should be cynical.

Matty: And that’s what we do. We look for the holes. Is how many places where what this is— because I look at it and I say, this is just churn, right? You know, so it’s sort of nice to say, yes, yes, I agree with you. They will buy the tool.

Adam: Yeah.

Matty: But will the tool actually get used once it hits the people in the middle that have to change? I don’t know, and let’s find out.

Adam: If it’s good, they will. And the problem is that very rarely is it good in the end. So look at the things that are sticky inside those large enterprises. There’s plenty of sticky things. Salesforce. Why is it so sticky? We love to talk shit about Salesforce, but the reason it’s sticky is because it’s actually kind of good.

Matty: [00:43:25] Good. And it does what it’s supposed to do.

Adam: It like does what it’s supposed to do. It like works culturally. If you kind of follow the way it does it, it’s like kind of works. And like, I don’t love it, but it, but it does actually work. And I think when you think about why will it be different this time, the answer is that A, we know more this time, and B, I don’t think we know today. So look, you’ve known me a long time. I got plenty of ego. I got no shortage of self-confidence. Do you know what I mean? I’m willing to do whatever, and I’m willing to go out on a limb, and I’m willing to bet on myself or whatnot. And so I certainly think that I have an angle on how a tool like this could be built and how it could make a difference, but I’m not so full of hubris that I believe I actually know for certain that it’s the only answer. What I know for sure, what I’m 100% convinced of is the status quo is not a good enough answer. And I think we have a lot of evidence that tells us the status quo isn’t going to work out. If our aspirations are to remain unchanged. If we still want what we said we wanted 14 years ago, something has to give. And I don’t necessarily think that it’s the same approach that the platform engineering folks are taking the way that they’re taking it now, which doesn’t mean that something that looks like platform engineering won’t be the answer in the end, because you can squint and sort of see platforms everywhere. I think, and it may be something like System Initiative, But it may be, it may be something we haven’t thought of yet. But what, what I know is that for a while here as a group, we’ve stopped thinking about them because we’ve sort of fallen into this trap of being like, well, we tried that already. And the part that I think is really real there is that when you look at each individual tool in the toolchain and you look at the shape of how we want to do the work, we have tried it already. We do know that if you just write another infrastructure as code tool that works roughly like Terraform, Pulumi, and the CDK, it’s going to work roughly as good as Terraform, Pulumi, and the CDK. We know that. You write another— you write another configuration management tool that works roughly as good as Ansible, Puppet, Chef, and Salt, it’s going to work roughly as good as those guys. And it might be different in some meaningful way. It might, it might be better in some vector, but its ultimate value will be roughly the same as what those things were. You know, you write a new CD pipeline tool. Is Argo CD an order of magnitude better than well-configured Jenkins? I mean, maybe, probably not, you know, probably not. Not if you’re being honest, right? It’s probably less work if you are like already in an ecosystem, you know, if it’s more targeted at a particular thing. But like, is it, is it like orders of magnitude better? So like what we haven’t tried changing, I think, is the shape of the system itself. We haven’t said that the problem is the shape that we’re asking people to work in. It’s not an individual tool. It’s not one box. It’s all the boxes. It’s the fact that there’s a pipeline in the middle. Why is that the right abstraction? Why? Like it’s not what we do. It’s not how you think about it, but it is how factories think about it. It is how job control thinks about it. But why is that how we think about it? Why is it that when we think about deployment, we can only think about it in terms of one layer of the application instead of all the rest of the infrastructure holistically? What even is an application? Which I know is like saying, what is— it’s like asking about a regular vocabulary word, but it’s different in every organization. And how do we express those semantics? And to me, what I want to see is I want to see a flood of people just breaking the shape of that system relentlessly and saying, well, what if we didn’t do it that way? What if what we started with was the supposition that we had to collaborate with each other? It had to be safe to operate at high speed. And maybe that’s it. And the rest, you could throw out whatever you wanted to throw out in order to make those 2 things be true. And most of the ideas we come up with will fail because most ideas are bad ideas. That’s just true. But then some of them won’t. And some of those things, I think, will deliver dramatically better outcomes for DevOps people than the status quo does.

Matty: [00:48:11] So how are you reasoning about this with the system. That’s sort of what I was getting at when I— and I appreciate everything you said. I want to be clear that I wasn’t sort of challenging and saying like, ah, this is never going to be different. I was very interested in the how, right? Not the— not the, well, why would it be different? Yeah, I want to know, like, cool, this is where I want to crack open the Adam’s skull in here.

Adam: Like, yeah, I— so I think it actually comes down to— I think a lot of what happens here starts with the supposition that— so I tend to start from the place that I know best. So for me, DevOps tooling starts with automation because that’s, that’s my home. That’s like the place where my heart lives. There’s lots of other pieces of DevOps, but that’s, that one is mine. So excuse that I’m going to sort of hang out there for a minute. But to me, we’ve basically been pushing in one unbroken line of thought on automation since Mark Burgess wrote the computer immunization paper in in the ’90s. And that’s a long time to be basically mining one train of thought, which just gives so much credence to what an incredible train of thought that was. So whoa, right? But then it becomes, I think, the source of a lot of what our ultimate problem is. So if our ultimate problem is a collaboration problem, then Code, it turns out, isn’t a super great way to collaborate. It can be a good way to express yourself, but when we think about the activities we want to do, you know, the things we want to do is we want to build infrastructure, we want to configure it properly, we want to see if it works, we want to relate to other people, we want to see if the application runs on it. The way that we think about that automation and the amount of effort that’s required for us to express a configuration is really, really high and requires a lot of expertise. And so for me, it starts with thinking, well, how can I lower the amount of effort required for an expert to express their expertise, which is a really important frame. So I’m not starting from the frame that says, how do I make it easier for someone who doesn’t know what they want to do something complicated easily? That’s not what I’m saying. I’m saying, if you are the hardest of the hardcore, if you are the bestest of the best, how do I make your experience better? Because if I can make it better for the bestest of the best, then everybody else, it’ll be revelatory. Do you see what I mean? Like, you’ll— it’s no problem for the people who know less, right? So, and to me, that answer started with saying, Well, I really can’t if the only way I can express it is code. Because code is code. It’s words in a file. You know, it’s, it’s, it’s, it’s going to check into source control. I can’t give it any new semantics. I can’t do anything with it. I can’t programmatically alter it really. I can generate it, but then I can’t mess with it later. Like it’s got a lot of pretty tough constraints. So the first problem is we express everything as code when perhaps we shouldn’t. And then if we’re like, okay, if it wasn’t code, so let’s just pretend a minute it doesn’t have to be code. Now what else could we do? And it’s like, well, the other problem is the feedback loops. So if you look at how long it takes to know if you’re right or wrong, if you’re doing automation or a complicated problem, it might take 20 minutes just to see AWS return an API call. It might maybe longer, right? How long does it take to actually tell you whether it’s going to be good or bad? What’s the information you get when you run like a Terraform plan, right? It’s like pretty good, but it’s actually kind of hard to read. It’s a little tough to understand what’s about to happen. It’s certainly hard to bring somebody else in who’s not an expert, both in the code and what it’s supposed to do, to then infer whether or not what they’re looking at as the output would be good or bad. So if we want to make those feedback loops go from 20 minutes or longer and not very informative to highly informative and immediate, what would we do? And so you go look at adjacencies and you’re like, well, the people who do that now in complex systems build simulations of real-world things. And so that leads you to digital twins. So you can say, hey, Formula 1 race cars can only be tweaked for this small amount of time on a race weekend. And so they build these incredibly complex simulators that simulate every variable of the car, and they actually have someone drive a car, fake car, in the simulator so that they can test all these different configurations of the car before they get to the track, so that only the most promising directions are the ones that they actually try when they get to the track. So what if we designed it to work more like a simulator? Because a simulator could tell us immediately whether or not it was good or bad. Okay, great. Well, that’d be awesome. How would we build a simulator of infrastructure stuff? And you’re like, okay, well, I guess we’d have to start by— simulators work because they’re really detailed. So in those Formula 1 cars, you get really detailed fluid dynamics. Every component has really detailed information available to it. And then in some cases of real digital twins, they get twinned to a physical piece of equipment with actual sensors and all kinds of stuff that feed back into the system. So that kind of leads you toward thinking, okay, well, what if instead of code, what we built was a simulation full of models of real-world things that were really high fidelity? And how would that work? You know? And then you just keep pulling on the thread of saying, okay, well, that solves the feedback loop problem. What about all the glue code we have to build all the time? So in order to do all this stuff, I got to glue together all these different subsystems. There’s all these different pieces that are in play. How do I remove the glue? And you’re like, well, I have to build the flow into the application. So, if the simulator understands what those things are, I have to understand, like, how do we decide to make a change? How do I propose something? How do I do this? How do I do that? System Initiative is our attempt at basically pulling on that thread until we get to the answer and just saying, okay, like, if that’s the direction, which I think it is, That’s a direction. How hard can we pull? And until what pops out the other side is an order of magnitude better way of thinking about running the DevOps flow of saying, well, here is the infrastructure that attaches to my application, and here is the way that they all relate to each other, and I can visualize it, and it’s in a simulator, and I can write code that describes how the simulator works, and I can apply that into the real world, and I can think differently about application deployment and work workflows and all of that stuff. So yeah, that’s kind of System Initiative’s angle. But like I said earlier, I don’t think it’s the only angle. And I think in order for us to win again as an industry and to make progress, we’re going to need more angles. Like we’re going to need more people deciding what is, what is the thread they’re going to pull on, you know, and let’s see where it goes.

Matty: [00:55:17] Well, I think that’s a pretty good way to tie all the pieces together. I have lots more questions and lots more things to talk about, but We are sadly running out of time, so I’m just going to have to chase you down on Twitter, X, whatever. By the time this episode comes out, who even knows?

Adam: What do you even call it? Yeah, yeah, this is— I’m— yeah, I’m easy to find, I think. And yeah, like I said, if there’s one thing I hope people take away from this conversation, like, look, I hope people love System Initiative because I’ve spent so much of my life now working on it and I, and I really believe in it. But also, whether they do or they don’t, I hope the idea that the problem with the— I’m not wrong, that the problem with how DevOps works is the shape of the system holistically. We’ve tried every other variable, we really have. And if we want a different outcome, we need to change the shape. And I hope that people become more willing to do that work and to throw some of the babies out with the bathwater. Maybe that’s a weird thing to ask people to do.

Matty: [00:56:22] But I think it’s really key though, and just to, to kind of put a, put a bow on that, I guess, is when we look back at this, we’re not saying that we did bad, right? We did the best that was available with the information at the time. So you made the point, you said, why, why is it going to work different this time? Because we know more.

Adam: Yeah.

Matty: So let’s accept that. Let’s say we know more so we can challenge the way that we’ve done things because we know more now. Not we should have done I would argue and say, should we have done it differently 15-whatever years ago? Maybe not. Maybe this was what got us to be able to learn these things, to be able to do the new DevOps. So if you go to arresteddevops.com/thenewdevops, we’ll have our show notes for this episode. We’ll have some links, probably maybe some cool System Initiative links to go check it out some more, link to that famous 10 deploys a day talk from Velocity back in 2009.

Adam: You should definitely watch it if you haven’t.

Matty: It’s seminal, right? If you visit arrestodevops.com/itunes, you can leave us a review in the iTunes store, which helps other people find the podcast. And as I like to say every episode for the last 3 to 4 years, nobody calls it iTunes anymore, but I refuse to change the 301 because I’m lazy. So, it exists displaying my lack of being able to change with the times. You can find us on Spotify, iHeartRadio, Audible, all the places fine and less fine podcasts are made available. Adam, thank you for joining me today. This has been a great conversation. I could do this again for hours, but longer than an hour apparently are not great ideas. So we’ll just have to, we’ll just have to do something else.

Adam: [00:57:56] Well, thank you for having me. And it’s always a delight to talk to you and, and, and to, and I love talking about DevOps, so I’m, I’m stoked to be here. Thanks. Thank you.

Matty: This is Arrested DevOps.

Adam: And remember, there is always DevOps in the banana stand.