Complexity with Michael Stahnke

Michael: [00:00:00] Okay, so the original problem was I couldn’t get my developer environments unified, and therefore I ended up with Kubernetes. What the fuck?

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton. We have a fantastic show today. I say this because I can predict the future, and I know how the next hour is going to go. But before we get into that fantastic future, let’s hear the current present from our sponsors. Thanks to our sponsor Gliffy, the leading diagramming solution for teams using Atlassian products like Jira and Confluence. Drag and drop shapes to quickly build a diagram capturing anything from code structure to a simple concept. You can start your free evaluation by visiting gliffy.com/arrestedevops and signing up via the Atlassian Marketplace. That’s gliffy.com/arresteddevops. Get started today. So Uffizzi is a platform for platform teams. You can stand up your developer platform in minutes, not months. What I like about Uffizzi is that it gives platform teams control and dev teams autonomy. It’s Kubernetes native and extensible. So you can customize it with tooling that meets your team’s evolving requirements. And these clusters, they spin up fast, like super fast. Out of the box, Uffizzi combines a great dev experience, secure multi-tenancy, and cost efficiency. But try it out for yourself at uffizzi.com. Download their CLI and you can spin up your first sandbox cluster in under a minute on their free starter tier. That’s uffizzi.com, uffizzi.com. So things should be simple, right? But really they’re complex, or are they? Are they complex? Are they simple? What the hell do these words mean? I have a guest joining me today and we’re going to delve into what we think about this, this word that comes up all the time, complexity. So Michael, you want to introduce yourself to our Arrested DevOps listeners and Sure. You know, kind of see what we’re going to do.

Michael: [00:02:20] Yeah. So I’m Mike Stahnke. I’ve, you know, been around this industry for a while, been doing the DevOps circuit on and off through, you know, the last, I don’t know, 13, 14 years, whatever, whatever, however long they’ve existed, I’ve been in and out of it. And, you know, the thing that I really wanted to kind of think about is, are we better off today than where we were kind of when we started from a, you know, operational availability, from the way we work, the how we debug, how we troubleshoot. It seems like what we keep doing is inventing new problems and then inventing new solutions instead of actually maybe looking at, do we even have to have this problem? Is it a real problem? Should we have solved it? Should we have just taken a left instead of a right at that time a long time ago? And a lot of my kind of theory around this, I guess, started when I was VP of engineering at CircleCI, and I was watching our struggles with availability and reliability targets and all these things. And it wasn’t because we had bad engineers, and it wasn’t because we had terrible practices or process. It was because we were doing really complicated shit. And sometimes it’s just hard to keep a lot of things online. I mean, that’s just the way it works. And how do you understand all this stuff? And so that’s kind of where I came about it. It was really, are we better off than what we were saying in 2004, 2005 with keeping things online and things like that? And I don’t think that the answer is a yes or no hard, but I think it’s a good thing to revisit and think about, like, where do we come from? How do we get there? And have we made the right choices to end up where we’re at today?

Matty: [00:03:43] So let’s think about that a little bit. Let’s put on the Wayback Machine 20 years. Oh my God. To like circa 2003, 2004, maybe like, and this is not, by the way, this is not like old tech dudes like going, in my day, I had this TRS-80 that did whatever, you know, and I hung my Sparks workstation from my belt. It was the style at the time, you know, none of that stuff. Michael and I are not spring chickens. We’ve been doing this shit for a while. We’re not any better than you. In fact, maybe we’re worse because we haven’t figured out how to get out of tech and go live on an emu farm or something like that. But that said, all right. So, Michael, let’s put on our Wayback Machine. What were you doing professionally? What did your world look like circa 2003 from a— as relevant to this conversation? I mean, a lot of things in 2003 that have nothing to do with this, but I mean, I was working on systems automation.

Michael: That was my primary goal. And I had joined a team recently, college graduate a couple of years earlier, and just saw the ineffectiveness of system administration. I had a computer science degree. I was like, I feel like we could write software to solve these problems that we’re solving largely through spreadsheets and manual labor. And so I started writing software to solve it and do administration and was using SSH and for loops and things like that. Obviously had some issues with it, but it solved a lot of problems and allowed us to manage thousands of servers with fewer staff than what we had started with and things like that as people moved around. And I got to learn a ton. It was super cool. But also, you know, we definitely had certain systems that were kind of cared for by specialized administrators, you know, like whether it was Oracle RAC or other, you know, like critical business systems where it was like, well, those people only touch Oracle RAC or those people only touch, you know, these very critical HP-UX clusters or Sun clusters or whatever. And You know, and so is that better off than today? Well, we’re not running HPUX nearly as much, so let’s call that a win. But, you know, but, but back then those things were, you know, millions of dollars were flowing through them every day. And so, you know, it wasn’t like it was a bad thing, but it’s just, it was just different. And so I was trying to do automation. I was trying to get us to, to work together and build toward a goal. And so a lot of the, those principles I think were, were there then as the same they are now. It’s just the tooling, the tech, and the systems and the scale of everything has changed significantly.

Matty: [00:06:03] I think the scale has a lot to do with it. When I, when I think about circa 2003, if I had been working for a financial company that was acquired by GE Capital and I got to have the fun thing of shutting down everything that I built, you know, which is always, always fun. But, and then thinking about from an automation perspective, but like, what was I doing circa 2003? It was like a lot of Active Directory and Microsoft Exchange. Like, the automation that I was doing was How do we move from our Exchange 5.5 to GE’s Exchange 2000 as part of the acquisition? And we still built a lot of other systems, you know, but they were all, again, it was a lot of, things were much more bespoke because you built long-lived things. And so I think leading up into that, it was things like, we were a pretty decent-sized environment and like we had a few dozen servers, right, of different, you know, you had your lab and you had your production and it was a computer room that I walked into. And but then also we sat there and had, you know, when you were on call the day that it was sort of your turn, you sat in that computer room all day long and they had a little like sheet you ran through that said, okay, it’s 5 past the hour, so go pull up all the event logs on the servers and look for things. And here’s that part. And then I think like after, again, around that time too, I was working at Allstate and one of my main jobs is kind of funny to think about now, but like was rebuilding dev servers constantly. So it was like, okay, there’s a new project. It had to be reimaged and I said, you know, the time I was at Allstate, I would sit there and you do a thing, you would create an answer file that went on a floppy disk and you go into the data center and you drop a Smart Start CD into the ProLiant, you’d boot the machine, you’d sit there, you’d wait. And then every now and again, hit enter, change the floppy, which meant I sat in a data center with my warm coat and a book, you know, and it was, it was kind of okay. And it was actually kind of a little bit radical that we even did that, right? That when it was like, okay, you have a new project, create a new server versus like the, okay, we just keep using shydev1 over and over again for all of our development. And while it sounds kind of obnoxious, it was reasonable to do because you were still, even at Allstate, we’re in our biggest data center, it was still maybe a couple hundred servers, right, that we were managing and dealing with. And then you kind of, again, still always had this like specific expertise, right? You’d sit there as we built those systems, but But because systems were long-lived and they could be, I think about actually very shortly after that was the big virus that made there have to be patching tools, right? And then I was at Aon Insurance, another insurance company around here. And then we were in this scenario where we had to go patch hundreds of servers and it was just to unleash an army of people into our data center who would go from the KVM and go this one, now this one, now this one, now this one. So on one hand, this is like incredibly inefficient, but you didn’t have that much to do. And you could hold— I’ve talked about, I’ve thought about this before too, about when we think about the complexity of distributed systems and maybe it’s complexity or just size. There was a time when you could hold your whole stack in your head. You could reason about the whole stack. You could say, okay, yes, maybe I have, okay, remember all of my frontend database servers, but they’re all the same. I have this part and this part and the application runs here and maybe I’ve got a couple SOAP endpoints that I talk to, but for the most part, it was a vertical scaling. And when do we get to the part that we feel like we need to still be able to do that and have to reason about the fact that we just can’t? And maybe that’s okay and maybe it’s not. Like, did we— I guess this is your question, right? Like, so we built all this great distribution and microservices and all these things and like, Are we any better off?

Michael: [00:09:48] I mean, the answer is unequivocally yes in many scenarios, but in many scenarios the answer is no. And I think that’s the thing that I keep trying to, to think about is this isn’t a one-size-fits-all game. And this, you know, I don’t intend for this hour to be a big inter-peds conversation because those are really fun. A lot of these tools were designed to solve certain scaling problems. The first thing you need to understand is like, do you actually have these scaling problems or do you just kind of think they’re cool. Keeping one thing online is easier than keeping 25 things online. So like, let’s just start with that.

Matty: I think sometimes we build for scale because we think it’s cool and it’s fun, but also for a problem we don’t have yet. Right. Right. We used to say like a Rails app is like up in 3 days, down in 3 months, you know? So it was great to do this. And then how do you, how do you scale it? And then we made a lot of snarky comments about, well, make sure you’re building for this in the first place. But Why bother until you have that problem? And then honestly, with the exception of something like a Slashdot effect, which again, I’m dating myself, or something like that, you’re not going to go from nothing to web scale overnight. And the Slashdot effect was very real, but that was a problem you would still not solve this by rewriting your app. We used to think about when I was at Classified Ventures and Cars.com was one of our properties, and every time they would do a Super Bowl commercial would do 80% of the site traffic in one day. But you knew that was going to happen. And did it make sense to like re-architect the entire application for that? Or to just say, you know what, we’re going to throw a lot of pizza and a lot of like, it’s going to be a shitty night and we’re going to rent a bunch of servers and put them in the data center for 48 hours and it’s okay. You know, but I’m sure the argument could have been made to be like, oh, well, we better re-architect the system to be able to scale it up for when it happens. But like, Dude, you know when it’s going to happen. It’s the same time of year every year for 48 hours. You know, so yeah.

Michael: [00:11:41] And I think a lot of, you know, a lot of this is, have you found a user base? Have you found product market fit? Have you, have you done the things that matter from a business perspective or from an outcome perspective before you get to this like technology investment that is just, it’s complex, you know, it’s like, I need a service mesh. I need a service discovery layer. I need, you know, observability tools. I need, you know, it’s like, Man, don’t underestimate the power of rsync and cron for things. It’s just one of those things where it’s not to say those tools are bad. I’m glad they’re available, but I don’t want to jump to them until it’s like I’ve got the needs for them. And I just think a lot of this is, well, you want to jump into the cloud right away because it’s infinitely scalable. It’s like, cool, do you have users? You can run a colo for quite a while. You can run a Linode or a DigitalOcean node or whatever. If you don’t want to touch the hardware, More power to you. I don’t either. But you don’t have to necessarily take this like auto-scaling, a Kubernetes cluster or whatever. It’s just how can you solve the problem? And I just think industry-wise, it’s like we’ve created a hive mind of like, there’s one way to do this at this point. And it feels like it’s not easier, it’s not better. And it’s not like ultimately, are you delivering the value you’re supposed to deliver is the real question. And if you’re spending more of your time figuring out if this minor version upgrade on Kubernetes is going to change your ingress controller API, that is a thing that is no value add that no customer is paying for.

Matty: [00:13:04] You said, are you delivering value? Sometimes you have that disconnect and maybe you don’t know how to answer that question. And, and again, you is doing a lot of work here. Like, what do we mean by you? Do we mean you the organization, you the company, you the CTO, or you the SRE, or you the ops engineer who is 7 layers down from anywhere? Again, depending on the size of the org, you’re talking about a 12-person startup, the world is different than also you’re inside JPMorgan Chase, right?

Michael: Absolutely.

Matty: You know, so, and you work in these little microcosms of, of value. And I, I think about this, it was, it was funny. I had, uh, someone who was a sysadmin who worked on my team and, you know, reported to me, again, close to back to the almost 20 years ago, you know, when we were at apartments.com. And very randomly, he, he called me yesterday and was like, I want to catch up on how do we— on this DevOps stuff. And We had this conversation and it was funny, like you would get a kick out of it, Michael, because I was like, I felt like I was repeating all this stuff that we’ve been saying forever and ever. And not like it was new to him. He’s like, you’re telling me all these things I already know, but I had to hear them. But a lot of it had to do with this, like, you know, I’ve said before, you know, my trope I like to do when talking is say, hey, do you know how your company makes money? If you don’t, go find out, we’ll wait. And he was like, that sounds so obvious, but like people, you know, we, we think about it within our world. And I, I remember a story that Sasha Bates told on The Ship Show, and I think Sasha’s told it multiple times. She was working for a large retailer and it was holiday time and they wanted to make a change, you know, whatever it was, the request came to make this change and she’s like, no, no, no, we have to keep the site up. That’s what we do. And her management said, actually, our job is to sell things to our customers. Like, it’s not within— and we, especially from an ops perspective, we get so in our Like, my job here is system stability. That’s maybe your job, but that’s not why your company exists unless your company is Linode or Rackspace or something like that.

Michael: [00:15:05] Right.

Matty: But you’re, you’re still providing that. And, and it’s while this is like the same tropes we’ve been saying for however many years of DevOps, and this is all wall of confusion, misaligned incentives, but the reason we keep saying it is because it’s damn hard. And a lot of times people don’t know. How your organization really provides value, or honestly, you said provides value. I’m just going to sit here and say how you make money, right? Like, what’s your key business metric and where does that go? And you’ll think it’s like, well, we sell this thing. That’s what you sell, but that’s not how you make money. Like you said, we introduce complexity because we’re also maybe visiting this in the world of what we know. And the, the, our, our map of the actual territory of our organization. And I think this becomes— it’s weird because I think it can happen disproportionately at a small organization where you’re trying to do all the cool shit, right? And that’s like where you’re saying, okay, do you even have a product? Do you even have users? But then you can go on to the other side where you’re part of a really big organization and your little monkey sphere is this little tiny part of, you know, Capital One or whatever. And I’m not picking on anybody for the record. I’m just trying to think of big enterprises I know that have complex multiple BUs, multiple organizational stuff. So you have to figure out some way to reason about it because you can’t reason about it at the MasterCard level. You have to reason about it at the operations team for fraud protection level kind of thing.

Michael: [00:16:43] Yeah, I mean, there’s obviously constraints in there. And so hopefully you can figure out why the constraints exist for your subsection of your organization. And it might be, hey, the constraints are you have these operating systems or you have these programming languages or you have these technologies you’re allowed to use or not or whatever. Everything has to ride on this platform, things like that. Again, those constraints are probably there for good reasons. And most of the time, they’re fairly well thought out. But other times you see things where, like how many developer workflows today involve containers? Well, a lot of them do. And are containers awesome? Yeah, they change the game completely. Are they awesome in every single scenario? No. Like every time I develop, I have to bind mount in my home directory. I have to set up a whole bunch of environment variables. I have to throw either a giant Docker Compose YAML file or have like shell script or a Makefile that automates it all. And it’s like, why am I doing all of that when I could just literally, you know, run the source code or type make or whatever and I could run it locally? Well, because the environment that I have locally, wasn’t the same thing that somebody else had locally. And it’s like, so rather than try and solve that problem, what we did was we say, we’re going to package up your laptop, we’re going to pass it around until it gets to production. And that’s like, that’s what a container is. And it’s like, that’s not necessarily the worst way to solve that problem, but it’s certainly not my favorite way to solve that problem. And, you know, what’s the— what the hell’s in that container? I don’t know. And so now there’s whole businesses around scanning those containers and scanning for vulnerabilities. And like, it’s like, so instead of solving why we made that decision, we just layer on new veneer to try and get us through it. And it’s like, Eventually, that’s going to make sense and it’s fine. And it does make sense in certain places, but it’s also like, well, what if we just all developed using the exact same environment? And then we would know that that’s the same environment that we would use in production. And when I say environment, it might be, hey, I need something that’s reproducible, that’s got non-repudiation all the way through it. It says, this is the exact stack, these are the checksums or whatever. And it’s just the same on everybody’s laptop. Okay, well, if that’s the case, then I know what it’s going to look like in production. And I don’t necessarily need a container to do that. And if I don’t need a container, I don’t need a container scheduler. If I don’t need container scheduler, maybe I don’t need a service discovery. If I don’t need a service discovery, maybe I don’t need a service mesh. It’s like you could just go back, back, back, back, back. And it’s like, wow, I could just type service start. That’s awesome. I miss that. I miss throwing strace at processes and seeing what’s going on instead of having to launch a sidecar pod and have my debugging tools go in there. And all this stuff, so many people have gotten so used to it, they don’t even think about kind of the affordances that were simple, where it’s just like, I know in DevOps where we’ve been telling people not to SSH into boxes for at least 10 years at this point, you shouldn’t be doing production changes manually and all that. Sometimes you just SSH, type strace, and you can see, oh, it’s hanging looking for this file because it’s not there. Why? Oh, there’s a permission problem. Cool. And it’s just like, that takes 10 seconds to figure out, or I have to launch an entire debug pod. Those are very different problems. And sometimes I just think about the problem of, okay, so the original problem was I couldn’t get my developer environments unified, and therefore I ended up with Kubernetes. What the fuck?

Matty: [00:19:34] Maybe it’s a little Conway’s Law comes into play, right? Because it’s like, how do we, with these organizations, and I was thinking about, you know, was it the AWS Summit in New York where I work, Ivan? We are, you know, managed data platform service, you know? And so when people come to the booth, they’re like, okay, what do you do? Because that’s what we do at a, you know, in the expo hall. We’re like, cool, awesome. You’re giving away something. And I feel obligated to at least ask you what you do. Cool. And so we say, hey, are you using any of these products? Because the way we talk about it, say that we only make sense if like you are using one of the 11 open source data products we have. Like, so cool to use Kafka, MySQL, Redis, blah, blah, blah. And people will be like, well, I’m the Kubernetes person. Like, I don’t know, we have a data team, they do shit like that. I don’t have no idea. And I’m like, that’s kind of like, again, I’m not expecting that you’re like a dope-ass DBA because you’re a CKA running this thing, but like, I would think you would want to know a little bit about some of the stuff that’s riding on your shit. And, but it’s kind of like, you can be like, not my circus, not my monkeys, data is someone else’s problem. And the reality is it so isn’t, you know, especially if you say you’re a platform engineer, because data is part of a platform. It’s not a failing. This is what I want to be really clear. Like, I’m not like, oh, these people that said this to me were like bad engineers. It’s really a reflection of the organizational structure and patterns within that organization and so many, which is I have to because of how I understand the world and not because I’m not enlightened enough to understand all these things. It’s just literally, it is not part of my job. And I will tell you a very old definition of DevOps was John Vincent Looses said DevOps is never saying that’s not my job. And you know what? I don’t agree with that anymore because it absolutely is totally fine. And I’ll go on the record to say my job is to do this. The problem is if your job, if what your job is, is limiting, that is a problem with your job and not with you as a person. So do not feel bad to ever say, not my circus, not my monkeys. It means that maybe you need a bigger circus.

Michael: [00:21:33] Yeah, I’m not going to go down that tangent because that one sounds complicated, but the—

Matty: isn’t this the episode about complexity? Right. If there’s a time to talk about it.

Michael: But complexity avoidance is actually maybe a solution. And so like the— but I think, you know, a lot of that is people make good decisions or they make the best decisions they can with the information they have available. And so it’s like, what context have we passed to them? What context do they carry with them? You brought back, how do you make money? That’s a good piece of context to frame everything else around. But there’s also those other constraints. Maybe you have regulations or you have other things that you have to figure out. If every one of your developers only knows C#, I would suggest that running on a Linux platform probably isn’t the best way to do everything. It might be, maybe Mono has come a long way and so has the .NET Core and all that. But realistically, let’s be real, probably running on Windows. You want to understand the context of where the rest of your organization is. But I think a lot of people in discussions that I’ve had of late, and I would say the last 3 years are probably dominated with, well, the industry went to Kubernetes and so we did too. And, you know, it’s like Kubernetes was originally designed to solve Google-scale problems. Unless you are Google, you do not have Google-scale problems. Like even large sites that are doing really complicated stuff don’t have those problems on a day-in, day-out basis. You know, have you ever had those issues? Sure, you’ve had some of them or whatever, but like, could you have solved that with a way simpler process? And I think the answer is unequivocally Yes. And so that’s what I want people to think about is rather than I need the best mountain climbing gear in the world because I’m going to climb Everest, it’s like, what if we just wanted to cross the street? I don’t need the best mountain climbing gear in the world. I can actually just take a walk. And that’s sometimes the difference.

Matty: [00:23:06] Hey, if we want to completely abuse every analogy we can think of, but I used to be really into photography and I had all this gear and kit and everything like that. And it’s like, no, you know what? I shoot mostly with my phone right now. And you know why? Because what’s the best camera? The camera you have with you.

Michael: Yep.

Matty: What’s the best container scheduler? The one you don’t need. Right, right, right. The best thing is the thing that does the thing you need right then. That’s the answer. That’s what’s the best. The best mountain gear is the gear that you need to cross the street. You know, because sometimes maybe if you’re like, okay, to climb Everest, I need to have this big backpack and a Sherpa and like these crampons and all this stuff. And I’m like, I’m literally trying to like walk in the mall and it’s actually harder to do that because I have all this extra stuff that does not serve me right.

Michael: Right. And again, it’s not that these tools are bad. You know, I would want the best gear if I was climbing Everest or I would want the best gear if I was doing some other, you know, analogous thing. But again, if I’m at the mall, do I want a 50-pound backpack to start with? Like, let alone everything else I have to carry with me? No, I don’t. And so those are the types of things that I guess I revisit a lot and I think about how did we get here and Why do we have platform teams that are doing all this to try and abstract this complexity internally? Well, it’s because we put this complexity internally. Could we have a smaller platform team? Could we have a smaller mandate for that platform team? Could we have solved these problems differently? And I come back to a lot of the ethos of DevOps early on. It was, hey, if you’re a developer, you need to be able to do operations and stuff like that. You need to have shared empathy and shared pain. None of that was a bad idea, like none of it. But then you look back at it and you’re like, man, operations had an expertise that was really good. And we’ve kind of let that atrophy in some cases to let developers take over. They rewrote a bunch of tools. And now we have developer points of view on operations. In some cases, that’s really enlightening and really eye-opening. In other cases, you’re like, what are we even doing? This was a solved problem in 1995. And we’ve reinvented the problem from the ground up in, you know, 2018.

Matty: [00:25:00] Go us.

Michael: I have a friend who says there’s always a 27-year-old willing to redo everything you’ve already learned. So, the old man yells at cloud. I’ll be here all day. So, from a complexity standpoint, I guess these are the things I just keep thinking about is I just pull up the AWS console and it’s like I’ve got 150 services I can pick from and half of them compete with each other at this point. How do I even select what to do? And of course, there’s courses you can buy because they’re happy to sell you anything on here’s where you use this and here’s where you select this and you can get certified in that or this or in architecture. And it’s like, man, I just wanted to host a website. You know, this isn’t that hard. I don’t know. I, I just feel like we’ve lost the plot. It’s like we’re sitting here playing with bits all day instead of solving the problems we’re supposed to be solving.

Matty: I think we’ve done a pretty good job so far of explaining what sucks, why it sucks, and why you shouldn’t suck at sucky things. Do we have any like theories on like maybe how to do something about this? Or is the answer just that is what it is and you should find a friendly neighborhood podcast you can get on and rant about it?

Michael: [00:26:03] I mean, that was step one.

Matty: So that was the last half an hour. Thank you for being part of that.

Michael: Right. No, but I mean, I think part of this is, you know, there are some tools that kind of didn’t make it that were much simpler. Like if you’ve ever used Docker Swarm, for example, next to Kubernetes, Swarm was amazing compared to Kubernetes, in my opinion. Did it have a couple of weird oddities? Yeah. Could they have been fixed if they were continued to be invested in? Probably. You know, like that was cool. But then it’s like, again, did I need those containers? Okay. Can I go back to having repeatable developer environments? Like, let’s ask where the pivot points were for those complexities. And if we revisit those decisions, like, is there a second market that can open up here on the simpler solution that maybe doesn’t solve 100% of all use cases, but solves the 80% case and gets us somewhere? And I don’t want to turn this into a pitch for where I work, but that’s one of the reasons I do work where I work is because there was kind of a vision on what if we did this differently? And I’m not even going to necessarily dive into that with too hard of a thing, but there are ways to do simpler things. Like, maybe I like, what if we took all the lessons we learned using containers and Kubernetes and tried to reapply them on a simpler format? I guess I’m waiting for that generation of tools that’s like, okay, we took the complexity curve, we rode it out, and now we’re abstracting the awesome patterns or the awesome things that worked really well, and we’re reimplementing them maybe without all of the complexity everywhere. I saw a thing the other day that was called, I think it was called Rust Process Compose. I might be wrong on that, and I’ll We’ll find a link and I’ll throw it in the show notes. But, you know, it was basically like Docker Compose, except I didn’t need containers. I could actually just run processes in different ways. And you’re like, Mike, isn’t that basically systemd? And I’m like, well, it could be, but what if it’s just in a developer environment and I can just do that locally? Because I wanna have my database spin up while I have my firewall or whatever, and I wanna have, you know, maybe I have a load balancer to do 2 processes coming in and out to test that. It’s like, do I need 18 containers to do that, or can I just run 3 processes on my laptop and actually model that pretty cleanly? If that’s the case, that’s a cool tool. And those are things that I’m really excited about. And of course it’s got all the Rust ethos behind it of the rewrite it in Rust and it’s safe and all that. And I’m like, I don’t really care about that. I care about the simplicity part of that. That’s really cool. And I just think there’s some of these other tools out there, they may even exist and I just haven’t been able to find them or the industry hasn’t found them and adopted them yet because I’m not the only person that’s looked at this and been like, you want me to do what to debug that process? If I am, I guess you can yell at me on the internet. That’s what it’s for.

Matty: [00:28:28] You heard it here first. The internet exists 100% to yell at Michael Stanky. That’s it. That’s all. All the other things are incidental.

Michael: Yeah, yeah. The, the additions to that that I could make— no, I’ll pass.

Matty: So what’s like— okay, so we sort of talked about the general idea. So I’m gonna, I’m gonna push you on this a little bit and sort of say like, let’s, let’s be like real practical. So like if you’re sitting there and saying, okay, and maybe you look at it from a couple different personas, right? And where we’re at, either in the point of that you’re an architectural leader, you’re, you’ve got the ability to, to, to really define these things. Like, what are some of the actual practical tips? You’ve been doing this for a long time, dude. You’ve got like some smartness.

Michael: I’ll give you a couple of examples of things that I’ve run into. So I work at a company of 18 people, right? And so it’s really small. We’ve been building up a new product. Again, I’m not here to pitch it. So that’s not what I’m going to talk about. But some of the problems we ran into, Our website is deployed through this convoluted process that uses a CDN. With the number of hits that I get a day, I could host this off a Raspberry Pi and a cell phone modem. So do I need a CDN in here and a refresh cache and bucket replication and all this stuff? No, I don’t need any of it. And my bill, instead of being $600 a month, could be like $5, maybe $10. It’s like, okay, there’s like— and if you want a real-world way to talk about complexity, you could talk about things like, what’s the cost of doing this in dollars? Because sometimes, you know, these things that have all these affordances for scale, but if you don’t have a scale problem, don’t pay for it. You know, there are some other things that are kind of in that same realm where, you know, metric hosting and analytic hosting and all this stuff, it’s like, you know, you’re trying to run it all yourself. It’s like, well, okay, can I just outsource that to somebody and pay nickels a month for it? It’s like that, that kind of stuff is pretty nice too. But there are, there are decisions that you’re making every day that are bound by some type of, you know, there’s the way that you think the industry is doing it, or, well, when we have 100,000 hits a day, here’s what we’re going to need. But you don’t have that today. And if you don’t, maybe optimizing for that is actually like you’re buying complexity early rather than getting it later. And if you go back to early CS theory, Donald Knuth will tell you premature optimization is the root of all evil, more or less. So let’s not prematurely optimize that. I keep telling people as we’re developing things, we have kind of a monolith and they’re like, well, this is never going to scale. It’s going to break apart. And I’m like, I hope that’s a problem we have. If we have a problem where we haven’t figured out how to scale, it means we have users. That’s a great set of problems to get to. Right now, I’m worried about how fast can we get users. So, let’s iterate as fast as we can to get those first 500 users or those first 5,000 users versus the 500,000th or the 5 millionth. Those are very different goals. And at that point, you have different business objectives. And so, it’s kind of, can you adopt it along the way? But the simpler things are like, don’t use a CDN before you need to, don’t set up all this load balancing. Keeping one thing online is easier than keeping 2. Keeping one thing online is easier than keeping 25. A lot of people do microservices and you end up with— microservices as a pattern are amazing in theory. When you actually get to practice, there are places that have a lot of struggle with it. And I don’t mean like, it’s not because the place sucks, it’s just because it’s difficult. If you have one team, one microservice, congratulations, you’ve probably done it right and you are succeeding. Most places end up with 10 microservices per team or 15, or eventually you get more services than developers because of the way you keep splitting things apart. Part, final, and like into smaller and smaller chunks. How do you keep that context in your head? You don’t. So you implement something like Backstage, and then now you have like the service catalog, and you’re like, well, do I need all that if I only have one thing that I really have to keep online? Maybe my one thing has 5 responsibilities instead of one, but that might be okay for a while. And I guess it’s how do you, when do you think about that scale problem, and do you like set up for it ahead of time, or do you do it later? And this even matters like at a small company, I think it’s really easy to draw that story in your mind and be like, yeah, of course you don’t have users, you start simple and you grow. But if you’re at that business unit at Capital One or, you know, insurance company or whatever, you know, and say they’re starting a new thing where they’re going to offer, you know, insurance for your pet for the first time. And it’s like, you don’t know how many users you’re going to get. What you want to do is say, what can I take off the shelf from all the other stuff we’ve already written? Because obviously not writing software is the cheapest thing I can do. But then also it’s like, do I need to go build this out to scale out to, you know, all of, all of insurance company X’s, you know, customer base, or should, are we only targeting one state to start with or things like that? And so can I shrink this down and say, actually, I’m just going to use an N-tier application like we did back in the day. It’s got a web server, an app tier, and a database, and we’re good from there. And if it starts to have big problems, we’ll expand and we’ll grow from there. And that’s just one of the things I think about, I guess. And the other side of that is don’t be afraid of that monolith if it’s making you money and doing all these things. You don’t have to be mad at it. It’s good at its job. One of the common things I’ll often say is only in software is legacy a bad word. Everywhere else in the world, legacy means it’s awesome and has done great things and you’ll remember it forever.

Matty: [00:33:18] And in software, you’re like, this is a legacy system.

Michael: And you’re just like, yeah, that means it made all the money. It’s still there. Let’s not get mad at it and say, let’s replace it with a Kubernetes cluster. Let’s replace it with a new complicated workflow. There are reasons that things exist. And I guess that’s kind of the stuff that I start to think about is like, what choices have we made collectively versus what choices matter more on an individualistic or a private basis rather than just chasing the industry? Because chasing the industry, there’s going to be a new thing in 2 years.

Matty: It’s going to be even more fun and more I’m not going to share the link to the repo because it’s embarrassing, but I have a project I was working on by myself. That’s the important part. The only developer on this entire thing was me. This was 6 years ago and it is basically, it was a tool for like call for papers, right? Like a, you know, so like a paper call sessionized kind of something like that. And it’s all microservices.

Michael: Why?

Matty: [00:34:18] Why? Because I wanted to fuck around with microservices. But like, also it’s called Blondie. Get it? Because it’s like, call me, you know, sort of like how I have a changelog generator tool called Bowie because it’s like, ch-ch-ch-changes.

Michael: I mean, naming the project is the most important.

Matty: It really is.

Michael: Like, yeah, I, I wrote a social media aggregator once to talk about all my different, you know, like profiles and how they were getting hits and stuff. And I called it Carly Simon because I was so vain. This was definitely about me.

Matty: I kind of love that a lot. You know, we’re all just trying to ride, you know, on Afeer’s, you know, Jepsen stuff. Yes.

Michael: Yeah.

Matty: That’s, that’s all right. That’s all right. Okay. So when we think about this from, so a lot of that was when you’re empowered to be able to kind of do this thing. What if you’re not? Right. What if you’re, what if you’re in that like, okay, so I am in this like kind of squad inside a squad inside a squad. I guess it’s like, how do you think, think locally, act globally in thinking about how to reduce complexity inside your large organization?

Michael: Well, I mean, you have your own workflows, right? Or you have the workflow, you either have your own workflows or the workflows that you’re servicing as maybe a platform engineer or an SRE or whatever. And if you can say, hey, this workflow has 12 steps, I can make it 10 today.

Matty: [00:35:30] Cool.

Michael: You’ve taken out 2 steps. That’s awesome. And some of that, like some of the complexity can be automated away. And I don’t like, I like to say avoid it, but like automation is still a powerful tool. Always has been, probably always will be. And so if you can make it so that you have sensible defaults and that those workflows are shorter, that’s great. If you can say, hey, locally, my thing that I do, you know, when I debug, the first thing I have to do is spin up this debug pod and I have to turn on this trace and I have to do all that. Like, okay, can I automate that? Like, if I have to do it all day, can I make it easier? Like, that’s, again, I think that’s kind of the veneer over the complexity, but if you don’t have the power to remove it, can you get it out of your way to the best of your ability? Like, that’s one of the things you can look at. I think another thing is kind of display, you know, if you have a technical, like if there’s a technical decision maker, can you have a discussion and say, hey, here’s the workflow when I want to debug this process using this method, and here’s the one where I want to debug it using another method. And maybe they’ll say, wow, I like that method 2 a lot better. What do I lose for that? Because, I mean, everything’s a trade-off, and any good technical decision maker is going to say, ooh, I like the upside of that. Show me where the downsides are, and then talk about it. And you may learn something that there’s a bunch of context for why that was chosen that you just never got handed. And you might say, oh, this actually is the simplest solution based on all the context that I didn’t understand until now? That’s totally a valuable thing to do too. And eventually it does come back to, do you understand what your business area is doing? Like whether it’s your company or your unit, your squad, whatever. But if you really understand what they’re measured upon and what their ins and outs are, like you can probably build pretty good decisions there. And so I would say if you’re not the technical decision maker, you’re still a technical decision influencer. And if you’re not, I would say that’s probably something you should work on to grow your career. You need to be able to influence people to make decisions. And that’s, that’s a really good thing to grow on. And so if you’re known as the guy who always wants things to be more simple or the person who wants things to be more simple, you could have a lot worse labels on your head than that.

Matty: [00:37:21] It’s doing that part when you understand what your, your org does and stuff. I think about when I worked at Chase and I was, I’d been there for, for a few years and, you know, was building and, and running production support on a bunch of servers. Because again, my job was to take care of servers and whatever. And then I was putting my resume together because I was applying for jobs somewhere else. And I was looking at one of my friend’s resumes and they talked a little bit about what that was like, build systems that did blah, blah, blah. And I was like, huh, what do we— And he was in a different line of business at the bank. And so I was like, how’d you know that? He goes, oh, I went on the intranet and I found the homepage for my business unit and whatever. And I was working in treasury services at the at the time. And I went and looked and it was like, oh, we process $1.5 trillion of wires daily. Oh wow, wow, I’m glad I didn’t know that before. You know, anything where you’re like, I don’t know, I was just building servers. But like, and I think that’s a great example of like, maybe we’ve gotten better about this, but I don’t know that we have. Is again, I was just like, push button, make server go. And again, I knew how— I knew everything inside my, my little Valowicken domain, which was I knew exactly how our our domain trusts worked, and I knew how we configured services and whatever. And I, damned if I knew what the applications actually did, you know? And maybe we’ve gotten a little better about that. I don’t know that we always have, but I’m sure there’s details from the business perspective that we just don’t know. And they’re not always the hardest thing in the world to find out, but you’re also not gonna necessarily get them told to you. Right. Right.

Michael: [00:38:48] And they’re not always interesting.

Matty: They are mostly not.

Michael: I wanna be straight up, they’re not always interesting, but sometimes you can convert them into something that is more interesting for your peer group or the people you’re talking to. Like when I worked at Caterpillar way back in the day, I was in a division that was highly profitable and pretty much every transaction went through one of 6 servers. And I knew that. And so it was like, okay, if our revenue this quarter was, I don’t know, like $3 billion in this subdivision, this subpart of Caterpillar, and it went through, everything went through these 6 servers. It was like, okay, every one of the, every day this server is processing at least $6 million a day of transactions. Like, okay, if that’s going on, how does that make you think about the availability profile for that service or the investment of, you know, we need to upgrade the backup license so that it can be replicated in 2 data centers instead of 1 or, you know, whatever. Like, and it was really easy to have those conversations with my leadership team once I kind of read the little press release that comes out about how much money we make, knowing how much of the stuff went through those servers and then being able to say, hey, you know, Mr. Division Manager here, I’m asking for an $80,000 upgrade to a piece of software for something that processes $6 million a day. Is this really a conversation that we wanna be having? And when I framed it like that, I can tell you the answer was, no, this isn’t a conversation I wanna be having. Please go buy the stuff and leave me alone. And that was exactly what I was looking for. And so, what I did was I took that business context and kind of reapplied it for my needs to get something done. And I think that those are things that, you know, if you’re sitting there thinking about how are you more valuable as a technical engineer, it’s all about, can I provide the context in a way that that is understood by people that are in my area. And so take those big business objectives and map them to yourself. And like, again, some of that’s really boring, but I also will say, if you want to learn how to do that, go find an S-1 for your favorite tech company. An S-1 is the thing you have to file by the SEC to go public. There’s a section called the risk section. It is hilarious half the time because it’s like, there are literally S-1s that say things like aliens could come invade Earth and that would cause a risk to our business. Like it’s stuff like that. So they’re not always the, like the most cut and dry, like just accounting stuff. Like in the risk section, just read the risk section if nothing else, because you’ll start to understand things and laugh a little bit. Like, I swear some people just put that in there because like, oh my God, I’ve spent 6 months writing this S-1. I’m going to put some bullshit under the risk section and just see who reads it. And anyway, but I used to get assigned S-1s to read as an engineering leader to go understand what other businesses were doing. And I can tell you they’re boring as fuck in almost all cases, but the risk section is amusing. And then you start to figure out, okay, there’s a couple metrics here that really matter. And maybe it’s depending on what business you’re in. It might be gross margin or it might be your revenue growth rates or whatever. And some of that’s boring and some of it’s not. But if you can understand how that applies to your business directly, you’re probably going to get more of the things you ask for if you can frame it like that. So I don’t know. I guess early on in DevOps, we talked a lot about value stream mapping. And I think that that’s kind of become less of a conversation in the last few years because people are way more interested in some of the tech. Now, maybe it’s not. I haven’t seen the conversation as much. I mean, you’ve been plugged into some of these conferences more than I have, completely, Maddie. But, you know, that stuff is really important to understand. How does the business happen at the other end of this tech? Because this tech is not here for the fun of it. It’s there to do something. And so what is that something? Try to understand it and try to care about it. If you don’t care about it, maybe find somewhere where you do care about it more.

Matty: [00:42:08] So just want to go back to the value stream mapping, uh, for a second. And what I was raising my eyebrows for that, that, that Michael was reacting to was, was really the early on in DevOps, we talked about value stream mapping a lot. And I was like, no, we didn’t. That’s new, but it’s not. But it was like, I mean, it was like 2012, 2013. Okay, well, you know, it’s all relative. Uh, yeah, but I, I— but then it was making me think, and I’m like, why haven’t we? And, and my answer is because Steve Pereira is not going to DevOps Days anymore. And so we’re going to blame this all on Steve. But that said, I’ll put a link in the, the show notes to a couple resources about value stream mapping. And I don’t think we have to do a whole deep dive into it. And I say we don’t have to do a whole deep dive into it because we’re not going to. But what the whole point of value stream mapping, and you can watch all these talks and do all this digging and all this stuff, but really it’s just like saying point A to point B. Like, where’s the value? Where’s the waste? Where is the most effective use of that? And it’s funny, I was thinking about it. I was like, I was talking about this a lot with customers and stuff a couple years ago, and maybe Maybe I’m not thinking about— we aren’t talking about it at DevOps Days as much because Steve isn’t coming around. Doesn’t mean that there’s nobody else that knows value stream mapping, you know, but we’re thinking of other things. Or maybe the new mapping is worldly mapping and we can only do one map at a time, you know. So, go invent a new map and then we’ll stop talking about worldly mapping. It’ll be, you know, like maybe we’ll go back to process mapping.

Michael: [00:43:34] I mean, I don’t really care what it is. It’s just you understand the outcomes at the other end. Even approximate understanding is good. It doesn’t even have to be like perfective. It doesn’t have to be, you know, where you understand it up and down. You don’t have to be the business analyst, but you should know, hey, these transactions amount in this revenue, or, you know, whatever the kind of the high-end goal is there. And if that’s the case, then you can say my complexity is built because it achieves this goal, not because I read 3 books about it or I got certified in it or there was a class in it. And again, I want people to think about why do we have the complexity that we have and do you need it? And it’s not just, can I put a veneer over it? It’s, can I avoid it? And I would love to hear some stories of people that were like, yeah, I thought about doing all this stuff and then I solved this problem using really simple solution. I put up an RPM, I installed it and I hit start. Cool. That sounds great. I miss those days. And sometimes that’s awesome. And other times it’s like, Great. You can’t do that 300,000 times. And if you can’t, cool. You actually do have the problems that some of these other tools were designed to help with.

Matty: [00:44:39] So I think that’s a, that’s a really good way to kind of wrap us up there. We’ve, we’ve had a whirlwind conversation about complexity. I don’t know if we solved any problems, but we certainly talked about a lot of them, but that’s what podcasts are for.

Michael: You know, I thought it was to sell stamps.com.

Matty: Yeah.

Michael: I was going to say me undies, like, you know, all of those things.

Matty: That said, make sure you check out our sponsors. Neither of which are MeUndies or stamps.com, but stamps.com and MeUndies will happily take your money if you’re looking to spend it.

Michael: Yeah. Yeah. Use some promo code. Try and put in this one and see what happens.

Matty: Yeah. Go, go to stamps.com and put in the promo code ArrestedDevOps and let me know if it works. And if it does, then they owe me some money. In the meantime, go over to arresteddevops.com/complexity for this episode’s show notes. If you go to arresteddevops.com/itunes and leave us a review in the Apple Podcast Store, we’ll probably read it. We may even read it on the air. But I don’t know if we will or not. Last time I remember we did that, it turned out it was actually Michael Ducey under a pseudonym talking trash about me and Trevor, which should tell you how long ago that was because Trevor was still on the podcast. You can also find us on Spotify, iHeartRadio, Audible, and all the places that fine and less fine podcasts can be found. Michael, thank you for joining me today. This was a great time. I’m so glad you were on the show. Any, any parting words? Any places people can find you on the internet or elsewhere if they want to, or you don’t want to?

Michael: [00:45:58] I am pretty much Stanma on every service that I’m a member of. I am a member of various services at this point. Who knows how social media is going to look by the time this podcast publishes. We have this anyway. I’m pretty available if you look up Michael Starkey.

Matty: It’s a fairly unique name, and there’ll be links to all those fun things in the show notes.

Michael: I’m working at a small place called Phlox, F-L-O-X, if you want to check it out, phloxdev.com. And, you know, we’ll stay tuned for more things on how we hope we can help you remove complexity in your future.

Matty: Yeah, go check it out. I think we’ve bent over backwards to explain that this episode was not for Michael to pitch his product, But it actually might be relevant to what we’re talking about. So maybe go check it out.

Michael: It’s why I care. It—

Matty: I like—

Michael: the reason I’m at the company is because I care about the problem. That’s why.

Matty: We believe you. It’s— I used to say I, I work at Chef because I believe in Chef. It’s not the other way around.

Michael: And, you know, I don’t work at Chef anymore, and I don’t believe in Beatles. I just believe in me.

Matty: So, and with that, this has been Arrested DevOps.

Michael: And remember, there was always DevOps in the banana stand.