Platform Engineering goes to Flavortown with Matt Kurtiz

Matt Kuritz: [00:00:00] It really is a result, I think, of infrastructure tools and software just becoming more like real software, which is great. They’re— it’s just better.

Matt Stratton: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton. We’re going back to a really hot topic in just a second, but before we get into all of the spicy takes, let’s hear a word from our sponsors. Let’s talk about one of the most exciting events in the DevOps community, DevOps World 2023. If you’re someone who’s passionate about learning, networking, and staying up to date on the latest trends, then attending DevOps World is an absolute must. So what can you expect from DevOps World? The list is endless. First off, get ready to hear from some of the most inspiring and innovative speakers in the industry. The sessions will cover everything from AI automation, cloud-native architecture, security and risk management, to continuous delivery. And the best part is that DevOps World Tour 2023 is coming to 5 cities across the globe: New York City area, Chicago, Silicon Valley, Singapore, and London. Find a city near you and register today at arresteddevops.com/devopsworld. So Uffizzi is a platform for platform teams. You can stand up your developer platform in minutes, not months. What I like about Uffizzi is that it gives platform teams control and dev teams autonomy. It’s Kubernetes native and extensible, so you can customize it with tooling that meets your team’s evolving requirements. And these clusters, they spin up fast, like super fast. Out of the box, Uffizi combines a great dev experience, secure multi-tenancy, and cost efficiency. But try it out for yourself at uffizzi.com. Download their CLI and you can spin up your first sandbox cluster in under a minute on their free starter tier. That’s uffizzi.com. U-F-F-I-Z-Z-I dot com. Thanks to our sponsor, Gliffy, the leading diagramming solution for teams using Atlassian products like Jira and Confluence. Drag and drop shapes to quickly build a diagram, capturing anything from code structure to a simple concept. You can start your free evaluation by visiting gliffy.com/arresteddevops and signing up via the Atlassian Marketplace. That’s gliffy.com/arresteddevops. Get started today. Okay, well, it is time for Arrested Platform Engineering, the podcast that helps you achieve understanding, provide self-service, and enhance your business and organization’s effectiveness for maximum— Okay, the joke’s not that good. But yes, that’s right, we are talking about platform engineering again. And I think we’re making our peace with that because this is a pretty important topic. It’s not a topic that’s going away, nor should it, nor should we want it to go away. And joining me today is another Matt. So we’re gonna have Matt and Matt. That’s gonna be a lot of fun. Matt, you want to introduce yourself to our Arrested DevOps audience or the APE audience? Since I guess that’s our new branding. Oh, ape.

Matt Kuritz: [00:03:32] Oh, wow.

Matt Stratton: It’s the ape. Oh, no, this is bad. This is going in a whole weird crypto way. Oh, it’s all bad. I’m sorry. Platform Engineering is over because the podcast initialism would be wrong. Anyway, welcome to the show.

Matt Kuritz: Yeah, absolutely. Thanks for having me. Just to give everybody a quick intro, I currently work for a company called The Farmer’s Dog. We make fresh dog food. If you have a dog, for your dog, obviously. I was going to say for your pet, but it is just dogs right now. And I work there as a staff engineer and the tech lead for the platform engineering team, which I assume why I’m here. And we’ve been doing this for about the last 4 years now is when we started our quote unquote platform engineering. Journey. The company has grown quite a lot in that time. It’s gone from a few engineers, very early-stage startup, to the engineering team now is about 50 to 100 people size-wise. And we’ve shipped over 100 million meals to our four-legged customers. So it’s worked really well for us as we’ve gone on our journey and excited to talk to you about it today.

Matt Stratton: [00:04:34] Very exciting. This being a podcast that is a non-visual medium, I’m gonna explain to you what, what just happened that was kind of funny. I have headphones on, so nobody in this room can hear what Matt said, but somehow when he said the word dog, Moxie, my Australian Shepherd on the couch behind me, perked up and went, what? And I was like, how the hell did she hear that? So Aussies have incredibly good hearing and they can hear either that or she can read lips. So cool. Well, welcome and let’s get into that a little bit. One of the things that Matt mentioned to me when we were talking about doing this episode is a blog post that Charity Majors published on the Honeycomb blog about a year ago called The Future of Ops is Platform Engineering. And we’ll put a link to that in the show notes so you can check that out. But a couple things when we think about what is platform engineering, like, again, we don’t have to— I was going to say we can be tired of redefining this, but we’ve been redefining DevOps for like a decade, so clearly we’ve got at least 8 more years to go of redefining platform engineering. One of the things I thought was really cool in this blog post is Charity has this table where it’s sort of saying, what’s the breakdown, right? Like in these different categories, if you’re gonna say what a platform engineer does versus, you know, someone in ops or DevOps, and we’ll have a whole other episode fighting about that. But one of the things I thought was interesting is the question was SSH, and for a platform engineer, it was no. For ops engineers, yes. But kind of when we think about, are these things that are really such an even split? Matt and I were talking before, and I said, you know, we’ve had a couple different episodes about platform engineering so far. Actually, one, I love the fact that if you go way back, and we’ll put links to all these in the show notes, Bridget had Kelsey Hightower and Andrew Clay Shafer on many years ago talking about platforms. And it was kind of the beginning of thinking this way. I’m not trying to claim that this whole idea came from Arrested DevOps podcast, but I’m not not claiming that because why not? But we also recently have had a couple other folks on the show. We had Daniel Bryant back in May, and then most, more recently in June, Pete Cheslock also joined us to talk about platform engineering. And one of the things I thought was kind of fun is none of us actually do it. We just want to talk about it. So what a novel idea. Let’s kind of look at what is the point of platform engineering at The Farmer’s Dog? What’s the point besides a CV-driven development?

Matt Kuritz: [00:07:03] Yeah, it’s a great question. I mean, for us, and I was the first person on the team, I had to think a lot about these things. I don’t know if I totally knew when we started going down this path. Honestly, when I saw the job listing, I was like, I really don’t like being called a DevOps engineer. It’s not, it shouldn’t be a title. So I was like, this is great. I’d prefer to be called a platform engineer. I don’t, really know what they want, but I’ll take that as a great starting point. But, you know, the thing that it reminded me of, you know, I get a lot of seeds of ideas obviously from other people, but a really random thing in the grand scheme, I think I remembered in a less maybe popular book than Continuous Delivery that Jez Humble wrote called Lean Enterprise. There was a graphic about, you know, self-service and operations building a PaaS, a platform as a service, right? And I never really wanted to do that because it’s like, hey, that’s a lot of stuff you have to maintain and do and you need a big team for that. So I never really followed through on that idea, but like the goal of that stuck with me is that an ideal, like in an ideal world, pie in the sky, there’s no handoffs between developers or operations at any point. And, you know, necessarily there’s very strict ownership by product teams end to end of all the software they’re creating. So it really starts with that for us. It’s about ownership and really trying to create, I guess, our own integrated socio-technical system around that that drives the right incentives and behaviors in our company. So for us, it starts with saying, hey, you’re in one of these 2 camps. You’re either on a product engineering team or you’re a platform engineering team. And the only difference is who the customer is. We don’t consider one operations, one development. Either building for engineers or you’re building for somebody else. And everybody’s concerned about site reliability, all the things that go with that. Everybody’s concerned about delivering software, all the concerns with that. We all have to do that for our customers. So for us, that’s the starting point. It’s really about creating those behaviors by creating the organizational structure and the culture and language that pops up around that, I think, is really important. And that’s what’s been helpful for us. And I think That is something teams can get potentially from following this approach. And I guess the first thing I would like to disclaim for everybody, like if you’re doing quote unquote SRE, DevOps, whatever you’re doing at your company, if it’s working, I’m not saying you should drop that and do what we’re doing right now. I’m also not saying this is going to work for everyone. I think Charity covers that nicely in her blog. Some companies have to go deep on infrastructure. This is not for you. You’re going deep on infrastructure. But for a lot of companies that are providing a customer-facing product, very high level, you can do these operations without the infrastructure is another way I think she puts it in the blog of platform engineering.

Matt Stratton: [00:09:46] So, I want to think about that. Like you said, you thought about in Jez’s book talking about building your own PaaS, and you’re like, well, that seems awful, but it seems like a lot. And I guess going into that nuance is sometimes you need that. But it’s sort of going back to the why. And I feel like we’re not allowed to have a podcast talking about platform engineering without referencing James Governor’s tweet that says the modern tech industry is basically folks just endlessly remaking remakes of Heroku. But maybe not, right? And, but then I think, and we think about the consumerization of IT, you know, going back to one of the challenges that we started to face in the last 10 years, With the advent of cloud and, you know, we think back to the quote unquote bad old days of a 3-month rack and stack for a server, you know, the old classic example, cloud example everybody loves to reference is whatever that dude was at the Washington Post who was like, I need to digitize all this microfilm. And IT was like, dope, we’ll give you a server in like 3 months. And he’s like, I’m gonna slap down my credit card and get AWS. And I did this whole project over the weekend. Ta-da, that’s cloud, right? So the idea is that our users expect Dropbox. They don’t expect, you know, a Remedy ticket that gets them a 3-month Rack and Stack, but we don’t necessarily have to go the full consumer level because your users are not necessarily consumers. So I think maybe if I’m distilling some of what you were saying is we don’t have to actually create literal Heroku. But what did Heroku provide? And can we get that same? And when we say that experience, it’s not the experience of like, cool, I click a form and it does the thing, but really just putting the abstractions where they need to be and minimizing the friction, right? Like getting as close, because that’s what, that’s what Heroku does, right? It gets you to value fast. And it’s not because it feels like using Dropbox. It’s just, it lets you do the thing, right? And I guess I was thinking, I’m glancing at Charity’s table again. And of course, you know, given that I’ve the last year plus continuing my career, you know, doing DevRel for the data world. And, but one of the differentiators Charity says on the database is one is, you know, in PlatEng, right, using hosted DBs versus running your own, blending your DBA expertise. And it’s just like, wow, this whole episode is not going to be quoting Charity, but it probably will be.

Matt Kuritz: [00:12:13] But right, you know, right, a lot of things I agree with.

Matt Stratton: There’s all the things, right? But like, what’s the other thing Charity loves to say is, said, you know, the The best tool is the one you don’t need. The second best one is a SaaS. And I used to say when I was at Red Hat and I had customers in the public sector, I’d say, you’re not the National Department of Continuous Delivery. Why would you build your own thing? You know, so maybe I’m going off on a crazy tangent, but a little bit is still getting back to you’re getting closer to value with what you’re trying to build here. And if it’s not something that is providing the value to your team, Don’t do it.

Matt Kuritz: Be super strict about that. And let me take it back to Heroku. Yeah. Because I think Heroku is very interesting. If Heroku works for you and your business use case, then when— I think there’s other concerns now, right?

Matt Stratton: But Heroku doesn’t work for anybody else because anybody anywhere anymore because Salesforce bought it and screwed it up. There we go.

Matt Kuritz: Right.

Matt Stratton: That’s my one big hot take. But yes.

Matt Kuritz: Back in the day. Back in the day.

Matt Stratton: I’m sure there’s someone, you know, but start with that.

Matt Kuritz: I don’t know. What’s today’s Heroku? I’m going to maybe just be shouting out companies and give them free advertising. But like Vercel, if you’re a very frontend-driven app company, your applications are frontend, you don’t need heavy backend stuff, you might get by just fully on Vercel, or at least that takes that concern away. Right. So there are more companies starting to fill the gap too, and we can get into that. But I think the point I want to make about Heroku is Do what works at the right scale and then iterate and evolve from there. And then the other part I would make is, the other point I would make is that like the tools are getting better. And I think part of the fact that they’re not all-in-one like Heroku is a feature and not a bug. That I think is just natural in terms of there is a diversity of complexity. If you’re going to have like one tool that’s going to be a platform that hosts all your infrastructure, It’s going to have to be very flexible because all different companies are going to have all different needs. The more flexible it is, the less it’s really going to do a platform engineering thing of you like defining golden paths, whatnot. So I think the fact that we do have tools coming along that are SaaS tools or, you know, PaaS tools, but they’re more focused on like individual parts of a system and they’re more flexible. They have better APIs. They have better SDKs. Or they serve a whole business function. Stripe is probably the best example of that. Do you want to run a payment gateway if you don’t have to? I think most of us prefer to use Stripe. And this is where I think my version of platform engineering, Farmer’s Dog version of platform engineering, the one we’re doing is potentially a really great pattern for people in terms of we’re not building a PaaS, we’re assembling a quote unquote loosely defined platform of saying, these are our preferred tools and we’re making the glue and putting the extra pieces in for the thing that works best for our business and our teams. Right. And that, that we’re creating something that is unique. I don’t know if anybody is using our exact combination of tools and stuff. And that means that we potentially can create an advantage for us by doing that. We’re not, we’re trying to create as little of our own software as possible, but we’re still able to create something unique that potentially makes us higher performing than another team and gives our business an advantage. So I think it’s kind of this nice in-between and like, it really is a result, I think, of infrastructure tools and software just becoming more like real software, which is great. It’s just better.

Matt Stratton: [00:15:49] We’re getting better to doing that. Had an episode that at least is a time of this recording just published with Sarah Morgan, when we were talking about product management approach to like how we do, whether it’s DevOps or anything like that. So when you’re thinking about what you’re building, so my first question I was starting to think about was like, how do you know that what you’re doing is doing well? Right? This is a key part of all this DevOps stuff is measurement and reflection and everything. So I had questions about how you’re evaluating the work that you’re doing on your team, but then also I guess that’s, maybe this is a bit of a leading question when I said, you know, I was talking about product management, but what are the things you think you do that make it really, like, how do you, if you are reasoning about it as a product, right? Like, what are you doing to do that? How’s that working?

Matt Kuritz: Yeah, that’s huge. So I think the first thing is we have to be aligned with the product teams on what our goals are as an overall org. So for us, the overarching like North Star is the ownership piece. And it’s taking steps towards that. Like what, what does this look like 1 to 3 years out is next in terms of where we’re going. At first, when I started here, it was really just about trying to get to a state of continuous deployment. It was really important to us that we set that as a goal. I think something else Charity harks on constantly, like it will change the way you work. To have every single commit have one true owner. So there is an ownership aspect of continuous deployment. If I push something to trunk, however you do your source control, and that’s going to trigger a deploy, there’s no ambiguity about who’s responsible for that deploy, who’s responsible for that pipeline, all the way through to our customers in production. Could even be in terms of validating, you know, the, the business metric. And we talk about our product teams as, you know, they’re product engineering teams, They are thinking more like from a product mindset too, not just more about operational concerns. So they’re really running the gamut. We’re trying to like put them in a position of they see the whole thing end to end and can do so because the cognitive load is so low across the spectrum. So yeah, that was a huge one for us. We got to 100% continuous deployment of all our applications. There was definitely a lot of, you know, technical, just fixing work we had to do. We had a distributed monolith that made it hard to get to that point. So we focused on a lot of those things with the team and in terms of like just strategizing how we’re going to get away from that. So the interesting thing about this, we started platform engineering, we’re like, hey, we can’t even start doing golden paths or anything because we have to get to a point where we have good practices. And this is where I think I get really frustrated with, you know, some of the things you see on the internet of people saying like, DevOps is dead. Platform engineering is the future. Charity gave this talk at DevOps Days New York about why that’s a little silly, but platform engineering is a good pattern. I’m saying it way, way nicer. Everybody should go watch that talk. It’s good. But yeah, no, the, we’re not doing, we didn’t do anything too different than, you know, anybody might do in a company that says DevOps. We just didn’t talk about it that way. It was like, let’s figure out how to deliver to our customers quickly and safely. For us, we just are organizing ourselves in a way that’s a little bit different. We’re not talking about devs, we’re not talking about ops, we’re talking about platform and product. And we’re sharing those goals across the team. And the individual goals are more about individual customers. Eventually we did get past that though. And I think probably the part people want to hear about is, well, how did you actually start building out the platform? When we did get to that, again, it was starting with the ownership. That’s our North Star. So how do we get better ownership for the teams? Well, we have too many people for just everybody to be on one team anymore. There’s not enough people aren’t aligned to the few monolithic apps. Well, we had to start thinking about, well, how do we create, how do we break down our system and create the boundaries in our system where we have teams mapped to the right business domains that have applications that live there so they can own them? Because platform doesn’t own anything. We’re not responsible for liability. We’re not responsible for operations. We’re responsible for building things for the product teams and helping them to do those things as the true owners. So that’s still, I think, a goal we’re working on of trying to get to a point where every single application in our system has one clear owner is our kind of multi-year timeline goal. Now, we made a lot of choices that were driven mostly by the product team. So for us, how do we do product management on these, I guess, more like quote unquote infra ops kind of topics. For me, it’s like, let’s work with people hands-on, like actually work together, watch what they do, get the feedback. So, you know, I’ve heard people say, I’m forgetting, like duct tape and rubber band product management, but it’s great. Like I can talk to my customers, I can work with them, I can pair with them. Like what is an actual use case we have? Let’s build that thing, do it all very manually, see if we like the result or not, maybe do it 1 or 2 more times. And then like, okay, we’re getting to a point where like maybe we do want to actually invest in building some software, doing some automation around this, building a code generator, golden path, whatever it is that actually makes it a standard now. So we don’t upfront really decide anything. It has to be done at least once and put into production. Before we’ll even consider building something for the whole org and saying, hey, everybody buy into using this thing now. So I think it’s a huge mistake teams make regardless of what you call them, if it’s DevOps SRE platform, anything else that we use is that, you know, we make the decisions within our own team. I don’t want to say silo. We can talk about my dislike of that term, but we make it without the input of our customers within our own, you know, kind of echo chamber. And then I guess rightly, like certain people aren’t happy with that and they push back.

Matt Stratton: [00:21:56] It’s kind of interesting about making decisions in an organization about the— we have this tension of on one hand, we want to enable squads to like pick the tool that matters for them and work in their thing. But then you also say, when does it make sense to make decisions that are at a larger perspective? And of course, sometimes depends on where you’re sitting in the org chart. If you’re in the procurement department, then you have a very strong opinion, but those are not what should drive them. But it’s not always a matter of, we like to say that and say like, oh, we’ll let everybody build whatever they want, use whatever they want. They know the best, the chop, the closer they are to the work, the more sharpened they are, the better they know about the exact right thing. And the only people who want to do things across the whole org is your procurement department because they want to save a buck. And I think that’s simplistic. And I’ll tell you kind of the way that I’ve thought about the decision point about when you can say you have within your squad or bubble, you have your do whatever the hell you want, we don’t care, you pick the right thing, versus when it matters to do it larger is, is this an interface point between groups? Right? Like, is it— if it’s an API, is this an endpoint or not? If it happens inside the API, who gives a shit? Right? So that’s why things like you don’t have a different source control tool across different squads, because that’s, that’s nuts, right? Because everyone’s interfacing. But like, which particular JavaScript library you use for like your form validation, who cares? That’s all standing within your thing. Like, so when you’re thinking about how you’re making those decisions in your organization, I guess I’m sort of saying, I’m like, let’s test that a little bit. Does that seem like—

Matt Kuritz: [00:23:38] Yeah, that’s a great example you brought actually, because like, that’s something I think we do behave a little bit differently. If it’s something that’s like interaction between teams, it’s not clearly owned by any team, that’s where platform actually steps in and is the decider and really makes the final decision and everybody has to kind of live with it. And I think that generally goes over okay because it’s the exception, but like a great example might be like tools for asynchronous communications, however you do it, right? There’s so many options. They have all different feature sets. And like, before you know it, you could have like RabbitMQ, SNS, SQS, Kinesis, Kafka, and it’s like every single time you want to do something, you need to be using a different tool. Every app has like 7 libraries in them to do these things. And so we’ve visited this many times and teams have asked for certain particular tools a lot of times. And it’s not like just us saying, no, we’re not going to do it yet, but we really try to get the temperature. From all the teams and really just facilitate what our feeling is as an entire engineering org and ask the tough questions like, hey, if Team X wants to introduce, say, Kinesis, and this ends up becoming a standard, is that something that would be beneficial to your team? Would you like maintaining that? Like, how would you feel about needing to use that when you try to integrate with this system? And we try to collect like all those opinions and take our own understanding, I guess, of what is now becoming more like systems engineering, being the platform team and being the quote unquote experts on that and try to make a decision and make sure whatever we’re doing actually is getting business value for whatever the cost of it is. And I think a lot of these things, there is some kind of like path dependence that we don’t want to admit. When we really should. I don’t know, like Kafka is a great example. Everybody loves to use Kafka. It’s a great tool. It’s also, I’ve seen so many incidents dealing with Kafka because it is such a powerful and flexible tool, and there’s a lot of ways you can get hurt using it. We’re just not ready for it yet. We evaluated Confluent at one point. We had a really hard time coming up with use cases that were like now. I think we’ll definitely use it at some point in the future.

Matt Stratton: [00:25:59] I think when you’re ready to use it, you should look at Aiven for your Kafka. Just so you know. If you’re just joining this show and you haven’t been listening lately, Ivan is where I work.

Matt Kuritz: Got it. Where I was going with that though is just being really critical. I think you actually brought this up first earlier on in the show of, are we going to get value out of it right now and not be too forward-looking? Because like part of it is, hey, there is a cost of doing this migration. Switching all the teams over. And you always have to do that, of course. And it means that we are— there is some kind of path dependence that if you haven’t used, or if you’re not familiar with that term, means that like you become dependent on the things you’re already using, the path that you’re already on. It’s hard to change things. The classic example is if everybody’s driving on the right side of the road and you wanted to change to driving on the left side of the road tomorrow, you can’t really do that because you have to change like all the stop signs, the street lights, blah, blah, blah, blah, blah. The cars themselves are not designed for it. Yes, but we have to be honest about those things. If not, we’re doing a bunch of work that’s not creating value for our customers.

Matt Stratton: [00:27:03] Say, hey, other team, if team, you know, team B, if team A was using Kinesis, is this going to be your thing? Like, are there times that you have to say, and they, when they might say, oh no, no, that’s fine. If you could say, you know, is it? Maybe you didn’t think about this because maybe you don’t know what this is. Like, I’m wondering if there’s a little bit of almost like that, that challenger sale type of thing to say, You know, because rather than sitting there going like, well, nope, they said it was fine, but, you know, say like, maybe, maybe you don’t know why this is going to be a problem or not even, or a consideration. So that’s—

Matt Kuritz: yeah, absolutely. And yeah, we’re challenging everybody’s assumptions. And like, when we’re going to those conversations asking for opinions, we’re trying to make it really clear, like, hey, this is how we foresee it happening. And like, dive in, hear what they say. If the response doesn’t really like— I don’t, I don’t know a good example— make makes sense to us or seems not like the reality we’ve experienced working with some of these tools, then, you know, we’ll challenge those things, push back and say like, okay, you’re imagining it working like this pretty smoothly, but in our experience, XYZ, there’s actually this ongoing maintenance and challenges. Or, you know, we’ll also just ask like, have you actually done this before?

Matt Stratton: [00:28:12] Have you even really know what it is? Because nobody knows all this stuff, right? Like you could sit there and say like, hey, is Kinesis going to be weird for you? And you’d be like, well, I’ve never heard of it, so it’s probably fine. Because it couldn’t possibly be relevant to my life because I’ve never heard of it. And you’d be like, well, this is what it does. And like, oh shit.

Matt Kuritz: And yeah, I’ll say like the thing is, I think most of the time when we do evaluate some of these tools, there is a real use case. The teams are asking for it because they really need it. And it becomes less about saying no to it. Although there are definitely times we’ve said no to introducing new tools. It’s more about like, okay, great. We have, we have the real use case for it. How do we actually first introduce it, migrate where needed, and then implement it in a way where we are getting the most out of it. No different than any other team would do, but I think that’s really where the meat of the collaboration with, with the teams happens when we’re going through this process.

Matt Stratton: Well, and I think that was, that was one of the other notes I, I scribbled down here was you talked about earlier about how some of this because it’s lessening the cognitive load for certain teams. And that’s certainly a complicated thing that we’re dealing with in today’s distributed system world. You said, hey, there was a time 15 years ago when you could hold the whole stack in your head because it was like the LAMP stack, right? There’s no— nobody can hold all of it in their head. And so having these abstractions in these places and this work that you’re doing is helping lift that cognitive load from a lot of the parts. But then also, you still have to have some folks because to be able to put that, because we’re not saying we’re in our silo, by having a lower cognitive load, but being able to be more focused is very freeing, right? So we’re like, okay, I don’t have to think about all that. But you still— someone’s got to have front of mind, like all of the, you know, places, because again, and that’s what I was getting at with that question about like, oh, Team A wants to start doing this thing. And to Team B, you might go to them and say, is this going to be like, do you see any ramifications of this? They might be like, no, I totally don’t. And you might be like, ah, but that’s because you actually don’t know what this is, you know, which is totally legit to not know. It’s not because team B is dumb. It’s just whatever. Everybody doesn’t know everything. How do you manage that as well? Like being broad, but not deep, maybe across your organization stuff besides being broad, not deep across tech in general.

Matt Kuritz: [00:30:34] I could talk a little bit about philosophy and the particular tactics that we’re using. So for one, I would. Just say for everyone that regardless of how you define the work in our field, we deal with a lot of double-edged swords, things that look diametrically opposed that actually aren’t, and then real trade-offs, right? Where we are picking something that is beneficial in one way and not in another. So I think really just understanding that as a starting point is really important. And I think my dislike of the term silos is exactly what you said. We have to bound our context in some way. I really appreciate, you know, kind of like not formal domain-driven design, but the thought process of domain-driven design from that standpoint of like, hey, we’re saying break down— people talk about breaking down silos. It’s not about just saying there’s no silo. If there’s no boundaries in your system, everyone needs to know everything. That’s impossible. Like small startups can’t do that anymore from a business standpoint and an engineering standpoint. So we have to put boundaries somewhere, right? I think functional silos are unhelpful. That’s part of what DevOps tried to help with, right? The functional silo between, or the functional wall between dev and ops. And we wanted to remove that, but yeah, kind of what we’re doing is like, hey, we’re just not even going to use those terms. We’re not even going to break down into functions, but we are going to bound our context. There are specific, like strong boundaries between teams. We don’t want teams to have to constantly communicate to get things done. So part of it is in the business from a business standpoint, where our boundaries are and our ideal is like, hey, the team working on fulfillment stuff shouldn’t really have to talk about the team working on the signup flow very frequently. They’re 2 separate parts of the business. Sometimes there is a collaboration that needs to be done. But also we should have tactics for how to do that. I’ll drop another really great blog. So it’s on Jessica Curry’s blog. I don’t, I think the name of it is Better Coordination or Better Software. And she talks about on the blog, like you can do all bunch of tactics to make handoffs and coordination more smooth, or you can spend more time building technical components of your system that allow people to work without that coordination, to work in an asynchronous decoupled way and create better software instead of spending so much time on the coordination piece. So for us, yeah, somebody looks at our system and they don’t see how it’s working, they may be like, hmm, they have a lot of, they’re trying to build business silos. And for us, that’s okay because they’re cross-functional, vertically integrated teams. They’re owning things all the way from idea to production. So it’s a quote unquote silo in some sense, but it’s not a functional silo. And they’re able to serve their customers very easily by, you know, as long as they don’t depend on other teams. So that’s always a challenge is keeping things decoupled enough where that actually works. So that’s our philosophy. That’s step one, like the business, like build these vertically integrated bounded context domains where people can work independently. Then from the technology piece, There’s a lot of things we do from an engineering standpoint. I guess this comes back to like building the platform, working with the teams to figure out like, hey, what can we do to make this easier for everyone? Not just like pick a particular technology. We’re going to use Fastify instead of Express, which is part of it. But like, how do we want to work? There’s a lot of different ways to work that work well. So one thing our team was very passionate about early on was following a monorepo pattern. And we made sure to like really do that the quote unquote correct way internally. We started out for better or worse with Node on the backend. So we’re full JavaScript. Another thing the team has really agreed on. And when I say team, I mean the larger engineering organization is let’s not introduce another language until we really have to. There’s a lot of challenges to working with Node at the scale we’ve gotten to, but we think the extra complexity of an extra language isn’t very helpful. The great thing about being on the one language and in the monorepo, there’s like a lot of really good native JavaScript tools for working with monorepos. The potential downside that you can get with that is that they’re used heavily in open source. So they have features for working with like publishing public packages that could lead you to what you might call an anti-pattern for doing a monorepo. And For me, the nice thing about the monorepo, I think the main goal— there’s actually a great website on it. If you— I forget if it’s just— no, I’m sorry. It’s not a monorepo website. It’s— I’m thinking of trunkbaseddevelopment.org. They talk about monorepos as a tactic to use to implement trunk-based development, but that’s really why we’re doing it, right? We want to practice trunk-based development, practice continuous deployment. This is a tactic that helps us do that. And it helps us, you know, be able to essentially do these lockstep upgrades is what it talks about without versioning and the costs that come with trying to slowly migrate versions across source code that lives separately from one another. So the main, the main thing with the monorepo you have to be able to do is like, if there’s only one version, it’s head, whatever the top commit is on trunk of your repo. That’s the version. If you’re doing that for an internal monorepo, you’re getting the benefit out of it. If you can’t do that and you’re doing versions within the monorepo, you’re probably not getting benefit out of it and it might be more cost than it should be. So for us, like this is great for engineers though. You talked about like upgrading a library in Node, I think at some point. We have dozens of apps in our monorepo now. If we’re using, I think I mentioned Vastify before. We want to do a Fastify upgrade. There’s just one atomic commit to upgrade Fastify across the repo. All of the apps that have that, if they have dependencies, that will trigger the tests of the dependencies before we actually push that out. And we can do big upgrades like that very, very quickly. That’s a huge relief of cognitive load from developers and just pure time spent. So that’s one thing. The other interesting thing is We don’t actually have anything that you would think of as like an internal development platform or like a, I don’t know if that’s what they call Backstage, the open source tool, but like everybody’s using Backstage, right? To like maintain where all their apps are. Like for us, we haven’t gotten to a scale yet where that’s really helpful because our monorepo is organized by our business domains. And, you know, all the apps are organized into those business domains. So you kind of know where to look. There’s a code owners file, you know, who owns what. That is kind of the directory of all the applications and you should know everything about it just by going into the repo. And to me, it’s a great interface, right? What do developers know? They know Git. They know how to work in source code. And so I joke sometimes and say like, what’s our platform? It’s a monorepo and a code generator. That’s our platform.

Matt Stratton: [00:37:39] What else do you need? As Pete Cheslock would say, that’ll get you where you got to go. I guess as we kind of think about bringing this around, we know there’s not one right way. There’s definitely— while there might be no one right way, there’s probably a dozen wrong ways to do anything. But if you wanted to give one thing to start doing, one thing to stop doing, and one thing to keep doing for folks who are trying to get similar results and maybe a similar kind of approach to what you’ve done, especially with your history and experience where you’ve gotten to today. So we’ll do one thing at a time so you don’t have to remember all of them. So first would be like, what would be one thing to say, I think the most important thing is really higher level than technical.

Matt Kuritz: It’s 2 mindset things. I’ll give you 2. So I like to think in terms of objective, strategy, and tactics. First, start with what is the goal we’re trying to achieve. The strategic in-between, which I think is easy to step— to skip and go directly to tactics of, you know, what’s the overarching I guess, kind of process of how we’re going to do that. And then there’s the individual, like, tactical choices on the ground you’re, you’re making. So there’s a great example, but I’ll just talk about, like, setting goals as a specific thing because I, I talked about that a few times, I think, before. I think this is something we all need to do better, including myself. I’m always trying to get better at this in terms of knowing what is our North Star, what’s this, like, long-term kind of vision. And, you know, I think I first picked up on this from Mike Rother’s book Toyota Kata, which is a really great resource if anybody hasn’t read it. If you have a dislike of Lean for some reason, I would say a lot of things were learned in the Lean community from that book and a lot of minds were changed by it. But one of the most interesting things from it is this idea of having a vision that you can’t ever achieve, so you always have to improve closer to your vision. And I think knowing where you want to go continuously leads to all these other things we talked about— continuous improvement, continuous delivery of product value. If you don’t have something you continuously work towards, it can’t be continuous. So starting there and then being able to break that down to smaller and smaller amounts. So I’d say do that. What’s your North Star? What is, you know, maybe your 1 to 3 year goal? That’s more concrete. And I’ll give you an example. So for Farmer’s Dog, maybe our long-term vision is you could say every dog lives their longest life. That’s, that’s what we want to achieve. We don’t really know what that is. So we can always try to do better at that. It’s never achievable. But we can always do something to try to help a dog live a little bit longer. And that, that’s our long-term vision. What is our challenge for the next 1 to 3 years, I’ll make something up, but maybe it’s like, you know, we want to make special collars that do health data and that helps owners. And that would be like, wow, that’s like totally not part of what we’re doing today. But if you know that long-term vision and it helps towards that, it helps you get there, then you know you’re not losing focus as a business and it gives you something concrete to work towards beyond like just setting KPIs and OKRs, which I think we get caught up on a lot. Where do you want to be in the future? Define that future state, work towards it. Then I think it’s a matter of like, okay, now we know where we’re going, we can iterate. What’s the next— in Kata, they would talk about target condition would be the exact word, but where are we getting to next? And I don’t think you need to like follow that exact framework. It can be at the right time scale for your business and you need to play with it. Like maybe you start out and you’re thinking about like, hey, my next— the next place we need to be is here 3 months from now, and you can define that more concretely now with a few pieces of data. It might be like, we have to help achieve this from an engineering standpoint. We want to get to deploying our apps 3 times a day because at the very least we have 10 engineers. Hopefully if we’re doing CI, everybody is committing to trunk once a day. So maybe we’re on the part of our journey where we need to get to, you know, 3 deploys per day, 3 commits per day from the 10 engineers and Eventually we’re going to want to keep iterating on that, but that’s our next target. We don’t know how we’re going to get there, but we’re going to try a bunch of things and maybe, maybe we’ll get there 3 months from now. That’s, that’s kind of the goal. And we’re going to keep kind of just going through those iterative cycles of trying things, learning, and trying something else. And every part of like the business can participate in that. It happens at all different levels, right? So there, I was talking about the overall business, then I jumped into engineering, but I think of the engineering level, you do it too, right? You think like, well, how do we help with these, I don’t know, IoT dog collars. And maybe it’s, well, we need to support IoT, we need to deliver faster. You come up with those things and you break that down so people can contribute to it. So, sorry for the long answer. That’s the one thing to do. Understand what your long-term vision is, break that down into goals to keep you focused and aligned and moving in the right direction.

Matt Stratton: [00:42:42] Excellent. What’s something people should stop doing?

Matt Kuritz: Copycatting, or the term I know people don’t really like is cargo culting, but I will say it just this one time because That’s the exact term of CC science that Richard Feynman used in his address, which is an amazing thing to read, I think, if you haven’t. But, the idea was that Feynman observed in actually doing scientific research that large swaths of entire fields were just copying methods but not getting actual scientific results. So what they were doing looked like science. They followed all the same processes that people doing science are doing, but the questions weren’t right. The underlying assumptions weren’t right. In some cases, we see this, the statistics aren’t even right. So the term I think that gets used a lot now is scientism or pseudoscience. And it’s a problem I think that we suffer from and a lot of fields suffer from. It’s not just something that happens in science. You see a lot of people copying artifacts from one context, but it’s not actually getting the results or the outcomes in the new context. And I mean, to some extent, you could say this is what Lean was of Toyota. And I think that’s kind of the point that came out of the Takata book was, hey, we missed the overall thought process, the mindset, the problem-solving, what makes it an integrated sociotechnical system, not just Kanbans and Adnan cords and these little things we saw, which can be really helpful tools, but that’s not really the point. And that’s not— when you actually slow down and observe that firsthand is what was being said is there’s way more under the surface there. And it probably wasn’t noticed because of a language barrier for one, but even without the language barrier, we do that all the time. Look at the platform engineering, what we talk about day to day right now. It’s, hey, I’m going to launch a Kubernetes cluster backstage, put a million controllers on it. There’s going to be one standard way to do everything. And it is really just copying the, the things we see other companies doing. And we’re not really getting, I think, the results unfortunately in a lot of those cases. So I’d say don’t do that. It’s hard to say don’t do that because obviously if you know you’re doing it, you want to be doing it. But maybe again, to turn it to a positive, think about your problems, solve those problems from first principles, don’t copy.

Matt Stratton: [00:45:21] Amazing. Amazing. Well, we are out of time. This has been a fabulous conversation, really digging into a lot of a lot of great success patterns and giving me a lot to think about, hopefully giving you, the listeners, a lot to think about. If you head over to arresteddevops.com/flavortown, you’ll find all of this episode’s show notes. I’ve been furiously scribbling down all kinds of links, and I’m going to be tagging Matt to provide a few of those as well as soon as we’re done recording here. So, definitely check that out. If you go to arresteddevops.com/itunes, you can leave us a review in the Apple Podcasts store, but I still make the link be iTunes. Also, I’ve been making that joke and reference for at least a year and a half here on the show, so I should find a new joke. I tried to find a new joke at the beginning of this episode, and it wasn’t that good, but that’s all right. And you can also find Arrested DevOps on Spotify, iHeartRadio, Audible, and all kinds of places that fine and less fine podcasts can be found. Matt, thank you so much for joining me today. This has been a real pleasure, and I hope you’ve enjoyed your time here on Arrested Platform Engineering.

Matt Kuritz: [00:46:29] Yeah, thanks so much for having me. It was a ton of fun. I, I really appreciate it.

Matt Stratton: This has been Arrested DevOps, and remember, there is always DevOps in the banana stand.