Platform Engineering with Daniel Bryant

Matty: [00:00:06] It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matty Stratton. Today, we are going to dig into the— a word, a phrase that might be on the tips of the tongues of our audience, or maybe in the inner ears of their listening. We’ll find out, and we’re going to talk about this idea of platform engineering. But before we do that, let’s have some words from our sponsors. Collecting compliance evidence shouldn’t involve spreadsheets and scavenger hunts. With automated controls and over 75 integrations, Drata automates the process without needing to be an expert. Drata supports 14 frameworks including SOC 2, GDPR, HIPAA, and ISO 27001. Companies like Notion and LemonAid have shared how Drata simplifies audits through automated evidence collection. Don’t let audits slow down your dev cycle. Request a demo today and get 10% off along with waived implementation fees at drata.com/partner/arresteddevops. Feeling like you have too many alerts, overwhelmed by vulnerabilities, and at the end of the day, not deploying apps as quickly as you would like? Sysdig hears you. Security in the cloud can be overwhelming and security posture is suffering. You need a way to prioritize what matters so that you can move faster. Shift left is the right operating principle, but you must shift left the right way. Sysdig roots everything it does in runtime insights. By knowing what is running in production, you can prevent, detect, and respond to threats and do it at cloud speed. To learn more about Sysdig, visit sysdig.com/arresteddevops. We’ve actually talked about this idea of platforms on the show before. I’ll put a link in the show notes. We had Kelsey Hightower and Andrew Clay Shafer on years ago with having a chat with Bridget about the idea of platforms before we had a name for it, a buzzword, if you will. And yes, we’ll call it that. But today, yeah, I’m really excited to be joined by Daniel Bryant. So I met Daniel at DevRelCon. In Prague, which was great. And he has a lot of great ideas around product-led growth and DevRel and things like that. But we aren’t talking about that necessarily, or we will, but we’re going to talk about this idea about PlatEng and all that. But Daniel, you want to introduce yourself to our listeners?

Daniel: [00:02:39] Yeah. Hey everyone. Hey Matty. Really appreciate the invite. It’s great to be chatting and such like great folks you mentioned to go before me, like Kelsey, Bridget, Andrew, like legends of the game, right? So you set me on a high pedestal perhaps, or high hopes. So yeah, my background is development. So I started my career as a Java engineer, did a whole lot of software architecture, then moved into some operations, built a few platforms on Mesos. I don’t know if folks remember Apache Mesos. And then I moved to Kubernetes as well, did a few platforms on Kubernetes. Now I run the DevRel team at Ambassador Labs, working very much in the Kubernetes space with our API gateway and telepresence fast feedback tools, and chatting to a whole lot of folks who are trying to build platforms. It is the, you know, it’s the thing du jour. And arguably we’ve been building platforms for many a year, but now there is this discipline emerging. Approaching, which I’m super looking forward to diving into today with you.

Matty: That’ll be great. And I just did check. So if you go to arresteddevops.com/platforms, you will find the episode that was recorded in January of 2016 with Kelsey Hightower and Andrew Clay Shafer. And distinctly what I remember most about this episode, which is unfortunately not any of the content, but being very, very jealous and feeling FOMO that I was not one of the hosts on this episode. Because after it was done and I looked at it, I was like, oh, What a conversation. Fantastic. So let’s kind of— so yeah, there’s— it’s interesting to me because I’ve thought about this idea for a while, you know, in a lot of the ways that I think about in my background with when I was working at Chef and other places and thinking about like, how do we have these teams that run the place where we put our stuff? And then it seems like in the last year or two, all of a sudden now we’ve given a name to this, which is platform engineering. And Maybe can, you know, definitions are fuzzy. You ask 12 people, you get 15 different answers. But for context of our conversation, if someone says platform engineering to you, Daniel, like, what does that actually mean?

Daniel: [00:04:34] Yeah, it’s a great question, Matty. And I’ve kind of formalized my definition, right? Because I’ve been asked so many times now, like, I’ve formalized it by saying it’s kind of the discipline of building, like, toolchains, workflows, platforms to support the team, in quotes, for Going from idea to observable business value in production. That is totally mindful. If you see my slides when I’m doing my talks, that is the definition because it’s that soup to nuts. We all come up with ideas, how do we get them actually in front of folks, and then clearly, how do we get that feedback? Because you’re a big fan of continuous delivery, and for me, continuous delivery does not end once you get that app in production, it just begins. You need to get that feedback either from a business point of view or from an operational point of view, get that feedback, keep iterating. So platform engineering for me encompasses that whole spectrum of your idea, stuff in production.

Matty: So it’s not just the infrastructure upon which you’re running your things. So one could say, you know, Kubernetes is not a platform in this way, right? It’s a part of that. So we’re looking at that whole delivery. To me, I guess it may be one of the ways to frame this too is something that James Governor from Redmonk has said before is everyone’s just trying to build their own Heroku. Oh, totally.

Daniel: [00:05:49] Yeah.

Matty: Is that— I mean, maybe it’s a bit of a glib way to do it. Not saying that James is glib, I’m being glib right now. But to say that sort of plat edge is like, that’s the DevEx that we want to have, right? You’re sort of saying we want this thing where we’re getting that out of the way, but it’s all of the pieces that go into making a Heroku that were abstracted away, or maybe not even necessarily abstracted away, but just sort of hidden away. And my question kind of going to that, I want to dig into a little bit more about what is involved with that, but You know, I think let’s, let’s maybe just address the elephant in the room, right? Like, I think we’ve— if you’re still on Elon Musk’s hobby website that used to be known as Twitter, you’ve probably seen there’s some sponsored tweets that are going to come across your transom about DevOps being dead. Platform engineer killed DevOps. I remember at KubeCon, yeah, yeah, yeah, from me, yeah, had their big DevOps is dead, and, and there was a lot of fun trolling to be had about that. But also Why, right? Like, I mean, is some of this idea about hiding that contrary to the principles of DevOps maybe? Is that the idea? Or, you know, where does that idea come from? And I’m not saying you subscribe to it either. I’m just— you’re observing this, like, why do we see these things being contraindicative of each other versus complementary?

Daniel: [00:07:15] Yeah, interesting, Matty. So, I come from, like, a Java background, right? So, for the— about 20 years I’ve been coding in Java, but for the past 10 or 15, like, Java is dead, long live Java.

Matty: I mean?

Daniel: So we’ve seen it so many times. And I think it’s the same deal with DevOps and platform engineering. Like, particularly like you and I now, we do embrace the marketing a little bit as well, whereas being engineers, and we kind of get it, like, no publicity is bad publicity to some degree.

Matty: Oh, I don’t think the publicity that that particular vendor got at KubeCon was what they were looking for, but you never know.

Daniel: I do hear you. And I think the thing is, like, It’s a shock thing, right? If you go like, X is dead, it does draw attention, undeniably, right? The t-shirts people are rocking at the conference, the chatter about the room, for better or worse, was about that. So, it does draw attention. I think I’ve seen this time and time again in my 20-year software engineering career. Folks, and I’m sure I was just the same back when I started, you kind of want to bury the previous generation of things, because you want to be seen to be innovating on the next, right? When you’ve gone a few years in the industry, you realize that history doesn’t repeat, but it rhymes. Heck of a lot. I think I often joke and say, in the engineering space, the history is very poetic because I come from CGI days back, Perl, PHP, and just today I was looking on the interwebs and there’s a WebAssembly CGI thing popping up and I’m like, whoa, 20 years come full circle. I think it’s just the same. Literally, we’ve been building platforms back when I started, it was racking and stacking, it was installing Windows NT or whatever on it, and then jumping a Java application server on top and doing that thing. Now we’ve come a long way, of course, and we’ve gone through Cloud Foundry, in Heroku, but I think we always have the same goals. We just hopefully get better at each step of the way, and we want to rename it when we do.

Matty: [00:09:04] I think maybe this goes back to one of the core problems, and we’ve talked about this a lot on the show, and some of us have to make our peace with it one way or another about what we actually mean by DevOps too. Because I think if you take the context of what a DevOps team does is build infrastructure for you, and using, maybe using some code to do it, which is what a lot of, a lot of times people think of a DevOps team as an automation team. If that’s the case, then that platform engineering replaces that. But what, what, and this I think is one of the things where to me, when I think about that, I was like, I feel like a lot of these principles are what this has been about the whole time.

Daniel: Yes.

Matty: Right. So is it replacing it or is it just, to your point, an evolution and just sort of a different framing of the same idea, which is what— how are we using these tools to, to create value in a collaborative way rather than siloing it? And then that’s the other, the other kind of a thing. And that’s what gets kind of fun about this is because sometimes you’ll read this and they say, this is what— how you keep your DevOps— you know, DevOps teams are siloed. And you’re like, well, the point was to not do that. So you were already doing it wrong. But that said, everyone doing it, quote, wrong anyway. So maybe we have to make our peace with that. Um, but when we— okay, so we kind of think about that. Well, I think we’ll keep coming back to this, like, idea of, um, of where that framing goes. I was thinking, you know, it’s a little bit like I, I come to bury DevOps, not to praise it, you know, kind of, I feel like, is the, the, the, the model of that. I, I, I will say that I think part of the reason that some of that marketing material in that particular place has also been not done well is yes, you do have that shock value piece of it, but a lot of the supporting information or supporting material that has been thrown at it has been factually erroneous.

Daniel: [00:10:53] Ah, yeah.

Matty: Right. There’s been a lot of like, kind of, we were, and, and I remember, I don’t wanna kind of get into this too, too much maybe, but keeping things slightly anonymous, some of these places, the conversation was had about, well, what do you, well, we’ve, we did this research and then the research is like, we talked to 3 companies.

Daniel: Right.

Matty: You’re like, okay, no, actually in DevOps, we take research very seriously. Have you heard of a thing called DORA?

Daniel: Yeah.

Matty: Right.

Daniel: Yeah. Yeah.

Matty: You can’t, you can’t come with this baloney and think you’re going to, you’re going to live in this world. That said, when we think about these platforms, as, as someone who, you know, is a conference organizer and reads a lot of CFPs, the idea of an IDP, the internal developer platform, is hot, hot, hot. And lots of chatter about the Backstage project that was created by Spotify and So, this again, at the end of the day, is an IDP still fundamentally your internal Heroku? Is that a way to sort of generalize, do you think? Or what makes up a good, or what qualifies something to say, this is what my internal developer platform, like, what are people looking for in something like that? Or what should they be looking for when they’re building something like that?

Daniel: [00:12:06] It’s interesting, Matty. I have seen some folks talk about IDPs as internal developer platforms, and some folks talk about it as internal developer portals. And I think it is an interesting distinction, right? Because Backstage has done a fantastic marketing job and like, love what they’re doing there. And like people are saying like, that’s the jumping off point for your platform. But in my mind, I actually did a presentation in Valencia at KubeCon last year where I talked about really that’s your UI, CLI, SDK, API on top of your platform. So I do double-check when I chat to folks now and they go, oh, I’m gonna build an IDP. I’m like, do you mean a platform like soup to nuts, or are you looking just to build a service catalog? And then I’m like, oh yeah, I was gonna say, do most people—

Matty: am I misusing the, the, the initialism as well when I say it’s internal developer platform? Is the convention more to say portal? Is that— again, obviously I think it’s great. Like you said, you always wanna say, what do you mean by this? Totally.

Daniel: Yep.

Matty: Because clearly that, that— but if that’s a clarification that most people maybe are thinking about as a portal. And I think that’s a powerful place. And I’ve thought about this idea, like you said, is it like a service catalog or sort of the next level of a service catalog? And the trick of like service catalog carries a lot of baggage because that comes kind of from the ITIL side where the idea is actually incredibly sound. And I’m a big fan of the idea of service catalogs. It’s just they sort of evolved out of a thing you can ask a person for, right? Which is actually why I like them, because I look at this as things I’m building within my DevRel team. Like, we need to have our equivalent of a service catalog that we provide to our account teams and our sales teams and our field teams to say, these are the things. And yes, we can— the difference is, again, because we’re not in this sort of ITIL framework, you know, it’s like, yes, you can also— we can do things that are not on this list, but this gives you an idea of what’s possible.

Daniel: [00:13:58] Yes.

Matty: And if— and if— so I think when you think about this model of a service catalog in this way, like in the ITIL way, it was very structured. It’s like, these are the 10 things you can ask IT for.

Daniel: Yeah.

Matty: And they are structured in this way. And that’s just sort of the way that it had to be for that type of IT service management. Whereas I, I like to think about these, these portals or these, these places as this should catch your 80%, right?

Daniel: Agreed.

Matty: 80% of the stuff is like, we do the same kind of thing over and over again. And if it’s clear and done in this way, we can, we can do this. You have a way you build it, we can automate it behind the scenes so it just all happens. And then the trick of it is if you need something that isn’t on this list, we can probably still do it. Yeah, but now it comes with cost, right? Time cost, effort cost. Yeah, yeah, you should always— that’s the— I think the trick of this is to sort of have, like you said, have that escape hatch, have that like, okay, because then also if we’re— but in order to be able to do those things, you have to have that 80% case handled in an easy way so that you’re not spending all your time doing the same stuff over and over. Again, this is the common toil reduction story. But now let me ask you this. You think about— anyway, so we’re talking about building these portals upon which are the entry point to your platform. At what point do we run into a challenge? This is where we’ve It’s a classic problem of, you know, the charity major statement, right? The best tool is the one that you don’t need. The second best one is a SaaS. And I look at organizations that really invest in building out so much of this and start to wonder at what point is that providing a certain amount of value. You know, when I was at Red Hat, I like to use this term that I would use there, but when I was at Red Hat, I was mostly selling to the government. I would say, hey, you’re not the US Department of Continuous Delivery. Why are you building your own platform of this, right? Like, what’s the thing that you do? So do you find, like, kind of when you think about that, I mean, there’s no SaaS really for this because it’s, I would imagine, just a pure— unless you’re, unless you’re at the level that you could do it. Like, you know, again, talk about come to bury. I mean, Heroku. Heroku was it. We had it. Thanks, Salesforce. You ruined it, right? But like, that’s— is that maybe the closest thing that we have to a true platform-eng-related SaaS? Like, if I wanted to say I, I want all the things of this, yeah, but I don’t want to have a 30-person engineering team that does nothing but build and operate this internal thing, like, is this— we can make these statements about don’t focus on things that are not your thing, but do you really have an alternative? I don’t know.

Daniel: [00:16:44] Yeah, and like, whenever I mention, or whenever we mention Heroku, I always like to chuck Cloud Foundry in the mix because I think they sort of blazed a trail as well. So I was doing a lot of Spring back in the day in Java, and like, they were really, you know, big. And then when I went Ruby, like Heroku was there. But I do think, like, it’s worth framing for the listeners as well, that they were different times. Because, like, we were building primarily web apps back then. Particularly, Ruby on Rails was the monolith, right? ActiveRecord, templated HTML, happy days. You know, I’m sure you and I have got fond memories, Matti, right? But, like, these days, we are building more complicated distributed systems, like it or not. Like, the scale, the requirements, you know, a good chunk of cases I look at, the distributed system is warranted. You can argue some things are, you know, we should just be building the monolith, right? And there, that’s when I think you mentioned about the 80/20 rule. That makes it a little bit more tricky because the 80% now is not just a monolith. Do you know what I mean? It is like Spotify again, and I talked about this in my KubeCon talk. Spotify talk about the golden paths and they’re like, on the platform, there is the right way or the best way to do things. But there is a golden path like Spotify had for machine learning apps. There was a golden path for microservices. There was a golden path for their frontend apps. So there was no like one true way anymore. Which I think we used to aspire to be like in Cloud Foundry and Heroku. I think that is a really key thing. That’s why I think we are genuinely struggling to get back to the halcyon days of Heroku. And I don’t think it’s possible these days. Every company I’m chatting to in Vasta Labs, our customers, they’ve got machine learning in the mix. They’ve got some frontend apps. They’ve got systems of record, systems of engagement. If it’s a biggish organization, looking for that one-size-fits-all, you’re only going to hit 50% of the apps at most, if I’m honest. Do you know what I mean? So that’s a really tricky thing to create a platform that supports all the different things.

Matty: [00:18:31] So I guess I would— so keeping that in mind, when you’re thinking about the challenge of doing this in an organization, that’s a pretty big ask of a group. Like you need a very— you’re creating your own separate software besides the software. You’re, for lack of a better word, fundamentally creating an internal SaaS, right? Yeah, that’s it. That you would do and that we are all struggling. And again, I can understand why creating a generalized one that a appeals to, you know, in the case of saying, hey, I’m gonna go build my new Heroku. And this is the SaaS that also— that’s multi-tenant across different companies, as opposed to a plat that’s multi-tenant inside your own. So you can at least do that. But that’s a, that’s a hefty, big, big thing. And you need— I feel like you need tons of cross-discipline and a total way to treat it like a product. Are people actually doing that?

Daniel: Yeah, for sure, Matt. It does depend on the organization. And we probably— it’s a good point to chuck in like a meme I’ve stolen from Matthew Skelton and Man Daniel Pace from Team Topologies, they always talk about building the thinnest viable platform. So before anyone gets too excited, I will chuck that one out there. So I like, I’m a big startup person, right? So like MVP and like the thinnest viable platform makes sense to me. But the companies like where, um, you know, I bumped into, had some good chats in Valencia again with Intuit, and they were at GitOpsCon. They were talking about their, what is effectively an IDP. And they, to your point, Matt, they have like a global team that supports this platform they’ve been building over the past, number of years. We’ve seen other companies at KubeCon, same deal, right? Once you get to a certain size, I think it’s totally warranted. Do you know what I mean? Back in the day when you had a team that was looking after your tin, maybe you don’t have that team anymore because your tin is now Amazon or Azure or GCP, right? Maybe you’re outsourcing that. But that team, it really is a value add in terms of, if we didn’t have this team, we would become super tightly coupled to a cloud vendor, or every team would do their own thing. It’s very And I’m sure that you’ve seen that in your time as well, right? Where there is like no one way to do the right thing. And then you just get so many different sort of versions of the platform out there. So I think it’s all about scale, right? And like any good architecture, any good operational question, the answer is often, it depends. And like it is, it does depend a lot on your scale in this perspective. And I think like, if you can get away with starting off just using something like a Cloud Run, or there is some like Knative, those kinds of things, where there’s some very basic kind of like that, for me, like, because it is a very basic platform, we do some GitOps or whatever using say, GitHub Actions to deploy. Like, for a startup of like 3 people, if you’ve not got product-market fit, please don’t build a platform. Please do that, right? But then if you’re an Intuit or some other company like that, rock on. Like, you can probably see like the ROI there in terms of the value that team, that platform team will give your whole organization.

Matty: [00:21:20] So that said, what you sort of mentioned, and it was interesting because like we go right to the tech often, right? And then my question about this is, is going back to the way that I’ve, you know, always think about like DevOps and always say, you know, the tech is easy, people are hard. We hear this a lot with DevOps when you think about the CALMS model. Everyone’s like, oh, you over-rotate on culture. And we’re like, no, they’re all equal, but I don’t have to convince engineers to play with automation tools. They’ll do that. So we emphasize the culture more because that’s the part that we have to push on because it requires more effort, but it’s just equally important. So I, and I think that that seems true with, with this idea of your platforms where, you know, immediately we’re going to sit there and we’re gonna be like, oh yeah, we could build this thing with, with, with Knative, we could do this, whatever. Okay, yeah, but you know what, the bits are the easy part. This is not solved by a tool. This is all the stuff. And this is when I was thinking about like what’s required from the makeup of that team. You need architects, you need people who understand software delivery, who understand how is this gonna— when I say how is this gonna integrate with our, with our source control, I don’t mean Oh, that’s fine, you know, because this has a connector to GitHub. No, what’s the flow? What’s the way? How do the people communicate? What I think about it this way, and I think this is, this is extrapolated, Jess Humble said years ago, and I love this, said source control is a communication tool for developers, right? And that’s what these are. These are communication tools. These are collaboration tools. So it’s not— the technology is frankly the least interesting part of it. It and the least challenging part, because that’s easy, right? That’s like, okay, we go install this thing, we configure this credential and maybe put some nonsense YAML in it somewhere, and then we’re done. But how do you, how do you connect what person over on this team does to the person over here? And how does this— I mean, just even thinking about, um, you know, coordinating, really, like testing, like how do these things overlap? This is all the hard problems.

Daniel: [00:23:15] Yeah.

Matty: And we want to solve that. We all want to solve them with it. With, with, with, with a tool, right? And, you know, you see this, you know, people say, you know, tools, you know, containers won’t break— fix your broken culture. But what’s interesting though, and this is where I think things get, get fun, Adam Jacob has said before— I, and I bring this up because it came up again on Twitter the other day— said, you know, tools influence culture and culture influences tools. It’s bidirectional. So oftentimes, and that’s sort of like a riff a little bit on— it’s kind of a variant on Conway’s Law, right? For people who maybe aren’t familiar, Conway’s Law says that your architecture is a reflection of the communication patterns in your organization, right? So, there are tools that, you know, you cannot adopt the culture of that tool and still— but you have to fight really hard in either way. But I sort of look at like, if it’s in the DNA of the tool to work this way, that will influence how you work. So, when we think about this, this tooling. And, and I wonder, not, not looking to say necessarily this is like a, you know, an eval of like which, which platform tool or which thing, but I think about things that I consider to be pretty heavy in, in, in the space. And heavy, I don’t mean like a heavy hitter, but just a lot to it. So like OpenShift or Tanzu, right? So, so I, I sold OpenShift for, for a year in a way. It’s so funny. This is not a video recording. But we have video with each other and my camera just like went totally out of focus and it’s driving me up a wall and doesn’t matter except that Daniel’s seeing me fuzzy and I am going to leave that in anyway. So the point is I sold OpenShift for, for a while and the problem that I see with these, these products, and it’s not anything technical, it’s just the reality of something at that scale. So first of all, like And again, I didn’t sell Tanzu, so, but I imagine it’s, it’s kind of the same story.

Daniel: [00:25:06] Yeah.

Matty: You know, Corey Quinn likes to say no developer can spend more than $50. No practitioner can go and buy OpenShift, right? Like it’s such a big thing that that happens at a very high level of procurement because it’s only even reasonable if you’re adopting it relatively broadly, right? It’s very hard to do land and expand with something like that. You know, you can, you know, some things like, for example, again, I always turn the reference— I like PagerDuty. A team of 8 people can start using PagerDuty inside a giant company and that can become 10,000 users, right? Yeah, you can’t really do that. It’s a lot harder with, with something like— because of its size. So what you run into is this challenge of that decision has to get made at this high, high executive thing. So you don’t necessarily have Like, how do you— this is talking about a sales problem, but still, yeah, you can run into— I’m sure, I guarantee, and this is not a dig on OpenShift or Tanzu, but I know in both of those and lots of other tools too, how many places are there where you’re the practice, you’re the platform, even the quote-unquote platform team, you’re whatever, and oh, well, I guess we’re doing OpenShift because the CIO has made this decision, because that’s the level at which the conversation has to happen. Happen.

Daniel: [00:26:21] Yeah.

Matty: And sometimes it goes— you do bottom-up, top-down, all these things. But I guess what I’m getting at is like, how do you— those big beefy platforms, how do you do your minimum viable platform? How do you— you can’t like get started small with something like that because it’s so all-encompassing. Do you see like this is the thing where you do have to sort of cobble together for a while and then you’re gonna eventually— because there is value in those things. Like, there’s a reason the batteries included type of solution of like a, you know, an OpenShift or a Tanzu is, is very real because again, you’re not the, you know, the Ministry of, you know, of developer experience, right? You’re, you know, that’s, you just want to ship some stuff, but it’s, it seems like a very large hurdle.

Daniel: Yeah.

Matty: I don’t know if I had a question in there, but I kind of was And it’s an interesting point, man.

Daniel: And I think that’s something like you sort of mentioned there, like I was definitely hearing you saying is like cognitive load, right? I would say things like Tanzu and OpenShift, I’m with you, love them. There’s definitely a time and a place, right? But like the cognitive load required to even understand what they offer, let alone how to use them is pretty massive. Whereas like if you spin up a Knative or whatever, like it’s a lot easier to get your head around, particularly as a small team. But I think like you and I, I think briefly touched on this when we were in Prague and I even mentioned it in my talk where I was talking about the golf course selling, as I jokingly used to call it, like back in my 20 years ago, I’d rock up and it’s like, we’re adopting this Enterprise Service Bus in like the job I was working in. And like, I’m like, I was a developer at the time, right? And my boss was suddenly like, you know, at the weekend I played golf with the, uh, whatever, TIBCO folks, right? Right. And I’m like, we’re buying this bus. And I’m like, oh, do we really need it? And that, to be honest, kind of sucked, right? I’m not picking on one technology there, but it was very much a top-down situation. But to your point, it kind of had to be because at the time I was working for the UK government and this was like the oil and gas division. Like, it was, you know, there’s a lot of folks involved in that, right? But I think these days I see requirements are driven more bottom-up, and then you need to get those sort of mid-level management folks that really bring these things together, but getting exec buy-in, C-level buy-in, is essential too. Do you know what I mean? So it’s that kind of like, there’s the top-down, sorry, the bottom-up, top-down, but then I think getting those middle levels, middle sort of layers on board is really key. And that’s what I see with the companies that are building their own things, same deal. They’re often like taking the requirements, starting small, getting a bit of buy-in at mid-management, and then some, you know, a couple of smart folks typically in the mid-management go, you know what, if we standardize our whole company on this, happy days. They get exec buy-in, and then it tumbles down. So it’s kind of like bottom-up, mid, top-down mashup of the things. Do you know what I mean?

Matty: [00:28:57] Yeah. And I think it’s because it is that, that, that kind of, you know, we always— the same thing you would say for consumer-facing software that you build, say, don’t, don’t worry about you know, building for scale till you have a scale problem.

Daniel: Yes, right.

Matty: You know, it’s like, hey, okay, you’re gonna do your minimum viable platform. You don’t have to worry about how is this gonna work for every single team at Intuit, but you kind of have to begin a little bit with the end in mind because it’s very easy to create yourself, to paint yourself into a corner, which— and I think that’s the, the challenge. And this again is why I think this type of work requires fairly mature software development approaches because that’s, you know, it’s because you have to see the forest for the trees, right? You have to say, or maybe it is more of a begin with the end in mind, which says, I don’t have to solve this problem.

Daniel: Yeah.

Matty: But I need to know, I need to make sure that I don’t create it as a bigger problem for myself later. I need to think about future me a little bit. I don’t have to fix it.

Daniel: [00:30:01] It.

Matty: We could do this minimum thing, but what I want to know is say, hey, ideally this will be used across every team.

Daniel: Yeah.

Matty: Okay, well, you know what, I’m gonna make a decision here that will actually make that impossible. That’s what you want to do. You know, I think that’s the thing is you don’t want to worry about making sure that it— figuring out how you’re going to do it. You want to make sure that you don’t set yourself up for something that’s impossible later because you’ve actually blocked, right? You sat there and said you know, okay, we’re gonna do this thing, and I can’t think of the perfect example, so I’m gonna just make one up, but it’s like, there is no way this would ever run on Windows.

Daniel: Yeah.

Matty: And I don’t give a shit about that right now, but, and, but I know that if I want this adopted across the whole company, we will have to solve that. Does it mean I have to solve it? No, but I have to make sure that I don’t put myself in a position that it is unsolvable. Like, there is this hard thing that will never work, you know.

Daniel: I think that, or like, the role of like the— I think what we’re sort of skirting around is the role of a platform product owner. Like I’ve seen, like I’ve chatted to Crystal Hershawn, she’s at Sneak now. Randy Schaub, who’s been sort of like in the early PaaS days. And they really do like start small, but think big. Do you know what I mean? And I did a fantastic podcast with Crystal on the Ambassador Labs podcast. And she was pitching exactly what you’re pitching, Matty, of like, I paint the big vision, get the buy-in bottom up and top down. And we all aim for that, right? And to your point, we give ourselves options along the way. Like we can at some point exercise the option, but we must never paint ourselves in a corner. So those kind of folks that are out there leading the charge, like, I’ve learned a ton from them.

Matty: [00:31:34] So I think that’s the— that I think you just, you know, again, like you said, what we’re skirting around is that there is— there are— I’m gonna— I’m not gonna say non-technical, there are non-engineering key roles in doing this, right? And, and I wonder if the outgrowth of like, where— okay, so in an organization, that is now starting to talk about wanting to do this, right? Like, and again, let’s, let’s, let’s put it— put aside your couple-person startup because there is no history in a company like that. Like, this is— this is my— just my question doesn’t apply there because this is about the, I have an organization that’s been around for a while. Yeah, we have a way we’ve been doing things because this is a— this is a change. It’s not a— it’s not a greenfield. It’s not a new thing. Number one, in a place like that I have my kind of assumption or guess or what I’ve also seen a little bit about where this comes from. Like, because it doesn’t come out of nowhere, right? You don’t, you don’t just go and hire up a whole— like, it’s usually this is a repurposing of a team or people are, you know, you’re bringing people around. Where does that usually come from? So like, the people on these PlatEng teams that are new, the, the new teams, where did they work at? Where did they work yesterday?

Daniel: [00:32:47] Yeah, so I mean, I think a lot of them were in the infrastructure space, or what we now call the developer experience space, like the tech enablement teams, dev enablement teams, those kind of things. There were folks that were, to your point, Matti, really focused on the problems, not the solutions, right? They’re really good at going, hey, like classic value stream analysis, your blocker’s here, let’s have a good poke around at that problem, understand what’s going on, and here’s how we fix it. Those folks, I think, are like the typical folks that are brought in. And it’s interesting to even look as in, Often there is a pain that triggers the creation of that team. Like an exec has discovered they’ve got 10 different versions of a platform out there. And then when a bunch of engineers left the company, say, then suddenly no one knows how to maintain version number 9 of the platform. And it’s like, oh my God, panic, which, understandable. So then they’re like, we need to standardize the way we do things across the org. And that’s often the trigger then for looking out for these people, creating the rec, saying, hey, we need this kind of platform product owner, this tech enablement owner, and then then this kind of person comes in and really starts looking at the actual bottlenecks and problems and specifying the vision and helping folks actually do the thing, right? The actual implementation too.

Matty: [00:33:59] Well, and that brings up kind of an interesting thing as well, maybe where we are in the adoption lifecycle, if we’re sort of looking at the crossing the chasm kind of thing. Because what you’ve just described to me, I still feel like is in that early adopters visionary part of like early on, because everything you described, I would go so far as to say is atypical of most enterprises. Most enterprises, you even said they have a DevEx team. How many of these places still have that? I mean, how many, you know, I was trying to think of some made-up companies because I don’t want, because I could be wrong, right? I don’t want to sit there and say like, does John Deere have a DevEx? And by the way, if they do, awesome.

Daniel: I reckon they do. They do really good work actually.

Matty: I was going to say John Deere is a terrible example. I was trying to think about like typical enterprises in my part of the country. Like, we tend to watch and see and then connect. But either way, people don’t—

Daniel: I think they’re maybe cool, like, to your point earlier about Toyota.

Matty: John Deere was a bad example, actually. John Deere is pretty damn innovative.

Daniel: [00:35:02] But I think the big organizations do. The SRE book from Google really revolutionized that, right? Folks are really thinking about SREs. A lot of, like, site reliability engineers are thinking about DevEx.

Matty: I will put it to you this way. This is the thing that helped me with my echo chamber. Was I remember I used to have these conversations when I was at PagerDuty with, with my boss, Alex, who was the founder. And I love— and Alex is great. And this is not a dig at Alex. It’s an interesting conversation we would have. I remember having this conversation about places that have a production support team, right? Like it’s basically a bug fix team. And he’s like, no, no, no, nobody does that anymore. That’s not happening anymore. And I’m like, no, Alex, enterprises. And he’d say, no, no, I talked to all these enterprises and they don’t do it. And I said, Alex, you talk to enterprises who A, are even aware of PagerDuty and B, have self-selected to want to talk about it. Yeah.

Daniel: Survivor bias, right?

Matty: Right. It is a survivor bias. And you know what cured me a little bit of our, my survivor bias was being a single dude going on dates and meeting people who also worked in IT. Yeah.

Daniel: Yeah.

Matty: And who were business analysts or who were software developers in pharmaceutical or in all these places we have around here. And all of these things were not how they did work.

Daniel: [00:36:12] Oh, interesting.

Matty: You know, so we have our own bubble because again, again, who are we seeing? Even the people that we see at conferences that we feel like, oh, you’re behind the curve. Yeah, again, there’s a survivor bias there. Those are the people who came to KubeCon. Yeah, you know, and whatever. So there is a lot, and even within organizations there, you will see those pockets, and they’re the beginnings of it, right? So there was a little bit, but I guess taking it all the way back to that crossing the chasm part of that is there is a forcing function that will eventually happen. I think what I was sort of getting at is in places that don’t have that, that are— want to maybe want to skip that part too a little bit. That also happens, right? Where like we see, okay, this evolved from this small thing that did this part, and it’s like, wait, now we actually— how many places jump over that though too, right? And say like, oh well, platform engineering is— I mean, this happened with DevOps, right? This happened with containers. This happened with all of this where there was an evolution path that happened a little bit, or there was this thing And then when you’re in that late majority, the late majority skips the pre-work.

Daniel: [00:37:15] Yeah, so to speak.

Matty: And it’s not necessarily a bad thing. I’m not saying like, oh, they missed the point, but the pain that got them there, that got the early market there, what is— you just sort of leapfrog over that to the implementation. But then that’s— but that’s what I think that leapfrog is what ends up with this idea of you take what was your ops team and you say, now you’re the platform team. Yeah, but that team is not necessarily made up of people And I think about the evolution of ops, you know, what was traditional tech ops, like you said, that was looking after your kit, that then became cloud ops teams in a lot of cases. Yeah, right, right, right. Which was fine. So you’re the one who manages all of the kit, but now it’s just VMs, they’re EC2 instances, or you’re the one who manages a relationship with Amazon. And then too often those teams just got called the DevOps team now because DevOps is a way to hire people. People, but you’re still doing that same thing. And then I believe those teams are now going to get rebranded, and then they were called SREs, but they weren’t SREs.

Daniel: Yeah, right.

Matty: We see this a lot, the Korean name only, right? You know, which is like— and again, you have to do this. Goes all the way back. You talk about the, the history of this podcast. I mean, I remember things I said, might have, might have even said this in our first episode, but it’s early on talking with Trevor, and I said, hey, it’s crazy. I said, nobody’s hiring sysadmins. This was 10 years ago. Right? So nobody’s hiring sysadmins in Chicago anymore. They’re only hiring people to do DevOps, but the job description is manage tape backups on a Windows 2003 server, right? Right. But the thing— now also, for the record, by the way, everybody who’s getting these better titles, good on you, man. Go get your bag.

Daniel: [00:38:50] Yeah, yeah.

Matty: Because yeah, this is my big thing about— I, I said there’s 2, 2 things that got me over the harping on the DevOps engineer title. One was Ian Coldwater pointed this out before, which said when you say that, the people who hear it are not the people who gave themselves that title. So all you’re doing is making them feel bad, right? So that’s, that’s not great. The other thing is, I— this, this data is suspect because this, this was a statement from a few years ago, but I think it’s, you know, so don’t hold me on it. But fundamentally, changing your title to DevOps engineer was a 20 to 30% pay rise. So I’m like, yeah, go get paid, man.

Daniel: That’s Cool.

Matty: Yep. Get your bag. Anyway, back to the thing about this evolution where you run into this. If you’re going to keep— if you sort of look at the same team that in some organizations has probably started life a decade ago as the TechOps team, that then became the CloudOps team, that then became the DevOps team, that then became the SRE team, and now is going to be the platform team, and their remit probably hasn’t changed, right? Like, how many have you— this is, again, thinking about dating ourselves, but I had a colleague at this financial company I worked at. And he had been there for many years before me. And I came in, we were a contract. I was working for CSC at the time, and CSC took over all the IT for this company, Heller Financial. And then we got acquired, all this stuff. And I remember my colleague Mark said, I’ve sat at the same desk for 10 years and I’ve worked for 4 different companies. Right? You know? And so that is real. And so I will tell you, this is, I’ve had, And the reason I know this— and there’s people that everybody wants to do this right, so they’re getting there. And I’m not saying like this is this, this fuddy-duddy thing where these people are never going to go, but I, I don’t think it’s too much of an outlier of that journey of that you’re going to have these teams are called the platform team that’s still at the end of the day these are infrastructure admins, which can have great ideas. And again, I’m a multi-decade sysadmin dude, right? Like I will never talk shit about sysadmins or infrastructure people, but we only know what we know and we don’t know what we don’t know. So, so if you were going to continue to take that sort of same group and continue to change what you’re expecting out of them without actually enabling that to happen, you’re going to end up with things that rhyme with everything that came before because you’re not going to have folks that are as architect-focused because you’re still talking about people that think about life in an operations way. I don’t know the answer to this. Listeners, if you’re in a situation like this, if what you’re seeing happening on that, hit us up on the Elon Musk hobby site, you know, and, and, and love to hear what you have to say. Can we talk about Backstage a little bit? Because I want to know more about it, and I feel like you know a lot more than I do. So I think it’s cool. I, I remember it sort of— I was hearing sort of like the, the— to me it seems like one of those things that I, I want to say was like a couple years ago sort of would hear about it. I, I will, I will admit I kind of put it off to the side the way that I do. I love Spotify, but it’s interesting that like there’s Spotify model that Spotify doesn’t even follow. So you always wonder about how much of, how much of this is the, is, is something that we take as inspiration and go, and how much is a model. And I kind of heard this, and then it feel like in the last year or so, all of a sudden it’s like backstage all the time, like this is a thing. And so I’d love to kind of get a feel for like what it’s about, from what you see, and then what you really see the, the use case for it. Like, what are people actually doing? Like, where are we in the, the hype cycle, if you will? Maybe not the hype cycle. That’s a Gartner thing. It’s the crossing the chasm, right? Like, where are we with, with, with something like that?

Daniel: [00:42:29] Yeah, so I mean, at its core, Backstage is really a jumping off point for the that IDP, building that internal developer portal. So it has got a service catalog, there’s a whole bunch of automation hooks you can hang off that. So I don’t know if folks remember like Rundeck back in the days, for me it’s like a modern Rundeck, if anyone, like again dating myself here, but it’s, but it’s tailored very much to developers. So you can very quickly understand the lay of the land, it’s got great search capabilities, you can say, you know, what services have I got in my estate, my portfolio, you can dive into those services, see who’s on call, who owns that service, who’s built, you know, that service in the past. You can also, there’s a thing called tech docs, sort of living documentation, which any of us who’ve developed software know the pain of keeping your documentation, A, up to date, and B, like easy to find in the right place. So a lot of the backstage work is sort of focused on these, on the tech docs and making it super, super connected to the, the actual services that it’s relevant to. So I think, you know, there’s definitely the, the notion of Backstage as a thing, like Spotify, for example, are, they’ve actually got a business model now where they’re selling extensions because it was highly extensible. You can build your own extensions to do pretty much anything, automate anything off the, off the like service catalog interface. But Spotify have got these approved, you know, subscription-based extensions. So I was like, oh, Spotify, interesting play there. Right. And then there’s companies like Rody who are like a SaaS offering of Backstage because Backstage is notoriously a bit hard to install and run. It’s like a TypeScript app and a couple of components that are a bit sort of tricky to stand up and keep, you know, keep up and running. So there is these kind of like, there is like straight up like backstage offerings out there. But then I think to your point, Matti, like a lot of it, definitely folks I’m chatting to, people either use it very much as a facade on top of what they’re building as a developer platform, or they use it for inspiration. Do you know what I mean? They look at what Spotify do, they look at what they, even like LinkedIn, other folks have got sort of similar things out there. And they say, I’m going to do a backstage-like thing within our org. So maybe I’m going to put on a web UI on top of some platform we’ve already got. Does that make sense? In terms of that, they put some kind of like, yeah, some user interface that easily connects the developers to what’s actually in the platform, all the services. And then very commonly, like tasks, you know, you want to automate is spinning up a new service, very common one. So you like template, and back in the day I used to use that Yeoman and a bunch of other ones that we kind of template your application, spin up a new microservice, and that microservice has all the observability, security, testing stuff baked in, like, so the best practices are super easy to follow because it’s all templated in the app. Other things that are very common, like in a backstage, like portal, being able to deploy the applications, being able to easily get access to observability. So you may have multiple kind of SaaS observability tools, but the backstage or your portal is your jumping-off point. You know, this service, you click on this link and you go straight to the Honeycomb dashboard, for example, or you go straight to the Datadog dashboard, that kind of thing. So I think it’s like a one-stop shop of once you’ve got your service up and running, if you want to understand, yeah, all the kind of operational aspects of it, you want to deploy it, you want to effectively manage the lifecycle of the service, right? That’s what these kind of portals and Backstage in particular is really good at doing.

Matty: [00:45:47] We’ll add some links in the show notes to a couple of the places we just kind of talked about, places to, again, not necessarily any endorsements or anything like that. We don’t have any discount codes for you or anything. This has been a really great chat. We’re kind of getting to the end. If you were to give like a couple pieces of advice to someone who’s saying, I really want to get into this, like we need this at my company, this is important, like what’s the— I’d say what’s the number one thing to do and what’s the number one thing to not do when you are thinking about investing in platform engineering?

Daniel: Yes, I’m gonna do a shameless plug yet again, Matty, for my KubeCon talk, right? So, because I like getting into that in show notes, by the way. Yeah, it was very much a standing on the shoulders of giants. So I’m not claiming I invented any of the things in that. It was very much a pulling together of things. I had a Twitter thread that blew up, my most popular Twitter thread ever, I think, about— and I talked about platform engineering, and then everyone chipped in, and I basically harvested all that great idea content and ideas and put it together in this KubeCon talk, which is still available on, on the CNCF YouTube channel. The 3 things that are key, Amati, are treating our platform as a product. And I said, you cannot have good developer experience without good user experience. That’s the key thing. And focus on the workflows and the tooling interoperability. Make the thing composable. They’re my things. Building a platform, like, we can definitely break down, like, should you build a platform? But if you are building a platform, those are the 3 things I recommend.

Matty: [00:47:10] And what’s the number one thing to not do?

Daniel: Oh, can I be controversial and say go out and buy a platform? We sort of hinted at that. And I’m not, again, not naming like Not naming any names here, right?

Matty: But I always like to say, you can’t buy DevOps, but I can sell it to you. There you go.

Daniel: 100%, right? I love that quote. Yeah. And that’s the one thing I’d say. And again, I like, I’m a big fan of the aforementioned paths as we’ve talked about, but I do see a bit of a knee-jerk reaction to your point, Matty, as we get towards the later adoption stage, folks are often worried about being left behind and they are like, if I just put more dollars or more pounds or more euros or whatever, if I throw more money at it, how can I catch up? And I think that that can be a dangerous place to be. And again, referencing or riffing off what we were talking earlier on, really, if you’re a bit of a late adopter, look at what is already out there and catch up on it. There’s so many blogs about how I built a platform. Pick a company. Expedia talked about it at KubeCon. Twitter talked about it. A bunch of companies have talked about it. So rather than try and buy a platform, my recommendation is spend a bit of time investing in understanding the problem space, potential solutions, and then go for it.

Matty: [00:48:18] That is amazing. As we just said, there’s a wealth of supporting links. We’ll have links to Daniel’s talk at KubeCon, a bunch of the other things we talked about, if you go to arresteddevops.com/platformengineering, that will have this episode’s show notes in it. If you go to arresteddevops.com/itunes and leave us a review in the iTunes Store, that will help other people find the podcast. Yes, I still call it the iTunes Store. I know it’s Apple Podcasts or whatever, but also it’s the Sears Tower, not the Willis Tower, and I will not change. It is Comiskey Park, not whatever it’s called Now, we are also on Spotify ourselves and iHeartRadio and Audible and all sorts of places where fine and not so fine podcasts are available. So check us out there. Daniel, thank you for joining me today. This was a really great conversation, very illuminating for me as an area I need to know more about. Hopefully, it was really great for our listeners. So thanks for being a part of it.

Daniel: I appreciate the invitation, Matthew. Always fun to talk about platforms with yourself. So that’s been great. Fun.

Matty: Absolutely. This is Arrested DevOps. And remember, there is always DevOps in the banana stand.