Platforms with Kelsey Hightower and Andrew Clay Shafer

Andrew: [00:00:00] My goal for 2016 is to be more like Kelsey Hightower.

Kelsey: Me too.

Andrew: Good luck.

Bridget: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m your host, Bridget Kromhout, @bridgetkromhout on Twitter. Arrested DevOps is brought to you by 10th Magnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. 10th Magnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/10thmagnitude. This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools, such as Chef, Docker, and AWS, so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. Today, we have 2 great guests. We’re talking with Kelsey Hightower, who’s got a lot to say in this platform space, whether it’s in 140-character tweetstorms or at every conference you wish you were at. Kelsey, want to tell our listeners more about yourself?

Kelsey: [00:01:27] I’m Kelsey and I look like an engineer. Just kidding. Yeah, so I have a background ranging from, you know, sysadmin through software developer with a lot of management thrown in between. And these days I kind of focus on helping people see the world the way I see it. You know, I think everyone has a perspective. And I kind of like some of the trends that are emerging around, you know, platforms, distributed computing, and this idea of just reusing these things that we’ve been building for the last couple of decades and standardizing on them.

Bridget: Great. Awesome. Thank you, Kelsey. And we’re also here with returning podcast guest Andrew Clay Shafer, with whom I have the pleasure and privilege of working these days. Andrew, what should our listeners know about you in this platform realm?

Andrew: I don’t know. Do I have to tell you everything? I don’t know, where should we start? I’d rather just get into it. If someone wants to know more about me, then Little Idea.

Bridget: [00:02:33] Little Idea on Twitter.

Andrew: Engage with my brand. That’s all I got today.

Bridget: Alright, awesome. Well, then I’ll tell you that Schaefer has spent a lot of time in this automate all of the things meme. So, Andrew, when did things—

Andrew: It seemed like a good idea at the time.

Bridget: Right. So when did things start shifting from this notion of just automating every single thing you do to all this talk of, you know, platforms?

Andrew: Well, I think personally that it wasn’t a moment when it all of a sudden just switched. It was seeing the patterns of success across many different projects and many different, you know, kind of scales of both project and infrastructure and organization, to realizing that in most cases what you’re trying to automate has as big of impact— the architectures that you’re working with have as big of impact on the outcome as the fact that you’re trying to automate it at all. And that there’s things that you can basically entrench as bad practices with your automation that are going to cause you more problems, which are worth revisiting and eliminating.

Kelsey: [00:03:47] Yeah, so I guess I’ll jump in as well on the platform side. Yeah, I think Andrew sums it up pretty well. It’s like usually when we find good patterns, those become foundational pieces, and, you know, I guess platform is just the right word to use there. So I guess over time when we do find, you know, there’s only so many ways that you can do certain things, especially at the lower levels, that it just doesn’t make sense for people to rediscover them over and over again. So at some point, whether it’s a standard library with a programming language or it’s a platform for infrastructure, you kind of get to the point where some things just move into the standard library. Not everything is a good fit there. You know, a lot of people want the platform to do everything for them, which I think is probably the biggest challenge. And just like a language’s standard library, putting everything in a standard library usually causes some friction where people think that’s the only thing available is what they see in their standard library. And sometimes you’re going to need those third-party packages, you know, depending on what you’re doing. Maybe you are really truly a unique snowflake and doing something truly new. And at that point, you’re going to have to probably develop your idea outside of the standard. And then maybe over time it finds its way into the standard for everyone else to use.

Andrew: [00:04:59] So I think there’s standards and there’s patterns, and those are slightly different ideas. And they’re slight— I mean, they’re related, obviously. But standards aren’t always good patterns, for one thing. And then I’d also add that, and this is a quote from someone else, but that I think what we’re actually seeing, what we’re participating in, is this general trend for complexity to move up the stack. And that as we as an industry, we as a body of practice, as practitioners, recognize these patterns, and then we start to abstract them in the way that we can leverage those underlying patterns without actually reimplementing them every time. That gives us the ability to move the problem up the stack and solve some other thing that creates value for our business, ostensibly.

Kelsey: Yeah, and I think all of these things are in perspective of the user too, right? You know, if you’re fresh out of school or you’re brand new to the industry, you don’t even know where to look for these things, right? You kind of, you see some things that are offered to you, you see services, you see these platforms, but you don’t really know what your problem is. Usually you may not have enough experience to make that decision. And I think that what delays the adoption of some of these things, whether they look like patterns or they’re standards, it’s really blurry, right, from the outside looking in, right? You don’t, you don’t have enough experience to say, hey, this is just a pattern, or this has moved to the standard phase.

Andrew: [00:06:23] Absolutely. I think I would add a little bit, and I don’t know how much we want to go on this tangent, but when you go into some organizations where they’ve essentially frozen their tech stack in amber like a decade ago, and inside of those organizations in some sense they have less experience with some of this new stuff that someone fresh from school might, especially if they’ve been paying attention to things like GitHub and, you know, the things that we actually mostly take for granted in our circle, probably. So it’s interesting to see. I don’t think that that limitation to evaluate the technology is totally unique based on the number of years of experience someone has, but there’s definitely this band of of experience and perspective that allows people to make better decisions. Because I think, and this kind of goes back to the original question about going from automating all the things to the way that I think more about the structures and patterns now, and that is that on face, when you don’t have some of these experiences, you haven’t got those scars, then 2 things look like they’re basically equal. They think You think they’re kind of equivalent solutions, when in reality one of them is far superior, or at least is going to cause you less problems. Not that any tech is perfect, right? Like, this is the joke about today’s best practices are tomorrow’s horrible legacy system. But the things that you see, your ability to evaluate those two and they appear to be equal is going to change as you get more experience.

Bridget: [00:08:05] Well, and I think to Kelsey’s point too, like, people who maybe have had that opportunity to step outside the organization that they’re in and even talk to people at other organizations, go to meetups, whatever, might have a different perspective both from the person who’s been doing the legacy thing forever and the new graduate, right?

Andrew: Absolutely. I think we all have different perspectives. That’s kind of the— that’s the beauty of it.

Kelsey: It’s very much situational, right? Like, I’m in the process of buying another house, and when you go house hunting, you know, depending if you want to live in the hills, in the city, in the suburbs, the same style of house usually is built slightly different. Number of bedrooms, the size, the foundation given to where you’re at. And I think that’s the missing part of most of the discussions around platform. You know, like, my given situation is that I get one user per hour. I can almost use anything for my platform, right? And it makes all other logical discussions irrelevant. But if you— or you may be on the other end of that extreme where I get a billion requests per second, and some things that just don’t matter to the rest of the world just don’t matter to you in your particular case. And I think those— we don’t have enough time to do those at conferences in our talks. And a lot of times as engineers, you don’t get enough time to evaluate the full situation before making a decision. It’s like, look, we need something by Friday. You’re like, all right, by Friday, this is all I can tell you, and we’re going to start here and see how it goes.

Andrew: [00:09:37] Yeah, I think the dimension of context is often lost in a lot of the discussions. Just basically to reinforce and echo your point that if you have a specific problem, then there might be a specific solution. That might be very different than what a general solution might be trying to accomplish, and that if you have built something that’s purpose— I mean, you’re seeing what I consider the platforms that are emerging are basically taking and distilling lots of lessons that people learned building their kind of one-off bespoke platforms, right. If you look at the things that evolved at a place like Netflix. At a place like Google, at a place like anywhere where they have— I don’t know if we want to go on this tangent, but just a quick tangent. I really think these are all parts of a— like, there’s a bunch of buzzwords we throw around, right? Microservices, platform as a service, DevOps, continuous delivery. I feel like these are all aspects of the same phenomenon, which is the patterns that emerge from these high-performing organizations that were delivering services that were highly available and also rapidly changing. If you don’t need to be highly available, if you don’t need to be at scale, and you don’t need to be rapidly changing, then you might not need all these other optimizations. But if you want all those qualities, then you’re going to converge on something that starts to look suspiciously like a platform as a service. Your opinions about some aspects of that might differ from organization to organization, but you’re going to have to have those qualities to get those results.

Kelsey: [00:11:20] I mean, I guess it boils down to when it works, it works, right? You know what I mean? Like, if you pick something and everyone else says it just is not the best thing to do, but it actually works for you, I mean, at some point—

Andrew: Well, Ken, the way that I think— I think this is slightly different because, like, not everyone has the same definition of works, right? Like, from the context, and we already kind of talked about a little bit of context. Organizations have different levels of what I’ll call thresholds of pain and what they’re willing to put up with. And so there’s lots of stuff that people think works, which I think is actually not working at all.

Kelsey: But it’s relative, right? You know what I mean? Like in that situation, you could go in and say, this doesn’t work, and everyone will look at you and say, we’re actually comfortable. In this. And the fact that you change something to bet that’s better in your terms can make them uncomfortable. And at the end of the day, as long as we have human beings making these, you know, decisions still, it sometimes comes down to that, right? I just feel different when I do it.

Andrew: [00:12:26] So the quote that I’ve adapted to kind of articulate how I feel about this scenario is a Tolstoy quote that says that that all happy families are alike and all unhappy families are unhappy for a different reason. They’re still families and they’ll probably still do lots of things that are familial, if you will. There’s tons of things that people do that are very different and totally dysfunctional. If you look at these patterns that have emerged in these highly functional organizations, then they all start to look very similar.

Bridget: You know, I think that this is, this is a good place to point out that these are a lot of tangents and this is so interesting that I don’t want to stop it, but I also want to bring things a little bit back towards the platform discussion just because, I mean, we have Andrew was, you know, standing at a workshop in DC, like live coding, pushing Spring Boot stuff to Cloud Foundry. I went to a workshop that Kelsey did at OSCON in Amsterdam where he’s doing all of the Kubernetes. And so I feel like let’s bring it back down to— let’s detangle and demystify some of all of this stuff, like this containers, microservices, schedulers, orchestration. Some of our listeners are trying to piece through, trying to make their way through all of the word salad and try to figure out what of all of that stuff takes me to the place Andrew is talking about where things work and things are better. I know there’s a lot there, so I’m just kind of thinking maybe it would be nice to get a little bit of perspective, like maybe Kelsey start, Andrew jump in, agree, disagree, but like what of all this stuff, like I feel like last year everyone wants to talk about containers. This year everyone wants to talk about how will they put all of those pieces together. What should people be paying attention to?

Kelsey: [00:14:29] I think most people have gotten— it depends on where you are on the spectrum. If you do not care about managing infrastructure, that’s not your job, you’re just going to find the platform that lets you do what you want. For some people, that’s Heroku. Look, I’m just going to write this app and deploy it there and be finished. Everything else is noise. Right, in that world. But it is very much down to your situation and your personal situation. Like, if you want to actually manage and tweak knobs on low-level infrastructure, let’s say you sat down independently of this salad, right, and you decided that you’re hungry. You need to decide what you want to eat. You know what I mean? Don’t go to restaurants and decide that they sell you something that makes you feel appetized. You need to kind of make a decision what you want to do roughly so. And I think what will happen is when you start to survey what’s available, You’re going to have to make compromises in order to avoid reinventing the wheel too much, right? And I think that is probably the reality of it. No matter what we say or what platforms we all favor, at the end of the day, you’re going to have to make those 2 decisions. Are you willing to compromise on your design and what you understand at this given point in time and adopt something that exists? Or are you willing to take one of these things and use a fraction of it and build on top? And reinvent the wheel to your liking.

Andrew: [00:15:45] That seems pretty reasonable. I mean, I don’t disagree with anything you just said. You basically have to look at what you have available, you have to look at your context, you have to match them up, and then figure out how you want to fill in the gaps.

Kelsey: Yeah, and I think the reason for all the debate is that there is a lot of emotional connection people make with tools, which is kind of silly in a lot of ways, right?

Andrew: That’s because everything in tech, even though it looks like it’s about logic and reason, is really about fashion and tribalism.

Kelsey: Yeah, we see that a lot, right? Like, if you learn to program, let’s say Bash, and you’re really good at it, so you feel that Bash is your comfort zone, you will attempt to attack all problems with your tool belt, right? If you’re Batman, you use your utility belt. That’s just what you do.

Bridget: I resemble this remark.

Kelsey: And I think a lot of people, that ends up clouding conversations. So you go into a new conversation, and you’re always thinking of an opportunity to use your existing skill set, and you’re going to try to make that shit work regardless, right? And if it doesn’t work, you kind of start feeling weird, right? You start to do very unnatural things to try to prove your point or why the whole project should be compromised to fit your skill set.

Bridget: [00:17:03] Wait, don’t they say that people can write C in any language?

Kelsey: You can write bad code in any language is probably more where we’re at.

Andrew: Yeah, and, I mean, in some ways this is more about the hammer and the nail, because you have this one hammer, and you’re gonna just start hitting things with it, and I think it’s a combination of that and then this level of abstraction. Like, you can literally do everything in C. Arguably, you could do everything in assembler, but those levels of abstraction that are available to you, the things that you’re going to need to rebuild to accomplish some of the things that people want to accomplish now, like, that might be the right tool for a lot of scenarios. It’s probably not the fastest path to victory for a lot of the things people want to do that, you know, like you said, I just want to have a Rails app. Heroku is probably a good choice.

Kelsey: Yeah, and then the other notion of this too is that none of these platforms actually do everything you think you want to do. And I think the part where you think you want to do is very critical because a lot of people make up situations that don’t need to exist, right? They’re forcing this, this thing on the platform. And I think since you have that bit of friction, that there’s still missing pieces from everything that’s out there and available, that it causes people to jump ship a little prematurely. Right? It’s like this winner-take-all. I got this platform, like Heroku for instance, but it doesn’t do this one thing, and I’m gonna jump back to rolling and reinventing the entire wheel for this one fraction of thing that I’m missing. So a lot of people have the zero-sum game in their mind.

Bridget: [00:18:38] And this is actually interesting because it brings up that, you know, what Kelsey was alluding to earlier about the whole opinionated and unopinionated, and then what Andrew was saying about hammers and nails actually reminded me of his tweet about treehouses. And I like that analogy, but it also, like, it doesn’t cover every— all the ground, obviously. But is some of it just like some people want to be building stuff and that’s what they want to be doing, and then other people want to be using stuff and have it exist? Or is it more nuanced than that?

Andrew: I think when you look at my Twitter feed and you see things like the infographic that you’re talking about. I mostly just want to troll Kelsey Hightower. That’s my main motivation there. I think that there’s a certain creative aspect to programming, and if you’re programming systems, you get that adrenaline rush just like if you’re programming apps. The people that have got that fixation with building systems, and I consider myself one of them, then You feel powerful. Going back, I’ve given some talks lately about just the evolution of my own thinking. The first time you figured out, okay, I can configure a server to do whatever I want with Puppet, you feel powerful. If you came from a world where you were used to typing doit5.sh on a command line, all of a sudden, now you can do one server however you want, and you can do 100 of them exactly the same way. And now you’ve got Amazon. I did a deployment of Rails, building Rails on Amazon Web Services, live coded at the Ruby Conference in Salt Lake City, Utah in like 2008. It’s like, that was powerful. And then you do that a few times, and then the analogy I’ve been using— and I didn’t get sued for Disney even though I gave this presentation at Disney— is like from Fantasia, where it’s, it’s, you got this spell book and you made, you made a broom and that broom started pouring water. And then pretty soon, like, the brooms are their own problem and there’s water everywhere and you’re just trying to figure out, like, how to manage that. And so then you, you start adding glue and, and, you know, I gotta orchestrate all these brooms and, you know, somehow connect them to my database or whatever. You just kind of go through this progression where you figure, okay, there has to be another layer to these abstractions. I think that’s the layer that we’re seeing emerge right now.

Bridget: [00:21:17] Are we sure it’s not just abstractions all the way down?

Andrew: Of course it is. That’s the point I was making initially, is the complexity is moving up the stack, and so we’re going to hide more and more of this complexity. All the words, the buzzword salad, orchestration, whatever, is just representing like a new layer to the abstractions.

Kelsey: Yeah, and I also think there’s a bit of people wanting to protect this world of— and it’s not quite lock-in, right? There’s always this fear about lock-in, and there’s plenty of discussions around various forms of lock-in. But I think people are really nervous about this idea of yielding this fully— a thing that works. Let’s say there was actually a service that worked 100%. You gave it your app. And it just worked 100%.

Bridget: Wait, this is software, right? I mean, this is running on computers.

Kelsey: Well, you know, like playing.

Andrew: And we’re talking about magics and spells here. It can work.

Kelsey: But just saying, like, even I think Heroku is a very close example, right? Heroku makes a lot of compromises to do a really decent job for most cases, right? Like, you shove your stuff there and you’re just shielded away from a lot. And I think that scares a lot of people because it’s like now you feel that there’s a control element that’s lost, right? If we do this long enough, we’ll never be able to do this on our own. I’ll never be able to buy a server. I’ll never be able to install software and configure it. Some people really want to keep that around, and I think there’s good reason for it in some cases, like Richard Stallman. It’s like, hey, if we all lose the ability to do this, you’re always going to be renting or paying outlandish prices because it will be no longer feasible. Element of people that are out there that are trying to prevent that from happening. Even though we want that convenience, we want that convenience on our own terms. And I think it’s probably slowing down the progress of platforms because we have to appease people that want to look under the hood.

Bridget: [00:23:08] Isn’t there an element of fear there too? Like, if— I mean, I gotta say, as a, you know, curmudgeonly sysadmin of yore, like, the idea of not being able to configure it and not being able to fix it when it breaks, which people will expect you to fix it even if, you know, like—

Andrew: I would take it a little bit farther and I’d say that it’s— some of the fear actually comes from the fact that they’ve attached their tribal identity to their tasks. Right? And so it’s not just that, oh, I’m going to lose the ability to do this thing. It’s that thing is me. The ability to do that thing is what defines me, my job, all these parts about how my own self-concept of the universe is.

Kelsey: Trust too, right? I think you said it earlier. Isn’t this software? We know that it’s going to break in some way, and we may not think it should break in the way that it’s breaking. I can do a much better job than Heroku. I can achieve 8 nines if you just let me do it. I can do 8 nines with a single data center. In my backyard, right?

Andrew: [00:24:16] That’s the Dunning-Kruger kicking in right there.

Bridget: Is this data center in your backyard? Because you’re in Portland, right? Isn’t there like snow right now?

Kelsey: Yeah, so the cooling will be cheaper this week. I think we’re still working through all of that right now. I think a lot of people are conflating this idea of freedom and my ability to own and run my own stuff where If you think about the platforms we build, they would be much easier if we didn’t have to support 80,000 install targets. In the case of Cloud Foundry, the fact that you have to have BOSH is so you can have people, look, I got these servers, and I’m in Amazon, and I’m in Azure. It’s like you have to support all of those things. But if you took that away, it will simplify even BOSH to this extent where we don’t waste as much engineering time there. We still have this element, even for the platforms we’re shipping. That do a lot of good things, there’s still unnecessary complexity to appease people that want to run it on their own or on their own terms.

Andrew: Yeah, and I would— I mean, it’s interesting you brought up the analogy of Bosch, but I think that this is something that replays itself every so often, right? And so when you look at the different layers of the stack, so when you went from everyone writing everything in assembler to people having C code, there was a conversation about how we’re losing control. This abstraction is so high level, I don’t even know what’s going on in these registers anymore. It’s this similar sort of idea where it’s like, oh, well, if we just had one architecture, we wouldn’t need all these complicated compilers to compile our C. Or whatever. It’s just kind of the natural evolution of these cycles of technology as things get adopted and innovated and created. I think it’s inevitable. You don’t fight the future. You just have to figure out where you want to live inside of it.

Kelsey: [00:26:13] Yeah, I think you’re exactly right. So we’re still on this timescale, right? There’s stuff from the ’70s probably still running in a lot of places.

Bridget: Definitely still running. I had meetings in recent weeks with customers who are still running things from the ’70s, and they’re not planning on stopping.

Kelsey: Right. So we’re supporting all these things in parallel. So I guess that’s also a thing that kind of slows adoption in some cases where it’s like, look, I have stuff on a 30, 40-year timeline that you need to either tell me what part of that timeline you can support with your offering, or you get ambitious where you try to take your tech that was built today and stretch it back to those use cases that were valid. 30 years ago, and I think that also adds complexity to the thing you’re building, right? And if you’re in the business to make money, a lot of times you’re forced to stretch backwards in time to also solve those problems, right? Because it’s not just greenfield, right? That’s not reality.

Andrew: Absolutely. I think the way my mental model of this works is that you see everything in terms of these sedimentary layers, and the layers don’t go away. And in fact, if you look at the finances of the businesses, the top line for the mainframe business grew every year for like the last decade, right, and those deployments aren’t going away anytime soon. But they’re there, on top of them you see the client-server model, you know, now we’re kind of in this other phase of the cloud model, which is actually accelerating based on the stuff we’re talking about, and that consumption will increase all the greenfield will eventually subsume. I think that each layer ends up being more and more nodes in the network, but that bottom layer isn’t going away, maybe ever.

Kelsey: [00:27:59] I think what we’re at now is that some of these things are allowing us to switch out what’s underneath a little easier. I think that’s the true benefit of containers, right? If you strip away the hype, it allows us to at least be able to swap out what’s behind us going forward, right? So whether it’s Docker—

Andrew: Well, let’s stop, and I think this is interesting to analyze because what containers actually represent, at least the way people are excited about them right now, to me is that Linux kind of won and became this ubiquitous standard. So this thing that got standardized there is that syscall interface to the kernel. And it gives you the ability to package these fully dependent deployable artifacts and then move them around. But that doesn’t necessarily backport Linux onto all those mainframe applications. You’re not also going to be able to package up that bottom layer of sediment.

Kelsey: Well, I mean, so I think it’s really more of that— you’re right, none of these things will go away and replace everything before it. But the new legacy that we create, I think, will be easier to deal with 40 years from now, right? 30 years from now, where these container things become become the new legacy. I think Docker is a good example. Docker has changed slight parts of that process of building a container, what it actually is, multiple times over its short lifecycle, but the user didn’t really have to do much to adopt the new standard. It’s like, just rebuild it, and you would just deploy to the new version, whereas now you’re dealing with so many low-level things that are just in your face that you have to make so many decisions about when and where.

Andrew: [00:29:38] Not to belittle that iteration, but I actually think that the majority of that benefit came from the fact that there’s the Linux kernel. That’s the substrate that people are on top of, and so as long as that is there and you can count on that, then you can do all sorts of things on top of it.

Bridget: Sure. I mean, you’re right, Andrew, but cgroups and namespaces have been around for a while, and what Docker did was make it a lot more accessible to the average developer, ops engineer, whatever, who maybe never played with zones or never touched jails?

Andrew: Yes and no. I mean, this kind of goes to the point Kelsey’s making. So what was made accessible is actually using those things. What was not made accessible is like the mental model of how those things actually work, right? Like maybe got, you know, some subset of the developer— the actually core developers on Dock are interested in that technology, and they’re exposed to it. But for the most part, going back to my original statement that the complexity moves up the stack, that is hidden from you from these abstractions. So what’s actually being iterated on is that abstraction.

Kelsey: [00:30:46] Right, but I think that’s the transition from pattern to standard, right? So whether we believe this will be the endgame, I don’t think it is, it’s just the beginning. But I think what happens is that some people say, look, these things work in a lot of use cases. Let’s put an API on it. And it’s a pseudo-standard at this point, right? You can just say, you know, you can say Docker in this case, and that brings a lot of meaning with it, right? We’re not saying it’s complete, Andrew.

Andrew: No, if you liked it, you should have put an API on it.

Kelsey: I was just thinking that.

Andrew: Someone needs to make a song about that and, like, do the little dance with the—

Kelsey: Yeah, well, that’s how we use these things, right? Is an API.

Bridget: And Kelsey has a really good point too when he says this is just the beginning. Do I dare say the U word? And I don’t mean Unix, but what about unikernels?

Kelsey: I don’t know, man. Like, when we say unikernels, at the end of the day, it’s like— so the thing is, when we say containers, we’re not even saying that because containers mean nothing without the API, right? Period. The reason why people are using containers in this way, what they really mean is Docker’s API. Full stop. Done deal. If you’re in the Joyent world, you’re talking about those APIs that allow you to actually do something useful. The problem with unikernels is you’re just swapping one backend containment or whatever technology for another. At the end of the day, people are still going to say, well, how the hell do I deploy it? If you say, well, we’re going to revert back to SSH and FTP, And it’s like, dude, no! Like, you can no longer introduce a new containment boundary without an API. So that’s where we start.

Andrew: [00:32:24] I agree. I don’t think that Unikernels are usable until they’re basically put into these fabrics just like the platforms are. And tooling also around creating the actual— so there’s a interview with Brian Cantrell that is actually quite good, and there’s a slight digression where he talks about the unikernels, and he basically makes this argument that unikernels are exokernels, which isn’t technically— I don’t think that that’s equivalent, and that everyone would have to become a kernel developer to use them. I don’t think that’s the case at all. Like, I don’t see unikernels ever being useful until you have a similar level of tooling that you have with Docker right now that gives you the ability to abstract most of that away from the average developer.

Bridget: APIs, like Kelsey just said. I think you’re agreeing with each other again. This is not much of a cage match, guys.

Andrew: [00:33:28] I didn’t know we were signing up for a cage match.

Kelsey: Me and Andrew agree on a lot of things, I think, a lot of times. I tend to delve delve into sometimes, like Kubernetes, for instance. There’s a romanticism with Kubernetes and this low-level abstraction. You should not be using Kubernetes directly for all of your needs. It makes zero sense. Andrew is living in this world where you’re higher up, there’s more definition, there’s more opinions. In that world of, look, stop effing around. If you’re doing little dances about Kubernetes right now, you’re totally probably wasting someone’s time, right? Because you’re going to try to get that thing and build a Cloud Foundry-like thing on top and call it your own. Andrew has a lot more experience in that higher level of the stack and the benefits of that thing. For me, I’m still living in this world a little bit low and saying, I’m not yet ready to shift focus to the endgame. The endgame to me is that I don’t care. Unikernels, VMs, containers. I have an app. It’s in this bundle. Run it and make it work.

Andrew: [00:34:39] I think it’s worth pointing out there’s some inherent stratification of all these personalities and what they’re going to care about. So, like, at that top-level application developer mindset, like, I just want Heroku. Like, I don’t care if it is Heroku or whatever. It could be Cloud Foundry. It could be Kubernetes underneath, like, whatever. I want to push a button, the easy button. I want the thing to work, and I want it to work over and over. If it has a problem, hopefully it restarts itself and the rest of that stuff. There’s another digression which might be fun to have around the 12-factor apps and then what I’m calling 12-factor ops, which is the other side of that contract. That kind of is how I’m framing this notion of opinions.

Bridget: But, and that’s, that’s actually related to something Kelsey just wrote a really good blog post about, this whole 12 fractured apps thing. It’s like, absolutely, we’re all realizing that, hey, there’s a lot of things we still have to worry about here.

Kelsey: Yeah, and I think a lot of times we’ve neglected things because we know that there’s humans involved and they will look at logs and error messages and repair the system, right? We still have a, we still have a lot of that assumption built in just in the people part of the organization, like I’m not going to spend an extra 2 hours thinking about how my application starts. You know what? Just do the basics, and then Bob over there or Julie will take a look at the logs and do the needful.

Andrew: [00:36:07] That kind of goes back, depending on where you live, on the threshold of pain that your organization is willing to suffer. The point I started making a moment ago is it’s like The people that are writing those apps at the top, they don’t want to care about what’s underneath it, but it doesn’t mean that there’s not going to be some kernel developers at the bottom and that there’s not going to be some platform developers in the middle. Each one of those layers definitely has responsibilities and passions about the stuff that they’re working on. They’re going to care about their layer.

Bridget: Yeah, that 12-factor ops thing that you’ve been talking about makes so much sense to me just because If in the 12-factor manifesto, the talk of— the Heroku propaganda that says, you should definitely log in this way, and it’s like, yes, asterisk, asterisk, asterisk. Like, yes, unless that’s going to totally eat up all the available memory and everything is going to be dealt with terribly by Docker or whatever. Someone is going to have to deal with whatever it is that the 12-factor app is doing.

Andrew: [00:37:18] Well, I think my point there is that if you go through each of the 12 factors in the twelfactor.net and read each of them, they kind of imply that there’s something else that exists that does a bunch of stuff to make this app work. Whatever’s going to fulfill that contract, whatever’s going to make the other side of that thing happen, better exist. If you’re going to have your application configured by things that are injected into the environment variables, There better be something that does that. That’s all I’m saying.

Bridget: Even if that something is Julie.

Andrew: Yeah, I mean, your platform— that’s the other point I’ve been trying to get people to— it kind of makes me laugh when people are like, we don’t need a platform anymore. We have Docker. It’s like, well, that’s adorable, but if you kind of logically unroll that back to whatever existed before Heroku came and people started talking about platform as a service, How did you deploy things before there was platform as a service? What is that thing that if you take away the quote-unquote as a service exists that allows you to deploy things, right? Like, let’s have a conversation about these capabilities. And so your platform might be Julie, because it’s not ops guys, it’s the ops girl, running do-it-5.sh over and over in a for loop or whatever, or it might be something like Heroku, or it might be something In the middle, you know, there’s all sorts of configuration management tools that I and other people have used to deploy software.

Bridget: [00:38:48] So if we’re, we’re, we’re coming up on time, we don’t have infinite time here, unfortunately. I always want these discussions to go longer than they do, but I’m gonna, I’m gonna ask both of you to be prescriptive for just a minute. And I know you cannot answer what the right answer for everyone’s use case is since you don’t know what their use case is, but for people who are trying to make decisions and have all these ideas swirling around in their minds about how they definitely need to get them some, you know, stuff from the shopping list. They need some Kubernetes and some Docker and maybe some platform, or maybe the Docker is the platform. Like, what— how do people choose? Like, what should people do?

Kelsey: Yeah, so I’ll start. Like, so I’ve, I’ve done this a lot of cases where, you know, sometimes I’ll tell them, hey, you should use Kubernetes, or you should just stick to what you’re doing. Um, and a lot of times I will just say, well, could we just draw out what you do, the fee, or how does your organization work? It’s like, well, there’s some people write some code, they check it in, we have this language, and, you know, this is what we want to happen. And usually you find a little bit about the scale that they want to do it in. And a lot of times you would just say, look, here’s what I know will help you kind of do these parts. I know of no tool that does everything, right? So here’s based on my experience, like, look, you need some of this, you need some of that, and here’s the part you’re gonna have to build. If you went this route? As of building that special part this way because we wanted something we believe is important to us. And I just think it’s really sitting down and saying, before you go and watch the YouTube videos and visit all the sites, just kind of go through and write, sketch it out, and you’re willing to do the missing piece, right? Are you willing to sit down and own that part that you’re going to have to write yourself? If the answer is yeah, then that’s probably the platform for you, but you gotta know what you’re getting into, right? People get into this halfway, hey, we’re gonna— like, what? When did you ever sit down and say that that’s part of the thing you wanted to do? So if you just take the tools away for a moment and have people whiteboard it out a little bit, key things fall out of that, and I think you lead people in the right direction based on what they—

Bridget: [00:40:54] That makes a lot of sense.

Andrew: The thing I would take away from what Kelsey just said, which I mean, I think we have this theme throughout the discussion is that you have some context and that you should work out what you’re actually trying to accomplish and then back into a solution rather than deciding the solution and then working forward. And that if you actually are— and that will force you to revisit a bunch of assumptions that I don’t think all people do a very good job of articulating, that will tend to lead you to better solutions. The other thing that I think is worth— I mean, I kind of made this point earlier, but it’s like, think about the architectures that you’re building and how automatable they are as much as whatever you’re trying to do. Like, there’s no great way to automate certain types of technology right now, certain types of architectures. There’s, you know, the example I’ve been giving people when I talk about this is if If your goal is to actually minimize the mean time to recover your application, your service, and you’ve got some sort of order-dependent stateful process to get it installed and running in the first place, and let’s say that takes X amount of steps and they have to be done in this order, and that gives you some number of time, X, Y minutes, whatever, then that’s the lower bound for how fast you can recover. It’s going to be actually longer than that because that’s how fast you could get that thing running if you know what state it is and you’re starting from a clean state. What you actually have to do when you recover those kinds of architectures is you have to figure out what state you’re in. You have to unwind everything back to the beginning and then build it back up in order to get it to run again. If you know you have to do that every time you install it, That’s now your lower bound. So if you can think of ways to eliminate those steps to get things where they’re more managed as stateful processes that can be arbitrarily moved around, that you’ve separated your state from your statelessness, and that you’re going to manage those things differently with whatever the kind of proper care and feeding is for each one, then it gives you a lot more power and flexibility with respect to the choices that you make with automation. You’re going to get a lot of those benefits regardless of whatever those choices are for the automation, just from changing the architecture.

Bridget: [00:43:22] That makes a lot of sense. Kelsey dropped off and was breaking up a little bit at the end there, but I think what I heard him saying is similar to what you were just saying, which is you have to figure out your map, your territory. You have to figure out what you have before you can make any kind of valid choices at all about where you’re going.

Andrew: You can’t build anything until you call Simon Wardley.

Bridget: That’s where I was going with this. Is that accurate, Kelsey? Is that the direction you were going, is figure out what you have?

Kelsey: Exactly. And also, at some point, people need to get over the hump over change. There’s this thing that kind of kills me a little bit. You’ll talk to a person— if your role is to write code and develop, I think there should be things that you’re willing to change to adopt better ways of doing things, right? Like the simple blog post with 12 fractured apps. Just recheck the database, right? Like, it may not be there to reduce some of the complexity Andrew was bringing up around this order thing to make the lower bounds very complex. Just what can you do? You may have to change some code, and I get it. Everyone can’t change code. You know, there’s people that don’t work there anymore, so now you’re just babysitting this thing until you go out of business. But if that’s not your situation, you really need to figure out some way of taking responsibility updating things to match or to reduce complexity in general. That needs to be a thing that’s on the table. It’s often off the table, and people have to build workarounds going forward, and I think that is just crazy at this point.

Andrew: [00:44:56] Here’s another thing I would add, especially in this context of orchestration and some of the stuff that’s going on. First of all, the orchestration and the code and the artifacts— there’s this whole pipeline of stuff that has to be you know, interface with human beings and ticket trackers and source code and all that is, is still closer to an art than a science. You know, certainly people are kind of converging on, on some sort of de facto standards, but I don’t think it’s a totally solved problem. But then if you, if you look at all this and, you know, Kelsey working at Google right now, if you think about all these projects that you see out there like the Mesos is out there, Kubernetes is out there, Cloud Foundry is out there. They all kind of have what I’ll call the Google diaspora. There’s these ideas that came from people that use Borg and some of these other internal Google systems. If you go read the Borg paper, there’s a bunch of interesting stuff about scheduling and orchestration and the rest of that. Then there’s this one sentence that I think is actually I think it’s maybe the most important sentence in the whole paper, but I don’t think most people pick it up as such. That is a sentence that says that everything that ran on Borg, for the most part, the preponderance of things that ran on Borg have an embedded web server that broadcasts metrics about the health of the application. If you just internalize that and you build applications that have some idea of their own health, and are telling you about their health, then you’ll start to get tons of benefits outside of whatever you choose to do with your automation.

Kelsey: [00:46:39] And that’s the main point, too, around you have to be willing to write some code, right? So before doing this, people did this with external Nagios pokes and probs. You know what I mean? Like, your app doesn’t know anything or report anything. People just build tooling around the app and probe it from the outside. It’s like, over time, we know that those probes need to happen for all apps. How come these are not standard things we do, just like listening on a port? This should just become a standard thing you do as a person that writes code for a living or a web server for a living.

Andrew: This kind of feeds into some of the stuff that we talk about from a Pivotal perspective, but I think it’s useful to frame it when you look at all the other frameworks, and that is what we’re calling the cloud-native layers. You have this cloud-native orchestration in the middle with the containers and restarting processes and the rest of that stuff. But then above that, we have discussions around the sort of cloud-native application frameworks. If you look at the way that Spring Boot in particular works, it comes with a lot of these embedded metrics built into the framework. You don’t need to use Spring because maybe you don’t like Java, but if you go look at those patterns, the other thing that comes with it is all the Netflix open source You know, whatever, like resilient, high, whatever, circuit breakers. There’s a long list of things that you get for free out of the box with Netflix open source right now. But taking those patterns, like, if you don’t have those patterns in your architecture at your disposal as sort of like components and Lego bricks that you can play with as a developer, then you don’t really understand what this kind of next wave of cloud-native architecture is actually enabling.

Kelsey: [00:48:21] You know what’s funny? When New Relic came out, maybe, what is it, 5 or 6 years ago, you would just import this one package, and then your app started just to broadcast all these metrics.

Andrew: Back when it was all Ruby, right?

Kelsey: Right, yeah, exactly. You just import this thing, and I think that’s what you’re talking about. We need, like, lib/howtododothis, right? You import lib/howtododothis, and then it’s just like, look, these are the things you just can’t compromise on in modern systems. We’ve learned too much in 30 years. No more excuses.

Andrew: Just out of the box, like, you should be able to get basic things about any of the nodes that are in the thing, you know, whatever kind of metrics around their own requests, blah blah blah. Like, all that stuff, put it on a dashboard, start to make decisions.

Bridget: Hopefully an auto-generating dashboard.

Andrew: Yeah, well, scientific visualization is like another thing. Like, that could be its own hour-long conversation. Like, what do you actually graph? That’s a fascinating conversation as well.

Bridget: We’ll have to have an episode about that at some point. I do know some people who are doing cool stuff in this space. I think both of you know Charity Majors, right? She and some other folks who, I think some of them were at Facebook with her, are doing a new startup right exactly in that space.

Andrew: [00:49:38] Oh, really?

Bridget: Yeah.

Andrew: What’s it called?

Bridget: Hound.

Andrew: Hound.

Kelsey: Yeah.

Bridget: It’s like hound.sh, I think. So it’s all about— I think the way she put it is she does not want to ever make another dashboard.

Andrew: I missed the memo. There’s some other stuff going on. I mean, obviously, like the Datadog stuff that’s a sponsor of the podcast. We’re using some of their stuff right now. That gives some of these abilities. There’s also like a Neil Signal Fuse or whatever.

Bridget: Yeah, Signal FX.

Andrew: Signal FX, that’s what we call it.

Bridget: We know all these people, so we’ll have to get them on and do an episode about that.

Andrew: That would be a fun conversation just to see like— I don’t know, I used to TA a scientific visualization class, and the types of things that you can encode and create, and also the types of optical illusions or the things you can obscure are kind of fascinating.

Bridget: So instead of lying with stats, you can lie with graphs, lie with your visualizations.

Andrew: [00:50:40] Lying with graphs with Andrew Clay Shafer.

Bridget: Well, in any event, so let’s get a tiny bit of Magic 8-Ball action here, a little bit of prophecy. What do each of you think is coming next? Shafer and then Kelsey, what’s coming next in this space?

Andrew: What’s coming next? I don’t know, I don’t know if I’m up for the challenge of making a real strong prophecy here.

Bridget: Trends, stuff you see happening.

Andrew: I mean, so I think like there’s different timelines and thresholds. I think that you’re going to see a massive adoption of cloud across a bunch of different verticals in the enterprise. I think you’re going to see A lot of them go to use these technologies. I think you’re going to see a lot of failures. I think you’re going to see a lot of struggle as their adoption is not always 100% enlightened and educated. That’s regardless of the tools that they pick. I think the way I frame that, and this is a conversation I’ve had more than once, particularly around the namesake of the podcast. Someone asks you to come help them because they want the DevOps, right? Like, we want to get the DevOps. Then you go there and you look at everything they’re doing and you try to, in earnest, give them some analysis of what you think would help them be better at accomplishing the goals that they have. After that conversation and you’ve articulated things that you thought were sensible based on your context and framing, they say, well, what we really wanted was some way to do DevOps without changing anything. Is there some way you can help us with that? And at that point, you’re just like, well, you should just use Puppet, whatever. Like, it doesn’t matter. Like, you’re not actually trying to solve this problem. Like, you have so many entrenched political issues that you’ve got to work through before you’re ready to be high-performing in this regard that like I can’t help you unless you want to just make me the CTO right now.

Bridget: [00:52:57] That’s— well, that is something that would probably make their lives very entertaining. I’m enjoying working with you at Pivotal, so—

Andrew: Well, there you are.

Bridget: So what do you think, Kelsey?

Kelsey: So I think people will be forced into this outsourcing of their compute infrastructure because of things outside of their control. So 5 years ago, you could say it was easy to say, make that decision based on how you felt. Like, hey, we’re doing this in our own data center. We’re just going to replicate that in the cloud. And that’s what most people have been doing with a bunch of virtual machines and one-to-one mappings, firewall rules, all the little things you did in your data center, you did in the cloud. I’ve talked to a number of people where they’re starting to get so much data because they want to process everything, or there’s new services that are coming out that require machine learning, that require storage of petabytes of data. This is the point where it’s now beyond your control. If I go to your team and say, look, we’re going to store a petabyte of data per day, and we have no money to buy a SAN array that can hold that much data. Oh, and I also want it to be globally available. At this point, you start to push the boundaries of what people can make up or assume or have, you know, this naive confidence that they can do it themselves. That’s the point where you get people to say, you know what, to be honest with you, there is no one to buy one of those from. Like, we’ve really reached the bounds of scale. We can’t get that. We can’t get that internet connection.

Andrew: [00:54:28] So, I mean, in some sense, I summarize this by saying that you’re going to see mass adoption of the cloud infrastructure. And, and I think that, yeah, I agree 100%. Like, basically you start to do some napkin scribbling of like what some of these IoT projects are going to generate from data perspective. Like, even a moderate, you know, not ambitious project, like the types of time series data that you’re going to generate, need to store and analyze, like just explodes, right? And like, you’re not going to build that.

Kelsey: Exactly. And I think that is what’s happening now. So I think in like the 1 or 2 year span, you’ll see more and more of these use cases being the reason why Not because they saw the light eventually, it’s because they were almost forced into it, which is different than what we saw 5 years ago, which is what I think will be different now. Then what that platform will be, I think there will be 6 or 7 of those things. There won’t be just one. There will be many, and then they will start to mature and actually work.

Andrew: I want to add one thing that goes back to something that I said earlier, and that is when I made this point that I feel like DevOps, microservices, continuous delivery platforms all really one phenomenon, and that the high-performing organizations kind of have very similar characteristics in this regard. The point I want to make to reinforce what Kelsey just said is that they didn’t just set out to do DevOps. That wasn’t their problem. There was actually Darwinian force. It was evolution that forced them. You couldn’t survive, like you couldn’t deliver those types of services without those types of solutions. So when you see the people that are, you know, quote-unquote healthy, quote-unquote with whatever their, you know, pain-inducing process is, it’s because they haven’t reached any sort of Darwinian force that will eliminate them if they don’t change their behavior.

Kelsey: [00:56:19] Yep, so that’s my call for going forward is that— and then I think at this point people will feel like these platforms, cloud offerings, are actually beneficial because now they actually see why they need to exist if it wasn’t clear before. And I think that will probably speed up a lot of things, right? You’ll see how many new players get into the space, new offerings. You know, if you think about these machine learning APIs, right, those offerings are targeted in a lot of cases at companies that are building self-driving cars. I don’t have 10 years to train my algorithms. There is no way you guys are going to do this from scratch. I’m not waiting 10 years for you guys to get done with this. So you’re going to start to see where people are pushing to this, and then it makes you feel happy, right? You use them and you’re immediately 10 years into the future, and now you actually feel as satisfied as getting an internet connection.

Bridget: Nice. So this is, this is a really exciting conversation. And if people want to have more of these conversations with you 2 folks, where can they find you in the near to distant future? Upcoming conferences, perhaps?

Kelsey: [00:57:21] Oh, so there’s tons of meetups that I like to show up at, but some of the big conferences this year will be KRAFT, OSCON, which is in Austin this year, which I’m a chair of, so I hope to see everyone there. And, you know, various conferences. I’m in Europe quite a bit, and I’ll be talking about not just containers, but what’s beyond that going forward.

Bridget: Nice. And how about you, Andrew?

Andrew: I think we’ll all be at KRAFT.

Bridget: So if you would like to see Andrew or myself or Kelsey or many, many fine other folks Craft Conf in Budapest, Hungary in April.

Andrew: Highly likely that we’re all at OSCON. What else? I’m gonna try not to travel till after March because my wife’s taking her medical boards, so she asked me to shut it down a little bit. So I’m gonna probably be doing a lot of local stuff and maybe writing a little bit more. The easiest way to get my attention if you wanna have conversations anytime is His little idea on Twitter. Other than that, like, if you send me an email, I might respond eventually, but I’m gonna go with if you send him an email and he responds, you will be surprised.

Bridget: [00:58:32] It happened to me today. It was amazing. He responded via email. I thought he was gonna respond on Slack or Twitter.

Kelsey: And I just opened up DMs from anyone, which has been awesome because I’m getting people now engaging engaging in longer-form conversation. And I didn’t know how much of a change that meant, but it is actually very powerful to give people a private space to have a public conversation, if that makes sense.

Bridget: You are incredibly brave and I salute you. I will absolutely never be doing that. I’m assuming I use the block button on Twitter more than either of you.

Andrew: I never block anyone. Yeah, I don’t block people.

Bridget: Yeah, sometimes people just want to keep talking and that’s okay, but I don’t want to keep listening.

Andrew: I didn’t say I listened to everyone. I just don’t block them.

Bridget: I just don’t want my mentions to be overtaken.

Andrew: Absolutely.

Bridget: Um, but yes, I will be at OSCON. I will actually be talking about managing distributed systems with BOSH at OSCON, which will be fun. I’ve got to ask, I know there’s going to be a Kubernetes workshop, but Kelsey, you’re the chair now. Do you have to pass off the baton? Does somebody else have to teach it now?

Kelsey: [00:59:48] Yes, I’m going to help make some really good materials, like advanced Kubernetes stuff where we go into design patterns when you do have a tool like that and looking beyond it. I think people are ready for the next phase. To be honest, I’ve come to the conclusion Most people do not want to run it themselves. It’s a fallacy. I think people really want to know what to do next, and that’s what we’re going to focus on.

Bridget: So is that one actually going to be taught by you?

Kelsey: It will have a lot of Kelsey sauce on it.

Bridget: All right. I suppose that I just— I’m just going to say right now that if you are teaching anything, which I doubt you’ll have time for, do not schedule me at the same time as that because I want to go to that.

Kelsey: Awesome.

Bridget: All right. Yeah, I think that that pretty much wraps it up. We have a newsletter at arresteddevops.com/bananastand. It’s the best way to know about upcoming podcast episodes and cool news with DevOps. We also have an iPhone app if you dig that kind of thing. You can download it for free at arresteddevops.com/iphone. Thanks to our sponsors. Be sure to visit them at arresteddevops.com. Thanks to Kelsey and Andrew for joining us. It was great having you guys.

Andrew: [01:01:02] Happy New Year.

Kelsey: Awesome.

Andrew: Thank you. My goal for 2016 is to be more like Kelsey Hightower.

Kelsey: Me too.

Andrew: Good luck.

Bridget: That’s pretty much everyone’s New Year’s resolution, I’m pretty sure. But, and loyal listeners, if you enjoy Arrested DevOps, we’d appreciate it if you’d visit arresteddevops.com/itunes and leave us a review in the iTunes Store. We would love to know what you thought of this episode. Please leave us comments at arresteddevops.com/platforms, which doesn’t exist yet, but it will at some point when I have any time whatsoever. Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter. We’re always happy to get your input, ideas, or feedback at shows@arresteddevops.com, which Matt will totally read, ’cause email. Please let us know any ideas you have for future episodes. I’m Bridget, @BridgetCrumhout. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.