Application Configuration with Tim Gross and Adam Jacob

Adam: [00:00:00] Every infrastructure program will grow until it becomes a full-blown, half-assed version of Kubernetes.

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m your co-host, Matt Stratton, @MattStratton on Twitter.

Bridget: And I’m your co-host, Bridget Kromhout, @bridgetkromhout on Twitter. Arrested DevOps is brought to you by Tenth Magnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. Trevor would usually read this part, but he’s on a plane for Tenth Magnitude right now. Tenth Magnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/tenthmagnitude.

Matty: This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 120 common infrastructure tools such as Chef, Docker, and AWS so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. You can get yourself a cool Datadog t-shirt by signing up for a free 14-day trial at arresteddevops.com/datadog.

Bridget: [00:01:20] So I’m super excited about this episode. I’m like bouncing in my chair here at the Fort Lauderdale Airport, because we have today a couple of great guests, and we’re going to talk about some stuff that I think is near and dear to all of our hearts. So, I want to introduce our first guest. Our first guest is Tim Gross. And Tim, I think I’ve embarrassed him on stage before by saying he taught me everything I know about containers. And I think I worked for Tim at Dramafee for a year, and it was pretty great. So Tim, tell us about yourself.

Matty: Sure.

Tim: Hi folks, my name is Tim Gross. I’m a Product Manager at Joyent. I’m also the lead developer of our tool Container Pilot, which we’re going to be talking about today.

Matty: Great, cool. Thanks for joining us, Tim. Then also joining us is Adam Jacob. Adam is an accomplished metal guitarist and sometime software developer. You want to tell us about yourself, Adam?

Adam: [00:02:21] Yeah, there’s no part of me that is an accomplished metal guitarist, but I really appreciate the attempt. I’m the CTO at Chef. I wrote Chef in the first place. It’s hard to say that one person wrote Chef. There’s thousands of humans who wrote Chef, and I’m far from the person who has the most code left, but I wrote it in the beginning. I did the same thing with Habitat. I met Tim in a hotel lunch thing a couple of weeks ago, was it a month ago?

Matty: Something like that.

Adam: I think Tim’s awesome, so I’m stoked to talk about Habitat and Container Pilot and and all the things.

Matty: Hi. So speaking of Habitat and Container Pilot, why don’t we talk a little bit about what they actually are? So, and I’d like to kind of hear maybe a little bit of a story about like why they came to be. So Tim, you want to tell us kind of the, a little bit of the story of Container Pilot to get started?

Tim: Sure. So my role at Joyent is to kind of look at the upstack stuff. So our technology is about running your data center, but we don’t have a lot of kind of the the kind of managed service tools that, say, our competitors like AWS have. So, my role there is to talk— is to kind of develop applications and demonstrations for how you would run containerized applications in production. And so, when I started working at Joyent, I’m working with a guy, Casey Bison, and as we started developing these, we realized that we were seeing kind of a pattern emerge in what you needed to do to make containers work in production without having human beings intervene in them. Container Pilot arose out of that work where we discovered what we really want is something that runs as a supervisor inside a container, and then just adds the behaviors that we want to allow containers to wire themselves up without having a human being intervene in them.

Matty: [00:04:19] Cool. Habitat, what’s the story? Why are we habitating the things?

Adam: Yeah, I mean, it’s a very similar story. I took a vacation after— it’s almost 10 years. I’m like 4 months away from being 10 years at Chef. And I took a month-long vacation and I was in Mexico and I was finally relaxed enough to think about new things. And I was thinking about whether or not the ways that we were approaching trying to get all of these companies to the next level. When you think about how do you— it’s one thing to build this. It’s hard to be a startup and decide that you’re going to build infrastructure for yourself. You spend a ton of time doing infrastructure and plumbing and network. It’s equivalently hard but has more moving parts to think about doing that in huge enterprises, right? In some ways it’s harder, but it’s probably really just different, right? I was thinking about whether the approaches we were taking from Chef, from Cloud Foundry, from Docker, from Google, was it going to work? Was it going to be great, or was it not going to be good, or how was it going to work? I had seen a lot of the same patterns that I think Tim saw, in terms of how we have historically thought about doing configuration management, about doing automation, When you think about how those patterns scale, I think the evidence was that they didn’t scale as well as we wanted them to once you got high enough up the stack. When you started talking about higher-level service abstractions, you started talking about how is this service going to behave, and how am I gonna think about it, and how do we reason about it, and how do the teams that work on them sort of across the organization, from developers who write the code early on to operations people who run it throughout its lifecycle, How does all of that map, and how does it work? Habitat was the result of a bunch of experimentation and a bunch of conversation of, can we build a system? Is it possible to build a system that makes that easier, and that makes them easier to reason about, makes them easier to run, easier to manage, easier to deploy? And that’s sort of where it came from.

Bridget: [00:06:42] That’s super cool. I wanna tease out a couple of things from there, because you were mentioning reason about, and you were also mentioning the amount of manual intervention. Intervention that ends up being necessary. And I know that Tim and I had a lot of experience with that, like, yeah, we had automation, but there were also places we had to poke at it. So Tim, is that where some of the same motivations come from for you?

Tim: Yeah, and I think that, and combined with this notion of immutable infrastructure where, you know, the whole— one of the big advantages of having Docker in particular, and I will say that Container Pilot is definitely focused towards the Docker ecosystem, So, one of the big advantages is this notion of an immutable artifact that you deploy to production. And so, that kind of says, well, we don’t want to have to tinker with this thing once it’s done. But in reality, of course, it has an environment that it’s sitting in, right? Some kind of network environment. It has the other services that it needs to be able to find. And so, it’s immutable, but it’s only immutable to a point. And so, the notion of saying, well, let’s do a lot of configuration management in kind of the traditional ways, which are going to be different per environment, rather than saying, like, let’s let it find out where it lives within the environment that it’s been placed. It’s kind of, I think, the difference in tack. Yeah. And so the idea there is to say, like, by allowing you to have this artifact that you can deploy and have it wire itself up, you have a lot less of the kind of throw it over the wall to the operations team, right? So all the behavior is inside the container, Now, that means that the application development team that owns that container and owns that service really owns the whole thing and not just, oh, well, I’ve written my code, and then how it wires up to everything else is kind of like, well, that’s an operational concern.

Adam: [00:08:27] Yeah, that’s exactly right. We talk about that as the automation traveling with the application, because you’re still going to do all the automation, right? It’s not like you needed that stuff. It needs to wire itself together, the internal service topology. The external relationships that it has, the way the software updates itself, all that stuff, you need all of those things. You can’t just say that they don’t happen because it’s immutable now. That’s false. But if it’s going to be immutable, because there’s value in that principle, then all that stuff and all of its behavior has to exist at the very beginning of the process instead of the very end. And that means that it has to travel with the application through its life. And yet you can’t rely on other external third parties.

Matty: That, that’s kind of an interesting thing, ’cause I’ve been myself trying to reason about Habitat quite a bit over the last couple of months as someone who needs to go talk to people about it. And one of these days I’ll be eloquent about it. That day is not today.

Adam: Uh, but it’s cool. We’re all trying to get eloquent about it. I think it’s like super new space.

Matty: [00:09:31] And, and it’s, it’s every day, it’s like a little bit more is sort of resonating to me, but what, what, what I hadn’t thought about before, that’s making sense to me, or helping with that, is we talk, when we’re just talking about infra code, or whether it’s app code or anything, right? But we talk about shifting left securely, right? About moving all this stuff to the left instead of it being like, okay, we do all this stuff that gets out there and then we’re catching it later, we’re doing whatnot. So it’s a similar idea here, right? Which is saying like, okay, the earlier, ’cause it’s just that common, well, I almost said common, but I talked to a lot of people, it ain’t that common, understanding that the sooner to introduction of a defect, right? The easier, cheaper it is to fix. So if we’re saying everything’s It’s an artifact. EAA, you heard it here first. That’s the thing, right? It’s an artifact like anything else. And I know that, like, probably to everybody else who’s on the show right now, this is all like, duh, that’s what this is about. But that’s what’s starting to help me understand it more is, again, it’s no different than shipping down a binary than shipping down anything. You don’t change it so much after it’s done, but it’s not like it’s different. But again, when you compile that binary, it’s different, right? Based upon the platform. Am I kind of way off base?

Adam: [00:10:40] I don’t think you’re way off base. I mean, I think, Tim, you should tell me if I put words in your mouth, but I think one of the things that I felt really strongly when I talked to Tim a while back was that complexity of, it makes so much sense that everything’s an artifact. It makes so much sense that you want your infrastructure to be immutable. It makes so much sense that you want to have that automation there. What I think is— why I think there’s a struggle and what I think was so fascinating to me about the conversation Tim and I had was I feel like we both came to really similar conclusions very quickly. And I think it’s because when you look at— when you flip your perspective over and you say that the problem you’re trying to solve for now is that application automation problem. It’s how are we gonna deal with all of the complexities of this application through its life with this constraint that says that it has to be about the application, not about the infrastructure, right? And I think that’s a very deep truth. And one of the ways I think you can tell that it’s deep is that I didn’t make it up, and Tim didn’t make it up, right? Independently, we didn’t talk to each other, we didn’t know each other, but, man, that thing is for real because if you if you have the perspective to be able to see it, and not everybody does, because Tim got to see all these customers at Joyent, he got to see all these patterns, he’s got this perspective that most people don’t have. I have similar perspective, right? I think that’s sort of at the nut of what it is and why it matters. It’s also why I think it takes people a minute to realize, what is it? Why do I need it? It’s not as easy a sell as being like, oh, look at how cool it is to just whip up some automation or whatever. It’s a little more complicated than that because hiding in there is a pretty deep fundamental thing.

Matty: [00:12:40] Tim, yeah?

Tim: Yeah. And I think the other reason why maybe people aren’t quite seeing it as quickly is that, particularly in the container space right now, is that there are a lot of companies who have some kind of product that they want to push that is saying, no, actually, let us do that at kind of a higher level of the stack and let us be this kind of overarching mega orchestrator. It’s kind of the opposite approach, right? Brian Cantrell, our CTO at Joyent, has been likening it to the framework versus library approach, right? We’re saying, you know, the control— these kind of mega orchestrators are taking the control at this kind of framework level, whereas we’re saying, you know what, invert the control and keep it down at the application level and have the behaviors emergent from that.

Adam: Absolutely. And I think that idea of where’s that control live, hiding in there, I think, is this really interesting question of what is the shape of the platform we’re going to build the future on? And I’m sure there’s not one shape, so I’m sure there’ll be multiple platforms in the future, blah, blah, blah, blah, blah. But that question of where does that control live, One of the things that’s interesting to me is when it’s only now, like, in, you know, what we’re in July of 2016, like, it’s July of 2016 that we’re finally at a place where we can think about what it’s like to really truly have to deal with the fact that every single organization on the planet needs to go through this transition, that it’s not going to be some isolated piece. And like, that need to make it generic is going to drive this much deeper conversation about what the shape of those platforms are, because not everyone’s going to be able to invest to the degree of making it bespoke like that.

Matty: [00:14:27] Sure.

Bridget: Well, and this is, this is a conversation that I’m having with customers a lot, because Adam made a point just a minute ago about how infrastructure is not where a lot of people should be focusing. They should be focusing on their applications. And I mean, it definitely could be argued that Cloud Foundry is one of the very opinionated platforms that you’re referring to. But for people who are considering their platform choices, whether it be Cloud Foundry or something else, there’s a lot of people in IT who are still very attached to their infrastructure. They’re very attached to the way they deploy their VMs. It’s like, okay, they got on the virtualization train, but that’s as far as some people want to go. I wonder if Container Pilot and Habitat are going to be really difficult for people like that to wrap their brains around.

Adam: Maybe. So, I mean, one of the things that’s true of Habitat is I question whether— I think that the principles that you find in things like containers, so, like, the principle of immutability, the principle of convergent behavior, like, those idempotence, right, that if I give you the same inputs, I get the same outputs, like, those principles are highly valuable. And they’re valuable if you are in the enterprise, doing VMs. They’re valuable if you are building containerized infrastructure of the future stuff. I think there are benefits to every technology choice we make. I mean, the thing about those people in those enterprises who are happy with VMs, they run multi-billion-dollar enterprises. So, they don’t suck, right? They’re doing better than most of us. So, by revenue, by size, by complexity, that’s hard.

Bridget: [00:16:15] Sure, but they’re also not talking to vendors about their platform choices for no reason.

Adam: Yeah, of course, they want to be better. There’s not no reason to move. But what I think is real is that the truth is we don’t know what the right platform is for most of them. We definitely don’t know if there’s a generic abstraction that everybody will wind up basically settling in on. The analogs we have for that transition, SAP in the early ’80s, where it was like, how do we put computers to run the back office? That’s the closest we probably have when we think of— and I don’t know, I’m not arguing we should all wind up with SAP, but you see what I mean? We don’t actually know. But what I think is true is that, is it too much for them? I don’t think it’s too much for them. I think it’s a question of where’s the right modulation between what they have and what they’re going to have, and how do we provide the flexibility to allow that experimentation to happen in a way that doesn’t start from a position that says that their end state is that they’ll look like Google, or that they’ll look like Cloud Foundry, or they’ll look like Chef, or they’ll look like Joyent, because they do have problems that I don’t have. Saying that their solution at the platform level will be identical to the solutions from the big web, like, A, it’s just ridiculous hubris. And then B, it’s probably wrong. I think one of the things that I like about Container Pilot, I like about Habitat too, is that it doesn’t start from that position. It simply says, hey, look, if you’re going to run these applications, they need to be smart, and how can we make them smarter?

Tim: [00:17:52] Yeah. I think a lot of the discussions that I’ve been having with folks that have heard when I’ve been going to talk about it or something, and they come up to me afterwards, they’re like, yes, this is what we want. Resonating with them is that notion that, like, they’re recognizing that even if they are using one of these mega orchestrators, that there are all these things that are still specific to the way that we need to build applications. Even the so-called stateless applications still have connection state, right? So, there’s all this hidden state in there.

Adam: Nothing’s stateless.

Tim: Nothing’s really stateless.

Bridget: Can we just agree stateless is kind of a lie? Sure.

Adam: I mean, it’s a useful lie. Yeah, it’s a useful architectural line. There’s— then you’ve got to save somewhere.

Tim: Yeah, yeah.

Bridget: And, and so, but anyway, sorry, please.

Matty: Yeah.

Tim: And so when people are, you know, when people are running into that, they’re saying, you know, we, we’re recognizing that we have to build this intelligence anyway, even if we are using this kind of mega orchestrator. So instead of saying, well, let’s split that, you know, behavior, they’re saying, well, what if we just take all that behavior on ourselves? And, and, and it’s kind of like follows really tightly along with this, you know, that, that DevOps model where you’re saying, well, you know what, the, the teams that are building these think should own them and have a more multidisciplinary approach to building operable production applications?

Adam: [00:19:15] Yeah, I think for me, I look at our history of automation and for the platforms we’ve built and how we have built them. And the analogy that I’ve been trying not to use, but I’m totally gonna use it because there’s no one listening, right? It’s only friends, so no one will ever leak. We’ve done a really good job of building enough automation that roughly gets us into outer space. We are in low Earth orbit in our ability to automate and build computer systems and make them be big and be distributed and to run these complicated systems. We can do that. That happened. And some of us are better at it than others. But essentially, the curve of that technology horizon is that it will be ubiquitous. We will all get to that level of goodness eventually. So forget about how every insurance company, every every furniture rental store, they’re all going to get to a place where they can essentially automate themselves into the moral equivalent of low Earth orbit. And then the next question is, if you wanted to go to Alpha Centauri, if you played Civilization, will you do that the same way you got to low Earth orbit? And the answer is no, because it would be an enormous rocket and it would use all the resources of Earth in order to try to get you there, and it would probably fail. And so you’re going to have to think differently about how you build those systems to do it. And saying that you already know what all the technological advances would be in order to make that work, like, it’s just not true. But it doesn’t mean we’re not going to do it. We’re totally going to do it. It’s just a question of how.

Matty: [00:20:41] So I want to take a second too and think about— because in both cases we’re talking about some pretty new stuff, both in just ideas but also just seeing it. And not that we can see it, by the way. Did I tell you guys we’re doing a demo? Did we warn you about that?

Adam: We can do a demo.

Tim: Yeah.

Matty: But maybe we can talk a little bit about the guts and kind of implementation. And part of the reason is I remember something that I thought was interesting when the day of the Habitat launch is, you know, kind of reading what people are saying. Everyone was looking to be like, how are— what pieces are you tying together? And like missing that, like, wait, no, there’s actually some like original code in here or whatever, you know. So I think it’s, we try to put things into the box that we understand, and that’s why I think it might be kind of important to take a couple minutes, kind of explain how Container Pilot and how Habitat work. I think that would be cool. Tim, you want to walk us through Container Pilot first?

Tim: Sure, sure. So Container Pilot has a somewhat smaller scope than Habitat, and it kind of— I’m gonna say it compares to the Habitat supervisor only, and not kind of some of the packaging stuff. It’s kind of agnostic to what you’re using for packaging, although in truth, all the demonstration work we’ve done with it has been with Docker, because that’s kind of been our focus lately. So, the idea is it runs— it’s a small Go application, a statically linked binary. You drop it in your container. It acts as the supervisor, the PID 1 of the application. It launches the application, and then it kind of spins up a bunch of loops, and those loops fire off user-defined behaviors on a periodic basis. And so, what those will be is, say, the heartbeats, health checks and heartbeats. It’ll watch for changes to a service catalog. It can do telemetry and a couple other lifecycle things, like what happens when it receives a SIGTERM and the application is going to stop. Do we do some kind of backup right before we actually shut the container down? Things like that. But the idea is that all the coordination is with some kind of service catalog. Most of our example applications are using Consul. It supports etcd as well. In theory, you can support any arbitrary backend because it’s pretty pluggable. We don’t have those actually written. There’s an open ticket for ZooKeeper support because some people ask for it, but nobody ever seems to wanna implement it.

Bridget: [00:23:07] Oh, that’s right. Wasn’t it John Vincent who was asking for that?

Tim: Yeah, yeah.

Adam: I’m shocked that no one wants to do it.

Bridget: You’re gonna be on a panel with him at Velocity New York with him and Charity, so you can talk ZooKeeper and Container Pilot there.

Adam: Yeah, strong argument to writing the ZooKeeper plugin.

Tim: Yeah, that’ll be awesome. Yeah, so that’s really all it does. The idea is that we say, okay, that’s the application that we’ve provided as a tool to help out with— really the idea being to help out existing applications because a lot of this logic, if you’re running a greenfield, Node.js application or Java application, you might have all this stuff as library code within your application. But, of course, it’s not just the things that you’re writing. You have, in a production environment, you have your log servers, you have your database, you have your NGINX, your HAProxy, all the other kind of stuff that’s around there. And so, the idea is if you have this to kind of be the common thread between them, you can get them to coordinate using kind of a global state. And so that brings—

Bridget: [00:24:11] sorry, I was going to say, you bring peace to the PID 1 wars.

Tim: Yeah, yeah, yeah. Well, because then, yeah, no systemd in the container. And so the pattern of kind of like an application built that way, we’re calling the Autopilot pattern. And so we have a GitHub organization, github.com/autopilot-pattern, that has a bunch of different applications where we’ve actually done this, and we’ve tried to keep it like It’s not so much comprehensive as it is, like, very different types. And we’ve really started with, like, the stateful application, with, like, databases first, because we think that’s actually a much more interesting discussion than stateless.

Adam: It also flushes out all of the hard edge cases of a supervisor.

Tim: Yeah.

Adam: Like, oh, I ran a process, and I changed its config when some shit happened. Like, not the interesting part.

Bridget: Nice.

Adam: Okay.

Bridget: So, Adam, if you wanna piggyback off that and kind of tell us about where Habitat came in.

Matty: Sure.

Adam: Habitat, like Tim said, is super similar at the supervisor level. We start a little earlier. When we started looking at the problem of how do we build the applications that are easy to reason about throughout their lifecycle, What we learned was we could solve a bunch of later-end deployment problems earlier in the cycle. So, we sort of had 2 requirements. One was we wanted to be able to work for software that wasn’t in a container. So, all the behavior that we can provide to something that’s in a container, we want to be able to do for software that runs on VMs or on regular servers. In particular, gross enterprise software was a target. I made a list of the worst software in the world to manage, and then I went and found some of that software, and then I tried to put it in Habitat. What we learned by doing that was that one of the things that was in your way is the way that our software distros are built. In particular, when you think about doing a security update, someday you’re going to want to do an audit of your applications. You want to know what libraries went into them and whether they’re running vulnerable versions. You also want to know whether or not that version is what’s in memory or not. It’s one thing to say I updated OpenSSL everywhere, That’s cool. It’s another thing to say the running version of OpenSSL in memory everywhere is correct. And that question is actually quite hard to answer with existing tooling. It’s really hard to know, like, well, did you really restart all the services?

Tim: [00:26:42] Were you sure?

Adam: And so Habitat includes a build system that goes all the way down to libc. So basically, if you can compile your software, what you wind up with is a Linux binary that runs on any modern 64-bit Linux, and all of its dependencies are explicit. We can tell you things like what your build dependencies were, that use a particular version of GCC, or of Go, or of Rust. Then we also have a supervisor that knows how to take those packages and run the services that you define inside them. In a very similar way to what Container Pilot does, you provide hooks to the user for different lifecycle events, For starting and stopping and reconfiguring and those sorts of things. And it runs as PID 1. It’s written in Rust. And that supervisor also is responsible for all of the coordination. So, where Container Pilot uses etcd or Consul or ZooKeeper, what Habitat does is the supervisors themselves use a gossip protocol to share their state with each other and to make decisions about what should happen in the topology. So, when you deploy them, there’s no infrastructure to set up. So, you don’t have to go out and build ZooKeeper or Consul or etcd. Instead, the supervisors will communicate directly to each other, both for their internal topology for a given service, and then also for doing the service discovery and connection sort of around it. And that was a— the early prototypes worked exactly the way that ContainerPilot works. So, we had working prototypes that worked that way, It was good. The only downside to it was that you had to know how to deploy those things, A, and then B, their behavior is subtly different. So, the ZooKeeper version versus the Consul version versus etcd, they behaved ever so slightly differently, usually around things like timings and stuff like that. So, Habitat builds a gossip protocol in. Then the last piece of Habitat is a build service that’ll launch here in a little while, where you can take what we call a plan, so the description of how to build that software and package it, and build all of that stuff. It will build it for you as a service, so you’ll point at it like a GitHub repo. It’ll take your plan, build you the package, and then support it.

Tim: [00:29:14] Yeah, what struck me as kind of really interesting about that model of doing the having the gossip protocol there is one of the challenges that we ran into deploying Container Pilot applications on Triton specifically is that a lot of those discovery services have assumptions about the underlying topology that turned out not to be true on Triton. And so there was like, oh, well, we should maybe kind of— and so we’ve come up with some workarounds for that that seem to work well. But one of the approaches that we were thinking about before we before I talked to Adam was, well, what if Container Pilot actually just implemented the console gossip protocol itself? Then I talked to Adam, and so I happened to tell him, I was like, oh my God, they’re already doing that.

Adam: That was one of the funnest moments of when we first met, because we literally had gone down the same road, and I think we had the same conversations about what you could or couldn’t do. I was very nervous about putting the gossip stuff in because The 2 things people tell you all the time— I’m a systems administrator in reality and I play everything else on TV. One thing that you tell systems administrators all the time, A, you tell us that we’re not software developers, and then only the best software developers in the world should write cryptography and distributed systems. That’s what they tell you. And they are right. And also they’re wrong a little. So you just have to be very thorough.

Matty: [00:30:39] Right.

Adam: It wasn’t as scary as I thought it would be. And also it was terrifying.

Bridget: Well, and is it a little bit less terrifying with it being open source? Because at least you know that if it’s terrible, other people might check you on it. Maybe.

Matty: Maybe.

Adam: I mean, maybe it’s less terrifying for the user. It’s not for me. Like, if it was closed source, I’d just be like, look at my magical unicorn gossip protocol.

Matty: It’s super safe and secure.

Adam: Suck it, customer. I mean, I wouldn’t.

Bridget: But nobody would really believe you. Would they?

Matty: I know this is hard to know.

Bridget: I don’t think so.

Adam: But someone would, right? Someone would believe me. I’m not saying we should do that. We shouldn’t. It’s better it’s open source.

Bridget: But I think it’s— I know Tim was telling me that this has been interesting for him just because this is your first time as an open source maintainer, right, Tim?

Tim: Yeah. I mean, I’ve made small contributions when we had, say, at Drama Fever, when we had problems with libraries that we were using or tools we were using, we were always pretty good about contributing back. But, yeah, I mean, a project of this size where— I mean, it’s not a huge project, but something where not just that it’s out in the open, but that I know that people are actually using it in their real-world production environments. In fact, right at this point, over half the code, I think, has been contributed back from the community rather than— and mostly, to be clear, mostly from one particular person who’s at Jet.com, Justin Walker, who’s— they’re not joint customers. They’re not, uh, they’re not using Triton. They’re, you know, he’s using, uh, Mesos and Marathon for their containers. And, but it solves the, you know, it solves the set of problems, uh, that they have. And, uh, so he’s been a really great contributor to that. And that’s been really fun to know that, like, people are really using it. And, you know, and we have a couple joint customers who are, who are using it in product, in their production environments as well.

Adam: [00:32:30] So that’s so cool.

Tim: Yeah, yeah. Oh, it’s like, it’s really, it’s really validating, right? And And even people who you haven’t— who we haven’t met, like Homejoy. Yeah, yeah, yeah. Like, they were, you know, I found out that at like Austin DevOps Days or something, they were at one of the DevOps meetups there, you know, there was a guy on stage from Homejoy saying, oh yeah, and we’re using this thing called Container Pilot. And I was like, wow, that’s amazing, right?

Adam: And then you find out it’s like everywhere and their entire business relies on it. You’re like, that is amazing and terrifying.

Matty: And also I want to hide under a rock.

Adam: Yeah, I had that experience at ChefConf where it totally like People came up to me and were like, oh, we’re using Habitat in production. I’m like, oh, are you? So good to know. You should totally do that.

Matty: It’s amazing.

Adam: Also, wow.

Tim: I was asked at Container Summit in Austin, I was giving a demo, a presentation, and somebody asked, is this production ready? And I said, people are using it in production. Brian Cantrell called me on that and said, yeah, I want to note that he said people are using it in production, not production ready. But he feels that it was production ready. So I was—

Adam: [00:33:34] yeah, he’s probably right. Like, I would still say that if it was up to me, no one would be using Chef yet. You know, like, if it was me, I’d be like, this is not— you shouldn’t— shenanigans. But like, it’s because I know like all the bodies. Like, I know all the moments where I was like, well, we cut that corner. Like, it turns out they’re fine. They were the corners you’re supposed to cut and everything’s cool. But like, man, if you ask me, I’d be like, I don’t know, man.

Tim: But if we’re being honest with ourselves, like, both Habitat, you know, the Habitat supervisor and Containerd pilot are a lot simpler than Chef, right?

Adam: I mean, they’re really—

Tim: absolutely. They launch something and they fire hooks, and that was really not a ton to them.

Adam: Well, so there’s an interesting point hiding in there though. That was one of the ways for me that I felt like when I started showing it to other people and getting feedback, where it was clear that we were onto something that mattered, was that it was so much simpler and the codebase was so much smaller. And it was because where the behavior happens, where the principles wind up coming out, are so well aligned that the code gets simpler and it’s easier to reason about. Because you don’t have to put— if you can rely on the build system, if you can rely on the inputs, if you can rely on the behavior of the supervisor, you stop having to put so much reliance at, like, the resource level that Chef does, right? Like, you don’t need— I don’t need an idempotent resource to put a file in the right place on disk if I can guarantee you that the outputs of whatever you ran will be the same if you ran it again at another place. And so, like, everything gets easier, and it all gets simpler, and it all gets smaller. It was really good. And I think, I’m sure you probably had the same experience.

Bridget: [00:35:17] Though people get a little bit nervous sometimes when you take away some of the knobs that they’re to turning, even if they shouldn’t be turning them anymore. Like, that, that is something that I have that— I have that conversation with customers again and again. Like, yeah, you want this, but you’re nervous about it. Like, yeah, how are you seeing— for both of you, how are you seeing people react to the way they have to think now?

Adam: I think there’s sort of 2 layers of knobs. So, um, for most people who just want to consume it, who, like, they’re just going to use a Container Pilot service, they’re going to use a Habitat-enabled service, right? What you want are only the knobs that they need for, like, safety criticality and, like, and the basics of their job. So, you know, my car has a button to start it, and I can turn off the anti-lock brakes, and I can turn off traction control, and that’s basically all of the options I have. And then I have a mechanic and he can plug a computer into my car and that computer can do things like change the engine timing and the gear ratio and he can do all these crazy shenanigans about its behavior. And you may or may not want to expose the knobs of that to people who are building and maintaining those things behind the scenes. So, for me, it’s about 2 layers of knobs. One layer is There’s what you need as a user, which should be small and just the ones that matter. And then there’s what you need if you’re building and maintaining, and then that needs to be a bigger set, but still constrained. Because, like, for Habitat, as an example, the gossip protocol right now has zero tunables in it because you shouldn’t need to tune it. Now, I’m sure we got something wrong in the implementation. And then the question will be, does that have to be tunable, or can I just change the defaults and then it works for everybody?

Tim: [00:37:08] I think what we’ve found is that, um, kind of we’ve ended up with, you know, effectively 2 levels or, or many levels of kind of knobs, inasmuch that, um, the, the kind of minimum configuration that you need for Container Pilot is very small. It’s like, okay, what service, you know, tell me where, um, tell me where the service catalog is, what service you’re advertising, and what backend you’re interested in, if any. And, and almost all of those can be We pare it down to, like, 1 or 2 lines of JSON to describe the thing. But it gets— those configurations can get larger, and because the hooks themselves are defined by the behaviors that are defined entirely by the user, they can be kind of arbitrarily rich. So, our NGINX example just uses curl for its health check. Our MySQL example is— we actually have a whole Python program that we’re shipping in that example that kind of does, like, okay, Hey, let’s look at things like set up the connection, look at various metrics, or how does it do the election between the— how does it figure out which of the replicas needs to become the primary at any given time? So, kind of that arbitrary richness, we kind of put that into the user’s hands. But there’s a lot of levels that you can do in there. And so, I think that’s kind of been the way that we’ve we’ve approached that. And so, as a result of that, because kind of the— it just really runs the hooks, what we found is that when people are using it, where they’re running into issues, it’s like, okay, because those hooks can be kind of arbitrarily rich, and they can be a single line, they can be a bash script, they can be a program, it gives— that’s kind of where people run into problems, because it forces them to then think about, oh, well, how is my application supposed to behave when its database goes away? Which is something they might have tried to abstract away. Behind, like, oh, well, the pool takes care of that, or the load balancer takes care of that. They actually have to think about what the behavior is now. So, there’s a little bit of a learning curve there, I think. But I think what it ends up resulting in is more robust software when you’re done.

Adam: [00:39:16] For sure. And I think that curve is lower than the equivalent curve in things like Chef and Puppet. It’s easier to look at a list of hooks and be like, okay, yeah, when X happens, Y. And then, there’s complexity in, like, well, okay, but did you cover all the edge cases of why? And you got to do that. But I also think one of the things that is so interesting was that we both wound up basically with the same hook design. That idea that, oh, it’s just lifecycle hooks and there’s reference, also speaks volumes to that as sort of the right approach. It feels like the right approach.

Matty: So, kind of going back a little bit to Tim, when you’re talking about how much of the code is coming from outside of Joyent, and just as these things are unleashed upon the open-sourciness of contribution, what kind of contributions that are interesting, or what’s been interesting to you about the contributions that are coming back outside of Chef or outside of Joyent?

Tim: Well, I think what has been interesting is we’ve seen kind of 2 sets of— of course, there are a few bug fixes and things like that. Which has been great, but also new features, or new hooks that we didn’t anticipate. One of the hooks that the Jet.com folks really wanted is to have some kind of periodic task that wasn’t part of the regular lifecycle of the application. So, something to say, like, you know what, for this database container, we really need to have a periodic snapshot taken. And so, adding that feature was part of the community contributions. And in fact, we’ve taken a lot of that code, because it was really well-written. And there’s an open PR we’re working on right now that says, okay, it had really good semantics around timeouts. We’re actually gonna move that, kind of refactor so that timeout code is now distributed throughout, is being used by all the different hooks that we have. So that was kind of like the first set. And then the second set, there’s been some folks who have tried to get, kind of expanded the scope more than we really wanted it to. And so I think we’ve kind of had this kind of balancing act we’ve been doing with the community where it’s like, we think that’s actually out of scope for this thing. In the container world, I think a lot of the time you still want some kind of thin scheduling layer on top of these things. So, a lot of our demo applications are just using Docker Compose because that’s a very easy way to launch a bunch of containers. Then if they’re all Container Pilot services, they can all wire themselves up. Some folks are trying to move more of the scheduling logic into this that we kind said, you know what, that’s actually— we don’t actually think that that’s where it belongs. And so, there’s always that kind of push and pull that you have to kind of gently do with folks.

Adam: [00:42:04] We’ve seen the exact same thing, especially on scheduling. Scheduling is the new email, where JWZ had that quote about how every program grows until it’s email. The new version of that is every infrastructure program will grow until it becomes a full-blown half-assed version of Kubernetes, right? Or of Cloud Foundry or whatever. We’re all going to build a scheduler. God willing, we won’t. Please let us not. I think you also need— you use the word simple, and I think you’re right. The simple scheduler is probably where we wind up in the end if the patterns of Habitat and Container Pilot turn out to really be right. Because so much of what makes the scheduler more complicated is that orchestration in the middle, where you push all the orchestration into the scheduler. Habitat, in terms of open source contribution, I think we saw most of what we’ve seen so far is people contributing plans. So people contributing software into Habitat that is Habitat-enabled and sort of building examples. We’ve seen a few contributions sort of to the supervisor, fixing small bugs and UX problems and those sorts of things. I think we’ll see more of that for sure. One of the conversations we’re having now in the Habitat community is, are we gonna see more shared maintenance of the plans, or are we gonna sort of see more, or do we wanna have more sort of way distributed behavior? I think we’re heading now towards saying that we want lots of core maintainers who are willing to volunteer to manage different pieces of software. And then we’ll sort of let the build service handle the rest. So, a lot of that stuff is happening now in the Habitat community. And we know, for example, that there’s a ton more of the topologies and the update strategies and those sorts of things that Habitat supports, where right now, we support one, right? And we know that there’s more than one, but rather than write them ourselves, I could make a list and go write them all, but then most of them wouldn’t get used. There’d be like 3 that matter, and like one weird stubby appendage that you wrote 6 months earlier because you thought it would matter and turned out not to. So we’re going to write those as people need them.

Bridget: [00:44:18] So this makes me curious when you’re kind of talking about, you know, how this stuff fits in, like in terms of other people contributing as well as people from your actual own organizations. Like, Tim, do you want to kind of talk to us a little bit about— I mean, Joyent was just acquired by Samsung. Like, is Samsung looking at Container Pilot? Like, how does Container Pilot play into the overall picture of what goes on at Joyent?

Matty: Sure.

Tim: So Container Pilot isn’t being used in our data center software. It’s not kind of necessary in that infrastructure because kind of the whole— because the entire box, basically the entire compute node is basically immutable in the Joyent infrastructure. I can say that Container Pilot is definitely something that we’re talking about as kind of like the core of this Autopilot pattern that we’re talking about with our customers and prospects. I can’t really talk too much about what Samsung wants to do with it, except to say that it’s definitely something that we’ve been talking with them about. I think that’s probably— they’re pretty constrained in what we want to share about the details.

Bridget: [00:45:24] Oh, so I will fill in with exciting things.

Adam: Yeah, nice boundary.

Tim: Exciting.

Adam: That was good. Way to set boundaries and hold the line.

Bridget: Well, you know, you know, Adam, I’m going to ask you the same question, which is, and this is kind of interesting because, because Chef is of course in the infrastructure automation business, and this kind of has this, you know, disrupting capability there. So how does this affect the whole Chef business? Sure.

Adam: So I mean, at the root of that is, like, what exactly is the business that you’re in? So I believe, and since I founded the company and I’m still here, the company gets to believe it too. Not that other people wouldn’t have come to those same conclusions. That was a joke. That wasn’t shade, that the business that I’m in is making successful customers. The truth is, people are like, you’re going through this transition, it’s complicated. There’s automation, there’s DevOps, there’s agile transformation. All that stuff is all tied together. You could have the best automation in the world. If the inputs to that automation are trash, then the outputs will be trash, right? And similarly, you could have the best idea in the world, but if the way that you want to build it and the organization you put it into is trash, then it’s probably going to be trash. So the business that Chef is actually in is helping these organizations go through that journey and being successful in, in, in that trip, right? So, you know, like Alaska Airlines got on stage at ChefConf and talked about this journey they’re on to be like the 62nd airline or whatever. And like, they didn’t talk a lot about Chef while they were there. What they did was talk about how all the automation and all the computer systems and all of those same principles and practices lead them to get to a place where they can open the door of the airplane in a minute from, from like gets to the thing, door comes in, door opens, 60 seconds. And like, that’s the business, that’s the business that we’re in. Like our business, like what we do is we help people do that. And if we do that for these businesses, then we’re awesome and we’ll be a company that lives forever and your grandkids will be like, wow, you had the Chef guys on your podcast or whatever.

Matty: [00:47:31] And it’ll be super cool.

Adam: Um, you know, it won’t be because I sold people Chef, like no one, Because in the end, it’s just software. And not that it doesn’t matter or it’s not great, it does matter and it is, but because it’s about helping people solve those bigger problems. So where Habitat fits in there is we see this really true need for open source automation. So if the things that you build your company on top of are not open source, I think you’re making a mistake. Both in your ability to grow and your ability to adapt to the challenges you have as an organization that are going to be unique to you. And so, one of those is Chef, which is incredibly good at infrastructure automation. Another is Habitat, which is super good at application automation, which, when you look at where Chef gets hard, if you graph the difficulty curve, Chef gets hard when you start talking about complex application orchestration. That’s where Habitat is really easy. Habitat gets really weird when you start talking about operating system configuration and shared state. And that’s where Chef is really easy. So, there’s this interesting curve there where they cross. We also have a thing we call Inspect, which does security and compliance automation. So, those 3 things, for us, roll up into the same commercial platform, which we call Automate, which does workflow. So, how do people work together to get changes through all 3? And then visibility, Which is, how do you know what happened? What’s going on with all of these things, with the infrastructure, with the application, with the flow of changes from left to right through the infrastructure? So, from a business point of view, it all flows into the same commercial product. So, I don’t really care if you use Habitat or you use Chef. You’ll use Automate either way, and you’ll probably use InSpec either way. And what I sell is you successfully transforming your business into being awesome, not so much what your infrastructure automation tool choices are. Yeah, that’s how it works.

Bridget: [00:49:35] That makes perfect sense. And I can see why both with Habitat and with Container Pilot, you’re solving this specific problem that the tools that we all have all been using, that have had— we’ve all had in place for a while, don’t solve. So, like, when— and I’m going to ask you, Adam, first, and then I’m going to ask you, Tim— like, when did you realize that you needed to solve this problem a different way?

Adam: Well, so I was on vacation, and I was smoking a cigar and drinking tequila.

Bridget: And you mentioned you were in Mexico. Yes, of course. But I’m just kind of wondering, like, you didn’t just get struck by, you know, a revelation in Mexico.

Adam: I mean, I kind of did a little. So, like, the challenge was, like, what I was thinking about and what then later in building prototypes became really real. Was, of course, I started from, how do I do this with Chef? How can I make Chef awesome in this space? And the truth is, when you went out, and you paid attention, and you looked at the space, and I spent 3 months not building Habitat, but instead, just using everybody’s kit. I used Docker, and Cloud Foundry, and I used super early versions of Kubernetes. I looked at as much stuff that was new and outside of Chef as I could. And the truth was, there just wasn’t a lot of value to add there, because they’re pretty good, and they’re getting better. But the problem was wrong. We built this diagram that said, applications, and then runtime, and then infrastructure. And so, it was like, you’re gonna have computers somewhere, right, in the cloud or in a data center. You’re gonna have some kind of runtime scheduler for at least most of your stuff, and that thing will eventually be I don’t know, one of N, right? And then on top, there was this big gray box, and it was just like, your shit goes here. And it was weird that that box was gray. It’s just weird that, like, for the thing we all wanted most in life, which is my application to be running and to manage it through its life, literally, it was just a gray box that was like, fuck you. I don’t know. I don’t know how that’s gonna work, brother. It’s on you. You just fucking sort it out, man. Sorry for the sweary, Spike. Really, that was basically what we were saying to everybody. It was just like, I don’t know, figure it out, Bob. And it’s like, well, what’s he gonna— so, what do you have to do to figure it out? And it was like, well, if you knew in advance that you were gonna run in AWS, and then you were gonna put Kubernetes on top of it, and then you knew in advance that you were gonna use etcd because you were using Kubernetes, maybe what you would do is build your application stuff so that you could have— so that you could then have the application do all the smart things it needs to do to deploy itself throughout its lifecycle, and it’ll all be immutable, and what could possibly go wrong? And it was like, you know what, maybe fuck that. Like, I don’t wanna do that at all. And for me, that was where I transitioned from being like, this seems fun, to it’s necessary. Because so few people have gone through that curve. And one of my experiences is when you do something hard, you love that you did it. You triumphed over the difficulty, you know? And like, that shit is hard. And so you meet people and they’re like, it was awesome. You should just— and what they tell you is basically to die in a fire. They’re like, you should suffer in this way as I suffered, and then you too could be amazing. And that for me was when I knew that we were onto something, because I don’t think you have to suffer like that if you think it— if you have technology that thought through the actual scope and breadth of your problem. And that’s what I think Habitat does, and I think it’s what Container Pilot does too.

Bridget: [00:53:09] What do you think, Tim?

Tim: A lot of the inspiration for the work came from the work that Bridget and I did at Drum& Fever, actually. I think where we had this 2 layers of orchestration that we had to do on everything. We had a Chef to Packer pipeline that was building all our VMs, and then we had this other thing going on that was building a bunch of Docker containers that we would then deploy. We, we had, you know, that really great infrastructure layer. And then we said, you know, which, you know, we, you have to put a lot of energy into build. And then we said, okay, let’s build, you know, and then it was like, well, and then the applications, it was like, well, that’s on the devs. And, you know, some of that didn’t actually work out all that well. And so there’s some, you know, some good lessons and bad lessons there. Um, and then when I came to Joyent, it was like, okay, well, you know, Joyent has the infrastructure layer covered, right? So it was like, okay, so my focus was just going to be on that application layer. And I started writing a lot of Bash, right? And I was like, wow, you know? And at every turn it was like, huh, so I’m really gonna like shell into the Docker container and like configure Couchbase to do this? And it was awful, right?

Adam: [00:54:15] So, you’re super R. That’s exactly what you’re gonna do.

Tim: Right? And I was like, no, that’s not the way. I’m like, there has to be a better way to do this. And so, it really started with like, we were writing all this like, you know, okay, well, we’ll have this shell script running in this thing that’s gonna do all this work. And then as soon as you start saying, well, I wanna have work that’s running in parallel to the application. It’s now become this, like, no, there’s got— and I remember Casey and I kind of being like, there’s gotta be a better way to do this, kind of on the phone. And I was like, well, what if we just built a thing that did that? And it was like, aha, we totally could do that. And that was—

Bridget: yeah.

Matty: And you could—

Adam: it’s a little bit lightning-bolty.

Tim: Yeah.

Bridget: You could write it in Go, and it wouldn’t have to be janky Bash.

Tim: Right, right, right.

Matty: Sure.

Bridget: And it does kind of strike you then. You’re like, wait a minute.

Tim: Yeah. Yeah. I mean, I think, I think it’s kind of like the— it’s, it’s, and it’s the same. I feel like it was the same, very similar sort of thing with, with, you know, a little plug for Triton. Like, I, it was very similar sort of thing for me for Triton itself as well, Joanne’s product, when I saw Casey Bison demoing it last summer here in Philly. And it was like, wow, you know, that whole like big chunk of infrastructure stuff, stuff, which isn’t direct business value, right? Kind of going into what Adam was talking about before, like, what is it that we really care about? All this infrastructure stuff is really clever, but it’s not what our businesses are supposed to be doing, right? That’s not our core focus. The ability to have something that’s— so, like, being able to say, wherever you can find those spots where it’s like, oh, So, you can build something that’s really robust and puts the focus back where the business value is, which I think is always gonna be at the application level. Wherever you can do that, when you can find those little bits of inspiration, luckily, you do it, I think you’re in good shape.

Adam: [00:56:11] I think that’s right. And I think one thing that’s really true about both Container Pilot and Habitat, and I think it’s true about things like Triton, and it was true about EC2. There are moments when you see technology that once you’ve seen it, you can’t unsee it. And you’re like, oh, nope, this clicked in my brain this way now, and that’s how it will be forever. And I’m going to have to explain it to you. And I felt that way when I first picked up CFEngine 2, where I was like, oh, nope, yep, that thing, it’s going to be like that now, some variant of that now forever. And I felt that way with Docker. And I think both of these things, and when you think about that stack, Like, we are definitely in a moment where, like, once you see those things, you can’t unsee them, and, like, you will forever now think about the problem in that shape.

Bridget: And that does make me want to, you know, we’re getting low on time here. We always run out of time on these episodes. It’s terrible or wonderful. I’ll just have to have you come back on another time. But, um, I, I do know that, Adam, you’ve been going around giving conference talks about the future of automation, and so I’m going to put you both on the spot. I’ll let you go first since you’ve been giving the future talk, but give us, give us the TL;DR.

Adam: [00:57:24] Yeah.

Bridget: Give us the TL;DR. What is this future of automation and application configuration? Yeah.

Adam: So, um, I’ll do the human answer and then the nerdy answer. So I think the human answer is the future of automation is thinking about the experience holistically, not of the siloed job of any individual or any team, but instead of, of how do we go through that entire process of knowing we need something and getting it out the other side, where sometimes that’s an application change, maybe it’s infrastructure, who cares what it is, but designing our automation not so much to solve the bespoke, the smaller problems that we have as a team, as systems administrators, as operators, as application people, but instead thinking about that holistic view of the problem, that’s the future at the human level. And it’s not how we built the tools. It’s not how Chef got built. It’s not how Puppet got built. I mean, look, retroactively, we can tell you it’s how it got built, but it wasn’t, right? We had a problem, we solved it. And so, when you think about how— so, I think that’s the future at the human level. I think at the technical level, what that means is you’re going to see tooling that starts to break out of the traditional boxes that we’ve put them So, what’s the boundary between the operating system and the application? That was defined for a long time by the vendors who sold us operating systems. And then the first free operating systems mimicked those choices because they felt immutable. They felt like they came down from on high because that was how it was supposed to be. Is it how it’s supposed to be? Is it right? I don’t know. What I can tell you is that it’s open season on thinking through what needs to change at all the different levels of the stack to support the right user experience. Because the user experience we all have right now is insufficient if what we’re going to do is alter the future shape of the platform for every business all over the planet. We won’t get there this way. It’s fine for us. It’s not fine for them. Even if we can get them to try, the actual lived experience of doing it is not good enough. So, the future of our automation is that you’re gonna see more tools that break that mold in the name of getting the experience of using them better. And they’re gonna do that wherever they need to. So, if that means new operating systems, that’s what it’ll mean. If it means new scheduler platforms, then that’s what it means. If it means that none of that makes sense, and we should all go back to, like, crossover cables and physical data centers, then that’s what it means. And I don’t know what it is, but, but whatever we need in the name of that, that’s what’s gonna be true.

Bridget: [01:00:07] Nice. Or, you know, or maybe it’s just all unikernels, right?

Adam: It could be. Your coworkers at Joyent love unikernels. I doubt that. I doubt the unikernel thing, but, you know, could be. By the time you’re done giving me an operable unikernel, it won’t be a unikernel anymore. It’ll be an operating system.

Bridget: So, right now you’re All right. So, Tim, tell us, predict the future.

Tim: So, I’m gonna try to keep my predictions a little less grandiose. And really, because I think what we’re going to see is a little bit of a backlash about the idea of everything being easy. And what I really mean by that is not that they shouldn’t have good user experiences or developer experiences, but that the notion that we can kind of abstract away the internals of how applications work, I think people have done that, and they’re like, wow, you know what? When we’ve done that, we’ve created these towering pillars of stuff that we don’t really understand how they work, and that’s not actually gonna give good production-operable systems. And so, if we’re taking an idea that we’re saying, okay, what if we instead move the intelligence down to the application layer, and we can do things where application— we’ve given the application developer a lot of responsibility in doing so, right? We’re saying, okay. This was something that you might have tried to, like, pitch off to somebody else before, and now you have more responsibility. But I think what we’re also doing at the same time is we’re giving them power to do so, and we’ve now empowered application developers to do that. I think that’s gonna be a trend that we’re gonna see, and it’ll continue.

Bridget: [01:01:42] I think what I’m hearing is the Netflix-style freedom and responsibility from you, Tim. Totally. Like, hey, with great power comes getting paged. Yes.

Tim: Well, or hopefully not. Hopefully getting paged a lot less. But yeah, yeah, yeah. Nice.

Bridget: I love it. Okay, so I think we’re, we’re probably going to want to move into our checkouts.

Matty: It is checky-outy time, but before we do that, let’s talk about some little community and event stuff. So first of all, remember, if you have an upcoming conference you’d like to see promoted on ADO, if you go to arresteddevops.com/conf, it will send me an email and I will ignore it, but maybe we’ll put it on the show. At least it will send me an email, I can tell you that much. Upcoming conferences. There’s a whole bunch of DevOps Days coming up because it’s that time. There’s Spring One platform is now basically.

Bridget: Oh yeah. I mean, I don’t even know if we’re going to get this episode up before that is happening, but that’s next week in Vegas. I’m literally sitting in Florida right now. I get to go home for like a day and a half, and then I have to go to Las Vegas because apparently I like going to really hot places.

Matty: [01:02:54] Yeah. Anyway, go to devopsdays.org, check it out. There’s a whole bunch of them. And they can also find a bunch of open CFPs. There’s a ton in some cool places, not that like, like Ohio, Ohio is cool, but also like Singapore and Cape Town and Nashville and Detroit and Madison and Berlin and Ghent and Bangalore. So go check those all out and write some cool stuff. Um, and I know Bridget traveling around is per the usual. So you can go, but I have now become boring, and I don’t go anywhere anymore, but everyone should come to DevOps Days Chicago, August 30th and 31st. A couple tickets left. ADO2016 will get you 20% off. It’ll probably do that at the other DevOps Days registrations too, but I’m giving Chicago some love, and this year with extra Adam Jacob. So devopsdays.org, come to Chicago.

Bridget: Nice. Oh, I should point out, I’m going to definitely be at DevOps Days Philly, which is where Tim lives, and that one is October 26th and October 27th. Everyone should definitely go to that one too. Yeah. Cool.

Matty: [01:04:07] Tim, what do you have for folks to check out?

Tim: Sure. Of course, all the stuff that we’ve been talking about is open source, so github.com/joyent/containerpilot. For Containerd itself. And then there’s a bunch of different demonstration applications, and some of them have been contributed by the community at github.com/autopilot-pattern.

Matty: Cool. Adam?

Adam: Yeah, you can learn about Habitat at habitat.sh. And what else do I have? I had something else, but now it flew my mind, so we’re gonna let it go.

Bridget: So, I didn’t actually prepare checkouts, but I thought of 2 while our wonderful guests were talking. So, while Tim was talking about being an open-source maintainer, I was reminded of that great blog post that Jesse Frazell wrote called The Art of Closing. So, we’ll have a link in the show notes, just advice to open-source maintainers. Like, you can’t say yes to everything because you have to support it forever. And then, while Adam was talking, I thought of And this book that Andrew Clay Shafer recommended I read, and he was right, called Badass by Cathy Sierra. And it’s all about how users don’t want to be an expert in your software. They want to be amazing. And so, if your software helps them do that, they will love it. So, those are the things I think people should check out. Nice. How about you, Matt?

Matty: [01:05:32] So, I discovered a cool game that I want to— I don’t know if anybody has seen it. It’s called Pokémon Go. I think, so you should try that. But also, along the— in keeping with the same theme of, like, me discovering a thing that everybody’s already known about, which is also known as my life, indirectly from friend of the show Jill Jubinski, I came across devopsdictionary.com, which I have now contributed a whole bunch of stuff to because I was bored. I’m pretty sure that the maintainers, like, abandonware’d it, like, 2 years ago, but I’m having a lot of fun with that. So you should all go add stuff, mostly just adding stuff about Michael Ducy, because—

Bridget: because, oh yeah, when it was first mentioned and it was like, there are 2 people in DevOps, Patrick DePoi and Seth Fargo, and I was like, what? Sort of just went on with my life.

Adam: Kind of right though.

Bridget: Yeah, I just figured somebody was— somebody was just trolling them, you know.

Adam: I think we’re all just pale reflections of Patrick and Seth.

Bridget: [01:06:35] Some of us more pale than others, so do not go out in the sun.

Matty: Some of us more reflective. So yeah, so we have a newsletter. We actually even just sent it, but you can sign up for it at rusteddevops.com/bananastand. It is the best way to know about upcoming podcast episodes and a fairly good way to know about some other DevOps stuff. And if you want to become a true ADO insider, you can check us out at patreon.com/adeo.

Bridget: Arrestedevops.com/arrestedevops. And this is all entirely because we keep doing live episodes at conferences and forgetting to say anything about the sponsors.

Adam: Yeah, we did.

Bridget: Sponsorship down. Yeah, it’s a problem. But all right, so thanks to our sponsors. Be sure to visit them at arrestedevops.com/10thmagnitude and arrestedevops.com/datadog. And thanks, Tim and Adam, for joining us. This was super fun. Thanks for having us. Yeah, it’s been awesome. So loyal listeners, if you enjoy Arrested DevOps, we’d appreciate it if you’d visit arresteddevops.com/itunes and leave us a review in the iTunes Store. I think Stratton reads them. I don’t because I have a policy of not reading the comments.

Matty: [01:07:48] They’re always good. The only negative review we got was a troll from Michael Ducy. That’s not a challenge to come troll us in iTunes comments, by the way.

Bridget: Nice. But we would love to know what you think of this episode. If you leave us comments at arresteddevops.com/app-config, then I think Stratton has some sort of recipe that will send those to our Slack channel. Yep, we’ll figure it out.

Matty: We have automation. I’m as surprised as you. Automation. So yeah, we’re on Twitter @ArrestedDevOps. We’re on Facebook at Arrested DevOps. We’re on the internet at Arrested DevOps. So blah, blah, blah. Anyway, we’d love to get input, ideas, or feedback if you have ideas for future episodes. Let us know, we might do them. I’m Matt at Matt Stratton. It’s possible. It’s possible that I’m Matt. Yeah, it’s fairly possible. But anyway, I’m Matt at Matt Stratton. Still, you’re still Matt. Now you talk, Bridget.

Bridget: [01:08:51] I’m Bridget at Bridget Kromhout.

Matty: We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.

Adam: This is not— you shouldn’t— shenanigans.