What Is New At Puppet? with Eric Sorenson

Matty: [00:00:07] It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and business for maximum DevOps awesomeness. On that note, if you haven’t noticed, we’re experimenting with having a new intro for the show. So let us know what you think. I’m your host, Matt Stratton, @MattStratton on Twitter. This episode is sponsored by VictorOps. From initial alarm to final retrospective, the mission at VictorOps is to make on-call suck less. Easily integrate with your existing monitoring systems and manage on-call schedules with rules for intelligent routing. In the live infrastructure timeline, get real-time context and see annotated alarms with resolution documentation. And when you’re in the firefight, collaboratively troubleshoot using native chat or bidirectional integrations with your favorite chat clients. Visit arresteddevops.com/victorops and sign up for a free 14-day trial. To see how they’re making on-calls suck less. This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools such as Chef, Docker, and AWS so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. So I recently had the chance to talk to Erik Sorensen of Puppet Labs about a whole bunch of new things going on in the Puppet universe. Erik’s a great guest and we love having him on the show. We talked about Puppet 4, Puppet’s new application orchestration, and we even pretended to be industry pundits around the Red Hat acquisition of Ansible. So I’m here with friend of the show Erik Sorensen from Puppet here to talk about, well, all things Puppet. So Eric, for people who haven’t had the pleasure of meeting you yet, can you give us just a little background on yourself and what you do at Puppet Labs?

Eric: [00:02:10] Sure thing, Matt, and thanks for having me on. Yeah, so I’m a longtime systems administrator. I’ve been doing configuration management for longer perhaps than I care to admit. My first CFEngine deployment was in 1998, back when it was still pretty raw. But I got really hooked on the idea of just lighting up an infrastructure and watching it come to life more or less on its own after I got my code working, I focused on that for the rest of my career. The last, my last job before I came to Puppet, uh, in 2012 was I built out a Puppet infrastructure at Apple for, um, MobileMe and for the iCloud services. And, uh, over time I just kind of wanted to, uh, move to Portland for various reasons, some personal, some professional. Uh, there’s, I’m a big bike nerd. And so I was happy to come up here in the land of, of cycling year round, get some fancy coffee and a good beer and, and of course work, work at Puppet. So I’m the technical product manager for Puppet itself and for the platform more generally.

Matty: Great. So, also wanted to talk a little bit, we did an episode recently about infrastructure as code that you were on, and if listeners want to go check that out, we’ll have a link in the show notes, or you can go to arresteddevops.com/infrastructure-as-code. We did a configuration management episode a year ago, but I’m interested if you want to kind of give us, before we go deep on any of the new stuff around Puppet, can you give us the, you know, maybe for listeners who aren’t familiar with Puppet other than it’s that thing for configuring some servers. Why should people care?

Eric: [00:03:40] Sure. So yeah, Puppet’s, um, you know, it’s a tool that’s been around for about 10 years. We just sort of like, uh, I gave a talk at PuppetConf where I went back to the earliest, uh, commit in the, in the Git repository, which was actually an import from a Subversion repository before there, there really was Git. Uh, and yeah, it was from 2005, so it’s been around for a while.

Matty: Well, happy 10th-aversary.

Eric: Exactly. But the idea is that Puppet lets you describe the desired state of your system and to write that out in a language, a domain-specific language, and Puppet will enforce that state on the nodes that it runs on. It can either run in sort of a master-agent mode or it can run standalone, whatever suits your fancy.

Matty: So basically what we’re doing, like a lot of modern configuration management, we’re kind of— I like to say we care less about how the sausage gets made and more about what is the thing that I want, right? I want to have my infrastructure look like this, and I’m going to trust Puppet to put it in that desired state.

Eric: [00:04:42] Exactly. I want to have a delicious bratwurst at the end of it.

Matty: Cool. So yeah, so let’s talk about what’s, what’s going on. So, uh, I know Puppet 4 came out— I don’t necessarily recently, but this year. Can you tell us a little bit? I mean, I’ll give a little background for myself. I haven’t really, uh, used Puppet in probably about 3 years or so, so my skills and knowledge are fairly rusty. So I’m really interested to know what’s What are some of the new things, first of all, in Puppet 4 before we get into the really new stuff that you guys just talked about at PuppetConf?

Eric: Sure, yeah, so Puppet 4 is the first major version of Puppet since, probably since you used it last in 2012. So some of that, some of the changes have been, you know, just sort of semantic version breaking changes that have stacked up over time, but there’s a lot of great stuff in there. I think the main thing is that we have a completely rewritten parser for the Puppet language, which has been available for about a year through a feature flag, like you could opt into it. But, uh, now it’s, it’s the default. I still have to break my own habit because I got into the, uh, into the habit over the past couple of years of calling it the future parser. That was the flag you had to set.

Matty: [00:05:52] Now it’s the present parser.

Eric: Now it’s the present parser and the previous parser is gone. So, um, the future is here in a sense, but, uh, it’s, it’s a really, so there was a lot of sort of, um, edge cases in the, in the language that the previous parser was against something that that Luke Kennies had written in a series of hotel rooms in 2006 and 2007 as his first developing Puppet. So there was a lot of emergent behavior that came out that was— sometimes people had come to rely on it, sometimes it was just sort of weird crap that happened. So it was nice to really get a fully formed specification for what that language ought to be and to sharpen it up. That people have been asking for for a long time that we just weren’t able to deliver in the previous one were some more languagey types of features for people that got into advanced Puppet module programming, like being able to do loops and iteration. There’s a type system in there so you can declare when you’re kind of declare an API for your module and say like, hey, we expect this thing to be passed in and it’s got to be a Boolean or it’s got to be one of 3 possible string values or it’s got to be a number. Those sorts of things that were sort of ill-defined in the previous parser.

Matty: [00:07:04] What’s the— what do you think the learning curve is for someone who hasn’t used the present parser, the current parser, and but maybe has a lot of experience with using Puppet over the last few years? What’s that? What’s that curve like?

Eric: Well, almost all of the language features are pretty much opt-in, like the looping and iteration and the type syntax is all kind of like you can sprinkle it in, you can go all in if you want to, you can sprinkle it in, or you can not use any of it if you don’t want to. So pretty much all with a few kind of edge cases where we had to change the way previous language constructs worked. All the Puppet 3 code is pretty much compatible with Puppet 4 parser. But generally, the syntax is pretty— it starts out pretty straightforward and looks more or less like a Nagios configuration file with curly braces and stuff, and then you can pepper in more languagey constructs like conditionals and loops and that sort of stuff as you need them and as you need more power in the language.

Matty: So being that this is out for a few months now and, and had been out there, what are you seeing? And I’m not expecting you necessarily to have like a hard number, but you know, what you see in the community and people you talk to, what’s the adoption been like for, for this new, for the new, the new parser and the new, the new Jam?

Eric: [00:08:20] Well, uh, there’s a lot of people who have gone all in with it. One of the examples that I talked through, um, at PuppetConf was, uh, uh, uh, organization called EvenUp, um, who are Puppet customers and literally went all in, like they reformed it all. Their modules, went all in with the type system, and Justin Lambert in our community was kind of the main driver behind that. And they really got a huge amount of reliability and were able to eliminate a ton of technical debt they’d accumulated in those modules over time. I would say numerically, the number of people running Puppet 4 in general is quite large. We did a cert, ran some stats, and there’s about 17,000 Puppet 4 installations out there that were— that have checked in at some point. And but I don’t know how many of those have converted their modules over to use the new syntax. I don’t have stats on that.

Matty: Yeah, I think that would probably be kind of hard to see. I don’t know if there’s any way that you could evaluate that in the 4, you know, because it’s kind of like you said, it’s really, are you using these features? Like, how do you kind of evaluate that? Like in some kind of static code analysis.

Eric: [00:09:34] Yeah, you know, for the Puppet Forge, which is our community content site, that there’s a, uh, uh, there’s actually a pretty awesome, um, sort of continuous integration system that runs in the background there when you upload new versions of the modules. And one of the quality checks that ends up, you know, uh, give assigning your module a, uh, a quality score of 1 through 5 when, when you upload a new version of it is to test whether it’s going to be, uh, Puppet 4 parser compatible. Now it doesn’t check necessarily whether you using, you know, new stuff that wasn’t available in 3, but it does make sure that if you’re, if you’re uploading it, that it’s gonna work for somebody who’s using Puppet 4.

Matty: Who’s using 4, so at least you’re, you’re, you’re, you’re compatible with that. So yeah, that’s definitely a good thing with that. So yeah, what are some, some of the other things that are new in 4 besides this fancy new parser?

Eric: Yeah, well, I would say the big— the other big shift is, you know, perhaps following Chef’s lead and kind of learning from some of the lessons that they all came to from moving over to Omnibus packaging. We’re also delivering the agents in an all-in-one style package that has everything that you need from a Ruby interpreter. Factor is included, OpenSSL is included, and Puppet itself obviously is all included in the same package. Plus, that package is unified between open source and Puppet Enterprise. It used to be that there were sort of 2 different versions and 2 different paths for various config files and SSL and stuff like that. Now it’s all kind of the one package to rule them all. As a delivery mechanism for the Puppet 4 agent. So that’s, that’s been a pretty, uh, it’s been kind of an interesting voyage for us, as I’m sure it was for, for Chef as well.

Matty: [00:11:16] Yeah, it’s, it’s, it definitely, um, I found, you know, having, having Omnibus and having kind of all the, you know, everything in the bowl, you know, depending upon the shop, the type of shop, obviously if, if it’s a heavy Ruby shop, they really like this because they don’t want, you know, their config management tool messing with their runtime. Absolutely. And even on the other side of the equation, if they’re not Ruby at all, they don’t want to have to worry about dealing with rbenv or any of the Ruby management stuff. So I’m a big fan of the Omnibus style.

Eric: And yeah, that’s been cool. I think on the other end of things too, it’s like you said, it’s been, it’s been great to be able to have a really consistent story for people who are running older operating systems like RHEL 4, which we support, and we’re able to say, you know, hey, you need If you need, we can guarantee that this is going to work the same as it does on more modern versions of Linux and even like Solaris and AIX, which we also support in the commercial version, that those are going to have a consistent user experience and you’re not going to have to rewrite your functions that you have used to extend Puppet in Ruby so that they take into account the fact that it’s going to be running on different versions, that sort of stuff. So that’s been a pretty big one.

Matty: [00:12:35] So hear that, all you DevOps AIX users. You have no excuses. You can do infra as code. You can use Chef or Puppet. So there’s no excuses.

Eric: You’d be— yeah, I mean, maybe you wouldn’t be surprised, but I was surprised, I should say, as to how much, um, you know, AIX there is out there in the world. And, and not even, you know, like, I guess, I guess if you’re running AIX, you’re going to run the version of AIX that came with your, with your hardware. But, uh, we go back all the way to AIX 5.1, which I think is, uh, Uh, quite, quite a ways back in the misty past.

Matty: Yeah, we keep, uh, joking about when we’re gonna— when someone’s gonna ship, you know, the kitchen LPAR driver, you know. So, and, and it’s a joke, but, you know, maybe, uh, we gotta test it.

Eric: And there’s some actually some pretty intense, uh, virtualization wins that AIX has if you’re— if you go all in with it, you know. You can, you can run a ton of, uh, a ton of LPARs on a, on a 390.

Matty: Waddleson, I know that you rewrote— and I think it’s Factor was rewritten in C++. Is that correct?

Eric: [00:13:39] Yep. Yep. So Factor is kind of the little bit equivalent to Ohi in Chef world that runs on all of the nodes and sort of gathers inventory information about them, gathers facts, and sends the facts along to the master, or uses that locally so you can make determinations about what the right stuff to do on that particular system is. And that was kind of an interesting experiment. So that, yeah, is rewritten in C++, and it allowed us to get some experience writing in that toolchain using C++, using Boost, which is a library, sort of like an enhanced standard library for C++. It lets you do stuff like command line parsing and regular expressions and all that stuff that modern languages have and that you’ve sort of been accustomed to just being there in the standard lib, but if you’re writing straight-up ANSI C, you know, you don’t want to have to go back to scanning through strings character by character to find a match. It’s like we have, we have tools for that now. And just getting experience and having, you know, build toolchain for all of the platforms that we support. And, and, you know, is this, is this really going to be a viable thing? And it turns out that it is. It’s pretty awesome. There’s— it’s super fast. It’s, it’s easy to extend. You can still extend it in, in Ruby if you wanted to. You can use structured data in YAML or JSON and just pull facts in that way. And, uh, it laid the foundation for a really awesome demo that, uh, was on the, the main stage at PuppetConf, uh, a couple weeks ago, where we have a prototype of the actual Puppet compiler that’s written in C++ using that same toolchain. And it’s something like, I don’t know, 50, 50 times faster than the Ruby one. It’s kind of ridiculous.

Matty: [00:15:24] And that’s running in Puppet land. That catalog compilation is something that happens on your Puppet master, right?

Eric: Yeah, it happens either on Well, it happens on the master if you’re running in agent master mode, or it happens on the edge if you’re running in standalone mode in Puppet apply mode. So it kind of— yeah, but that’s a necessary step for transforming the input of the Puppet language code that you write to the thing that’s actually consumed by the agent code that goes through and looks at your system and determines whether or not it’s in desired state.

Matty: So yeah, so 50 times speed increase, that’s non-trivial.

Eric: Yeah, and you kind of get to thinking, well, what would it look like in a world where you were basically running Puppet all the time? If it ran fast enough and consumed few enough resources on the system, that as soon as you pushed out a change, it more or less converged instantaneously across your whole infrastructure. There’s some interesting possibilities there that are still obviously pretty early on, but cool to think about.

Matty: Yeah, that sort of ability, like you said, it’s sort of watching for a change and just going. Found when I think about— there’s, there’s certainly, I think, organizations that would be ready for that. I think there’s still organizations where that would scare the hell out of them. These are the orgs that I’m, you know, work with who are saying, wait, no, I don’t want Puppeteer Chef running except maybe once a month.

Eric: [00:16:45] Yeah. Oh yeah.

Matty: Which is also scary as hell, by the way. If you’re listening, don’t do that.

Eric: Well, but I, you know, you can understand where they’re coming from. It’s definitely driven by, driven by a fear of fear of change. But I think that’s also interestingly sort of a shift that I’ve seen just over the course of the last decade or so, where we started out wanting— the model came from CFEngine, honestly, where you wanted CFEngine to run more or less continuously. And if somebody edited a file on a system, you wanted that to be reverted almost immediately. And we talk about that today in terms of configuration drift. And I think there’s been a shift to where now those kinds of changes, we want to be able to understand them and incorporate them into the, into the running config, you know, into this ongoing state enforcement, but not necessarily have them auto-reverted in a bastard operator from hell sense of like, no, you did the wrong thing and we’re going to beat you down for it. On the one hand, like, I understand, I am empathetic towards the idea that people want to have their configuration captured and not have random changes going on. But on the other hand, there’s definitely a real use case there where people either want to run in no-op mode or why-run mode and have the tool show us what did drift, but not actually remediate it until the user takes an action to do that deliberately.

Matty: [00:18:08] Yeah, and there’s a lot of pieces with that where I see, you know, and I kind of, just to back up, especially for people who are listening who are new to the space or are mostly used to using a tool like Puppet or Chef as a, like, this is the thing I do for provisioning, but not having it be something that’s happening a lot. The reason that when I was making what sounded like kind of an extreme example of saying I only want Puppet running once a month, but the thing is when you talk about every time that your config management agent runs, you run a risk— not a risk, there’s a possibility that something has drifted and it’s going to have to make a change. And the longer the interval between the last time it ran and now, the greater the chance of a large number of things that have changed. So you could be making a very large change if you think about how much your system changes over 30 days versus over a day versus over an hour. So I think part of it, like you said, just also as we’re moving and getting comfortable, especially in the infrastructure space, of understanding that actually, no, more frequent, smaller changes really are safer. And I think that’s a thing, as we— like you’re talking about in the continuous delivery model, of understanding that even if it’s not, like you said, in complete real time, but if your limitation, it’s a decision based on your business as to how often the, you know, Puppet agent is running, not because, well, it takes a long time, so I know it’ll overlap runs if I have it more frequently than every 30 minutes or whatever. Like, it should be a conscious, deliberate decision as to how often it runs, not a limitation of the technology or perceived thought on how it would work.

Eric: [00:19:46] Absolutely. We’ll talk about that in a minute on the— or in the application orchestration side too, because I feel like there’s a corresponding shift towards like Hey, I know my configuration management tool is really a means to an end to deliver the business app out, new versions of the business app or new configuration out to the nodes. It’s not so much about the bare system provisioning or ongoing state enforcement of the way the operating system was set up because those become more disposable. But it’s really about the user initiating some change through their infrastructure and the infrastructure responding to that rather than this CFEngine Puppet automatic convergence model.

Matty: Right. And that’s the thing I like to think about too, is that we don’t make changes to our infrastructure for the sake of making changes to our infrastructure. They’re driven by changes to our services, to our applications, to things of that nature. And that’s, again, when we think— and we talked about this, I think, a little bit on our Infrastructure as Code episode when you were on about tying those together, about an infrastructure change is driven by an application change of some kind. And yeah, so when we get into that app orchestration, which I’d love to start talking about that because I really haven’t had a chance to look at it more than anything than to know you have a thing called application orchestration.

Eric: [00:21:06] Sure.

Matty: So, and I think that was announced at PuppetCon very recently.

Eric: Yeah, yeah, it was, it was kind of the headline thing there. Let me hit on one more point. Oh yeah, about that. Of course, sort of as a concurrent change along with the agent rewrites or the moving into, into into C++, we’ve also been sort of shifting the server stack over. And traditionally Puppet runs, the Puppet master side runs under Apache and Passenger or some kind of Rack-compatible app server. And while that is awesome for a lot of people and works pretty well, there are certainly some limitations. And kind of the worst-case scenario that we started seeing more and more frequently as people ran Puppet at large scale was that the Apache Passenger stack would sort of fall apart and get kind of very erratic response times, and you kind of wouldn’t be able to know how to pull it back from the gamma quadrant. So a couple of years ago, based on the successes in the PuppetDB project, which is kind of a data warehouse project for Puppet, we started rewriting some of the server components into Clojure and running that replacement stack for that Apache and Passenger piece on the JVM, going through Jetty, and then running the Puppet Masters inside JRuby. And that’s been pretty awesome since it’s been rolled out over the last 2— I’m sorry, last year. It was announced at PuppetConf last 2014 and has been out since then. But we’re really consolidating on that on the server side, so you have kind of this really bulletproof server-side stack that runs mostly on the JVM and is written in Clojure, and then the agent-side stuff runs in C++.

Matty: [00:22:59] Yeah, because I mean, there’s the— I don’t wanna say the calculations, but again, if you’re running in that Puppet Master scenario, you’ve got that catalog compilation that needs to be happening on the server side. So anything that you can do to optimize that, right? You know, because you don’t want that— again, as you’re looking at saying, okay, great, now I can have my, you know, factor is faster and the catalog compile is faster, but if I’m waiting for something else. So you’re kind of doing the usual, right, you know, value stream. You’re like, okay, bottleneck, this bottleneck solved. What’s the next one? Right.

Eric: It certainly is in terms of like, if you’re running in an agent master mode, the bottleneck in Puppet is definitely the catalog compilation. I mean, it’s like, you know, factor of 10 or 20 more expensive than any of the other operations that we do. So optimizations that both, you know, lower the max values of the time that it takes to compile a catalog, as well as provides consistency if you get a thundering herd problem is pretty beneficial.

Matty: [00:24:03] Awesome. Yeah, that’s, that’s pretty cool. So yeah, so let’s, let’s talk application orchestration because orchestration is the magic word of the DevOps as I understand it. Forget containers, it’s about orchestration.

Eric: Well, even, even if you have containers, you still need orchestration.

Matty: You still need that. I like to say choreography instead of orchestration. That’s my new thing. I feel that’s more of the case, like a Swan Lake kind of choreography. Exactly. Yes. And then you go here and then you go there and we do it together.

Eric: I’m a big George Balanchine fan, so yeah, suits me too.

Matty: Maybe that’s right. We’re going to, we’re going to re— we’re going to turn the industry on its ear. You and me, Eric, we’re going to change the word because that works. Yeah. Yeah.

Eric: So we call it the— unfortunately, the product, however, the product is called Application Orchestration. And I, I couldn’t dip my oar into that particular stream before it hit the, hit the world. But yeah, so this is a, this is a pretty awesome thing. It’s a pretty— we think it’ll be a pretty big game changer, if I can use that word. That’s one of those. Content warning words like thought leadership that makes me cringe every time I say it. But in this case, it might actually be true. I don’t know. This is a prototype. There’s been a prototype that Luke wrote several years that’s been kicking around, and we actually focused this year on productizing that and making it a real, a real thing. And the idea is that the same kind of model based approach that you apply to a single node’s resources in regular Puppet, you can now apply across your application infrastructure using the orchestrator. So it kind of moves everything up a layer and lets you describe the relationship between hosts and between nodes or between containers, between different moving pieces of your application, and lets you share little bits of information about them. So you can sort of inject the idea of, hey, here’s my database connect string, let me tell my application servers what that is without having to necessarily hardcode it into the app server’s configuration. And then a piece that actually is the orchestration part of it, which lets you, when you kick off a job in response to a code change that you’re rolling out, it builds a graph just like Puppet does on a single node, but it builds a graph of your environment and then runs Puppet on the nodes that are involved in that application deployment in the correct correct order and taking into account that, you know, some of them need to finish before the next one starts and making sure that they have the right, um, the right information in their catalog before they run. So it’s, it’s pretty— there’s a few, there’s a few moving pieces there, but, um, it’s a pretty awesome thing to behold. Yeah.

Matty: [00:26:53] And, and I think that’s, that’s the thing that a lot of people are looking for is being able to talk about their topology, you know, like you said. So sort of moving, moving it up a layer for sure. And, and one thing that I, I find that’s really interesting with, you know, talking to customers and people who are using these tools too, is that sometimes I think there’s an interesting layer of when you need orchestration and when there are orchestration problems, what you think you have an orchestration problem and you can actually solve it with an architecture fix. So I think there’s a little bit of both, right? Like you still need the ability to have things happen in a certain sequence. But I’ve also seen a lot of times what people are looking to do, it’s kind of like, well, but you could, you could build a little more resiliency around this one particular piece so that you can can handle it. You still need to handle it, but there’s still dependencies.

Eric: Sure.

Matty: Right. You know, it’s just sometimes I think the— when I, when I see what people are expecting, and again, in my case, it’s just because of Chef, because I’m talking to pre-sales people in Chef and they’re like, oh, what do you mean that it can’t go and then automatically detect that this particular thing was done and this was done and this was done and this was done? It’s like, well, that’s what you can know that Chef was done because you should know that it put the thing in the proper state. You know, a lot of times they’re looking— because what people are used to doing, if you’re taking— this is, I guess, what I’m getting at is A lot of times when people are looking to solve an orchestration problem, or they’re saying that, what they’re doing is they’re taking a manual release process where there’s a long runbook of the things you do to release that have a lot of manual pieces. And if we think about it from the analog to config management, when we think about how the system is able to be idempotent, right? You can trust that you’re telling it Do it till this is done, but you don’t have to write in— we talked about this a little bit on the InfraCode episode. If I’m putting a package resource into Puppet or Chef, I don’t really have to write a test that says, did the package get installed? Because that’s what it does. So similarly, if I’m sort of saying, I’m writing my thing that says, okay, the first thing that has to happen is the database servers have to be updated, and I have a module that does all the things it needs to do, I don’t necessarily have a whole bunch of checks that need to happen before it moves on to the next thing. I— and correct me if I’m getting this kind of wrong— I can kind of assume that the node is converged and it is now ready. It is updated to the newer version of my infrastructure. And I think if you’re used to your process being, and then the sysadmin goes and copies a bunch of binaries, and then the DBA goes and runs some SQL scripts and then runs a query to make sure that it’s getting back the results it’s expecting, and all these things, it’s kind of like shifting left a little bit to the— my infra code. I can trust that it would do the thing it was supposed to do, but I need to know— it’s more about saying, do this bit of stuff. But I think there’s this challenge sometimes where— and I’d be interested to see when people are looking to implement this implementation of like, but how do I go in and add all these checks? Because they’re used to doing that in a manual way. They’re used to having a, you know, they’re looking at orchestration as a way of doing a script recorder of what their sysadmin usually does, which I think is different. Right? That’s not what we’re talking about here. You’re just talking about making the graph bigger.

Eric: [00:30:12] Yeah, exactly. Making that graph take, take into account the fact that there are some things that need bits of data that exist, that are, you know, exist on other nodes and might be determined dynamically by the, by the fact that Puppet ran on that node. You know, might have, might have produced some bit of information that needs to be consumed by, by another node in that, in that application. But I mean, but I think in theory you’re absolutely right. Like, and there is still, you know, there’s still that mind shift just Just like running Puppet or Chef on an individual node, you don’t want to just take your bootstrap script and line for line transcribe it into your configuration management tool, right? There’s a little bit of awareness and understanding of the way that tool needs to work, and a little bit of a mind shift that you have to go through in order to make use of it in the proper way.

Matty: If you think about it, if you say in Puppet you’re going to send notifications through your graph of saying, okay, you need to restart Apache. After I update this config file. But what you aren’t doing is you’re tying that notification together, but you’re not writing something into the service resource in Puppet that says, check to make sure the file is properly there. You’re saying, I’m assuming that if somebody told me to do a thing, that I should do a thing. It’s ready to do it. And I think that’s wrapping the head around that, where that power is going to come in. Again, automation, like you said, we’re not taking our Bash scripts we already had and just writing them in. Can, but you lose a lot of the power when it comes into that, and then you, you’ve, you’ve done it. So I, I’m— I think that’s going to be a— when we look at the next generation of what this orchestration piece means, and why I think the, um, this type of an orchestration story like what you’re talking about, which is just taking this same convergent idea and expanding it, versus saying, well, how do I just take steps that I’m used to doing as a person and make a robot do them instead?

Eric: [00:32:10] Yeah, yeah. I think the, the neat thing about the about the orchestrator today too is that it’s a way to tie together a lot of the content that already exists out there in the ecosystem. So, so like your description of your application is kind of like you can use modules that you’ve already written or that are out there on the Puppet Forge and link them together in a way that actually describes what that application looks like, even if those pieces live on, on different hosts in your infrastructure. So you get to leverage the kind of ecosystem which I think think it’s pretty cool.

Matty: So how does it work? Am I like writing like kind of an uber manifest, like a big manifest that has all the things in it, or like how do I actually do this?

Eric: Yeah, so there’s 2 parts to it. The first one is, uh, that it— which is, it’s kind of a nice conceptual split that I think, uh, is, uh, similar to the way that the, the Puppet and Chef, uh, resource providers work, in the sense that there’s one that’s sort of a description of what that application looks like. That describes what the— this is what my database server looks like. This is what my app server looks like. This is how they communicate. This is the order in which their relationships need to be satisfied. There’s another couple of— they’re called meta-parameters in Puppet language. I’m having to mentally translate in my head to Chef lingo.

Matty: [00:33:29] Oh, you don’t need to worry about it. Our listeners aren’t necessarily Chef people. They’re everybody, all the people.

Eric: I believe they’re like attributes on Chef resources. That look like, you know, that say this is going to produce a particular bit of data that will be consumed by the other one. This is the— and these are the— you basically model the relationships between those different pieces of your application component, the different components of your application. And then there’s a second part of the app description, which binds those component roles into individual nodes that are running Puppet. And those could be physical machines, it could be virtual machines, it could be containers, But that second step, that’s the binding, actually is the input to a version of the catalog compiler that actually compiles an environment graph. The consumer of that environment graph is a new service, that’s the deployer service, that actually is responsible for figuring out, hey, let me— okay, so these are the machines I need to contact. This is the order I need to contact them. And if the earlier ones fail, I’m not going to try to contact the later ones. I’m going to report that failure. To the user and abort this particular job.

Matty: [00:34:46] Okay. I was going to ask about what’s actually running this manifest. So it’s this deployment service, right? The deployer service. The deployer service. And then can that be either something you run on demand or it could also be potentially looking for just a change in a new version of the module or manifest rather?

Eric: Yeah, so normally this is, you run the deployer as either, you know, either out of CI or from a command line. Once there is a new commit that you’ve promoted into the environment that you want to change, and then the deployer will go out and make that live.

Matty: Gotcha. So easily called from the end of something like Jenkins or whatever, you’re like, okay, it’s ready to go. Now we send that and then through some type of web service call or some type of service call, right? And says, now go forth and orchestrate.

Eric: Yeah, exactly. There’s actually a bunch of interesting tech under the hood. One of the pieces is this new communications protocol that does use WebSockets, but it’s a WebSocket that’s open from each of your nodes back to the central service. And that’s the communications channel by which we say, hey, you know, now it’s, now it’s time for you to run. This is what the job is that you’re running. Um, report back your success over the, over that WebSocket.

Matty: [00:36:12] That’s very cool.

Eric: It’s, it’s pretty neat there. The, uh, the demo is pretty awesome. Uh, my, my teammate Ryan did the, uh, did the demo live on stage during the keynote. And, uh, like all live demos, uh, he kind of, he kind of flipped into, you know, Bill O’Reilly mode and said, fuck it, we’ll do it live. Yep. A chunk of the configuration that was actually commented out from an earlier trial run. So he had to, in front of 1,500 people, had to fire up vi and uncomment a chunk of, a chunk of Puppet code. So props to him for being able to pull that off. But yeah, it was pretty, pretty cool to see that.

Matty: Uh, yeah, awesome. We’ll, we’ll hunt down the— if there’s a recording or that, we’ll put that in the show notes.

Eric: Yeah, absolutely. The video is up on, on the PuppetConf site. Uh, And the rest of the session videos will be up pretty soon. I’m actually a little— I’m sure like you for ChefConf, I get to see very little of the actual conference because I’m pretty much working the whole time. But there’s a bunch of interesting, both more in-depth app demos than we did on the main stage as well as David Luederkorff who’s been around in the community for a long time. He was a Red Hat employee for a long time and wrote the— he was kind of the lead developer on the project, did a really in-depth tech talk about the technology that’s behind the application orchestrator.

Matty: [00:37:40] So speaking of Red Hat, I’m not quite sure when we’re releasing this episode, but as of now, it was last week was Red Hat acquiring Ansible. So I don’t know if you want to join me in completely talking out of our ass where we don’t know anything about what we’re talking about. About what this means to the world.

Eric: So we’re moving to the pundit part of the show.

Matty: Sure, let’s do it. And if it doesn’t work out well, we’ll just— I’ll just cut it.

Eric: Yeah, I mean, that’s, that’s pretty interesting. I think my favorite, my favorite response to that was that they probably could have gotten Robin Bergeron back to work at Red Hat for less than $150 million.

Matty: We all know that’s really why it happened, you know. So sometimes people have to be sneakier, that’s all.

Eric: From— so let’s talk about this from a couple of angles. I mean, from a business standpoint, I think it makes a ton of sense for Red Hat. It was really unsurprising because there was so much Red Hat DNA inside of Ansible from the start. It’s not an out-of-the-blue move. I think from a technology standpoint, too, it makes sense. There’s definitely some gaps that Ansible is a good glue to fill. It fits in really well with the rest of the Red Hat infrastructure ecosystem, which is pretty much all Python apps and the series of tools that have either been developed in-house and open-sourced or were acquisitions that they rolled back in, like Ansible itself. So, uh, you know, not, not, not surprising. And I think it’s, uh, it should be a good move for them.

Matty: [00:39:11] Yeah, I think that’s— I think it was a pretty, pretty easy buy. And it’s, it’s been funny kind of, you know, cruising the web spheres and, you know, people saying, well, I wonder why they bought Ansible instead of Puppet or Chef. And we’re like, well, there’s, there’s kind of not the same thing. And part of it is like, I think I don’t think either of us want to be bought by Red Hat. I mean, nothing against Red Hat, but we’re doing our own jam, right?

Eric: You know, and it also seemed like if you look at the positioning and kind of the blog posts that came out from where they talked about what, you know, what it was that they were going to do and why it is that they did this, it really is around that early lifecycle in the machines, the provisioning and the stuff that Ansible does really well, which is kind of— it’s almost like what you described earlier about taking the steps that you take as an administrator running a series of SSH commands and capturing those into a repeatable way. And when that— the fact that there’s no— you just need Python on the machine, which of course a Linux box is going to have out of the box, and you just need SSH as your transport. You just need a root SSH key in order to get in. Into it, it becomes really easy to be able to run a sequence of commands across your fleet and to do that in a really low-friction way.

Matty: [00:40:32] Yeah, and I think that’s one of the things that when we kind of look at the ecosystem that I would never deny is that the ramp-up time on Ansible is way faster than it is on Puppet or Chef. Now, in my mind, that runway might run out pretty quick, but if you’re saying, I just want to get started right away and do a thing, so if you’re talking about, like you said, being able to get into that place of saying, what was I doing before to be able to get that quick provisioning, to get that day zero, get my machine going. It’s really kind of taking that extensibility of being able to do that in a really powerful way. And then throwing Red Hat’s resources behind it and having that a little more embedded in their ecosystem. Some of the things that like come to mind to me, and again, nobody can predict, but it’ll be interesting to see what, you know, it does kind of around the larger ecosystem. Um, I know Ansible has, has Windows support, some Windows support. You know, the blog post was like, hey, no, we still love Windows, etc. I would be interested to see what that does for Windows adoption around Ansible these days and, and where that investment happens, because that’s a, you know, you, you know as well as I do that, you know, being that type of cross-platform brings a lot of challenge. And that’s something that Red Hat has really never had to do at all, right?

Eric: [00:41:52] Like, certainly they—

Matty: I mean, they have no relationship there.

Eric: Well, and indeed they have a vested interest in people not running Windows because they pay Microsoft for a license. They’re not going to buy— not going to buy a RHEL license. Yeah, it’s, it’s, it’s pretty interesting. It— and, you know, we— there’s actually a, uh, a Puppet integration with, um, Red Hat Satellite 6. They— and yeah, with them on Puppet Enterprise, there is a, you know, a great, um— there’s actually a really cool, really cool, uh, integration that we’ve got going with them that we talked about a couple weeks ago at PuppetConf, Carl Kallmann and the product manager from Red Hat co-presented on this. I don’t think that— well, they’ve stated publicly that doesn’t change that. It’s just there’s a separate piece around that command and control and command execution that Ansible fills just really, really well. I think that was the point of the acquisition.

Matty: Absolutely. I would agree. I think ultimately, we’re right. They were just trying to get Robin to come back. That was accomplished, for sure. It’s going to play out interesting. The good thing is, I do remember one of the things that I saw in some of the notes I saw from Luke’s keynote is, he talked about the fact that, what is it, there’s less than 15% of enterprises are using any of these kinds of tools right now. There’s plenty Plenty of stuff out there. That’s a great thing for those of us in the space, what he said. I’m going to paraphrase, because I was just reading notes. As I remember from my understanding, it’s a great thing for those of us in the space because, hey, there’s all sorts of customers, all sorts of people that we can help. But it’s also kind of a little— when we understand it, it’s also a little terrifying. It’s really challenging to think about that there’s that much out there that really needs, needs help and needs to be in this space. And we need to be able to figure out how we can, you know, and that’s why I always kind of look at it and I, I say, you know, when I’m not wearing my actually, where does my paycheck come from, you know, hat, I’m like, and someone says, should I use Puppeteer or Chef? I say, my answer is yes. Yeah. Like, do the thing that makes you awesome, but not using a tool like this will not make you awesome. So there’s plenty of people out there that need it, right?

Eric: [00:44:03] This is the genesis too. And this is why I, I put promote #HugOps at every opportunity that I get because I feel like there’s a small number of tool partisans who feel like there ought to be some kind of deathmatch rivalry or something like that, and it’s just not the case. It’s not— I feel like my old boss, Scott Johnston, who has gone over to Docker and runs product over there, describes that other 85% as being whitespace. In the market. It’s just empty space that’s out there. I think now, certainly more so than 5 years ago, they have an awareness that there’s a problem and that there are things out there which can help them solve that problem, but haven’t yet made the move to actually dig in, figure out what the options are, and help find something that’s going to get them out of the, uh, manual hell that they’re living in.

Matty: [00:45:08] Yeah, I mean, I think a huge indicator for me is, I mean, I’ve spent my whole IT career in the Midwest, so I can speak with authority about this, and it’s not speaking down, but we are slow adopters here in the Midwest, right? You know, and the fact that there is so much interest in this space and my calendar is so full makes me feel real good. There’s still a lot of challenge, you know, it’s like sort of like you said, I feel like it’s a lot of the Well, we know there’s a thing we should be doing, so Matt, come in and talk to us about this thing. And then we kind of talk about it, and then sometimes you get the, oh yeah, we’re totally ready for that. And then you’ll— or sometimes you get the, not yet. And that’s also my approach, is if, if I come in and we’re talking and you’re, you’re looking to want to do something with Puppet or Chef or whatever— I mean, if I’m talking to you, it’s going to be about Chef, but whatever, you might be talking about both— and there are times I can tell, you know, it can be evident that you’re not ready. And that’s not a slight, light. It just means organizationally you might not be there yet, right? And in a case like that, it might mean you got to move some stuff before you can. And, you know, I, I was talking to a customer about this exactly today, and I said, I don’t want to sell you the wrong thing because first of all, I, I do the work that I do because my life was made better because of Puppet and Chef, right? That’s absolutely true. So it’s super easy for me to believe in what I’m selling and what I’m promoting. So I just want you to be awesome. It’s great if you pay me because, you know, new iPhones and stuff don’t come for free. But, uh, and I love to get paid for doing a thing that makes me happy. But by the same token, I don’t want to sell you the wrong thing because, A, it’s going to make me feel bad and dirty. But mostly, it doesn’t even— from a business perspective, it’s bad business, right? Because if you’re not ready for it and I can probably convince you to buy the damn thing. Then you buy it, and you hate it, and you don’t implement it well. All that means is, next year, you don’t buy it again, and now you hate me. That did nobody any good. I think there’s a lot of making sure that just because it’s the new hotness, that your organization’s ready. But I think that’s way harder than the technology. But we know that, right?

Eric: [00:47:19] Matt, you have no future as an Oracle TSM. Yeah, I might get in trouble for saying that, but that’s okay. That’s, that, that is not a, that is not a moral quantity quandary that a lot of large, uh, enterprise software vendors have. But yeah, absolutely, you’re absolutely right.

Matty: What else is going around in the Puppet ecosystem that, that people should know about that they might not be aware of that’s, that’s rad and cool?

Eric: Okay, well, there’s, there’s a couple of things that have really caught my eye recently. Um, I, I worked, uh, like I said, I worked most of the time at PubCon, but I did a, I did a stretch at our demo booths there, which was actually really eye-opening for me because I got to poke around at some things that I don’t through the course of my regular job, which was really cool. One of them was Gareth Rushgrove, who does the DevOps Weekly newsletter, and is just a great, wonderful human being, first off, but also a great proponent of DevOps and of making sure that people are, like you said, both culturally ready as well as technologically savvy enough to help change their organizations. But he’s been doing a lot of work on a Puppet module for managing resources in Amazon, and Puppet Labs-EC2, I believe, is the module name. I hadn’t really seen this since the very early days of it, and the thing that blew me away is because Puppet kind of builds this model, like we’ve been talking about, about your resources, it has the ability to go out and query the state of the system and present you back with what that model looks like. So there’s a subcommand if you have the Puppet tools installed, it’s called puppet resource, puppet space resource, and you can just run that over users or you can run it over the files in your system, and it’ll literally print out Puppet code that you can save out to a file and then apply that to a different system or modify it and reapply it to the same system, or, you know, kind of has this transportable format, kind of like, um, what PowerShell does around passing around objects on the network.

Matty: [00:49:22] Yeah, no, I remember I, I, I like co-taught the weirdest like workshop ever at Interop last year with Ben Ford from Puppet. For listeners who don’t know, Ben is at Puppet. And it was really weird because they wanted us to teach a workshop that was the basics of Chef and Puppet. And we’re like, how are we going to do that? But it was fun. But he talked about Puppet, really, like I hadn’t seen it because I don’t think either it didn’t exist the last time I used Puppet or I wasn’t aware. And I was like, that’s super rad. I really like it.

Eric: Puppet resource has been around for a while. The thing that blew me away about the EC2 work is you can use that same puppet resource subcommand to describe resources in your Amazon infrastructure. You can go through the AWS web interface and build up a VPC, build up your instances, and then run puppet resource against the API. API and be presented with Puppet code, which you can then save and check into a Git repository and do whatever you want with, because you now have your infrastructure as code for your EC2 resources. And yeah, it just blew my mind.

Matty: [00:50:27] That’s really cool.

Eric: So that was pretty awesome. And I guess, let me just— one more thing too, is like, there have been some— this ties back something you said earlier about building in the health checks and the verification steps of a particular resource actually being brought into compliance into the Puppet code. Well, I saw a great talk by Dan Bode, where he talked about using Consul from HashiCorp to do exactly that, because Consul has this idea of of querying a local system for resources which that system would provide to the rest of your infrastructure. The way it does that is by, as you describe what the resources look like on that system, you build in a health check to it. So you say, this ought to have a web server running, but don’t publish the fact that the web server is running unless it starts passing this health check that TCP port’s opening and it’s returning a right HTTP status code, that sort of thing. And it does that really dynamically in a really fast, fast loop and publishes that information both out to Consul’s central key-value store, which you can query, or out to DNS. So if you have like other, other machines in the infrastructure that are using, that are using service names to get at it, it’s kind of a pretty reactive service discovery. But Dan showed a really cool integration in between Puppet and Consul that I thought was, was, uh, was pretty next level, and that actually does that, uh, that thing that you described of building the health check into the infrastructure so that, uh, into the provider almost, right?

Matty: [00:52:15] That’s how it, that’s how it knows that it converged that resource was. Am I getting that right?

Eric: Yeah, uh, well, it does it as a separate— so it publishes it through the Consul service. So Consul’s running as a separate separate daemon, and it’s doing those health checks continuously, and it updates the central registry once the health check succeeds. And then if it starts failing for whatever reason, it’ll take that out. So it becomes a very reactive, very dynamic sort of data store for what services are active on your network right now, like within a, you know, 5 or 10 second convergence time.

Matty: Very cool. Well, this is— we have— this time has just flown by, I’m gonna tell you. You know, I was like, yeah, we’ll probably only talk for like half an hour.

Eric: Get rolling, you know.

Matty: Yep. And it happens. So, uh, but thank you very much. Uh, it was great to have you on the show again. We’ll do this again. I just, uh, wanted to share one, one story that, that came to mind when you talked about promoting HugOps. They’re both actually— there are 2 stories that are related to this, and they both involve Sasha Bates. Okay. Uh, which comes as no surprise probably. But one was at DevOps Days— actually, they both involved DevOps Days Minneapolis last year and this year. So I remember this year Um, Eric tweeted— and listeners, Eric tweeted, uh, that, uh, he and Sasha— so Sasha Bates works at Chef— were talking to each other and someone, bystander, said, oh, are Puppet and Chef gonna fight now? And he said, no, we’re gonna hug, you know. And that’s the truth.

Eric: [00:53:42] And that’s how it goes.

Matty: Last year at the first DevOps Days Minneapolis, at the evening event, uh, just happened, there were a bunch of Chef folks there and we were sitting around with, with a Puppet employee at the after hours, just sort of, you know, drinking some beers, doing whatever. And it was, I mean, just in very good-naturedness, we’re sort of teasing about like, oh, you know, you’re outnumbered, but, you know, back and forth. And then Sasha, she said a very smart thing that, you know, Sasha says many smart things that I remember and quote her on and she doesn’t remember ever saying. But this was, friends are more important than where you work. And that’s absolutely true. I think that in this industry, we have a lot of respect for each other. I don’t really care where it is that you’re, that you’re working if you have smart ideas and you’re open and empathetic. And yeah. And I’ve made great friends with, quote unquote, who, like you said, there’s a small percentage of people who would think that Eric and I should hate each other because we work at Puppet and Chef. And actually, I think I love to learn from people who work at Puppet and people who do stuff with Ansible and all these things. It just makes us all better at what we do. It makes us better people.

Eric: [00:54:45] The metaphor I generally reach for is the fact that it’s all of us together in this tiny little boat trying to get across a giant sea of stupidity. Yes.

Matty: And with that, I think we’re good. So thanks a lot, Eric.

Eric: Thanks so much for having me on, Matt.

Matty: Yep. So as I forgot to mention at the beginning of the episode, this— we are also sponsored by 10th Magnitude, a cloud services company that figures if you’re listening to this podcast, you must be pretty cool. You can find out about joining their cloud services team at 10thmagnitude.com. And don’t forget about our newsletter, The Banana Stand, which you can subscribe to at arresteddevops.com/bananastand. It is the best way to know about upcoming podcast episodes and cool news with DevOps. We also have an iPhone app if you dig that kind of thing, which you can download for free at arresteddevops.com/iphone. And thanks to all of our sponsors. Be sure to visit them at arresteddevops.com/10thmagnitude, arresteddevops.com/victorops, and arresteddevops.com/datadog. Again, massive thanks to Eric for joining me. If you yourself enjoy Arrested DevOps, we’d appreciate it if you’d visit arresteddevops.com/itunes and leave us a rating or a review in the iTunes store. We would love to know what you think of this episode. Please leave us comments at arresteddevops.com/puppet. You can always find us on @ArrestedDevOps on Twitter, and we’re happy to get your input, ideas, or feedback via email at shows@arresteddevops.com. I’m Matt, @MattStratton. This is Arrested DevOps, and remember, there’s always DevOps in the banana stand.