Steven Murawski: [00:00:01] It’s time for Arrested DevOps, a high-level biweekly panel discussion of DevOps concepts, giving you a tantalizing taste of the basic technologies and ideas of DevOps to entice you to try more. Here are your hosts, Matt Stratton and Trevor Hess.
Matt Stratton: Welcome to Arrested DevOps, Episode 9, Fast and Furious: Configuration Drift. I’m your co-host, Matt Stratton, @MattStratton on Twitter.
Trevor Hess: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.
Matt Stratton: This episode of Arrested DevOps is brought to you by 10th Magnitude, helping businesses realize true agility through DevOps and cloud-enabled innovation.
Trevor Hess: Tonight on Arrested DevOps, we’re going to be talking about configuration management.
Matt Stratton: Exactly. So joining us today, we have Sean O’Meara of Chef. So Sean, what’s your favorite food-related pun with regards to Chef tools?
Sean O’Meara: Food-related pun. Kind of like the— there’s a new tool that came around recently called MEEZ, M-E-E-Z. It generates like a cookbook skeleton for you. It’s a pun on mise en place, which is like what line cooks actually do to prep their stations before they actually get to work. It’s like, oh, here’s a pile of onions, here’s a pile of Whatever. So I like that one.
Matt Stratton: [00:01:22] And we have Chris Webber of Demand Media. So Chris, as a certified puppet master, do you prefer marionettes or paper bag puppets?
Chris Webber: Definitely paper bag puppets.
Matt Stratton: And finally, we have Steve Murawski of Stack Exchange. And Steve, what do you find to be more useful when configuring Windows servers, PowerShell or a firearm?
Steven Murawski: I, well, so that depends. I used to work for a police department and the cops there, whenever the computers weren’t functioning, they’d always offer to shoot it. So that always made it difficult to return, to get stuff warranty work done on the thing. So Panasonic Toughbooks were supposed to be able to take a shot, but I never wanted to put that through the paces. So I’ll stick with PowerShell.
Matt Stratton: Stick with PowerShell.
Steven Murawski: Cool.
Matt Stratton: So tonight, we’re going to be talking about configuration management. For me, I look at it as basically configuration management is different than necessarily software configuration management when we’re talking about things like version control in some ways. But ultimately, to me, it’s creating a predictable and what I like to call kind of a trusted target, which is to say, we know what our systems look like. We’re defining what they should look like. And letting them be as much as possible self-managed. And we’re just— like I said, I really like the term trusted target because when we look at how we do things like software deployment, it’s really hard to automate deploying software in a safe way if you don’t know what the systems look like that you’re sending them to.
Chris Webber: [00:02:57] So I kinda take a different approach when I talk with people about it, especially in terms of the way Puppet DSL is written. I really describe it in terms of configuration management to me is executable documentation that keeps reinforcing itself. So we take the— this module that has an exact layout of what’s supposed— that system’s supposed to look like, and it’s well documented as what it’s supposed to look like. So that’s kind of the angle I usually take with people. As far as it’s executable documentation. You now know what the system is always supposed to look like, assuming no one SSHs to it and fixes it.
Steven Murawski: Yeah, I actually like the executable documentation line as well. And the thing I like to stress around configuration management is it’s an ongoing process. It’s not something that happens once like a setup script. It’s something that, you know, will watch for configuration drift, will bring things back. But it’s only— and this is something that, you know, a lot of the audiences I talk to that are just kind of getting into this space miss— is it’s only explicitly configuring what you tell it to configure. If I go and turn something on or on a server that is not explicitly being configured by my configuration management, I have no visibility into that. So it’s the things that we’re being intentional about configuring on a system. In either the positive sense, like asserting something’s there, or in a negative sense, as in, you know, we don’t want it to be there.
Chris Webber: [00:04:37] There’s one other kind of direction I’ve taken this as well when I talk to people, and this is, I think, a little more true of Chef than it is of Puppet or Ansible. Some of the other tools, but the idea that you take a step back and start treating your servers as software objects that have just really crappy APIs. Chef and Puppet become this awesome API to managing the software object. So if you can start thinking of it in terms of, oh, I just have this software object that happens to have a really crappy API and I can use something like Chef or Puppet on top of it to manage what it does instead of this, oh, we went and configured this thing. Well, it’s an object. It’s a software— it’s, you know, something you’re interacting with in code.
Trevor Hess: I really like the documentation description because for me, that’s— I’ve only started using configuration management recently, but what it’s allowed me to do is instead of saying, well, What did we do to that one server to make the thing work finally? It’s all, it’s all there in the scripts to set up and update and maintain the servers, and there isn’t any lost knowledge because it’s, it’s all there.
Steven Murawski: [00:05:55] Trevor, that’s a huge point there. Uh, you know, I talked, I’ve talked to a lot of guys who work in medium-sized and small shops, and they’re like, you know, I don’t need this stuff. I only have a handful of servers. Until the point where you have to reimplement that system or hardware fails or you have to expand the environment. And then it’s, okay, what did I do 6 months ago, a year ago, 4 years ago when I stood this box up? What was that one trick to the registry key that I had to find or the service I had to disable or whatever config file that I had to tweak to make this thing work? And not losing that and surfacing that and having that under change control. That’s one of the huge benefits of this of these types of platforms.
Trevor Hess: And it’s really great because, for example, we were trying to enable WebSockets on one of our server setups a few weeks ago, and I got pushback from one of my coworkers, you know, saying, why, you know, why should we try figuring out how to write a script to enable WebSockets? We can just go into the Server Manager and do it. And I finally got him to agree to do it, and it was 2 lines, and he’s like, oh, that was way easier than I expected it to be. And it’s like, yeah, now we don’t ever have to worry about it.
Steven Murawski: [00:07:11] Well, even if it is a complex script, if it comes down to, you know, once it’s done, it’s done. And now the next time you have to do that, you don’t have to do that whole thing again. You know, even if there are multiple steps. And that’s actually one of the things is it starts to abstract away the, you know, those huge multi-step processes and bring you just kind of a surface view of, okay, what is the end result of those things?
Sean O’Meara: Like most things, tend to overthink this kind of stuff. So, I kind of give config management a slightly more generic definition that includes basically anything you can possibly do, any techniques and strategies for managing configuration and its complexity, you know, and things like manual configuration and execution management and all those that non-policy-driven executable documentation, CFEngine-derived stuff, gets to live underneath the configuration management umbrella as well. That was just me.
Matt Stratton: [00:08:18] So, in our experience, when we talk about what business problems are we solving with configuration management, I mean, I gave the example of being able to Or I alluded to, I guess I can put it into more detail, to be able to speed your automation of your software delivery because you’re having a trusted target. So I think that’s one business problem that I see that it solves, is it’s an enabler of more automation in your delivery process, and then therefore reducing cycle time and shortening a feedback loop. But those aren’t the only business problems we have, right?
Chris Webber: So I basically, going way back, when I first implemented Puppet, the simplest thing was just time savings. It was, how can we get it so that the SSH config is the same on all the boxes? And very quickly, it’s funny, the Puppet aspect of things didn’t actually win over a lot of people. It was the stored configuration that won everybody over because all of a sudden, because we had the Puppet client running on all of the boxes, I now had a database that I could go query information about those boxes and write— I wrote a couple facts right off the bat that answered questions about, well, what does this box do and what, you know, those types of things. So you get the CMDB side of it and, you know, for me, in the enterprise, in the traditional IT shop, it solves problems more around user access management and things like being able to do auditing and all of those kinds of like just standard IT things. Then when you get into the development side of the house and the engineering side of the house, it’s the DBAs now get a box that looks exactly the same with all the users because if you’ve ever installed Oracle, you know that there are like 30 users and all sorts of crap that has to be done just to get it so that they can install the database. Like, I’m not even talking installing Oracle, I’m just talking the baseline to be at a point where they can install Oracle. That was a huge win when I was at UCR. Now it’s, you know, we’re talking about going the Netflix model and, you know, pre-baking AMIs, and it is a DSL that lets us quickly iterate on what those things should look like, and we can take it from Vagrant to Packer to all the other things. So those are the kind of the big business problems I see it solving.
Matt Stratton: [00:10:46] I was thinking too, when you talked about being able to give the boxes to the DBAs that were set up that way, I think a great advantage of using configuration management is again predictable environments. And again, coming from the TechOps background, you know, it’s— it can solve for you the whole, well, it worked in QA, why didn’t it work in production? Or hey, it worked on my laptop. That’s the worst one. Right? You know, I work on my laptop. Why didn’t it work in QA? And config management lets you extend that to say, hey, if I’m developing an application, I don’t have to worry about configuring, you know, a LAMP stack on my laptop or getting IIS configured the same way that it’s supposed to be or all the pieces and parts. I can use a local virtualization solution to get myself a machine that looks just like production except for scale. And then I know it will work all the way through the process. I’m a believer that developer laptops, outside of the VMs running on them, they should be used for email.
Trevor Hess: [00:11:47] I agree.
Steven Murawski: So the other, you know, kind of tack along there, the other business problem that I see config management helping with is I have an app that behaves differently on one box than the next. And if I have, you know, 3 boxes are behaving the same way and 1 box that’s not, well, something’s out of whack on that other box. Config management narrows down where that variation can be. And rather than, you know, maybe I’ll dump the logs off that box and reimage it because it’s going to take me, you know, 20 minutes to get a new physical box into, you know, with an OS and fully configured and ready to go and lay down the config management on it so it’s ready for apps versus going in and spending, you know, and spending 2 hours troubleshooting the thing to maybe try to bring it back into compliance. I can return it to a known good state much, much faster, and I can grab the logs for troubleshooting at a later time offline when it’s not causing a problem.
Matt Stratton: [00:12:51] Like they say, it should always be easier to build a new environment than fix a broken one. And you can’t do that if you don’t have config management, because if you’re doing everything by hand, it can take days or weeks. Or I’ve seen too, like you said, where you see that drift, and that’s where, you know, the name of our episode comes from, right? Configuration drift. You know, where you run into that scenario where you say, okay, so I got this one server in production, I’m going to go and maybe I’ll take it out of my load balancer so it’s not serving traffic, but I’m trying to troubleshoot and I want to use a node that’s in production. But then you forget that you gave someone access to it so they could troubleshoot it, and then you put it back in there, and now you don’t know that that one of your 300 web servers is different than the other 299 until weeks or months later when it’s misbehaving and you have no idea.
Steven Murawski: Right, and that’s, you know, and that’s where config management is kind of one piece of the puzzle, you know, and try— it’s trying to get away from giving these, you know, having a reason to interactively interact with a particular machine. Because I forgot who said it first, but, you know, once somebody interacts, you know, logs in interactively on a machine, you don’t know the state of that machine.
Matt Stratton: [00:13:58] And it could be— You just beat me to that one. That’s Mark Burgess.
Chris Webber: Yeah.
Steven Murawski: Yeah, because, you know, there’s even, you know, even if the person just logs in, you know, there’s startup scripts that could run or Group Policies that could apply if you’re in a Windows environment, or any number of things can happen to change the state of that system. And if that impacts the application that’s being served or the purpose that that machine’s serving, you know, now you have that one bit of variation. So, you know, config management, one piece of the puzzle, then you have to be able to replicate the other services of why somebody would need to get onto that box. You need to provide troubleshooting tools and rich logging and all, you know, all sorts of other capabilities around that to keep people off the boxes.
Matt Stratton: I had an experience once when we were first— I was at a shop where we actually were implementing Puppet, and I kept burning myself because I was the one who put it out there, and we were using it to, in this case, to manage some of our CMS platform. And I would sit there and then, you know, I was troubleshooting something. I, you know, kind of had all the deployment going through the configuration of the systems through Puppet, and then one of the developers comes over, one of the content developers comes over and says, hey, you know, the images are all showing broken in QA. And I went and I looked at, you know, of course, what do I do? Because in my defense, I’m new to this. And so I, you know, SSH in the box and I take a look at it and I’m like, oh, you know, I’ve got this alias is wrong, you know, the symlink is wrong, I’ll fix that. And I say, hey, is it working? Oh yeah, it’s great, you know. And then 15 minutes later he comes over, goes, it’s broken again because we had another Puppet run, it came back and it undid everything I did because I made the change manually. And I think that’s a, that’s a really hard shift you have to make that you can’t do these things manually because your config management tool will actually potentially come and undo what you did if you don’t make the change through your configuration management.
Trevor Hess: [00:15:51] Absolutely. If you’re not used to that, not used to that process, and your gut reaction is still to, to go in and change it because you know how to change it in the interface, or you have a script sitting on the server that you would run, or you have a set of, you know, any number of different things that you are used to doing, you’re gonna break your state.
Sean O’Meara: Yeah, this is why the term infrastructure as code is also pretty popular, right? Because it kind of like beats home the fact that, you know, like you should be treating the policy description of your machines the same way that you would anything else, like in, you know, like the software kind of world, right? So like you edit files and put them in version control, and you test them, and then you deploy them, versus, you know, logging into machines and manipulating them with your fingers, right? Which is one of the— which is the way people have done it since, you know, like the beginning of time, right? A cool side effect of this is, like, so if you look at a number of the strategies that typical operations shops employ to actually, like, reduce the amount of complexity in their environments, one of them is to be a monoculture when it comes to, like, you know, their servers. So, like, we are a Windows 2003 shop, or we are a RHEL 5.8 shop. 5 shop, and it will be that way until, you know, the next 5, 10 years comes along and we need to refresh everything. And configuration management comes along, and suddenly, like, that monoculture is not necessarily that important anymore. So if you have all of your, your server configurations described in a CM language, right, the upgrade between RHEL 5 and RHEL 6 Yeah, you’re gonna have to shake a couple bugs out, but you know, you’re starting at a really good place where you can go in and just like figure out what the differences are between the 2 systems and adjust your policy accordingly instead of having to go back and be like, oh, we need to boil the ocean, so like we’re going to be stuck on this platform for no reason, right? And it’s a huge bummer when you want to run software and you can’t because you’re on a 10-year-old operating system, you know. And I think CM tools alleviate a lot of the reason people like to stick with kind of older, kind of older operating systems, which is kind of cool.
Steven Murawski: [00:18:32] Excellent point. I, I, that is one of the biggest selling points for config management solutions like, you know, like Puppet, Chef, or Salt, or CFEngine because number one, you make it easy to test out the new OSes because it’s easy to get that replica of your environment down. And then 2, it’s easy to tweak it because you’re following these— you’re following practices of you have version control, so you’re not— you don’t worry about making a tweak or change to something and testing it out because you can always get back. You can always see that history of the changes you’ve made. Enabling that capability to upgrade OS or migrate platforms and have a strong starting place is absolutely huge.
Sean O’Meara: Yeah. And another kind of related topic is— so something that’s been— I forget who coined this immutable infrastructure, immutable servers term, but I kind of hate it. Well, I mean, the whole point is, like, okay, we’re going to build this system, right, and then never touch it until, like, we want to completely rebuild it. That’s fine. Totally cool. Do that. But it doesn’t make it immutable, right? It doesn’t mean, you know, like, something can’t change on it, right? And very, very often inside of an actual cluster of machines or, like, an infrastructure with a capital I, like, you need to make little tweaks, right? My favorite one is, like, let’s say you have the classic load balancer pool, right? You want to add another machine to the pool. Okay. Are you going to rebuild your load balancer, or— no, you’re going to go in and adjust the 4 bytes of— however many bytes of data, 32 bytes of data in the load balancer config that you need to point it at this new machine and not completely throw the thing away. What if you’re doing local packet filtering between all the nodes in a cluster? You just need to update one firewall rule. You’re going to completely destroy the machine and push another 4 gigabytes of data down a pipe and bring up a new one for every machine when you need to change literally one line of config on all your machines? It just seems a little over the top to me.
Matt Stratton: [00:21:02] And I think it’s— if I’m following you right, and I would agree— is that you could take this, what was probably a good idea and take it to an insane conclusion. Like you said, I’m going to blow everything away because I’m making a small change. To me, and I think where everybody is kind of getting the idea, as usual, is from Netflix, right? You know, where they— the concept is when I’m releasing software, instead of deploying it to existing infrastructure, I’m going to deploy it to new stuff, and I’m going to do a canary release. I’m going to bring up some new machines. I’m going to deploy to these new machines, send 1%, 2%, 3%, da-da-da-da-da, and I basically just destroy the boxes that had my old code on it. And to me, that makes a lot of sense because it’s a safer way to do it. But I also don’t think that necessarily— like you said, and maybe if you’re someone like— if you’re doing the 10 deploys a day, then yeah, your machines don’t live long enough that you have to manage them very long. But most people aren’t doing that.
Sean O’Meara: Well, I mean, they should be, right?
Matt Stratton: You know, they’re getting there. They’re not—
Sean O’Meara: [00:22:03] I like to include other things under the config management umbrella is because like People like to— everybody’s excited about Docker right now, right? So, containerization, that’s actually execution management, if you nerd off on it for long enough. It’s not mutually exclusive with the convergent policy description-based configuration management at all, right? Somehow like to think that they are or they should be. Like, no, like you can totally apply a Chef recipe or a desired state— what do they call them in Windows land?
Steven Murawski: Configuration. Desired state configuration.
Sean O’Meara: Yeah, yeah, or Puppet module.
Matt Stratton: You can apply DSC to a Docker container because you can’t containerize Windows, right?
Sean O’Meara: Oh, there’s nothing stopping you from building that with your config management. Yeah, then like the snapshot and throwing it in like your EC2 load balancing pool at all. As a matter of fact, you should do that, right? You don’t need to start at that base state and spend 15 minutes converging a node every time you bring like a new image online. You don’t have to, as long as the details about the image are captured in your config management policy. That’s great. Like, that’s what you should be doing. Then, when you want to upgrade to the latest Amazon Linux or whatever is going to come out next month, fine. Do the small amount of debugging on your config policy that you need and have fun.
Matt Stratton: [00:23:42] Yeah, and I agree, too, where you talk about you don’t necessarily have to start from nothing. There was— actually, it was the most recent episode of Food Fight. Lucius was talking about this a lot, and he— I don’t know if he coined it, but I’m going to credit him with it— the term a leaden image as opposed to a golden one. It has to be turned into gold, but it’s almost there. And we were talking about it today, and one of my coworkers said, you know, I would prefer the term brown and serve because it’s not quite baked. So we’re going to start talking about brown and serve images instead of living ones.
Sean O’Meara: Images aren’t bad. It’s the loss of the resolution about the details of the image that’s bad, right? So like images became bad. Like, back when everybody first got their shiny new VMware cluster last decade, like, the first thing they would do is they’d log into machines and they’d start manipulating with their fingers, right? They’d make with the clicky-clicky and, like, type the box and do the thing, and they’d snapshot the machine and treat that image as the artifact that they’re using. Like, that was the config management strategy that they were using. And the reason that’s bad is they’re losing the history of the configuration It’s baked into the image, but it’s nowhere else, right? They can’t reproduce it easily. So, you’re stuck with these, like, forks of these images and stuff that you can’t even update, right? So, yeah, fine. Use images all day long. Don’t spend half an hour building new VMs when you don’t need to, but you need to have the details captured somewhere, and that’s what configuration management is. It’s that executable documentation, that policy.
Chris Webber: [00:25:22] Exactly. And so the way I’ve been kind of communicating this to some of my developers so that they understand like what’s going on, because they get frustrated, it’s like, well, I need to go write some Puppet code and make this happen and push it out. And they’re like, why do you need to? Just, you just go in and yum install. And when I explain, okay, well, think of it in 2 ways. Number one, I go and modify config on that box, it’s effectively like me going and modifying what’s running in production in your codebase and not checking it back into version control system, like it doesn’t make it back into the repo. Or you’re— the other way to think about it, if you look at the system as the software object, it’s effectively like going and connecting to using like a debugger to the running software process, making a change to an object, and then disconnecting. There’s no record of whatever happened there.
Matt Stratton: It’s—
Sean O’Meara: yeah, it’s like, I’m going to log into your web server and hand edit this HTML file. Okay, like, so why should my NTP configuration be any different than your HTML file? You know, like, stop it.
Matt Stratton: [00:26:31] So everyone here kind of has a different— from a tools perspective, kind of comes from a different background. We’re all talking about the same kind of thing, but one of the things I thought would be interesting, and this is certainly not a bake-off or a competitive kind of thing, but feel free to throw down if you want. Just wanted to talk about— so we kind of are saying, like, I know Sean is from Chef, Chris is Mr. Puppet, and Steve is like one of the thought leaders, I would say, around PowerShell DSC, or Desired State Configuration. And I kind of wanted to talk a little bit for people who don’t know, so what’s the difference? And there’s other tools, there’s CFEngine, there’s Ansible, there’s Salt, there’s all sorts of other stuff, other ways to do this. What do we see as kind of the big differences between the 3 ones? Not that— because people might be wondering. You hear about it, you’re like, hey, why are they different?
Sean O’Meara: Yeah, I’ll take this one to start, I guess. So something I’ve always liked to do, because I kind of have a long CI background— I’ve used CFEngine, I’ve used Puppet, I’ve used Chef. Played around with Ansible. So I’d like to start this conversation off by kind of like showing how they’re the same, right? And then talking about the differences, right? Like what makes them CM, like for real, right?
Matt Stratton: [00:27:52] So what’s the joke? Should I use Puppet or Chef?
Chris Webber: Yes, right? Exactly.
Sean O’Meara: What are the differences there? And what makes config management tools, like the modern crop of them, kind of all the same is Like, they’re all kind of derived from this idea that was first delivered with CFEngine of convergent operations, right? So, like, test and repair mechanisms, right? So, every instruction that’s executed on your machine for the purposes of actually configuring it, right, it’s a test and repair mechanism, right? Like, I’m going to test the state and then take action to repair it if I need to, right? So, test and repair, test and repair. If you look at what CFEngine Promises, it’s exactly that. It’s a test and repair. And then along came Puppet. Again, test and repair, right? A lot of people like to call this idempotent. I think they’re wrong, but, you know, whatever. So, I’m not going to actually write this file, because it looks the way it needs to, so I’m going to move on to the next thing, right? And I’m not going to install this package, because it’s already there. I’m not going to attempt to start the service, because it’s already running. I’m not going to install the user, because it’s already correct. That’s what makes all the tools the same, is because they all behave in this way. And what the DSL provides, right, the DSLs actually restrict the instructions that you can execute on your machine to these convergent test and repair operators, right? And then you build policy by putting these into groups, right? So, like, promise bundles, and Puppet modules, and Chef recipes, they’re all just names for groups of these test and repair operators, right? So that’s how they’re the same, right? How they’re different is they have different attitudes around, you know, like ordering, like how is that accomplished? Just there’s different philosophies around different kind of like nuance and subtlety around it, about various ways that they think that this should behave.
Chris Webber: [00:29:57] Right?
Sean O’Meara: And then, of course, you have, like, the language stuff, right? You know, like, oh, like, Chef’s Ruby, and a lot of people detest Ruby with the blaze of a thousand hot suns, right? You know, so they’re going to go to Python, and because it’s written in Ansible, right? Like, completely overlooking the fact that the meat of it is, like, you know, YAML, right? You know, and the Windows stuff, like, it’s really cool, but if I want to write a new you know, resource type for it, I have to break out the .NET to actually do it, right? You know, so that’s, so that’s kind of how they’re all like the same and different in my mind.
Chris Webber: So the way I normally, because I get this question a lot, the way I kind of answer that question obviously is yes, just pick one and go with it. The big place I kind of differentiate is with Puppet you are more limited in what you can do, so you have less of an ability to shoot yourself in the foot. But at the same time, with— and there are some interesting things about the way it handles ordering that’s a big differentiator as far as the way the graph works. So I generally point out to people when you’re talking about Chef, so Chef uses a run list, and if you’re an idiot like this guy sometimes, and you put your SSH after your, say, your web stack, and your web stack blows up while you’re provisioning a box, and your SSH config didn’t happen first, you now can’t get into the box. That’s an example of one of the problems you can run into. But on Puppet, the way the graph works, in concept, if you, you know, once again, if you do it right, SSH doesn’t depend on the web stack, so if the web stack blows up, it blows up separate from SSH getting completely configured because the graph handles separating those concerns. On the flip side, because Puppet is limited, you can’t do things that just will drive you batty, so, like, string manipulation, which seems to be 90% of configuration management sometimes, is very difficult to handle in Puppet because you don’t have access to native functions like gsub and all of that goodness. So those are the kinds of things that I differentiate them on. And generally, and I don’t know that I agree with this anymore, especially after having some time to kind of hang out with some Chef folks at SCaLE this year, but for a while there, I absolutely, for For older, old-school ops people, I tended to push them towards Puppet. I don’t know how true that is anymore, especially with Ruby 1.8.7 being deprecated and the Omnibus installer being amazing. But I absolutely, for devs, I almost always say, you know, go look at Chef. It will, it’ll be a little bit more of a natural feel to you.
Steven Murawski: [00:33:02] Well, I guess that leaves me up here to talk a little bit about DSC. And so I would actually say that DSC probably isn’t ready for prime time yet as a standalone config management thing, and I don’t necessarily know if that’s really where it’s headed to be. So the primary part of Desired State Config is the local Configuration Manager, the agent on the box. And basically what that is, is that everything that has Windows Management Framework 4, which can be Server 2008 R2 and up or Server 2012, it’s there by default. 2012 R2, I’m sorry, it’s there by default. But basically every Windows box will have a Config Management Agent available to it on the box. And it’s accessible over WinRM. There’s an API and a standard way to define a configuration and send that on down the pipe. And that agent can be leveraged by whomever, whatever, however. And that is— that’s really the key part of Desired State Configuration, that there’s this, you know, a standard way to manage Windows boxes, and that the hope is that other config management tools will embrace that and leverage that so that way when resources are written for a Windows box, they can be leveraged by Chef or by Puppet or CFEngine or Ansible or whomever because they can all connect in and because the local configuration manager, that local agent is doing the work, it’s, you know, it can use those shared resources. And then the DSL that’s used to generate those configurations can be whatever. It can be, it can be Ruby, it could be the Puppet DSL, it could be what CFEngine uses, whatever. That’s one of the big differences there in DSC. Now there is some basic infrastructure in DSC to actually kind of do a self-contained thing, and that’s something I’m playing with at Stack Exchange. Number one, because I have the latitude to do it, and so I can kind of— there’s some stuff there so that because there isn’t any product out there currently that uses the Local Configuration Manager, there’s kind of this minimum viable product worth of infrastructure around it to be able to— there’s a PowerShell DSL to generate configurations. There is a minimal pull server to distribute them and then an imperative way to push those configurations out to machines. So there’s, you know, enough functionality in there to kinda highlight it, to prove it out, to test it. I don’t necessarily know if— unless you’re a total Windows shop and you can’t stand up a Linux box in your environment and you can’t use a hosted service, I don’t necessarily know if I would push anybody towards being solely a DSC-style shop.
Matt Stratton: [00:35:57] You see DSC working— and this is kinda my thought, and I’d like to see if you think I’m on the right track— is working in concert with letting something that’s— it’s almost like you said, it’s the agent piece, right? It’s something that something like Chef or Puppet can kind of dish off to and handle it safely. Then, like, today, right, I can write a Chef recipe that’s going to execute a PowerShell script, but it’s not going to be idempotent. Sorry, Sean. Because it’s just going to tell me, did the script run or not, and what was its exit code? But it’s not going to— I’m going to have to put smartness in my script, whereas if instead of that I’m calling some type of DSC provider, DSC is going to then be able to come back and basically say, yeah, everything’s the way it should be. That’s kind of how I’ve been seeing it, and I think one of the things I like is, you know, doing a lot of work in Windows places with Chef is it’s just the stuff’s not necessarily always there, and Windows without DSC doesn’t always lend itself to being configured in a really nice way.
Steven Murawski: [00:37:05] Yeah, yeah, start— starting with Server 2012, the reach of the PowerShell ecosystem really grew. You could do a lot of stuff on 2008 R2 and earlier. It was somewhat painful, and you were reaching into WMI a lot to do things, or you’re calling certain .NET APIs or classes and You know, once they hit Server 2012, they started— they restructured a lot of the internal APIs, and they came up with a way that you could layer basically a config file and auto-generate PowerShell commands off of those APIs. So teams that didn’t necessarily have the time to spend building standalone commands could expose a PowerShell management surface very easily. And in a very standard way. So, and that influenced too the structure of their APIs that they rebuilt. That was all part of that plan. So you went from having like around 200 commands to cover things to 2,400 commands covering, you know, your network and your firewall and Active Directory and all basically, you know, top to bottom. There’s very little that is untouched now by command line coverage on Windows Server, but that’s Server 2012 and up.
Matt Stratton: [00:38:28] And that’s the OS. And I don’t want to go too much in the weeds on DSC, although I love that we’re talking about it because it’s new and it’s not the same old stuff. And I know we’re putting Chris to sleep here.
Chris Webber: I actually have a question around this because this is— I’ve done— ironically, I’ve become the token Windows guy. My environment. But so I did a bit with Puppet on Windows 2008, and I’m gonna bite my tongue a little bit, but so my question to— and this is probably for Sean and for Matt as far as how does Chef interact? Because one of the things that killed me with Puppet is you get in and it’s like, oh, I want to set up IIS, except That is so not intuitive because you’ve got to go grab this module that handles DISM roles and you have to do some magic to figure out which of the 50 million DISM roles you actually need. Then the cool thing is that the web.config file actually gets created as part of your application in most cases, but that doesn’t handle configuring the initial site. I think my biggest problem with Windows has always been, when I’ve gone to do it, is the disconnect. When I’m on a Nix-based system, any one of them, whether you’re in Chef, Puppet, Ansible, you effectively have this package config service trifecta that, I mean, literally HTTP is package httpd, /etc/httpd.conf, /httpd or whatever and, you know, service httpd. And I’ve never been able— like, that hurdle, even with the CM tools, I’ve never been able to get over other than with lots of bourbon.
Steven Murawski: [00:40:24] So you’re working too hard, man.
Matt Stratton: Yeah, so I got a couple, couple thoughts because this is— and again, as much as we say we don’t want to like trash on stuff, but I spend most of my day configuring Windows systems with Chef and trying to make it work. And there’s a couple things that get in the way, and this is where it would go into a question I have about DSC. So one of the things you mentioned that makes this a big challenge is there is no real good package management for Windows. So it’s super hard when you want to talk about getting a piece and part there. It’s fine for all the stuff that’s part of the OS, because you can use features, and like you said, the question about Chef, it’s pretty easy. The Windows Cookbook exposes these things super easy. IIS is actually— and I haven’t tried to do— to be fair, it’s been over a year since I’ve done anything with Puppet on Windows. One of the main reasons that I kind of initially started doing more with Chef is that I couldn’t do anything but POSIX-based ACLs when I was using Puppet, and that caused a bit of a problem for me. Chef could handle it, and therefore, that’s where I started down that road. Configuring IIS is not terribly challenging with Chef because There’s an IIS cookbook, but it’s still not as easy as you would think because it’s still running AppCMD. It’s still just doing— it’s sending commands, and it will send them over and over again, whether they need to be run or not sometimes. But then things that really start to get to be a big pain, and this is leading into what I’m going to ask you, Steve, is say SQL Server. There is a SQL Cookbook, a Microsoft SQL Cookbook. I don’t remember if, Sean, I may have actually bugged you about this already. I don’t remember if I sent you a note or not. It works great with SQL Express because SQL Express is one EXE that gets run. If you want to install SQL Server Standard or Enterprise, it’s a whole bunch of files that have to be pulled down. And it, you know, you can’t just say— I mean, again, SQL has a great unattended install. It has a silent install, and we’ve done stuff where we’ve broken out, you know, the config.ini for it as a template in Chef and populated that with attributes, and that’s all fine and good, but it’s certainly not— it’s not going to go and change, right? It’s great for installing SQL the first time, but it’s certainly not going to go back in if somebody changes something in the core configuration of the SQL Server, it’s not going to fix it. And so my question to you, Mr. DSC, is what is the support for the non-OS-related things in Desired State Configuration these days?
Steven Murawski: [00:43:03] So there’s not a ton of official support yet. So this product officially just RTMed, you know, like 6 months ago. And the way things tend to go, in my experience at Microsoft, is that until something’s around as a feature, other external teams won’t take dependencies on it until they know it’s actually a thing. So, like PowerShell. When PowerShell first came out, Exchange was one of the first big people, big customers to leverage it. And it was a lot of work to get them to buy in on that because they were building the PowerShell management infrastructure for it as PowerShell was still working its way to RTM. And so there’s not a— the total dev time for DSC to begin with was a very short cycle. Between Server 2012 and Server 2012 R2, you know, that was like about a year. So it was a very short dev cycle for the OS. So feature-wise, there wasn’t a lot of time for a lot of internal teams to take dependency on it. Now that it’s actually a feature that is part of the OS, that’s there, it’s on by default, other teams can start building on that. Now, product teams at Microsoft, whether they’re part of the OS or whether they’re external like SQL Server, they respond to customer pressure. If their customers aren’t asking for it, it’s not as high a priority for them. I know that there are— that internally the PowerShell team has been very active in lobbying for the use of DSC around Microsoft, but it’s customer voices that really start getting it driven. And I know I’ve asked for DSC support around SQL Server, but There are some community resources already around SQL Server. I curate the PowerShell.org community repository.
Matt Stratton: [00:45:16] We’ll put a link to that in the show notes.
Steven Murawski: We have some resources now growing there. The PowerShell team has actually sent out a bunch of their demo resources. Everything I’m building, pretty much we’re putting out there. We’ve had some other contributions as well. But this is an area that we want to grow. And, you know, if the product teams are slower to get stuff out, well, I’ve got needs I need to deal with now. So, you know, SQL Server is one of them. So there’s some basic stuff around the install now. But as far as like ongoing management, that’s, that’s a need that needs to happen. And, you know, Matt, if you want something around that, file an issue on GitHub. Seriously.
Chris Webber: ’Cause that’s—
Matt Stratton: Absolutely, I’m—
Steven Murawski: I mean, that’s, yeah, I’m getting more and more interest from people, you know, in that space and in playing around with it and wanting to try the stuff out and try building some of these resources and, you know, it’s a wide open space right now, but it’s a very, very young space, so.
Matt Stratton: [00:46:20] Cool, so I want to have another question. To kinda throw around there. So we’ve been talking about the advantage of it. We actually got kinda deep on some of the different ways the tools work. So what are— so let’s say, alright, someone’s listened to this podcast, they drank the Kool-Aid, they said, alright, config management starts awesome. How do they start?
Chris Webber: So the answer really becomes it depends on the tool you start with. If you are on Chef, you are going to— and I know I’m I’m the Puppet guy, but I’ve been doing this a lot, answering these questions a lot. So Chef, there are some amazing tutorials on, I wanna say it’s learnchef.com, and Puppet Labs has a set of tutorials as well. And those videos and those, and I wanna say Puppet actually provides a Vagrant box to get you started, which may or may not make life easier for you. That’s the way to start is to just jump in and and go to the 2 sites. Obviously, I’m just saying Puppet or Chef, and then play with it. You know, it’s just like code. The best thing to do is actually write a little bit of it and see what happens.
Trevor Hess: [00:47:35] Yeah.
Sean O’Meara: My answer to that has always been start small, you know? Like, this stuff does have the power for you to, you know, like, When wielded properly, you can do amazing things, right? Like, you can run disaster recovery scenarios in the cloud and restore everything. You know, you can literally automate your entire business with these tools, but that shouldn’t be where you’re starting, right? You should start as small as possible. You should pick one class of machine and try to model it. With your tool, start there, right? So you can take the original one and throw it away and bring a new one up, that kind of thing. One thing I would recommend to anyone using any tool from any community is start trying to— like, don’t try to, like, use a lot of the community modules at first. A lot of the community Chef cookbooks, a lot of the community Puppet modules. I’m not really sure what the Ansible landscape looks like right now. Start by writing your own. There’s a couple different reasons for this. The first one is, if you don’t, you need to be an expert in both that technology that the cookbook or the Puppet module represents and the tool itself. To figure out what’s going on. And that is a huge barrier to entry for people when they go to run some random cookbook and they see some random error and they’re like, oh, what’s this? I don’t understand it. They just get frustrated, right? Versus if they start to write their own and it’s like, oh, this is really easy. I can install a package and drop a file off and start the service. Like, I get this. You know, it makes that learning curve much less steep, right? So my advice is just to start small, but jump in and start small.
Chris Webber: [00:49:38] That’s amazing advice. I keep wanting to like mash a +1 button, but I don’t have a +1 button to mash. And so that’s what I’ve actually done is I generally, for very specifically for people like in the sysadmin, the Linux and Unix sysadmin world, I generally suggest, okay, we’ll start with something that you know is going to be the same across all sorts of things like SSH. If you’re coming from the academic environment like I’ve been in, you’ve got 200 machines and maybe 100 of those are running the same— I’m sorry, maybe any given set of like 5 of them have the same application on them. Otherwise, they’re all their own unique snowflakes. So I always find, you know, find something that’s gonna be the same everywhere and you don’t have to do a whole lot of, uh, what is it supposed to do? So usually I suggest SSH or hesitantly sometimes resolv.conf, but that can get you into trouble too.
Matt Stratton: [00:50:42] Everything’s a frickin’ DNS problem, right?
Sean O’Meara: Or DNS. Don’t try to start DNS.
Chris Webber: I absolutely— one of the first things I do when I build out a new Puppet infrastructure is I create— so when I’m like setting up how the Puppet config should look in the Puppet module, the Puppet module to manage the Puppet config on the box, I add a host entry that points back to the Puppet master that it— that’s responsible for serving that box. Because I know that if I screw something up and resolv.conf goes wonky, at least there is a host entry on that box that can get me back to the Puppet master to fix it.
Matt Stratton: It can always go back to the mothership.
Steven Murawski: Yeah, just to kind of tack along the direction suggested by Sean and Chris there is to also, instead of picking a single server, maybe start with your server deployment checklist if you have that in your environment. You know, for me it was, okay, we had on the Linux side of things when we were deploying servers, there were like 4 things on our checklist. You know, is it in our CMDB? Is it in monitoring? Is it, you know, and then is Puppet enabled? And I always forget the 4th thing.
Sean O’Meara: [00:51:55] I don’t know why.
Steven Murawski: On the Windows side—
Matt Stratton: that’s why it’s not a checklist.
Steven Murawski: Right, exactly. On the Windows side of things though, it’s like 30 items. And so my velocity in being able to deploy Windows Server was much, much lower. And granted, we do a few more things on those because they’re specifically tuned for particular workloads. But as things have grown, we still do a lot more on— or we’re doing a lot on the Linux side of things as well. It’s just all masked by Puppet. And so what I started doing was I started building resources because there’s only a handful out of the box in DSC. There’s nothing— nothing else existed. I started this back when DSC was first announced in the Server TAP program, so it was in the early adoption program. So I started building the resources to do the things that I would normally do in a deployment script and, you know, and making that match the pattern for Desired State Config. And that was how I kind of got started. And I started, you know, with one server and then how do I get this to multiple servers and kind of building and learning from that.
Matt Stratton: [00:53:04] Cool. Okay.
Chris Webber: I have a confession to make because this just gets me thinking about it. You get to a point when you’ve done this long enough that you stop forgetting how the normal commands work. So, like, check-config is a perfect example. I literally will run puppet apply -e service name enable is true on a box instead of trying to figure out how to run check-config, because I always forget.
Sean O’Meara: Yeah, so it’s like service atom and then like service name, like Spark, and then like, yeah, they would reverse the arguments around depending on what platform you’re on. And just, oh Jesus. Yeah, Steve’s advice is super duper duper effective for getting people ramped up with CM to take existing documentation that you have, whether it’s a broken wiki page or like a runbook, and like translating it into the executable documentation is huge, right? It really clicks light bulbs on over people’s heads about like what exactly this is and why it’s good. And very, very quickly, can I just—
Matt Stratton: [00:54:23] I wanted to follow up on one thing that was important. You said taking that and translating it. Into configuration management, not pasting it into configuration management, not taking your setup scripts and just shoving them into Chef or Puppet or into something else, but turning it into that new language, adapting it.
Sean O’Meara: Yeah, so we have one customer who had a runbook for setting up a server. It was literally like an inch and a half thick, right, where it was like, okay, you know, your provisioning team just handed you this box Page 1, right, like, install this software from this— you know what I mean? Page 2, like, edit this configuration file. It was that detailed, and I’m like, it was really impressive, actually. But yeah, it was a perfect kind of thing for them to go through and be like, oh, okay, so I’m going to write a package resource here, I’m going to write a template resource there, I’m going to write a service resource here. And you also find out that you actually— you’re missing resolution from the documentation, too, right? So, like, install Apache.
Chris Webber: Okay.
Sean O’Meara: [00:55:24] What version of Apache, right? You know, like, you’ll find actual holes in your, you know, documentation this way, too.
Steven Murawski: Yeah, what modules do you need to enable? And all— yeah, all that sort of stuff.
Chris Webber: This is a conversation I actually had when PuppetDB was released with the folks over at Puppet Labs of, They have amazing technical writers and their technical documentation was awesome, but they didn’t release a module at the same time. And it’s like, guys, your docs are great, except I’m confused on this part and this part. Whereas if you had just published the module, and they did so within a few days actually, so kudos to them. But it was the, give me— you’ve got a language that defines exactly how to configure things.
Sean O’Meara: Things.
Chris Webber: Give it to me in that format instead of English, which sucks for describing the actual state of the world.
Matt Stratton: Before we go into the checkouts, just want to get everybody to reply and say, what is your thought on the future of configuration management? Where are we going?
Chris Webber: [00:56:29] I think I’ve already stated it, but I’ll reiterate. Servers are going to become software objects, and it’s just going to be one gigantic development codebase. Maybe not one codebase, but you get the idea. We’re gonna treat everything more and more as just a development project and with slightly different requirements, just like we treat UI projects slightly different than we treat backend projects.
Matt Stratton: Anybody else got a crystal ball?
Sean O’Meara: I think it’s gonna eat the world. I think software is eating the world, and I think that config management is gonna be a huge part of it, and We’ve already— we’re somewhere along the adoption curve, right? I mean, you gotta remember, CFM has been around for 20 years, right? LConfig, you know, people have been— there’s obviously a problem here that people have been trying to solve, right? And we keep inventing tools, right? And like tools are cultural artifacts, and you know, like as developer culture eats the world, you will see these artifacts everywhere, right? And it’s gonna be fun, and I’m excited to be a part of it.
Steven Murawski: [00:57:44] Yeah, I definitely think we’re, you know, amongst the group here, everyone’s, you know, very into this space, but I think, you know, in general, there, there’s still a long way to go in ramping up the use of some, some sort of configuration management tool. But the benefits that you get from it are just so tremendous. You know, having that executable documentation that won’t get stale. You know, it’s not that inch-and-a-half-thick binder that’s way out of date by the time, you know, by the time it’s sent to the printers. And, you know, that there’s new things that have changed and been updated over that time. I think we’re just going to see a continued drive in the space, even if, and even, you know, even as, you know, things like Docker become more popular, because these things don’t just manage provisioning, it’s configuration drift, it’s good, it’s, it’s that, it’s the, and it’s the iteration on that environment, it’s the changes as things go forward, you know, that these tools really address. And so yeah, that’s, I definitely think, and I think the Windows space is ripe for an explosion in the use of config management tools. It, you know, everywhere I go, there is more and more interest in this space.
Chris Webber: [00:59:06] Okay.
Trevor Hess: Matt, I think you, in one of our early episodes, really kind of hit the nail on the head about where this is taking us to. It’s taking us away from really— and I’m going to totally fail at paraphrasing what you said—
Matt Stratton: But essentially, if you have to remember what I said, I have no idea what you’re going to say right now.
Trevor Hess: It’s going to be— it marks the end of caring about individual machines and really like tending to specific devices and boxes and talking about things in terms of those sort of things. And it just being something you spin up. It doesn’t, you know, you don’t care about the actual physical box and where it is and who’s touching it because nobody touches it. You spin up a machine as you need it, and because you’ve built your executable documentation, it’s gonna stand up exactly the way you expect it to.
Matt Stratton: I couldn’t agree more with myself. So we’re gonna wrap up with our checkouts.
Sean O’Meara: I have FactMag Mix 432. It’s Luke Vibert. If you’re into dance music, you should probably check it out. I get to work from home now, so I get to spend a lot of time listening to music, and I listened to this today, and it was very good. You should too.
Chris Webber: [01:00:24] A couple of things just that I’ve been playing with quite a bit recently. Packer, if you are building those images, it’s been awesome because we can basically take our config that we use to build our base images in for Vagrant, and you can do things with VMware and with AWS, and they all kind of stay the same. Heroku, for the ops folks out there and the dev folks, the ability to just write some code and do a git push and, hey, your app is running, is awesome. And finally, if you are running Puppet code, please, please, please test your code. Puppet RSpec Or I guess it’s RSpec Puppet, and Puppet lint are amazing tools and will make your code better. So please check those out. All right.
Steven Murawski: So I guess I’m up then?
Matt Stratton: Yes, sir.
Steven Murawski: How many can I have?
Trevor Hess: As many as you want.
Steven Murawski: How fast can you talk? So I’ve been doing a lot of interviewing lately. We’re hiring a bunch of people at Stack Exchange, and so I wanted to point out our CEO, Joel Spolsky, had written a book, Smart and Gets Things Done, about interviewing and staffing. And it’s a lot of good stuff in there about finding and hiring quality people. A book that’s come up a lot recently when I’m talking to both dev and IT is Release It! It’s an old book. It’s been around for a long time by Michael Nygard. And it’s about designing and deploying production-ready software. Excellent, excellent read. It’s been a favorite of mine for a number of years. And as we’re moving more towards, you know, infrastructure as code, for those ops folks getting up to speed with what clean code looks like, Clean Code by Robert Martin, Uncle Bob, is an excellent read as well. Oh, and last but not least, you mentioned containers for Windows. Microsoft Research has a project out there called Drawbridge that they’ve had out and been writing about for like the last 6, 7 years or so. So I wouldn’t, I wouldn’t hold your breath, but I bet you some kind of containerization for Windows is on its way.
Trevor Hess: [01:02:44] Yeah, I’m sorry, I was gonna say, I think it’s the 3rd time that Clean Code book’s been mentioned on the show. I’ve been on vacation, so I haven’t been doing too much. I’ve been really tech down this vacation. So one thing I saw was somebody put together a little application to graph your TV— your favorite TV show’s ratings from IMDb. It’s interesting because you can see the trend lines for how your shows have progressed through the years and the seasons. It’s neat. I’ll put the links in the show notes. Also playing a lot of board games. I got Monty Python Flux. It’s a wonderful variant of Flux. It involves singing Monty Python songs and talking in outrageous accents. Can’t go wrong. And I also picked up the Star Trek Federation— Star Trek Catan Federation Space Expansion, which really brings some different mechanics to the Catan game.
Matt Stratton: Nice. Okay, so I’ll wrap up. One that I am putting as a checkout, and it’s not because of our guests being on this episode, but I would have thrown it up anyway, is a pretty awesome podcast that I’ve recently discovered called Ops All the Things. And that just might be hosted by, I don’t know, Steve Murawski and Chris Webber. So it’s, it’s off to a pretty good start. Chris mentioned when we were green rooming earlier that because they’re ops guys, their publishing schedule sometimes is a little challenging. And unrelated, or at least as far as I know unrelated, you guys can tell me, is there’s a documentary which is available on Netflix called Bronies, which is about adults, both male and female, people who enjoy the new My Little Pony television show. It’s a really interesting documentary. I watch the show, I watch it with my kids sometimes, and sometimes with my flatmate. But the documentary really kind of was eye-opening, is really entertaining in the way it’s put together. That’s called Bronies, it’s on Netflix. I want to thank our amazing panel, Sean O’Meara, Chris Webber, and Steven Murawski. Thanks for joining us. Everybody, please feel free to check us out, or please check us out at arresteddevops.com. You can follow us @ArrestedDevOps on Twitter. You can like us on Facebook. If you use Google+, you can hunt us down over there. I’m Matt, @MattStratton on Twitter.
Trevor Hess: [01:05:05] And I’m Trevor, @TrevorGHess on Twitter.
Matt Stratton: We’re Arrested DevOps, and remember, There’s always DevOps in the banana stand.
