Test Driven Infrastructure with Arthur Maltson and Michael Goetz

Michael: [00:00:00] I wasn’t sure, but I was trying to channel my inner Matt in saying like, well, you know, it’s not just the chef show.

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and business for maximum DevOps awesomeness. I’m your co-host, Matt Stratton, @MattStratton on Twitter. Arrested DevOps is brought to you by 10th a cloud services company that figures if you’re listening to this podcast, you must be pretty cool. You can find out about joining their cloud services team at arresteddevops.com/10thmagnitude. This episode’s also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools, such as Chef, Docker, and AWS, so that dev and ops teams share their key metrics data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. So I did have the opportunity recently to sit down and talk to Arthur Maltson and Michael Goetz around the topic of test-driven infrastructure. When we’re writing infra code, testing is really critical, but we wanted to talk about how you actually go about implementing these tests. And who better to do that than talking to some of the experts in the field? Arthur and Michael. So I’m really excited to have a couple great guests to talk with me about the idea of test-driven infrastructure. And a little pre-warning, I guess, we’re probably going to talk a little Chef-specific because my guests and I know Chef better than, than some of the other tooling. But these ideas should still apply to pretty much whatever you are applying for your configuration management. And so first, I’d like to introduce our first guest, who is Arthur Maltson. So Arthur Arthur, tell our audience a little about yourself and your background with test-driven infrastructure.

Arthur: [00:02:14] I am Arthur Maltson. I’m a software developer at Ontario Teachers’ College. I’ve been doing software development for about 8 years now. Started doing DevOps full-time here at Teachers’. I also gave a talk on test-driven infrastructure at DevOps So maybe that’s why I’m here. So yeah, really, I’ve been doing software development. I kind of take the software development infrastructure side.

Matty: Also joining us is Michael Goetz. So Michael, tell us a little about yourself.

Arthur: Hi.

Michael: Yeah, thanks. My name is Michael Goetz. I’m the manager of the Solutions Engineering Group at Chef. That is our consultants that go out and help people with their daily uses of Chef and other automation and infrastructure as code activities. My background is kind of an old-school release engineer, so I’ve got a lot of personal angst around properly validated infrastructure and application changes going out into production. So happy to kind of chime in on this conversation.

Matty: [00:03:32] Great. So we talked about this a little bit in our infrastructure as code episode, but I want to— which we’ll have link to in the show notes, and you can listen to it at restdevops.com/infrastructure-as-code. But before we get into the details on how to really do your test-driven infrastructure, let’s talk a little bit about why you should bother testing your infra code in the first place.

Arthur: It’s really around getting that software, whether it’s infrastructure as code, or it’s the software that you’re releasing, giving you that confidence that when you do make those changes, going to work the way you expect it to. I think that’s really the value.

Michael: I think when you talk about testing infrastructure and the value behind it, I think you really just need to understand what it is that you’re trying to test, right? Because if you think about you’re just testing in general, you have the signal in, right? Which is, did I— is the thing doing what I told it to do? You have the signal processing, which is your configuration management tool of choice or whatever application you’re using to take the instructions you’re giving and do some kind of execution. You shouldn’t really want to test that because hopefully the application you’ve chosen has a pretty solid test suite. And then you have the signal out processing, which is validating what came out the end. I think to what Arthur was speaking to is exactly what you wanted to have, right? The end state. I think what’s really important is just to make sure that you’re testing the things that you or the people you work with are inflicting change on a system and not so much trying to recreate things that may be validated elsewhere.

Matty: [00:05:23] I think one of the other things that’s important to think about is that it’s also giving us predictability. We’re going to— that we understand what these configuration changes are going to do when they get applied into our production realm rather than doing exploratory testing. And it gives us confidence in being able to deploy these things. So the thing is, let’s talk a little bit, when we talk about being test-driven. So for some people who might not be familiar, especially if you’re not coming from a development side, or even if you are a little bit, because not everyone does this, I want to talk a little bit about what it means to be test-driven. Fundamentally, as my understanding, and panel, jump in if I’m getting this wrong, but I always think about test-driven development traditionally means this idea of, sometimes we talk about this idea called red-green refactor, which means we’re gonna write all of our tests first, and they’re all gonna fail. And then we write code until the tests pass. That’s the green part. And then we refactor it, which means, okay, we wrote it well enough to get it to actually work. Now let’s go back and make it pretty. So the challenge that I see with this when it comes to writing infra code is I have found sometimes it’s really hard to write all my tests first because of the dependencies that different pieces might have on each other, being able to get from one to the other. And I’d like to know, in your experience, if you actually, A, have had experience actually following a true test-driven approach to infra code, and if so, how do you approach that?

Arthur: [00:06:54] I guess maybe I’ll jump in here. I think just a little nuance there for the test-driven development definition. You don’t necessarily write all of your tests first. You usually write 1 or 2 tests, and then you write the implementation code. Right after. So it’s the same thing, and I agree it’d be really hard to come up with all of the tests up front for your infrastructure, test-driven infrastructure. But I think if you break it up into small little chunks, like, okay, for this application to run, first I need to define a user for that application to run as. So you might write a test that says, you know, make sure that this user and group exists, and then you’d fail, and then you write the code to make it pass. And that, that’d be kind of the cycle that you’d get into, whether you write the tests first or you write the implementation code first. I mean, personally, I’m not, I’m not very religious. As long as the tests come shortly after or shortly before the implementation code, then you’re golden.

Michael: [00:08:08] Yeah, and I think, you know, you kind of identified 2 real use cases there, because there’s greenfield development where you’ve never built a thing before. And I think Arthur’s approach here is spot on. You can’t test what you don’t know. So you identify what you’re going to work on. You write tests to validate what you would perceive as the outcome. And then you write tests against those. And that’s great, again, when you’re inventing something new. Another approach that’s quite interesting that I’ve seen some folks use is, As you are coming into— you have infrastructure, right? It’s not like your business just started. You have infrastructure that needs to be validated. And maybe it hasn’t even been automated yet. And people will use the test-driven infrastructure model. But what they’ll do is they’ll use production and write tests against production, saying, I know the state that I want it to be because I have a working copy, and use that as a blueprint to develop against while they write their automation code, which has been super successful. Successful for a lot of organizations that have the whole legacy systems that need to be migrated.

Matty: [00:09:25] Yeah, I want to talk about that. And that was— I’ve definitely seen people be successful with that. And again, speaking specifically around Chef, thinking about using audit mode, and actually, like Michael said, using your production as your testbed. So what you can do if you write your audit tests, which are you’re saying basically, this is what I know a compliant system should look like. So I’m going to write some audit cookbooks. Which don’t— and for those of you who haven’t used audit mode before, an audit cookbook is a set of controls that simply says if this is true, then the system is compliant. So I’m going to write my compliant— basically, it’s my compliance tests. And I’m going to unleash them upon my infrastructure in audit-only mode. So one thing, again, if you haven’t used audit mode in Chef, the Chef client has 3 flags for how it works. It can run with audit disabled, which is the default, it can run in audit enabled, which means it will run any conversion code, any recipes you have to actually with resources and put resources in state as well as run your audits. And then it can finally run an audit only, which means even if the run list of the node has convergence code, it won’t do it. All it’s going to do is run any audit controls that are in the run list. So what you do in this case is sit there and you say, okay, I’m going to sit down and write controls that say what I think a machine should look like. And I’m gonna run Chef client in audit-only mode on my production infrastructure. And what this is letting me do is actually truly be, like Michael said, test-driven, because what this is gonna let me do is determine, first of all, which of my machines out there in my, you know, fleet migration that’s happening are actually not compliant. So I know that I need to write some Chef code to actually make them be compliant, but it also lets me determine what’s gonna be the impact of doing that. Because if we sat down and said, well, I know what I think these machines should look like, I’m gonna write Chef code to converge on that, and I’m gonna unleash it on my 10,000 nodes. And then I determined that I actually broke 9,000 of them because there’s 9,000 snowflakes out there that I didn’t know about, and I didn’t know these particular idiosyncrasies. Whereas if I start with audit, which is basically just a test, I can then say, okay, well, on these particular machines, this resource is not the way I expect it. Now I can do some investigation with the owner of the system or the owner of the application that’s involved and determine what might actually be the reason it’s that way, ’cause maybe my understanding is incomplete. And/or at the minimum, I can know that I definitely need to test it against a configuration that looks like that before I roll that out. So I think that’s a really powerful way to do test-driven infrastructure. Again, it sounds silly, but you’re treating your production as your test environment. But it’s not where you’re testing what the change will— you’re not rolling out the change and testing it. You’re, you’re letting that generate what you need to do. It’s driving your code change.

Michael: [00:12:13] Yeah, and I think, you know, Matt, to your point, just to— I mean, we talked about audit mode, which I think the name itself is a little misleading. I know the original intent of audit, right, what it’s behind, but there’s other tools, the frameworks, just to be— try to be a little agnostic here, right? You can do the same with ServerSpec, with Leibniz, with some other things that are out there open source. But I think the real key here is, you know, Matt, to your point, you have a known good or a system that is known good, and you write some sort of validation checks against that and use that to drive your test-driven. It’s a really good pattern for sure.

Arthur: And I mean, I guess with audit mode, that’s really Matt’s idea there of writing all the tests ahead of time. And you can— I think it’s the same thing when you’re trying to refactor a legacy system. Touching a legacy system that has no test is terrifying. And so you really want to write some tests first to check, usually an out-in approach. So you write some integration tests saying, hey, the system’s still in a good state. And then you start to, as you’re drilling into the class or whatever it is that you’re trying to refactor, you want to have some tests to tell you, okay, what does it do today? And then how can I either change it, or if you’re doing a big rewrite, and I guess in some sense with audit mode, you’re essentially rewriting what’s a manual system of server with a configuration management system like Chef. So that’s definitely a really powerful tool.

Matty: [00:13:51] The other thing is I’d like to think about, so we can, we can take this again, talking a little bit about, I’d like to talk a little bit about, about, about the how do I do this. We already talked about the how I can approach it from kind of a global looking at my whole infrastructure. But if I’m looking at it and saying, okay, so I am, I want to write some, some infra code, and we may again use some Chef-specific terms here, but I think the approach is similar. But I want to talk about. I actually wanna specifically talk about how-to. And I think for each of us, maybe we can kind of share our approach that if, as a cookbook developer, what are the tools of the trade and the process that you follow to say, how am I gonna test my infra code? And approach that. I’m writing a brand-new cookbook, maybe, you know, it’s a new thing, and I wanna make sure that I’ve got— I hate to— I shudder at saying that I’ve got test coverage because that’s a whole other ball of wax we could unwind. But thinking about the the steps that I go through. So I’ll kind of turn it over to Arthur first to kind of walk us through your workflow, and then Michael and I can follow on and either nod our heads in vehement agreement or add on our own thoughts as well.

Arthur: [00:15:01] I guess to be a bit Chef-specific, the first thing that we use with any new cookbook is the Chef Generate, and we have a custom generator template that we use. But anyway, we try to be very prescriptive with how you write. Really push writing tests. So really right from the get-go, the minute you generate your cookbook, there’s already test stubs and all of the configuration for Test Kitchen. So we use Test Kitchen heavily. That’s, that’s a tool from Fletcher Nickel and really lets you test your cookbook, or it actually has plugins for any configuration management system out there. It lets you test those cookbooks locally on your So the first thing that we’ll usually do is potentially either start with the test or start right away with the implementation code and write the test shortly after. And we just iterate with our Test Kitchen builds, running them locally. So we use Test Kitchen with ServerSpec. So we write our tests in ServerSpec today. And eventually I think we’ll, we will be switching to audit mode. But ServerSpec really gives you this kind of English-like syntax to check the state of the system. So you say, you know, you expect a file to be here and its content should be this regular expression. And that’s pretty much exactly what I said is, is the way that you write this server spec code. And then we, we really use Kitchen Docker a lot. So get that fast feedback, which as a software developer, I really want to be able to run tests that execute in under a minute, under 2 minutes. And if the feedback’s longer than that, it’s kind of painful. So I found that with the regular, with the default Test Kitchen Vagrant approach, destroying and recreating the machine really takes a long time. You really get hit with the, with the Vagrant initialization and startup. Really, Kitchen Docker has saved our bacon a bunch by really making tests super fast, and we can always start with a clean slate with a destroy and then verify run again. And so once we finish writing our tests. This is actually another piece that we could probably talk about for another long time. We follow a very developer-centric workflow of using branch-based development. So somebody sits there, creates the cookbook with the tests and the implementation code, pushes that up as a branch, and then somebody code reviews that. We have automated builds that start to run. So again, we can get into And once it’s fully reviewed, we merge it in and it gets usually automatically deployed to production.

Michael: [00:18:02] Yeah, so that’s pretty close to what I’ve done for a while and what we recommend for customers. I’ll say something a little provocative here. I don’t actually destroy my test kitchen instances until I feel like I need to do a clean run. And, you know, that’s probably going to anger some TDI purists. But I find, to Arthur’s point, right, you don’t— when you’re trying to do development, you don’t want to wait for kind of spinning up new instances or even new containers, right? Like building from scratch over and over again. Even if you have a container to use as your image, can sometimes take take a little while depending on how complex some configuration code you have to work on is. But what I also always find is at certain points, I’ll make a judgment to rebuild from scratch to make sure that I can, A, rebuild, and I can run it again and that nothing changes. So I don’t have any test and repair loops where I’m constantly making a change because I don’t have— I don’t have any kind of idempotent logic in place. But I think, you know, to Arthur’s point, you know, using— I use Test Kitchen. Depending on what I’m trying to do, I’ll do Kitchen Docker or Kitchen EC2 or Kitchen DigitalOcean or even Kitchen Joint, right? So like a bunch of different providers. But I will say that For me, I test as I go. So I write one test, I write the code, then I write another test, and then I write the code instead of trying to write the whole suite ahead of time. But yeah, so I think the key thing that I would add on to this is that we’re talking about local developer workflows for testing. And I think people get a little scared and maybe confused when they start to automate that into maybe a continuous integration pipeline. And really, it’s a little bit more complex. But it’s about as simple as you think. If you can do it locally, you automate it with robots and your CI pipeline. And try to stay as close as possible. Don’t add in new things that you didn’t do locally. And if you have to, try to figure out how to make them— how to have those tests and testing frameworks being utilized locally as well.

Matty: [00:20:38] So I just have a couple things to add to that. So one is I’ve gotten into the habit of instead of referring to it as local development, referring to it as individual development. Because local development definitely implies that I’m doing all this running Kitchen locally, like on my actual laptop. And it may make more sense for me to be doing this in my organization to say, we have VM workstations that everybody uses, because I know that then that way, they all have the same version of ChefDK, they’re all configured the same way. Or even sometimes local development implies local virtualization. And I could still be using maybe something like Kitchen EC2, or, you know, a VMware driver for Kitchen or Hyper-V to actually spin those instances up. So I don’t need all of the you know, these resources on my laptop to be able to do that. So, and the other reason I like calling it individual development is it really kind of like thinks about how that workflow works, that now I’m working as an individual before I move into collaborative development into my pipeline. One thing too, and I’ll put a link in the show notes to this, is, uh, so Sasha Bates gave a talk at ChefConf 2013 called How to F Up Your Configuration Management Adoption, and she talks about a lot of mud pies she made made. But one of the things in there that she talks about is how when she’s testing, and this goes back to a little bit with Michael’s thing about how he doesn’t do destroys, and because I think if you do a destroy every time, that’s not necessarily a full test either, because you’re not necessarily building new machines every time. So Sasha’s point is she’s like, I run it against the existing new machine, then I run it again, then I run it again, then I destroy the VM, then I run it again, then I run it again. Because you want to be testing for idempotency, you want to make sure you understand how it works against a VM that already has some configuration on it. So I So I think that’s an important piece. And just talking a little bit more about some tools that should be part— again, to echo Michael’s point that whatever you’re doing, I think it goes both ways. Whatever you’re doing in your test suite that’s happening in your automated pipeline, you should be having that happen in individual development. But you also should be having whatever tests you consider things that you test for individually should be repeated in your pipeline. And to a lot of people, sometimes that feels really redundant because it’s like, oh, hey, I already know that I ran food critic, so why am I bothering to do that in the pipeline? And it’s for 2 reasons. One is trust but verify, right? You wanna make sure, I don’t always trust myself to run all my individual tests. And then also a matter of because my code, maybe once it gets into a collaborative development, it may be now merged with somebody else’s code, and I wanna be able to run those tests as well. And that’s a core precept in Chef. Delivery of how that actually works with our different phases that it goes through and the different stages that it goes through to be able to do that. What about performance? So where does performance come into play? And that’s kind of an open-ended question, but I’m just going to ask it that way. How does performance come into play in test-driven infrastructure?

Michael: [00:23:37] So, yeah, this is Michael. I’ll go first. I think performance is really impacted on— I think there’s 2 aspects. There’s what amount of infrastructure are you willing to put into making your testing environment look as close to production as possible? And I would argue, with someone who doesn’t have to worry about an ongoing maintenance budget for an environment, you want to make it as close to production. So if you have 15 systems in production, you should have a 15-system cluster in test. So from a performance perspective, you want to do that. But I also live in the land of reality, so sometimes you can’t do that. So I think the goal that people should strive for on that aspect is as close to production as possible. But I think in the individual or isolated development perspective, you are going to want to try to give as much battery power as possible to your isolated development also looking as close to your production environment as possible. This may mean having multiple Docker containers or multiple images running locally on your system, and that can be kind of a resource hog. But again, if you are validating on small chunks, you can kind of reduce the footprint need for what you’re actually spinning up to test locally.

Arthur: [00:25:07] Yeah, I mean, I think the performance— so there’s really a difference between unit tests and integration tests, right, from the software development world. Unit tests are really things that execute in memory, they execute really quickly. And then there’s kind of your integration. And then I guess end-to-end is probably the slowest ones where say you’re opening up a browser and poking around and testing your application that way. So from a software development world, we really run into performance issues if you lean really heavy on the end-to-end browser-based tests. And that can really get you into multi-hour-long test runs. So, in some sense, I think maybe to Michael’s signal-in, signal-out analogy, if you have a lot of the signal-out tests and you’re spending a lot of time setting up, say, a gigantic infrastructure, you really can get into a lot of performance pain. So, one of the cookbooks we have sets up the ELK stack, the Elasticsearch, Logstash, and Kibana stack. And we actually test all of that locally. With Docker in a multi-Docker environment. And those tests really do take a long time. Like, we’re talking about 20, 30 minutes for getting some feedback on a change that you made. So, you know, performance, especially from the infrastructure side, you don’t really have too much to play with aside from beefing up your machine or beefing up the test environments like Michael alluded to, to really get that feedback to come back faster.

Matty: [00:26:53] Absolutely. And I think that’s one of the things that people, when you start talking about doing test-driven infrastructure, they see that the whole process makes everything feel like it’s slower. And that can be from the perspective of you have more things to do, right? You’re like, I just want to write some code. I don’t want to have to test it. I don’t want to have to wait and go through these iterations and all these pieces. And one of the things to that is just a matter of the reality is in the gestalt, Right? Like, while it may feel like it’s making things slower, the reality is, in practice, it’s making you be able to move with greater velocity because you have more confidence in those changes, and you’re not having to go and say, oh, I rolled out this cookbook, and oh crap, it broke something, so I now need to really quick write some remediation to fix the thing that I just broke. So making things safer overall makes it faster. One of the other things I’ve seen is that people talk about all the work that’s involved in writing a test, especially for infra code, because you might have to write some really complicated stuff to be able to determine the state. What I found often too is that actually it makes your development faster because you’ve probably figured out a lot of the logic simply by writing the tests. There’s, in addition to test-driven development, there’s another idea that people have written or talked about called README-driven development, which I think is interesting, where you would write the documentation for your cookbook before you do anything. And the funny thing is, when I think about how I apply cookbook development, that is how I do it. I don’t write the README. But when we think about, you know, when I’m helping customers figure out, you know, go through a POC or learning how to do it, the approach is we say, okay, first figure out what that desired state of your infrastructure is. And I said, write that down. And that’s actually, in a lot of ways, your README. And then the next thing that you do is you determine the resources needed to put it in that state. So you really, and then you can write test to say, is this resource in the state that it’s in? But I think that’s really the hardest part, is it’s like you have to— it’s sort of like my thing when I was in school and I never wanted to write outlines for my papers. I just wanted to start writing, and that’s actually how I always write. But it’s tough to write infra code that way. You don’t just start and sit, you know, do chef generate cookbook, fire up your default.rb in your recipes folder, and just start hacking. You have to have a plan. And if you have a plan, the test will come out naturally, and all of that kind of pre-work makes the writing of the code actually much faster and much easier, I believe.

Michael: [00:29:19] Yeah, Matt. Actually, you said 2 things that I kind of wanted to jump in on, which was some people complain about it takes so much time to write a test, right? And I always— that kind of grates on my nerves a little bit because how much time do you spend on an incident bridge when the thing is broken? And I guarantee you it’s probably going to be longer than the time it took you to write a proper test for your infrastructure code. And so I get it, it’s tough. You’re trying to do some work and you want to get a change out fast, but you really are trying to, uh, um, to protect yourself from that safety.

Arthur: I, I, and I, I guess, sorry, just to, to build on Michael’s point, uh, how much time do you spend SSHing into the system after you’ve converged it or run your playbooks or whatever to see, hey, Wait, did this actually work? Yeah. Oh yeah, it’s okay. Okay. Yeah, that’s okay. And then you’re going to do that for the other 100 systems that you run on, right? So in some sense, like you were saying, you’re saving a bunch of time. You’re paying it forward ahead of time by writing those tests. But then again, it pays you forward later on when you roll out the changes.

Michael: [00:30:39] Yeah, and I think another thing that can kind of help with the— it takes so long, right? The other thing is the image that you use to validate the change can change, right? There’s no one saying that you have to build from scratch every single time. I think this goes for everything, right? I mean, if you have infrastructure code that that defines the policy, and that’s well-tested, and it’s available, then there’s no reason to just execute it again to get more green lines in your output at the end of your Test Kitchen run. If you really want to kind of short-circuit and really focus in, you could add another layer to your Docker container that has all of the other 15 steps you’ve already completed done, or bake a new image that has a lot of the initial configuration done. But you’re still running cookbooks against it.

Arthur: [00:31:39] Oh right, so a real-world example of this that we actually have, um, we’ve been rolling out Sensu as a monitoring, uh, service, and we really want people to write checks really quickly. And you really don’t want to sit there on your workstation waiting for the entire Sensu, uh, infrastructure to configure in inside of a Docker container. So really, some, some of the devs built, like Michael suggested, a Docker image that already has Sensu inside of it. And so if you want to write checks, you just spin that up, Sensu’s already there, and then you write some checks against it. So it’s really, it’s, it’s really power. You can really sit down and hash out those performance issues and really solve them and give you faster feedback. Because at the end of the day, with testing, just like in software development, you want the that fast feedback.

Matty: I want to just— the next question, just as a thing kind of to wrap up, is what are— what really gaps do you see in the approaches to test-driven infrastructure, whether they be in tooling or in practice, or things that you think that we as a community could be doing better in order to be able to have better predictability? Where are the gaps?

Michael: [00:33:00] You know, the gaps that I see are a lot of times people being too purist and trying to apply decades of software development experience and TDD approaches, which, to be fair, are kind of in debate anyways, to a group of people that may be new to testing infrastructure code and being a little bit too draconic, right, and letting people learn those lessons similar to the way— I’d say that’s a gap because I think sometimes we feel that we can teach people or let people avoid the things that actually taught us in the first place as it relates to testing. I would say that’s a people gap. From a technology gap, I think we still don’t have a good fleet-wide validation mechanism. You have tools that allow you to validate one system. You have tools that will let you validate the output of a web page, which I guess is technically fleet-wide validation. But we don’t really have anything that, you know, you spin up an app, a web, and a database system, and you can validate all 3 of them or have the configurations that you’re expecting at the same time after having the whole thing managed as infrastructure as code. There’s been talk about trying to do that to some extent, but, you know, I don’t really think that that’s out there. I think that’s the big story that most people want to get to as they evolve their infrastructure testing.

Arthur: [00:34:34] So I think one of the ones that I’ve definitely experienced is the multi-node cookbook testing, right? So when you’re trying to stand up an entire infrastructure that consists of several different components that all talk to each other, what we’ve done is we have them talking to each other on localhost in one Docker container, but that really doesn’t represent the real state of the system. And again, this might just be something that we’re not doing well, but it’s not— I haven’t seen it talked up, talked a lot about, and really how people spin up a cluster of machines, test the bejeezus out of them, and then tear them down. So that’s probably the biggest gap that I’ve seen so so far. And then at the end of the day, it’s really— we touched on performance. It’s how to get that performance and that feedback cycle as fast as possible and kind of ways to optimize that.

Matty: So thanks for that. So let’s go ahead and move into checkouts. So why don’t you— Arthur, do you have anything you would like our audience to check out?

Arthur: [00:35:41] Sure, Matt. I guess I splurged a little bit on the checkouts, so I can go pretty quickly.

Matty: Oh, take your time because I actually couldn’t think of anything this week. So, the checkouts are all about you. So, you are— you’ve got full rein here.

Arthur: Sweet. All right. So, I guess from the technical picks perspective, I’ll pick a book, tmux: Productive Mouse-Free Development. So, I read this book a couple of years back now, and it was really the way the way that I’ve dove into using tmux on a regular basis for just about everything I do, and tmuxinator as a gem that makes using tmux a joy. So check that out. If you’re old school, then screen is probably the tmux equivalent you’ve heard of. So check out that book. It’s really short. It’s 100 pages or so and really dives you into how to use tmux. The other one I’d like to pick, a technical pick, would be private supermarket. So here, here at OTPP, we set up a private supermarket a few months back and we’ve been loving it. It’s really, it’s really actually opened up Chef development to more than just the people who have the keys to the Chef server, because now we can share cookbooks in an open fashion within the organization. So Private Chef, as set up through the Omnibus Supermarket Cookbook, really makes it a breeze to set up Private Chef— Private Supermarket within your organization. And from a non-technical pick, I wanted to pick the Headspace meditation app. So this is at headspace.com. It’s— I got into meditation a few months ago now, and And this was— I mean, it might sound a bit cliché, but, uh, in the app era or whatever you want to call it, there’s an app for everything, and one of them is meditation. And probably, uh, meditation gurus are, are, you know, throwing up in their mouth or something, but, uh, this is— this has been the way that I got into it, and I found it really fantastic. It’s run by, um, Andy Hunter. I think his name is, that he did a TED Talk on meditation and he runs this application and, and this company and they just do fantastic work. Uh, it’s really helped me in a lot of ways to kind of de-stress and actually sleep better. That was, that was a big one for me. Um, and just there’s lots of different, uh, different meditations for different circumstances in life. And, uh, finally, I, I guess I’ll pick my favorite wheat beer. So I’m a big fan of wheat beers, and my favorite one is Paulaner. It’s a Munich wheat beer. It’s just fantastic. It’s kind of hard to find, to be honest, especially here. I live in Ontario, Canada, where we have a giant monopoly that sells all, all of the liquor. So it’s a little bit hard to find, but hey, you know what, it’s awesome. I guess one last quick one is if you want to work off that I’ve started to do weightlifting recently. I’ve always done cardio, intense cardio, and weightlifting actually turns out to be a really great cardio exercise. And I have a great guy teaching me how to do that well. So great.

Matty: [00:39:13] Actually, it’s funny that you mentioned that. I actually need to get back into the gym again. I was— I’ve kind of fallen off the wagon. But one of the things, again, you know, when it comes to, you know, lifting is great and doing those kind of exercises, one that a trainer I was working with— I don’t know if it’s your gym— they have something called battle ropes, or as my trainer called them, the shoelaces of death. But they’re those big heavy ropes, and you can do a lot of exercises where you fling them up and down and they just kill you. But they’re really good for strength and cardio and gets your heart rate really up there. But, you know, it’s a strength exercise as well. So I lied, I actually have 2 quick checkouts. So one is I wanted to first of all echo Arthur’s point. So that TMUX book, and that’s by, um, Pragmatic Press, I think— we’ll put a link in the show notes— I’ve read that book as well. It really turned me on to TMUX and understanding And I was using tmuxinator at first based upon that book, but I’ve changed to— I use a tool now called tmuxil. And I’m not quite sure exactly why I like it better than tmuxinator besides the fact that it’s an Arrested Development reference. But it works out really nicely to be able to define my— just like tmuxinator, you define your spaces in YAML. And I think maybe it just for some reason echoed with me a little clearer on the configuration of them. But it’s really nice. So I can sit down and say, I have a whole thing I’m working on for a particular cookbook maybe, and it’ll fire up all the different panes and have one that’s got, you know, Vim ready to go. Another one, I actually can even have it go and fire up Atom to open the cookbook in Atom, where if I want to edit it there, have Kitchen ready to run, have Git running over on the other one. And then the other one, which is a little bit of— I may have mentioned this on the show before, I don’t remember, but there is— so Telltale Games, they’ve done a really great game about The Walking Dead, but they also are in process of a game around Game of Thrones. It’s a very interactive storytelling game that I just realized I think there’s another episode for me to download, and I’ll be playing that. I don’t know if there’s Android tablet versions of this. There might be. If I find them, I’ll put a link in the show notes. We’ll definitely put a link in the show notes for the iOS or the iPad version of it. But it’s really fun if you’re into Game of Thrones. It’s a really fun multiple branching story kind of thing. Thanks for joining us, Arthur. Hope you enjoyed being on the show. We certainly enjoyed having you.

Arthur: [00:41:29] Thank you so much, Trevor. This was awesome. Great.

Matty: Awesome. My name is Matt.

Arthur: It’s the application that says Trevor.

Matty: Yeah. Yes, our recording app makes me look like I’m Trevor, at least in the recording. So, oh, it’s okay. All right. And on that note, we’re out. So an interesting thing that happened was when we initially recorded our interview with Arthur and with Michael, that was about 12 hours before Chef Software released a whole bunch of new products that can be used to really enhance your test-driven infrastructure workflow. And we weren’t able to talk about it in the initial interview, but Michael and I wanted to add a couple things to share with you about these features that we announced, especially around things like what we call Chef Compliance and a couple other tools. So we’re gonna go ahead and talk about that stuff really quickly and give you some resources to learn a little bit more. One of the things that was released is a new product called Chef Compliance. We want to talk a little bit about what that’s all about. So Michael, what do you think when we think about Chef Compliance? What’s that all about?

Michael: [00:42:50] Yeah, so what’s interesting is when people talk about compliance, they’re usually— it’s because they’ve been impacted by having to do some kind of audit, right? And so when, when people first started doing some server spec tests and then we had the audit feature that came through, it was very much in response to, you know, how can I make security or compliance officers happy that this automation that I’m doing is actually compliant with, you know, PCI or SOX or, you know, STIG or CIS benchmarks. And for those of you that actually dove into or have a security or compliance background, you’ll know that what you’re doing is taking this doc and translating it into root user must not be 0. So what’s the check to validate that? And so when I think about Chef compliance, I think about how do we actually bring security and compliance folks into an automated world so it’s not just DevOps? Right? It’s everybody kind of moving along with the automation and validation of the changes going through infrastructure.

Matty: [00:43:56] Totally. And I think that’s one of the challenges when we think about compliance is you’ve got a bunch of different actors that care about setting compliance or the challenge, you know, think about the challenge of audit, right? You’ve got your compliance group, your compliance officers, and they care about a lot of things like process compliance and running audits and patch level. And you’ve got your security folks who are caring about, you know, that your systems match to security policies. They’re doing things like pen tests, doing system verification, and your ops folks are, you know, they’re doing monitoring and logging, they’re rolling out apps or applying patches. And when we kind of think about the tooling that we’re all using to do that, your compliance folks, you know, their stack is PDF and Excel, right? You know, so they’re writing policies in PDF form and PDFs that they’re being shared. They’re keeping track of things in Excel. It’s a lot of human words, right? Your security folks, are writing, you know, using shell scripts or penetration tools or security tools, and then your ops folks are, you know, using maybe some other testing frameworks or things like Chef. And then what you don’t have, just like you said, we don’t really have a common language. And so, that was one of the things we really wanted to look at with, you know, Chef Compliance was thinking about, like, how can we have a common language so that we’re not translating, you know, we’re using a common set of tools to meet all of those pieces. And so, When we think about how compliance lets us do this idea of continuous audit. So we’ve talked about before, right? Security and compliance are just another aspect of quality. They’re baked into our workflow. And one of the things about Chef compliance, the tool, is it doesn’t actually require Chef client to be running on the nodes that you’re testing, right? So it lets us test against our compliance on our existing systems and use that same compliance during our creation of our infrastructure and our apps through our pipeline. Now, one of the other things that was, like I said, we released a whole bunch of stuff. And one of the key things, in addition to the Chef compliance tool itself, we released some open source items. And one of them was something called InSpec, which is really, it’s a testing framework that’s built on, it’s kind of heavily influenced by ServerSpec, for example. But it’s, and so InSpec, As we have seen it, we’ve referred to it as both meaning like infrastructure spec or— and it’s really giving you kind of this idea of being able to have built-in compliance. So you’re not existing at the end of the release cycle. But a big part of it is it’s including metadata that’s required by security and compliance pros, right? So we’re bringing in things like how important is this particular compliance test? And we can both run these tests at a command line or they can be baked into our testing cycle. Because another thing that we released released was Kitchen Inspec. So you can use, you can run Inspec tests directly in Kitchen. And in fact, it doesn’t even depend upon Busser. So it’s a really nice piece of that. And then the last bit of open source stuff that got released was something called Train, which is leveraged by Inspec and some of these other pieces. And that’s like this abstraction for being able to talk to local or remote instances in a unified manner. So it supports SSH or WinRM or Docker or Mock. And it’s letting me just sort of say, run this command or connect to this machine. And I don’t really, I can abstract away whether it’s via SSH or via WinRM or whatnot. And the other thing that’s happening in Inspect that’s really cool to me, that I think is really cool, is that you’re able to kind of create these custom resources that are abstractions. So instead of writing a test that says, let me write a regular expression to make sure that this particular SSHD config file contains this string, ’cause if I’m the compliance officer, I don’t necessarily know that level of, I may not wanna get that deep into it, I just wanna be able to say that my SSH config file, whatever that happens to be, or the thing that’s in charge of SSH config, its key-value pair matches to this, right? This particular parameter is indeed set. And it gives, again, there’s no problem in computer science that can’t be solved without a sufficient layer of abstraction. It’s just making it a lot more accessible to write these common, and things that are translating the human requirements into something that’s expressed as an outcome is what I see with that. And I don’t know, I haven’t had a whole lot of time, a chance to really play with this a lot, other than, I mean, I’ve done a lot of stuff with playing around with Chef compliance. I haven’t written a lot of InSpec yet. I don’t know, Michael, I don’t know, what have you kind of just, what has been your gut feel on this? I know this is all a lot of really new stuff.

Michael: [00:48:46] Yeah, no, I mean, when Audit first came out, I was super excited about it. I think it started to solve a need. I think the benefit from Inspect is that it adds this metadata layer, which is super critical information when dealing with compliance, right? When you talk about traditional compliance checks, right? So like security validation or specific regulation compliance checks, they have severity levels, right? And sometimes you can fail a compliance check and that’s okay, right? Like that is not going to block you from release. And one of the things that was difficult was to really get an understanding of like, what’s the criticality of this compliance failure? And so I think one of the greatest things about the inspect piece is all the metadata that it adds, and it’s not locked into some arbitrary, you know, low, medium, high, right? You can make the levels as you need them to be. And I think what the general flexibility of it is great in that just thinking about a compliance check is just a test. We talked about test-driven infrastructure and what does that actually mean? Anytime you’re writing a test, you are checking for some sort of compliance. It may not be a standard reg, but you want to make sure the thing that you are doing is compliant with your expectation. So you can kind of leverage this and create profiles for all kinds of things that aren’t specific to compliance. So that’s, for me, playing around with Inspect so far has really been awesome. It’s based on Ruby. It is very similar to the ServerSpec DSL. And it’s very behaviorally written, right? So you can read the checks as opposed to having to have for loops with all kinds of complexity in there. And then the last little bit about Inspect that I think is really good, and it meshes really well with Chef’s general philosophy, is if we don’t have the thing you need, you can totally build your own thing. So it has its own ability to create custom resources, just like you can do within Chef. So overall, I’m super happy with it. I’ve played around with it for a little bit. I’ve tried to translate a CIS benchmark in for Red Hat, and that went over pretty well. I’m kind of excited to see what kind of profiles people start creating and publishing out to the, out to the community.

Matty: [00:51:21] Another thing that InSpec has into it is when you’re writing these InSpec rules, you can create descriptions, and that when you build that into Chef compliance, where that’s really powerful is that when I see that something failed, it’s providing the context and description and explanation as to why, and even including how to resolve it. So if we think about the way this works in your workflow is you start by having, by running your compliance scans against your machines to see if they’re compliant according to what you’ve defined. And then when they’re not, you then go into the workflow of your kind of local or individual development, like we said, using ChefDK to write your remediation. And then again, being able to leverage those same inspect tests using Kitchen Inspect to make sure that you’ve resolved them, move that stuff through something like Chef Delivery to be able to ensure that it’s— that these changes that you’re integrating are integrating well with your other infra code, and then letting Chef Server deploy them appropriately for the actual remediation, and then wrapping something like Chef Analytics around it to ensure that those nodes are now actually continue to be compliant. So, it’s really about it also the whole workflow, right? Like, these, all these things like Chef compliance, delivery, analytics, inspect, they all work together to give you really what the important thing is. Because again, it’s not about the tool, it’s about how you’re doing work. So, it’s giving you a workflow and an approach to say, I really am, I’m test-driven or I’m compliance-driven, I’m outcome-driven, right? Are my things the way that they’re supposed to be? Is the outcome of my systems the way that is dictated according to the policy of my organization. And it’s also providing that ability for the people responsible for those systems to be able to remediate them because maybe they’re getting alerted by Chef Analytics with the context of the rule that says, here’s how to actually fix it. So they can go in and do their own remediation through Chef and push it through your proper pipeline with all your right approvals and governance. So that’s really, I think, the approach we’re taking with this is creating all these building blocks, which are awesome, but then also tying it into a workflow that’s usable And that’s what’s giving you, rather than just a lot of pieces, giving a way to actually be successful with that. So, it’s exciting stuff. And I think it’s giving us that ability when we think about our infrastructure and our systems and how they interrelate, and especially being able to be collaborative about that, right? Everyone has their piece that comes into their area of responsibility and of expertise. I’m the compliance person. I have the expertise of knowing what things should look like, and I can, again, communicate that in a reproducible, consumable way.

Michael: [00:54:14] Yeah, I think one thing we didn’t talk about too is the remote execution aspect of the compliance thing. So, you know, we’ve been talking a lot about Chef products, right, and how it ties into the workflow, and it’s super exciting. But there is an aspect of being able to run inspect validations via remote execution, which is why train comes in against your systems. So speaking back to what I talked about earlier about folks working through TDI, trying to get to test-driven infrastructure, and that approach for existing systems and how do you start migrating them to be managed by infrastructure code? Chef compliance seems to be, from what I’ve spun up and played with it so far, has been very much like a super easy way to just go and, here, write some inspect tests and validate them against your existing infrastructure, and then you get a nice report that says, here’s all the things that are out and the critical things. Criticality of whatever it is in the descriptions that Matt was saying earlier.

Matty: [00:55:24] And again, some of the questions that have come up is, you know, InSpec is, and just like Kitchen, are not necessarily chef-only. You know, InSpec is a framework and a tool that you can use, just like you said, for forever that might come into, for testing your infrastructure code in this really expressive way. So, it’s, you know, obviously, Chef built it as a building block for a larger workflow that we wanted to do for Chef, but it’s an open-source product, and it works independently of that. So it’s something we’re sharing with the community and hoping that the community is gonna help us make it even better and better. So, I mean, hopefully this gave some context and color around some of the new things that are happening around test-driven infrastructure, especially in the Chef world. It doesn’t really necessarily replace, it just enhances the things we talked about earlier in this episode as well, and just gives even more tools in your toolkit around that.

Michael: Yeah, I think this has been an awesome discussion, and I just encourage everyone when they think about test-driven infrastructure to keep an open mind about what you’re actually testing, when you’re testing, and why you’re testing the things in your infrastructure.

Matty: [00:56:42] This was a great conversation. And don’t forget, in addition to great conversation, we have a newsletter as well, which you can sign up for at arresteddevops.com/bananastand. It’s the best way to know about our upcoming podcast episodes and cool news with DevOps. We’d love to thank our great sponsors. Please be sure to visit them at arresteddevops.com/10thmagnitude and arresteddevops.com/datadog. Thanks to Arthur and Michael for joining me tonight. And loyal listeners, if you enjoy Arrested DevOps, we’d appreciate it if you would visit arresteddevops.com/itunes iTunes and leave us a review in the iTunes Store. We would love to know what you thought of this episode. Please leave us comments or feedback at arresteddevops.com/tdi. Also want to give some attribution to the music you heard during this episode came from Bensound, which you can check out at bensound.com and is distributed under a Creative Commons license. You can check us out at arresteddevops.com or on Twitter @ArrestedDevOps. We’re always happy to get and put ideas or feedback at shows@arresteddevops.com, if you like that email thing, and let us know any ideas you might have for future episodes. I’m Matt at Matt Stratton. This is Arrested DevOps, and remember, there’s always DevOps in the banana stand.