Matty: [00:00:00] That was automation 20 years ago, was a tape roll.
Robyn: That was like the greatest day of my life.
Matty: So hey, welcome to Arrested DevOps, episode 44, Infrastructure as Code. I am your co-host Matt Stratton, @MattStratton on Twitter.
Trevor: And I’m your co-host Trevor Hess, @TrevorGHess on Twitter. Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that figures if you’re listening to this podcast, you must be pretty cool. You can find out about joining their cloud services team at arresteddevops.com/10thmagnitude.
Matty: This episode is also brought to you by VictorOps. From initial alarm to final retrospective, the mission at VictorOps is to make on-call suck less. In the live infrastructure timeline, get real-time context, and see annotated alarms with resolution documentation. Visit arrestedevops.com/victorops and sign up for a 14-day free trial to see how they’re actually making on-call suck less. I said that— I read that ad really badly, so I apologize, VictorOps. But that’s okay. We’re going to move on with our lives.
Trevor: [00:01:00] This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools such as Chef, Docker, AWS, so that dev and ops teams can share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a 14-day trial at restofdevops.com.
Matty: So tonight we’re talking about infrastructure as code, and, and this is not really talking about— it’s kind of a follow-up on our episode on configuration management, which you could check out at restofdevops.com. /9, but really thinking more about this idea of treating our infrastructure as code. So our— one of our first panelists is Josh Timmerman from Chef Software. So Joshua, can you tell us a little bit about your background?
Joshua: [00:02:00] Sure. I’m a software developer at Chef, and I’ve been here for a while, one might say. I’ve done a lot of different things here over the last 7 years that I’ve worked at Chef, and my main background is in systems administration and operations. And when I’m not working, I like to drink craft beer and play games of all kinds.
Trevor: That sounds awesome on all counts. Thanks, Josh. We also have Eric Sorensen from Puppet Labs. Tell us about you, Eric.
Eric: Hi, so yeah, I’ve been doing production operations for a long time. Probably my first real job was running an ISP when 14.4K modems were the new hotness. I used CFEngine for a long time, and I got into Puppet— my last job before I started at Puppet it was doing a public implementation at Apple, and now I work here doing product management, which I’ve almost got figured out what that means.
Matty: Awesome. Sounds like most project managers I talk to. And it’s funny, I did come up the same way, working at an ISP when, yeah, 14.4 was lightning fast. So I’m sure we can sit and compare abacuses sometime. And so rounding out our panel is Ansible’s very own Robin Bergeron. Robin, you joined Ansible recently, but tell us a little bit about yourself before then as well as now.
Robyn: [00:03:00] Sure. And look, I’ve got this handy-dandy name bar that you guys won’t be able to see when you listen to this podcast, but if you’re watching it on the internets, look. So hi, I’m Robin. I work at Ansible. I’ve been there for a month. I guess we get in the Wayback Machine. I was a sysadmin from 1996 to February 2000, which was obviously the right time to quit, not like in December or anything. I’ve been a business analyst. I spent 7 years not working and just raising my kids, which I, you know, attempt to continue to do while now having a full-time job again. My last big gig was working at Red Hat. I was the Fedora project leader for 2 and a half years. systemd is not my fault.
Matty: [00:04:00] We were definitely talking a little bit before. For listeners, you know that sometimes we have shows that feel interview-focused, and this is definitely going to be much more of a panel discussion. I wanted to get some folks that I wanted to have on the show and talk in depth about this topic of infrastructure as code. That first topic basically is going to be like, what the heck is infrastructure as code? Just when we use that word, What do we mean by that? To me, I know when I talk to people about what I mean when I talk about treating my infrastructure as code, it goes down to this. I like to tell— because usually when you’re talking about this, you might be talking to operational folks, and I say, DevOps is a two-way street. A lot of times, those of us who are SysOps or sysadmins or come from a TechOps background, When we hear DevOps, we think it means, like, make those devs carry pagers and give a shit about uptime, right? But there’s also that other thing of, hey, there are things that we can learn. Some of those devs over there, like, that part of the industry, over the last 20 years, or at least the last 10 years, have developed some really good practices, and we can steal them and do things like, I guess, at a high level, To me, treating the configuration and the way that you look at your infrastructure is that it’s— to me, it means that it’s versioned, it’s modularized, and it’s testable in an automated way. That’s what I think, but I could be wrong.
Eric: [00:05:00] There was a cool little conference here in Portland last week called AtomicOn. It was the first edition of that conference. Heavy Water Ops put it on, and we had back-to-back keynotes from Luke from Puppet and Adam Jacob from Chef on subsequent days. It was actually really cool. There was a meta-talk halfway through the second day where the speaker, whose name I don’t recall right now, did a survey of the previous talks, and of the 10 previous talks, like 7 of them started off with a definition of what infrastructure as code was. So, yeah, very similar to kind of the establish your ground rules and define what DevOps is before you start talking about it. But, I mean, I thought, for me, the most compelling definition that came out of that was, you know, if your entire infrastructure is destroyed by some natural disaster, if you’ve done infrastructure as code properly, you’ll be able to rebuild it in a new place using just the contents of a version control repository.
Matty: [00:06:00] Yeah, that’s the— Joshua, isn’t that the Jesse Robbins quote that I’m going to quite mangle, which is that it means that you can reconstruct your business as long as you have access to your version control system, a data backup, and compute resources.
Joshua: [00:07:00] Yes, but it was actually Adam Jacob came up with that originally.
Matty: Oh, and Jesse just says it all the time?
Joshua: Or said it all the time? We put it in slide material early on, so it just got propagated out. But Adam’s—
Matty: Joshua is my chef historian.
Joshua: Yes.
Matty: Every time I say something wrong, because I know it anecdotally, he’s like, Dude, you don’t understand. I was in the shit. I was there, man.
Joshua: And I’m also pedantic, so there’s that.
Matty: Does anybody have— I don’t want to say it necessarily— well, I mean, it could be a contradicting opinion or thought about that.
Trevor: I think you get started with infrastructure as code by having a repository. So a lot of the infrastructure folks that I’ve worked with have, you know, said, well, what’s Git? What’s TFS? Like, I just keep my, I just keep my, my scripts in Notepad. That’s fine, right?
Robyn: When you guys were talking about, you know, how you went to Atomicon— am I saying that right? Was it Automicon or Atomicon?
Eric: [00:08:00] Yeah, there was money riding on the pronunciation of that.
Robyn: But yeah, it was like, is it Automatacon or Atomicon? Or, you know, Optimus Prime and Megatron. When people You know, when you go and you see that, like, all of these people have a different definition that they’re laying out at the beginning of their— at the beginning of each of their talks, just to sort of level set, you know, what, you know, sort of a common language. I think what that really sort of indicates is that people don’t just have, like, one set meaning for themselves of, you know, what this actually means, right? Every time you think about it, it’s always subject to a new experience or a new thing that has, you know, either gone very well for you or occasionally has been, like, a complete utter disaster for you, which kind of changes how you think about what that actually means for yourself. So I think in a lot of cases, it’s making sure that you can, whatever it is that you’re doing, that you’re doing it in a way that you can continue, that you’re open and willing to evolve, you know, that what you’re considering to be infrastructure as code and what that actually means to you.
Matty: [00:09:00] Yeah, I mean, I think that’s why some of those definitions we’ve talked about, they’re kind of purposely high-level, right? They don’t go in as— and you can probably swap some of that out. And I think there’s even been some stuff with that, now that I know it’s an Adam quote, but whatever, where I’ve even seen us move the words around a little bit as things evolve, but the intent is still very similar. So I’m actually interested, not having gone to Automatomicon, as to just kind of what are some of the other definitions people had, because I’m just curious, you know, at a conference like that, you know, what were people considering that was different than what we’ve kind of just talked about?
Eric: I think there was pretty widespread agreement that, fundamentally, it meant using good DevOps practices and pulling software— as you mentioned, pulling software development practices into your infrastructure operations and trying to make things go both ways. There’s a, you know, I think there’s an interesting— you mentioned 10 years. I think it’s actually way further back than that because, like, if you think about RCS, that’s been around for a really long time, and yet, as Robin pointed out, it can be a novel experience for people to see your infrastructure, you know, your scripts and whatever it is that you use to run your systems, checked-in versions, having good commit logs and a good audit trail and being able to go back through it from an archaeological sense and understand why something is the way it is in your Kickstart configuration or your Chef or Puppet configuration and be able to divine intent from the current state of things. Just that can be a fairly revolutionary concept for a lot of people.
Matty: [00:10:00] Yeah, I think that’s like what Trevor was saying is a lot of admins, what they know about Git or TFS is, well, there’s that one server that my devs use that I have to keep running for them. That runs Git on it, I think. Or again, sometimes you have folks keeping scripts in source control, which is— but they’re not necessarily versioned, but at least it’s a place to keep them, which is a start. Kind of what I was talking about, again, I was just sort of throwing the data out there. I was thinking kind of like the dawn of extreme programming and things like that, not necessarily— because it’s more those overall practices. And the other thing is, like, I’ll talk about something like being 10 years ago and then realize that I’m older than I think I am. And something that seems like it was only 10 years ago was really, like you said, 20 years ago, and then I go and I’m sad.
Robyn: [00:11:00] Yeah, I think about 20 years ago and it was really cool when we automated all our things with Expect and when we got the big tape robots so we didn’t have to go around with little DLT tapes to the 200 servers, the 4 of us.
Eric: But I think the next level up is that test-driven development, and that didn’t exist when I started out doing configuration management even. The concept of unit testing your CFEngine code was pretty revolutionary. And now it’s kind of like— and I actually really credit the Chef community for really pushing this forward and being like, if you’re not testing, you’re doing it wrong, and making the tools and the sort of supported ecosystem around that so that it’s easy to do the right thing.
Trevor: [00:12:00] It’s always so interesting and powerful when I introduce somebody to testing the infrastructure code. Because they’re like, wait, I can actually like make sure the thing I did actually was the thing I did? That’s incredible.
Joshua: One of the things I like about the unit testing aspect of testing the configuration management code is that you’re testing the inputs because we— I don’t know anybody who has the benefit of working in an environment where they only have one single version of one single operating system. It’s all different everywhere. You’ve got, especially if you work in big enterprises, you have everything and it all has to work together. And if you’re using Puppet or Chef or I’m sure Ansible has cross-platform like logic in some way, you have to do conditional branching and find out, you know, if I’m on SmartOS then I have to do these weird things because SmartOS is different than Linux and both of those are completely different than Windows and well, we just might as well just toss everything that we’re doing with that, because it’s just not going to, like, map the same way. So being able to test that kind of stuff in a unit testing framework has been super valuable, because we can catch regressions. Like, somebody messes up the support for our popular— our most popular platform, right, whatever that may be.
Matty: [00:13:00] I think the thing is, like, the idea of testing things is not foreign to sysadmins, but the idea of being able to automate the testing of them. So we’re used to saying something like, well, I’m going to try my script and see what it does, and I’ll do exploratory testing and things like that.
Joshua: [00:14:00] That was my life.
Eric: Yeah, this was another great AtomicCon quote that everybody has a test environment. Some people are fortunate to have that environment be different from the one that runs your production. Different.
Robyn: My test environment and the way my tests are confirmed is via Twitter when everything goes down. Bye-bye.
Matty: I was thinking about that, too. There was a conversation on Reddit. I wanted to bring this up, and there were some good follow-ups on that, but I’d like to, I think, surface this a little bit more. Again, we keep talking about— it goes back to this idea of test-driven development for infra code. I think sometimes that can be kind of hard because There’s so much dependency that happens. It’s not as easy as it— and not that I’m saying that it’s easy. I don’t write real dev code. But presumably, if I’m sitting and writing a module or a class, I can put a breakpoint in and say it only needs to go this far to see if the test is green versus there’s so much stuff that could go down the path in my config code. I like to write my tests, But I have to write some code to get them around. One thing that I always thought was really interesting when I was helping people learn this is a lot of the, oh, I don’t have time to write those tests. I don’t have time to write the tests. As I say, the thing that you don’t realize is that after you’ve written the tests, writing your code actually got— your infra code actually got really easy because you did a lot of the big thinking in writing the test. You’re implementing that thing you wanted to do. So, in some ways, it’s kind of test-driven, but maybe not necessarily in the mechanism of red-green refactor test-driven. What are your thoughts on that?
Eric: [00:15:00] Well, if I could go— I think the critique that you’re talking about, if it’s the one that you put in the show notes there—
Matty: Yeah, I didn’t get to actually reading it out loud yet. If you would like to, you can.
Eric: [00:16:00] The comment is, if I can sum it up— I’m not going to read a Reddit poster’s thing word for word. It’s just dangerous.
Matty: We’ll put a link to it in the show notes.
Eric: But the point of it is, like, just— I would sum it up as just doing unit tests for your configuration code is not sufficient, which I totally agree with. That’s totally true. It’s good. It’s good, but it’s necessary, but it’s not sufficient to get good coverage. But I think that the world has kind of evolved since the post came out, and sort of the state of the art has advanced to actually doing integration-style tests with infrastructure code. And I think this is a fairly new development. There’s a lot of stuff that’s still in flux, certainly in the Puppet community. You know, there are some— we have, like, RSpec Server, we have ServerSpec, we have a few different tools that people are using. But the upshot is that once you have a framework which allows you to spin up a set of machines that are systems under test and to provide assertions about the way those machines interact with one another using the infra code, then you’ve gotten a lot further down the road, and that’s something that you’re doing more than just, like you said, like Trevor said, testing that what you said you did was actually what you said. You’re actually testing the interaction with those systems.
Matty: [00:17:00] It’s one of the things where I’ve seen, and this is something that sometimes we get into, you know, I’ve gotten into internal arguments with people of the technical level and expertise with the tool, like Joshua, who are really good at this. And I’m not saying I’ve had this argument with Joshua, but maybe we have. But where they’d say, like, I wouldn’t write a unit test to say, is nginx installed? Because Chef just took care of that. Why do I have to test for it? I only would write that unit test if I’m doing the branching logic. And my argument is, A, for people who are new to Puppet or Chef or whatever is the technology that you’re using, have them write that test. It’s useful. But it’s also useful down the road for regression, because sure enough, you’re like, yeah, well, I know, but maybe someone later is working on your module or your playbook or something, and they do something that actually would make that not happen anymore. But they don’t know that that’s a thing that matters, so it still has to pass your testing. That’s that thing. Then it’s also the way that I’ve kind of heard it described is you think about the signal-in, signal-processing, signal-out stages of testing. Signal-in is like when we talk about unit testing, which is, did I write a properly formed piece of code? Does my code— that might include things like loading, but also unit testing, like if I call Puppet to do this, would it actually do that?
Trevor: [00:18:00] Yeah?
Matty: Cool. Signal processing is the— does Puppet actually work? Does Chef actually— does Ansible actually work? And the good thing is, unless you work for one of those companies, you don’t have to care about that. We’re doing that for you. But then the signal out is the things like the acceptance testing, like the— okay, so if I did install and configure NGINX on a website, there should be something listening on port 80. And as was described by my colleague Chip, it’s the, is what I said what I meant kind of testing. And then you can take it even further to, Eric, to your point of saying you can test actual full infrastructures as well. But even if you don’t have to do something that complicated, it’s about— because what you said may not have been what you meant or what you needed.
Trevor: [00:19:00] So I think you bring up another interesting point, Matt, which corresponds to something else you just said. When you’re mentioning not necessarily knowing what it is you’re trying to test, and that’s why it can be challenging. Challenging to do blue-green, going through with somebody who’s just getting started and teaching them how to test that NGINX is installed or that the service is running, that helps them learn the patterns so that they can identify solving a more complex test in the future.
Matty: Oh, for sure. It’s— yeah, you don’t want to make your, your first time you write and, you know, a Chefspec test something that’s crazy branching, because then when it doesn’t pass, you don’t know why. Like, is it because you wrote bad spec tests? Is it because you did whatever? Or just like do just a simple file matcher so that you know how to do that so you can move on. I think that’s something in the community that we need to think about is that— and I think a lot of the tools do allow for this— is being embraceable by someone who’s new to it, but that we have to remember, and it goes back to that whole, like, well, why would I write a unit test for something? This goes back to that Reddit post. The person’s answer was, why would I do that? Because isn’t that what Puppet does in the first place is make sure that package is there, so why would I need to test that it was? And it’s because it’s a way of learning the more complicated way of testing.
Joshua: [00:20:00] So on the whether you should write tests for simple resources in those, you know, in your recipes or in your manifests or whatever, I’ve come around on that and found that I like the confidence of having 100% test coverage for all the resources in my recipes because That way, if one of them gets deleted accidentally because we go on a refactoring spree, then I don’t have an unexpected result where, oh, now the package doesn’t actually get installed and I actually needed that. So I have to make a conscious choice to go and modify the spec in order to account for that.
Matty: [00:21:00] And I think that brings us into thinking about using those principles that besides being able to test as the individual developer, but having this test coverage and having these tests need to pass in order for you to actually release is a key of that. That especially goes into why, if you’re doing something like that, if you’re deploying your config code through a pipeline of some sort where it has to pass tests, then you very much care about writing regression tests or making sure they’re in there. But what you don’t want to do is regression tests should never be tacked on later. You build them You put them because you had a problem. A lot of times, well, sometimes they’re tacked on later because you discovered a bug, right? I think it’s Paul Reed who will tell the story because I think he was at Mozilla at the time where, I think it was in 2005, on the first day of the World Series, Mozilla released a version of Firefox that broke MLB.com. So there’s still, at least up until the last time whenever I heard him talk about it on the Ship Show a year or two ago, there is still a regression test in all builds of Firefox to make sure that MLB.com still worked.
Joshua: [00:22:00] Nice.
Matty: Because they would never break the World Series again. So when we think about this, and I want to think about this idea of using a pipeline for infra code, and both from the perspective of a greenfield, I’m not doing infra today and I want to do my Ansible project now or whatever, I’m going to start doing things the right way, not the kind of canned way I’ve been doing it. Versus the, I have a really nicely embedded implementation of Puppet, but now we want to do this in a DevOps-y way. So what are the differences and challenges you might have in either/or of those scenarios, do you think?
Eric: Time is one of them. I mean, the more tests that you have— the more tests you have and the better tests you have, the more time it takes to get through them. And I mean, that’s— sorry, that’s a little tautological, but it can get pretty extreme. And, you know, if it’s been said that a process, the accumulation of process, is like your organizational scar tissue, if that’s the case, then just like your MLB example, your acceptance tests are your codebase’s scar tissue. It’s just stuff that has gone wrong over time that you want to make sure won’t happen again. And over time, that can be like, you know, like 2 to 1 in terms of lines of code versus the actual codebase itself. That can be That means that if you have a lot of platforms and you do a lot of tests, it might take a lot of time to get through a run, and that can slow down your ability to release. It can slow down your organizational agility.
Joshua: [00:23:00] Yeah, it turns out that developing software is hard, and that more and more, the way that we do infrastructure, the infrastructure itself looks like an application. So, you know, especially when you’re talking about doing things with infrastructure as code, so having proper testing and doing proper software development is super important, and it’s worth the time that it takes to do it and to do it right. Because when you accumulate that technical debt, like everybody does, when you come back to it in 6 months or a year or even longer, it’s gonna bite you, and you’re gonna have to refactor the entire world anyway. So you didn’t really save any time by taking shortcuts. Now, there’s a difference between that and things like, you aren’t going to need it, right?
Matty: [00:24:00] Like, yeah. So, correct me if I’m wrong here, but a lot of it, I think, was talking, again, like you said, the time, especially the time when you’re talking about migrating something that’s already there to the new way of doing it versus doing it the quote-unquote right way the first time. I think that’s usually why when I talk to folks who are getting started, I say, start doing it right the first time because it’s going to be— it’ll seem like it’s making your life a little bit longer, but if you ever want to do it the right way— I’ll usually say something like, released through a pipeline that does no tests, but you have it there, so now you can add those tests later, right? But have the habit so that from day one, you know that the only way that you can push up a new playbook or cookbook is by committing it to source control, not by uploading it directly to a server somewhere or just doing a thing with it. And then it builds those habits, because habits are hard to break, whether they’re good or bad. And I think that’s the challenge with moving folks who’ve been doing things one way because it works, right? I mean, if these tools didn’t work and didn’t make people’s life better, everybody except Trevor wouldn’t have a job that’s on this podcast right now, right? You don’t have to do it the fancy DevOps way to get value out of any of these things. In fact, I don’t know, so does it provide that much more value? I mean, let’s play devil’s advocate there. Why do I give a shit about doing this the DevOps way? Why can’t I just use Puppet the way that I want to, or just run playbooks? I’m a sysadmin. I keep them. I know what I want. I’ll keep them on my laptop and just go.
Joshua: [00:26:00] That’s kind of cool to do that if you’re the one person that can get away with those kinds of things, but in the real world, we want to be professionals and we want to do good work. Because if we— when we don’t do good work and we don’t act professionally, then that’s where we run into problems, where we have a deployment that goes wrong, mistakes that get made. You know, these are complex systems that have a lot of cascading factors. So, you know, if you say, oh, I’m just gonna push this quick change up, and then, oh wait, why is it taking that long? Right? Like, who hasn’t had that experience anyway? And, you know, when you don’t have proper automation around that, being able to recover is it becomes super difficult.
Eric: I think it’s fine as long as you don’t have any other— any coworkers or any customers.
Robyn: Yeah, if you’re a one-man startup, then that’s perfectly fine because you are the person who is having a bad day. But instead, otherwise, you’re basically being sort of disrespectful and making other people have bad days.
Joshua: [00:27:00] Totally works on my Minecraft server down in my basement, but you know.
Matty: So what about Aligning your thinking about— now this is thinking about a more advanced topic, but it’s something that I think we like to talk about is this great idea that we aspire to, which is aligning your infra code to your app code. Because in theory, I’m not making changes to my infrastructure unless it’s in service of this application service that is the reason that my company is here. I guess I’m curious to see, how advanced of a feature— and by feature, I don’t mean feature of the product, but I mean feature of the idea— like, if it’s like a ticky box of advanced feature to do that, like, how advanced do you need to be? And really, what are some of the things to think about if you’re gonna pro and con or positive/neg when you’re gonna kind of look at that?
Trevor: [00:28:00] I think it depends on whether or not you have a mature deployment tool in place already or not. And it also depends on whether or not you have a level of complexity that would, that would make it so that you would need to create your own tools in the infrastructure automation tools to deploy your code, because oftentimes you’re really just putting files on a system and making sure that services are started, which is just more system configuration. And if you can, if you can use the tools you have for infrastructure code to version everything and use your packages the way you would. I’m starting to find that it’s easier to scale letting the infrastructure tool deploy the code. Like, I’ve been, I’ve been bashing my head against Octopus for the past 2 weeks to get it to, to, to work with arbitrary environments while not giving it access to— not giving arbitrary environments access to production. And it’s just been a pain in the neck, and I would be so much happier if I let a tool besides Octopus that was managing the servers deploy the code.
Matty: [00:29:00] So that’s, so that’s argument for just saying at the end of the day your application is just another configuration point anyway, so use your configuration management, you know, tool or your, you know, your infra code tool to actually do software deployment as well is definitely a pattern that can exist. I’m also thinking one of the challenges that comes up when we go to talk about this is that, again, first of all, I will tell people, they’ll say, well, can you use Chef to deploy software? I’m like, well, of course you can. I say, but if you already have something, if you have a rich Capistrano thing set up already that works for you, that’s great, then keep doing that. Because just in general, I always recommend to people, I’m like, when you’re getting something new, solve your gap problems first. Don’t go replace the shit that actually works. With another tool because you didn’t accomplish much except get something different, right? Fill your gap. But I think it requires a super high level of DevOps, if DevOps is a thing that can be measured, to be able to do this because whoever is writing the infra code and managing it, so to speak, or is authoring it, needs to be now very tightly coupled from a pipeline perspective to the app code if you want to really be able to align them together You can’t have the infrastructure team using Git and Jenkins and the application team using TeamCity and Mercurial because those pipelines— it’s almost like a common pipeline or parallel pipelines. I don’t know. Am I making sense?
Trevor: [00:30:00] I think that helps.
Robyn: I was just going to say, you used the magic, the D word. DevOps, and I think sort of the substitution there might just be like you have to have a high degree of sort of trust and transparency. I think, and I think this is where Trevor was going to go, is I think for those things to be even remotely effective or possible, like there’s got to be a base level of simplicity, right? Like if these, if one group is like, we insist on using this tool, and it’s a tool that other people are just like, dude, I do not have like a month to even learn that, like, you’re just— that’s sort of the beginning of an unhappy relationship. So I think it depends on, you know, how simple your common languages are between language tools, whatever, are that are, you know, sort of shared between those 2 groups. I mean, for that to be like a surmountable problem rather than a maybe someday when we all know how to use each other’s things.
Trevor: [00:31:00] Right, and that’s kind of where I was going, was You don’t have to immediately stop and say, okay, let’s all converge on 2 tools. Let’s use Git, and let’s use Travis CI, and everybody will be happy. You can, if you’re— using Matt’s example, if the devs are using Mercurial, they can package up the software and give it to the infrastructure team initially, but that’s just another wall that you need to take down. But doing it all at once is not necessarily a solution either.
Robyn: [00:32:00] I think the key is, well, and Matt touched on this, definitely baby steps. Like, you can’t just burn everything down and, you know, don’t fix what isn’t broken and take steps here and there because otherwise you start, you know, if you have like 25, we tried this and we failed, we tried this and we failed, people get very sort of demoralized about things like that. Small successes one by one like not only make people feel good, but it’s like you’re getting like 5 minutes here and 10 minutes there. Next thing you know, you’re like, whoo, I’ve got like a whole week to, you know, I’m not patching fires anymore. I actually have time to like move us forward and do amazing things.
Matty: Yeah, you’re chipping, you’re chipping away. It’s continuous improvement for sure. And I’m a— I tell people all the time, and my coworkers are, I’m sure, tired of hearing it. It’s one of the, one of the Matt Stratton and solution architect tropes. Which is the crawl, walk, run, right? Don’t have the first thing that you want to do with Chef be go manage your big storage array. Learn how to install Apache first. Do something that— don’t pick the most complicated thing to be your very first thing. For often good reasons, that’s why people want to do that because they’re like, well, that’s a thing we haven’t had a good answer for, so we better do something with that. But you have to crawl before you walk. Like Robin said, you get small successes and they work for you from a time perspective because you can do little things, which then give you more resources to do more things. But it can also be very much when we think about making any kind of a transformation in an organization, evangelizing even in spite of yourself. Yeah, it’s just proof of it. When I think about organizations I’ve been part of or seen who go through agile transformations, You don’t do the whole thing. You have a pilot project and you stack the deck on that project, right? You pick something about this part of the company that has value but is not going to be some super legacy thing, and you’re going to staff that with the people that you know are the most open to change and everything. So it’s going to be really successful, and then everyone’s going to say, wait, I want to do things the way those fellas over there are doing it because they’re being super successful. And then the same thing, if you start aligning to these small successes, and then people say, wait a minute, We’re not having these outages, this particular outage we used to have all the time, or now when we build servers, this one thing is a little better. It’s like, now we can make some more things better. Trevor in the chat just said it’s trickle-down DevOps. So I don’t know why I didn’t say it out loud because that was worth it.
Trevor: [00:34:00] I don’t know. I thought it might’ve been a bad connotation.
Eric: There’s definitely been some good, you know, high-functioning organizations that I’ve been a part of that, um, almost certainly by accident, but when we were able to get to a place of really collaboration, what that looks like in terms of infrastructure as code is stuff like the application developers start adopting in their pipelines the ability to output versioned native operating system packages, like versioned RPMs. Instead of it being like an obscure tarball that’s plopped down somewhere, a WAR file or whatever, it’s actually a package that can sit alongside the operating system packages and can be managed by the configuration management system, the same thing as the rest of the code. So there’s a more seamless continuum and not that hard line in between, like, well, you just manage the operating system, we’re going to take care of the application ourselves. And in that sense, that really goes back to that idea of the shared responsibility and that the blending of, you know, sysadmin practices like building RPMs and spec files and that sort of stuff into development practices of like, you know, here’s the ultimate— the end state of our CI pipeline is something that’s actually consumable by the team that’s going to be responsible for pushing that out onto the machines.
Matty: [00:36:00] Yeah, it’s like adjusting those inputs, and I think that goes back to thinking about what Robin was talking about, having the transparency and the trust is— and this, by the way, for anyone who’s listening, is when we talk about empathy in DevOps, that’s what we’re talking about, right? Is that as a developer, I understand and empathize that it would make my ops person’s life easier if I gave them an RPM versus something obscure. I mean, there’s more to it, but that’s what that means. The reason I’m kind of ranting about it is there were a lot of talks about empathy at DevOpsDays over the course of last year, and this was used by quite a few people to talk about why the DevOps community was just about feelings and hug ops and not doing real work. But that’s what we mean. I think getting to that place helps you really accelerate your velocity because it’s one fewer steps, and you’re able to now focus more on doing smarter things, right? Like, hopefully, you’re— the challenge of most sysadmins is that they have to be very technically advanced, and you have them do very mundane things. And it’s that great quote in the Jez Humble, Dave Farley book, Continuous Delivery, which says, having a very skilled person do mundane topics is the surest way of ensuring error. Short of sleep deprivation or inebriation. So the more that we can— not only does it like free up people’s time so their work life is better, which just as a company we should aspire to, we actually want to get our money’s worth, right? You know, I mean, I, I, I— your ops people are, and your devs too, but you probably have a lot of people that are doing things that, that they have the ability of providing even greater value to your organization. They’d be happier, sure, and I would be happier knowing they’re happier.
Eric: [00:38:00] Imagine a world where Jordan Sissel never had to write FPM.
Matty: Yeah, I’m just— I’m glad that at least I live in the world where he did write it.
Joshua: You know, be better if it would never have to be written.
Matty: It would be worse to live in the world where he never wrote it. Yes, but because—
Eric: but that’s a response to— I guess my point is that’s a response to that kind of low-trust environment where things come out of, you know, that we don’t feel like we don’t have control over or the ability to influence what’s delivered from on high to us. And so we have to write these tools, which is total, you know, FPM on its own is totally awesome. It saved my life on more than one occasion, but it saved my life in an environment that was a low trust environment where I couldn’t go to the team that was delivering me that code and be like, hey, this thing would really be a lot better if it was, if I had the ability to install it in a repeatable way.
Robyn: [00:39:00] I think it’s interesting that, Matt, you brought up the comment about, you know, we talk about this stuff at DevOps Days. I think a lot of it is those of us who are from companies who are also open source projects, right? So we see this on both sides of the coin, where you have to have trust and transparency about what people are sort of developing and contributing in a community, and then we see those patterns sort of repeat with, you know, our customers or even in our own companies because we’re all sort of, you know, doing these things on our own again. So it’s like we realize doubly how important these sort of cultural things are and how important they are to making things happen, like in our communities and in our workplaces. And I think maybe we get so doubly exposed that we do talk about it a lot, but I think it’s also because it’s important and matters a lot because without it, it’s, it’s, it’s Yeah, no transparency to be had.
Trevor: And quickly, with the empathy and the DevOps days, we have to be careful to remember that we claim we understand empathy, but then we don’t acknowledge the whole picture. Like, I know one of the things we talk about a lot is remembering that as part of this whole movement and empathy, that we also remember about our sales folks and our marketing folks and including them in the picture. And I know I actually had a conversation with part of the marketing team at Tenth today about— they were really upset about a comment that was made at a DevOps event they were at recently, where somebody had tweeted, why are the companies here? They don’t get it. Why are they here? This isn’t about— this is about empathy and culture. Why are the companies and their salespeople here?
Robyn: [00:40:00] Well, and I mean, the answer to that is so that we can develop empathy for people who are using it and understanding what the heck is going on. I mean, without that, it’s a gigantic feedback loop. It’s like meta-meta-meta-DevOps. It’s like unicorn.
Matty: [00:41:00] I would direct them, and not to go way off on topic, but direct them back to our marketing episode. We just talked about all this stuff. There were actually some great open spaces. There was a great open space at DevOps Days Chicago last year that was about that. I think that the point is, again, whether— wherever you’re— as part of the process of delivering software, thinking about where those inputs and those outputs go, and Eric, to your point, thinking about when I’m passing something to you, or just as much, like, I might sit there and say, like, yeah, if we totally automated all this infrastructure with Chef, it would be super rad, but not realize what the impact might be on some other people in a different part of the pipeline and figure out. That doesn’t necessarily mean don’t do it, but it’s don’t go and steamroll it and make it this thing. It talks about any type of transformation. Again, the bigger you can make the tent and realize that these things are not really— when we’re talking about our infrastructure as code versus just pure straight-up config management, which I think are 2 different things, one encompasses the other. But if we’re doing this, first of all, if we’re smart, people, we’re going to absolutely involve people more than ops, if for no reason than we probably don’t know crap about the right way to use some of these tools like Jenkins and Git and stuff like that. Maybe we do, maybe we don’t, but hey, or they may already— our dev friends may already have a beautiful continuous delivery infrastructure that we can just utilize. It also is impactful to that, and I think at any time, I think this is— This is an area where it can work, that collaboration and transparency can— it’s either going to pay massive dividends for you if you do it that way, or you whiff on it and it’s going to really hit you. There is no middle ground on that one, I don’t think. I think you either do it well or you do it poorly.
Robyn: [00:43:00] You’re either an open source software company or you’re not. You can’t just be like, here’s some code, but we’re not willing to listen. Because that’s just not an effective, you know, and it’s the same thing in the workplace. Like, yeah, we’re all about the DevOps, but we’re not actually going to talk to you. Just go back to doing things the old way. Don’t pretend and lead people on because that just makes people cry. They think they’re going to get a unicorn and they’re just going to get a box of Kleenex for their tears.
Trevor: I think most people get it, but I had to kind of remind myself teaching some people about how to use Git and how to use Jenkins recently that, you know, these may be people that you— that know lots of things about certain areas, but Jenkins and Git are completely new to them, and it’s, you know, you have to, you have to go from the beginning and explain it. You can’t assume that those, those things that are now intrinsic to you are intrinsic to them, even though, you know, they may, they may have equivalent knowledge about something else and that may be how you’ve come to know them, you still need to understand that it’s a new thing for them.
Matty: [00:44:00] So before we move into our wrap-up, I just wanted to know, and especially for— I’ll kind of direct this anywhere, but in a little bit of a way I’m directing it more towards Eric and Joshua just because they’re more historians on their particular respective place. But what do you think is really different in the world today of treating your infrastructure as code versus when you first started working on something like that? I don’t mean first started, like, going back to when we were using Expect, Robin. I mean, like, when we were first— your first experience with truly treating your infrastructure as code in the way we describe it today. What’s different? What’s changed?
Joshua: Everything. Everything has changed. I’m not actually being snarky about it because there’s— the way that we build infrastructure, the way that we think about infrastructure has completely changed in the last 5 years, let alone in the last 10 years. You can’t be a systems administrator or operations person and survive in the technology field today without having an understanding of how to write some kind of code. I don’t care what it is, and I don’t care what your sophistication level is, but you have to have an understanding of being able to do it. Otherwise, you just can’t keep up.
Matty: [00:45:00] Everything has changed.
Eric: I would say there’s— I mean, for me, there’s really 2 kind of pieces to it. One of them is kind of the cultural aspect, and it’s the movement, you know, the part of DevOps that involves operations people taking development processes and development folks understanding what it means to build, say, monitoring, you know, exporting monitoring out of their application as a first-class thing instead of having to try to poke through it as a black box. And the other part is the technological aspect of it, where there are tools and frameworks for testing infrastructure code, for testing, you know, your Puppet code or your Ansible code or your scripts even. I mean, I looked at this Bash test framework the other day, and so even if you’re running your automation in Bash, you have no excuse for not doing tests now. And that was just sort of unfathomable even about 5 years ago.
Robyn: [00:46:00] Yeah. I’ll throw in, even though I’m not totally a historian, if that’s okay. I think the other big thing that’s sort of changed, and I think it’s sort of a cultural thing, but 10, 15 years ago, if someone came around and said, I’d like to use some crazy open source stuff and maybe use some ideas that other people put on the internet, it would have been like, oh my God, absolutely not. That’s just not even an option. I think, you know, the fact that open source is sort of an acceptable thing these days and that there’s a huge community of, you know, DevOps folks out there who are sort of exchanging all their ideas and evolving what, you know, infrastructure as code actually means to each other or, you know, in their own ways over time, like, that’s not something that None of these ideas would have been implementable, you know, 15 years ago. It would have been like, yeah, no, sorry, those— that’s no free hippie code. Like, that’s just not kosher here at the workplace. And that’s completely changed. I think without that piece of this, just like, you know, having not 14.4 modems and everything else that’s changed over 10, 20 years, we wouldn’t be where we are now.
Matty: [00:47:00] I think that transparency, and especially with regard to enterprise companies. I mean, not necessarily that they are— now, again, their openness to open source and things like that is one thing, but I think back to even just over the last couple of years ago, the reason that I would go and talk to places, this was pre-working at Chef, but just even a couple of years ago when I was at Tenth, and they would say, you are the 5th person to come in here today and tell me that I should be like Netflix. That was all the success stories around doing these things were Netflix, or Etsy, or Facebook, But the— and I always had this sneaking suspicion that I’m like, this is working for other people, but they’re not allowed to talk about it. So the only people that can talk about it are people from Etsy and Netflix and Facebook. And then so we have things all of a sudden, you know, company— and again, because companies move slow with their ability to realize that the stuff they’re talking about is just plumbing, it’s not competitive differentiation. So you have things like Doze, right? You have things like Target coming out and saying like, this is exactly how we do it. GE, this is what we did, all this success. And it gives a lot of these companies courage to do that because it’s— and, and it’s, to me, that’s like a massive thing besides— because we knew that open source stuff was behind the firewall, you know, over the last 10 to 15 years. That’s been happening. But what hasn’t been happening is them sharing the stories. And that happening now, I think, is what’s going to— yeah, well, and Netflix helped with that in the beginning. Netflix sharing that stuff, I mean, even them sharing their code, but drove a lot of things. And yeah, I think that’s a transformational moment.
Robyn: [00:49:00] And I think companies being willing to let— I think companies have also recognized that, you know, if you want to be able to retain your employees, like, they like to be able to put their stuff on GitHub so that other people can see that they’re awesome and they can write things and that, you know, people like to go out and talk about what they’re doing. And when they can’t, sometimes maybe they think like, this isn’t the company I want to work with. So I think some of it’s also just like, we’d like to listen to our employees and make them happy. And so I’m glad that people are feeling like, yes, people want to hear my story, even if it’s, you know, like my tale of failure or my tale of awesome, whatever it is. Like, you always have lessons learned, and I think companies are even okay with people going out saying like, yeah, we totally sucked on this one. We’re waiting for VW on that this week.
Matty: [00:50:00] Yeah. So we started late and we’re running late, but we’re— so I’m going to move us into our wrap-up stuff, but we’ll— and we have a, we have a whole bunch of it. But we’ll go through it at a reasonable pace. Some upcoming conferences. Operability, a whole bunch of DevOps Days, Velocity New York and Europe will be in October. For the DevOps Days that are, if you go to devopsdays.org, you can see all that stuff that’s happening in October. Very first DevOps Days Detroit will be November 11th through 12th. If you’re a listener to ADO, you can get a 10% discount with the code ADO10. It’s ADO and the number 10. Right after that, the very next week, is the very first DevOps Days— oh, DevOps Dies? I got it— DevOps Days Ohio, which is in Columbus, and that’s November 18th and 19th. You can save 10% there with the discount code Arrested. Yes, I tried to ask them to both use the same code. Don’t get me started. The Chef Community Summit, if you use the code ArrestedDevOps, you can get 20% off your registration. The Seattle version of the summit will be October 14th and 15th, and the London version is November 3rd and 4th. Go to chef.io/summit. PuppetConf is going to be October 5th through the 9th in Portland. It’s not too late to register. Eric, do you know where people go to do that, or just search for PuppetConf and they’ll find it?
Eric: [00:51:00] PuppetConf.com, unsurprisingly.
Matty: There you go, it’s that easy. PuppetConf.com. And Ansible Fest is November 19th in San Francisco. Go ahead and register away. Robin, where do they just go?
Robyn: ansible.com/ansiblefest.
Matty: There you go. That easy. The Config Management Camp in Ghent is going to be in February, but the CFP should be coming soon. If you follow @cfgmgmtcamp on Twitter, you’ll find out what’s going on with that. Coming up, I am traveling a whole bunch. I’m leaving. I’m going to spend a few days on vacation in Palm Springs. And where it’s going to be like in the hundreds while it’s like nice and balmy and 75 here in Chicago. But next I’m going to be in Phoenix for a chef event on Tuesday and Wednesday. And this time maybe I’ll see Robin and we’ll recognize each other.
Robyn: [00:52:00] Yes, unlike last time that was—
Matty: Robin and I like were at karaoke together with a group of people and introduced each other only— introduced ourselves to each other only by first name and we hadn’t met in person before, just on Twitter. And then I realized like later in the night that I was like, oh my God, that was Robin. And I sent her this like either a DM or a tweet and I was like, oh my God, I’m such an asshole. And she’s like, I didn’t know it was you either.
Trevor: Yeah.
Robyn: Well, I think, I think the worst part was that I felt really bad for Mr. Doocy who was like, this is Robin and she works at Elastic. I was like, um, we should—
Matty: About that.
Robyn: Yeah, about that. I just got a new job. Where are you working?
Matty: Yeah.
Joshua: Oh well.
Matty: [00:53:00] You’re at the sales offsite event, Jeff, and you work friends. It’s okay. We’re all friends. Hug ops all around.
Robyn: It was drinking time.
Matty: Exactly. I’m gonna be in Austin. I’m going to Austin City Limits, and I will be at the Chef Community Summit as well. So Trevor?
Trevor: I will also be at the Chef Community Summit, but before that, I will be in Norwalk, Connecticut, the 5th through the 9th.
Joshua: October.
Matty: You have a lot of listeners in Norwalk.
Trevor: It’s actually just outside of New York City, so we might have some—
Matty: Norwalk represent! If you are from Norwalk and you listen to Arrested DevOps, send us a tweet and tell us.
Trevor: And hang out with me. You can do that too.
Matty: If I release this show before— I don’t know if this is going to get released before you’re there. So besides hanging out maybe when I’m in Phoenix, Robin, what do you got going on?
Robyn: So I’m going to OpenStack Summit in Tokyo at the end of October. So I’m like, it’s one of those like I’m leaving on Halloween, I’m flying for 27 hours, and I’m still getting home in time to take my kids trick-or-treating, which is awesome if they want to go trick-or-treating this year instead of playing Diablo 3. And God only knows what else in the interim, and then I’ll Be it Ansible Fest, obviously, and all the other things. But yeah, this is sort of like a lay low month.
Matty: [00:54:00] Yeah, we’re saying like the next couple weeks.
Robyn: Yeah, AWS, I might go to re:Invent. I don’t really know yet. Haven’t decided.
Matty: Eric, I assume you’ll be at PuppetConf for sure.
Eric: I will be at PuppetConf.
Matty: Do you want to make work cool? Any cool events in the next couple weeks?
Eric: Well, for any listeners who are also runners, we’re doing a 5K the Thursday of PuppetConf. So if you’re attending and want to run, check it out on the schedule page and hit me up If you want to volunteer to help people guide around the waterfront. It’s going to be a good time. We had 50 people do this last year. It was fun.
Trevor: Oh, sweet.
Robyn: Running?
Joshua: Hmm. Yeah.
Robyn: I hear it’s a thing.
Matty: So let’s— it is a thing, this jogging. Let’s go into our checkouts and retro. We’ll start with you, Joshua. So just checkouts, I mean. I don’t know why it says retro.
Joshua: [00:55:00] Maybe we’re doing retro checkouts. Maybe I should pick some music from the ’80s or something. First checkout is Chef policy files. This is a new feature that we’ve been releasing various parts of over the last few months. I’m doing a webinar next week, which is probably gonna be last week by the time the episode is published. So, September 30th, there’s a webinar, and there’s a link in the show notes. So, you should be able to get access to the recording of that for that. And then ChefDK, which should probably be out by the time you hear this if you’re listening to it on, you know, iTunes or whatever. And that’s got some new features and fun things for policy file usage. And then my last checkout is to check out Fat Scotch Ale by Silver City Brewing. And if you ever go to Seattle, you can get it at the Sea-Tac Airport in the Seattle Taproom. It was delicious.
Matty: [00:56:00] Awesome.
Trevor: So I’ve got a mission then for some of you.
Eric: My first checkout is this podcast that I’ve been listening to pretty obsessively lately. It’s a dark techno drum and bass kind of thing. It’s called Gray Area from the UK. You can find it on soundcloud.com/samurai-music-group, all one word. Pretty awesome. And I’ve actually— tech-wise, I’ve been working with Ansible a lot lately, which might sound kind of amusing, but it’s pretty awesome. And I’m particularly interested in talking to people who are using Ansible in conjunction with Puppet and Chef. So just me on Twitter. I’m @ahpook, A-H-P-O-O-K, on Twitter if you want to hit me up if you’re interested in talking more about, about your use case and what you’re doing with that.
Matty: Robin.
Robyn: So first I’ll say, Eric, I know lots of those people, so I will be happy to funnel them towards you. Uh, let’s see, my checkouts. I was sort of curious what this part was, and now I, now I know because I listened to everybody else. Uh, so I guess my first checkout would be, uh, Ansible 2.0 is coming out soonishly. Uh, but if you want to go, you know, look at the devel branch and try things out, that would be awesome. Related to my past life, Fedora 23 beta is coming out, and as I always recommend, try these things out because they are the future of Red Hat Enterprise Linux. So if you would like to complain about something that, you know, might affect you in the future, the time to look at it is now. So go get it. It’s like Fedora or getfedora.org, I think. I don’t know, they’ve changed all this.
Matty: [00:57:00] We’ll put a link in the show notes.
Robyn: Yeah, we’ll put a link in the show notes and I’ll pretend like I’m a conscientious former Fedora project leader here sitting on my, uh, Mac. Uh, and then finally, I guess, uh, I’ll, I’ll pimp out Splatoon for my daughter because she thinks it’s the greatest Wii game ever in the history of mankind.
Trevor: Cool.
Matty: All right, Trevor, take us home.
Trevor: Well, all right, give mine. Take us home to you, to the dungeon.
Matty: Penultimately, penultimate checkout is Trevor. It’s been that kind of podcast.
Trevor: [00:58:00] I, uh, I just picked up a new board game called Welcome to the Dungeon. It’s a super quick game. You can play it at lunch. It’s lots of fun. It’s like a little dungeon crawler. Everybody agrees on a hero for the dungeon, and then everybody kind of bets with monsters. And then eventually everybody kind of folds, and the last person in who didn’t fold has to fight the dungeon. And if they win, they win. If they don’t win, they lose some health. It’s really interesting and it’s really kind of stab your buddy kind of fun. It’s, it’s good stuff. Secondly, I’ve been playing Rocket League a little bit lately and that’s been a good time as well. Matt?
Matty: Okay, well, so my favorite little game to be playing on my phone lately has been Pac-Man 256. It came out a little while ago. It’s a new version of Pac-Man that has different upgrades and weird stuff. It’s super fun and addicting and has very much perfect for the, like, I’m just sitting here in the lobby waiting for someone for 5 minutes. As listeners know, I do love audiobooks. So the 2 audiobooks I listened to recently was— one was The Martian, which by the time you’re listening to this will be a movie in theaters starring Matt Damon. I think it’s coming out on October 2nd. Look, I was going through old episodes of ADO recently and realized that Bill Joy on our culture episode his checkout was the same book but not in audiobook form. And I also just finished Felicia Day’s audiobook, which is called You’re Never Weird on the Internet . And so if you’re a fan of any of her work with Geek and Sundry or The Guild, or just a fan of nerd stuff, it was a super fun book and I really enjoyed it. I’m sure the, the non-audiobook version would be great too, but it’s read by the author and that was great. So I expect— and we’ll have links to all these things in the show notes, of course. We’re gonna be adding a feature to the show where every week we’re going to be talking about a couple of different other podcasts you might want to check out in the future that are DevOps related. But that’s just a preview for me to tell you that’s happening rather than listing them all off as a way it’s listed in my show notes. So Trevor, your turn to talk.
Trevor: [01:00:00] Thanks. Thanks again to our sponsors. Be sure to visit them at arresteddevops.com/10thmagnitude/victorops. Thanks so much to Eric, Josh, and Robin for joining us. This is where if you guys would like to say any kind of things along those lines, this is an opportunity to say so.
Matty: Thank you. Wow, that was like just begging. We’re like, please say thank you back to me. This is when you should say, it was a pleasure to be on the show. It was my favorite podcast to ever be on. You guys are so rad!
Robyn: This is the best non-echoing podcast I’ve been on all day.
Matty: Right?
Trevor: Fantastic! I’m so glad that that was an experience for you.
Matty: Robin, I think you should go into iTunes and leave us a review that says, this podcast does not echo and I give it 5 stars. And if anyone else who listens to this show has an iTunes account, go to the iTunes Store and give us a rating and review. We really appreciate it. It would be super awesome, and we’d love to know what you think of— what?
Trevor: [01:01:00] I was just going to say the next part. Or you could— we’d love to hear what you have to say about the episode. You could leave us a comment at arresteddevops.com/44.
Matty: You can follow us on Twitter @ArrestedDevOps. We are always happy to get your idea input or feedback in general at shows@arresteddevops.com. Let us know any ideas you have for future episodes. I’m Matt at Matt Stratton.
Trevor: I’m Trevor at Trevor G Hess. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.
