Making The DevOps Transition

Matty: [00:00:07] Welcome to Arrested DevOps, episode 12, Making the DevOps Transition. As usual, I am your very congenial co-host, Matt Stratton. You can find me on Twitter @MattStratton.

Trevor: And I’m your very here co-host, Trevor Hess, @TrevorGHess on Twitter, but I’m not usually on Twitter.

Matty: Yeah, it’s a misnomer to say that Trevor’s on Twitter. So I try.

Trevor: I’ve been, I’ve been trying.

Matty: He’s tweeted like 3 times. It’s awesome.

Trevor: I said something irreverent the other day. It was, you know, I was sad that nobody acknowledged it.

Matty: As usual, Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that believes DevOps makes their clients more awesome. You can check out their 360-degree cloud services at 10thmagnitude.com.

Trevor: Tonight on Arrested DevOps, we’ll be talking about what it actually means to implement DevOps at your company. With a panel of folks who have some real-world experience with it.

Matty: This episode is sponsored by PagerDuty. PagerDuty eliminates the noise, chaos, and manual processes across the entire incident lifecycle to decrease resolution time. PagerDuty is trusted by companies like Etsy, Nike, and GitHub. To sign up for a free 30-day trial, visit arresteddevops.com/pagerduty.

Trevor: [00:01:19] Additionally, we’re pleased to share with our listeners a special discount for the upcoming Velocity Conference in Santa Clara, California, June 24th through June 26th. Velocity is about the people and technologies that keep the web fast, scalable, resilient, and highly available. For a 20% discount on registration, visit arresteddevops.com/velocity and use the discount code arresteddevops20.

Matty: So speaking of Velocity, we’re pretty excited that we’re giving away one free 2-day pass to the conference, and here’s how we’re gonna do it. The first person to tweet us @ArrestedDevOps the full answer, including the name of the session and the speaker, gets the pass. So pay attention. Who and what is happening at Velocity Santa Clara on Wednesday at 3:30 PM in the Grand Ballroom GH? That is Wednesday at 3:30 PM in the Grand Ballroom GH. When you tweet, include the hashtag #VelocityConf and of course @ArrestedDevOps in your tweet. First person that gets it gets the pass.

Trevor: [00:02:24] And now let’s introduce our panel. First, we’ve got Jeanne Steinback. Hey, Jeanne, would you like to introduce yourself?

Jeanne: Sure, I’m Jeanne. I’m currently the Director of Software Delivery at Rewards Network, and then I just joined there recently. For the previous 6 years before that, I was the Director of the Corporate Systems Development team at Redbox.

Matty: And we also have with us Chris Andreen.

Chris: Hi, I’m Chris Andreen. I’m the Director of Software Development for the John D. and Catherine T. MacArthur Foundation. I’ve got a lot of past experience in higher ed, as well as startups, financial industry startups, and corporate insurance. So kind of a mix of goodness to bring to a foundation that’s trying to make the world a better place.

Matty: Awesome. Let’s get started and talk about getting started with DevOps. So I guess one of the first things to talk about is, if we’re going to talk about making a— we’re going to go through a transformation and apply DevOps to our organization. That can mean a lot of different things to different people. So I’m curious to go— because we’re pretty opinionated on this subject here at this podcast, but I’d like to know, Chris and Jeanne, when you take this and you look to apply it to a place where you’re trying to make a transformation, what does that actually mean to do the DevOps?

Chris: [00:03:41] Well, for us, I think the foundation that I work for, which does a lot of great work, there’s also a lot of history that’s kind of led us to where we’re at at this point. A lot of technical debt And for us, implementing some form of DevOps is really about trying to pull that thread on that sweater, see how far it goes down, clean out the cobwebs, and really kind of reinvent ourselves in a much cleaner fashion. A lot of the way we’ve done things historically has been manually. So manual deployments, manual compilation, manual everything. So this journey for us is really just about moving us towards a better, We’d like to have a best-of-breeds technology organization, but I think just getting to a point where we could be more effective in our technology delivery would help us be more effective in doing the creative work, you know, the development side that might better support our grantmaking efforts and our grantees as well.

Jeanne: Yeah, and I think from my perspective, we sort of came to DevOps— we had fairly mature agile teams. We had pretty much already implemented all the low-hanging fruit to make us more efficient, and then when we really discovered what DevOps could do for us, especially lowering our cost of deployments by, first of all, by automating, and then having DevOps actually take deployment responsibility from our devs, that was a huge time saver. Our velocity almost doubled when we had a full-time dedicated DevOps person on our delivery team. We didn’t think that there was that big of a chunk of efficiency left to have, and we were very surprised that it turned out, you know, that well, and that it actually did strip so much time. Because as we were deploying 2, 3 times a week, the cost of those deployments was very high before we had a dedicated DevOps person.

Trevor: [00:05:31] It sounds like one of the problems you guys are both trying to solve is that with one of efficiency. Is that right? What other problems were you guys trying to solve by implementing a new philosophy?

Jeanne: Trying to free up dev time to do dev and to move other aspects of support, you know, checking after deployments, you know, running tools to make sure our deployment was good. All of the things around deployments, moving those off the dev’s plate and onto someone else’s plate freed up an enormous amount of dev time for us.

Chris: Yeah, and I think on top of that, consistency. Just having a consistent framework from which to work going forward. I mean, I know that for us, nothing breaks the same way. So, knowing exactly those steps that are being taken and taking them consistently makes it a whole lot easier and a lot faster to diagnose issues and resolve them, I think.

Jeanne: Consistency— I’m sorry, keeping environments consistent, because that was also one of the responsibilities of our DevOps engineer, was because he was deploying to all the environments, that keeping those in line— and that used to be just an enormous headache for us, but he eliminated almost that entire problem. Problem for us.

Matty: [00:06:40] Sounds like the implementation there is, you know, kind of saying it’s putting a practice in place around release engineering, so to speak, as a way of implementing these DevOps practices under someone who’s responsible for consistent software release. Am I getting that right?

Jeanne: Yeah.

Matty: In your case, Jeanne?

Chris: Yeah, I think that’s a fair statement for sure.

Matty: Very cool. And then, so when we’re talking about this, because this is a real important one to me. Is this idea that when you’re looking to make a change of some kind, you know, we always want to start with the problem. We want to start with— actually, a lot of times we want to start with the solution, but then we think we’re going to be— we’re going to say, oh well, we’ll figure out the problem before we think of the solution, which is great. But then I like the idea of that we start with success criteria. We say, how do we know when we’re going to be happy? And then that drives your problem and that drives your solution. So when you’re looking to make these improvements in an organization to kind of address these things, How did you know when you were happy, or how will you know when you are happy, assuming this is continuous improvement, you are always working?

Jeanne: [00:07:42] When your velocity goes up. The only reason to even have a development team is to provide business value to the business, and when you can increase the amount and the frequency and the repeatability of deploying valuable software to the business, that’s a hard statistic that you can track and show the value of what you’ve done. So it’s not even just a question or someone going, hey, you know, I think this is working great. We have actual statistics to back up, you know, the success of our DevOps team.

Matty: And what are some of those metrics that someone could use? Because I think that’s a great point, is you want to be able to show it so it’s something that you can measure, right? You know, it didn’t measure it, you don’t measure it, you can’t manage what you don’t measure, right? So you can’t make it better if you’re not measuring it. And I know when I’ve done this before, it’s kind of always been hard because you— if you weren’t measuring it the old way, it’s hard to compare the two. So like to show that the needle moved if you didn’t have a needle at all. So anyway, but I’m curious, what are some of those metrics that can be used to show that the need— like how can we see that we’re moving the ball forward?

Chris: [00:08:45] From our standpoint, I mean, really some of the metrics we’re trying to look for are kind of shifting the amount of time we spend on operations versus development, you know. So a lot of times when you don’t have a good solid kind of DevOps practice in place, you end up spending a lot of time cleaning up things that you accidentally introduced. So eliminating that and making our developers much more efficient and working on things that are actually increasing the value. Now granted, I know from a business standpoint it’s very similar, right? We actually don’t take money in, so we do the opposite. We just give money out. So, from our standpoint, what we’d like to do is be able to be more efficient at giving money out and more effectively track how productive that money was that we did give out, as well as, you know, make it easier for our grantees to be able to do that same thing. So we can go back and look and see what we’ve learned from what we’ve done. And I think that technology could really play a big role in that. So, we’re kind of in the beginning of our DevOps transition, but it’s very apparent the changes that are coming down that are going to help make us more successful?

Jeanne: [00:09:59] So the metrics that we kept around— well, first of all, I’m a huge advocate of keeping metrics on our— a lot of metrics on our agile teams. The only way to know if your team is healthy is to have something to baseline off of and then base your improvements off of. So we keep cycle time, which is the time something goes into analysis to the time it comes out into production. We keep velocity, which includes all of the interruptions and bug fixes and everything. So how much value did you actually put out within a time box? We track numbers of bugs, we track the number of deployments that we do within a time— we just timebox it 2 weeks just so we have a way to keep metrics. We keep pretty much any metric that helps us determine the health of our team, and then anything that— an area that we find is a bottleneck, we’ll take a baseline on that area and then we’ll try different things to try to improve that area. And so we always are improving off of a baseline metric for whatever fits it. QA issue or if it’s a DevOps issue so that we can tell if we’re successful or not.

Trevor: [00:11:00] So it turns out— sorry to hold off on a response for a second, but we do have a winner.

Matty: Yes, so Twitter user Blanche the Beach responded correctly. So Aaron is his name, and you will be the recipient of a promo code for a free 2-day pass at Velocity. You just have to get yourself there. So Aaron, I’ll send you a note offline with that code, but congratulations. And now back to our show.

Trevor: As you guys started implementing this, these new methodologies, this new philosophy, and I get the impression, Chris, that you’re still going through it, are you being met with any resistance, or did you meet any resistance in this, the process?

Chris: We have not met any resistance because fortunately I kind of am over the software side and the infrastructure side, so the only resistance would come from me, which makes it a lot easier to move forward. In all honesty.

Trevor: That helps.

Chris: I know in other places that I’ve worked, there’s been a lot of territorial kind of things that go on that make it kind of difficult to move forward, especially when you’re kind of merging those, those two together. But yeah, we’ve been really fortunate.

Jeanne: [00:12:08] Yeah, we met significant resistance when I was at Redbox. We didn’t have an actual DevOps team, and we figured out we needed one just sort of by looking at where our current bottlenecks were. It wasn’t a general practice there at all, and we did have to fight, but I did fight with statistics. So when we could prove where our bottlenecks were, when we could prove where our time was being spent, it was a pretty easy fight after I accumulated enough data and then, you know, made a presentation saying that, you know, this is— I think we can have, you know, a 20% improvement in the amount of time that we spend on deployments. It actually turned out to be much higher than that, but it was a pretty easy sell after I accumulated the data. So I think that One of the ways to get this passed, if you have resistance, is to really track your metrics, accumulate the data, and then make a logical presentation. Just not, hey, I hear DevOps is really hot right now, we should have one, you know, we should have a team. It really helps if you can back that up with some data and also maybe experiences from other companies, you know, blogs like this, stuff like that, where people are, you know, saying, you know, this is what the DevOps team has done for my company. But any evidence that you can present will make your job easier.

Trevor: [00:13:20] So it sounds like you had a lot of resistance from the top down?

Jeanne: Yes.

Trevor: Okay, so like you were saying, you presented your statistics on velocity. Did you have a team that you had that you kind of used as a pilot, or did you find statistics from other teams to make your case?

Jeanne: At Redbox, I was in charge of all of the corporate applications, so I used my team as kind of as the guinea pig. I was able to control that aspect of it because that was my team, so I used us as the guinea pig. I took the results, like I said, they were actually better than I had anticipated, and then it turned out that we ended up assigning DevOps to each of the delivery teams after we were able to prove its value, and now they are big proponents of the DevOps. And then when I came to Rewards Network, it wasn’t even a question, it was Where is our DevOps team? Oh, we don’t have one. Well, we need to get one. I actually got almost no resistance there because they were just like, how can you make our delivery faster? And I’m in charge of all of delivery there, so I didn’t have anyone above me that was trying to resist. And I knew from experience that one of the best ways that we could speed delivery would be to have dedicated DevOps teams.

Matty: [00:14:30] And this is where we can kind of get into like differences of how we talk about words, and words aren’t that important. When we talk about kind of like the DevOps philosophies, one of the changes that you made— because I really want to now get into saying like, okay, how did we do this, right? Because that’s what everybody wants to know. They say, hey, you were successful, what did you do? And so we’ve given some beginners— or beginnings of it, which is show metrics, show the value. You know, I always say I’m from Missouri, show me. You know, don’t point me to some Gartner report or something like that. And like Trevor said, when you have a team that pilots it, that’s a great way to get adoption too, because then they say, wait a minute, why is that corporate apps team so fast? Why is their stuff getting out so quick with so few defects? I want that. Oh, well, look at what they’re doing. They’re doing continuous delivery. They’re doing, you know, they’ve got automation there. These are the things they’re doing, and then people want that. So that being said, so we said metrics is a good way to start, right? You want to be able to show, but what are some of the changes that you had to make? Like we talk about when we talk the philosophy of DevOps, or one of them, right, is that developers have to change how they approach their services and things they create the same amount that operations has to change. And so what, if any, kind of mind adjustments did your— you know, we know the value that your developers are getting is that you’re freeing up cycles for them, but what did they have to change? What did you have them do differently in your DevOps world?

Jeanne: [00:15:53] Letting go of the responsibility for DevOps. I mean, getting them to release that to someone else’s hands, because it’s hard to get developers to trust other people. So, getting them to let go of that and trust someone else, and we just happened to get extremely lucky in getting our first resource was a fabulous resource which I now have at Rewards Network, and it was just such a great experience all around. But that was kind of getting guys to let go of some of the things that they were currently responsible for.

Chris: Yeah, I think that’s— I think I would piggyback on that and say that is definitely a challenge that we’re still kind of going through, and that is things have been done this way for a long time and they’ve worked just fine, so why are we changing it now? Type of mindset. But I’m very hopeful that when we finally make it to the other side, that it’ll be really apparent to everyone why we took this path. So I think I would like to go back to the consistency thing. So, I mean, I know a lot of developers have their own styles, but somehow they still are pretty good at kind of doing things a little bit differently here than they did over there. And really, in order to make this really successful for us, we really need a good consistent methodology, whether it be, you know, how we’re checking into source control or how we’re defining how feeds will run versus actual full applications, you know, things like that. And bringing that level of consistency is kind of difficult because it’s kind of been freeform, and you feel like, well, I’ve got all the time in the world, or I don’t have enough time, so maybe I’ll, you know, take some serious shortcuts this time around. But I think that that is kind of a struggle that we’re working ourselves through, but I think right now that there’s an understanding that that’s helped us out in the long run.

Jeanne: [00:17:37] You know, Chris, that’s a really good point. Consistency is one of the things that drove us in this direction, too, for doing the same actions over and over again when you’re going to deploy something, so that— and not only deploying to each environment, not just doing your builds and then deploying to an environment, but doing the same thing when you’re deploying. Deploying environment to environment. And I think one of the big drivers here was that implementing continuous delivery goes hand in hand with your DevOps team. Once you figure out that you want to be a continuous delivery team, once you get to the point where you can be a continuous delivery team, you almost can’t live without DevOps. So, if— when your teams get more mature and more sophisticated and faster and more, you know, all your waste is pulled out of your system, you realize that you it becomes very apparent right away how important DevOps is, because it just streamlines your workflow, it separates responsibilities, and people can work on the things that they were hired to work on instead of having to do all these surrounding tasks that aren’t really providing value to the business. So, and your developers are usually your highest paid resources, so you want to be having them produce the most value, and this is one way to you know, help accomplish that goal.

Trevor: [00:18:47] So, Jeanne, my focus is that I’m a developer. It’s interesting to me to hear that you’re— hear you say that you’re finding that your developers are doing better kind of siloed off from—

Jeanne: Not siloed off at all. So yeah, I don’t want to give that impression. No, the DevOps guy sits right next to our developers, and there’s a constant conversation between them. I mean, he has stopped us from making so many stupid mistakes, I mean, I can’t even tell you. You know, when he overhears— this is part of the reason we sit at Redbox, everyone was sitting together in an open environment, and this is what we’re moving to at Rewards also, is that when people hear you say things that don’t make any sense, you know, if you have a trusted Agile team, they’ll call you on it. And I can’t even tell you how many times, you know, our DevOps guys kept us from taking a silly shortcut just to get something out fast, or doing something that wasn’t the right thing to do. I mean, and that’s what Agile teams do, is they call each other out when they see, you know, it’s a safe environment, but they call each other out when somebody’s going to do something that isn’t the best thing for the team. So definitely not siloed, but it’s a separation of duties in the essence that our deployments no longer needed 4, 5, 6 people in a room to get something deployed. It’s a hit the F5 button. I mean, you know, that’s basically our application deployments got so simple and so fast, but that’s because our DevOps guys helped us build in the CI architecture and the software putting the whole program together with us. They work very closely with the developers to put that whole system together, and the end result was that the separation of duty was that developers no longer were hardly involved at all in the deployments, but not siloed at all.

Matty: [00:20:27] So you got a resource or resources that are focused on developer tools or release engineering or that kind of thing, which if you call it, you know, DevOps team, whatever you call it, Frank, I don’t care what you call it, right? But they’re focused on that, but then you’re— because someone’s paying attention to that That’s putting some responsibility, or at least giving some direction, it sounds like, to your development team to say, okay, you want shit pushed out? This is how it has to happen. Right? You know, we’re not going to just go and— you’re not going to hand me a Word document and tell me to copy a bunch of files. This is how we work. And then someone is focused on that, but it’s not handing off. And that’s kind of one of the tricks of the DevOps philosophy to me is that it’s, you know, we’ve name-checked John Vincent’s blog post before where he says DevOps means giving a shit about your job and never saying that’s not my job. And I mean, developers need to understand what’s going on in operations, operations need to understand development, and there’s also this kind of thought now that we need to stop calling it DevOps because it’s— what’s the term now? It’s BizOps, right? You know, where it’s not just about development and operations anymore. It’s like everybody just needs to give a crap.

Jeanne: [00:21:30] It’s all a team. I mean, they’re part of the delivery team. They just have some functions. Sometimes, some of those get interchanged. Sometimes, devs will do something and sometimes the DevOps guy will do something. They understand how each of the other’s jobs work. There’s no siloing of even understanding at all. It’s just finding the balance of the most efficient way to get software out the fastest. We tried many different things before we ended up where we ended up at the end of 6 years. It was a constant evolution and a constant improvement over time of even the smallest little details deployment, checklists. And that’s one thing the DevOps guys did, they kept us on track because we did have checklists for deployment. I mean, there’s a whole list of things, but I mean, did we accomplish everything on that checklist? And he would make sure that, you know, he talked to the developers to make sure that they had checked every one of those boxes off before we deployed. You know, did we check with the user? Has there been training on this? I mean, he actually took an enormous amount of responsibility for putting that out into production.

Matty: [00:22:31] Very cool. Absolutely. Before we just— when we were green rooming earlier, Chris and I got into a— we started talking about a bunch of nerdy infrastructure things, and Trevor made us stop and said, you guys are doing the whole podcast before anybody’s listening. But I’d like to kind of go back to that a little bit and talk about, you know, Chris, you’re on a journey. I mean, we’re all on a journey, but you’re very specifically, you know where you are on your journey. And you and I have talked offline about this, and I’d like to kind of get an understanding of where’s your nirvana, right? Like, you’re never going to be done But where’s your magic pixie dust, rainbows, and unicorns, and where are you on that continuum?

Chris: That’s a great question, actually. You know, I’m not 100% sure I know where my nirvana is. I can tell you kind of what our starting point is and kind of how I see things down the road a little bit. But to begin with, we’re really trying to leverage cloud the best we can. We’re looking at both Azure and AWS. We started with Azure. AWS is fantastic, and we are going to get to AWS as well, but I think there’s a slightly lower bar of entry to Azure. Our plan is essentially to move all of our applications out to the cloud, use a cloud identity service provider, and we’re going to rebuild our infrastructure weekly from the latest server images, Windows or Linux, so we don’t have to think about patching. They’ll all be locked down. You know, there will be no SSH or RDP access to any server. And they’ll live for a week. Essentially, at the end of that week, we’ll turn off the existing infrastructure, drop any infrastructure older than a week, and turn on the new. I think later on this year we’ll have it so we can point it to either Azure or AWS. Essentially, we’ll just go one cloud service provider to the next. It’s interesting because we’re really trying hard. There’s all of these great features that both cloud providers have, but we’re really trying hard to keep things as simple and as vanilla as possible. So if I can’t just spin up a box and turn on, you know, the couple of features I need and run my app on it, then I’ve over-engineered my app. I want to keep it as simple as humanly possible. And then I think our longer-term discussion will be, so what do we do with our databases? You know, right now we’re going to be slightly hybrid just because we’re keeping our data on-prem, especially if we’re going to be moving it out to— the goal being to go back and forth from one cloud to another. But, you know, I think that will be something we begin to tackle next year is how can we move that out there. My ultimate goal would be that we don’t have any infrastructure on-prem to support any development or any applications to live on because there’s no reason for it anymore, you know. I know that there are a lot of infrastructure guys that love their bare metal servers, and I heard a great term for it. Late last year at a conference, and that was server huggers. And so we’ve taken it very seriously. We do not want to be server huggers at the MacArthur Foundation. You know, we’re more than happy to just, you know, roll through servers like water.

Matty: [00:25:38] You keep your servers at arm’s length. They’re going to put them in therapy later in life because they didn’t get enough love. Yep. That’s a good thing. And one of the things, Chris, we talked earlier when you were telling me about your idea of being able to flop between potentially cloud providers like Azure versus AWS back and forth. And I— there’s an article, and I’ll put it in the show notes, but it’s called How Boeing Merges Its Data Centers with the Amazon and Microsoft Clouds. And it’s the exact same idea, right, where they’re saying this is how we’re going to maintain some type of open capability and not be locked in and be able to leverage them and But, like, Chris, like you said, if you’re going to do that, you have to keep yourself somewhat at arm’s length from the, like, really, really special snowflake stuff that that particular provider might have.

Trevor: Right.

Chris: Keeping things vanilla is key, because if you go any further than that, then all of a sudden you’ve added this new piece of complexity to your builds and your deployments that just I feel like that flies in the face of what DevOps, or whatever you want to call it, really is for.

Matty: [00:26:49] I really like your plan with when you’re refreshing weekly or as you deploy to new images so you don’t have to deal with patching, because that’s a huge thing. I remember back when I was making my peace with DevOps, having started life as an ops guy. I hesitate to say when I was first learning about it because I wasn’t learning, because I thought what I thought. The first thing I would think is, again, like most ops people when they hear DevOps, especially a couple of years ago, All you thought was, oh, you want to have Root in production. I get it. Then the question I kept throwing up was, well, who’s going to do all the stuff that’s not sexy like patching and blah, blah, blah? The real answer is not that patching isn’t important, but you take those things away. I’m not going to take credit for this thought, but this came up on our Mythbusters episode 6 episodes ago, which is arresteddevops.com/6 to go check it out. Sasha Bates pointed out that if someone’s saying what you’re doing that nobody wants to do, Why are you doing it? Why have you not automated that away? And so that’s, that’s a great approach, Chris, where you’re like, okay, patching is a thing, but how can we make it not be a thing? And the answer isn’t don’t use Microsoft stuff, because you always have some type of infrastructure change you want to do. And so, and for those of our listeners who are kind of learning about this, this is this idea, it’s a buzzword that not everybody uses the same way, but immutable infrastructure, right? Where you’re saying, I don’t really have to care about managing the configuration of my infrastructure. Because I’m just gonna shoot it in the head when I’m done with it. And if it needs to change, I’m not even gonna bother changing it. And that’s, you know, what we would call like the Netflix model, right? Where they deployed new machines, and now everyone’s talking about containers, but it’s the same thing, right? Where you’re gonna just say, I’m not gonna worry about making this look the way that I want it to look because I’m not gonna let it live long enough to ever get screwed up. And, you know, I love that you’re saying, you know, you can’t SSH, you can’t RDP. Steve Murawski from Stack Exchange likes to say RDP is not an administrator tool. And I think more people should embrace that. And we could make the same comment about SSH. You know, it’s not just talk smack about Windows admins, please. So I think those are pretty good points. So we kind of talked about— so Jeanne, you gave us some background on what you did, and like you said, you got it. I want to play off of— you said you tried a lot of things. This was a journey, and we talked about what worked and what was successful. So what maybe worked less well? What were some of the mud pies you guys made that our listeners could avoid?

Jeanne: [00:29:15] We did a lot of things over and over again that could have been automated that we weren’t automating because everybody is so rushed to get everything done. You need to get software out right now. Instead of stopping and automating some of those repetitive tasks that you do over and over again, it’s a mistake not to stop and do those. What we finally did is we put a big piece of paper up on the wall. Every time we did a task, for the first couple weeks, we wrote most of them that we thought might be repeated on the wall. When something got to the point where we did it 5 times in 2 weeks, we went, okay, that’s a candidate for automation. We’ve now had to stop what we’re doing and do this same thing over and over 5 times in 2 weeks. We did this not only in software development, but for our interruptions. When people come up and, hey, can you do this for me, or can you tell me this, or can you get this information for me? We did it not only for infrastructure and for our deployment process and for our dev process, but we did it for actually, you know, other processes too in the company where we just noticed that people are asking us the same thing over and over. And then we would stop and we would automate that thing, whether it’s just a tool or a job or a script or whatever. We just stopped and took time to automate it. So one of the things is, if you see things you’re doing over and over again anywhere in your process, stop and take the time to fix that thing because it’ll have rewards for the rest of the time that you’re there instead of just manually doing something over and over and over again because you don’t have time to stop and fix it. And that’s a— that was a lesson learned the hard way, definitely.

Trevor: [00:30:45] Chris, how about you? Do you have any missteps you’ve made so far that you’ve corrected?

Chris: Well, sure, yeah. I mean, in all honesty, I think, you know, I heard someone describe it you know, getting into this kind of mode as iteration zero, right? And I think the hardest thing about it is that it’s constant mistakes, right? I mean, no one goes into this knowing exactly how to make it work for their organization right out of the gate. So we have gone through 5 different source code providers, right? We didn’t— we decided we wanted our source code to be in the cloud, right? So we have tried every different source code— well, not every different source code provider, but we’ve gone through TFS and Git and Subversion and then multiple carriers of all of those different repos and figured out some of them are really consistent with how things come down and some aren’t. We’ve restructured our repos 100 times to try to come up with the best route of being able to pull our code down and compile it without ending up with a bunch of garbage that gets deployed as well. I don’t think it’s possible— I don’t know, maybe it is. I haven’t met that consultant yet that gets that Hail Mary on every throw and it works.

Matty: [00:32:03] If you know one— That’s me, right? No.

Trevor: Matt, you’ve been stuck long enough for us— long enough for you to figure out how to get it done.

Matty: That’s right. You know, if you can’t dazzle them with brilliance, baffle them with bullshit. So, not that I would ever enact consultant at all.

Trevor: So, what kind of tools did you guys use? You know, you guys have talked about repositories and continuous delivery, continuous integration. We’ve had a few people reach out to us and ask what kind of tools people are using as they start working in this kind of space. So what tools have you guys been using to better your process?

Chris: We’re using Subversion as our repository. We are using— actually, we’re using Jenkins, you know, and I’ll tell you, I haven’t found anything Jenkins can’t do. I mean, it’s the ultimate application. I can make it do everything.

Trevor: You know, sometimes I feel like I want to bash my head against Jenkins, and I really— I want to use anything but Jenkins, but you’re right. It really can do just about anything if you find the right plugin for it.

Chris: [00:33:13] But, you know, I’ll tell you, as far as— so we don’t— We are not using a ton of plugins aside from Subversion and the Chuck Norris plugin, because you’ve got to have Chuck Norris letting you know when your builds are successful. But aside from that, we’ve got it— it’s really stripped down. I mean, a lot of our code is all Microsoft-based, right? So we essentially have a PowerShell library that we’ve created, and we just use Jenkins to orchestrate those PowerShell scripts to spin up infrastructure, deploy code. Rollback, you know, if regression isn’t working properly or whatnot. So we definitely— we use it for a lot, but certainly the plugins, we keep it kinda light on that side.

Jeanne: Since we’ve started building out our CD at Rewards, we were using Subversion also, but we’re gonna be switching to Git because we want distributed source control. As the teams get bigger, you find that you have a need for distributed source control. And then we wanted to kinda stick— we’re using Git and Go and Mingle kind of all integrated together, so we kind of wanted to stick with that toolset because it fits within our agile methodology really well. There is, I think, you know, you can pretty much plug in any code repository you want to into most of the systems. It’s whatever you want to use, and everybody’s got their loves and their hates for— I let the developers use whatever they want to use. They’re the ones that have to deal with it on a daily basis. In fact, I pretty much want them to be happy, so I let them choose, you know, whatever tools they want to use. Sometimes we try tools and they don’t work and we get rid of them and we try a new tool if that one doesn’t work. It is a constant. Something great and new comes out, we want to try it, we give it a try. If it works, we keep it. If it doesn’t, we move on. I think it’s just constantly searching for the next better thing. We don’t want to be on the bleeding edge, but we want to have current technology, but we want it to be stable and high quality at the same time.

Matty: [00:35:04] Pull something you just said that really resonates to me and is something I’ve been thinking about lately. And you said you want your developers to be happy. And I’ve been looking at, you know, because a lot of times people ask me, they say, especially because I just started a new job, they’re like, so what’s your job? And I’m like, I work for Chef. And they’re like, well, what does that mean? And I’m like, it has nothing to do with the movie or cooking or any of that stuff. But one of the things that I look at is I think about, it’s, maybe it’s the sysadmin in me, but so much of my approach and thought has always been like, how can I make things suck less? I’ve consciously made myself do a 180 on this and say, my job here is not to make things suck less. My job really is to make work fun. I’m not going to take credit for this idea. This is totally— Jez Humble said this. Somewhere I heard it. He made this comment in a podcast or in a talk. He said, what we’re trying to do with DevOps and with continuous delivery and all these things is to make it fun to be at work. It doesn’t mean like, oh, hey, Super Soakers and Nerf fights and stuff. But let’s enable folks to do the things that are fun. What’s fun? Creating new things, right? Moving the ball forward, moving the needle, whatever metaphor you want. Doing fun, innovative things and making progressive change to the business and making the business more awesome. That’s fun. Patching servers is not fun. Troubleshooting a deployment script is not fun.

Trevor: [00:36:24] What is it that comes up almost every time you start talking, Matt? About ops people and bringing in DevOps methodologies, they always say to you something along the lines of, well, what am I going to do?

Matty: Yeah, yeah, that’s it. It’s a good point.

Trevor: You’re going to do the fun things.

Matty: Absolutely. You’re going to do this. I had this come up where when I was doing DevOps or moving towards this automation at Apartments.com, I had people on my sysadmin team who said, well, once you’ve automated all this, what the hell are we going to do? And I said, you’re gonna do what we’re paying you to do and use your big brains and innovate, right? You’re not gonna copy files, you’re not gonna do this.

Trevor: You’re not gonna babysit the system.

Matty: Yeah, and, and I was validated when we talked to Etsy last time and John Allspaw said the exact same thing. He said people say to him, you know, what’s gonna happen if we, if we do all this great automation? What does your web ops team do? And he said, are you kidding? There’s tons of things for them to do that are innovative and leverage knowledge. So that’s what we’re trying to do, right, is we’re trying to say let’s make it It sounds cheesy to say let’s make it fun, but I’d rather have that approach. And with that, we’re, we’re coming up to our end, so it’s time to move into our checkouts. And this is the part of the show where we’re just gonna say something that we think is cool and that the listeners might want to investigate and enjoy. This could be anything. It could be a website, it could be a book, a blog, a whiskey, a beer, a TV show, etc. And we’ll start with our guests. So Chris, what do you got for us?

Chris: [00:37:53] So I think that people know about this, and maybe I’m just kind of late to the party. But I recently downloaded an app called Bands in Town. And I’ve got to tell you, so I love music, and I love going to see shows all the time. And I downloaded this app, and it connects to Pandora. And basically, it figures out all the bands you like, and it’ll tell you whenever any of those bands you like are coming to town. As well as make suggestions to bands it thinks you might like. It’s really pretty good, actually. I haven’t seen any bands that I don’t like pop up in there so far, which is pretty impressive considering I’ve been using Pandora for, you know, like 9 years, and it still gives me some crap suggestions every once in a while.

Matty: Awesome. Jeanne?

Jeanne: I read all the time, so I have 2 books, actually. One is a business book called Zap! And it’s really about how to empower your employees. And just on a point you were making before, people are happy when they feel productive. And the only way they’re going to really be productive is if you empower them to make their own decisions and to control, have as much control as possible over their own environment and allow them to make their job fun by empowering them to make decisions about their environment, about their tools, about, you know, the people they work with. Anyway, Zappos is just a book that kind of talks about how to empower your employees and improve employee satisfaction through empowerment. And then for the truly creative side, there’s a book that I just read called Night Circus, and this is completely different. It’s completely fantastical and magical on a completely different level. If you’re creative, if you have a crazy imagination, you’ll love this book. It’s kind of otherworldly but very enjoyable, and if you have a good imagination, it creates worlds in your mind that you can see. So I would recommend reading that if you just want a fun kind of creative experience.

Matty: [00:39:48] Awesome.

Trevor: Trevor? First of all, I just want to say, Matt, your introduction at the checkouts this time started off as the worst Christopher Walken impression I’ve ever heard.

Matty: Really? Now I’m gonna have to listen to it.

Trevor: So for me this week, I actually just read about this before we started. Apparently JetBrains has released a new database IDE. So it’s called OXDBE because they have to have witty names. ’Cause they’re JetBrains. The icon’s pretty. So there’s that. And also, I was playing around in some CSS, and I was trying to figure out how to look at the print media state for one of the CSS tags, and I discovered that buried in Chrome Tools is a little checkbox that lets you select the CSS media state that is currently being rendered on the page. So that’s super useful for debugging. I will post the article on the show notes as to how to get to that buried setting.

Matty: So I’ve got 2. My first one is a book. It’s a book published by Pragmatic Programming. It’s called tmux: Productive Mouse-Free Development, and it’s basically a how-to book on using tmux, which is a terminal multiplexer. Yes, SSH is not an admin tool, but those of us who develop on Macs, we use a lot of stuff at the terminal, and I’ve heard that tmux is the way to go and I want to be a tmux ninja. I’m going to shout out to Chris Webber, who told me about that book. Also, I want to reference a new website called burnout.io. At DevOps Days Pittsburgh recently, Jennifer Davis gave a talk called From Hero to Zero. It’s been making the rounds on Twitterverse. It was a really amazing talk. I can’t recommend it enough, and I’ll put a link to it in the show notes. It’s about basically the impact that not having a real life has on a knowledge worker. There’s a lot of science to back it up and a lot of stats. She talks about how we want to be the hero and we want to be the awesome person. She kind of says, we all see this, but this is why it doesn’t work. This is why you have to take a day off sometimes, and this is why you need to sleep. This inspired a website called burnout.io, which is a collaboration site to talk about burnout in IT and what we can do about it. Actually, our buddies at The Ship Show have an episode up right now that’s all about burnout. We’ll put a link to that in the show notes also. I completely forgot about that with my checkout. As we wrap up, reminder, we have a newsletter. You can subscribe at arresteddevops.com/bananastand.

Trevor: [00:42:24] There is no money in our banana stand.

Matty: There is no money in our banana stand. This is the best way to know about our upcoming podcast episodes and cool news with DevOps. We won’t send anything more than once a week. We won’t sell your email address to evil people, I promise, because we hate that stuff. Right now, if you take our survey at arresteddevops.com/survey, you will be entered in a chance to win an Amazon gift card. Only people listening to the podcast can enter, so don’t, you know, tell anybody about it. If you are listening to this later, the contest will end on June 15th, which is Sunday, I think, of this week.

Trevor: Yes. Thank you guys for joining us. We really appreciate you being here tonight, Jeanne and Chris.

Matty: And also, if you did miss out on winning the Velocity Pass, remember we are still able to offer you a 20% discount to the conference. Go to arresteddevops.com/velocity, use the discount code ArrestedDevOps20. That’s the number 20.

Chris: I won the contest, right? If I remember correctly, that was— you announced I won?

Matty: [00:43:27] Sure, you win 20% off. I’ll give you that for free.

Trevor: You also win 20% off.

Matty: Yeah, hey, everybody wins 20% off. But yeah, our actual winner, just as a reminder, so that I remember, it was Blanche the Bleach on Twitter. So congratulations.

Trevor: Yay.

Matty: Thanks for joining us. Thanks for listening. Thanks for watching. Thanks for being here. Check us out at arresteddevops.com. Follow us @ArrestedDevOps on Twitter. Look for us on Facebook. I don’t know. We’re all over the internet, you know, but Twitter is where we’re at. Matt likes social media a lot. We do. Speaking of, you can find me on Twitter @MattStratton.

Trevor: You can pretend I’m on Twitter @TrevorGHess. I will respond if you tweet at me, but I probably won’t tweet otherwise.

Matty: And we are Arrested DevOps, and remember, there’s always DevOps in the banana stand.