dev to ops

Matty: [00:00:07] Welcome to Arrested DevOps, episode 19, Dev to Ops. I am your co-host, Matt Stratton. You can find me on Twitter @MattStratton.

Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.

Matty: Arrested DevOps is brought to you by TenthMagnitude, a cloud services company that figures if you’re listening to this podcast, then you must be pretty pretty cool. You can find out about joining their cool cloud services team at 10thmagnitude.com.

Trevor: We want to let everybody know that the Chef Community Summit is coming up October 2nd and 3rd in Seattle, or if you’re in Europe or other continents—

Matty: or you like the UK in general—

Trevor: well, yes, October 15th and 16th in London. Use the discount code ArrestedDevOps for 10% off your registration at arresteddevops.com/chefcommunity.

Matty: Also, speaking of conferences, registration is available for the first ever DevOps Days Chicago. Arrested DevOps listeners can get 10% off their registration with the code ADO10. Visit arresteddevops.com/devopsdayschicago for all the details. So we all know that being on call sucks, but what if there was a tool out there that allowed you to route incidents to the right team and mention specific people to ask for help and hop into chat with your team from an easy-to-decipher incident timeline that gave you full context of what was happening? That tool is VictorOps, and they are different. From setting up global on-call rotations to creating a post-mortem report, VictorOps is there with you through every step of the incident lifecycle. Our real-time collaboration platform helps your team solve problems faster. Sign up for a free 14-day trial to see how they’re making on-calls suck less. Visit arresteddevops.com/victorops to sign up.

Trevor: [00:01:57] This podcast is also brought to you by Datadog, a monitoring service for dev and ops teams who work collaboratively to write and run applications on highly scalable infrastructures. Datadog brings together system metrics, events, and alerts from over 80 common infrastructure tools to provide teams with graphing, correlation, and data analytics. Datadog is available for a 14-day trial for free at arresteddevops.com/datadog19.

Matty: So in last week’s episode, Mike Fiedler pointed out that basically nobody ever starts out wanting to be a sysadmin, and we all find our way here some other— in some other direction. So tonight we’re talking to some people who started their life doing dev stuff and somehow got suckered into learning about ops. So we’re calling this show Dev to Ops. All apologies to Damon Edwards, this has nothing to do with DTO. But it’s a great name.

Trevor: Well, so just so everybody knows, Matt busted my chops because apparently my ad-libbing sucks and I don’t vary up our introductions very much. So I wrote some very special introductions just for Matt for tonight’s episode.

Matty: [00:03:04] So tell them that.

Trevor: We’d like to welcome Aaron Blythe. Aaron, will you tell us a little bit about yourself?

Aaron: Sure, yeah. I’ve been working in software for about 12 years now. I work for Cerner Corporation. We’re always hiring. We’re trying to transform healthcare every day in every way, so it’s an excellent field to be in. Initially, like, the biggest frustration for me, for years I was writing software, and like we would get client issues and it would be nothing like our development environment. So the big thing, like, the move towards ops is software has changed quite a bit, and now we can actually stage like production a lot more. We have big data. Things that we’re working on now. So, exciting stuff. Glad to be here.

Trevor: Awesome. Thanks for joining us. We’d also like to thank for joining us John Smyth. And would you be so kind, John, as to inform us who you are and what you do?

Aaron: Sure.

John: Being kind is not really my thing, but I’m a senior consultant at 10th Magnitude. I also started life doing development. I’ve kind of worked on to a A lot of operations stuff. So yeah, that’s about it.

Trevor: [00:04:10] Awesome. I probably said the same thing again because I didn’t add lib— what I was— I didn’t write what I was going to say after people introduced themselves. So sorry, Matt.

Matty: You are so fired.

John: Your webcam is so bad, it reminds me of Max Headroom.

Trevor: Nice. I guess I need a new webcam.

Matty: This is just all a trick for Trevor to try to get new crap. I just bought him a new microphone yesterday. Now he wants a new webcam.

John: Well, Matt, your voice doesn’t match your video, so I expect a group of ninjas to break into your hotel room or something and start a karate war.

Matty: Yay, hotel Wi-Fi.

Trevor: Well, finally, now that we’ve neglected him, who are you, Nate Burleson?

Nate: Hello, thanks for having me on. I’m also a senior consultant at 10th Magnitude. Started software development about 18 years ago, back in the internet boom when Microsoft technologies were new, you know, I’m talking classic ASP, that kind of stuff. A lot of times developers were forced to dive into the server itself. It wasn’t by choice, it was by necessity. There weren’t always sysadmins around, so a lot of times we had to go in and administer IIS, etc. So that was my first real introduction to sort of the ops side of things, and as of late I’m doing some automation in Azure, and I look forward to talking about it.

Trevor: [00:05:25] Excellent. There, I said a different word than awesome, even though—

Matty: Very good, Trevor. We’re very proud of you. So I’m thinking the first thing, I know we just kind of did introductions, but kind of thinking about more of where— like, I’d like to hear a little more details about, like, what was your first exposure to operational stuff? Like when you really had to get your hands dirty like a sysadmin.

Aaron: For me, it was— I moved into a Rails e-commerce application that we have here, and we started deploying the code by ourselves as the development team to just see what it was like and have, like, kind of a new challenge. So I was running a bunch of Bash commands. I learned Bash kind of on the fly as I was building our production environment. I always felt like there had to be a better way, so we started looking at that time— this was 5 years ago— we started looking at Capistrano. Then from there, we found out about Chef. Since then, it’s been on. We built a team where we do the DevOps type of thing. We both write the code. And we actually deploy the code.

John: [00:06:26] Yeah, I don’t know, I think, you know, Nate, you mentioned it before, but I think we kind of came up in about the same time period, right? Classic ASP and early internet stuff. And it was just kind of a natural progression. It wasn’t so clear-cut, I don’t think, between operations and development. You had to know the platform that you were developing on. And I think, you know, the first time I started doing, I guess, what would be pure operations type work was because there were some problems on SQL Server and our application wasn’t performing the way it was supposed to. When I started digging into performance on the SQL Server and doing some administration on it, and I don’t know, it seems like there’s a— this kind of artificial division now where it’s either operations or it’s development, although that’s kind of going away again. And I think, you know, years ago, I don’t think it was so cut and dry. I mean, You had to do what you had to do to get your work done.

Matty: Yeah, I think I remember, John, you and I had a conversation one time, or I’ve seen you say this before, which is you’re like, hey, I’ve been doing this DevOps thing for a long time, but we used to just call it working.

John: [00:07:31] Yeah, no, that’s really true. I mean, if you think about it, really, before there was pure operations, you had programmers and you had systems programmers, right? The people that took care of the systems were also programmers. And, you know, I don’t know when it became operations people don’t know how to write any code, but That’s not the background that I came from. You know, it seemed like everybody knew how to at least write some sort of scripts if nothing else, and I don’t know, it wasn’t so divided, I don’t think.

Nate: I will say that when I first started my career, I first started in Perl, and so at that time there were plenty of Unix sysadmins. You know, this is when Linux was fairly new, but BSD and other flavors, I guess they were at the time, I don’t even recall. The Unix sysadmins actually taught me a lot of Perl tricks. So kind of back in the day, and then I think that’s still true today, the quote-unquote sysadmins back at that time were pretty good programmers. I mean, they really knew how to work the command line and some of the programming languages, etc., to automate their jobs. So as John stated, I think this division is sort of modern and somewhat artificial. I do know that sysadmins didn’t want programmers to touch the systems very much. For fear that we didn’t know what we were doing, which was true. But I do feel that largely this, this division that’s come up in the last 15, 10 or 15 years is really artificial.

Matty: [00:08:50] Definitely, I can see that because I think that— and this was one of the things we talked about a little bit last week when we’re talking about sysadmins— we talked about kind of the idea between computer operator, you know, system sysops and sysadmins. And it seems like a lot of ways, like, kind of that move to being more of the administrator versus the operator was a place where some of that distinctions started to change a little bit. So I’m interested to know— so again, you know, so John, just background, I mean, John and I have worked together on and off for years, and like, it’s funny because I’ve always known John as a systems guy. Like, I didn’t know John started his life as a developer until he told me that like a few months ago, and I was like, oh really? Well, that explains a lot, in a good way. That wasn’t snarky, that was like, that explains why you’re so good at certain things.

John: No, it’s cool.

Matty: So I’m just kind of curious, like, what do you— so being kind of classically trained, if you would, for lack of a better term, with a development background, like, what do you think are the kind of skills that— where’s that applicable to doing more of the ops stuff? Because like, John, when I worked with you before, I mean, we were system engineers in a financial institution. It was all back office crap. There was no programming except for all the cool Perl scripts you wrote.

John: [00:10:00] Yeah, I was going to say, but there was because there’s always a place to automate things and always has been. I started off as a developer because I went to school and learned how to write code. When I came out of school, the first job I had actually— I actually was a consultant and the first job I had was actually doing testing, writing tests. I did some work writing programs and then I did some work in infrastructure. I guess I never saw them as being too distinctly different.

Matty: From a development background, right, I was saying, where is that applicable? And so your thing was like, well, hey, I’m always writing scripts. But just thinking about besides automation that you do, being a sysadmin and being an ops person, it’s not just about automating too. There’s task-oriented things. There’s a different kind of way of approaching it. So I’m curious, what skills or background, how does that apply?

John: I mean, everything fits together, right? It’s just layer upon layer of abstraction. And the deeper you know into those abstractions, the better you’re able to do your job, the better you can troubleshoot things. So you’re right, it’s not just automating things, although, you know, that’s a big part of it, being able to do, you know, 10 things instead of 1 with the same amount of time. But it also gives you insight into how things work and how the pieces fit together. I think the more you can learn about computers, the better you can do your job when you work with computers. The more you can learn about the business, the better you’re going to be working with that business. People tend to, I think, kind of pigeonhole themselves sometimes, you know, that people sometimes complain that they’re stuck in whatever they’re doing, but you have to find a place to apply those things. And as far as having a development background, my knowledge of computers starts with that background that I had, you know, I took assembler in college and learned about the hardware by, by programming it, right. And everything else is— everything’s just the next layer of abstraction on top of the top of the previous one.

Aaron: [00:11:51] Yeah, I totally agree with that. It seems like on the development side, like being more classically a developer, even still, the thing that we’ve been doing is going and comparing our processes from our development teams. For years we kind of did waterfall, and then we moved into Scrum and Agile and kind of do the Scrummerfall thing now. The operations teams that we work with are kind of like ITIL-based. At the base layer, they’re kind of the same thing. You want to get people reviewing it to get quality out of it, but the big thing that we’ve seen between, like, the development team teaching the ops team is sometimes, like, code reuse. Like, we’re doing the same task over and over. Let’s find a common way to do this. And the big thing we’ve gotten from the ops team is just, like, this vast knowledge. Like, when I was doing development in more of a cookie-cutter type of situation a few years back, The development environment was set up for me, and I was just kind of— I never really knew what— how to set it up. I couldn’t set the whole thing up from scratch. I was just supposed to write code that worked in that system, test that code, and then we would ship packages to one of the many domains it went to.

Trevor: [00:13:00] I think it starts— it starts with a necessity. So, you know, when I was working for Dave, who was on the podcast, I think, 2 episodes ago, You know, we’d be working with this sensor fusion system and one of the systems wouldn’t be responding and we’d have to reimage the whole machine and we had to figure out, okay, what do we need to do to do that? I mean, I remember back when I really had no clue what I was doing in IIS and I manually changed the permissions on the wwwroot folder and completely blew up IIS and then I had to figure out how to fix it. Which was a fun challenge. But I think that kind of the transition from Dev to Ops kind of starts with necessity. You know, you see something that you need to automate because you’re tired of doing it. You see something that doesn’t work the way you think it should, so you investigate it. And eventually you figure out it’s not your software, it’s something else. And if there’s not a person who can immediately facilitate that change and you have the power to do that change, you become the vector for that change. Start the transition.

Matty: [00:14:03] True. One of the things that I think is interesting when we think about on an operational side, and I know that pretty much we’ve all kind of, I think, done some config management stuff here. I know one of the connecting points of most of us is Chef, actually. And so think about infrastructure as code. You know, Aaron, you brought this up and you said, hey, one of the things that was great was being able to kind of bring some of this rigor around application development to the ops folks who weren’t as familiar with that. Adam Jacob has a great quote where he said, you want to really see cultural change, see what happens to sysadmins when you tell them they have to do a code review before they make an infrastructure change. I’m kind of interested to say, what do you think is one of those besides that? What are some of those real big cultural differences when you look at someone who didn’t come up with a dev background that’s getting into kind of this DevOps piece when you’ve interacted with— either because you’ve interacted with it or you just would speculate? Some of the more interesting challenges around that?

Aaron: One of the most interesting things that we’ve been doing culturally lately is explaining that on our development teams, about 50% of our time is spent reviewing someone else’s code or testing someone else’s code. Like, you develop half the time and then you review or test someone else’s so we can push quality out the door. And some of the operations people that we’ve worked with, they’re like, are you serious? Like, you spend that much of your time? Like, that just doesn’t even seem feasible. How do you have so much time? And so like in Lean, what we talk about a lot is like you got to slow down to be able to speed up because you’re going to pay that time somewhere. So that’s been an interesting like eye-opening thing. So culturally, I think that’s probably been the biggest central point that we’ve had the most discussions in the last couple months.

Matty: [00:15:43] Yeah, I remember Nate, I don’t know if you remember actually, I remember distinctly like looking for guidance from you a few months ago around code review especially, where I was like, so I know code review is a thing, but I don’t know how to do it. I don’t know if you remember that conversation. I think, John, you might have been there too. Actually, I’m curious, what would be your— around that, because I think that’s the thing that’s like, again, Aaron, you alluded to it as well, that that’s super foreign to ops-type people. What advice would you have, or what’s a way to kind of ease that curve?

Trevor: I think first and foremost, the first thing you do is don’t immediately say no. You know, I’ve known plenty of people who I’ve worked with who you come up to and say, hey, we need to make change X, and change X needs to be facilitated by change Y. And the person just will turn around and say, nope, we can’t do it. No.

Matty: Is that in your code review? Because that was like the real split, because I still don’t completely understand how you do that. So maybe I’m asking you guys to teach me how to do a code review.

Trevor: How to do a code review?

John: [00:16:44] I don’t know.

Matty: I think I know what it means, but no one’s ever shown me. I’m going to admit that I’m dumb.

Trevor: I mean, code review is totally subjective if you don’t have something to review it against.

John: Yeah, yeah. And there’s no— I mean, different people do code reviews different ways, right? Right. And in the past, I’ve been— mostly it’s been kind of a— we didn’t review every piece of code, but we would pick somebody’s stuff to review for style to make sure that that was right, and to make sure that everybody understood what was going on, and would, you know, on a weekly basis or so, walk through some people’s code. I’ve also seen people just review their code kind of in an offline fashion, right? Like, be responsible for reviewing another person’s commits before they get committed. So, I mean, there’s no right or wrong way. Well, there’s probably a wrong way. There’s always a wrong way.

Trevor: Part of it, too, is taking the things that we kind of have agreed upon as a culture. Like, on some level, depending on where you work, some of that culture might be some of the clean code methodologies. So, you know, you’re— when you look at your code, you should be able to, you know, the method name should be English so that you understand what the method is actually doing. You know, it should not be polymorphic. It should be, you know, it should serve a single function. You know, you want to kind of— there’s these things that we’ve agreed upon and been introduced to that you kind of work into that. On top of that, like I was trying to— like I was saying before, you may have a code review document, or not a code review document, but a code specification document where you say, You know, hopefully nobody does this anymore, but, you know, all methods must be prefaced by function, or something that you’ve agreed upon as a team that you have to do as part of the code because of whatever reason you have to back it up. Now I think all those decisions should be discussable. You know, there should never be anything that’s like, oh, all our functions are gonna start with the word function because otherwise if the CEO looks at the code, he won’t know what’s going on. Adding the word function is not going to make it any clearer to the CEO. We should talk about why that’s really relevant. And at that point, it may be we just don’t go back and remove all the other functions that say function in them. But going forward, we name our functions something more understandable, like combine users or something, whatever your function might be doing. Add.

Matty: [00:19:04] So, OK. So for an ops person, I just went— I just heard womp, womp, womp, womp, function, womp, womp, womp, womp, polymorphic.

Trevor: Like, wah, wah, wah.

Matty: So I’m like, but if I understand it correctly— and again, I’m really only like 5% being an asshole to you right now, Travis. I’m really saying I want to understand. You’re basically saying the point of the code review is to say, OK, so I wrote a chef recipe, and John is my code reviewer. So he’s going to look at it and say— first of all, I would guess, and this is just like, he’s going to make sure that it’s passed If we say that I have to test my crap, right, he’s going to make sure it’s passed the tests. So he’s a human gate for that, right? Right. And then also he’s going to look at it and go, like, OK, this is just ugly, right? Or this isn’t clear. Because I would think that would be a big piece of it too, right? If he doesn’t understand it, that’s a bad sign, right? That’s exactly right. John, you’re super staticky, by the way.

Aaron: The biggest change I’ve seen over the years is, like, I remember when I first started writing code. Is we would print off, like, reams of paper and get in a room and, like, review code that way. Then we’d throw stuff out on network drives and do some diff tools and stuff. The tools have gotten amazing in the last few years. You can do inline stuff on GitHub. We use an Atlassian project called Crucible, so I can say this one line where you left a line after your rm -rf/, then you had the rest of the stuff, like, yeah, we don’t want to run that on production. That’s gonna cause a bit of a problem, so I can hone in on Here’s the pieces that we want to make sure we get cleaned up before we move on.

Trevor: [00:20:38] Largely, it’s an accountability thing. You may see somebody who came up with, you know, the solution that they provide you uses a pattern that doesn’t match what the rest of the project is using. So you might see someone switching between different design patterns. Like there’s a project I’m working on now where there’s 100 design patterns for no reason, because the person who wrote the code had no one to be accountable to. Except themselves, and it’s just— it’s a hairball, and it is collapsing under its own weight. There are people in this room who know who I’m talking about, and so I’m not going to go any further.

John: Yeah, I mean, as far as code reviews go, there’s no better way to make sure the code is readable than to make sure somebody reads it other than you, and I think that’s a big part of what it is, really. I mean, you can have a linter pick through a lot of this stuff, but you have to agree on what the lint rules are first. You iron those things out at code review. When you’re talking about just the way your code’s laid out, Chef, as an example, Matt, one of the things that we review in code reviews with Chef is that you’re using attributes appropriately and that we’re using them the same way, that we’re following the same patterns so that everybody can understand what’s going on, that things are functionally decomposed and all that type of stuff. The bottom line for the code review is really to make sure that this is readable, reusable code that, that’s going to work in the infrastructure and that follows at least the highest level standards where everybody’s kind of working off the same set of rules.

Trevor: [00:22:07] Right. And on top of that, like you saying reusable, John, made me think of another point too, is that there may be some things that aren’t obvious to you because you’re in the heat of the moment. So you may look at my code and say, hey, you’re adding these things together in a specific way multiple times. Maybe you should extract that as a method itself, you know. So I don’t know, you’re spinning up IIS in like 4 different ways. Why don’t you just have a specific recipe that spins up IIS in a configurable way instead of doing it specifically every time?

John: I have to say too, I probably learned more reading other people’s code than just about anything as well. So, oh yeah, you know, the other thing you get by reviewing code is a different perspective.

Aaron: That’s another benefit that we’ve seen for whether it’s new people or someone that’s experienced that comes to the team. Like, that’s just a learning experience. Like, when we onboard new people to the team, it’s like, ask questions. Use this opportunity. We got the code up here. You got 2 days. Grab this line and ask what it does, and we start conversations, and even though we’ve been doing it that way for 6 months, we see a new perspective. We’re like, we should maybe go a different direction. You got— you have a new way to look at this.

Trevor: [00:23:13] I mean, that was one reason why I really enjoy pair programming is you kind of get that feedback Always. You get, you know, like I was saying, hey, you can pull this method out and use it, make it reusable. Why are you naming this function that adds things together subtract? It doesn’t make sense.

Aaron: Have you successfully pair programmed? Like, I’ve tried with varying results different times. Sometimes it works out great.

Trevor: It truly depends on the person. I mean, I found that with someone who is, for lack of a better term, on a parallel with you, you get a lot better results. I’ve worked with people who just were kind of unmotivated and just did not fit. You know, they didn’t fit culturally to begin with, so they didn’t work pair programming because they weren’t part of— they weren’t willing to be part of the culture that existed, and it didn’t— it just didn’t work because, you know, if 2 people are doing 2 different things, it causes conflict. But on the whole, there’s just been— I found that pair programming was always much faster, and you caught problems a lot quicker when you pair.

Matty: [00:24:15] So on the topic of pairing, have you experienced doing kind of collaborative development across that wall of confusion, if you will, right? Like kind of having kind of an opsy person developing some infrastructure code with a devy person? Have you actually, like, had that exercise, Aaron?

Aaron: Both our deployments and our code, there’s, like, one particular ops person that works here, like, We would stay late together and, like, work through problems and go from the whiteboard back to the keys and then to the system and everything, and it just really meshed. And I think that that’s kind of what you were talking about, Trevor, is, like, if you get that person you mesh with, you can bounce around and, like, kind of pair on solving a problem. I’ve had other instances where it hasn’t worked, right? But, yeah, we’ve— I’ve written code, like, sitting next to most of the developers on my team, but I’ve also done it with ops people. When you hit that magic, it’s pretty sweet.

Trevor: It’s wonderful, really. I was supposed to start pairing with an ops person, but he works for Chef now.

Matty: [00:25:17] Sorry about that. So what was something maybe that was surprising to you? Like, right, so when you kind of— maybe some notions you might have had about what it really— what ops was all about, like that you kind of thought that’s what ops was, and when you started doing stuff with it, did you have any surprises, or was it pretty much what you expected?

Nate: I think it’s pretty much what I expected. I think I was surprised at some of the immaturity, at least in the Windows space, when it comes to some of the tooling and some of the inexperience when it comes to things like source control and code reviews and some of the things we’ve covered already in this discussion, because you’re so used to that. You’re so used to that, and the in the quote-unquote dev space. The fact that that didn’t exist in the quote-unquote ops space, you know, I hate to do air quotes too much, but I felt like there was a certain level of immaturity that surprised me because in my experience, the most technically adept people tended to be the ops people. So I was just kinda surprised by what seemed like a lack of tooling and kind of a lack of experience there. Now granted, it’s new. Ops people have traditionally just done one-off scripts to do something, or they might get a little more advanced, and that’s kind of been evolving over time. But it was really, I guess, some of the immaturity at certain places. But generally it was exactly as I expected, with that exception.

Trevor: [00:26:35] I think for me it was the thing that was surprising was that I never knew that there was— like, I knew that there was a line, but I didn’t really realize how far over the line I’d already gotten towards ops when I finally started understanding what the ops role means. So like, you know, I used to build machines. I used to— well, I mean, I still do build machines, but not professionally really anymore. I configure machines.

Matty: You’re maintaining your amateur status so you can compete in the Sysadmin Olympics?

Trevor: I think that’s going to be my new tag when I remember to do that. But yeah, I just, you know, I was adding users. I was main— you know, setting up firewall rules. I was doing all this crap. That, you know, was just necessary. And I didn’t— there wasn’t any ever anybody where I was working who was truly in that ops role. Probably the closest would be when I was working in the lab at Columbia, where there was an infrastructure team, but they didn’t have time necessarily to, for lack of a better term, babysit the smart lab. So we had to do it ourselves.

Matty: [00:27:36] So I— we had a question that came up on the Twitter. So Ben Cody asked, are there— is there any type of incentivization or measurements that are in place to help bring your dev and your ops together in your org?

Aaron: That’s an interesting thing because there is and there isn’t. It’s always a mix, and a company decides that, like, we have grassroots type of things that we’ve put together, and we want developers and operations to work together, but then there’s also top-down type of things. So one of the challenges we’re facing right now is I work at— we have multiple campuses here in Kansas City. I work at one of the campuses, and then the operations people are at a different campus, and that’s like 15 miles away. So my team goes over to their campus one day a week, and they come over to our campus 2 days a week. And I don’t think anybody in management has said that we have to do this. We do this because we want to be physically in the same location and humanize each other. So there’s incentives to actually make sure that we’re collaborating in ways that’s going to make things work a lot better. And we found quite a bit of benefit from that. But yeah, the incentive thing, that’s a hard one. What would you measure? What would you, what would you say is like success as far as trying to incent people to work together? So we don’t have anything concrete.

Matty: [00:28:55] Yeah, I wonder about that, because again, you kind of want to be able to show result, but metricizing collaboration seems awfully tricky. Well, that’s—

John: it is. And, you know, I’ve seen organizations where dev and ops were pretty close together, or even kind of pushed together, and other places where they were more formally separate. And that doesn’t necessarily dictate the interpersonal relationships between the people, right? And that has as much to do with the cooperation as anything else. You know, even if you, you know, you have 2 teams that are functionally separate, if the people that work on those teams work together, then you’ve accomplished the goal. And if you have an organization where, you know, dev and ops are working together on the same team, it doesn’t prevent somebody from being a dick either.

Matty: Yeah, you can’t just— we’ve talked about this before, right? You can’t just put everybody in an open workspace and expect them to suddenly be buddies. By the same token, and this has actually been kind of a bit of a conversation, there’s the blog post about why you suck at technical interviews or something like that, and I’ll put it in the show notes, that talked about this whole thing about hiring for people that you— we say you need to hire somebody you want to work with, but that doesn’t necessarily mean you should hire somebody that you want to be your friend, because that doesn’t necessarily mean that they’re good to work with, right? Collaboration doesn’t necessarily mean that we’re friends. This was again, I kind of think about this like, actually, John, I consider you my friend now, but when I think about when we worked together back in the day at the bank, we worked together for years. We didn’t socialize outside of work. It was okay. It was like we didn’t have to go out on beer blasts and booze cruises and do all this stuff, and we collaborated just fine. It’s not to say you can’t do those things either, but having this expectation that that’s what collaboration is, is everybody’s like social, can actually be counterproductive, I think.

John: [00:30:44] Yeah, and I think, you know, you asked that question earlier about, you know, what in your dev background helps in operations. And, you know, part of it is if you understand the other side, there’s an amount of trust by speaking the same language. And I don’t think organizationally, you know, being separate or being together is as significant as people trying to learn a little deeper and taking responsibility for things that are maybe not necessarily right in their job description. I think when you have people that work with other people kind of hand in hand and kind of try to really understand what’s going on and get to the bottom of problems, that’s when you have success.

Matty: It wouldn’t be— it’s been too long since we’ve had a— you basically just did a call out to Lucius’s blog post about that DevOps means Never saying that’s not my job. So, time for that.

Trevor: I think I mentioned this on our last episode with devs involved, but I remember there was one person I worked with who graduated with a degree that had some programming in it, like some intense programming in it, who didn’t understand what the RAM was in the system. Like, didn’t understand it at all. And then the other thing, they didn’t know what the boot menu was. They didn’t know how to go into the BIOS or any of that. And I was just like, how have you worked with computers for, you know, we’ll say 5 years and you have no idea how to get into the BIOS or what the BIOS is? I was like, what? So I like not knowing— I feel like knowing some of your basic keywords, and this is probably actually different from what I said, I think before I was defending them, but in terms of being able to communicate across the line, if you don’t know the basic terminology, you’re, you know, off to a bad start.

Aaron: [00:32:31] That’s one thing I’ve been blown away with, is the amount of networking terms in the last 2 or 3 years that I now use in conversation with people. 2 or 3 years ago, I had no idea. Like, I mean, an ACL was like 3 letters to me or whatever. Like, you know, I need—

Trevor: isn’t that that thing you tear?

Aaron: I need this machine to talk to this machine. Can we make that happen? Now I understand, like, what’s actually going on between between those.

Trevor: So to end the suspense—

Matty: oh, what happened in your data center when you had a keycard?

Trevor: You know, just figuring out what the hell was wrong with the server when it would go down. Like, I remember there was one time there, the video card that was in the system— if I remember correctly, it was a Quadro 8600— and it had just completely died, and we couldn’t get our hard line to the server working because it was interfering somehow. I had to go into the server and actually pull it out of the rack and Actually, you know what? I actually put the server in the rack, now that I think about it. That was fun. It used to just sit on top of the rack. That was also interesting. But no, nothing catastrophic happened because I had keycard access to the data center, Matt. Thank you.

Matty: [00:33:37] This is the difference between you talk to developers and talk to sysadmins. Sysadmins have way scarier, more fun stories about all hell breaking loose. So if you want to hear awesome stories, go back and listen to arresteddevops.com/18, The Sysadmin Show. So I’d like you guys, what would be the advice or tips you would offer to someone who’s got a development background that wants to understand operations better, wants to have some empathy for ops? What are some of the things they could do?

Trevor: Talk to an ops person, start doing things. There are plenty of resources online if you don’t have an ops person to start learning.

Nate: There’s a—

Trevor: I think I actually have it bookmarked and I’ll put it in the show notes, but there’s like a YouTube channel or something that’s like how to learn Linux or something, and it’s like from like this Linux guy who just wants to teach people who don’t understand Linux how to use Linux.

Matty: I’m gonna call a little bit of BS on your talk to an ops person thing, because that’s not really advice. I mean, I think it’s the right thing, but that’s— I’d like to like, how do you talk to an ops person? Like, how do you—

Trevor: [00:34:40] Like a human being?

John: It depends what language they speak, man.

Trevor: Yeah.

John: Well, I’m just sort of saying, like, I mean, the question I guess is, it’s kind of a weird question. Like somebody that’s doing development probably doesn’t just want to do ops. They have some specific interest in some specific thing. Right. So I guess it’s kind of hard to answer the question.

Matty: So maybe I should color it a little differently. So like one of the things we talk about in DevOps that like one of the squishy culturey things, but it’s super true. That’s important is empathy. Right, is understanding where the people on the other side of the wall are coming from. So I’m trying to like say like, what’s some tactical things? Because we’re technical people, we like things to do. So like, what’s a tactical way that if I’m a developer and I like the ops people I work with, I don’t think they’re jerks, you know, I’ll have a beer with them or something, but like, what are some ways to speak their language in a way that I can start to understand what it’s like to be an ops person?

Trevor: I’m going to cycle back 2 episodes again and say, ask for help. You know, will you explain to me what you’re doing? What are you working on right now?

John: [00:35:42] The big—

Trevor: I saw you were fighting with, you know, I saw you were talking to the boss and he looked angry. What happened? Why did it happen?

Aaron: I usually ask like, what’s the most frustrating part of your job right now?

Trevor: Right.

Aaron: I’ve been blown away by the answers. I’m like, I didn’t even know that that was part of your job or that was something you had to worry about. I didn’t know anybody had to worry about that. Like, I mean, just hearing story from another person that like— because sometimes it minimizes your problem. You’re like, well, I have a lot of things to worry about, but I don’t have to worry about that. Like, I mean, that’s always been interesting to me is like, what’s the big one that you’re worried about right now?

Matty: You know what, and that can lend itself to some rubber duck debugging too, right? Because if you go to them and you say like, what’s the biggest thing you’re worried about, they’re going to explain it and then that can sometimes help them solve their problem.

Trevor: I think the piece that goes hand in hand with that is don’t ask the question to ask the question. Ask the question because you care.

Aaron: Listen to the answer and engage in possible answers that can help them.

Matty: John, you’ve been known to be a cranky dude sometimes. Tell people, if you’re in the middle of putting out a fire, someone comes up to you and says, hey John, looks like things are stressful. What’s going on? How are you going to react? I mean, they don’t come up to you and be like, so what’s going on? But like, more like, hey, wow, this looks stressful.

John: [00:37:02] Wow, that looked like, that looked like a mess. Yeah, I would probably be a sarcastic ass like I usually am, actually.

Matty: So I guess, well, the lesson is don’t interrupt them in the middle of a fire. But maybe the fire, maybe you say, hey, John, it looks like you just had something stressful happen and now your heart rate may have reduced. What happened? So now I would—

Trevor: that’s the empathy piece you’re talking about. Don’t, you know, don’t be the asshole who’s interrupting somebody. You know, like, I see you running around with your head cut off. How can I interrupt you and make things more difficult for you?

John: Yeah, I mean, it’s kind of— again, it’s a question without a situation, and the opportunities to collaborate are situational. It’s when you have a problem. It’s, you know, you will have problems, and there will be problems that you have to work with other people, whether you’re a developer working with operations staff or vice versa. And those are the opportunities to ask questions about the environment that get you familiar with how the pieces fit together and to again kind of build that trust is, you know, ask questions. Don’t pretend like you know the answer when you don’t know the answer. Nothing makes me crazier than when somebody feels the need to answer a question despite the fact that they don’t know what the answer is. But if you work together, there’s plenty of opportunities for people to work together, and, you know, that’s when those things arise that allow you to, to see the other side and kind of get more familiar with what other people are dealing with.

Trevor: [00:38:26] Yeah, I think it was— I want to say it was somebody who was on with us during the Etsy episode mentioned something along those lines where he just started, you know, he had the pager at night and would be, you know, they would have him be, you know, he asked to be responsible or was made to be responsible for ops crises. He held the pager. Well, for what? So when things went wrong, he was on call. And then, you know, maybe they had somebody else to support him if he needed it, but, you know, it was going to go to him first. He would, you know, he was accountable for the fire in the middle of the night.

Matty: No, you’re right. That was, that was definitely on the Etsy episode. Yeah, they brought up, they said on the web ops team there was one of their developers asked to be part of on-call for the web ops team to learn about that. And so if you’d like to learn more about that show, it was arrestedevops.com/11. It’s one of my personal favorites. So one last question before we go into our checkouts. We asked the sysadmins last week for what’s their favorite crap-your-pants thing they had. And so we’re still talking about ops. So what’s your most badass war story that’s operationally focused, that’s around that case? So Aaron, you get to go first.

Aaron: [00:39:35] Let’s see, the first thing that comes to mind is I just deployed one of our applications. It was Friday, 5 o’clock, and I went out to dinner with, with my wife And then I, like, get a text from my boss that his boss said the application’s not running, and it’s not painting. None of the CSS is painting. And I’m like, I’m sure it is. I’m absolutely sure it is. So I have to pull it up on my phone, and I look, and it’s there or whatever, but it was deployed in an HA fashion where I had 2 nodes. One of the nodes, the CSS was working, and one of them it wasn’t, so I had to pretty much leave that dinner and go fix it. Immediately tried to get it back up because it was kind of a— I was like, how in the world is he looking at this at this particular time? I was like, how often does he look at this? And why is it Friday, like, while I’m having dinner with my wife?

Matty: But we—

Aaron: I mean, I got it, I got it back up and running after a little bit, but it was just like, what, right now?

Matty: What about you, John?

John: Oh man, they all kind of run together. So many late nights, right? We had a storage array a couple jobs ago that used to periodically just grind to a halt and pretty much stop responding and would take down everything. Like, everything was on it. The entire data center pretty much ran on it. Everything was virtualized and all the storage was on the one host. And so whenever the thing would go down, man, it was a nightmare. We had to basically bring up the whole infrastructure. But I mean, I don’t know, everyone’s got those type of stories, right?

Matty: [00:41:02] What do you got for us, Nate?

Nate: So this is somewhat mixed app ops, I guess. I released— we had a SOAP service to do all of our app error logging. And the day before my honeymoon, I decided it was a good idea to release a new version of that, which essentially caused an infinite loop calling back to itself to log its own errors. And the plane lands in Argentina, and my BlackBerry starts lighting up. It almost caught fire, I think. This is going back a ways, and, um, you know, I had to do that emergency call to get that rolled back. And that taught me to never do a release on a weekend or prior to leaving town, for sure.

John: People disappear in Argentina, but they managed to find you, huh?

Matty: Yeah, yeah, right. Trevor, you didn’t get to tell a war story last week, so what do you got?

Trevor: Well, I was rebuilding all the computers for the Smart Lab and We merged a few XPSs together. So these are desktop machines, not servers. So less fun maybe for you, but it was pretty fun for me. And there were ATI NVIDIA— sorry, ATI 5770s, which were brand new at the time. And we put them in the new box. There was, you know, this was a 12-core i7 boxes with, I think at the time, 16 gigs of RAM. And, you know, they were just beasts. We were putting dual 5770s in there. I get the machine up. I try to get the 5770s to install their drivers, and they just completely fail. Absolutely completely fail. And I call ATI, and they’re like, well, we don’t know what’s going on, but that driver is probably broken. You have to uninstall the driver. Then you have to download this third-party tool to actually uninstall the driver, and then you can install the real driver. I was like, really? Screw you, ATI. I am never buying another product from you ever again. Eventually the machine did work, but those cards were never ever reliable.

Matty: [00:43:06] So I just have one, one quick story that it’s actually about someone else, but it’s one of my favorite sysadmin ops panic stories. So back in the olden NT 4 days, that just to put some context into it, one of my coworkers was in charge of our lab, like our development lab. And so what would happen was when machines would recycle out of production, they would get turned over to this coworker who could then use them in the lab. So what happened was a machine had been— database server had been pulled out of production and given over to this person who we’ll call RefJack. And he was tasked with saying, okay, fine, this can be your thing now. And then of course, and this kind of thing would happen, they They came to him the next day and went, oh my god, we need to get some data off that thing. We totally forgot. We need you to boot it up real quick. So he calls me in a panic, and he’s telling me this. And he’s like, I plugged the storage array into this thing, and it’s not seeing the external data. It’s not seeing the disk. It’s not seeing the disk. And this was with those Compaq external arrays that would have 2 disk subsystems that were connected to 2 different RAID cards inside the machine. So we’re kind of looking at it. Sure enough, it’s not seeing it. And I kind of do the troubleshooting thing. I’m like, well, let’s just make sure. I’m like, you sure the cables are tight? Yeah, the cables are tight. Right? So I’m like, well, I’m going to double-check. So I checked the SCSI cables, and in his hurry, he had plugged the 2 RAID cards into each other and the 2 storage arrays into each other. And from that point on, for at least a year, I called him Loopback. So— and I also, if we ever get him on the podcast, he can tell you lots of stupid crap that I’ve done too. So it’s all fair. But let’s move into our checkout. So at this point, our panelists will share with you something awesome, something interesting, that they think you would dig. It can be a beer, it can be a book, it can be a tool, all that kind of stuff. So we’ll start with you, Nate.

Nate: [00:44:56] So what I picked is something called ConEmu. I suppose that’s how you pronounce it. Console emulator, ConEmu. It’s a great console tool for those Windows people out there, you know, that command executable is horrible, the CMD. This allows you to have tabbed windows. You can also do different types. You can do Bash and PowerShell You can do different types of console. That’s ConEmu. The links— I provided the link. It’ll be on the website. It’s just a great tool. You can have a PowerShell window open or a number of them, and at the same time have a command line open or Git Bash or whatever.

Matty: John?

John: So yeah, I was trying to think of some checkouts that I could share because pretty much all I’ve been doing right now is, you know, running the kids around to their stuff, and I don’t think anybody wants to check out Wheaton Rams youth football as much as I do. So, but Jeff suggested that I check out Nuskie’s Bacon, so that’s gonna be my, my checkout. I also need to check it out. As you know, Matt, I love bacon about as much as anybody possibly could love anything. So Nuskie’s Slab Bacon is gonna be my checkout.

Matty: [00:46:03] Awesome.

Trevor: Sounds wonderful.

Matty: Aaron, what do you got?

Aaron: The big thing I’ve been having people watch, because it just blew me away, is Jez Humble’s talk. ChefConf this year, because we’re trying to do a lot of organizational change right now, and he talks about, I think it was HP, and just how they said, you know, this isn’t working. Let’s do something different. And they spent a lot of time making their development just really home again. So I’ve just been blown away by that talk. I’m telling everybody to watch it.

Matty: It is a great talk, and the HP LaserJet team, that story is one of my favorites, and it’s probably like the 4th time that it gets named. It’s been name-checked on this episode because it’s amazing. So Trevor, what do you got for us?

Trevor: All right, so first and foremost, I want to tell everybody to check out TeamCity because I got to play with TeamCity some more this week, and I just, I forgot how much I loved it because I’ve been so bogged down doing other crap. It’s just, it’s amazing the stuff it just does out of the box, and it’s like, unlike its competitor that is like a butler of some sort. Kinda sucks a lot. So check out TeamCity. Also, I found on Reddit today this neat little Visual Studio extension called Encourage, which is supposed to like— I installed it, I haven’t seen it actually pop up yet, but it’s supposed to give you little encouraging words when you are coding, like relevant to what you’re doing. So I’m hoping that’s as interesting as it sounds. And as seen in Patrick Stewart’s ALS Ice Bucket Challenge. I’ve been drinking during the episode today, Chivas Regal, or Chivas Regal, however you want to pronounce it. It’s good scotch. I’m enjoying it. It’s cheap. Speaking of which, I plan on doing the Ice Bucket Challenge that Matt brought up this weekend. So, uh, so we’ll post that soon.

Matty: [00:47:51] Awesome. Okay, so my checkouts real quick. It’s this app for the iPhone called Lumosity. It’s supposed to be for exercising your brain. So every day, so you basically go in, you say, what are the things that you want to be better at? If it’s remembering names or solving problems or not getting distracted. And then it gives you little workouts, brain workouts that you do every day. I’ve been doing it for a couple of days. I don’t know if I’m any smarter, but it’s kind of fun. So that’s Lumosity. It’s in the App Store. I don’t know, maybe it’s for Android. I don’t know. I don’t know these things. So reminder, we have a newsletter. Yes, I didn’t send one out this week about this episode. But you can subscribe to it at arresteddevops.com/bananastand. It is the best way to know about upcoming podcast episodes, when I remember to send it, and possibly cool news with DevOps.

Trevor: Also, don’t forget that it’s conference season. We’d like to remind you about Flowcon, which is happening September 3rd and 4th in San Francisco. Flowcon is going to bring together technologists and industry leaders passionate about innovation through lean product development, continuous delivery, lean UX, and DevOps. We’ll be exploring the role of culture, technology, and design in growing organizations that thrive in an environment of continual change. We’ll be providing inspiring and actionable information for key decision makers responsible for products and services that depend on software. The full program includes speakers from Google, Netflix, Heroku, Nordstrom, SoundCloud, Macy’s, HP, Joyent, ThoughtWorks, and IBM. The second day features an open space unconference and workshops from Don Reinertsen, Mary and Tom Poppendieck, Whitney Johnson, and Sarah B. Nelson. Use the special promotion code ArrestedDevOps50 for $50 off at arresteddevops.com/flocon. And I want to add, I will be attending Flocon, so if anybody else is going to be there, let me know and we can say hi or whatever.

Matty: [00:49:48] Remember, as we’ve talked about, O’Reilly Velocity New York is happening September 15th through the 17th in New York City, and it is the place where DevOps, WebOps, and performance professionals gather for a legendary learning and networking experience that explores why a faster, stronger web is no longer an option but a necessity. Hear from the best speakers in the industry who will delve into topics ranging from hardcore math and stats, to monitoring, clustering, analytics, and organizational culture. Attend Velocity and you’ll view your work, the technologies you use, and your organization in completely new ways. Use code ARDEV20 to save 20% at arresteddevops.com/velocitynyc.

Trevor: Thanks again to our sponsors, VictorOps and Datadog, and to our loyal listeners. If you enjoy Arrested DevOps, we’d appreciate it if you would visit arresteddevops.com/itunes and leave us a review in the iTunes Store. We’ll read our favorite ones on upcoming episodes. So far, our most favorite one reads a little bit like this.

Matty: [00:50:54] Am I supposed to read something?

Trevor: That’s because nobody has posted a review, so please go to iTunes and throw a review in there.

Matty: The most recent review was in May, so yeah, we’re begging for love. So thanks to Nate, John, and Aaron for joining us tonight. That was awesome.

Trevor: Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter.

Matty: We are always happy to get your input, ideas, feedback, or I don’t know, gift certificates, or I don’t know, pictures of puppies at shows@arresteddevops.com.

Trevor: I am Matt, @MattStratton, and I’m Trevor, @TrevorGHess.

Matty: We are Arrested DevOps, and remember, there’s always DevOps.

Trevor: In the banana stand.