The Sysadmin Show

Matty: [00:00:07] Welcome to Arrested DevOps episode 18, the SysAdmin Show. I’m your co-host, Matt Stratton, on the Twitter known as @MattStratton.

Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.

Matty: Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that figures if you’re listening to this podcast, then you must be pretty cool. You can find out about joining their cool cloud services team at 10thmagnitude.com. Also, don’t forget that registration is available for the first ever DevOps Days Chicago. Arrested DevOps listeners can get 10% off their registration with the code ADO10. That’s ADO and the number 10, like for 10%. Visit arresteddevops.com/devopsdayschicago for all the details.

Trevor: So if you guys remember way back in episode 3, So longtime listeners or new listeners, that was arresteddevops.com/3. We focused on developers. Now, 15 episodes later, we’re finally talking to sysadmins, the ops in DevOps.

Matty: [00:01:12] So yeah, it’s, uh, I’m pretty excited for this one. Um, this has been kind of a— Trevor, I know you just came back from vacation or something. You’ve been traveling. I’m actually not even in Chicago right now. I’m out in Newport Beach, which is where Arrested Development takes place. So there’s that. So yeah, coming to you from the OC.

Trevor: Don’t call it that, but I’m pumped.

Matty: So that being said, this episode is sponsored by PagerDuty. PagerDuty does not endorse that bad, bad joke, but PagerDuty does eliminate the noise, chaos, and manual processes across the entire incident lifecycle to decrease resolution time. PagerDuty is trusted by companies like Etsy, Nike, and GitHub. To sign up for a free 30-day trial, visit arresteddevops.com/pagerduty.

Trevor: And this podcast is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, events, and alerts from over 80 common infrastructure tools so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a 14-day trial at arresteddevops.com/datadog18. That’s arresteddevops.com/datadog18.

Matty: [00:02:33] So like Trevor said, we did a dev show, was one of our very first ones we did back in, I think it was like in January. So I’m surprised it took this long for us to do the sysadmin show, but I’m really excited because These are my people, right? The sysadmins. You know, the ones who do all the real work. I know that’s not very DevOps, but tonight I don’t care because sysadmins rule.

Trevor: You guys can’t see it, but Mike Wright just did the rock-out horns.

Matty: That’s right. Metal. So speaking of which, we should probably introduce the panel of awesome sysadmins.

Trevor: Guys, you guys want to go ahead and we’ll go roundtable? Brian, you want to introduce yourself first?

Brian: Sure. My name is Brian Wagner. I do a couple of other podcasts. I do linuxinstall.net and I do DevOps Mastery. I’ve been doing the sysadmin thing for almost 20 years. Started off at FedEx doing email administration and system administration and just kept doing it. Been supporting web ops type stuff for most of my career and doing a whole bunch of security stuff too.

Matty: [00:03:39] Awesome.

Trevor: Thanks for joining us tonight, Brian. Hey, Chris, want to introduce yourself?

Chris: Yeah, I’m Chris Read. I’ve been doing the sysadmin thing for just over 20 years. Done a fair bit of consulting at IBM, the great evil, as well as many years at ThoughtWorks, just spending a lot of time flip-flopping between being a dev and being a sysadmin or being a network admin. Now I get to, get to do all of those things.

Trevor: Awesome. Thanks for joining us as well. And Mike, you want to introduce yourself?

Mike: Sure. My name is Mike Fiedler, @MikeFiedler on Twitter. I’ve been in this field of computing. It seems like you’ve invited everybody who’s got about 20 years or more experience. So I’m a member of that club. Done this across a couple of continents, a bunch of different companies, both larger and smaller. I currently work for Datadog as the Director of Tech Ops, and we build big things and I love doing what we do. So this is kind of, you know, coming full circle. I’m also a sometimes co-host on the Food Fight show and pretty vocal on a bunch of different other topics. So you’ll see my name around at some point.

Trevor: [00:04:50] Awesome. Thanks for joining us. Matt, you want to kick off the discussion?

Matty: Of course, I would love to. So one of the things I think that’s kind of a good start is— so we keep talking about, we’re like, all right, it’s this admin show, it’s this admin show. So what exactly does it mean to be a sysadmin? Like, how would you define a sysadmin?

Brian: A sysadmin is, is the person responsible for maintaining the operating system, all of the network connectivity, making sure the disks don’t run out of space, that there’s enough disks. In a DevOps type way, we’re generally the ones that have to figure out how to take the code the developers are doing and making it fit on servers, figuring out what size they need to be, that kind of thing. And pretty much just responsible for the care and feeding of all the virtual or physical hardware, depending on what you’re talking about.

Chris: Yep, I’d agree with that. I’d say it’s also keeping the lights on, keeping everything up and running. I think traditionally also it’s always been we take what the developers are throwing at us and against generally the developers’ best efforts, we make sure that it stays up and stays running. Something that I I like to try and at least teach my PFYs when I’ve got them under me is to actually have the conversations with the devs as well and help them understand the concept of mechanical sympathy for what their code is having to run on. Because sometimes they just don’t understand the limitations of what’s there.

Mike: [00:06:13] So I’m going to take a different tack on the definition. I did a talk at Velocity New York last year. Might have been last year, where I kind of tried to delve down the history of what systems administration entailed and where it’s gone. And just breaking down the word of administrator, what is an administrator? It’s someone who’s typically tasked with managing shared resources for the better good of everyone. So whether that be, you know, hard disks or operating system patches, but beyond that, even full data centers, so that way an entire company can continue to survive, to support Chris’s keeping the lights on metaphor. It’s taking whatever the company needs to do to grow and sustain itself and expand on that. And if there are shared resources, then, then kind of manage the pool of shared resources and figure out what needs to be allocated when. That’s a more touchy-feely description, but I like it.

Matty: [00:07:16] Okay. It’s interesting too to think about how kind of historically, right? Like going from like the idea of computer operator to computer administrator, you know, because I think about our system operator, right? Like to think about kind of when I was coming up and I think we’re all kind of again, you know, around the same vintage that it was, it was sysop, right? Like it wasn’t sysadmin, it was sysop. It was, you were the operator of the computer and it’s, I kind of, kind of like to think about Again, right, like Mike, you were talking about where it gets into that touchy-feely. It’s like where that evolution goes, a little more of the— I don’t know. I’m just kind of curious what you think is the implication of that etymology word, that word change from being the operator of the machine to the administrator of the machine.

Chris: I think there’s a difference between the two. My brother-in-law is a sysop for IBM in the Global Mainframe Support Center in Johannesburg. He is a SysOp, and talking to him about what he does day to day, it’s very much, are the batch jobs running? Is everything still up? Is everything going? That’s about as far as he gets to go within the hierarchy. He doesn’t get to understand or spec or do any capacity planning or have a look at upgrade cycles or any of the kind of general stuff which I think separates the SysOp from the SysAdmin. Sysadmin is— the more I think about Mike’s definition, the more I like it, where it’s beyond— it’s that whole greater good keeping everything going. It’s that wider range. I think it’s gone from the days where it was possibly some kind of manager who was allocating the budget to actually having someone who’s looking after the systems and understands them technically, starting to specify these are the classes of servers we need for the workloads that we have. This is how we need to scale them. This is how we need to grow them. This is how far we can go with our current capacity before we need to look at Better cooling, better power, things like that.

Mike: [00:09:13] Yeah, and to go back to kind of like historical definitions, right? Some of the earliest computers had computer operators, sysops, or just, you know, computer operators, the people who moved the wires and plugged things in and then pressed a button to program a calculation that other people had designed aka developers, in order to execute that kind of program and produce the results, whether those be, you know, lights being lit up, printouts, etc. And those were the operators. As it grew, as more and more operators and the programmers kind of merged because we developed these higher-level abstractions of programming language to enable that interface to machines, now you had a big computer room that had a bunch of fans and a bunch of tape reels that had a punch card thing. So the guy who, or gal, whoever was in charge of running that room had to make sure that everybody got the right amount of timeshare at their university on this computer, that if the card feeder kind of broke down, that they knew how to change the belt or change the thing. And these kind of skill sets have changed over time to map to the technologies that are in play. But the actual practice of balancing the needs versus the supply and demand, really, of what we have has always been the systems administrator role.

Matty: [00:10:44] I concur. So, but here’s an interesting thing. So I was thinking about this a little bit, and a lot of when you’re kind of, when you’re saying, when you’re talking about like what that evolution from operator to administrator, or just the, maybe it’s less about the evolution from operator to administrator, because again, like Chris pointed out, there’s computer operator is still a thing and it’s not necessarily, but it’s as we evolved to require that extra thing, it’s kind of like DevOps, right? You know, and my friend John Smith is fond of saying, because he, when he’s introduced to DevOps, he’s kind of like, huh, I’ve been doing this a long time back when I just called it working.

Mike: You know, so we could definitely get into an hour-long argument over what DevOps means. I don’t know if that’s the best use of our time.

Matty: No, but I wanted to kind of go into it, but thinking about it, but I think it’s interesting. But my point was sort of like that evolutionary thing is like, in a lot of ways to me is similar to that. And so that being said, you know, obviously we’re a show about, about DevOps. And so the question I came up with, I’m kind of curious. I’ve never asked this one before, but what’s your favorite thing about DevOps?

Mike: [00:11:54] It’s shorter than sysadmin. That’s 3 syllables versus 2. That’s, that’s my favorite thing.

Matty: What was it? I think Patrick DeBois said at Minneapolis, he said something about that they called it DevOps because it was shorter than Agile System Administration, which was the original name.

Mike: Yep.

Brian: Yep.

Mike: Does anybody else have a favorite part about it?

Brian: My favorite part about it is just, it’s the collaboration and getting to learn to play with other things, right? I wouldn’t normally have a chance to talk to somebody like Trevor, who’s a developer, who’s focused on doing development and learn about how Java’s inner workings work. But because I do web ops, DevOps type stuff, I have to learn how those things work because I have to be able to support those guys. And so just like I don’t think it’s right for developers not to understand how things like the network stack works or anything else basic, I mean, I don’t expect them to have deep knowledge, but at least basic knowledge of how the operating system works. Being able to do that collaboration and that crosstalk and learn some things as well as teach some things is one of my favorite things about being in the DevOps world.

Chris: [00:13:04] Yeah, for me, my favorite one is having a term that people can gather around and try and get. You know, in the past, I’ve been the developer or been the sysadmin trying to talk to the other side, and it’s, oh, you can’t do this because X, Y, and Z. Now it’s at least a point that managers are now aware of. You can go and wave it. It’s like, well, They’ve all agreed that they want the DevOps hotness, and so now it’s an excuse that you can wave in front of them to say it’s good for people to talk to each other.

Mike: I mean, that’s fair. Those are all probably better definitions than my 30% syllable cut. I guess the favorite part is that it’s become— it’s opened up the discussion for me, that it’s no longer, here’s these guys who sit in a closet and tinker with things that we don’t understand and we don’t necessarily want to play with them. Now it’s opened up the platform of discussion to say, all right, here are these, all these kind of realms of informational knowledge that we just don’t have proficiency in, but there’s these other people who do. So let’s talk amongst ourselves and figure that out.

Brian: [00:14:09] And just to go on and build on that, it’s also that tearing down those walls. I mean, I don’t know about you guys, but when I started off doing the thing that is now DevOps, but when I was doing it before, It was pretty much developer create code, developer hand code to sysadmin, sysadmin deploy code. And there was really not a lot of communications that was going on in that stream, which made life freaking hell for everybody. The developers were never happy because they never had what they needed. The sysadmins never knew what anybody wanted, but they just knew everybody was pissed off at them and shit ran downhill and they were standing at the bottom of the hill to catch it. And now that we have something, to Chris’s point, now that we have something that we can say, you know, this is what DevOps is, and this DevOps thing that we should all be participating in is about collaboration as much as it’s about automation and how we do things. So, you know, having those collaboration points to point to and say, hey, this is what we really need to do, and have it there to back us up, I think is just incredibly useful.

Matty: We have a question came in from our viewers. So Patrick Cable says, Interestingly enough, I wonder to what degree of intentionality AWS used the word SysOps for one of their certifications. So there’s an Amazon cert called Certified SysOps Admin Associate. So I don’t know how much that’s a question as it’s just an observation.

Mike: [00:15:29] It’s what do you guys think about this term?

Matty: I guess.

Mike: Yeah.

Matty: Yeah.

Mike: I haven’t heard of it. I’m going to look it up now, but I’m going to guess that it probably has something to do with how do you run systems on this vendor’s platform. It seems like most certifications that are out there apply to a vendor who is providing them. So, you know, if you were to get Red Hat certified, Red Hat is not going to teach you a lot about Oracle software. They’re going to teach you about Red Hat stuff. So I’m going to guess that AWS Certified SysOps is going to teach you about system-level operations on the AWS platform. Obviously, I don’t know that. That’s just my guess. But that seems to be a pattern of certification that I’ve seen in the world.

Matty: Yeah, it says the concepts that you should understand for the exam include deploying, managing, and operating highly available systems on AWS. One of my favorites is one of them is estimating AWS usage costs.

Chris: So yeah, it kind of sounds the same to me. My experience with Amazon in training is the last project I actually did at ThoughtWorks before I left them was actually working with Amazon. To develop the very first AWS training courses. And so I basically— me and one of the EC2 core devs initially, we sat down and developed a full-day workshop. And Amazon were doing this very, very reluctantly because they were just getting hounded by people left, right, and center. And at least at that stage, this was middle of 2009 we were doing this, they were constantly having people bugging them to do training and to get certification. And they were just like, it’s not our core business, we’re not interested. So, we’ve got ThoughtWorks out to develop it initially. Given that history, I’d say it’s probably people coming to say, we will give you large amounts of money for some certification, please. So, they just outsourced it to someone else and said, there you go. Do what the PHPs are asking for and take it from there.

Matty: [00:17:25] Listeners, if you are a Certified SysOps Admin Associate, please let us know. What you think of that program and how much of it’s operation and how much of it is administration in your opinion. This is called audience feedback.

Mike: Yeah, it looks interesting. There’s an exam. There are no prerequisites, so if you’ve had some experience in AWS, you might actually be able to take it without taking any training and pass and become certified. Maybe I’ll do that tomorrow.

Matty: I don’t know.

Chris: Yeah, take a, take a practice exam, see how far we get compared to this.

Matty: I had a, the, uh, a guy that I worked with who was like the director of network security at this insurance company I worked with. And this was probably like 10 years ago or something. Or actually when he did it, it was probably a good 15 ’cause it was NT4, but he just decided to get his MCSE in a day. He didn’t study. And again, and this was not because it was super easy, it was more because he was like, but he was just like, I bet I could do this. That’s just a lot of tests.

Mike: [00:18:26] Yeah, MCSE took a while. It’s been years since I thought about that. But the more fun one that I had with was Novell. This is going back some 15, 16 years. And that was a lot of fun. Novell NetWare, the original networking.

Matty: Yeah. Dude, I remember when Novell was big enough that— remember when you would call in their support and they had like a radio station that was the hold music? That had like, instead of traffic reports, it was like they would list like the hold times for the different lines you might be calling in on. Because there were that many people that were using Novell that it mattered. Good old Red Client. Here’s the thing. So I sort of have this belief that sysadmins are inherently cynical. So what do you think about that?

Brian: Why are you limiting it just to sysadmins? Because I’ve met more than a few cynical developers, and I just think it’s us as IT people in general are just cynical. Like we always—

Matty: I think pessimistic is more of the word I’m looking for. Like, I feel like the shops I’ve worked in and the sysadmin teams I’ve worked with or had working for me, we tend as a breed. So, so this is my theory and they can tell me if I think it’s right or wrong, or well, my theory of course is right, but is that it goes with the job, right? Because like part of the job of the sysadmin is to think of all the things that could go wrong. So you can guard against them. And if you’re doing that all the time, you start to see shadows everywhere, right? You know, I don’t know. I just, I just feel like it’s like the last team that I ran, the thing I would say all the time is, okay, look, I know you guys can think of 100 reasons this won’t work. Let’s just make believe it will.

Mike: [00:20:05] You know, I mean, I’ll differ a little bit on that, on that approach. I try to be very optimistic and realistic and try not to let cynicism creep too much into my day-to-day. I feel like that is definitely a stigma that exists around, as Brian said, not just systems administrators but also tech people in general. And I think that might be because we see how badly things are broken on a day-to-day basis. Like anybody who sat down at a system that they inherited is like, oh, this is terrible. Anybody who’s inherited some code and looks at it, oh, this is terrible. If you looked at your own code 2 years later, oh, this is terrible, right? So we deal with that kind of struggle of everything is just crap and we have to deal with that, whether it be fixing it or replacing it or just walking away and having a stiff drink, just dealing with these kind of real life, and this is our job. Like, this is what we do day in, day out. So I definitely see how cynical attitudes can help brave the storm of basically— I think Brian mentioned shit running downhill, you know? But that doesn’t apply only to systems people. It just is probably more common with the communities that you hang with are systems people, so you see the cynicism there as opposed to the people who are inheriting broken code as opposed to inventing code every day.

Chris: [00:21:36] There’s a slight theory I have carrying on with that is part of being the ones at the bottom of the hill to clean it up. We’ve got no one else to pass it on to, so you kind of have to vent back up to the rest of the world that it’s all broken, try and kind of aim the fan to spray some of it back up.

Matty: So I wonder, I guess my other part of that too is it’s a matter of, and it’s funny, it brings to mind that this part in High Fidelity when he says, what came first, the music or the misery? What came first, right? The system administration or the pessimism? And by that I mean, like, is it a thing that you develop because of, like, you know, like you said, it’s because you’re cleaning up the messes a lot, because you’re seeing it, because you’re in it. But then at the same token, is it something where you kind of have to like that, I think, or else you’re going to go do something else with your life, right? And Mike, I’m like you, like, I tend to be a much more optimistic person when it comes to things, which is, you know, when it comes to those things. So I don’t know why I still do this. That actually is probably why I got why I don’t do ops anymore directly, but do you kind of have to— does that have to be a thing that’s fun for you, do you think?

Mike: [00:22:37] We’re in this field for a variety of reasons. Everybody has come to systems administration through some path. Nobody goes, studies computer science, and is immediately a systems administrator. In fact, very few people actually follow that path. We all are drawn to this. Through some sort of personality trait or kind of curiosity to see behind the ether. Maybe some people are on control power trips. But like you said, Matt, if you don’t like it, then you go and do something else. Like Chris, you’ve been doing system stuff, you’ve been doing development stuff. Now I kind of straddle the line. I do a lot of coding, but it’s usually infrastructure-focused coding. So you kind of have to find where you want to go, and then that’s the part that you enjoy about it. If you don’t like it, you know, what was it, there’s a popular Ignite and series of talks by Nathan Harvey about quitting your job. You know, if you don’t want to do that because you don’t like doing that, somebody else will hire you to probably do stuff that you want to do.

Chris: [00:23:45] If you do it, you’ll do it better. I mean, for me, it’s the having to put up with the stuff breaking and the pain and suffering is, to me, it’s just a side effect. I do it because there is the joy in seeing a perfectly running system under heavy load doing what it needs to do and users not noticing that it’s there. Yesterday we did a cutover of one of our big data systems and we flipped it over intraday while people were using it and no one noticed. It’s that quiet, no one noticed, that fills you with the most joy. That’s like, yes, we did this complicated thing and no one cared. That’s where the joy comes in. The key thing for me is I agree with pretty much everything you said there, Michael, except for the guys that are in it for the power trip are the guys you must be scared of the most and try and get them wheedled out as fast as possible because they are the ones that cause the most pain and suffering for everyone because they enjoy the pain and suffering that others endure due to them.

Mike: [00:24:46] Hey, I didn’t say that everybody in this field is perfect. There are a lot of different types. But like you said, yeah, having people not notice what you do. So I tossed in a link that will go in the show notes. This is a group of guys who change a tire on a car while it’s up on 2 wheels. And, you know, they change a flat and replace a tire. That’s how a lot of like live operations feels like to me. It’s, you want to keep that car going. And there is definitely a thrill there of being able to do this live and sometimes, you know, go out, hang out with some friends and be like, hey, guess what we did last week? We yanked out 15 different components and nobody knew.

Matty: Yeah, that’s exactly it.

Brian: See, you guys are talking about it from that end and I’m sorry, I’m going to go back into this cynical thing, but I’m so used to getting dumped on, right? I would get stuff. Sorry, Trevor, but I would get stuff from really wonky developers who really probably shouldn’t have been developing code and would throw it my way and I would put it on systems and it was like, wow, what can we write that’s going to make this system stay up long enough for them to figure out what the bug is so that they can fix it, but we won’t impact production? And, you know, that whole rush of just, can I beat the really bad code I just received? Or the latest Microsoft patch that’s decided to totally screw up IE and have it flood with all kinds of other stuff it was never supposed to send, and it’s really messing with the code. And can I beat those problems back and nobody knows? Because to me, like you guys are saying too, you’re failing as an ops person if anybody notices. It’s kind of the way I feel. Like, if you notice that we had a problem, then that’s my— and I feel responsible for that, and that’s why I want to try to prevent. And it’s that rush of trying to beat the system and make sure that I can keep it up and come up with a solution that’s going to work without putting the company out of business because we’re going to spend so much money. At the same time, not allow people to actually use whatever systems we’re talking about. There’s more to it than just all the— you guys were being very shiny-happy, but there’s also some negatives. Like, when things are going bad, there’s also that rush too. And I think as a sysadmin, you can’t not enjoy fixing the shit.

Matty: [00:27:04] Oh yeah. So when, when Mike was talking about nobody says they want to grow up and be a sysadmin, so that was the thing. So in one of my last jobs, when I was at apartments.com, I kind of started this intern program in our tech ops. And for the first few that I did, it was really, really hard. It was a really hard hire to hire a kid that was in school to come do tech ops. And what I’d have is I’d have someone who was like, studying to like do application development or project management, or they want to be, you know, this was sort of like, okay, here’s a thing I can do where I can go get a little bit of experience. And then this dude came in and interviewed for one of the slots. And I was like, so what is it you want to do when you grow up? And he’s like, I want to be a sysadmin. And I was like, bullshit. No, tell me the truth. I’m like, you’re just kidding. He goes, no, this is what I want to do. And I was like, I hired him. And dude was like the best hire I’ve ever had. This was a couple of years ago. So, I mean, he just finished up school. He is a DevOps engineer now at a gaming company in Chicago and making more money than I made when I was like 38, which was a year ago.

Mike: [00:28:09] I just realized, maybe not 38, but you know, you made that sound like so long ago.

Matty: Like, oh, I was just sort of saying numbers. I’ve been up for like 36 hours at this point or 38 hours. Yeah, exactly. But yeah, it was just— and it was because— and so what made him an awesome intern was that he would do anything. And it wasn’t like just whatever, because it was interesting to him, right? It was like, oh cool, I get to go do this. And I remember talking about— he’d be like, yeah, and like he’d be like in a class in school and they’d be talking about virtualization. He’s like, dude, we just completely migrated our entire virtualization network, you know, and you guys are just talking about it and I’m doing it. And And I think like that’s a rare thing. Like I feel like a lot of us got into this side of the world not necessarily on purpose. So when someone really wants to do it, we kind of, I think at first, I don’t know, for me, I have that reaction like I had, which was the, for reals? You know, just because I think it doesn’t occur to people as a choice, maybe more now than before. I don’t know.

Mike: [00:29:12] Part of the problem that exists in the world is that it isn’t a profession that’s visible or like shown to younger people as this is an option for you. People hear about, you know, oh, I could go be a doctor, I could be an accountant, I could be an architect, I could be a fireman, but nobody really hears about systems administrators. There was a systems administrator in a movie that was played by Jack Black whose last name was Fiedler. The character’s name was Fiedler, that’s why I remember it, but he was a systems administrator, Enemy of the State. And I was like, wow, there’s a sysadmin in a movie. Like, that is new to me. And oh, and yes, his name is Fiedler. Yay me. But it was, uh, it was just exciting that this is a niche of the technology world, which if you really think about it, is kind of foundational. Uh, again, like, like somebody mentioned earlier, maybe before we were live, like characters are like, yeah, in, in some TV shows are like, yeah, you don’t know what we do? We build the internet. We make all of those fancy things that you love to poke and click and point and apps and downloads and likes. We do that. This is what we do. We build that. Maybe not for a service or a company that you’re actively using right now, but at some point in kind of our history, we’re helping you do what you want to do from, again, behind the scenes.

Trevor: [00:30:36] You know, and I think Matt’s probably, you know, at least in terms of sysadmins that I know well, Matt’s probably the closest to a pure sysadmin that I’ve ever met.

Matty: Wow, that’s scary.

Brian: Is that a compliment or a—

Trevor: No, it’s not meant to be detrimental in any way, but to kind of highlight this, I think I’m by far the youngest in the room. Growing up over the past 20 years, as opposed to having worked for the past 20 years, everybody I know who maintains systems was also a developer. So every system administrator in the school also taught the programming classes. I know a person who works for a naval shipyard as a systems administrator, but he also runs all the development network at the grade school I used to go to. One of the best programming teachers I had at Columbia is also the system architect. I can’t think of anybody, you know, Matt codes for Chef. I can’t think of anybody who’s a pure sysadmin and doesn’t do code also.

Brian: [00:31:39] Well, Trevor, I think this is the reason why I wrote an article a few months back on DevOps Mastery saying basically in 5 years, if you weren’t scripting, then you aren’t going to have a job as a sysadmin. And the shot across the bow on that was Microsoft saying, okay, that’s it.

Mike: We give up.

Brian: We’re not doing a GUI anymore. And we’re going to ship 2012 without a GUI. And if you didn’t wake up and realize at that point that you need to learn how to at least program in PowerShell, if not one of the other languages— Ruby, Python, Perl, pick your favorite language— but as a sysadmin, if you’re not doing that, you really don’t have a choice. Because I have a feeling maybe not the next release of Windows Server, but the one after that, there isn’t going to be a GUI option. They’re going to expect you to move that over. And a lot of that has come from Azure and Microsoft learning that Hey, you know what? It’s really hard to do a GUI across the internet, and being able to run scripts and shells is much more efficient and much more consistent. And I think that’s why a lot of what you’re seeing is there. I think it’s more that they just don’t have a choice. Like, you have to learn how to develop. And I actually pulled a developer into my team at my last— when I was a team lead, I wanted a developer to come to my team because I wanted him to teach my scripters how to be more like a developer and how to like use version control. Because, you know, I’m really tired of going into places and everybody’s backup scheme is first version of script .back, second version of script .back.date, third version .back.back.date. I mean, you know, those aren’t version control systems and it’s not ways to manage your code. And then nobody knows which is the right one.

Matty: [00:33:19] So somehow Nathan Harvey just felt this like Wow, I want to give someone a hug moment right now. Because that’s like one of his— I love his thing. He’s like, if you think that version control is .back, then stop what you’re doing and go install Git. I’ll wait.

Brian: Yeah. I mean, was that because of your— was he on your last show? But I saw some tweets.

Matty: He said it on there and I’ve seen him do it in talks. It’s, it’s, but yeah, it’s totally true.

Mike: Nobody wants to use RCS anymore. I thought that was the latest and greatest.

Chris: Now the latest and greatest is the AS/400 file system where it does the version controls for you.

Mike: Thank you. Thank you very much. I knew it. I’m going to go run an upgrade.

Matty: So a question from our listeners. I’m going to butcher this. Matteo Safranco says, how many of you were introduced to the notion of an admin or a sysop from the BBS days? So let’s really blast from the past here.

Chris: Oh yeah.

Trevor: Yeah.

Matty: Who wants to explain to Trevor what a BBS was?

Trevor: [00:34:22] I know what it is. I didn’t use it.

Matty: I know what it is. It’s like, I had a history class. Yeah.

Trevor: Yeah.

Mike: It’s about how young Trevor is. So who wants to field the BBS?

Chris: Interesting story with the BBSs is I downloaded my first copy of Linux through a BBS. It was in ’93. I downloaded over my 9,600 bits per second modem. Over a week, I could download one floppy every day, every night, overnight while it came down. But once I had that, I actually then had the tools to connect directly to the internet because the BBS that I was on had a slipdoor. So we could go through the slipdoor, bring up the slip client, and behold, an IP address on the machine. And the world was the most awesome place back in those days.

Mike: So did you ever run your own BBS?

Chris: I did try and run my own BBS for a little while, and then I actually went and started doing work for one of the local BBSes down the road who became one of the big ISPs around. And when they built their new offices, they had their modem banks in a glass room right by reception, so you could walk in, you could see all the lights. It was fantastic.

Mike: [00:35:37] Those were the days. Yeah, I ran a BBS back probably in ’92, ’93, and I could only run it at night because we only had one phone line. And so it would be live from, I think, about 10 PM till 6 AM. Otherwise my mother would be upset that I was running up the phone bill. But this was before Linux. This was— I was running it under an MS-DOS system with a QEMM to split memory management so that way I was able to run this BBS process in the background and still do other stuff in the foreground. Like, I remember kind of being amazed at our Golden Mail kind of network route where it took 3 days for somebody to get a response, to get your email, and it was so exciting when they did because, you know, that was faster than snail mail. Now, If it doesn’t happen in 10 seconds or less, oh my God.

Chris: Yeah, yeah, I had a fiber internet address, but the joy was the same.

Matty: [00:36:40] Oh yeah, we had like, yeah, my college, like we didn’t have a direct internet connection. So it was like UUCP, right? It would like dial up and do UUCP to like pull mail down from somewhere else, like several times a day or whatever. And you just— the deluge of email all at once.

Brian: Did you guys have BitNet and JNet too, which were like other university-only inter-mainframe connections? Because that’s what Kent and I think CMU had BitNet addresses and some other schools in Ohio had those addresses too. Didn’t take quite that long, but yeah, for me, I was a user and abuser of BBSs. I didn’t really ever try to admin one. And I got lucky because in Ohio, Case Western Reserve set up one of the first Freenets, which was supposed to be a whole community thing around that, and it was all text-based, but it came out kind of too late. The internet started to take off right about the time it came out, so it kind of got crushed by the internet.

Mike: So Brian, I’m curious, you said that you were a user and abuser of BBSs. Did you ever get caught by an admin? Did you ever have your account locked out or get a nasty message?

Brian: [00:37:48] I became friends with a lot of admins because of that, yes, they would just, hey, look, you’re only supposed to be on for so long and you figured out how to get around it. And I’m like, well, you know, and then we would go out and hang out and everything would be cool, but I would be locked out for some point in time. A couple of them just didn’t like me. So yeah, they kicked me in. There were so many at one point that it didn’t really matter.

Matty: You just pop to the next.

Chris: Yeah.

Matty: One of the questions we got, so Alex Howlser at Nick’s Geek, his question to the panel was, how often do you think, fuck it all, I’m going to be a plumber or an electrician?

Mike: That’s a good question. So I probably started out with a little bit of electronics and stuff before I got into like more computing just because that was available. Like I always had a computer at home. My dad worked as a software engineer. So I was lucky in that respect. But the schoolwork and courses did not have any computing where I was, so it was more about electronics. There’s something very satisfying in working on something with your hands, seeing the physical manifestation of something work, and in this day and age of cloud computing, who knows if you’re ever going to touch hardware again if that’s what your company uses. So there’s something very cool about that, but I don’t come across the desire to change professions because I think that overall I’m pretty lucky where I am in this field. I get to play with some of the underlying technologies of everything, and an electrician or a plumber feels so constrained compared to what I get to play with on a daily basis.

Chris: [00:39:30] Yeah, I enjoy the electronic side of things, and thankfully at least I still look after a lot of metal, and so I get to, I get involved with with a lot of the electrical stuff. It was actually about 3 months ago, our data center guys came to us and said, yeah, we just realized on one of the PDUs on your cabs that you’re running at about 115% of the rating on that thing. So, it was 3-phase coming into the cab, and so we had to add an extra phase and rebalance our systems on the fly. I still enjoy this. There’s definitely that physical manifestation. Whenever I’ve At least when I was younger, when I had a full-time dev job, that’s what I missed the most. I missed going into the server room and actually, you know, having a look at sorting out the cables or figuring out where we can add capacity or what was going on. Just seeing the blinky lights makes me happy.

Matty: It’s funny, like, to watch, like, the HipChat conversations and stuff when, like, someone goes to a data center for some random reason and then just starts like, oh my God, it was so awesome to see racks again. You know, because you’re like so— and like I did this, we had a— when I was at 10M, like just as a gag, like because they’re a cloud company, right? So it was like they were doing some giveaway thing at some career fair and, you know, we’re like, do we have a— and I had like an old server and I was like, here, you can have it and see if people can guess how much it weighs. And if they are, they win a thing. And so I brought it in. It was like this G5 DL380, I think, or something like that. And I swear to God, it was like I’m freaking drooling because I’m sitting there going, oh, they’re so pretty inside. Because I mean, I still love the way appliance are designed, you know. And it was— and I swear everybody else in the office was like, seriously? I’m like, yes, I don’t understand, you’re developers.

Trevor: [00:41:13] It’s only a matter of time, you know. I think Chris was saying earlier the company had the, the servers out behind the glass so that everybody could see them blinking. It’s only a matter of time before there’s just, you know, an art installation that is you know, a replication of what servers once looked like blinking in an office lobby. You know, it’s just, you know, a half-inch facade of what once was.

Mike: Well, I mean, not for nothing, a few years back I was in London and I dropped by the Museum of Science in London and they had a whole setup of, what was it, home computers. And my first computer, the TI Texas Instruments 994A, was powered up behind that glass. And I was like, okay, now I’m a dinosaur because I’m in a museum with everybody else.

Matty: Oh no.

Mike: But considering that in the device that you have in your pocket, the iPad, the computer that you have today is so overloaded with—

Trevor: Yeah. Yeah.

Mike: [00:42:14] Okay. It’s not an iPad per se. But these have so much technology in them. Your television has more technology than the space shuttle had, right? So, you know, microcomputing and cloud computing and all of the services that you’re used to kind of touching and feeling are kind of going away in favor of something else. So, you know, at some point, what was it? Mitchell Hashimoto, the creator of HashiCorp and Vagrant and a bunch of other cool tools that are really important to modern-day development and stuff like that. He had at one point said, here I am having done all of this and I’ve never seen the inside of a data center. And to me, that was like mind-boggling. Like, oh wow, you can’t even conceive of what the rest of us have been, you know, like working in and doing.

Matty: Which you were hoping to save us from.

Mike: Exactly. Like, you don’t— oh, you don’t know. And eventually I think he did get around to seeing the internals to see how it actually works. But it was just like mind-boggling that in 5 to 10 years you may have the kind of specialization in the field of technology where the only people who see inside of data centers are people who solely work in data centers, kind of like the only people who see the inside of the sewers are the people who work in the sewers. Like it’s turning into that specialization field over time where Matt’s desire to kind of like sit there and drool over a ProLiant server is going to be foreign to more and more people.

Matty: [00:43:55] I had a really interesting thing. So, uh, see what you guys think about this. So when, when my tech screen for sysadmins, um, when I was in apartments, like my softball question was like, describe RAID levels to me.

Trevor: Right?

Matty: Tell me about the different RAID levels because I figured I wanted to start with something easy and blah, blah, blah. And I get so many people that couldn’t do it. And at first I was like, oh my God, who are these people that think they can come do this? And I was like, you know what? There are people that are really good sysadmins who live in a virtualization world and they don’t touch hardware, but they’re good at managing systems. You know, it was like, to me, I was just like, oh my God, how can you possibly, you know, not know that? And it’s like, you actually could be really good at what you do. You know, you could be a really good admin and not know that kind of hardware. And then this was like a huge conversation. I feel like, Trevor, I feel like we’ve talked about this on the show too. Have we?

Trevor: I think on some—

Matty: We’ve talked about it a lot. I think, oh, we talked about it on the Dev Show. Full circle.

Mike: You planned that one.

Matty: It’s like I did it on purpose.

Brian: [00:44:58] Yeah.

Matty: So before we move into our checkouts, I want to go around. And your best scary war story. What was your best story of when you absolutely crapped your pants as an admin? So, Mikey.

Mike: Oh, me first? Oh, I got to come up with one of the worst. So this, it’s not the worst, but it was pretty weird where we’ve got everything running in a data center that’s all managed and cooled and everything like that. And one of our core mail servers just Bombs out. It’s gone. You know, can’t go, can’t hit it. We obviously did not have a failover email server at that time just because we were transitioning. And it would always be transitioning. That way you can write it off as that’s why we weren’t planned. And so we hop in, go over to the data center and pull, you know, look at it. It’s just dark. It’s all, it’s not, it’s one of the ones that’s at the bottom of the rack. You know, it’s a, 2U, I don’t know, what is that, 5 inches now size server, pull it out and pop it open and there’s a dead mouse inside there that had apparently urinated on the motherboard and shorted it all out and that’s why we lost it. And it was like, oh well, not a quick fix here. Gotta order a new motherboard and power supply and Get all that replaced. Not a fun thing to find.

Matty: [00:46:34] Chris Treat.

Chris: Yeah, my one’s actually also email related. I think it was— it was about ’97. I was an email administrator for a big multinational based in Johannesburg. I looked after an MS Mail 3.5 system running on NT 3.51 and had a couple hundred Well, actually, we had about maybe— I think it was about 20 mail servers spread. One was as far away as in Hong Kong on the back of an analog leased line. All the email, internet email, came through a little SparkStation Classic that we had sitting on the back of our 64K line that everyone used for the internet. That was down, and so internet email wasn’t working. I was working in the head office, and a whole lot of people had been bugging me for, when’s the email coming back? When’s the email coming back? It’s like we’re waiting for the telco to come and bring the line back up. Eventually, I’d gotten fed up with people coming to ask me, so I decided to send an email to everyone saying, listen, there’s a problem with internet email right now. We’re waiting for the carrier to come and fix the line. It’ll be back. Now, we had about 6,500 mailboxes in the system, so I just went MS Mail and just— I had kind of the very first— I think it was probably one of the precursors to Outlook. It wasn’t the full-blown MS Mail client. It was one of the newer ones. I just said, right, everyone in the address book, To address, click Send, go. Then maybe about half an hour later, the IT director comes screaming into the room saying, who let this fucking cowboy loose on my network? Screaming and shouting because what had actually happened was that there was a bug in the older MS Mail mail clients where if the To header was bigger than 64K, the mail client would just segfault. What had happened now is this email was now in everyone’s mailboxes, and pretty much everyone throughout the company’s mail clients started to segfault and crash on them. They had no email at all. It was basically me and one of the guys on my team basically had a 36-hour day basically putting together a batch script to go through each and every mailbox, find this message ID, and delete it from the mailbox so that everyone could get a hold of their emails again. That was a very educational moment in bad software and sysadmin.

Matty: [00:48:50] All right, and Brian?

Brian: I’m gonna continue the trend. I don’t know why, it seems like we all started doing email stuff, but my first official job as an email admin, I got pulled over from the help desk because I actually knew Unix and understood Solaris and could do all that stuff. The company that I worked for had just been purchased by FedEx. The 2 email admins who convinced me that, hey, come over, we’re gonna teach you Lotus Notes and we’ll teach you how to do all this sysadmin stuff and it’ll be great. Both of them just do this real big sales pitch on me. I go, okay, cool, let’s go. The day I start, they both put in their 2 weeks notice. Oh, the worst part about that was, oh yeah, by the way, we’ve already prepped you for the first upgrade that you’re gonna have to do. You kind of need to do it soon because there’s some issues with the version of Notes we’re on. You need to do this upgrade as soon as possible. Okay, and you’re sure it’s gonna work? Yeah, yeah, it’s been in dev for, for months now. It’ll be fine. It’ll be totally fine. Okay, so I do the release. Fine. Saturday night, Sunday, no problems. Monday morning, about the 101st person connecting to email would crash the server. So the server would recover, come back up, and Hit 100 people, crash, recover, come back up, 100 people, crash. I was on the phone for 3 days with IBM trying to get this to work correctly and eventually just had to back it out. But 3 days of talking to, like, here I am, was I like 21, having to explain to the president of 3 different divisions who have no idea how the internet or email works. Why they can’t get to their email and why they can’t connect to it. And, you know, various yelling and screaming at me, to which I looked at him and went, you can yell all you want, not going to get it fixed any faster. There’s nothing I can do. So, uh, yeah, that was my worst. It taught me to test the stuff myself. Let’s put it that way. And do load tests.

Trevor: [00:50:48] Matt, what about you?

Matty: Okay. So I think I’m going to do this kind of quick. This is a, you know, kind of crap your pants because of my own fault thing. So when I was really just early starting out, So I was working for this company, this mail-order computer retailer that at the time was a competitor to CDW, because believe it or not, there was a time when there were competitors to CDW. And so our main like kind of financial order system was this little, you know, System 5 box and running some ridiculous custom software that had been built for us and blah, blah, blah, that I didn’t really understand how it worked. And I was also like all of probably, yeah, I don’t know, very green in my career and my reach was definitely exceeding my grasp when it came to my knowledge. And so we had this issue where it was at the end of the day and some files had appeared to have gotten corrupted. And so I was like, well, I had my backup from last night. I’m like, let me, I want to see if those backups are there. And so I tried to run, you know, whatever the System V command was to basically index the tape, but then it appeared that it was starting to restore. Like, and so what? So I sat there thinking that I had basically blown away, but of course it was taking hours to do. This is like after hours, so now it’s just me and the owner of the company, and I’m sitting there going, thinking that it’s this countdown that’s going to result in basically like, hey, we lost $50,000 of orders, are totally gone. And it basically took like hours and hours and hours to then tell me he couldn’t read the tape. But I was sweating like a crazy person during that time. So that was one of the scarier ones. Then there’s, of course, I think we’ve probably all done the power off the wrong box. Like I was at an insurance company here in the, well, not here, not in Chicago, but in the Chicagoland area that rhymes with mall fate. And one of my jobs was just rebuilding boxes in this data center. And you kind of would go there and since you were building it from scratch, you’d just be like, boom, shut it down. So I went in kind of boom, shut it down. Then I go on the KVM and I’m like, wait a minute, Wrong box. Don’t know what this thing does. I’m just going to leave the data center and kind of watch from the outside and see if anybody comes and freaks out. And nobody did. Yes, you all should trust me with your systems. So that said, let’s move into our checkouts. So Mike, what do you got for us?

Mike: [00:53:06] The first thing that I think everybody should check out is very relevant to today’s show. It’s opschool.org. School is an open-source curriculum for anybody who wants to learn anything about the practices of systems administration and operations in general. I’m currently the chief editor because nobody else wants to do it. And it’s open-source, it’s on GitHub. We have a bunch of people who contribute when they can. Contribute as you can. Read it, fix my typos, and and help us all out. The second one is codeschool.com. They’re not anybody I know personally, but I’ve checked out their site, I’ve used it before. They’ve got a few decent courses on just introductory to development and some other kind of fields. They’re very fun and helpful to kind of get your feet wet if you want to try a new language or a new framework. They’re pretty, pretty useful. They have some paid courses as well. That I haven’t tried yet, but I’m pretty sure are better than the free stuff. And the final is, everybody’s always talking about beer. I don’t. I drink wine. So check out the Benziger Family Winery. They’re from Sonoma, California. They’re really good. They’re at benziger.com. I’m a customer. I love it.

Matty: [00:54:27] Chris?

Chris: Yeah, I got something old, something new. Something old to look at, which I’ve recently been getting a lot of mileage out of, is the IMFile plugin module. For rsyslog. Basically allows— gets rsyslog to tail a file for you and spit it out to a remote syslog server. So I’m getting a lot of mileage out of that, just tailing log files and spitting that into Logstash for me. And something new is io_snoop for Linux. If you’ve ever been sitting on a Linux box with a fairly new kernel and wonder what processes are are writing what to which file systems, which block devices. I/O Snoop is a piece of beautiful Bash and awk which trolls the perf events and the debug file system on the Linux kernel, which gives you all those answers as well as service times for each I/O operation. They are very, very good. Don’t confuse it with— originally Brendan Gregg wrote it for Solaris using dtrace, but he did a port of it to Linux recently, which is fantastic.

Matty: [00:55:32] Awesome. Brian?

Brian: All right, I got, um, I have 3 of them. A site that I thought of while Mike was talking was, uh, TrueAbility. It’s a cool little site that, uh, they run little contests. You can go out and test your Linux skills and see if you can, can beat other people playing it. They usually have some sort of giveaway with it. My second item is I love the Pogo Plug. This is probably one of the most hackable and definitely one of the cheapest Linux boxes you can get your hands on. It’s not very powerful. It’s only got 128 megs of RAM and an ARM chip, but they’re like $10. I put a link in the show notes. It takes about 20 minutes to hack it. And then you’ve got actually a cool little Linux box that you can play with. It’s got USB and an SD card slot, so you can put some extra storage in there or you can do whatever you want with it. I’ve been playing with those. Those have been a lot of fun. And then my last one is Ohio Linux Fest is coming up in October and I will be speaking at Ohio Linux Fest. So I wanted to give it a little plug and say that registration for Ohio Linux Fest is open now. So you can go out and we had a little talk about this on the linuxinstalled.net podcast last night. The registration process is a little wonky, so bear with them. It’s all a volunteer organization, but if you can figure out how to do it, even if you can’t, just show up on the day of the thing. It’s only $5 to get in. If you don’t pre-register. So that’s cool.

Matty: [00:56:54] Trevor?

Trevor: Well, I only got one tech-related thing because I’ve been on vacation, but there’s this app for iPhone which— so because it’s for iPhone, I will probably never use it, but it sounds cool. It’s called Human, and it’s—

Matty: Oh yes, it’s awesome!

Trevor: Oh, you’re using it?

Matty: Yes.

Trevor: Apparently it lets you kind of do semantic lookups of your contacts so you can kind of associate like locations or nicknames and that kind of stuff, if I understand it correctly.

Matty: Yeah, it’s like a replacement for your phone app, so to speak. I mean, it dials through the phone app, but like, right, because you use that instead for like direct calling. It’s pretty cool. I’m getting used to it.

Trevor: But other than that, I kind of spent the past week and a half going through historic New England, going through Salem, Mystic Seaport, the Gilbert Sullivan House— or not Gilbert Sullivan. I keep saying Gilbert Sullivan. Gilbert Stuart. He’s the gentleman who painted George Washington that’s on the $1 bill. Probably the coolest thing was technology related was I went to the USS Nautilus, which was our first American nuclear sub. It’s sitting in the harbor in Groton, Connecticut, and you can tour the ship. It’s really neat. Lots of old equipment on there. That’s pretty much it for me. Matt?

Matty: [00:58:09] Yes. Okay. So I have a couple real quick. One is, it’s called Lal Root. So if you go to lolroot.ca, it is a certificate authority for self-signed certs. So it’s totally a groaner joke thing that’s completely ridiculous and made me laugh a lot. So go check it out. Also, in honor of the show, this is an old school thing, but I bet there’s a lot of people who listen who have never seen this before. And those of you who have, it’s always, you know, good for a laugh. And that is the Bastard Operator from Hell. So I’ll put a link in the show notes, but if you can’t wait, it’s bofh.ntk.net/bofh. And it was out on, uh, just, uh, back on Alistair back in the day. And it’s the story of Simon, the bastard computer operator from hell, and how he tortures his users. And to this day, I read it and it literally— I mean, I don’t laugh out loud at things by myself usually, and I laugh. Like crazy, which might be a reflection on me as a human being. Also, as a checkout, is that registration is open for the Chef Community Summit, which is in October 2nd and 3rd, I think. But if you want to know for sure, go to arresteddevops.com/chefcommunity. And I have a discount code, I think, which I’ll remember in a minute.

Trevor: [00:59:28] But in the meantime, while I’m looking for it, as the, uh, the, the opposite of what Matt’s saying, there’s, there’s also a subreddit, Tales from Tech Support, which tells the horror stories from our side of the thing, of the game.

Matty: Oh, actually, well, you know, one other one just along those lines, if we’re just doing this, I should also— is Chronicles of George. I will post a link to that. You guys should all read Chronicles of George. But while I’m looking for this, Chris, you said you had another checkout, another thing.

Chris: Yeah, well done on the BOFH thing. I still refer to guys that I’m training as my PFYs, my pimply-faced youths. Yeah, if there’s any developers out there in the Chicagoland area who want to learn more about the ops side of things, one of our internal teams here at DRW who’s looking for a developer to help some of our sysadmin folks with improving our monitoring and systems like that, we’re looking for someone good. So we’ll have a link to the job ad in the, in the show notes.

Trevor: [01:00:29] Cool. And I don’t know if Matt found—

Matty: I didn’t. I’m this— I, oh my God, I’m so bad about this. So I will tweet it. So if you, so make sure you watch our Twitter among Matt’s many tweets. No, no. Arrested DevOps will tweet it, not me. So follow Arrested DevOps, not we, whatever. Anyway, so let’s wrap it up. Reminder, we have a newsletter. That’s also a reminder for me that we have a newsletter. You can subscribe by going to arresteddevops.com/bananastand. It is the best way to know about upcoming podcast episodes and occasionally cool news with DevOps when I remember to send them. But I definitely will be making a point to send more updates, not like a ton that will spam you. So go sign up for our thing.

Trevor: Also, don’t forget it’s conference season. We’d like to remind you about FlowCon. FlowCon brings together technologists and industry leaders passionate about innovation. Through lean product development, continuous delivery, lean UX, and DevOps. They’ll be exploring the role of culture, technology, and design in growing organizations that thrive in an environment of continual change. They’ll provide inspiring and actionable information for key decision makers responsible for products and services that depend on software. The full program includes speakers from Google, Netflix, Heroku, Nordstrom, SoundCloud, Macy’s, HP, Joyent, ThoughtWorks, and IBM. The second day features an open space unconference and workshops from Don Reinersten, Mary and Tom Poppendieck, Whitney Johnson, and Sarah B. Nelson. Use the special promo code ArrestedDevOps50 for $50 off at arresteddevops.com/flocon. That code was ArrestedDevOps50 and the website was arresteddevops.com/flocon.

Matty: [01:02:19] Also, the O’Reilly Velocity conference is happening September 15th through 17th in New York City, and it’s the place where DevOps, WebOps, and performance professionals from Fortune 500 to exciting startups gather for a legendary learning and networking experience that explores why a faster, stronger web is no longer an option but a necessity. Hear from the best speakers in the industry who will delve into topics ranging from hardcore math and stats to monitoring, clustering, analytics, and organizational culture. Attend Velocity and you’ll view your work, the technologies you use, and your organization in completely new ways. Use code ARDEV20 to save 20% at arresteddevops.com/velocitynyc.

Trevor: And as Matt and I are both forgetful, I should also mention that Flowcon is September 3rd and 4th, so it’s coming up pretty quick.

Matty: So, oh my God, do we not actually say when it is? Into my spiel?

Trevor: Nope.

Matty: Well, now you know.

Trevor: Yes. And so thanks again to our sponsors PagerDuty and Datadog, and to our loyal listeners. If you enjoy Arrested DevOps, we’d appreciate it if you go to arresteddevops.com/itunes and we’ll listen to you in the iTunes Store. Uh, we’ve, we’ve not— we’ve only had a couple, I think. Matt, have we had recent reviews? You’re on iTunes.

Matty: [01:03:38] We haven’t had anything recently, so we could use some more. If you write a good one, we’ll read our favorite ones on episodes, or hell, any of them at this point.

Trevor: We just want some love, guys. Come on. Thanks to Mike, Chris, and Brian for joining us. Thanks, guys. It was great having you on the show.

Chris: Thank you, guys.

Brian: Thanks for having me.

Trevor: Absolutely. Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter.

Matty: We are always happy to get your input, ideas, or feedback at shows@arresteddevops.com. So if you have ideas for upcoming shows, let us know, or just thoughts, two cents. And this is your lucky week because— lucky or lucky week, I don’t know— but we’re gonna have a whole nother episode next week because calendars are hard. So join us next week at 7 PM Central on Thursday, August 28th for Dev to Ops. So we’re going to be talking about people who made the transition in their career from being a developer to moving into more of an operations role. The URL for the show is arresteddevops.com/19. So as always, I am Matt, @MattStratton, and I’m Trevor, @TrevorGHess. We are Arrested DevOps, and remember, there’s always DevOps in the banana stand.

Trevor: [01:04:50] And Matt’s going to burn down the banana stand while he’s out in the, in the OC. Yeah.