Matty: [00:00:06] Welcome to Arrested DevOps episode 36, Dr. BOFH, or How I Learned to Stop Worrying and Love the DevOps. I’m your co-host, Matt Stratton, @MattStratton on Twitter.
Bridget: And I’m your co-host, Bridget Kromhout, @bridgetkromhout on Twitter.
Matty: So Trevor’s not joining us tonight because he’s camping in the rain, so hahaha, sucks to be him. But it doesn’t suck to be 10th Magnitude, who’s one of our sponsors, and they’re a cloud services company that figures if you’re listening to this podcast, you must be pretty cool and probably aren’t camping in the rain with Trevor. You can find out about joining their cloud services team with Trevor at arresteddevops.com/10thmagnitude.
Bridget: This episode is also sponsored by PagerDuty. PagerDuty eliminates the noise, chaos, and manual processes across the entire incident lifecycle. To decrease resolution time. PagerDuty is trusted by companies like Etsy, Nike, and GitHub. To sign up for a free 14-day trial, visit arresteddevops.com/pagerduty.
Matty: This podcast is also brought to you by Loggly. Loggly is an agent-free logging solution that can save your site and your sanity. Whether you’re troubleshooting site issues or monitoring external services, Loggly allows you to easily discover root causes for specific errors while managing and centralizing your logs. Get started with all features included for free at arresteddevops.com/loggly. All right.
Bridget: [00:01:31] So tonight we’re talking to some reformed BOFHs, AKA Bastard Operators from Hell, about how our experiences have changed over time in the industry. Our first guest is friend of the podcast, Chris Read. How’s it going, Chris?
Chris: Hey guys. Good to be back.
Matty: So we also have Kevin Hubbard joining us for the first time. So welcome, Kevin. Can you tell us a little bit about your background?
Kevin: Hey guys, thanks for having me. Kevin Hubbard here. I’m currently the DevOps engineer for B-Cycle at Trek Bicycle Corporation. I have about 15 years as a former sysadmin.
Bridget: And also joining us is Yvo van Doorn. Please tell us about yourself.
Yvo: Thanks, Bridget. So, my name is Yvo van Doorn, and I’m also a reformed bastard operator from hell.
Matty: Previously worked at classmates.com and McGraw-Hill Corporation, where I definitely practiced those I know we’re obviously all familiar with Simon, aka the Bastard Operator from Hell, but first of all, listeners, if you’re not, you can go to arresteddevops.com/bofh and there’s a link to the BOFH archives. But really, kind of the history of it was, it was, I believe it was, you know, kind of some stories traded around on Usenet in the mid-’90s that were supposedly the diary of a computer operator named Simon who lived with no purpose other than to terrorize the users of his systems and had a very antagonized— antagonistic relationship. And to be honest, that was— it really resonated for a lot of reasons, I feel like, for most sysops and sysadmins at that time. Makes me always think about— there’s a line in Clerks which is, this job would be great if it wasn’t for the fucking customers. And I kind of feel like—
Bridget: [00:03:18] how great would this job be if it weren’t for all the users?
Matty: Exactly. Like these people, like touching my stuff and messing up my stuff. So I’d like to kind of hear some, some stories or experiences from the panel about, you know, again, we’ve all lived in this type of relationship in our career. What was it like when, when we were starting out and you kind of had these barriers?
Yvo: I don’t know.
Kevin: To me, it was sort of operating in a scarcity model. You had limited resources, and my background was more hardware-based, provisioning storage, supporting the hypervisor. It seemed like anytime there was a new ask for a new application, I immediately went to, okay, how am I going to go back to ask for the capacity to run this? I would get so frustrated, I boiled it down to, how are we going to support this? And that was like my standard line anytime someone brought up something new, which, you know, I always wish that I could have been, you know, or what I would tell people is I wish I had like a trophy to give you for a good idea because I just can’t get that off the ground with the resources that we have. You know, it just kind of, I would say, was not a fun way to operate, but I thought that was the most realistic view.
Chris: [00:04:35] My introduction to it came from Back when I was— my last couple of years of high school and back at university, I was the system administrator for— at least when I was in high school, for the school systems. Then in university as well, just helping out with adminning on the computer science side of things. And to me initially, it was just seeing what damage ignorance of a system can do, how people trying to do something, how badly they could affect shared resources and what the knock-on effect of that can do. And most of the time, it wasn’t malicious. It was due to ignorance. But it kind of built up this mental attitude of all users are just out there to break things. We need to constrain them as much as possible because when things break, you’re the one that gets shouted at.
Bridget: And also remember, at the time, the systems didn’t have as many safeguards built into them. I mean, remember how terrible the locking was in the mail systems? Yeah, you had to restore people’s mailboxes from backup all the time, but they weren’t bad people or stupid, and that’s why they tanked their mailbox. It was because the tools were not actually all that helpful.
Chris: [00:05:46] That’s because you were using the new shiny inbox. You should have just left emails as individual files as God intended.
Yvo: Until the file system filled up.
Chris: Exactly. Or you run out of iNotes.
Matty: I remember years ago I had a good friend of mine who’s on the dev side of the house, and I would get emails or calls from him, and it was in different companies, you know, we working at different companies, and he’d basically be asking me how he could get around the things the sysadmins put in his way. He’d be like, well, I really need to do this thing, and so how can I get around this thing? And I was like, dude, I can’t do that. I can’t sell out my I’m like, I don’t even know your sysadmins, but those are my brothers, man. I’m like, I’m not going to screw them over by telling you how, even though I totally could. And that, to me, resonates to the way that I think a lot of us thought about it, which was— and again, to Chris’s point and to Bridget’s point, not necessarily malicious, but there was this belief that it was like, just devs are stupid, right? Or whoever, and whoever your users are, whether you’re supporting an app dev team or if they’re end users, if you’re supporting, you know, backend infrastructure stuff, all they’re gonna do is, is break things because they don’t, they don’t care, right? Like, nobody cares about the systems like a sysadmin does because they’re ours, right?
Bridget: [00:07:10] Now I keep seeing smiles from Yvo, so tell us about your background. Which of these camps were you in?
Yvo: So I was definitely in the sysadmin camp and Our devs, while they didn’t care, they were also incentivized not to care. So the place I came from, they had deadlines, and it was very— I mean, let’s talk about software methodologies or software development methodologies. It was very much waterfall, and they had a deadline on, say, a Thursday, because it was like every 3 or 4 weeks and sometimes ended up being on a Thursday. And a developer, they got actually paid on— and this was a The social network that never became good. Because they’re still around, I don’t want to refer to them by name, but this company incentivized their developers to ship code that was intentionally bad. And I’ve had some nightmare stories and some nightmare late nights trying to figure out— and one in particular that I can remember is We launched this new yearbook-type app where someone could upload their photo, and the developer shipped code to the sysadmin team or the system engineering team, and it didn’t include any logic for when an upload would fail. That doesn’t seem bad, except for that this also went on a brand new NetApp that was set up, and it had restrictive permissions on it because I’m a sysadmin and I care about locking down permissions to the the operating user of that app, which is at that point was a resin app that was running as a resin user, which was under a JVM. And, it did not have write permissions on certain directories on that photo— that new NAS. So, it could create directories if certain things happened in the right order. But, if that— if for some reason it didn’t happen, the photo would say it was uploaded successfully to the user, and it would also put it in the database as like it would link to the photo that it was uploaded successfully where it was at, but there was no photo.
Matty: [00:09:25] I think we had the same code at Apartments.com because we had the exact same problem with our for rent by owner where the same thing where there was a photo upload capability and then you would end up with all this junk in the database that would end up with like 404 images on the site because it would like generate the URL that pointed to no actual image file.
Yvo: Yeah, yeah, so, so that was one of my like Bastard Operator from Hell moments. It was just like, fucking developers.
Bridget: But you made a really, you made a really good point. It’s okay, we’ve made our peace with the explicit tag. Yeah, um, you made a really good point about, uh, misaligned incentives. I’m wondering if the rest of our panel wants to address what they’ve seen with incentives that may or may not have been diametrically opposed, or at least not aligned very well, and how that made the sysadmins versus the rest of the organization, or even users or devs or whatever, behave.
Matty: [00:10:26] So I can tell this is, this is a fun story I like to tell. Smarter folks than I have said this before. I mentioned in the show notes that Tom Limoncelli talks about this and Chris already corrected me in the green room that actually someone else said it first. But again, it’s just this idea that, you know, in development, commonly you’re incentivized to deliver features, and in ops, you’re incented to provide stability. And what provides stability is things not changing, right? Like, changes make shit break. I think back to a colleague of mine at a firm, and basically, literally, her bonus was tied to a percent— based on a percentage of uptime of the site. So this was downtime on that site, was cash money out of her pockets. And do you think that this person would ever let anybody touch anything? Hell no. And could you blame her?
Bridget: That sounds like a recipe for never upgrading for vulnerabilities.
Matty: Yeah, it was totally— it ain’t— it literally, it ain’t broke, don’t you dare try to fix it.
Kevin: [00:11:27] Yeah, I guess along similar lines, you know, my experience with developers or that, you know, maybe it was a small group that really had some sort of territorial you know, sort of mindset around the application. And I don’t think the sort of viewpoint that users are losers is just for sysadmins. I think that developers often have the viewpoint that, you know, users don’t know what they’re asking for. So I often found myself like sort of siding with the users on like, okay, this is obviously a bug, this is not working the way it should, and just running up against a wall against developers. It’s like, well, and you know, they had stuff in their pipeline, typically, you know, new features or a new release rewrite, a complete rewrite, and shit that was just broke was left to the sysadmin to either restart services or figure out some other way to sort of explain away in a very lengthy phone call of why it works this way.
Matty: Yeah, do the workaround, right? I mean, this is— let’s, let’s talk about the loveliness of ITIL, right? What’s one of the things ITIL gave us? Literally the word workaround, right? That’s like a thing in ITIL is that there’s a workaround, which means we’re not gonna fix it. But somebody is going to have to do some work. I remember at Apartments, and again, this could happen anywhere, so I’m not saying bad things about the company. It’s more about maybe some of the developers that are still good friends of mine, but I can talk shit about them. We had a scenario where there was a memory leak in the web app, in this .NET web app, that basically required at least several times a day actually having to recycle the app pool. And this went on for weeks and weeks, and it was a matter of, well, it’s no big deal, sysadmin will do it. And they’re like, for real?
Yvo: [00:13:13] Right.
Matty: You know, it’s like, no, we can’t. We’re not going to try to troubleshoot this because we have to ship features. Never mind the fact that we are constantly, you know, again, it’s that common, like, things aren’t being operationalized or being thought about in that way. And I think that’s a big thing that caused that divide, right? Which was not, you know, so you do end up with that, like, that feeling like nobody gets it, man, right? And I’ve worked with lots of sysadmins who, again, have this personal attachment to their stuff. And I remember writing— this is the thing, so let me tell you, this might be the way that my thinking has changed. So when I was a manager, I remember a review that I wrote for one of my sysadmins over and over again, who worked for me for years, and his review always said, so-and-so takes the uptime of our website personally. And I thought that was a grand compliment. That’s not actually a thing to drive, right? Because then you’re talking about, like, again, even though it wasn’t a monetary incentive, he had a personal investment in that kind of thing. So that’s going to echo through how you approach allowing change to your systems. And I think it comes out subtly too, right? Like, you sort of project this, like, cowboy developer thing on people just by virtue of the— because they’re touching your stuff, whether they’re cowboys or not.
Bridget: [00:14:39] There’s so many valuable things that you’re saying there, Stratton, and I want to hear more from our panel about this too, but I just, I want to point out too that in reality, there’s a lot of perception of, like, Oh no, the whomever will touch our stuff and it’ll go wrong. But reality, anyone who touches anything, it can go wrong. Hey, AWS is having AWS-like things happen. It can go wrong. Totally bizarre interactions between some sort of kernel bug or a misconfiguration in your VPC that leads to ARP cache problems. Not that that— it’s oddly specific because it happened to us recently. There’s all sorts of things that can go wrong that are often not a specific person’s action.
Matty: And the irony of it is the same sysadmins who will love to talk about like how developers are going to fuck up their shit love to sit and tell war stories with each other about when they fucked shit up, right? Chris was on, we did a whole episode about this, right?
Bridget: So we’re complete hypocrites. Yeah, reality, we all fuck shit up.
Chris: [00:15:43] But with the memory leak one that Matt was talking about, it reminded me of kind of one of the first times where I actually started to see the value of educating users. And it was, again, it was a web app. It was a big ecommerce.com. And it was, again, leaking. This time it was just leaking sockets. We had— it was a Java web app. And this thing was just leaking, leaking sockets. Every 3 hours, we had to restart the JVMs and just cycle them through the cluster one by one to keep uptime. Eventually, I happened to be the— we just migrated from ClearCase to CVS, thankfully, and so I was happy to be rid of ClearCase. I decided to go and dig through the code and see what this guy was doing. Basically, he’d create sockets and just leave them there. He’d never close them, and he’d never clean them up. That’s why they were leaking. Eventually, I went over and spoke to the developer, and it’s like, why are you not closing your sockets? He’s like, garbage collection handles that for me. Then the penny dropped. It’s like, okay, so he thought he was right. He wasn’t being malicious. He wasn’t trying to be a pain. It was just he didn’t know any better. This was also at the stage where I had a 6-foot pine pole that I carried around, which the developers titled the Clue Stick, because whenever I walked around with the Clue Stick, Then, you know, me walking towards where the developers carry the clue stick, everyone was nervous. Everyone would look up. It was like, oh dear, whodunit, whodunit. And, you know, it reached a stage, but then I realized that, you know, we need to have these conversations. We need to be talking with the developers and saying, hey, do you realize that this thing you’re doing is causing this massive impact for everyone else?
Matty: [00:17:28] So basically, a 6-foot-long pine stick of clue is not DevOps.
Chris: It is not DevOps. It is the conversation that is the DevOps.
Bridget: Nope. So how is this DevOps compliant?
Matty: So how has this changed though, right? So, so things are— well, first of all, that’s going to be kind of my question is, is has it changed in your experience? I mean, Chris was, was, was going right to that, so I want to hear a little bit more about how do you look at— because we’re all supposedly reformed BOFHs here. So how did, uh, how did you make that transition? How did you, you know, did 12-step program? Did they put you into Simon rehab in the desert or what?
Chris: So me, at this stage, I’d spent a good 3, 4 years each side either being a developer or being a sysadmin or being a network admin. I would flip-flop between the two because whilst being elbows deep in cables in a data center, I would miss writing code. While waiting for my code to compile, I’d I’d missed kernel optimizations in the hardware. And so it was realizing that I could have these conversations with developers and that I would actually— their reaction to me would be better if I went without the clue stick and had the conversations and started speaking to them about, so do you realize that some feature in your code is doing X and Y. And probably to a bigger extent, it really took off for me when I joined ThoughtWorks about 10 years ago, where, you know, the very first project I was on was with Jez Humble and Dan North, and that’s the one that we kind of wrote the initial continuous integration pipeline paper that came out of, and eventually that’s the project that Jez’s book came out of. Because then there was 8 of us sitting in a room on this project, and our whole purpose was to get code written by 2 different development teams, get it out into production onto the client systems. And it was only through, you know, we had the relationship with our developers, but we didn’t have the relationships with the client sysadmins. And it was only through getting them involved in all the questions that I really decided You know, really realized that it needs to go both ways. It’s not just sysadmins telling developers, hey, do you realize your code is doing X, Y, and Z? But it’s also, you know, from being the developer’s advocate going to the sysadmin saying, hey, what’s our network config look like? What’s our hardware look like? What are our options around X, Y, and Z? And building up the relationship and being able to test when we find Again, one of the first deployments we did on that project was one of our first prod deployments out to a cluster. Everyone was writing state to local file system, but then as they didn’t have sticky bits on the load balancers, so as they flipped over, it carried on. It’s just this keep on going, keep on trying.
Kevin: [00:20:44] I totally agree with Chris. To the newbie here, it just seems like DevOps has, you know, sort of formalized the relationship between sysadmins and developers, and I’m sure you guys have covered that before, but to me, being new to the role, it also just seems like a much more natural, you know, sort of iterative process working closely with devs. I think a lot of what we’re talking about is this sort of dichotomy of users, and sort of I’ve traded up, you know, the end user with people that are actually classically trained IT pros. And so the conversations that we’re having now that the relationship is more formal is just— I don’t know, to me it just seems much more valuable. We’re working side by side, we’re sort of kicking ideas back and forth. So to me it feels like there’s less of the time spent— are you asking, you know, I have to do the math, are you asking for what you think you’re asking for? You know, that was my sort of approach to users, end users. But now that my customer base has changed and now my customers are devs, you know, we kind of cut to the chase in a sense, and we really are starting to have a meaningful conversation around what’s the best way forward, what’s the smart decision, and we’re working— I don’t know, anything else I say is going to sound really sort of kumbaya and cheesy, but I think you guys catch my drift.
Bridget: [00:22:06] Hey, we’re all about the kumbaya and cheesy here.
Kevin: Nice.
Bridget: I was actually going to just say, like, that back-and-forth collaboration, I think, is super valuable. And also, like Chris said, that it’s not so much just Chris is the one dispensing wisdom from the mount, you know, like bringing down stone tablets to hand to ignorant folk or whatever. I mean, that’s not the way it works anymore because of the kind of people that Kevin’s talking about who we’re collaborating with. I mean, like, at Drama Fever, we had some horrible cash machine problems, and Art, one of our developers, is the one who put a lot of effort into solving a lot of them. Yeah, I mean, maybe that’s something that in some organizations would have just been the, you know, role of ops, but we actually have a pretty collaborative relationship and we had a developer who had the skills and the ability and dug deep into a problem. It’s like, I like having that kind of collaboration on a team.
Chris: I was just going to say, Bridget, have you ever seen handing down stone tablets on high work at all?
Bridget: [00:23:10] I don’t know. What do you think, Yvo?
Yvo: I have not. But so one of my— so I’m going to completely cut back to why I went to the anti-BOF camp is they actually fired our PMO organization. And that made things a lot better. It’s actually interesting that they were sort of like they existed. This is at my last job. They existed to sheerly separate dev from ops, and we were going through a downturn, and they got rid of the project management team. I’m sorry, project managers out there. We all love you, but in this case, you didn’t do a good thing. The moment they got rid of them, they actually had to invite someone from system engineering into the development meetings because It was in the past, it was someone from project management that would communicate the needs and the wants from development for the next release. But now it was actually someone from IT operations that sat in the room and it became some— it immediately became more collaborative. And immediately we saw and foresaw things that were happening early on development of a sprint because we were actually using agile development, but we had a PMO organization. So it was like a hybrid agile development because it’s not truly agile, we actually foresaw things coming down from development that we could actually support them early on, and we knew exactly what it would take to actually undertake that initiative. At that point, I saw the light. Without reading anything about DevOps, without— there was stuff about DevOps already published at that point, because DevOps is 5 years old today, right?
Bridget: [00:24:57] It’s about 5 and a half years old at this point.
Yvo: This is well into DevOps actually being a thing, and I actually remember reading about DevOps at Netflix, and I’m like, well, that’s never going to apply to me because that’s Netflix.
Matty: I think we’ve all had that. Like, I’ve told the story before about the first time I heard Jez on DevOps Cafe talk about continuous delivery, and I’m literally driving down the Eisenhower Expressway yelling at my radio, that, that would never work at Apartments.com, and then he drops the HP LaserJet firmware story, and I’m like, oh, never mind.
Yvo: So for me, it was getting rid of PMO actually allowed us to start collaborating with the team, and that turned into a lot of good things.
Bridget: That makes sense in that even if you have project management as a focus in your organization, that doesn’t replace the actual cross-functional cooperation.
Matty: Absolutely. I think I can see, you know, and Yvo, not to put words in your mouth, but see if I’m kind of unpacking what you said, is it wasn’t so much having a PMO in general was the problem, but a catalyst that forced the collaboration was removing them. So previously the PMO had fulfilled a certain role of communication. When those people weren’t there, maybe the point wasn’t to enhance all this awesome collaboration, but it was just like, well, someone can’t take the requirements from the users and give them to the engineers thing, right? You know, so it was like, I guess people have to talk to each other, and then you see dividends pay out of that, right? You didn’t start coming to the meetings so that you could do the DevOps. You started coming to meetings because you had to, and then there was a benefit that came out of it.
Yvo: [00:26:34] Absolutely. So there’s this great example of where we had to do a data center migration. So my last employer, they were a managed services company, or they were paying a managed services company about $14,000 a month. To host about 15 servers. I kid you not, 15 servers, and it cost them $14K a month. And it was like with this outfit in, in the New York area. And like, I came in and the first 6 months I’m like, what? Why are we spending this much money when we have a staff of 6 in Seattle supporting 15 production servers that we could easily do? And so I came and I went to the VP of operations. I’m like, and technology, I’m like, hey, so we could easily do this for about a third of the cost if we did this on our own. And they were anti-cloud. So I even proposed AWS at the time. I’m like, okay, so I got shut down on AWS, but they were totally open. And it was because of business stuff. So there’s a bunch of accounts and they couldn’t do AWS because of accounting, whatever. So I proposed like something safe. It was VMware, and it was about a third of the cost. They would have ROI on the entire thing in about 9 months because they were paying so much money for 15 servers. We started this project, and I actually reached out to the head developers at the time, like, so we’re going to move everything over, and I got shut down by my PMO organization. On top of that, I actually got spanked. I hate using the word, but I actually got verbally assaulted by people because I approached the developer directly about getting this done, which would have direct savings to the business on top of— it gave us the ability to deliver faster.
Bridget: [00:28:26] But that sounds like an org— I mean, that sounds like you’re describing an organization that’s fairly dysfunctional, right?
Yvo: Yeah, exactly.
Bridget: They don’t want you to save them money because instead they want you to preserve some power structure. Yeah, some real or perceived internal organization.
Yvo: Yeah, so like we— and I ignored it because I’m pretty stubborn, and Matt knows this. I’m pretty stubborn as I can be, and I just kept on going. I’m like, well, I saw the light. I’m like, I’m going to keep going. And I remember it was about 2 weeks before we had to cut over from this managed service organization to the new data center, and the developers were all on board. We had some minor issues, and they’re minor issues in that I think some of the IPs didn’t translate because we were moving the static app from one place to another. There were some changes. The development organization was all on top of it, but the PMO organization wanted to put a block on it. They’re like, no, we’re going to delay by a month. I’m like, no, we know what the problems are. We’ve realized what the problems are. We have a plan forward. I was using a different— I was using a configuration management tool at the time that was different than what I currently work for. We had identified the issues in that. It’s okay to say where you work, Yvo. Okay, great.
Matty: [00:29:39] Yvo’s my boss at Chef, by the way. He also loves it when I call him my boss.
Yvo: You thought you were going to get through this show without that coming up. Anyway, so I was using Puppet, and we had everything written in Puppet, and it was like we were going to completely stand up the infrastructure with Puppet. We knew exactly what was wrong, and we identified it, and we even fixed it within 6 hours. But the PMO organization, because we had a mistake that happened now 2 days ago, they’re like, They wanted to delay the entire thing by a month, which would cost the organization another— because we’re now, we put in our notice, they were going to charge us $20 grand to stay there. And like, this is ridiculous. We ended up moving forward. We overruled the Pima organization, but to this day, it’s bad times.
Bridget: Stay away from that. So, Kevin, tell us a little bit about the dev/ops cooperation or, you know, whatever it is that you have where you work. Do you have project managers standing in the way of your ability to talk to each other?
Kevin: [00:30:46] No, we don’t, but I don’t know if it’s an exact sort of similar situation, but we kind of have inherited the application we’re working on from a vendor that was developed, you know, according to a certain spec back in the day, and we’ve taken it on. And, you know, I hear stories— this was before my time at Trek— but, you know, the relationship before, we basically write some code, throw it over the wall, or rather the vendor would write some code, throw it over the wall, and it either did or did not do what, you know, it was tasked to do. And oftentimes it did not do what it was tasked to, but we just didn’t have the group in place to sort of say that, hey, this is not what we basically paid for. It could be so much better. Since we’ve taken it in-house and created the DevOps side, I just feel like, as we’re looking through— plug for a sponsor— through Datadog and seeing that there are definitely opportunities to make the application run more efficiently, that we’re all looking at this together. There’s less, you know, sort of team division where it’s, well, this is a, you know, resource hardware issue versus no, this code could definitely, you know, be written better. I think we’re all sort of focused on those things together. We’re in a small room together, so maybe that’s part of it. And, you know, we sort of work on our solution, you know, throughout the day. We have a test BeCycle kiosk in our office. It’s like 5 feet from my chair. It’s about 5 feet from everybody’s chair. And the normal sort of process is we— it starts out working in the morning, but sometime around noon it’s definitely broken, and hopefully we get it working again by 5 PM. But we’re all working on that. We’re sort of working against instances in Azure on the server side, deploying manual fixes and DLLs to the application on the station itself. For me, I don’t know, I have a little bit of a panacea. I love what I do. I love who I’m working with. I’m learning a ton. I think that was one of the worries. I know you guys covered what it means to be a new DevOps in a previous episode, but one of my big concerns is I’m not going to—
Matty: [00:33:06] Kevin, you can talk about it now, too.
Yvo: Sure.
Kevin: It was really sort of a leap outside of what I was comfortable with, and I knew that in the interview. I’ve just found that there’s just this level of excitement in not knowing and a level of, well, I’ve got to figure it out because no one— obviously people will help me, but I’m not one to punt. I like to dig in and figure things out. It definitely seems like if I do get stuck, I have the people there that know the answer on the code side or can help me with whatever I need. For help with on the development side. So I guess, you know, what I maybe take it to some other place, the things I know that one of the questions we have on the agenda is what’s changed for me. I think, you know, all the tools have definitely changed, but they haven’t changed so drastically that I’m completely lost. I mean, I definitely, once I, you know, get in and see what, you know, okay, this is a little bit different than it was before, I guess I’m not in, you know, UCS Management Console, I’m now in portal management, everything you guys are intimately aware of, has been just such an awesome shift for me because I don’t feel like I’m sort of in this camp that has to defend against all other attacks from other camps or users that don’t know what they’re asking for or financial sort of barriers to to accomplish what we want to get done. I don’t know if I answered your question, but I yapped for a bit there. It’s just been— for me, it’s been really cool. I think we’re working on a product that definitely has a long ways to come to mature, and I don’t see that we have any sort of major blocking or need to go outside of our group to solve the problem. I think everybody that has the different, you know, knowledge or, you know, background to solve the problem is working on the problem at the same time.
Bridget: [00:35:17] That’s pretty great. That’s awesome. All right, so that’s the perspective of you’re doing this stuff, you know, lately or recently. Chris, I know you’re— I’ve seen you smile a couple of times there because you’ve been doing this stuff for a while. You want to give us your perspective?
Chris: Yeah, yeah, a couple of things, just especially with the talk of PMO. Pretty much one of the first things that we got rid of at DIW when I joined was just the project managers getting in the way. We still have 2 project managers that we use. They are very, very valuable when it comes to doing things like data center migrations, setting up new data centers, big bits of work where timelines are needed and bits and pieces and the rest, but they’ve got nothing to do with software deployment, software delivery, software management. They are really focused on data center infrastructure, circuits and leased lines and all the rest. Also, a lot of the talk I spent today at GOTO Chicago— GOTO is kind of the current generation of one of my favorite lines of conferences— and I was going to save this for the The point is at the end, but one of my favorite talks today was a good friend of mine, James Lewis, talking about how he embraced Conway’s Law. And his talk’s initially on, or primarily centered around the microservices-oriented points of view, but everything that he talks about applies to what we’re trying to do from the DevOps point of view. Conway’s Law, if anyone’s not familiar with it, Conway’s Law states, and he stated this in 1968, that organization software reflects the communication lines that happen within the organization. James points out that, you know, a lot of companies that are successful with microservices, what they do is they actually combine their systems. They reduce the overhead of communication between teams, and so you get these cross-functional teams that are focused on delivering a product And so there is no PMO involved in there. Within the team you have your developers, you have your infrastructure guys, you have your networks guys if needed, any QAs and testers and all that kind of thing. There is a single team working together and they have complete ownership of what’s going on. But this also works outside the software industry. One of the great examples he uses is around the team at the Royal Free who cured the 3 Ebola patients that made it into the UK. Where they’ve got exactly the same thing. There’s a team of 33 that focuses on each patient, as they focused on the Ebola patients when they came in. But of that 33, only 7 were clinicians. You know, they had their own porters, they had their own facilities guys, they had their own finance guys that were all kind of dedicated on this team to, okay, this is a serious infectious disease that we need to kind of iterate around very, very quickly because You know, how are we going to cure this? How are we going to stop it from spreading? And these same principles that we’re talking about within software are also principles that people are dealing with outside the techie realm. Some of the other bits and pieces that we have around kind of the age, the changing of the BOFH mentality for me is how I treat my PFYs. So the PFYs, for those not familiar with The Pfy is the pimply-faced youth. The first Pfy or two that I had, you know, the important lessons that I felt I needed to teach them was, you know, defensive. You know, how to preempt the evil things that the users would try and do. My last, my recent, well, my current Pfy, you know, now is At least the one before, the one I had in London, he would often complain about the history lessons that I would give him. But often it’s once the history lessons helped him understand how we get there. And so some of those history lessons would be hardware-oriented or network protocol-related, but some of the history lessons also how we got to the stage now where we need to talk to everyone involved in focusing and having everyone focused on the goal of where we’re trying to get to. Rather than have this antagonistic approach, try and see it from the other person’s angle and try and think about the problems that they’re trying to solve and see how can we help us as a group collective move forward.
Bridget: [00:39:50] That’s, that’s a really good point, Chris, because I know that a long time ago in the ’90s when I was learning by example and learning by, you know, reading Alt-Sysadmin Recovery, You remember old System in Recovery. You’re smiling.
Yvo: Oh yes, oh yes, the memories.
Bridget: But when, when that was the environment that was shaping my attitude, I had to actively work to change my attitude, to move past that, and to start thinking about things from the perspective of other people and how can we all work together to try to meet the organization’s goals more effectively. And like that, it does require like making a deliberate decision to shift, but I’m glad that you’re taking that approach and teaching that to your PFYs these days because hopefully then more people aren’t going to start out in the BOFH mentality that we started out in.
Matty: So how can we do better? We used to be this way. We’re better now. But what are, what are some of the things when you kind of look at and I would say, you know, only 3/5 of the people on this panel right now are actually practicing sysadmins or SysOps or whatever right now. Yvo and myself don’t do that necessarily directly for a living, so it’s a little easier for us to talk about, like, it’s more hypothetical. But for those, you know, Bridget and Kevin and Chris who are doing this work. How— what do you want to do to get even better about this? Unless you’re totally perfect now, and then you can ask.
Kevin: [00:41:34] No, I mean, for me, I definitely would like to become more familiar with just development standards and, hey, pick up a language and start writing code. PowerShell right now is where I spend most of my time. I’d like to take a stab at building something. Not that I haven’t done that in the past, but it seems like now, in my off hours, when I’m not working, I’m more likely to do that. That’s kind of my playbook here coming up. Just being able to steer DevOps and make smart decisions on what tools we’re using and You know, not to, you know, just take a little bit from here, take a little bit from there, as far as the plethora of tools that are out for, you know, whatever, you name it. There’s just a lot of focus on tools for DevOps. So, I don’t know. Again, there’s, you know, I want to maintain this new flexibility I’ve found in DevOps. But I feel that there’s some decision-making that needs to be made on what tools we’re using, what we’re standardizing on, and how to keep that going forward. So taking a page out of my old playbook, how are we going to support that? But yeah, finding a way to support that because it makes sense and it is in this sort of new realm of cohesion and working more closely with all the different Well, not all camps, because we don’t have the typical infrastructure teams or email support groups in what I’m doing. Just taking this new sort of relationship between ops, being able to go to the cloud to spin up new instances, create more capacity for our applications. And yeah, just sort of being able to keep going down that successful path, which has been successful in the— well, it’s been about 10 months now, almost 11 months. But sort of keeping the agility, keeping my eye towards what’s coming down the pike as far as new tools, how to use some of the different I just— my head’s swimming. I spent the last week at MS Ignite, so—
Bridget: [00:44:10] You have all sorts of ideas.
Kevin: I know. Like, I want to talk about Release Manager and how excited I am for that as a platform to build all these other, you know, link in all these other cool tools that we’re using— Octopus Deploy, Chef, TeamCity— to sort of, you know, use that as a sort of base platform to start building up some more in-depth release process and metrics around how frequently we’re releasing. To me, I think it’s important to show to the business leaders how important this DevOps notion is. That’s not just a notion, that there’s some traction being made having this sort of focus on constant development, constant deployment. I think at Trek, it’s kind of a neat organization because their products are built around that. They’re constantly releasing new product and new bikes, apparel, what have you. So constant improvement is a thing at Trek, and I think DevOps, you know, fits that as well, and just being able to show that with some metrics. So, yeah.
Matty: [00:45:20] I think that’s—
Bridget: I was going to say, Stratton, I bet you’re probably thinking what I’m thinking, which is it’s really exciting that Kevin just told us all this stuff, and this is from a Windows perspective. And I know that those of us who have been doing this for a while, like, we didn’t think of Windows as a platform that you could even talk about, you know, continuous delivery or configuration management on, but there’s been so much progress in that area. It’s amazing.
Matty: Yeah, it’s, it’s, you know, I want to just kind of throw this in here a little bit following up on some of the stuff Kevin was talking about. Because initially this episode was supposed to be our Microsoft Ignite wrap-up episode, but I— that was a big fail for me. So we put this one together and I’m actually really excited. This turned out pretty awesome. So I love that. But one of the couple observations that I had about Ignite that I do want to share, the amount of interest that we had at the Chef booth, and that’s not patting ourselves on the back. That’s not supposed to be like, oh yeah, because Chef is awesome, everybody wanted to talk to us. But compared to previous events, just how many folks that were at this, basically what used to be TechEd, you know, so this is what Microsoft would call the IT pros, you know, so the sysadmins of the Microsoft world, the admin, you know, it’s not mostly developers and stuff. How many came to us and maybe they didn’t know what Chef did, but they understood, like when we said, oh, this is something that helps facilitate DevOps, they knew what that meant in kind of a very real way. And it was a lot less of— it was a lot more talking about Chef and what we could do than here’s why any of this matters at all, which was really heartening. And especially just to see how much of the talk— you know, I didn’t— unfortunately, I didn’t get to really see very many talks because I was standing in the booth all day having the same conversation with 800 people, which was great because it was a great conversation to have. But talking to people about the talks they went to, these were topics I would not expect to have seen at a Microsoft event a few years ago. And it’s— people have tweeted and made these comments that, yeah, we are definitely living in an alternate timeline right now, you know. So somewhere there’s an alternate universe where it really sucks to work with Microsoft stuff. Fortunately, we’re not living in that in that alternate timeline, right? At this point, we’re in the good one, or maybe it’s the evil one. I don’t know, the goatee.
Bridget: [00:47:49] I’m just on that topic of how we can improve. I’m still interested in hearing what, uh, what Chris and Yvo think that we are going to use. Just a line or two. How are you working on iterating to get even better at this, you know, doing the DevOps as opposed to doing the BOFH thing?
Chris: From my point of view, it’s— I’ve kind of dropped off the speaking, writing scene for the last couple of years, primarily because moving continents with a family is hard. But one of the things I’ve been thinking a lot about lately is much of what we see in the DevOps world, the implication underneath is you need to be using a PaaS of some sort You need to be somewhere in the webby space. I don’t deal with any PaaS. I don’t have any TCP connections facing the intertubes. My stuff is all purely internal. And for a lot of enterprises, especially when I was a consultant, these guys are really suffering out there. So kind of for me, the next thing is to take what I’ve learned you know, in my consulting days in the webby world and the 5 years I’ve spent having my own hardware and kind of distilling those principles and kind of figuring out ways that we can roll it out to the hurting masses out there who really need some help and love. And as much as I hate the term enterprise DevOps, there is much we can do to help people in enterprises learn the lessons and love of DevOps.
Yvo: [00:49:31] I would actually also target the other spectrum of that, is the lone sysadmin, the one that works in a small or medium business, and they’re one sysadmin supporting 50 to 200 to 300 users. We need to start embracing them as well. That individual is not a happy camper. They clearly still think, and they’re working under a pressure where they believe that it’s just them versus the world. I feel like things like DevOps and actually having a he or she understand, know what DevOps is all about, may actually make their lives a lot easier. A great way of seeing sort of like a temperature reading of that is, And Matt, you and I joke about this actually, is like r/sysadmin on Reddit. Like half that crowd is a lone sysadmin and they’re so unhappy and they’re so unhappy. And like, if you think about like, if we can make their lives easier, they’re going to eventually go into an enterprise shop or they may go for a startup, but they’re going to come in with these preconceived notions that being alone and supporting developers or supporting another organization is not a bad thing. And right now they do. Right now they really don’t like life. And so I feel like that’s one area we can really improve on.
Matty: [00:51:07] I agree. And I think you would say basically the same thing that I would think about in the role that someone like you or I being on the vendor side and how I look at what I do again, you know. I mean, sure, it’s great to, you know, get paid and everything, but I really do feel like we do what we do because maybe not everyone who we sell Chef to or that we help out or do things with is it really gonna make a difference, but when it does, it’s pretty fucking awesome, you know, to know that you’re doing some work that— because we’ve had this pain, right? And you’re not gonna get everybody there. And, you know, Jez has said before about that, you know, he’s gonna be preaching this for the rest of his career. And the good news is he’s always gonna have a job. The bad news is there’s so many people that don’t want to hear it, but you still get through. And that’s what I’m hoping, and I hope that happens. You know, I’ve given up on r/sysadmin. It’s just— it’s a place where people just want to fight. They just want to be bitchy. And they just want to complain and they just want to vent. And maybe that’s okay. And maybe that’s what that needs to be. And it doesn’t accomplish anything by getting into a fight there. So let them, that’s empathy to that. But there’s other mechanisms that we have to help show. And I think like this show, this episode, I think can be great because it says, hey, look, we’ve all been there. This is 5 people that this was most, you know, we’ve been in our career for a fair amount of time, you know, and we all felt the way that you feel now. And we got there. And once you know something can be done, it’s a lot easier to do it. So hopefully this helps reach some people.
Bridget: [00:52:53] And here’s my solemn promise. If you’re less BOFH-y, I guarantee you’ll be happier. Guaranteed.
Matty: I guarantee your money back. We will happily refund you the money spent on this training. Offer not eligible for our sponsors. So that said, let’s— we need to wrap up here. So let’s go into— we’ve just got a couple— some community and event stuff coming up. Happy to announce— actually, Bridget could say it. It’s your conference, your thing.
Yvo: Sure.
Bridget: Okay. There is a lot of DevOps Days coming up. You should go to devopsdays.org and check them all out. If you would like to come to DevOps Days Minneapolis, it’s July 8th and 9th. Unbeknownst to me, Matt went and signed Arrested DevOps up as a media sponsor for the conference.
Matty: [00:53:55] I would like to point out it’s not completely unbeknownst to you, but to be fair, when I mentioned it to you, it was when we were in a bar and you were cranky about something, and I said, hey, ADO should be a media sponsor, and you said, talk to Duffield, he does the sponsors.
Bridget: I really don’t remember this conversation.
Matty: Yeah, you were—
Yvo: yeah, it was—
Matty: I was there.
Bridget: It was—
Matty: yeah, this is so—
Bridget: I’m not familiar. But in any case, ADO listeners can get a 10% discount on, um, a DevOps Days Minneapolis ticket, which is still on its early bird pricing until the end of May. So we can get 10% off with the code ADO10. Okay. And, uh, DC in June, Pittsburgh in August.
Matty: The very first DC, right?
Bridget: Very first DevOps Days DC, um, at the US Patent and Trademark Office of all places. Should be pretty cool.
Matty: Is this going to be the new thing now where you like have to do DevOps Days like in somewhere, you know, like everyone has to raise the bar? Like Rockies was in a data center. They’re going to do theirs in the Patent Office. Although I’d like to say, you know, Chicago, we did ours in the Sears Tower, but then you discover that those weird spaces are not really actually that good and you go back to something normal.
Bridget: [00:55:01] You know, you did it in the Sears Tower, which was really cool, but I did not know it was possible to be too high for LTE network.
Matty: Yep, that’s, that’s why we are not— talk to Chris. Chris was our network guy. Exactly why we’re not doing it there again.
Chris: But browse on the network problems there.
Bridget: Yeah, that was, that was pretty wacky. So yeah, Dev Upstairs Chicago, Matt could tell you a little bit more about that.
Matty: Yeah, actually, so that’ll be August 25th and 26th.
Kevin: 26th.
Matty: It’s at the Gleacher Center in Chicago, Illinois. The end of August should be actually pretty nice. Registration is available. I think we still have some early bird tickets available, so if you go to devopsdays.org or devopsdaysshai.org, you can register. We also just opened our CFPs, which I think are open until the end of June. If you go to cfp.devopsdaysshai.org, Go ahead and you can submit an Ignite or a talk. We’d love to see what you have. So let’s do our checkouts. Chris, what do you have for our listeners to check out?
Chris: [00:56:05] Yeah, keeping with the topic that we’ve got today, there’s an awesome talk about the evil hat by Liz Keogh. I’ve got a link to a recording of, I think it was QCon London. Where I saw her give this one, but it’s fantastic. She gives some fantastic examples of various different ways perverse incentives have bitten people. But yeah, also just keep an eye out for when the QCon Chicago talks make it out onto the internet because there’s some really, really good ones. The closing keynote today was from one of the JPL NASA group who she was actually in charge of working on the parachutes from the Mars landers with the whole sky crane system there, which was a very educational talk. I really enjoyed that.
Matty: [00:57:10] Great. And Kevin?
Kevin: So non-tech related, well, it is somewhat tech related. The new Trek mountain bike that’s out is the new Stache. It is a 29er+. It is super sweet. Basically, the cool thing about this bike— so I ride a very large bike, and on top of— I’m 6’4. You know, with a 29er, you have larger wheels. Typically, those bikes are up a lot higher, and with a larger wheelbase, they’re a little bit harder to guide down a downhill path. So they’ve Solved that problem by basically raising the rear chainstay above the drivetrain. Allows the— when you pull up the bike, you can view it on trekbikes.com. Again, it’s the new Stache. You can see how much lower the bottom bracket is by allowing the drivetrain to be below the chainstay.
Matty: I am looking at it right now. I love that the tires are called Chupacabra.
Kevin: Yeah, it’s a super hot bike. Just came out this month. All very excited about it, and I’m hoping that they still have one in my size.
Matty: [00:58:16] I need to buy a new bike, so I think, Kevin, I’m gonna be hitting you up for some suggestions.
Yvo: You have a friend.
Kevin: Yes, absolutely glad to, anytime.
Matty: Yvo, do you have any checkouts? I know we kind of hit you at the last minute, so do it live.
Yvo: No, not right now. I don’t, because I had no idea that was coming.
Bridget: It’s okay, I’ll give my checkouts, and then if Yvo wants to tell us about something cool he’s read or bought or thought about or tried lately, that would be great. You know, new favorite whiskey or whatever.
Matty: Yvo drinks IPAs.
Bridget: What?
Matty: And martinis.
Yvo: I do drink IPAs.
Bridget: Hey, at the DevOps Days Minneapolis organizer meeting tonight, I had some delicious Lift Bridge. It’s a very hoppy IPA from Stillwater. Or is it Stillwater? Yeah, it’s from Stillwater. Okay, so I have 2 checkouts. One is we’re about to that, if you live in zone 4B like I do, we’re about to the time of year where you start thinking that maybe you could start planting some things, but maybe not because it’s overnight lows in the 30s right now. But if you’re into gardening at all, which I am, seedsavers.org. Organic Heirloom Seeds is super worth checking out. You can get really interesting seeds for, you know, like, I don’t know, types of carrots that you’ve never seen before and, you know, things like that.
Matty: [00:59:48] My son wanted to plant cantaloupes this weekend. He found cantaloupe seeds at Walmart. He’s like, can we plant these? I’m like, I’ve never grown a cantaloupe.
Bridget: Can you grow cantaloupe in Chicago?
Matty: I don’t know. We’re sticking with carrots and cucumbers and tomatoes.
Bridget: Oh, I’ll have to see. But anyway, so if you haven’t, you know, sated all your seed desires, it’s worth checking Seed Savers out. And then also, because it was a really interesting PDF that came up in a discussion that Allspaw actually linked to, because as BeerOps likes to say, as Catherine Daniels likes to say, he mostly communicates by sending PDFs to people, whether in email or other. But he linked to this interesting paper by David Woods and a couple of other people from 2004, I think it was, called Common Ground and Coordination in Joint Activity, which doesn’t sound as exciting as it actually is. It’s basically all about communication, collaboration, cooperation, you know, all of this inter-team stuff that we were talking about today on the Hangout.
Matty: [01:00:52] So that’s awesome.
Bridget: And full disclosure, I have not read the whole thing yet. I’ve read like 17 pages in. It’s very long. It’s hard to make it all the way through really long PDFs, but it’s really interesting so far. There might be a twist ending.
Matty: I don’t know. My first checkout, which I have to admit I haven’t read yet because I just got it today, but we want to share it, is that the early release, unedited, whatever version of the O’Reilly book Effective DevOps by Katherine Daniels and Jennifer Davis, has been released. And most amusingly, O’Reilly chose to have the animal on the COVID of it be a yak, which was pretty awesome. So go ahead, you can go pick that up and put it on your Kindle or your ebook thingy or whatever. And then another one is— I’m not quite sure how to pronounce this. I think it’s ASCII cinema. ASCII cinema. It’s basically ASCII cinema, and it’s a tool for recording terminal sessions. For playback later, like for a video of them. So like if you’re showing a demo or doing a bunch of commands on a terminal. So I’ll put a link to that in the show notes, or it’s asciinema.org to check out.
Bridget: [01:02:08] That’s really— that’s interesting. But okay, so we can’t let Evo get away with not telling us anything whatsoever that’s cool or awesome. So here’s a gimme, here’s a softball. What is, in your estimation, the most delicious IPA?
Yvo: Hop Venom by Boneyard. It is the best IPA ever.
Bridget: Why?
Yvo: It’s a perfect balance of flavor, so it’s not like overly— even though it’s called Hop Venom, it’s actually not overly hoppy, and it hides the fact that it’s a 9%-ish IPA. Double IPA by actually like being nice and it goes down nice. And yeah, I don’t know, like they have a— they actually have the Notorious, which is a triple IPA. While it’s really good, it is actually— you can actually taste the alcohol versus Hop Venom where you don’t.
Matty: So I’d like to point out that Evo on Untapped has tried 558 different kinds of beers. So when he tells you that’s good, you should, uh, yeah, we’ll have a link to that.
Bridget: [01:03:17] We’ll have a link to that.
Yvo: So my local, so my local bar in Seattle, it’s called Chuck’s Hop Shop. They actually have it on tap every Monday and they save a keg to do it on Mondays. And at this point I actually go there and call it Hop Venom Mondays because it’s just that good.
Bridget: Boneyard Beer Company, you heard it here. All right, fantastic. Thank you.
Matty: Yeah, great. So wrapping up here, we have a newsletter. You can subscribe to it by going to arrestedevops.com/bananastand. It is the best way to know about our upcoming podcast episodes and cool news with DevOps. We also have an iPhone app if you like iPhone apps and don’t have another kind of pod catcher, but hey, you can download it for free at arresteddevops.com/iphone. I’d like to thank our sponsors. Make sure you visit them at arresteddevops.com/pagerduty and arrestedevops.com/loggly, which is L-O-G-G-L-Y. Thanks to Chris, Kevin, and Yvo for joining us, and to our loyal listeners, or disloyal listeners depending on the case. If you enjoy Arrested DevOps or don’t, either way we’d appreciate it if you visit arrestedevops.com/itunes and Leave us a review in the iTunes store. We’d love to know what you thought of this episode, and you can leave us comments at arresteddevops.com/36.
Bridget: [01:04:43] Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter. We’re always happy to get your input, ideas, or feedback at shows@arresteddevops.com. Please let us know any ideas you have for future episodes. I’m Bridget, @BridgetCrumhout.
Matty: I’m Matt, @MattStratton.
Bridget: We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.
