managing systems in the cloud

Matty: [00:00:07] Welcome to Arrested DevOps, episode 23, Managing Systems in the Cloud. I’m your co-host, Matt Stratton, @MattStratton on Twitter.

Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.

Matty: Arrested DevOps is brought to you by cloud services firm, 10th Magnitude. Not only are they cool enough to sponsor Arrested DevOps, But they bring you the Azure plus Chef equals awesome webinar series. The next episode is Wednesday, October 29th. Check it out at tenthmagnitude.com.

Trevor: We all know that being on call sucks, but what if there was a tool out there that allowed you to route incidents to the right team, @mention specific people to ask for help, and hop into chat with your team from an easy-to-decipher incident timeline that gave you full context of what was happening? That tool is VictorOps, and they’re different. From setting up global on-call rotations to creating a postmortem report, VictorOps is there with you through every step of the incident lifecycle. Our real-time collaboration platform helps your teams to solve problems faster. Sign up for our 14-day trial to see how we’re making on-call suck less. Visit arresteddevops.com/victorops to sign up.

Matty: [00:01:15] This episode is also sponsored by Redgate Software. Redgate makes tools that bring the benefits of continuous delivery safe releases, efficient deployment, and fast feedback to your database. Find out more about database lifecycle management, download free trials, and browse the database delivery learning program at arresteddevops.com/redgate.

Trevor: Now, as you may remember, all our loyal listeners, we invite you all to leave us reviews in the iTunes Store, and we’ve been promising you for months that we would read any reviews that we got. Well, we finally got a glowing accolade, and we’re We’re going to have Matt read that to you now.

Matty: The first thing is we actually did read one a couple episodes ago, but this one is way better. So this is left by listener Bob Farley, and the title of this review is Two Chimps and a Mic. He says, I stumbled across this podcast the other day and I am dumbfounded. It’s basically two chimps blathering away about something called DevOps. Now I have no clue what they’re talking about, but they really thought they were clever. Especially that one Matt that can’t seem to stop talking. Don’t get me wrong, I think it’s great that kids these days have this outlet. In my day, we had to have our voice recorded into a wax cylinder if we wanted to do a podcast, and we didn’t have the iTunes to distribute our wax all over the internets. Maybe if I had the iPads for podcasts, I would have done it myself when I was a young’un. And that’s why I give it 5 stars. I had no clue what they’re talking about, and they weren’t even funny. But I love that they would get behind a mic and make complete fools of themselves. Keep it up, Mark and Travis. So thanks, Bob. If nothing else, thank you for the 5 stars.

Trevor: [00:02:55] I would like to point out a factual error in this review. There’s 2 microphones typically, at least between the 2 of us.

Matty: That’s true.

Trevor: There were no other factual errors in that review?

Matty: Especially the part about me not being able to stop talking.

Trevor: Or us being chimpanzees.

Matty: So, on with the show. Tonight we are joined by one of the authors of The Practice of Cloud System Administration: Designing and Operating Large Distributed Systems, Tom Limoncelli. I hope I pronounced your last name right, Tom. So, A, can you tell me if I pronounced your name correctly? And B, can you introduce ourselves to our audience and tell us a bit about what drove you to write this book?

Tom: Hi. Well, thanks for having me on. I’m really excited. I’m a big fan. Longtime listener, though I do listen to it on my iPhone at 2x, so I’ve actually never heard the theme song at regular speed until today. I thought maybe I should do that. So I want to start with a time management tip since that’s my other thing, which is listen to your podcasts at 2x. You’ll get through them a lot faster. Okay, so my name, yes, Limoncelli or Limoncelli, I answer to both. Actually, I have relatives that say it both ways, so Limoncello is how I generally say it. And what was your other question? You had an actual good question in there.

Matty: [00:04:09] Oh, what kind of was maybe the impetus for the writing for the book? Maybe it’s a good start.

Tom: Well, system administration has really changed. My first book, The Practice of System and Network Administration, came out in 2001, and it’s amazing how much things have changed since. So what we wanted to focus on was DevOps things and all the stuff that I learned while working at Google and other companies, which are really a bigger focus on service administration. If you look at my older books, they’re really oriented towards people that are doing like help desk and system administration work. And I think modern system administrators, most of them are focused on service administration, and the help desk at large companies has been outsourced And small and medium businesses, you’re doing some of that, but it’s not a huge fraction of your job like it used to be.

Matty: So one of the things we talked about a little bit before was whether or not, you know, we said it’s about managing systems in the cloud, but kind of the focus of it is not— of the book is not necessarily about kind of what we might traditionally think about as cloud providers like Amazon or Azure or Dito or whatnot. But that being said, like one of the things I had on here, I’m kind of curious your thoughts on this. As I said, you know, let’s get it out of the way right now. I mean, okay, what the heck does cloud mean anyway?

Tom: [00:05:35] Well, what does cloud mean? Cloud means whatever you want because the word cloud has been taken by marketing people and kind of destroyed. In fact, you know, when we were trying to decide on the title of the book, we were trying to figure out— the book is mostly about DevOps and service architecture and service operation, and what’s the one word that combines all of those? There isn’t one, so— but the marketing people at the publication were like, oh, if you could get cloud into the title, that would be awesome. So the first time we actually use the word cloud in the book, it’s to explain what we mean by cloud. And the first thing we say is, you know, cloud has been destroyed by marketing and it means nothing. So when we talk about the cloud, what we’re really talking about is really distributed computing, which is the concept of systems so large that the work is divided among many machines. So whether that’s a web server that is a website that’s really, you know, 10 or 10,000 web servers behind some kind of load balancing mechanism, or Hadoop, or even just a service-oriented architecture where you have many machines all cooperating to get the job done. As opposed to the old paradigm of, you know, back in the 1990s when the web was new, a website was generally a particular server And as the website needed more power because it was getting more users, they got a bigger and bigger machine. And eventually we exhausted that route. I mean, you know, we were buying these huge, you know, Sun R10K— what was it— the 10,000 model and these huge, huge machines. And that wasn’t gonna scale because machines only get so large. But distributed computing, you know, if you’re dividing the work over dozens of machines you can scale that larger than any one machine by virtue of the fact that even if you did find a super huge machine, you could make a distributed network out of them, you know. But interestingly enough, that brings all new challenges, not just architecturally, but how we run our services. And so the book is really about these new challenges, and I, I think it applies whether you’re at, you know, Facebook or Google or Yahoo or if you’re at a site with, you know, one web server and a database.

Trevor: [00:07:52] So as this sort of new paradigm evolved, how did you start shifting from this world of vertical scaling to a world of horizontal scaling in terms of systems?

Tom: Personally in my career or in my writing?

Trevor: Both.

Tom: Okay, well, my career, it happened because I left my old job and went to work for Google where everything is horizontally scaled, even, you know, internally we would make jokes about, you know, even the way we handle our cafeteria is horizontally scaled.

Trevor: But before we go too far, just in case, can you define horizontal versus vertical scaling? Good point. I realize that that might be a term that not everybody gets.

Tom: And I actually—

Matty: I want Trevor to define it because he’s the one who brought it up.

Tom: You know, I admit that I have it on the palm of my hand, a little cheat sheet. I always have to look, wait, which one is the vertical scaling? Okay, so correct me if I’m wrong. Vertical scaling is taking the one machine and getting it bigger and bigger and bigger, so buying bigger machines, more memory, more disk. And horizontal scaling is dividing the work over many machines, so a website that’s split with a load balancer that’s split over, you know, 8 machines, 80 machines, 800 machines, that’s scaling it horizontally. You can do that with storage, with CPU, with memory.

Trevor: [00:09:16] That would be basically what I would say too. So at least we have the same shared understanding of what we mean local to this conversation. If someone disagrees, they can tweet at us about it.

Matty: So you made the transition because of, like you said, you went and you worked at a company where that was the way, but skill-wise, or maybe skill or kind of philosophy? Actually, let’s start with skill. Like, what were some of the skills that you had to adjust being kind of more classic system administration moving into this more service-directed world?

Tom: Right, good question. So, well, the first change was monitoring became so much more important because if you’re in charge of a few machines or a simple system, monitoring is important, but you kind of know what’s going on because you built the system and you’re— you understand the system. But as systems grow and these distributed systems become incredibly large and incredibly complex, no one person can understand the whole system or have visibility to it. So you have to instrument everything so that it can be monitored. You have to put a huge effort into making it observable. And then have the monitoring and the dashboards and everything. And as a result, you make decisions based on data instead of hunches. And actually, let me tell you a story. So when I am— you know, I worked at Google from 2006 to 2013, and on my like second or third week, they were talking about a problem, and I thought Oh, I know the solution based on my experience. And you know, this was a very humbling experience because I was kind of like, well, you know, I’m a senior sysadmin and typically the solution is XYZ. And someone who was a very junior sysadmin pulled out this graph and he said, well, actually with the monitoring system I’ve collected data. Here’s the graph. Here’s the trajectory of the line. When I made this change, the line changed like this. Therefore we should do, you know, the opposite of what Tom says. And I was like, Wow, that is such a better way of doing system administration. And, you know, I was humbled. I was like, yeah, that’s the way we should be doing things instead of, you know, hubris and ego and the experience of senior people. And so I’d say a lot of what I learned was that. And I should say in the book, we have— the monitoring chapter got so large we split it into 2. Part one is the architecture of monitoring systems and the other is kind kind of the practice of monitoring. And even then, throughout the book, there are more chapters about being more data-driven in your team management and systems management and everything. So I’d say that’s the biggest behavioral change. The second biggest change is just when you’re dealing with huge scale, downtime is thought of differently. So, you know, in the old days, I’d be running a mail server for my 100 or 200 users, and if it’s down, my monitoring system tells me it’s down and I rush to fix it, or whoever is on-call rushes to fix it. And we kind of pat ourselves on the back because it used to take us 2 hours to fix it, and this time we did it in 1 hour, and wow, what a great improvement. But when you’re running a service for thousands or millions of users, every bit of downtime is, is so much more visible. And, you know, at Google, you know, the first time they had a big outage— well, first time after they were a well-known entity and had a big outage, it was headline in the New York Times the next day. So instead, you try to do your system administration where you sense when a system is sick and you fix it before that turns into an outage. As opposed to being really good at responding to outages, you get good at preventing outages.

Matty: [00:13:08] So I was wondering if— you went in a different direction than I was expecting, not that I disagree with it at all. But one of the things when you talk about operating at scale, an anecdotal story that I heard, and it’s a a gaming studio that will remain nameless for 2 reasons, one being I don’t think I remember who it was. But the idea was talking about the number of instances that get spun up. We’re talking about we’re gonna need to fire up for scale, you know, another 10,000 or 100,000 instances. If 10,000 of them don’t come up, we’re okay with that. Like, it’s just sort of a scale thing. Like, again, with that horizontal scale, you can be like, hey, I can have that margin of error because in a way, right, these nodes, these systems, these servers are simply cogs, right? You know, and, you know, Christopher Webber likes to talk about his idea that, you know, everything will become an API, like servers will all become API-like components in code, you know, that are all just sort of these cogs that talk together. And that’s the thing that to me, again, thinking about that shift from the cloud way of thinking of the— that they’re really all just resources as part of a larger gestalt machine that is your service like you said, A, you care a lot more about the service that you’re providing, but I think you tend to care a lot less about the individual— I don’t want to say hardware, but right, the individual systems themselves, right?

Tom: [00:14:27] The components don’t matter, the system matters. So, and this doesn’t even have to be true at scale. I mean, even if you’re small, you’re familiar with RAID. And you know, before RAID was popular, a hard disk was either running or it was dead, and that was your storage system. A component failure equaled a service failure. Your hard disk died, you were gone. But now we’ve decoupled component failure from service failure because RAID lets us keep running even though one disk has failed. And as you build systems larger and larger, it’s not just RAID, you know, it’s many replicas of your web service and your database service and your other things. So You know, another way of thinking about it and something we— there’s a whole section in the book about this, is the fact that the old way of thinking of computers is that they’re up or down. It’s a binary thing because they’re digital and therefore, you know, it’s yes or no, it’s up or down. But with very large systems, you have individual components fail and then things are just degraded. So it’s more like an analog system. It’s not that You know, Gmail is never up or down. Not that I ever worked on Gmail, but it’s a good example. Gmail is never up or down. It’s running at 90% up or 80% up. It’s an analog thing. And as more components fail in a large system, it gets slower or features disappear or whatever. And yeah, so that’s a whole different way of thinking of systems.

Trevor: [00:16:02] Yeah, there was a lot of discussion both at DevOps Days and I’ve been having these discussions with some of my clients as well about exactly that. The, you know, getting away from the all on or all down and, you know, they’re trying to create these distributed systems and they’re realizing the software they already have is up or down. You know, they have all these discrete systems within their system, but they are all reliant on the whole tower being built before any of those individual components can be running. And so they’re working on that split.

Tom: Yeah, you know, I teach a tutorial at the next USENIX LISA conference that covers a lot of this stuff, and for each I go through 8 different case studies, and for most of them I realize that the audience is going to say, yeah, but the software I’m using can’t do that, and my response will be, that’s why we need to architect things differently, and you need to be there at the architectural level, not to just say, oh, that’s depending on, you know, this technology, we don’t use that, which is what sysadmins often find themselves saying. But instead to say, hey, we need to distribute this. If it’s going to scale to 10 million users, we have to start from scratch thinking about how are we going to do this in a distributed, more analog way. And I think when— even if you’re taking off-the-shelf packaged software that is totally the opposite of this kind of thing that we’re talking about, you can still maintain that sensibility as you’re designing the system and designing how you’re going to use those components. I mean, most things work behind a load balancer nowadays, and when they don’t— things that aren’t, you know, a web service— you can work with the vendor or you can plan things. And when you can’t, the dirty secret of distributed computing is Every, even the most perfect system where everything is distributed and replicated, there’s always that one thing that can’t be replicated, right? There’s some lock server or something that can’t be replicated, or it goes against the grain of the rest of the system. But you can keep that culture and that sensibility as you’re doing the design.

Matty: [00:18:11] There’s a, I think it was on an episode of Food Fight Show where Jez Humble made a comment that I reference all the time, which was the, If you say I can’t do that because of my architecture, then the answer is, well, your architecture is wrong. You know, it’s just like—

Tom: I want to respond to that, but first I want to explain that this book is very much— it’s a textbook that I hope will bring— that we, the 3 authors, hope will create a new generation of people that do think in these terms. And if you are a system administrator that listens to these podcasts and says, I don’t understand half the things they’re talking about, I hope that this is the book that will bring you up to speed on those things. So I don’t wanna say it’s training wheels for this, but it does go over the basics of these things. And now directly in response to your question, one thing that I found is that system administrators often don’t have that background in large system architecture that lets them have these conversations like, your architecture is wrong, because then someone’s gonna say, well, how should we do it? And they say, I don’t know, it just should be done some other way. So part 1 of the book, which is about the first third of the book, is distributed architecture explained in terms that system administrators will understand. There’s an overview chapter and then a, you know, design patterns for resiliency, design patterns for scaling, platform selection, these kind of things. And it’s really kind of like a boot camp for the architectural decisions that you need to understand so you could have these very high-value conversations with developers. And then part 2 of the book is how to operate large complex systems. So it’s a DevOps overview, how to manage on-call, which is very different in a large system, how to do monitoring, how to organize the work that your team does. You know, how do you structure a team when it’s 5 teams of 5 people each maintaining this huge system, or even just 5 people maintaining one small system. And then there’s a surprise ending. Should I spoil it for your—

Matty: [00:20:20] I don’t know, I would think not. We want people to read your book.

Tom: So it’s not called part 3. Well, there is a part— part 3 is the appendixes. But so the surprise ending is we then have an assessment system that helps people rate the service that they’re running, and that reveals where they could make improvements. And that’s cool for 2 reasons. One is that I’ve talked to a lot of people who say, oh man, I work at this place and it’s really screwed up and we don’t know where to start. Well, here’s a questionnaire, 12 questionnaires, you average each of those and you get 12 values, and you could say, oh, so, you know, number 3 we suck at worse than anything else, that’s You know, maybe it’s not where we’re the worst, but that’s definitely where we’re feeling the pain the most. So it helps you visualize, or it helps surface your areas of improvement. But then if you make a spreadsheet of those 12 items, say that’s one column, and every month you reassess yourself, and the ratings are from 1 to 5, we have a suggested color scheme, and if you do this every month, you get a heat map that shows your improvement. Each month you add a column and you see the colors fill in and they’re going from green to red and you can actually see the improvements you’re making on that service or not making.

Trevor: [00:21:40] That’s awesome.

Tom: Yeah, and if you really want to get crazy, you can have a spreadsheet for each service that you run. Say you have, you know, 5 services. You could do this for each of them and as happened recently in where I work now at Stack Exchange, our Director of Engineering said, hey, this particular service is our top priority. We could take a look at our assessment for that and really focus on it, or you could roll up all 5 of those services and get an overall number to describe what you’re doing.

Trevor: So it sounds like, it sounds like in the book this rating system is targeted towards sysadmins, but it also sounds like when in practice you’re using it with—

Matty: the book is called Cloud System Administration.

Tom: Well, it’s targeted at whoever’s doing the operations, and in a good DevOps environment, both your developers and operations people are— I’m sorry, it rates— oh, good question. Is this targeted towards developers or the sysadmins? I guess—

Trevor: [00:22:46] I mean, it sounds like it should be a shared conversation.

Tom: It really is. Yeah, now I’m thinking, I bet the wording is in the book sounds more sysadmin-y, but that’s just because I come from an environment where I expect everyone to write code. So I don’t think in terms of who’s a developer, who’s operations. I just think in terms of there’s some people on the team that are more operations-heavy and some are development-heavy. So, you know, in every sysadmin team you have some projects that are obvious coding projects, like we have to automate something, and some that are more operations focused, like, you know, we’re moving a data center, you know, and different people have different skills and you need a mix of those. So anyway, this rating system is really at the overall operations capability of your team, and if you need to improve that, sometimes you improve that by writing code and sometimes you do that with a more operational project. And also, if you’re the CIO of a company and you have a dozen teams, you can take this assessment, if you have each team do this assessment, you could roll up to the team level and then roll up to the organization level. So you could in theory use this to understand, like, if you’re the CIO and you have a dozen sysadmin teams or operations teams underneath you, you have no idea how well they’re doing. But if you have each of them do a self-assessment, it gives you the visibility that you need to figure out where to move resources and where your improvements need to be made.

Trevor: [00:24:14] So, that sounds fantastic. I mean, it sounds like you’ve used this in practice a little bit. What are some common starting places that you’ve seen? Like, you know, typically when you hear about distributed systems, you hear about, you know, you can distribute everything but the SQL Server, or whatever it boils down to. What are some common areas you see that are good candidates for abstraction and such?

Tom: Well, I think it’s important to There’s 2 ways that I look at this. One is you can do like a lean analysis, which is essentially a fancy way of saying we’re going to find the bottleneck. So what is distracting you the most, or what’s the choke point that’s preventing the important stuff from getting done? Another way of doing it is— this is kind of a hybrid of a bunch of different techniques, but one thing that I really enjoy doing is, I guess it’s my way of breaking down silos. So you have, you know, some operation that involves 5 different silos, and I like to have meetings where the different silos come together and talk about what are our pain points, what bottlenecks are we causing for each other, and that’s really powerful. I was actually in one of the projects I was working on, when I joined they had just done an analysis. It was very complicated They were having trouble getting their projects done on time, and they did analysis of what does it take to get one iteration done, and they found it was 27 handoffs by 15 different teams. And in the years that they had been doing this, no one had ever drawn a picture of, you know, a graph of all 27 handoffs. So they didn’t— A, they didn’t know how many handoffs it was, they didn’t know how many teams were affected, but they had just spent a couple months trying to write down the whole process. So this is kind of like Gene Kim’s 3 Ways of DevOps, way 1. And then they met, they had sit-downs with all 15 of those teams and showed them the graph and said, this is what we’re dealing with. We think that, you know, item number 10 on the list is something you’re responsible for. This is the input, this is what we expect out of it. We think we can’t bring this to you until the following dependencies are completed. Is this correct? And they went through each point in the graph with all these different teams, and they found tons of dependencies that weren’t needed. They found tons of dependencies that hadn’t been documented that explained tons of delays that they’d never understood before. But most importantly, it bred a sense of empathy between the teams, and that— it just organically broke down the silos. I don’t think an executive standing in front of everyone saying, we have to break down the silos, ever has broken down a single silo. But sitting down one-on-one with another team and getting empathy makes people say, oh, you know, we don’t have to do it that way. Oh, you know, oh, we always give you the database in an unsorted order and every person down the line then sorts it themselves and that takes 2 hours because it’s huge. Oh, we could sort it once and hand it down the line. You know, these simple empathy things are so important. And one of my favorite comments from these conversations was, we had just shown the graph and walked through it with this other team, and we took a break and someone cornered me in the hallway and said, Tom, you know, I’m embarrassed to say this in front of everybody, but in all the years we’ve been working together, we just would get a ticket saying, you know, we need this, and we would do it, and we always thought, gee, like, once a month or a couple times a month we get the same request or a similar request from you guys. We had no idea that we were just one node in this graph of 27 handoffs, and we feel so bad that we’ve, you know, not responded to them so quickly. We had no idea that our delay caused 5 other delays in the graph. And I was like, that’s nothing to be embarrassed about. Let’s— when we’re done with the break, let’s start with that, because I think everyone hearing that would be really powerful. And we just kept having that kind of experience, and we were able to take this crazy graph— when I left the team, it wasn’t a linear, you know, 27 handoffs. It was still this spaghetti mess. We actually understood what the handoffs were and where the dependencies were, and things were getting so much better, and it was becoming more of a repeatable process.

Trevor: [00:28:36] That’s a really interesting story, and I mean, it makes me think of— I had a conversation recently about somebody who’d gone to, you know, works in a fairly large large organization and had gone to their first DevOps conference. And, you know, the thing that— they were a developer and the thing they said was like, I never really thought about, you know, what I’m doing to the ops team. Are the errors that I’m sending them useful to them? You know, are they actionable? Are they, you know— and just that you’re absolutely right, the empathy is— it’s funny how all these conversations in some manner tend to wind up back about how we treat each other as people.

Tom: Yeah, yeah. So at Stack Exchange, we have one of our failover processes. If we lose our main data center and have to fail over to our DR site, it’s just a huge, huge process. And not everyone on the team knew how to do it. You had all these common problems. I know I was— when I was on call, I was just praying that that kind of outage wouldn’t happen because I knew that I was the new person and I didn’t really understand the process. So we started doing drills. And the first time we did a failover drill, it took 10 hours to fail over to our disaster recovery site. And the next time it took 5 hours, and we can now do it in 1 or 2 hours. And after the next couple drills, I’m sure we’ll have it down to maybe 10 minutes. But the interesting thing about doing these drills was that even though we filed— like, one of the drills we filed 30 bugs related to how we would improve the process. Documentation fixes, etc. Half of those bugs were closed by the time the drill was over because there was enough wait time that developers were just, you know, coding up solutions, fixing scripts, improving the user interface to something, fixing documentation just in real time because they were seeing like, oh, operations has to do this in the failover situation. Oh, I can make a button that’ll do that in one click. Right, because they were seeing what we were going through and it was improving empathy.

Matty: [00:30:34] I just wanted to read a tweet that there’s— we’re having quite a few people are really enjoying this episode, which is awesome. But Eric Shamow just said that talking to Tom is like talking to Patrick Debois, a lesson in how work, tech, and empathy are not divisible.

Tom: Thank you, Eric.

Matty: But I think that’s so true, right? Like, you know, Trevor said it’s, you know, all comes back. To how we treat each other, but it’s more than that too. It’s also, it’s those, all those things, right? You know, and I think, you know, if we hand wave past like the, oh, okay, well, we should all treat each other nice. It’s like, it’s being able to see how those things actually functionally intersect, right? You know, and I think that’s kind of one of the, maybe the ironic thing of people who don’t understand that not being empathetic to the people who do that happens in the community a little bit, which is the, well, how is this measurable? You know, I can’t, go to a CIO and say, I’ve increased empathy by 10%, you know, and I think it’s still just going back and looking at like the— but when you gave that example, you’re like, okay, so we did this thing and what it did at the end was increase empathy, but we didn’t say let’s have the empathy project.

Tom: [00:31:39] Right, you can’t have the empathy initiative. You have to have— the goal has to be improved service or improved ability to roll out new features or improved uptime or, you know, some— something that is manageable and critical to the business, the way you get there is through empathy. And it’s kind of like— I can’t remember his name, but continuous integration people are often talking about how the goal isn’t to get your entire build and deploy time down to, you know, 2 hours or 2 minutes or 2 seconds. I mean, you can brag, oh, it was 5 weeks, now it’s 2 hours. You can brag about that, But that’s not really the important thing. The important thing is the confidence that comes from moving to continuous integration, and the fact that it’s not just that the builds are now 2 hours, but the builds are now so well tested that you have the confidence to just constantly be pushing code. And when a developer writes a new feature, it’s not 5 weeks later that it’s available, it’s 5 hours later. That it’s available, and that’s, that’s a business value. I mean, imagine if you were a car company and every car you made you put in a warehouse and it sat there for a year before you sold it. That would be crazy. But we in the software industry, we write software— well, so 20 years ago we would write software and it would sit for a year before the release was done, and then we would shrink wrap it and put it on floppy disks or CD-ROMs and put it on stores. And so imagine what a revolution in the car industry would be if they went from warehousing stuff for up to a year and then selling it to selling the cars, you know, within a couple days of making it. And you would be like, well, of course they did that. What idiot would have made cars the old way? Well, here we are in the software industry saying, hey, look, we now write code and within, you know, in the same decade people are using it, right? So yes, we should be very proud that we’re doing things better now. Oh, and you mentioned Patrick Debois, who is brilliant and awesome, and I have to say the comparison is unwarranted because he, you know, he’s smart and has invented a lot of this stuff. I feel like I’m more of a journalist that has collected a lot of great things and writing it down. I think I just want to just, you know, deflect that comparison. I’ve learned from some really awesome people. I’m still learning and I’m still, you know, trying to gather that up and, you know, maybe I found a better way to explain things, but I think that’s just because I’m dumb and I understand how to communicate to dumb people and that’s my thing, man.

Matty: [00:34:23] You know, I was thinking about when you’re talking about what your focus with the book was and that’s a lot of like kind of our intent with this podcast. Like you said, you know, kind of when you’re, when you’re starting out on this DevOps journey or you’re doing this and you’re like, I don’t even know where to get going, right? Like, I don’t even— people are throwing around terms and they’re talking about stuff. And, you know, I wrote a blog post the other day that was about hipster DevOps, right? About hipster DevOps happens to the best of us, which is we don’t do it on purpose, but it can be very intimidating coming in because we have this jargon and all this stuff. And so I think that’s really important because You have to get kickstarted somewhere, right? Like, and so just the whole way you described what your intent with the book was, I’m like, oh, that’s sort of what we’re trying to do. I don’t know how well we’re necessarily always doing it, but you know, that’s—

Tom: Yeah, I mean, and we need that diversity of skills in our community. And technology has always been like this. There’s the, you know, top 5% that are inventing the new stuff, and then someone’s got to push it to the other 95%. And right now DevOps is— oh, I don’t wanna say we’ve gotten ahead of ourselves. We’ve accumulated a huge amount of good news to share to the world, and we now we’re seeing that part of the innovation cycle, as they say, where that’s getting pushed out to so many other people. And I’m really glad to see the DevOps community mature that way. I think a couple years ago, no one here on this podcast obviously, but there were certain people in the DevOps world who were talking as if DevOps is so good that no one else could possibly understand it, so we have to keep it to ourselves. And, you know, these enterprise people would never understand.

Trevor: [00:36:05] Those are the true hipsters.

Tom: That has disappeared, luckily, and now any talk you see on DevOps, it’s all about how we’re going to save the world by sharing this with the other 98% of the world.

Matty: Along those lines, I think, so Paul Reed gave his closing talk at DevOps Days Chicago last week. One of the things he talked about was enough with the unicorns already, you know, which was— I mean, we talked about that.

Tom: So I didn’t see the talk, but I’ve heard people use that expression now. So I’m wondering, unicorn can mean a lot of different things. What did he mean, that DevOps is only successful at unicorn companies like startups and we need to get rid of that attitude?

Matty: Yeah, he had a couple different things, and I’ll link to it in the notes. But one of the things was that he— the point that he made was, he’s like, you talk to the— it was actually funny, he like called out to the audience, like, who are the unicorns? And they named everything, and he flipped up his slide, and he was all over it. But it’s the typical, right? It’s the Etsy, the Amazon, the Netflix, whatever. And he said the first thing was, he’s like, you go talk to these unicorns, and they’re like, we’re not unicorns. Let me tell you all the things that suck, right? Let me tell you our challenges. Let me tell you our things. So A, But then he talked a lot about the whole— this idea of creating the culture of the other, the concept of the other, and about the othering and the divisiveness that the idea of these unicorns, you know, this thing. So that was— his point was, A, first of all, there’s not necessarily such a thing as unicorns in the first place, and B, when we create this division, it creates this idea of the other and it makes it feel like it can only— to your point, it can only be— it’s propagating the myth that DevOps can only be successful at a unicorn company, and the truth of the matter is there is no unicorn company anyway.

Tom: [00:37:44] Right. I mean, Todd Underwood gave a talk at last year’s Leesa. He’s a manager at Google, and he spent the first, like, 10 minutes of his talk explaining— well, debunking this myth that the problems that Google has, no one else has. And he said it so well, I’m not even going to try to repeat it. Which is an interesting flip on what you were saying. The problems that these unicorn companies have are universal, and if you think you don’t have them, think harder, because you do or you’re about to have them, because everyone is facing the same technology problems. They might just be dealing with dozens of computers instead of hundreds of thousands.

Trevor: Right, I mean, it boils down to, in a certain aspect, it’s that grass is greener on the other side of the fence thing. You know, oh look, Google’s talking about all these success stories and Amazon’s doing all these things, Netflix, Etsy, and you know, all, you know, you’re always thinking about the problems where you’re at just now. So, you know, you can get to these things, but don’t think that these other companies don’t have their own problems or the, you know, they don’t have the same problems that you do.

Tom: [00:38:49] Right. And all companies have the biggest problem, which is a problem I like to call people, you know, everyone— those damn humans. One of the first movies about computers going out of control was called The Forbin Project, and the computer eventually realizes that, you know, it’s been programmed to achieve world peace, and it eventually realizes the real problem here is people. So it basically becomes a dictator and takes over the— enslaves humanity so that they can’t cause problems for themselves anymore. So where am I going with that?

Trevor: Yeah, so humans—

Tom: yeah, damn humans. But those are the problems that we, we have to fix the most. I mean, that’s, you know, my second book was a time management book for system administrators because everyone that I talked to was saying that they don’t have enough time, that my first book was too long and they didn’t have enough time to read it. And I said, ah, time management, I should write that.

Matty: That book is on my shelf, by the way.

Tom: [00:39:51] Thank you.

Matty: Having read it as well, I’m not just saying it’s on my shelf.

Tom: And I’m terrible at time management, but I have all these coping mechanisms that let me fake my way through it. So the book is actually a list, you know, it’s what, 10 chapters? It’s all of my coping mechanisms, and people think I have good time management, but if you think about it, no human actually has good time management skills. If you think a friend has good time management skills, it just means that they have these coping mechanisms. So it’s really just a, you just need to learn the coping mechanisms.

Trevor: Yeah, I got to get through my uncut book list.

Tom: Well, it’s also a very short book. In fact, O’Reilly, when we handed in the last chapter, O’Reilly said, you know, we don’t generally publish books this short. We’re going to have to extend it. And I said, think about the audience. These people don’t have time for a big book. Let’s keep it at 120 pages or whatever it is. And, you know, the most important 3 tips are in the first like 20 pages. So if you haven’t read it yet, just read those first 20 pages.

Trevor: That’s just good organization.

Tom: [00:40:52] Always front-load. I found that’s— I try to front-load books and presentations I give. I recently attended someone else’s presentation and they saved all the good stuff for the end, and I was like, you could basically reverse the order of your slides and it’d be an awesome presentation.

Matty: So before we get into our checkouts, I just wanted to kind of wrap up with one kind of final question, which is Trevor had said, like, where’s a place to start with certain things? So what would be like one thing to keep, like if you say there’s one thing to keep in mind when your boss says, bring us to the DevOps?

Tom: I guess the most important thing is to figure out what the boss means that way, because you get in this situation where they say, you know, bring us the DevOps, and you spend a month doing something and you’re like, there, we’ve improved things, and your boss is like, I don’t see any difference, because they were looking for something else. So if the boss thinks that it means continuous integration or more modern ways of handling on-call or whatever. You have to kind of fish that out of them. It’s, it’s kind of like, you know, bring me the cloud, because to a lot of executives that means elastic computing. That means they know that projects keep getting delayed because it takes 6 months to get a new server, and they’ve heard that with the cloud I can get a server in 15 minutes. So what they’re really talking about is time to set up a new machine, which, you know, if your security department takes 3 weeks to approve your request, then it’s still not going to be super fast. But, you know, that means working on that kind of thing. To some people, the cloud just means data not stored on my machine, you know, like most consumers think of the cloud as this thing that backs up my photos and my music and syncs my calendar and stuff. So I guess that’s where I’d start. And then if they don’t know, then you get to insert what you want the DevOps to be, which could mean like, well, cynical Tom says, so take your biggest pain point and fix it, claiming that that would be DevOps. But non-cynical Tom would say, fine, start with, you know, what are the business priorities and turn them into, you know, measurable results, and then work towards, first of all, figure out an automated way to measure those things, and then start working on projects that improve those things. And you can now see if changes actually make improvements or if they just sound like they do, which I think is chapter 20 of our book. Sorry. Hey, since I’m plugging things, can I plug my— I’m speaking at a conference soon.

Matty: [00:43:25] Yeah, and you know, we also really didn’t properly plug your book. And we said the name of it but didn’t give like— I know you have a URL too, but yeah, plug.

Tom: Okay, well, the— so plug time. So the book is called The Practice of Cloud System Administration. It’s from Pearson, and I co-wrote it with Strata Chalup and Christina Hogan. And it’s basically mostly what I’ve been talking about tonight. I’d also like to plug the USENIX LISA conference, which is a big system administration conference. It’s November, what, the first or second week in November, usenix.org is the website, and I’m gonna be teaching 4 different tutorials there, all based on the book, and I’m also giving a couple other talks, so they’re working me hard this year. But outside of my talks, the grid this year is amazing. I think the LISA conference is a different conference than it was 5 years ago. So if you’ve heard that it’s overly academic or not hip to the whole cool DevOps thing, take a look at this year’s Grid. I think you’ll see it’s really evolved. One of the keynote speakers is Gene Kim himself. The closing keynote is by a woman who’s been studying the technology and the people problems around robotics at NASA. And drool. Yeah, she has some really interesting things to say. And just the committee And the chair, Cole, really just put together an amazing conference this year. So take a look. It’s November 9th through 15th, I believe, in Seattle.

Matty: [00:44:58] And 9th through the 14th.

Tom: 9th through the 14th.

Matty: Yes.

Tom: Those are my big plugs.

Matty: And that actually brings us nicely into our checkouts, which I think your checkout goes right into that. So this is the part of the show when we tell you something cool. So Tom, why don’t you tell us about your checkout?

Tom: This is what I think your listeners should check out. In a couple weeks, Stack Exchange where I work is open sourcing the monitoring system that we’ve been developing. It’s gonna be called Bosun, B-O-S-U-N, and there’ll be a big presentation about it at USENIX Lisa, and I’ll include the link in the show notes.

Matty: Great.

Trevor: Matt, you wanna do yours?

Matty: Yes, so I’ve got 2. The one checkout that I think is kinda cool is Jay Mondrala wrote a buster for Test Kitchen using The Pester framework, which is a PowerShell testing framework. It’s pretty neat. I mean, it’s very new. Matt Rock wrote a long and involved blog post that talks about, like, all kinds of the things. It’s got the longest title ever called Configure and Test Windows Infrastructure Using PowerShell Technologies, DSC, and Pester Running from Chef and Test Kitchen. I will post a link, but it’s pretty cool. It’s very in-depth if you’re looking at doing more Chef-y, Windows-y, test-y things. The other thing is my latest way of annoying everybody in the world. In iOS 8, you can have multiple keyboards. Yes, I know Android has had that for like 20 years. Well, now we have it in the iOS world. There is a keyboard for animated GIFs by a company called Riffsy, riffsy.com. It’s just a keyboard that basically searches animated GIFs. Now that’s the only way I communicate on Twitter and in text messaging now is animated GIFs. I’m sorry, everybody.

Trevor: [00:46:39] It’s okay. It’s awesome to talk in GIFs. You know, iOS may have just gotten multiple keyboards, but you guys still got the first GIF keyboard. So I guess that makes it my turn. There was a post on Jeremy Morgan’s blog which I saw on Reddit about the great unicorn hunt, and this time we’re not talking about companies, we’re talking about people. So it’s about the different ways that we kind of fail to hire people by, you know, the ways we filter people out and how maybe not all those are valid. And maybe we need to be more inclusive on some level of everyone and not looking for the rock star necessarily. That was neat, which is also funny because today I got an email from CodeProject, which I’ll also put in here, which I clicked on. I don’t usually click on those emails, but this one said the war for, like, the developer war. And I was like, what am I, what am I going to war against? And it wasn’t, it wasn’t actually, it was actually from a CIO’s perspective. Or a CTO. I forget at this point because it was this morning and the day has gone by. But like the war for finding all this talent. It was interesting. And finally, today was the launch of Borderlands: The Pre-Sequel. Don’t tell my best friend, but I played to level 5 as Claptrap and it was hysterical. And that’s it.

Matty: [00:47:56] Great. So just a reminder that we have a newsletter, arresteddevops.com/bananastand. It is the best way to know about upcoming podcast episodes and cool news with DevOps.

Trevor: Thanks to our sponsors, VictorOps and Red Gate, and to our loyal listeners. If you enjoy Arrested DevOps, we’d appreciate it if you would visit arresteddevops.com/itunes and leave us a review in the iTunes Store. As you heard today, we will read the reviews when you post them, no matter what they say. Thanks, Tom, for joining us tonight. Everybody, please check us out at arresteddevops.com. Or @ArrestedDevOps on Twitter.

Matty: We are always happy to get your input, ideas, feedback, or animated GIFs at shows@arresteddevops.com.

Trevor: I’m Matt, @MattStratton, and I’m Trevor, @TrevorGHess.

Matty: We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.