Matt Stratton: [00:00:07] Welcome to Arrested DevOps, Episode 11: DevOps at Etsy— not a unicorn, just a sparkly horse. I’m your co-host, Matt Stratton. I’m @MattStratton on Twitter.
Trevor Hess: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.
Matt Stratton: Arrested DevOps is brought to you by TenthMagnitude, a cloud services company that believes DevOps makes their clients more awesome. You can check out their 360-degree cloud services at 10thmagnitude.com.
Trevor Hess: Tonight on Arrested DevOps, we’re going to go take a flash to the past. Anybody who’s listened to our first episode, you might remember Matt saying something like this.
Matt Stratton: We’re going to really try to steer clear of name-dropping, so you’re not going to hear me say, you know, when I was talking to John Allspaw at Velocity.
Trevor Hess: Matt, do you remember saying that?
Matt Stratton: It’s possible.
Trevor Hess: Well, it just so happens we’re talking to the ops team at Etsy today, who whom John Allspaw is a member of. So that’s going to be a lot of fun. But before we get started with that, I just got some personal news. I’m actually going to be joining, or have already joined, the 10th Magnitude team. So I’m going to be working in the cloud with DevOps and with Matt, and I’m really looking forward to it.
Matt Stratton: [00:01:18] Yeah, I’ve got some news too. So I swear this is not because Trevor is joining 10th Magnitude. But I am actually leaving 10th Magnitude and I will be joining Chef. Sad to leave 10th Magnitude, but leaving them in capable hands with Trevor and real excited to go join Chef. This episode is sponsored by PagerDuty. PagerDuty helps businesses increase their reliability by connecting people to their global operations with actionable intelligence. PagerDuty makes it easy for engineering teams to manage the entire lifecycle of their instance in order to decrease things like resolution time and improve quality of life for everybody who’s on call, which as someone who’s been on call before, I dig. PagerDuty is trusted by Nike, GitHub, thousands of other companies, including Etsy. And PagerDuty helps keep the systems you’re using every day online by making sure the right people know at the right time about issues so they’re resolved quickly and before they become major customer-impacting failures. You can check them out at pagerduty.com. To quote Bridget Kromhout on Twitter, she says she likes to level up her game by learning from Etsy. We were gonna make that our title, but John Cowie from Etsy said that was far too unicorn-ish, so we went with sparkly horses. And let’s get started. First on our panel, we have John Cowie. John, would you like to introduce yourself?
Jon Cowie: [00:02:37] Hey everyone, my name is John Cowie. I’m a senior ops engineer on the team at Etsy, and I resolutely maintain we are not a unicorn. We have the same problems as everybody else does, but what we’re gonna try and I guess, talk a bit about today is how we deal with them and how we do some of the engineering stuff we do. So I’m kind of— I guess my primary jam is Chef stuff. I’ve written a lot of our Chef tooling at Etsy, and I’m working on a Chef book for O’Reilly at the moment, which will hopefully be coming out soon.
John Allspaw: My name is John Allspaw, and I manage the infrastructure and operations organizations at Etsy, which is roughly about half of the engineering org at Etsy.
Pete Bellisano: My name is Pete Bellisano. I’m a web operations engineer and newest member of the team.
David Yurkiewicz: And I’m David Yurkiewicz, web operations engineer. Been roughly here for a year and, yep, loving every moment here.
Matt Stratton: [00:03:38] So for our listeners who don’t know, Etsy really does enjoy a reputation in the industry for being a shop that gets DevOps, if not really setting the standard. So I’d like to know, guys, why do you think you guys enjoy that reputation?
Jon Cowie: So to kind of answer that, this is tailing on from the discussion we had on Twitter the other week about when somebody mentioned Etsy was a unicorn. There’s kind of a bit of a— there are a couple of companies, of which I think Etsy is one, who tend to be considered unicorns in the industry. But what I was saying on Twitter, what I believe, is that I think calling Etsy a unicorn is a bit of a misnomer, because that implies that we somehow have magical powers that other people don’t. And really, I think, I guess the secret to the way Etsy does operations is not that we have magical powers that people don’t, or we have these innate abilities that no one else has. We have all the same problems as every other company out there does. We have servers that break. We have software with bugs. We get paged in the middle of the night. I think if I could put my finger on it and say anything that lets Etsy do what we do is that we have, you know, from the CEO down, IT and engineering is recognized as a core competency of the company. And we’ve put the resources and practices in place to make sure that— I mean, basically we’ve solved quite a lot of the easy problems. So the combination of those lets us be more proactive than reactive. So we’re spending less time firefighting and more time, you know, doing things, you know, like we get to speak on podcasts and we have time to contribute to open source software because we aren’t— I guess we aren’t spending quite so much time fighting fires because we’ve put in the work and the resources and the headcount to make sure that we have the space to do, you know, proactive stuff as well.
Trevor Hess: [00:05:28] What would you guys say would be like the short version, the SparkNotes of DevOps at Etsy?
David Yurkiewicz: Well, I mean, I would just say pretty much each company kind of does it differently. I think one of the greatest things here is how it’s leveled between different teams and between operations, and we have all developers, and it’s how seamless we interact with each other. The fact that if someone’s working on something, there’s really no barriers between Collaborating between each team, I think that’s probably the biggest thing that we can probably say is the seamless interaction that happens and how it all works together real nicely. We have people— we have specifically to our search stack, or even in our own— we have in-depth tools, and we have different teams. If we have a meeting for a new feature that’s coming, we have that collaboration. It kind of naturally, organically happens. And I think that’s what actually just makes it work so well.
Jon Cowie: [00:06:46] I think an even shorter version is we— I had this discussion, I think it was with Matt at ChefConf. We trust our colleagues, and we communicate with them. I think those 2 things pretty much underpin the entire thing.
Matt Stratton: What’s been the history of DevOps at Etsy? So we see where you guys have gotten, but kind of what did you see as some of the transformative things that you had to do or that were necessary to make it be successful?
John Allspaw: It’s a good question, and I think that the A answer would be that there wasn’t a single concerted effort. It’s not like we— when I first got to Etsy, there was a bunch of the infrastructure, a bunch of— actually, basically, somewhat of a reinvention of engineering as a discipline as a whole. But it’s not like we had to do some campaigning. It’s not like, well, there’s this thing called DevOps, and let’s have some lunch and learns about DevOps. There wasn’t a lot of marketing that needed to happen. In fact, it still is very, very weird to hear the word DevOps. If you hear— if somebody says DevOps in the office, it feels very weird. You’re probably referencing something that someone said on the internet, and you wouldn’t hear it in your colloquial— in regular conversation. It just feels weird to say. So I wouldn’t say that there— I can’t boil down to like a pithy, well, here are the 3 or 5 or 10 or 100 steps that we went with. I think what it meant was that decision-making needed to recognize domain expertise and finding a sort of shared mutual understanding of various problems that we needed to solve, but in that high-trust way that Cowie pointed out. I realize I’m being vague, but outside of specifics, there wasn’t much other than that and don’t be an asshole. Really, that was put in place. How’s that for an answer?
Matt Stratton: [00:09:20] I think that’s great. I think the thing that resonates with me was I love the fact that you guys don’t use that term. Like you said, it’s weird, right? It’s just sort of business as usual, and if other people have put that name on it, awesome, but whatever, this is just how we work.
Trevor Hess: It really helps to underpin the concept as, you know, a series of concepts and best practices so much as quantitative thing that, you know, you know, there’s the same reason everybody gets— who really believes in the concept and the best practices gets angry when people say we need more DevOps, we need more, you know, we want to buy DevOps.
David Yurkiewicz: Yeah, it’s like a— it’s a buzzword. It’s like in the— in our industry, we have a lot of buzzwords that kind of like come in and like either they’ll, they’ll turn into like these marketing terms and like it’s sadly that The whole movement has turned into, like, oh, well, hey, we’re marketing and we’re giving you DevOps. You can have DevOps this way. That’s why I kind of agree with John. It’s kind of a joke if you say, oh, well, let’s— No one says, hey, let’s do DevOps today. That’s just kind of—
John Allspaw: [00:10:26] There’s something about that the word is underspecified. That’s all. It’s that it’s no different than You know, it’s no different than cloud, it’s no different than— but it’s no different than compliance, it’s no different than scalability, it’s no different than resilience. I think that part and parcel of getting good engineering work done is paying attention to language, and engineering as a whole generally has a low tolerance for things that are either underspecified or or don’t have enough specification to be useful in conversation. So you wouldn’t say, oh, well, we need more— oh, let’s take a look at this architecture. Okay, well, we need to store information in a database, so we need to maybe cache it, and then we need to retrieve it. Okay, well, you never say— somebody would never say, oh, well, I think what we need here is more robustness, so let’s just add some more robustness to it, right? That would be the start of the conversation, not the end of the conversation. And so, like, the second question after that is, okay, what do you mean by— like, agreed, it needs to be more robust. Now let’s talk about what that means, because we need to sort of unpack it. That’s the reason why it’s just sort of a waste of time to say DevOps, because if the real thing that you want to get at is, hey, this revelation of of something new or a departure in the way we wrote an alert or something along those lines, you wouldn’t— you would just say, oh, hey, here are the reasons why this is surprising to me. I saw this design document or I saw this metric and it went a little weird and that was surprising to me. You wouldn’t say, there’s not enough DevOps in this graph, or along those lines.
Jon Cowie: [00:12:51] You see what I’m saying?
Matt Stratton: Yeah, and I absolutely agree. I like the idea that, you know, we can use a common term as a starting point of the conversation. And I think that— and I also really, really agree, David, with your point about how it’s like we have these buzzwords and they’ve been latched onto from a marketing perspective. And it’s— to me, it’s a blessing and a curse. I mean, the good thing is it’s getting the idea out there, so it gives us a shortcut to start a conversation. But then, like, John, like you said, when it becomes the end of the conversation, then you haven’t accomplished anything. So I really do struggle with the word in that way. But I think once you’re doing work, you know, then it doesn’t matter as much.
John Allspaw: Yeah. And it only becomes problematic if you end up being satisfied with with leaving it there, right. It’s no different than the terms human error or situation awareness or complacency. These are all just— they’re fine words, they’re just underspecified, that’s all.
Matt Stratton: [00:14:09] My understanding is that, you know, Pete, you’re certainly not new to technology, but kind of this is a different type of a role for you. And I’d love to have a little bit of background, like where you came from, at least philosophically, and what you’re seeing is different being at Etsy.
Pete Bellisano: Absolutely. Actually, about 10 years ago, I was in IT. I actually worked with some of the guys that are here now. And actually, I left to start my own business totally outside of technology. I did that for about 7 years, and I actually became a fan of Etsy during that time, not aware of the fact that I still had my friends that were working here, guys I used to work with. I missed the technology, so I got tired of being my own boss, and I actually joined the team of Corp IT in February of 2013. Did that for about a year, and I just joined the ops team in January. So the transition has been amazing. One of the things I guess John and David were touching on, and John Allspaw too— sorry, there’s 2 Johns— was that the thing that made it possible was the culture that we have. And from my perspective, that’s the reason why I’m here, both from seeing it outside and also being a part of it internally. The culture is very nurturing. There’s a lot of collaboration. There’s really no limits to what you can do if you have the drive and the want to ask a lot of questions, because everybody’s more than accessible. Cool.
Matt Stratton: [00:15:42] And then, you know, David, I understand you’re relatively new to Etsy, but you’ve been in this web ops world for quite some time. So any thoughts that you have about things that you’re seeing that are really done differently where you’re at now?
David Yurkiewicz: Yeah, I mean, sure, definitely. I mean, I’ve been doing IT, regular system administration for quite a while, and it’s one thing that I find interesting here is that, like what Pete touched on, is culture. And you can’t really buy culture. You can’t really just kind of get a consultant to come in and like, hey, we’re going to change how we do things, and we’re going to do either DevOps or whatever you want to call it. Here, it’s actually the culture actually enables the openness to have people to learn a lot and touch new technologies. And actually, like I said earlier, one thing that I love about this place is the natural way that we kind of solve problems together. We don’t have to be in the same team, but how cross-teams, they work together. And like I said, it’s organically, it just happens. It’s very natural, and like what Allspaw said, it’s pretty much like, everyone’s— it’s pretty much don’t be an asshole. We’re solving things together, and we kind of actually like who we work with, so it works really well. In other places that I worked, we were working with similar technology, but there were a lot of limitations. There was a separation between the system administrators and the operational people. And developers and DBAs and stuff like that. So that segregation hindered the culture, hindered a lot of the projects, a lot of the things. If there were new ideas where you needed to take a DBA or a certain developer for something else, it would hinder that. Here, if you have an idea, I can go to a DBA, hey, what do you think about that? And then, hey, I need help with frontend stuff. I can talk to people in frontend. And so forth. So it’s that natural collaboration that happens here because of the culture. It’s what makes it pretty awesome here.
Trevor Hess: [00:17:59] So if I’m understanding what you’re describing of the culture, you guys have a very open workspace. You don’t separate developers from ops from IT. Everybody kind of works in the same space, I’m guessing? Or could you put some more definition around it?
Pete Bellisano: There’s a lot of cross-pollination.
David Yurkiewicz: Yeah, I mean, we have a lot of that. We have teams, of course, specifically under the operation infrastructure umbrella. We have specific teams, or like I said earlier, DevTools, we kind of deal with tooling internally, core, and some that work with the search infrastructure and performance and stuff like that. So we have those teams kind of separated, but as far as, like, you know, it’s open space, like, for example, in our office, like, I sit pretty much with all the people in operations, but I have corp IT sitting next to us. John sits not too far from us, so it’s a very open space. If I have a question, I can just get up and like, hey John, what do you think about this graph? Or John might come, hey, I like that graph. Can you tell me what’s going on here? It’s very open as far as the office. It kind of enables that cross-working experience, I’d say.
Jon Cowie: [00:19:19] But then on the flip side, I mean, so I actually live in the UK, so we have a lot of remote employees as well. I think one of the most powerful, I guess the most powerful thing about Etsy’s culture is that what David was just describing there about the office culture, cross-pollination between teams and stuff. It is an open-plan office, but I don’t think the communication is a result of the layout of the furniture in the office. So the same cross-pollination happens with me as a remote employee. I mean, I work 5 hours behind and several thousand miles away from most of the people I interact with on a day-to-day basis, but there’s still the same— albeit on video chat or IRC or email— there’s still the same cross-pollination of ideas. Kind of intermingling of ideas within teams and all that sort of stuff, you know, within remote teams as well as sort of in the same geographic location.
David Yurkiewicz: Pretty much all the communication, most of it happens in IRC, so we have all these channels where all the general communication happens, and, like, I could be talking to someone that’s right here in the Brooklyn office or someone that’s, like, in the West Coast or, like, someone like John that’s in London, and it’s very seamless. It works really well, and I think that actually helps.
Matt Stratton: [00:20:32] So you talk about the culture. The culture’s awesome. And again, I mean, I think we could probably agree that culture’s the hard part, right? The tools are easy. But when folks come in to start working at Etsy, do you sometimes see them get kind of a culture shock if they’ve come from a place that doesn’t have this level of trust? I mean, I think about, I’ve been doing sysadmin for a long time, and I always would make the joke that your best sysadmins are super cynical, ’cause their job is to assume that everything’s gonna break, and that sometimes corresponds into that they don’t trust people, because, and I don’t know, I just kind of, I think about like a lot of the sysadmins I’ve worked with and could see them coming to Etsy and be waiting for the other shoe to drop the whole time, being like, what’s the catch? This sounds too good. Or developers coming in and having a different approach. Do you see that happen?
Jon Cowie: I might grab this one, because I’ve actually been in exactly the position you’re describing. So my background before Etsy— I’ve been at Etsy about 2 and a half years now. Before Etsy, I always worked at very, very small startups where I was the only ops person, the only sysadmin. And I developed a very— I guess you would call it a very sort of B-O-F-H attitude. Like, I am the sole— the infrastructure is mine. I’m solely responsible for it. Nobody else can touch it or they’ll break it. So for me, when I joined Etsy, one of the— it wasn’t quite a culture shock. It was more of a— I had to adjust the way I was thinking. Because whereas in the companies I was used to working for, if a server went down, I would have someone screaming at me on the end of the phone. I would say one of the biggest adjustments I had to make coming to work at Etsy was this idea that— I mean, largely thanks to people like Allspaw and Mike Rambetsy and I— who was my direct boss at the time, promoting this idea that, like, essentially shit happens. Like, things are going to break. With all the will in the world, you cannot defeat human error. So rather than yelling at people for when the inevitable happens and something goes wrong, let’s try and figure out, you know, how to learn from that. And, you know, okay, a mistake happened. What were the assumptions that led you to do what you did? How can we avoid that next time? Can we improve our monitoring? So I think for me, it was this thing I was saying before of trusting your colleagues and communicating with them, particularly on the trust side, given I was used to being the guy who got yelled at. I’d say the biggest adjustment was— it was a nice adjustment to have to make, just this idea that all of a sudden I’m not the only person interested in keeping the systems running. Like, developers own their availability too, and, you know, there’s developers on call I can wake up if something goes wrong with the code that I can’t fix. And there was definitely an adjustment to make, but I would say it was definitely on the good side.
John Allspaw: [00:23:17] One thing that I wanted to say, yeah, it’s great to hear all of these other guys, including exactly what Cowie just said, because it helps validate my own sort of thinking, at least my hopes when I first got here. Inherent in your question, you had mentioned, okay, so the culture is awesome, and the question was around whether or not there’s culture shock when people come in. Is the culture awesome? I think the question has a premise that there is something to be discerned. And I think that there’s probably a fallacy in there, that the culture is somehow— I’m not going to dismiss the idea that an organization has inertia and culture, I mean inertia and momentum, but that’s on behaviors, not on culture. I think one of the things that we do well is not that we just indoctrinate people, like we scare the assholeness out of people. I think that instead we turn our focus from— instead of preventing a bad culture, it’s by making misunderstandings easily repairable. Faulty information and misunderstandings, if you can’t detect when they’re happening and repair them quickly, that’s what allows the practices to sort of drift from where you expect them to be. And bad cultures happen when practices and behaviors drift from where you expect them to be. We’re here in New York, us 3 are here in New York, and if you were to ask a New Yorker if they think that people in New York are terrible drivers, they’re not going to say yes. They’re going to say they’re just drivers. Whereas if you ask the same person who comes from Kansas City to drive in Manhattan, or at least be in the back of a cab in Manhattan, they’re going to say Those people are fucking crazy. And so there’s a normalization that happens, and it’s not just a normalization of don’t be an asshole, because that sounds like more of a preventative measure. Instead, if we just eschew the idea that we’re gonna set it and forget it, and we make it incredibly easy for people to be like, hey, you said that thing in IRC, I think it sounds like you’re really frustrated because you had a really shitty day. That’s totally happened to me. I want you to know that you’re kind of being a dick. And so, and, and the other person hearing that can say, oh, that sucks. Sorry about that. Thanks for telling me. That’s, that’s what we’re talking about. There’s, there’s, there’s no like special chip that we can stick in the back of somebody’s neck that says that you’re going to be a nice person and be Buddha-like in your abilities to sort of bring your best self every day.
Jon Cowie: [00:26:55] You should tell the NFS over WAN story. That’s a good example of this.
John Allspaw: Oh, yeah. I completely jumped down Cowie’s throat. This is at least maybe about a year ago. He made a suggestion. He made a suggestion in full enthusiasm. He was helping, wanting to help solve a problem. And I think my exact words are, that is the dumbest idea that I think I’ve ever— that we could possibly go with. And I was on a video, and I was on a conference call with way more— way, I think, even larger than the operations team. I wasn’t— I was an absolute asshole to John Cowie, and I didn’t realize it at the time. Actually, it wasn’t too much longer after that, but my team made it very clear, hmm, that probably didn’t come out the way you expected it to come out, and they’re right. And so being able to repair those things—
Jon Cowie: [00:27:58] you have to bear in mind, like, in the organizational chart of things, Allspaw. Well, at the time, Allspaw was my boss’s boss. But Allspaw is now my boss’s boss’s boss. So it’s— in some companies, somebody— I mean, Allspaw is a senior vice president. Some companies, that level of person, you can’t interact with like that. You can’t say to them, hey, you’re being an asshole. Or hey, that didn’t come across the way you said it. Because they’ll probably fire you. So it’s a pretty cool thing that we were able to say to Allspaw, like, that really didn’t come across quite the way you intended it. And Allspaw was like, You know what, you’re totally right. And now we’re telling a story about it, and I still work here.
Matt Stratton: Well, and you know what is an interesting thing I’m thinking about too when you said that about saying, oh, in another company if you tell an SVP, hey, you were kind of being a jackass, he’s probably going to fire you. And part of me thinks like, no, you know what, he probably isn’t, but you’re going to assume that if you did he would, so you’re not going to do it. And it’s that thought, like I was in an organization once where when we were trying to do a lot of this automation stuff, and put some good reliable software release in place, and one of my sysadmins said, well, what happens when the CTO comes running down the hall yelling and screaming, get this goddamn release out now, I don’t give a shit. And this was always his answer, and I said that to our CTO once and she goes, when have I ever done that? You know, but it’s this underlying thing that is not necessarily based upon real examples of that, but the other stuff around it makes you think that’s where it’s going to go.
Trevor Hess: [00:29:26] Right. That’s— I think the behavior and the culture are somewhat cyclical. They have to support each other. Yeah. Because you can have the intuition to, you know, to want to, you know, be expressive of your own emotions and say, hey man, what you just said was not cool. But if you have the bigger emotion of I’m fearful for my job if I say something, because the perceived culture is that, you know, you’re going to lose your job.
Matt Stratton: I would love to hear a little bit about what your infrastructure looks like. I know one of the big things that usually shocks people is that you guys are primarily bare metal, or at least the last time I heard this, that was the scoop. That was at ChefConf when Cowie turned to me and said, I might be interested in looking at some cloud stuff, even though it doesn’t matter for Etsy because we’re all bare metal. So what’s it look like? At the low level for you guys, for the geeks out there?
Jon Cowie: That is still the case. Our infrastructure is pretty much entirely bare metal. We do use S3 for some image storage and some backup stuff. I mean, a large amount of the reason for that is not we’re averse to the cloud per se. It’s that we do a lot of capacity planning. I mean, Allspaw literally wrote the book on it. We do a lot of capacity planning and things like that to plan. We understand our seasonal traffic patterns quite well. We know what our traffic does at each times of the year, we capacity plan accordingly. And honestly, we use the capacity on the physical hardware we have. So our main code base is a monolithic PHP application. We run the LAMP stack— Linux, Apache, MySQL, PHP. And for example, we understand the workload of our applications. So this isn’t the case if we had a physical web server. We’d only be using 5% of the CPU. And, you know, we could just save a ton of money by virtualizing and running 10 VMs on the same box. The physical hardware we have, we have to fill our capacity requirements, and we use it accordingly, and we actually utilize the infrastructure we have. So it’s partly a case of, for us and our particular workload and our particular requirements, I don’t think it doesn’t make commercial sense to move it, for example, to EC2.
Matt Stratton: [00:31:44] What do you look at when you’re— from a software delivery practice, could you speak to that a little bit?
Jon Cowie: We do do a lot of continuous delivery. We don’t actually deploy the Etsy.com codebase with Chef. We have a separate tool we developed called Deployinator, which we deploy. Essentially, Chef is everything under the application code, so the operating system and all the software that supports the application. So we’re big on continuous delivery. We’re currently deploying somewhere in the order of 50, 60 times a day, I think. And largely the ethos is make small changes often. So if something goes wrong, we recover quickly. We have generally when new code is deployed, it will be behind a feature flag so you can turn it off quickly or you can gradually ramp it up. Our entire kind of process and way of thinking is built around small, rapid changes. And although we don’t deploy the application code base with Chef, that’s kind of transferred to the way we think about Chef as well. We tend to make small, rapid Chef changes much more often than, you know, making one massive Chef change once a month. We have somewhere in the order of probably, I think the last count was somewhere around 40 people who have knife keys and can, you know, make changes to the Chef server. And we’re talking, I think, a couple hundred chef changes a month, less than code deploys, but we make a lot of them. And it’s just the same ethos of small deploys fast and often. We have the monitoring and, you know, the metrics in place so that when something does go wrong, we can find out quickly and we can recover quickly and move on with our day rather than it being this massive scary event when a bug, you know, a bug is shipped to production or, you know, we make a cookbook error or something like that.
Matt Stratton: [00:33:29] We talked about how a couple years ago, Cowie was on a similar episode of Food Fight, and we’ll put a link in the show notes. I recommend listening to it. It was a lot of fun. And I’m curious, Cowie, what’s changed? You’ve been at Etsy. You were a newbie to Etsy at the time, and now you’re the elder statesman, right? So what’s—
Jon Cowie: By elder statesman, I’ve been at Etsy 2 and a half years, so it’s not exactly like I’m a seasoned pensioner. Or anything. I think for me personally, I mean, I had been at Etsy roughly probably only a couple of months at the time I did the Food Fight show. I think one of the biggest changes has been the company as a whole has pretty much doubled in size since then. Back just after I joined, the company was somewhere around 250, 300 people, and we’re now close to 500. And the engineering team has doubled in size, and correspondingly, the amount of infrastructure we have has gone up massively as well. So it was kind of interesting for me seeing how— because when I started, I was obviously still quite cynical of things, and I was thinking, you know, Etsy has all this nice culture and stuff, but surely it’s not going to last. As the company gets bigger, there are going to be procedures and compliance and all this stuff that’s basically going to suck the joy out of the job. So it’s been quite interesting for me seeing how that wasn’t the case and how As the company’s got bigger, you know, there have been challenges. For example, I mean, the part I’ve been most involved with is our chef workflow. We’ve had to evolve certain things. We’ve had to adapt, because as the company gets bigger, different challenges arise. But the interesting thing for me has been that how with that kind of culture Allspaw was talking about, about misunderstandings being easily fixable, and this kind of continuous delivery and communication and trust thing has been how we’ve, you know, we’ve come up with these challenges, people have sat down and been like, okay, this needs fixed. So one example that particularly relates to me was KnifeSpork, one of the tools I wrote. We use for all of our chef stuff. That’s our kind of chef workflow tool. And a while ago, probably about 6, 7 months ago now, we were sat in the ops meeting and it turned out the way we were using Spork kind of didn’t completely work. Like, there is— as we got more Chef users, it was beginning to kind of slow people down slightly, some of the hoops we were having to jump through. And it was an interesting experience for me, because whereas in a previous company I could see me like, no, no, no, you’re attacking my process. This is change. I don’t like change. It was being able to— I guess it was having gained the perspective that comes from working in that kind of culture to be able to take on board what people were saying. I’ll be like, okay, I can see what you’re saying. This does need change. And rather than being defensive, it’s like, okay, let’s look at how to change this and make this better. And I’ve seen that happen time and time again with different challenges that have cropped up as Etsy’s got bigger, and that is largely the way it happens. If something comes up that needs change, then you go and speak to somebody or the manager in charge of that bit of the company or whatever. You just have a discussion and talk about, well, this is what’s happening. How can we make this better? Everybody seems genuinely interested in advancing the state of the art in what we’re trying to do. There’s no sort of defensiveness or empire building, like, no, you can’t touch this. This is mine. You’re eroding my power, sort of thing. So it’s been interesting seeing as the company has basically doubled in size how we’ve managed to keep that culture. That’s not saying we’ve stayed the same. Like I said, there’s been a lot of changes, but we’ve been able to kind of utilize that culture to move through all those changes and end up where we are today.
Matt Stratton: [00:37:16] Do you think that this was— this philosophy that we keep talking about, this openness, this collaboration, and like you said, this actually giving a shit, is kind of inherent to the DNA of the company, like from the beginning? Was there some type of— and also that’s probably a little more towards you because you’ve got more historical bent on this. Like, was that almost from day one, or was there an adjustment that kind of happened? And I’m thinking for people that are wanting to make this type of transition or transformation in their organization today.
John Allspaw: It’s a good question. So I’d say sort of piecemeal. I know a lot of people say— a lot of people have asked me, okay, well, How do I do this? I want to do some of that stuff. How do I do that at my company? Usually what I say is, I don’t know. I was in charge, so I don’t know how somebody— I haven’t been in a position where I didn’t have the requisite authority to sort of make those changes, or at least it’s been a very long time since that was the case. Actually, 10 years ago, that was the last time that I wasn’t in a position of authority that could make material or at least large-scale changes in practices. Having said that, even so, having learning reviews or after-action reports, or at least I think what we generally refer to as postmortems, that was something that we introduced, that I introduced here, and the requirement was that it needed to be filled with diverse perspectives. And sort of the tenets of what are now known as blameless postmortems required having diverse perspectives in the room. It’s essentially a debriefing, and that helped. It helped a lot because then people from all over the organization would be able to sort of give their own narrative and then add it to others’ perspective, not just in a technical realm, but finance and legal and customer support and fraud detection and design, all of those types of groups. So that made for a decent place for people to try to at least attempt to understand other people’s perspectives. The other thing that is just very, very— it’s sometimes easy for me to completely forget— is when developers deploy their own code, there is there’s a bunch of things that can happen. It means that development cares much more about how their code is running in production, and then they are challenged by themselves to come up with answers to questions that they may not know. So those 50 or 60 times a day, or however many times a day we deploy, I’m not sure. I used to say that we’re not so secretly turning software developers who come work at Etsy into ops people. We don’t even— I’m not even— I don’t even know how many times we’ve deployed because I haven’t been— even when I was on call and was working on tickets when I first got here. You guys don’t know how many times today?
Jon Cowie: [00:41:25] No.
David Yurkiewicz: There’s a dashboard right over there.
John Allspaw: There’s a dashboard we can find out, but It’s because ops is too busy. So many people have said to me, well, if developers deploy code, then what do operations do? Like, are you joking? If operations isn’t tasked with shepherding or chaperoning or releasing or anything like that, there’s an immense amount of proactive work. I don’t know, so I think that when that happens, you get developers having that— it’s almost sort of like, you know, not forced empathy, but they don’t have to simulate. They are taking the perspective of an operations engineer, and I think that’s hugely motivating, and it brings a huge amount of creative problem solving. To the table that wouldn’t ordinarily have happened if operations were in the critical path of that. Does that make sense?
Matt Stratton: [00:42:33] It does, and actually it was funny that you mentioned that when you said, hey, if ops isn’t deploying code, what the hell are they doing? And when I was running tech ops at another e-commerce site, that was exactly what came up when we were talking about a lot of automation. The sysadmins on my team said, well, wait a minute, if this is all automated, then what are we going to do? I said, we’re going to do cool stuff rather than copy files around all the time. You know, I mean, there’s all these things that, like you said, you don’t— you never get around to because you’re doing all the stuff that either developers could be doing or even better robots could be doing. Yeah.
Trevor Hess: And from my perspective as a developer, it’s absolutely, you know, if somebody— if there’s an ops team able to think for me, And I have no motivation to try and solve that problem. But if I’m included as part of that process, then it is my responsibility. I have to take that challenge on.
David Yurkiewicz: Yeah, and that’s one cool thing we also do here is that we have what we call designated ops people for different teams. So, like, if you can be assigned to, like, I don’t know, somewhere in the homepage to with the homepage team. So the philosophy there is pretty much to help them out if they have anything like ops-related questions, anything with setting up new Nagios alert checks or stuff like that, or graphs. But the idea, the main idea is actually to show them how to do this stuff so they can actually bring that to their team, so they enable them to actually, you know, if you want to do Nagios checks, you know, send metrics to Graphite and start graphing stuff, like, we’ll show you. We’ll hear you. We’re not, you know, we’re not trying to hold anything, you know. Let us show you so you can actually do it, and actually, we’ll be there to help you out, too.
Jon Cowie: [00:44:28] One of the more interesting examples of just what David was talking about there is in our on-call rotation, we actually have a member of our DevTools team who is a, like, by by occupation a developer. He’s not technically part of the operations team. He was sufficiently interested in that side of operations that he requested to be in the on-call rotation and has been, I think, for the last year.
Matt Stratton: Somebody asked to be on call.
Trevor Hess: Wow.
David Yurkiewicz: I thought you guys were doing something. Yeah, it’s kind of shocking.
Trevor Hess: So what kind of— so Matt wrote this question. I don’t know if I would have used the term mud pies, but what mud pies did you guys make that might help other organizations learn? So, you know, what were your missteps that you guys became aware of that you solved? John, beginning of the episode you mentioned, you know, there are these little— you guys have been solving these little easy problems. Could you maybe touch on a couple of what those problems were?
Jon Cowie: [00:45:33] I guess, so one we’ve been focusing quite a lot recently, one of our other relatively newer members of the team, Ryan France, his middle name has become Context. Ryan Context France. And his whole thing, what’s got him really excited at the moment, is taking our Nagios alerts and adding context to them. So we still get paged when things happen. We’ve put a fair amount of effort into trying to make sure that when we get paged, it’s for something we actually need to deal with. So that was kind of step 1 of the process. Step 2, what Ryan’s been working on really, really hard, is when you get that page— we’ve all probably seen Nagios pages that just tell you, oh, disk space on the server’s broken, and you get that big long line of meaningless disk stats. What Ryan’s basically been working on is— so when you get that alert, it will say, OK, disk space on web200 is warning. Then it will give you a ribbon graph of all the partitions. And you can see which ones fill up. Then it will give you the ganglia graph of disk space on that box. So you can see if it just tipped over the threshold or if it’s massively gone up really, really quickly. And then it will give you a sensible breakdown at the end of exactly how far over the threshold it is and where the threshold was, like what percent was the threshold and things like that. So when you wake up— You can fit— when, you know, 4 in the morning you just got paged, you can just look at that email and you can see, oh hey, this is— it gives you the information you need to be able to make the decision on what you have to do next rather than having to wake up, get to your laptop, turn it on, log on to the VPN, log on to Nagios, then try and figure out what you’re supposed to do. Because it might be the case that you can just act the alert by email if it’s just tipped over the threshold. You can act the alert by email and fix it in the morning. If you have to get out of bed and go to your computer to figure out that first, it just makes the experience much more sucky. So it’s things like that. It’s actively in our interest to do this stuff. But it’s only by spending time on it we’re actually able to solve this problem. It’s not going to disappear by itself. So I would say in terms of quality of life, certainly, that’s been one of the biggest improvements that we’ve been able to do.
David Yurkiewicz: [00:47:48] I mean, I think another one that probably we kind of are spoiled on and we overlooked a lot is our host building process. It’s so automated that at my last company, we used to spend so much time trying to build multiple hosts. We have so many automated tools. We have pretty much a host building stack. Pretty much we can bring up a server in a matter of minutes, 5 minutes or so. Little stuff like that where, hey, we’re thinking of getting a new data center. All right, we don’t have to worry about building 100 servers. We’ll just run a few commands and they’ll start building up like that. That’s another big one, I think, that we solved. We overlook it a lot because it’s so simple for us, but it’s very time-consuming if you have to do all the manual steps of Then the RAID configuration, then actually, you know, PXE booting the server. We have it right down to like, you know, I think it’s, I want to say, like 5 to 10 minutes per server. So it’s, you know, that’s pretty lucky to have that.
Matt Stratton: [00:48:54] And that’s talking bare metal too, right? You know, I mean, you’re like, hey, great, you guys in the cloud, you can spin your VMs up in 15 minutes, whatever. We’ve got hardware going too.
David Yurkiewicz: So that’s huge.
Matt Stratton: So I guess we’re coming up towards our tail end here. I wanna give a chance for everybody to, before we move into our checkouts, if there’s some thoughts that you would have about what you’ve, maybe the thing that you’ve learned the most from being part of this team at Etsy.
David Yurkiewicz: I guess I can go first. I think honestly, what I’ve learned the most here is that, you know, I always had the philosophy, you always learn something new, always learn something new, and then here it’s like, It’s okay if you don’t know everything. There’s so many resources around you. Always have that drive and that hunger to learn. You can broaden your skill set by just touching new technologies and working on new things you probably haven’t done before. Just that right there, I think it’s pretty powerful.
Pete Bellisano: [00:50:03] Pete? I can say what I’ve learned from here is the value and I guess the importance of just asking questions. Everyone here has been more than accessible and willing to help. To David’s point, that will just accelerate your skill set and ability to learn more.
Matt Stratton: Allspaw, what have you learned?
John Allspaw: I’ve learned that every time that I think that some sort of size barrier exists, I’m always proven wrong. Quite often you’ll hear, oh well, we can’t do that because the company is too big, or this will work now but it won’t work later. Bullshit. That’s the one thing over and over and over that I continue to— things fail at a certain size because you expect them to and aren’t putting in enough effort to do the work to make them not break down. That’s my pithy Learning that I’ll give you, Cowie.
Jon Cowie: [00:51:24] I would say probably the standout thing for me I’ve learned since I’ve been here is quite how much you can get done. The kind of the things you can do when the consequence for failure is learning more stuff.
Matt Stratton: Oh, nice.
Jon Cowie: That’s my even pithier one.
Matt Stratton: Awesome. Well, let’s move into our checkouts. So, Cowie. What do you got for us?
Jon Cowie: So I’ve been— one of my frequent bugbears over the last year has been chefing stuff on Macs, which is a horrible experience since Apple seems determined to make it hard to do anything without clicking buttons. So my mentions this week are 2 Chef cookbooks. One is the DMG Cookbook, which is maintained by— not maintained by Chef. Which basically has a whole bunch of resources for things like unpacking a DMG and installing a PKG file inside it, and just the whole— all of the stuff you would want to do with packages on Macs. And the second one I want to call out is Fletcher Nichols’ rbenv Cookbook, which has single-handedly saved my somewhat long hair, which I’m quite pleased to have kept, which I did not have to tear out by trying to set up multiple Ruby environments. Environments on Macs via Chef in an automated fashion.
Trevor Hess: [00:52:46] Awesome.
Matt Stratton: David?
David Yurkiewicz: Actually, I just recently became a fan of pushover.net. It’s this utility you can use to do push notifications to your phone, different applications. And I run a terminal client for IRC. And I’m getting all my— like, anytime someone messages me on IRC, I get a push notification on my phone. Set it up to go to emails and other applications. You can do a lot of cool things. The API is very accessible. You can do a lot of things. So pushover.net, it’s a great utility that you can use. And it works for Android, iOS. They even have a desktop application, which I think is only like $4.99. It’s pretty cool. You can do a lot of things with that.
Matt Stratton: Cool. Pete?
Pete Bellisano: I’m going to switch gears here and go with libations.
Matt Stratton: Whiskey?
Pete Bellisano: Buffalo Trace?
Matt Stratton: Oh yes, yes, favorite here among us.
Pete Bellisano: So if you haven’t had it, try it.
Matt Stratton: And, uh, Mr.
John Allspaw: [00:53:47] Allspaw, um, I, I just recently rewatched, um, a presentation by a very smart guy named Lloyd Taylor, um, and, uh, it was, uh, the name of the talk is called Hacking Your Organization. It was in a 2000— I think it was 2011. Pretty sure it’s 2011, QConf in San Francisco. And the QConf site has the entire video and slides. And so yeah, it’s very well worth the 45 minutes or so to watch, especially for anybody who’s interested in the stuff we’ve been talking about today.
Pete Bellisano: Cool.
Matt Stratton: I’ll trace— hunt that down. I’ll hunt down that link and we’ll put it in the show notes. Trevor.
Trevor Hess: All right. So I just found a site the other day called Shortcut Foo. And I totally, totally suck at Vim. And I don’t want to totally, totally suck at Vim. And so this site actually— it doesn’t just do Vim. It does other applications as well. But Vim is one of them. And it teaches you the shortcuts of getting around different IDEs and other environments. So I’m gonna teach myself how to get around Vim with this tool. And secondly, I finally sat down and watched Agents of S.H.I.E.L.D., and I was very happy to have done so. It tied in really well with the Dark World and with Winter Soldier.
Matt Stratton: [00:55:20] Mine actually— so Trevor, once you actually learn all the Vim shortcuts, you can take advantage of my checkout, which is a Google Chrome extension called Vimium. Which gives you keyboard shortcuts for navigating and controlling Chrome using Vim shortcuts. It’s amazing. And it’s at vimium.github.io, and we’ll put in a link. The other thing that I just heard about this morning is there’s a Kickstarter for a card game called Release. And so it’s a card game about software release, and it looks absolutely hilarious and intriguing and interesting. They have specialty cards for Patrick Dubois and Jez Humble and a bunch of other people they aren’t telling us about yet. So for all we know, Allspaw might be in there and he just is keeping it quiet.
Jon Cowie: I don’t know.
Matt Stratton: So I also want to make sure everybody knows or remembers we do have an email newsletter. It’s at arrestedevops.com/bananastand, and it’s the best way to find out about upcoming podcast episodes and cool news with DevOps. We promise we won’t send you something more than once a week, and we never, ever, ever distribute that email address to anybody else. And you can take our survey at arresteddevops.com/survey for a chance to win an Amazon gift card. Now, only listeners to the podcast can enter, so don’t tell anybody that secret link I just told you. And if you are listening to this later, the contest will end on June 15th, 2014, so don’t bother trying to do it like a year from now. Thanks to our amazing panel for joining us today. Thank you to John Cowie and David Yurkowitz, Pete Bellisano and John Allspaw. And I hope that a lot of people are going to get a lot of value out of it. I hope you enjoyed yourselves. If you’re new to us, you know, find us at arresteddevops.com or @ArrestedDevOps on Twitter. I am Matt. I’m @MattStratton.
Trevor Hess: [00:57:18] And I’m Trevor, @TrevorGHess on Twitter.
Matt Stratton: We are Arrested DevOps, and remember, there’s always DevOps in the banana stand.
