Trevor: [00:00:07] Welcome to Arrested DevOps, episode 40, Building an Ops Team. I’m your co-host Trevor Hess, @TrevorGHess on Twitter.
Bridget: Hey, you’re not Matt.
Trevor: That’s what I just said.
Bridget: This is the first ever episode of ADO that Matt couldn’t join, so I’m your co-host Bridget Krumhout, @BridgetKrumhout on Twitter.
Trevor: Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that figures if you’re listening to this podcast, you must be pretty cool. You can find out about joining our cloud services team at arresteddevops.com/10thmagnitude.
Bridget: This episode is also sponsored by VictorOps. From initial alarm to final retrospective, the mission of VictorOps is to make on-call suck less. Easily integrate with your existing monitoring systems and manage on-call schedules with rules for intelligent routing. In the live infrastructure timeline, get real-time context and see annotated alarms with resolution documentation. And when you’re in the firefight, collaboratively troubleshoot using native chat or bidirectional integrations with your favorite chat clients. Visit arresteddevops.com/victorops and sign up for a 14-day free trial to see how they’re making on-calls suck less.
Trevor: [00:01:13] Today, we’re going to be talking with 3 great guests, starting with Charity Majors of Parse and/or Facebook. Charity, can you tell us a bit about yourself and your background?
Charity: I am an accidental computerer. Basically, I never set out to do this. I was a classical piano performance major in college, and then I dropped out because I realized I really love computers and fixing things, and also I really love making money. Money is really nice. Pianists, not so much on the money-making. So right now, I was the first infrastructure hire at Parse. We were acquired by Facebook about 2 years ago. So I handle all of the backend operations and DBA work for over half a million mobile apps, basically. And I have a team of 7 engineers who are amazing.
Bridget: Thank you, Charity. And now a repeat engagement with Team Etsy. So, MCR, Patrick, tell us your stories.
Patrick: [00:02:15] I’m Patrick McDonald. I’m one of the web operations managers at Etsy. I’ve been there about 5 years now. I was a pretty early hire on the team and was actually hired in as an individual contributor for a couple years before heading over to management. So I have a bit of that story of the transition and the building and all of that stuff.
Michael: Hey there, Mike Rembetsy. I am the VP of Technical Operations at Etsy. I joined Etsy in 2008. I was one of the first first new ops people in 2008. There were only 2 of us, and the team has since grown to over 50 that I’m responsible for, which includes day-to-day operations, all of our corporate infrastructure, our data centers around the world, as well as our security team, DBAs, and working for great people like Patrick and making sure that the team has what they need from a day-to-day perspective.
Bridget: All right, so we’re talking today about building an ops team. So it sounds like maybe we should start with, okay, Mike, Charity, you’ve done this. Maybe Charity, if you want to start with your perspective on where do you even start?
Charity: [00:03:27] Yeah, that’s a really good question because it’s hard, right? There’s a lot of really fierce competition for really great engineers. I mean, I would say you should start with the question, do you actually need an ops team? Because there are a lot of places out there who think they need traditional operations engineers when all they really need is someone to really care about their infrastructure. That doesn’t have to be someone who has really strong traditional operations engineering strengths. I mean, the DevOps movement, as a buzzword, I can’t really get over how much I still really hate it. But if you’re a tiny shop and you have half a dozen engineers and you’re not doing anything that is really pushing the bounds of scalability or reliability or security, you probably don’t need an ops team. Just wanting to not get paged in the middle of the night is not a good enough reason to start hiring operations engineers. You should have actually genuinely hard operations problems before you even start looking to hire engineers.
Bridget: [00:04:32] All right. I see Mike nodding at that. So as an early ops person at Etsy, has that matched? How well does that map with your experience?
Michael: Um, so I definitely agree. I think that the early startup of a company really kind of allows you to have that flexibility that Charity is talking about, right? Taking a look at what the problem is that you’re solving for is always a good thing, right? You take a step back and for a lot of companies these days, um, compared to when I started in the field, most people are starting on AWS, right? They’re starting inside of this, um, this environment where they have the flexibility to not fully immerse themselves in what traditional ops is and what they do. And I think that taking a look at building an ops team really does depend upon where your company is in its lifecycle, right? So if you’re in a very early lifecycle, it may not make sense to enter into the world of competition, as Charity mentioned, right? The ops competition of trying to find good hires is definitely very competitive. It’s not impossible, but it’s definitely competitive. So, you may want to lean more towards someone who has a focus of dev, and I agree, the whole buzz term of DevOps is a— for me, I’m like, oh, DevOps! Right? And being involved in the DevOps community, for me, it’s just a culture. It’s a way of working and a way of how you get things done in the long run and how you respect other people inside of your organization. But, back to the question at hand, I feel that making the decision to hire traditional ops— and I keep saying traditional ops, but let’s say operational-minded folks who are are used to running data centers is not necessarily always the right choice depending upon where the company is. As you grow and as you start hitting some of those scalability problems that Charity talked about, or you have other security concerns, or you have, oh, we’re going to do this or that and we need more manpower or just power in general, I think that it’s important to take a look at what the problem is you’re solving for. Because once you start down that path, it’s not a hard one, but you have to maintain it. You have to grow the team. You have to grow the individuals. You have to grow the culture. And I feel that that takes time and effort that, in a startup phase, may take away from the goal of actually getting the company off of the ground.
Charity: [00:06:56] And it can let your developers be really lazy, too, right? Like, if you don’t hold them responsible for the reliability and the maintainability of their own code, if you’re just like, oh, we’re going to solve this problem by hiring an ops team, That’s probably the wrong answer. You should have a much more specific goal and vision than that.
Trevor: Charity, you had said specifically, you have a hard operations problem. Can you tell me what you mean by a hard operations problem as an uneducated developer who may occasionally cause some of those issues you just mentioned?
Charity: Yeah, totally. For example, reliability is a thing. Anyone should be capable of building and maintaining and being responsible for reliable service. But if the core of your business revolves around having more than 3, 3.5 nines, you probably want someone with deep expertise in making that happen. If you are scaling incredibly fast, if you are 2x-ing, 3x-ing, 5x-ing, 10x-ing year over year, then you need people with deep expertise and a history of scaling systems really fast. I come from a history of— I’ve never actually worked on a team that had a dedicated DBA team. Operations has always included being DBA. If you’re doing something new and exciting with your databases, you want people to be focused on making that work really well. I think another area is deep security concerns. Security is an operational specialty that needs a lot of expertise, if that— I mean, this all comes back to like, what is the mission of your business, right? Your responsibility is to make your mission succeed. Now, if your mission is to run a website that doesn’t deal with a lot of e-commerce or whatever, you don’t need to hire someone who has deep security specialties. If your mission is to build a service that is growing steadily but not, you know, exponentially, your mission does not require hiring someone who has the specific expertise of rapid scalability. So I think it comes back to like really deeply understanding your mission and looking for people who can fill those strengths rather than looking for people who lack weaknesses. So I’m really—
Bridget: [00:09:27] I find this fascinating from a perspective of scaling both your team and scaling like, your infrastructure while looking at your mission. And it was interesting that you mentioned retail, because obviously Etsy is in that space. Patrick, can you address when you came on board, like, at what point during Etsy’s scaling you came on board, and how that intersected with how the team was scaling?
Charity: Sure.
Patrick: I think I came on board when the company was about 120 employees in 2010. The ops team was, I believe, 4 people at the time. And we actually were— we had a bunch of hires come on right around that same time. But one of the things, you know, I guess one of the things that we already had the luxury of was having a bit of an ops team already formed. And I think luxury is kind of a good word because having ops people kind of is a luxury. You know, it’s when you’re first starting, you know, a business, usually You know, it’s great if you can spend time like really thinking about the mission statement, who you want to hire based off of that, but a lot of companies are sort of just going with the flow. They’re scaling as quick as they need to, and they’re scaling with who they have on hand. So I think that given where we were, you know, we already made some decisions. Like, we had already decided that we’re building this on bare metal infrastructure. Of course, back then the cloud was a little bit less in everyone’s face, so, you know, companies were often going the more traditional route there. We already had a pretty decently functioning network, which wasn’t the case a couple of years before that. But kind of by the time I had come in, things were kind of stabilizing. So I think at that point, that’s right around when we were really starting to focus on what the culture should look like, right? There was We all kind of had ideas of where we wanted it to go, but I think that it wasn’t really until we passed a certain barrier, you know, a certain level of comfortability with our infrastructure, a certain understanding and having the right talent on board, that we really were able to sit down and focus on those things. So I kind of came on right around that time. And it’s been a pretty crazy ride. The team’s scaled 4 or 5 times up from that since then. And yet, I think what we did back then, we’re actually still doing a pretty good job carrying that forward today and making it so it kind of seems like a smaller team and has a really solid culture.
Charity: [00:12:09] I think you said a couple of things that were really interesting, which is that having the time to think Having an ops team already is a luxury. I totally will not disagree with you there. And having the time to think about your mission is kind of a luxury too. Because you’re right, most startups fail. And most startups do not fail because they had insufficiently awesome ops teams, right? They fail for external reasons or for leadership and vision reasons or for funding reasons or for totally random reasons. So, getting to care about those things and getting to actually guide the culture and the direction of your technical team is totally a luxury. I think that’s really important to call out.
Trevor: Another piece that I think we’re kind of gearing towards here with the way we’re discussing this, one of the things we had outlined was talking about— we’re talking about everything being related to code now. Infrastructure is code. Obviously, your software code itself is code. I mean, the idea we were going to discuss is what is an ops team now, but what we’re talking about needing to hire to the need you actually have and not necessarily to the general bucket that everyone talks about being a developer or being an ops person. Really, like you said, you need a security engineer to address your PCI concerns, or you need someone who’s familiar with the cloud so that you can discuss how to scale in the cloud. Do you think it’s less about the bucket and more about the person?
Michael: [00:13:47] It’s funny. I mean, people can learn. People are, you know, hopefully they can learn or they want to learn, and that passion is great. I think that one of the areas that we focus in when we actually hire, we actually have been building, you know, the operations team, which is very wide in terms of responsibilities and what they do and what where they focus is, is that it is about the people. You know, the, the glue that holds a team together is how well people interact with one another, how well they respect each other. Are they able to actually walk up to another teammate and say, hey, look, that statement that you made, I really didn’t like too much. Let’s have a conversation about that and figure out how we’re going to continue to work together, whatever that end goal is.
Trevor: Right.
Michael: And that end goal can change every 3 months, 6 months, year. It depends again on the company. But at the heart of it, if you have a team that bonds together, if you have a team that is capable of working out their problems, whether it’s around language, whether it’s around how people perceive each other, I think that you’re going to build a long-lasting operational team. You’re going to build a long-lasting development team. You’re going to build a long-lasting management team that can adjust together at the chaos that is the internet that we all live in today. And I think that when you actually do go through your hiring process, for us at least, and I’ll talk about us, that’s a really big component when we do hire someone. And it doesn’t matter what job they’re in. We do very heavily weigh on the cultural aspects, how people interact with one another, whether it’s negotiation, conflict resolution. Things along those lines. So I think that, you know, out of your question, that’s what popped into my head. I was like, yeah, technical skills are great and there has to be a level of technical skills depending upon what job you’re hiring for. But at the end of the day, do you get along with the person? And when you don’t get along with the person, do you feel as though you’re going to be able to actually work through that with them? That is something that I hold very important because let’s face it, we all say silly things sometimes. I certainly do. And that’s not, you know, who I am all the time, but if you have someone who can come up and say to you, hey, look, this is not cool, this wasn’t kosher with me, and the person on the other end receiving that feedback can turn around and say, you know what, you’re totally right, I didn’t see that perspective, I apologize, thank you for calling me out, and they walk away with a bit more knowledge and you have that bond and that experience with someone, that’s what you build good ops teams on. That’s what you build any good team on, quite frankly.
Trevor: [00:16:27] Let me ask the question that follows on to that. How do you interview for that? Because that’s something I’m trying to figure out also. And so I’ve been working with one of my coworkers to kind of put together— we’re calling it the Socratic interview method— to force people to reveal their thinking through conversation and expose would those cultural concerns be an issue. Because most of the people who come in are people who are willing to learn and are capable of learning. You’re absolutely right. It comes down to, can you be a member of the team and kind of fit into the culture?
Michael: Patrick can certainly jump in and help out with this if I leave anything out, but to say that we got it right 100% of the time would be a total lie. You’re not going to get it right all the time, and it’s a learning process even during interviewing. During hiring that you go through. We certainly do break up within the process. People have certain focuses, whether they’re technical or whether they’re cultural. The questions that we have are certainly focused around things like, for example, tell me about a time when you had an awkward situation when you were working and how did you actually resolve that situation? You’d be surprised. You’d be surprised how many people are like, yeah, I had this conflict because someone said something, and we dive into that and we go a little bit deeper and a little bit deeper. Questions like that. We also look at writing skills. We also look at communication skills in a variety of different ways, from whether it’s homework with essay-style questions in addition to just in-person interviews. ICs are interviewed a little bit different than managers.
Charity: [00:18:15] I’m a big fan of soliciting a wide range of interviewing feedback and interviewing types because there’s no one template that everyone is going to fit. I totally agree with what you were saying, Mike, about— and Bridget— operations engineers, good ops engineers are good at learning things. One of my pet peeves in this industry is the laundry list of technical skills that you’re expected to have coming into the door for any given job. When I started at Parse, I had never used Chef, Ruby, AWS, Mongo, Cassandra, Redis. The list just goes on and on and on, and that was absolutely no barrier to success. I was fortunate that they didn’t go, oh, we’re a Mongo and Chef shop, and you’ve never seen these technologies, so therefore you’re not capable of doing this job. It’s like, no, that was one of the most exciting periods of my career. Learning all this stuff at once. This is why I do ops. I like learning new things. And so on the one hand, it’s really easy to screen for specific things that you already understand and you expect someone to already understand. But it’s also the laziest form of interviewing. Because someone already having been exposed to any given question or any given problem set is not— it’s like they say in the financial world, past performance is no predictor of future results or whatever. There are some people who are really good at written communications, and some people are really good at thinking on their feet and problem-solving on the spot. Some of us completely freeze up. I really like to give at least 50% of the questions that I want to talk about to the candidate ahead of time. Give them a little bit of time to think about it in a non-pressury environment and work out some preliminary stuff before interacting with me in a very formal setting. I think you want people to be bringing the self that you’re going to be working with on a daily basis, not the self that is freaking out and wondering how they’re being perceived and totally stressed out, right? And anything that you can do to make it a more welcoming environment, it helps people perform at their best, which is what you want them to be doing as your potential coworker, right? And I also, I think it was Ben Horowitz, the guy who wrote from Andreessen Horowitz, he said recently, he’s like, you know, if I was given the choice between interviewing someone for a few hours or just getting feedback from people who had worked with them before, I would take the feedback and no interview Any day of the week. I couldn’t agree more. In Silicon Valley, it’s kind of a small town. I find that reaching out to people who have worked with the people that I’m going to be interviewing gives me so much valuable information. It’s like the highest signal-to-noise ratio if you can get it as far as how good someone is going to be as a coworker.
Bridget: [00:21:32] Though, of course, that brings us to the problems of selecting people from our networks and culture fit and all that sort of thing that can eliminate candidates that we otherwise would— they would be great candidates, but we don’t know them or know of them. They don’t get necessarily their fair shake.
Patrick: That’s a really good point. That really does, I think, in many ways start with the job posting that you put out there. Making sure that it doesn’t have biased language in it, making sure that there’s nothing in there that makes people feel like you’re an unapproachable company. Really, you want to be as inviting as possible to keep on building that network. There are a lot of great places to meet people too, at events and conferences and everything, where you can start to get to know them, like Charity was saying, outside of the formal setting first. I find that we’ve had a really high success rate with hiring people that we’ve gotten to know through other channels first before actually bringing them in for the interview. I think it’s interesting too because I know when I was first hired the interview process was really nothing like it is now. Obviously a much smaller company. It was much less formalized. I remember that there was a pretty big technical interview where I was just asked a bunch of difficult questions. And it was kind of like— I thought I didn’t do very well, but apparently I did do pretty well as a percentage of— or as a percentile of the applicants. But I did think that as we’ve evolved, we’ve moved really beyond— we still obviously do technical interviews, but I think that most of the value of the interviewing we’re coming to find is in a lot of the questions that MCR was saying that he likes to ask. Things about specific situations, times where you exhibited certain qualities in communicating, in leading, in all of these non-technical areas. Because we all know that our skill set turns over so quickly, you don’t really need to have someone that has deep experience with the exact flavor-of-the-month database you’re using or something. You do need someone that understands what it means to work in that environment, in a collaborative development operations environment, and knows how to get things done, essentially. We also have another component of our interview process that— we have this idea of Etsy-ness, as we call it, which is actually something that we’re retooling right now, given that it obviously can be up to interpretation. But as we continue to iterate on this, I think it’s really important for people to have something like this, an idea, sort of a shared value set that you define that your company is looking for, and making it pretty clear before the interview process what to expect from interviewers in order to pull those qualities out of people and be able to evaluate them on that in a relatively short period of time.
Charity: [00:24:51] I also think that I value things like enthusiasm so highly. If people are under-informed, whatever, you can make them informed. But if people don’t care, if people are apathetic, if people— so one of my main things that I look for in the interview is how people talk about their past jobs. You ask them questions about their past roles and what they liked, what they didn’t like, what went well, how they reacted when something bad happened, because something bad happens at every job. I try to look for a sense of like, They try to fix things, whether it’s human things or technical things. The sense of learned helplessness that just reeks off of some people who have been in bad situations for a long time, you know, it sucks and it’s like poison to a team if people don’t feel empowered to ask questions and to probe and to fail and to correct things and to make the team better. I will take an energetic person who feels ownership over themselves and their environment any day over someone who has a technically superior ability. Like both of you are saying, these soft skills are what make a team. The technical skills may be what make for a strong individual, but the interpersonal skills are what make for a really powerful, long-lasting team that can really execute on a mission.
Michael: [00:26:28] So we have, we have a saying, right? Because everyone— well, lots of people, I should say, not everyone— but lots of people have great ideas. Hey, we should do this, we should do that, right? And to speak to your point about the enthusiasm or the willingness to go and take a problem and fix it, whether it’s human or technical, absolutely couldn’t echo that more. So the saying we have is patches welcome. It’s like, hey, look, if you have a great idea and you think you could do it, Do it. No one’s going to hold you back from trying that new thing that you think is going to be awesome for a set of VM servers. Go for it. Prove it out. Build a prototype. It’s basically a human-centric engineering approach where if— excuse me— where a human being who has the idea, we don’t ever want to dismiss that idea. We may find out later it’s maybe not the best idea, but for that initial energy, that initial passion, that willingness to get in there and try and do something, Patch is welcome. Get in there, let’s do it, show me how that thing works. I am more than willing to say I am not the smartest person in the room, and most times I’m not, which is totally fine. But if there’s an idea that we can foster and we can build together as a team, let’s do it, let’s see how that works. And, and I totally agree and echo that I would take that any day as well.
Patrick: [00:27:44] I think our jobs are really— when we’re talking about defining what our roles are, well, in the end, your role is really, it should be optimally, to do whatever is best for the business regardless of what you think your job description is. If you have an idea that’s going to further the business, you should absolutely go after it. I think we do a pretty good job of encouraging that at Etsy, certainly a lot of interdepartmental collaboration on things. Like, we’re doing our Hack Week right now, for instance, and there’s— everyone at the company participates in Hack Week, absolutely everyone. So you get ideas coming from all sorts of different people, and you get these big teams from all different departments that work together to create awesome hacks. And I think that kind of spirit needs to be always fostered in the day-to-day setting, too.
Bridget: Okay, so that kind of leads us towards, we’ve talked a lot about the great qualities we’re looking for and want to encourage in individuals, but as managers, I’m wondering if you can all address, maybe starting with Charity, all address the actual, you know, hands-on role of the manager in making this stuff happen.
Charity: [00:29:01] Yeah, totally. So I think it, It depends. It depends on the size of the team, the size of the org, and the specific characteristics of the individual manager. You have some managers who are— my archetype as a manager is I tend to go in as the early engineer, build up the infrastructure, hire a team, and segue into management roles, and then get bored a couple years later and do it all over again, right? So I know my archetype. There are other super valuable archetypes. I feel that as the team grows, it’s doing the team a disservice if the manager stays in the critical path of the technology. You have to take yourself out. It doesn’t mean you can’t still do technical work. I find it so rewarding and relaxing and grounding to do real technical work. But I, like, around the point where there were like 4 or 5 people on my team, I realized I was holding everyone back if I still tried to be the tech lead for everything and, like, reviewing diffs before they got shipped, right? Because I’m getting pulled out to different kinds of crises, right? I went from firefighting on the platform all the time to getting pulled into ad hoc meetings about things like branding and marketing and cross-functional calibrations and crap. And, like, it isn’t fair for my team to be gated on me, I shouldn’t— I should never expect them to do that. And I would be holding them back in their technical development if I clung to the role of tech lead and person who is driving the technical roadmaps and person who is the expert at everything. It’s A, impossible, and B, not desirable. Nobody wants to work for a manager who doesn’t give them the space and the breath to grow and to lead things and to make mistakes and to develop into a more senior engineer. I would say in the early stages, if you’re lucky as a startup, you have an early hire who can grow into that manager mantle and who is willing to put down the technical lead stuff that they started out doing. In larger and more— I don’t even know where to put Etsy on the— on the spectrum between startup and Facebook. But at Facebook, they’re just like, you’re a manager, you don’t fucking do technical work. It’s like you get a demerit if you’re doing technical work because your job is your people. Your job is tending to each person’s career trajectory and making sure that they know how they can improve, how they can meet the expectations, how they can get to the next level, how they can grow as an engineer into a more senior role, and you’re cheating your team the more time you spend on technical things. And that’s, that’s not a role that I personally really want to play, which is why I stick to like the smaller and middle games, and I’m not like a director at Facebook. So those are my thoughts.
Michael: [00:32:09] So yes, I’ll let Patrick talk about where he is in the org and his level. I think for me, there comes a point in time when you do take that manager path, right? And you have that fork. And something that you were talking about, Charity, kind of sparked the idea of motivation in my head and what motivates people to work. And I feel that my job is to help motivate, help set up guardrails, and then let people get it done. Let them kind of explore how they’re going to do it. If they hit a bump here, maybe they bounce over to here, but the path of which you know, the direction that we’re going is certainly something that I feel a manager, a director, VP needs to take a step back and to let people succeed and fail and learn from those failures in order to grow themselves, in order to grow the organization, so on and so forth. I, you know, I, I, I, for whatever reason, Daniel Pink’s book Drive pops into my head about motivation and how to motivate people from moving from an old Carrot and stick, you do this, you’re going to get this, to the 3 pieces of science that motivate human beings. That’s the autonomy to do what you want to do and how you’re going to do it. Obviously guardrails. Make sure you’re not totally going off the reservation, even though sometimes that might be fun. To the mastery of it. Are you learning? Are you investing the time? As a manager, as a leader, am I empowering people to actually spend that time to become masterful of something. It could be code, it could be cassette, it could be anything you listed earlier, like technically, or on the other side, the human skills that people always need to work on. I do, everyone does. And then finally, the purpose. Do people find that they— that what they’re doing has a purpose? And at the end, do they feel comfortable of what they did, how they did it, what they learned? So for me, it’s funny, people— I’m actually pretty awkward about saying that I’m a VP. Um, and, and the reason why is that I’ll just be like, hey, I’m head packet and paper pusher. You know, like I just, I’m there to provide the tools for people to do their jobs and to make sure that along the way, if there is a concern or there is a problem that they feel as though they have a safe outlet to come to, to talk about things and, um, and know that they’re going to be helped, not that they’re going to be screamed at, not that they’re going to be yelled at or any of these other horrible stories that I’ve heard during, you know, any sort of, conference interview process, anything like that. So I feel from my perspective, that’s really my job in a lot of ways.
Bridget: [00:34:49] So packet and paper pusher— are these TCP or UDP packets?
Michael: Well, if they’re UDP, you’ll never get the paper.
Bridget: All right, so let’s get the reality check from Patrick. You work for this Mike guy. Tell us your perspective.
Patrick: Yeah, I have to— I really do agree, especially with what Mike was saying at the end, where the manager’s role is really— it’s a facilitator. My job is to essentially take some of the, you know, plan, obviously business requirements, you know, with, you know, the entire organization, bring that to the team level and say, all right, here’s what our goals look for the year. But pretty much outside of that, everyone should just be doing what they think they need to do. It’s kind of for me to remove obstacles that come into their way, to make sure that I can smooth things out if need be. But really, it’s to encourage and to allow people to reach their full potential. At Etsy, actually, we do something quite nice where we I think a lot of companies are starting to do this, but you have the traditional view that management is sort of like a super layer on top of the rank and file, right? But what we do is we have 2 separate and equal career paths: individual contributor track, manager track. So when you go from, let’s say, a senior engineer, let’s say an individual contributor 3, right, you’re actually sidestepping into a manager 3. You’re not going up because you might have proved yourself in one arena, but you are really acquiring a completely new set of skills. You’re moving into a different career entirely, and you’re not going to be awesome at it from day one. No chance. There’s a lot of mistakes that you sort of have to roll with, and it does need to be Like Mike said, you need to be constantly learning and acquiring new knowledge about what it means to be a modern manager. And I think that you’ll end up, if you’re really listening to your people, you’re going to learn anyway just through them, because they will be— they’ll be the first to tell you what you’re doing well, what you’re not. And the transition’s been really nice. To management for me, I think, because of how Mike has allowed me to basically— he’s given me enough leeway to make mistakes that I need to make and to build myself up. I remember one time I brought a problem to him, and he was just like, well, take care of it then. That’s what I pay you for. And I knew at that point I crossed the line, and it was like, all right, I now have the trust to just execute on this. And I think that’s, that’s really what I’m trying to do for, for all my reports as well, is to get to that point where they just say, alright, I know Patrick says I can just run with it and it’s all gonna be good. He’s got my back.
Charity: [00:37:59] Yeah, Patrick kind of touched on this, but I, I think I wanna just say it like super plainly. I think you have to say again and again and again and again to your org that management is not a promotion. It’s a career change. And people will forget it, and new people coming in won’t have heard it, and people won’t believe it. So you have to keep repeating it. It is not a promotion. You are not being promoted to management. You are doing a career change. This means you’re responsible for a different set of deliverables, you have a different set of skill sets to build, and you are not superior to your reports. Right? This is important. For a lot of reasons. Number one, because it’s true. But also because if you don’t emphasize this, you don’t build a compelling story for how ICs reach their full potential. Like without eventually, inevitably having to become managers. Lots of people don’t want to do that. Lots of people want to do that but shouldn’t do that. Lots of people would be better off becoming better and better and stronger ICs while still building their leadership skills, because manager and leader is not synonymous. There are so many ways to exercise leadership skills that don’t involve having direct reports.
Trevor: [00:39:26] Now you’ve made me question how I want to ask my question. So when you’re a small company and you’re the only person on a team, how do you, how do you balance bringing on more people with the emergence of a leader, with the eventual need to, instead of, instead of having a lead, just a leader, you wind up needing a manager and that is, that is a separate role. How do you, how do you manage that transition when you’re growing?
Charity: Uh, wow. This is so situational, right? It’s really hard to predict how a situation will evolve when you don’t know the personalities who are going to be there. I think that the— that’s true, right? I think that it’s usually easiest for the team, it’s most natural, it feels the most organic if a leader can emerge from the crowd, even if it takes extra mentoring, extra support, extra training or whatever. That’s not always possible. It’s been possible everywhere that I’ve worked because this is something that I naturally do, so I can’t really speak to the flip side. Mike, maybe you can.
Michael: [00:40:35] I’ll go back to stages of the company in some ways. I think that in the early stages, you know, working towards a goal and everyone working together is really where leaders emerge. Managers, for me, I’m not sure if people know this or not, but I actually was lucky enough to be able to hire my boss, who’s John Allspaugh. So I think, you know, John was an advisor before he was at Etsy, so he and I kept in contact and we were able to create a, you know, sort of a relationship earlier. And I think at a certain point, having a manager, having a— whatever the definition of a manager is that we just all went through— having that person come into an organization could be the most disruptive thing that anyone will do. Right? It’s, it’s, it’s also the scariest thing. Um, hiring a new manager to come in versus some people growing inside of an organization. So again, going back to how we hire and how we look at people and, um, how we interview and what we look for, things like that. It doesn’t have to be a bad thing. I mean, it’s worked out pretty well for me, I think. Um, and I think that You as a, as an individual and the company, you’ll get to a point where you will need to hire a manager. You will need to have someone in charge of empowering people to do their job, providing them with the tools, looking at strategy, looking at longer-term goals and objectives. And that’s not something to shy away from, but being flexible enough to understand when you need to actually have that. And that’s always the tough part, is when you make that decision. I can’t say when the right time or wrong time is for anyone, but I think that as long as a company, as long as a group of people are feeling that that is the right time, then that’s the right time, you know. And if it’s not, you, you learn to adjust, but you’ll get to that point where you need a manager.
Charity: [00:42:31] Yeah, totally. I feel like there’s a real divide here between how do you grow a technical lead into a team manager versus How do you find and sort for amazing, compelling, charismatic executives like Allspaw, who, goddammit, I wish I could have Allspaw as a boss. Don’t we all wish we could report to John Allspaw? Can we just agree to clone him and then everybody gets an Allspaw? Everybody gets an Allspaw.
Bridget: You get an Allspaw, and you get an Allspaw, and you get an Allspaw.
Charity: Yeah, if only, if only that would work.
Patrick: Yeah, I think it’s really important not to conflate leadership with management, because a leader is a leader regardless of what their title is. And I think management is more a necessary evil in many ways, that it’s a layer that comes into existence typically when the organization gets to a certain size and requires it. One of the things that I like to ask if someone says that they’re interested in becoming a manager, I like to immediately ask, well, what do you think you can accomplish as a manager that you can’t accomplish as an individual contributor? A lot of times I actually find that a person might actually not realize that they might lose some influence moving into the management role because of how they’re positioned amongst their peers as an individual contributor. They’re already leaders. And because they’re 2 different careers, you can be a leader in one space, not necessarily the other. So there’s a lot of things to think about there, certainly. And when— with a smaller organization, I think leaders emerge pretty clearly. With a bigger one, though, you sort of have to find ways to, as Charity was saying, like mentor them and encourage them. And what we do at Etsy, we have a program called Learning and Development. And what’s really a department, and we have all sorts of, you know, training programs and dens and, and various other, you know, classes and workshops and stuff that you can attend regardless of your role, really. I mean, we have something called a leadership intensive too, which is kind of where you— it’s sort of to groom someone to potentially move into management down the line, or really just to kind of help someone that is kind of a leader in the technical space to kind of boost their management-type qualities. So having these programs, I think, is really important if you do have a potential manager emerging. And throwing people into the fire, yes, it’s how most of us learn most of the time. But that’s not really the best path to take. I remember that when I was moving into management, Mike would just drop books off at my desk and say, you should read this. And this was before it actually happened. This was kind of in the ramp-up too, so that I kind of had some good ideas and we had one-on-ones and talked about what the transition would look like before it actually happened. So when I actually got there, I felt like I was in a good spot. So I think having that support network is really important.
Trevor: [00:45:49] Yeah, I think having a support network is definitely something that’s valuable. We are getting close to the end of our hour though, so let’s wrap things up a bit.
Bridget: We have so many things we want to talk about and we always say we’re going to do follow-up episodes and that’s usually a lie, kind of like the sushi in our last episode. We did an episode we called Eating Sushi with Andrew Clay Shafer. Was no sushi. It was very sad. But what I guess— there’s, there’s so many topics around, you know, how you manage and guide and review and pay and recruit people that I feel like I could just kind of throw a general question to all 3 of you. One at a time, please give us your absolute best advice to someone who may find themselves in the position of building an ops team?
Patrick: The stakes are pretty high when you’re first starting out, and I think, you know, you don’t want to make bad hires early because they can cause a lot of damage. So I think when you’re moving into building an ops team, really focusing on hiring good people. Hire people you like, hire people that you trust, hire people that Maybe they need to do a little bit more research, maybe spend a little more time on Stack Exchange than the next person, but you know that they’re going to get the job done the way that you need it to get done, and they’re going to do it, and they’re going to be nice. They’re not going to be assholes, and that’s okay. You should be able to say, we’re going to sacrifice maybe some of these— I think people are tempted to jump in and be like, I need to hire the most senior person I can find. They’ve got to be able to do everything. And in reality, like, that’s often really not the place you want to start. Those are kind of people that you want to acquire toward the end, toward the more mature stage, when you actually have a place for people like that. So I think also just talk to people that you know are already running successful ops teams and find out how they did it and ask for their advice. Can’t hurt to talk.
Charity: [00:48:02] I would say if you are really trying to build a team, Build your networks. Go to meetups. Talk to people. And don’t just talk to the popular kids. Reach out to diverse communities and diverse crowds. And go meet people who are doing cool and exciting things who are slightly off the beaten path. The more people you know, the better you’re going to be at hiring. It’s kind of a somewhat regrettable fact, as Bridget was saying, but I think that if you are conscious about the types of networks you’re building, you can mitigate that to a large extent. And I would say don’t be afraid to take risks on people who seem enthusiastic and ambitious. I love telling this story, but when we were at Linden Lab, our receptionist looked bored one day and we started taking him to the colo to rack servers with us. He’s now one of the best network engineers that I know, and he was just He was just always there, always up for the next thing. We were totally happy to mentor him and offload shitty work onto him. He just kept kicking ass and knocking it out of the park. I’ve seen this happen so many times with people who didn’t have formal schooling, who were English majors or music dropouts like me. There’s so much untapped potential out there that the big companies aren’t really utilizing or exploiting. And, you know, if you’re willing to take a risk on someone, it doesn’t always work out, but a lot of times it does, and the world is better off for it. So I look for untraditional, like, sources of talent, and I believe in nurturing people and training them and learning together and collaboratively. And also I believe in poaching my friends from all their successful startups.
Michael: [00:49:52] If I had to give advice, Plus one to what everything that Charity and Patrick said, for sure. I think that that absolutely echoes with me as well, no doubt. The one thing I would say, or some additional advice, is if you are building an ops team, make sure that you address conflict. Make sure you create a safe space for the people that you do hire to have open, honest conversations with one another. There’s nothing more toxic to a team than people chatting behind other people’s backs. And I think that all of the things that everyone has said today, as a leader or a manager or an individual contributor, from the highest to the lowest, I feel it’s everyone’s responsibility to create a safe environment where people can be open and honest with each other, where people can feel comfortable enough in who they are, no matter who they are, to be able to approach and have conversations about things that they absolutely disagree with or things that they want to try, whether it’s technical or whether it is human-centric. And I feel that that is one of the many cornerstones that we’ve all said throughout this whole entire podcast that people should take away from here, is that it is everyone’s responsibility to create an environment for people to learn and grow and feel comfortable.
Trevor: [00:51:14] Thank you, everybody. Let’s, let’s go ahead and move into checkouts. Mike, you want to start us off?
Michael: Definitely. So the checkouts that I had were— we run ELK at Etsy: Elasticsearch, Logstash, and Kibana. And recently we’ve had to move some things around, and draining those servers of the data is a bit time-consuming. So we open-sourced a utility Katherine Daniels, BeerOps on Twitter. We open-sourced a utility that would actually help us when we’re working on draining the cluster. My other checkout would be a 2012 Velocity talk from Dr. Richard Cook about how complex systems fail. It’s always nice to go back and remind myself once in a while that we all do work and live in a complex system that is the internet. That is not only computers but also humans. So I would urge everyone, if you haven’t seen it, go check that out as well.
Trevor: Thank you. I just want to remind everybody that all these notes are going to— all these links and checkouts will be in the show notes for further review in the future. Charity, you want to go next?
Charity: [00:52:26] Absolutely. I would like to share with everyone a guided meditation for adults. It is— it will make you feel much better. Breathe in strength, breathe out bullshit.
Bridget: That is awesome.
Trevor: It’s a wonderful video. Patrick?
Patrick: Well, I guess we’ve been focusing on this theme of building teams. I want to switch gears for a minute and talk about maintaining teams. One thing that’s really been resonating for me for quite some time now, even from the months it’s been since Velocity, was Dr. Christina Maslach’s talk on burnout in tech. It’s a lot of times for reasons that you don’t think it is. It’s really not mainly about how much work you have. It’s much more about how much ownership you feel. Do you feel like you’re in a safe environment? All these kind of things. I’ll post a link to a little self-assessment you can take. I think it’s just fun to do this whether you feel like you are in a burnout phase or whether you feel very comfortable. It’s like a 5-minute little thing and it kind of scores you. You can get a good idea of where your risk is for burning out. I think everyone should— I would encourage everyone to take that and think a little bit more about what you can do to make your job a little more tolerable every day. Nice.
Trevor: [00:53:54] Thank you. Bridget?
Bridget: Okay, so DevOps Days Minneapolis was this past week. I am editing the videos and getting them posted so everyone can enjoy the fabulous talks that we had. I would encourage everybody to check those out. I can’t even name one specific one that was the best because they’re pretty much all the best, but relevant to the interests of the folks here, we did have Katherine Daniels from Etsy and Jon Cowie from Etsy talking. At DevOps Days Minneapolis, which was amazing. And Jon Cowie had a bonus rant at the end of his talk all about privilege and inclusivity in tech. So everyone is gonna have to watch that once I get that up.
Trevor: Nice. I guess that makes it my turn. So lately I’ve been, I guess I just finished Batman: Arkham Knight, which was, quite fun. My computer was capable of running it, and I happened to get it before they stopped selling it on computer. Besides that, there’s a tool that’s for a new Git client that somebody was showing me that is currently invite-only called GitKraken, as in release the Kraken, but also GitKraken as a, you know, kind of a double entendre, which again, wordplay, always fun. So take a look at that. We’ve got some conference updates as well. DevOps Days Pittsburgh, Ohio, Detroit, Raleigh, and Charlotte are all in the works. Take a look at devopsdays.org to get more information about those DevOps Days events.
Bridget: [00:55:34] Oh, and Stratton’s not here, so I should mention Chicago. DevOps Days Chicago coming up August 25th and 26th. You can use the code ADO10 for 10% off registration. You should go. It’ll be amazing. There’s basically a ton of DevOps Days right now. Trevor will be there.
Trevor: There’s another Chicago one.
Bridget: I will not be there, sadly, but DevOps Days Melbourne’s going on right now, and there’s, you know, Boston coming up, all sorts of ones around the world as well. So definitely worth checking out. Also, there’s OSCON next week, so maybe it’ll already be going on by the time this is posted to iTunes. But I’ll be speaking at that. You’ll be at DevOps Days Chicago, Trevor. People can also find me at OSCON, Velocity New York, Operability.io in London, and Velocity EU.
Charity: I’ll be at Operability too.
Bridget: Yay! So we’ll be seeing folks at a lot of different conferences this year. And we have a newsletter, if Matt ever sends it out. Matt isn’t here to give himself a hard time about this, so I have to say it for him. arresteddevops.com/banana-stand. I still haven’t watched any Arrested Development, so— It’s the best way to know about upcoming podcast episodes and cool news with DevOps. We also have an iPhone app if you dig that kind of thing. You can download it for free at arresteddevops.com/iphone.
Trevor: [00:56:57] Thanks again to our sponsors, 10th Magnitude and VictorOps. Be sure to thank them and visit them at arresteddevops.com/tenthmagnitude and arresteddevops.com/victorops. Thanks to Charity, Patrick, and Mike for joining us today, and thanks as always to our loyal listeners. If you enjoy Arrested DevOps, we’d appreciate it if you’d visit arresteddevops.com/itunes and leave us a review in the iTunes Store. We’d also love to know if you’ve got any thoughts about upcoming episodes and any thoughts about this episode. You can leave comments for this episode at arresteddevops.com/40.
Bridget: You can also check us out @ArrestedDevOps on Twitter. We’re always happy to get your input, ideas, or feedback at shows@arresteddevops.com, so please let us know any ideas you have for future episodes. Uh, I’m Bridget @bridgetkramhout, and I’m Trevor @trevorghess. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.
