← All episodes
EPISODE 55January 20, 2016

Open Source with Phil Dibowitz

Read the transcript

Phil: [00:00:00] But it drives me nuts when people talk about, oh, look at this new idea that we just came up with, and it’s never been new.

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m your host, Matt Stratton, @MattStratton on Twitter. Arrested DevOps is brought to you by Tenth Magnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. Tenth Magnitude empowers businesses to better collaborate across teams and achieve IT transformations using the cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/tenthmagnitude. This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools, such as Chef, Docker, and AWS, so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. So back in October of 2015, I sat down with Phil Dibowitz of Facebook to talk about his thoughts on open source and how someone can get started with it. I hope you enjoy the conversation as much as I did.

[00:01:37] Okay, so I am here with Phil Dibowitz from Facebook. We’ve been excited to have him on the show. He hasn’t been on before. We figured, you know, we were hanging out here at Chef Community Summit and I cornered him and said, Phil, we’re gonna sit down and we’re gonna do an interview. And he didn’t run away fast enough, so we’re now here. Sitting here. So now you’re here. So Phil, you want to tell our audience a little bit about yourself and what you do?

Phil: Sure. So I’m a production engineer at Facebook. And I spent a couple years building a pattern at Facebook of people being able to own their own tiers of machines. So my team built sort of a low-level set of core cookbooks that basically generate an API on attributes. So if you, for example, want to set a sysctl You have one variable assignment in your own cookbook and then you’re off to the races. You don’t have to understand all kernel tunables. If you need, you know, a cron job, you don’t have to know about all the other cron jobs in the system. You assign some stuff into an array and the magic happens. And we did that specifically not using a lot of the internal resources, using templates and notifications so that when someone deletes those 3 lines from their cookbook, the recipe, it all just goes away and gets cleaned up magically. And so sort of whole systems get managed as one as opposed to piecemealing it through. And it took us a couple years to sort of get everyone to migrate onto that system. But since then it’s been, it’s been really, really, it’s been really good. And, you know, through that we got, I got super involved in the Chef community. You know, became a maintainer, started adding features, little things that we needed, bigger things that I thought were useful to the community. Most recently the multi-package support in Chef. And then also a lot of our internal tools that people have found useful, Taste Tester and Grocery Delivery and stuff.

Matty: [00:03:29] How long have you been at Facebook?

Phil: November will be 5 years.

Matty: 5 years, and what did you do before?

Phil: Before that, I worked at Google. I worked on the Gmail infrastructure. And before that, I worked at Ticketmaster, which we were also really big into configuration management and trying to figure out how to scale that better. At the time, Puppet and Chef didn’t exist, so we actually wrote, a bunch of us wrote a configuration management system, which you can find on GitHub today, called Spine. And for its day, it was really fucking cool. But it doesn’t really have a community, and it wasn’t maintained. And so after I left, by the time I came back, it wasn’t sort of the right tool for the job anymore. Like, the industry had moved on.

Matty: Gotcha. Very cool. One of the things I wanted to talk about, too, is I remember at ChefConf this year, you sat on a panel talking about open source. And it was, to me, I don’t know if you had the same thing. I remember being at ChefConf this year and thinking, you know, we’re living in this very weird alternate universe, you know, where we had, you know, Mark Russinovich and Jeffrey Snover up there talking about running Linux on Azure, and The Canonical folks were talking about building stuff on Windows. And we’re in this— I always say, I’ve spent a lot of my career working with Windows. And I feel like Michael Corleone in The Godfather. I’m like, every time I think I’m out, they pull me back in, because they kind of have their stuff together. And I’m interested, just in general, on your thoughts around how you’ve seen, in your experience, open source as community, and just also as product, and within various industries, maybe evolve throughout your career.

Phil: [00:05:02] Well, so open source has always been a driving in my career, in my personal life, and the things I care about. So part of the reason that I was able to get started in this industry very young was because I was hacking on open source as a kid. You know, there’s no— you don’t have to get a job to do it. You just get to go and do the thing you want to do. And so that model was always very appealing to me. When I came to Facebook, a big part of my interview process was, hey, if we’re going to build this new way of doing things, I want to open source it. I want to talk about it. We worked with, you know, Chef, to make sure that everything we were doing from a scalability perspective in Chef for Facebook was gonna be available in what at the time was Open Source Chef. Now there’s just one Chef server. So it’s something that I feel super passionately about and I try to pay a lot of attention to. And what I’ve seen over the last 2 decades is, right, so open source started way before me, but it was the only thing. There wasn’t proprietary software really. You had maybe some local punch card things that somebody had written for their own internal processing or whatever, but it wasn’t a generic piece of software. Generic software was by and large distributed on BBSs across universities or whatever. And then you had this sort of long period of time of Windows and these proprietary apps and operating systems, and the thing that sort of changed somewhere in the late ’90s was as Linux became more popular, popular, and as you had Apache coming out and whatever Mozilla Firefox was called at the time, Phoenix or something, you had this awareness amongst people who were in industry who were kind of like, man, I kind of always thought that story that I heard about everything being open was neat, but that would never work. But you started to see that it was not only going to work, but it was going to be the better solution. So what you sort of, in my mind, have seen is people not having a choice, right? Like people, industries, companies, communities realizing that you, in order to really compete, you have to be able to move fast. And the best way to move fast is to let everyone else help you move fast, right? It’s not about some altruistic, this is the better way. As much as I like open source and I feel passionately about it, I think the thing that’s changed is people, it’s no longer the thing you do because for some morale or moral thing, right? It’s the thing you do because that’s the way to get the business done faster, right? If you write, like, I’ll take Facebook as an example. We realized after putting a lot of effort into trying to make our PHP apps scale better, we needed a runtime that was different from what people were doing in the real world, and we decided to do it in a way that other people could leverage it. So we wrote the HipHop PHP to C compiler. We open sourced it. And we didn’t just throw some code over the wall. We really tried to build a community around it. We tried to engage the Zend community, who has been great actually about, you know, kind of working with Facebook And now there’s a shit ton of people who will contribute to it and use it, and yeah, we have a bunch of people who maintain it internally, but a bunch of their work gets done for them, right? And things move forward without us having to shepherd everything. You know, people talked at Facebook about writing a config management system. Like, why would we do that, right? There’s great ones out there, people are already maintaining them. So when it comes to Microsoft, I think they’ve been forced to see this light, right? And they’re doing more and more open source stuff. They’re releasing more and more of their libraries and their frameworks open source. And I don’t think it’s, I think for a long time it was lip service. For a long time there was a dude somewhere who said, well, if we don’t open source the thing, you know, we’re not gonna be able to say that we’re open source friendly.

Matty: [00:08:59] I’m trying to think of like the term when it’s not like read open, but it’s like open source, the open source where you don’t accept PRs. You know, it’s like here’s the code, but we don’t accept contributions. And I think there was sort of that, and I think then that, like you said, that would be the lip service.

Phil: I know a lot of companies do it, yeah. And I think that was what they did for a really long time because they didn’t understand the community, they didn’t understand the benefit, and then you look at like, you know, what Azure’s doing and what they as a company are doing now. And I don’t think the whole company gets it, but I think there’s large portions of that company who have bought into this model, see the benefits, and to me, as much as I’ve been, you know, sort of anti-Microsoft because I was an open source advocate, I do, to me it’s exciting to see a big company that’s sort of been the antithesis of our model actually coming around. I think it’s going to make a better Microsoft. I think it’s gonna make a better Linux. I think it’s gonna make a better everything. When, you know, they may not do the same things the exact same way the exact same way we do, right? I don’t think Linux and Windows are gonna merge, but we give each other ideas and we build off each other in the same way that like Chrome and Firefox are better because they’re both there, right? They can look at the other one and go, well, I don’t like the way they did that, but it gives me an idea, and then they do a better thing, and then it goes back and forth, right? So to me, that’s what’s changed in the industry over the last 2 decades is people realizing that just ’cause you see my code and take my idea doesn’t mean that my product is no longer cool, because I’m gonna look at what you did with that, and then I’m gonna come up with my next idea, and we’re just gonna get better faster. So yeah.

Matty: [00:10:25] Yeah, and I think that’s one of the things that we’ve talked a little bit about. There was in our— we did an episode recently, and Robyn Bergeron was talking about this whole thing of where things have changed, and kind of open source behind the firewall thing. And one of the things that I was thinking about, and this was true with kind of the DevOps stuff as well, is I said, I think it’s been going on a lot longer than we kind of think about, but up until relatively recently, a lot of enterprises don’t— wouldn’t talk about it. I remember thinking, you know, there was a point, you know, I’ve talked about before, about when I was doing consulting, and I would go in, and I would talk about DevOps transformations, and, you know, people would say, you’re the 5th person to come in today to say I should be like Netflix. And the problem was, and that’s where kind of this unicorn model or idea came from, is because the only people you were hearing from were Facebook, or Netflix, or Etsy. And I remember thinking, and I’m like, but these other companies are doing it, they’re just not talking about it. And that’s what I think has been great with the DevOps Enterprise Summit. And when you hear about when companies like Target— and Nordstrom actually has always been very vocal about it— but even GE and all these big companies are willing to talk about it, that really helps it see, like, OK. And I think what has become the realization is that a lot of this stuff that we’re talking about is not the things that provide your competitive advantage, right? You’re not talking about— like, Netflix doesn’t open source their video encoding software, right? I’m sure there’s plenty of stuff at Facebook that I’m sure you do not open source, because it’s your competitive advantage. But the stuff that’s plumbing, you know?

Phil: [00:11:58] Infrastructure scaffolding, those things.

Matty: Right, yeah. It doesn’t make any— like you said, it could just make you better. It makes everybody better. And it doesn’t matter if other people know how you are scaling your servers. They’re not going to— because you’re not in the business of scaling servers. If I want to come disrupt Facebook, it’s not going to be because I’m better at building 300,000 servers fast. Right. Absolutely. There’s other things. And I think that’s the thing that I think within a lot of enterprises is challenging, because they, within their IP, within their normal legal, it’s figuring out how do they not set a precedent or understand that just because it’s written here doesn’t mean that it’s special sauce.

Phil: I think there’s 2 things I want to comment on that. One is, you were talking about sort of the history of DevOps becoming a more well-known and accepted thing, and then also this sort of leveraging and releasing open source software, right? So on the one hand, I really agree with one point that you sort of brushed over there really quickly, which is I don’t think DevOps Like, the word is new. I don’t think the style is new. When I was a junior admin in the ’90s, and we were working on this old dot-com, and I was trying to learn anything I could from this guy who was really smart and had been doing this for a long time, and he had said something about a junior and a senior admin, and I mean, this really stuck in my head. I said, well, what’s a senior admin? What will make me a senior SA? And he’s like, well, you gotta be able to read the code of the applications you’re deploying. You’ve gotta be able to work with developers and bridge that gap. You’ve gotta be able to go talk to the database guys and bridge that gap. You’re never going to be maybe the C coder that they are, or the DBA they are, but your job is to not just be the systems guy, but to be the systems guy who can also touch all the other things in a meaningful way that bridges that gap. And I was like, oh, OK. But that’s what DevOps is, right? We’ve added a bunch of stuff, and we formalized it, and yada, yada, yada, but senior admins could always code. Senior admins could always work with other teams in a meaningful way. And it’s nice to see it formalized, but it drives me nuts when people talk about, oh, look at this new idea that we just came up with, and it’s never been new.

Matty: [00:14:05] It’s never been new, but I think what’s interesting— well, we’ve made the joke before. I have a friend of mine who says, I’ve been doing DevOps for a long time, back when I just called it work.

Phil: Right, exactly.

Matty: But what I think is, and I absolutely agree with you that coming up, this was the thing, but I think there was a pendulum swing where for a while what you were saying was actually not the case. I mean, we were getting into this that, yeah, there were very senior sysadmins who— and again, this happens a lot within the enterprise— where it’s like, I care about the box. Right. That go, so it’s not necessarily that it’s a new idea. It’s a very old idea, but it’s actually kind of a correction for changes that potentially were made that—

Phil: My take on that has always been those people are just bad at their job.

Matty: But the problem is there are a lot of them.

Phil: Yeah, absolutely. So then on the open source side, I think that’s much more difficult. I think you’re trying to work with a legal team inside your company to explain why we should or could give away a thing, and there is a lot of complications around that. And I’ve been very lucky to always work with lawyers who get it and go, okay, here’s our risk, what do you think, like, what’s the value? Let’s try and compare those and then just do it if we think it’s a good value and not do it if we think it’s not a good value, right? Which, that’s what you should do as a business, right? You don’t just release things ’cause, and you don’t just not release things ’cause. You make a value judgment. Which means you need lawyers who understand this entire space, which is a thing a lot of lawyers were not trained on until recently, right? You didn’t used to be able to go do an IP specialization when you get your law degree that covered the concept of releasing your own internal intellectual property as an open source project, right? So I think that that’s— I think we’ve always gotten it, right? But there was so much in the way between business and lawyers and marketing and all of these other things that to get through all of that was way too much effort. So unless you worked in a 3-person company where you could say, hey, I know the CEO because he’s my buddy since college, you know, that was fine, but if you’re in a big company, there was just way too much uphill battle to do, right? Whereas now we work in a world where there’s a pretty good chance someone in marketing’s gonna get it, and there’s a pretty good chance someone in business is gonna get it, and someone in the legal team’s gonna get it, and you sort of find the right person, and you have some chance of getting through that gauntlet. I mean, again, if you’re working at a Facebook or a Netflix or whatever, that’s probably not an issue. But in an older or more traditional company, yeah, there’s probably a path you can find.

Matty: [00:16:33] I think that’s where that shift happens too, is kind of the, the realization of that, you know, every company is a technology company now, or every company is a software company, you know, but whereas if you were inherently, by definition, according to what everybody at the company already thought is, well, we’re a software company, then you probably hired a lawyer who had a background in software. Right. Right, but if you are now coming to that realization, again, it’s getting the rest of the business up to that understanding of what does it mean to be a software company.

Phil: Yes.

Matty: Even if you’re, you know, it’s the CIO of United has the quote, which is, you know, we’re not an airline, we’re a software company with wings.

Phil: Right.

Matty: You know, and so you have to think about where that goes, but it’s challenging, ’cause again, and you also, I think a lot of times, you know, there’s a lot of fear of precedent, right? It’s, you know, certain companies that have very protective IP, I think their lawyers are concerned about if we do this, does this set some kind of a precedent that this other stuff that we’re super protective of is a thing, but—

Phil: That’s where policies come in handy, right? Like guidelines on what you might wanna open source versus what you might not want to open source and what the risk there is. And the other thing I think that’s changed a lot is I’ll be on a plane sitting next to a random person, and they’ll go, oh, what do you do? I work in technology, blah, blah, blah, blah. And then somewhere in there I’ll mention open source, and they’ll be like, oh yeah, yeah. And they won’t, you know, they’ll be a doctor, or they’ll be a business executive, or they’ll be whatever, and they’ll have heard of it. Like, they won’t know all the details, but they know that there’s this concept that people can release the things they do for free. For the public consciousness to have some concept of that just shows the massive shift we’ve gone through recently. I mean, that, the first time I had somebody sitting next to me, I think she was a flight attendant or something, who just happened to be taking a flight, and she was just like, oh yeah, yeah, yeah, yeah, I was reading a couple articles about that the other day, and it was just like, you know, mind blown, like, wow, somebody who doesn’t work anywhere in neurotechnology has heard of this thing, that, we’ve hit critical mass, kind of, you know? That was really exciting to me, and I think that’s the world we live in right now, where not everyone knows, but a critical mass knows. It makes our life easier.

Matty: [00:18:41] What do you think, through your, you know, you get like, spent, like you said, pretty much your career and quite some time in the open source space and with open source communities. So what’s changed? Or how do you feel that, as far as especially around the people involved, the community, we talk so much about community these days.

Phil: I think it’s really interesting. 2 things have changed. One is the tooling. It used to be really hard to run an open source project. You’re taking in patches, but then you’re taking in 4 patches and the patches don’t apply anymore. And then you got sub- 7 replies in your email thread and you forgot to reply to the dude who came back and said, actually, you thought this code was wrong, but it’s right. And it was a lot of, you had to really, really be on top of your shit. You know, things like GitHub makes this really easy. It’s not perfect, but oh my God, it’s a lot easier to run an open source project now. So I think the tooling in general makes it easier for the people running the projects, but also makes it easier to contribute. It used to be that you had to know what patch policy they had. Like, do you do patch -p0, patch -p1? Like, how do you generate your diff? Blah, blah, blah, there was like, there was just all this stuff you had to know about how to contribute. Now you send a pull request. You don’t even have to join the mailing list, right? Like, the barrier to entry is way lower. So I think that’s one thing that has enabled better, a better community kind of across the board. The other thing I think is we are struggling as a community to both grow in terms of inclusivity and welcoming people who are not of our culture. And when I say culture, I don’t necessarily mean you’re from another country. I’m a loud, obnoxious, aggressive guy, right? And if you’re not by any one of those axes, and you’re trying to push for your change on whatever mailing list I’m on, like, you may feel like I’m steamrolling you, and you may just go away. And like, maybe that’s your fault, maybe that’s my fault, maybe that’s nobody’s fault. But we sort of recognize it’s a problem, and so I think we’re, as a community, we’re struggling with the, okay, On the one hand, we need to find a way for that person who had a great idea to not be intimidated, to be able to have their voice be heard, and be a member of that community and feel empowered to be able to get their ideas across and to fight for the thing that they want to fight for, while at the same time also allowing the sort of culture of aggressive, and I don’t want to use aggressive, that’s not the right word, but like really detailed ripping apart of patches and ideas, right? That ability that I am going to tear down everything you’ve done, not because of you, but because it’s my job to give every single problem in your patch so that you can go back and make a better patch, be a better contributor, and our project gets better, right? That is the thing that has made open source code awesome, and we want to keep that, and I think it’s important we keep that, and we be able to keep that in a way that no one’s ever going to hold back on that kind of feedback. We need to do it in a way where we’re helping people to understand why that feedback comes that way and make sure no one’s ever using that as an excuse to attack people, right? Because there’s also that, right? There’s the, I’m just giving you a hard code review, but really I’m just being a dick, ’cause I don’t like you, ’cause your English isn’t as good, or you’re a woman, or you’re this, or you’re that, whatever the case is, right? So I think that there’s these two things that we’re trying really hard to bridge that gap, and to make something awesome, and it’s hard. I think people problems are hard. We’re really good at technology problems. People problems are much more difficult, and we’re getting so much better. You know, here at ChefConf, right, we got people all around who will help you deal with personnel problems, with code of conduct problems, and nobody feels oppressed. Nobody feels like, oh, I can’t go give my friend a hug, right? It’s a great balance of I can kind of be myself, it’s not a stifled office environment, and yet it’s a safe place, right? But it’s a hard balance, and not every community’s figured it out, and I don’t think there is any perfect answer, but I think that’s the thing that open source communities in general are struggling with right now, is how do you strike that balance? And that’s what I see. The tooling kind of was stage 1, and this is stage 2. Now you got all the new people coming in.

Matty: [00:22:58] How do you foster that?

Phil: That community that you want to foster in a way that still lets you sort of have some semblance of the way you do whatever you were doing.

Matty: The way you do the work, right, absolutely. So what would you say, so for people who, you know, I think about myself, and we get this a lot too when we talk to people who really take, I don’t want to say take advantage, but benefit from open source projects and open source software, and they really want to give back in some way, but it can be really challenging I think, because some of it’s intimidating and not necessarily, it could be partially because because of the things you mentioned, just by virtue of that, but also maybe just thinking, like, hey, there’s this— it’s such a huge thing. Like, who am I to do this? Like, what are some things that you would— if someone says, hey, you know what? I get so much value out of Chef, and I want to be able to contribute back in some way from an open source perspective. Like, how do I get started? Like, what’s the— you know, because I’m not probably going to go back and contribute to core directly as my very first thing. What are some good ways to get started?

Phil: [00:24:01] So there’s sort of 2 questions there. One is, what do we do community, and the second thing is what do you do as a guy who wants to be contributing, right? So I think I’ll tackle the second one first. When I was first getting involved in the open source community when I was, I don’t know, probably 12 or 13, I didn’t like— I could write a little bit of Perl, but like I wasn’t a coder, right? And I couldn’t get involved in any of these projects that I thought were cool. I’d find a bug and I’d be like, I can report the bug. God knows how to fix it, right? So what I started doing was writing docs. So I was a big fan of IPFilter back in the day, and I maintained the IPFilter FAQ. I wrote it, I maintained it, and I would just sit there and watch the mailing list, and like the questions that came up 3 or 4 times, stick them on a webpage, you know? And it wasn’t like, you know, it was like an HTML3 webpage that I wrote, you know? It could be a wiki page today. You know, I wrote FAQs, I would contribute, hey, your docs say this, but that’s not the case anymore, just an email to mailing list. Little things that, you know, people started to know my name and appreciate the fact that I was there even though I wasn’t contributing to the code. And from there, as my coding skills got better, I would see a thing, I’d be like, oh, maybe I can fix that bug, right? And I think that there’s no meaningless contribution. People are like, oh, that contribution wasn’t big. Who cares, right? Any contribution is a contribution, we’re a community. So I think that’s the one thing. The other thing is, have you been to DEF CON?

Matty: [00:25:23] I have not.

Phil: Wiseacre gives this really good DEF CON 101 talk, and one of the things that, there’s many things that they cover, like, you know, please take a shower every day. But one of the things he covers is, you know, some of the biggest name speakers in the world are there, and none of them are above any of you people who, you know, this is day one in the industry for you. If you walk up to a speaker and they don’t give you the time of day, if they are acting like they are better than you, please tell us, ’cause we don’t want that speaker back, right? And I say that to people at the Chef Community all the time. Go talk to Adam. Adam’s the nicest fucking dude you could wish to meet, you know, and people are super intimidated by him. I’m like, there’s nothing to be intimidated by. Go talk to Adam, that’s why he’s here, he loves talking to people. People get intimidated by me because I did a keynote. And like, I’m just a dude. I write some code. You can come talk to me, right? I don’t think there’s anyone in this community who won’t sit down and chat with you for a little while. And I think what we need to do is remind people of that constantly. Guys, there is no one above you or better than you. Go talk to whoever you want to talk to. And if they somehow come off like assholes, please tell somebody so that we can have that conversation with them, right? Because I think everyone comes off like an asshole by accident sometimes.

Matty: [00:26:38] Yeah, sometimes, and it can be the same thing. It’s like you may be on your way to go use the restroom or something, and I’m gonna try to talk to you, and you’re like, hey, not right now. And it’s not because you’re being a jerk, it’s because, you know, I mean, you just had like 3 Diet Cokes.

Phil: Exactly, yeah.

Matty: Sorry.

Phil: Right, and so I think as a community, the thing that we can do is constantly reiterate that that’s of value to us, that anyone can talk to anyone. There are no special kings up in the corner that you can’t approach. And I think if we continue to do that and continue to live that, not just say it, but do it, I think the community will just continue to get easier and easier to contribute to, whether it’s you want to contribute to Core Chef or whether you want to, you know, rewrite the docs or whether you just want to, like, be a guy who, you know, when we’re looking for a volunteer to help run one of these discussions, we’re like, oh, hey, could you help us run this discussion, right? And so I think that that’s— those 2 things are what’s next in terms of continuing this forward.

Matty: I think that’s great. I mean, I think that the point is, right, you know, by default, a lot of times people probably think contributing to open source means writing code, and there’s so many ways to contribute to an open source project that could fit into your particular skill set. I mean, just like you said, helping facilitate a talk, running, you know, a local meetup. Yeah, you know, for something like that. I mean, if that’s where your skill set is better, writing docs, I mean, it’s sort of the— I mean, it’s not a joke, but whitespace fixes, whatever. I mean, just even if it’s little stuff, because that little stuff is still a problem, right? It’s still a thing that needs to be done, and that’s a really great point.

Phil: [00:28:11] There’s a couple projects where they’ll have have people who are not great coders, but they understand what a good bug report is. So as bug reports come in, their job is to go, hey, please grab debug logs, this is how you do it. Or, hey, you didn’t include this information, please include it. Please tell us what operating system you’re on and what version you’re on, whatever. And they know how to make sure those bug reports that are coming into Bugzilla or the mailing list or whatever it is be awesome bug reports so that those users can get an awesome experience without some developer having to go back and forth 6 times with some guy who made a bad bug report. That is an amazing service that those people are providing even though they’re not coders. So there’s millions of examples of things you can do to contribute to a community without actually having to write code.

Matty: Great. So I have one more question before we wrap up. And so just what would you think from having been at Facebook for a while, and obviously everyone’s very familiar, very high-profile company, people have— what would be something that in your experience, and maybe just from your day-to-day work, from whatever, that might be surprising to people that they might not know?

Phil: [00:29:13] About Facebook or about the industry? About Facebook. Oh, um.

Matty: Or just—

Phil: I would say about Facebook, the thing that probably surprises people is the same thing I would say about a lot of the big companies I’ve worked for, which is internally, the goal is to do the best thing we possibly can for the user. You know, we have constant discussions on like, how do we best ensure people’s privacy? How do we best do this? How do we best do that, right? Like, it’s like, our users are people and we care about them, right? And I think that that’s the thing people don’t believe, ’cause there’s a lot of companies out there that don’t operate like that, but I think most new companies do. Like, that sounds surprising, but, you know, I’ve worked for a lot of big kind of dot-com-y companies, and I think most, I mean, when you live in a world where people are sharing on a platform, whatever that platform is, user trust is your bread and butter. And so it’s the thing you care about, because it’s, why would you do it if you didn’t care about that, right? It’s, yeah, there’s a quote in our S-1 that says something to the effect of, we make money to build products, we don’t build products to make money. You know, and like, you don’t put that in your S-1 if it’s not something you don’t believe in. So I think those sorts of things where, you know, yeah, there’s some really just fucked up companies out there, but there’s also a bunch of companies who like, work really, really hard to do the right thing. Sometimes they do things well, sometimes they make mistakes, whatever.

Matty: [00:30:41] I mean, that’s all human. I mean, the thing, these companies and all these things are made up of people.

Phil: They’re made up of people. So the first day when you’re in a meeting, and what they’re talking about is, like, how to be the best steward of data, or how to be, you know, how to care about user privacy, or whatever it is, you know. You know, that makes it a fun company to work for, when you can get behind the ideals of that company. You know, I’ve never been in a company longer than 2 and a half years, and I’ve stayed here because I love the people, I love what we do, I love the technology. You know, social networking isn’t a thing that is critical in my life. Like, I use Facebook like everyone else.

Matty: That’s not why I came to the company.

Phil: I came to the company mostly for the technology, and I stayed because of the company itself. So I think that’s probably, I’d say, the most surprising thing. Not probably to me, but to most other people.

Matty: Awesome. That’s great. Well, thanks for taking the time today. I really appreciate it.

Phil: Yeah, it was fun. Thank you for having me.

Matty: Absolutely. A little housekeeping. Remember, we have a newsletter, arresteddevops.com/bananastand. It’s the best way to know about upcoming podcast episodes and cool news with DevOps. Thanks to our sponsors. Be sure to visit them at arresteddevops.com/10thmagnitude and arresteddevops.com/datadog. And loyal listeners, if you enjoy Arrested DevOps, we would appreciate it if you would visit arresteddevops.com/itunes and leave us a review in the iTunes store. We’d love to know what you thought of this episode. Please feel free to leave us comments at arresteddevops.com/open- Be sure to check us out on Twitter @ArrestedDevOps. We’re always happy to get your input, ideas, or feedback via email at shows@arresteddevops.com. Let us know any ideas you have for future episodes. I’m Matt, @MattStratton. This is Arrested DevOps, and remember, there’s always DevOps in the banana stand.

BROUGHT TO YOU BY

Matt is joined by Facebook Production Engineer Phil Dibowitz to talk about the state of Open Source today, the changes it has gone through in his career, as well as some of the best ways to get started in the world of Open Source.

Matty corners Phil Dibowitz, a production engineer at Facebook, at the Chef Community Summit in October 2015 and talks open source: how it has changed over Phil’s career, why companies do it, and how someone who benefits from it can start giving back. Phil hasn’t been on the show before, and, as Matty tells it, didn’t run away fast enough.

Phil’s Background

Phil has spent a couple of years building a pattern at Facebook for teams to own their own tiers of machines. Phil’s team wrote low-level core cookbooks that expose an API on attributes, so setting a sysctl or adding a cron job is a variable assignment in your own cookbook, with no need to understand every kernel tunable or every other cron job on the system. They deliberately used templates and notifications so that deleting those lines from a cookbook cleans everything up. It took a couple of years to get everyone migrated. Along the way Phil got deeply involved in the Chef community as a maintainer, most recently adding multi-package support, and mentions internal Facebook tools people have found useful, like Taste Tester and Grocery Delivery.

Phil will hit five years at Facebook in November. Before that came Google, working on Gmail infrastructure, and before that at Ticketmaster, where Puppet and Chef didn’t exist yet, so a group of them wrote their own configuration management system called Spine. It was very cool for its day, Phil says, but it never had a community and wasn’t maintained, and the industry had moved on by the time of Phil’s return to it.

Open Source Because It’s the Faster Way

Matty describes the odd alternate universe at ChefConf, with Mark Russinovich and Jeffrey Snover on stage talking about running Linux on Azure, and says it feels like Michael Corleone: every time the Windows chapter looks closed, they pull Matty back in. Matty asks how open source has evolved over Phil’s career.

Phil got into the industry young by hacking on open source as a kid, and when interviewing at Facebook wanted to be able to open source what they built. Phil sees a shift since the late ’90s, as Linux, Apache and the browsers showed open source would not only work but be better. What changed is that companies no longer do it for moral reasons: “the best way to move fast is to let everyone else help you move fast.” Phil’s example is the HipHop PHP to C compiler, which Facebook released and tried to build a real community around, including engaging the Zend community, so a lot of the work now gets done for them. It is also why nobody at Facebook wanted to write a new config management system when good ones were being maintained.

On Microsoft, Phil says it was lip service for a long time, but thinks large parts of the company have bought in and finds it exciting. Like Chrome and Firefox, each side makes the other better, and Phil doesn’t expect Linux and Windows to merge. Matty adds the open source that doesn’t accept PRs as the lip-service version.

Plumbing Isn’t Your Competitive Advantage

Matty says enterprises used to keep quiet about how they worked, so the only stories were from Facebook, Netflix and Etsy, which is where the unicorn idea came from, and clients would say Matty was the fifth person that day to tell them they should be like Netflix. Matty credits the DevOps Enterprise Summit for getting companies like Target, Nordstrom and GE to talk. Most of what gets shared is plumbing, not competitive advantage. Netflix doesn’t open source its video encoding, and if someone is going to disrupt Facebook it won’t be by being better at building 300,000 servers fast. Phil calls it infrastructure scaffolding.

DevOps Isn’t New

Phil wants to comment on the history. As a junior admin in the ’90s, Phil asked a mentor what made a senior sysadmin, and the answer was being able to read the code of the applications you deploy and to bridge the gap with developers and DBAs. “That’s what DevOps is,” Phil says, and “it drives me nuts when people talk about, oh, look at this new idea that we just came up with, and it’s never been new.” Matty quotes a friend who has been doing DevOps for a long time, “back when I just called it work,” and suggests the current wave corrects a pendulum swing toward admins who only care about the box. Phil’s take is “those people are just bad at their job.” Matty says the problem is that there are a lot of them.

Lawyers, Precedent, and Policy

Getting a company to release code is the harder part, Phil says. Inside a company you have to persuade a legal team to give a thing away, and Phil has been lucky to work with lawyers who weigh the risk against the value and decide. Lawyers weren’t trained on releasing internal intellectual property as open source until recently. Now there is a decent chance someone in marketing, business and legal will get it, and you can find a path through the gauntlet even at an older company.

Matty says that’s the realization that every company is a software company, and the rest of the business has to catch up. Matty adds that lawyers worry about setting a precedent for the IP they are protective of, and Phil says that is where written policies come in, guidelines on what you’d release and what you wouldn’t. Phil’s sign that open source has reached critical mass is that a stranger on a plane, even a flight attendant, has heard of it.

Tooling and Inclusivity

Phil says two things have changed in communities. The first is tooling. Running a project once meant juggling patches that no longer applied and email threads you forgot to reply to, and contributing meant knowing a project’s patch policy. With GitHub you send a pull request and don’t even need to join the mailing list.

The second is inclusivity, which Phil says the community is struggling with. Phil describes being a “loud, obnoxious, aggressive guy,” and says someone who isn’t may feel steamrolled and go away. The project needs both: people who feel able to bring ideas, and the culture of ripping apart a patch so the contributor can make a better one, without hard code review becoming an excuse to attack someone. Phil says ChefConf strikes a good balance, with people around to help with code of conduct issues, and that “the tooling kind of was stage 1, and this is stage 2.”

How to Start Contributing

Matty asks how someone who gets a lot of value from Chef but isn’t going to commit to core as a first step should begin. Phil started around 12 or 13, when a little Perl was readable but bug fixes were out of reach, by writing docs. Phil maintained the IPFilter FAQ, watched the mailing list, and put questions that came up three or four times on a webpage. Phil’s view is “there’s no meaningless contribution,” and adds “Any contribution is a contribution, we’re a community.” Matty adds meetups, facilitating a talk, and even whitespace fixes. Phil mentions people on projects who aren’t coders but triage bug reports, asking for debug logs and version numbers so developers don’t go back and forth six times.

Phil also wants people to remember there are no gatekeepers. Phil cites a DEF CON 101 talk that tells newcomers that no speaker is above them, and says the same goes at Chef: go talk to Adam, who loves talking to people, and “there is no one above you or better than you.” If someone comes off as a jerk, tell an organizer, because everyone does by accident sometimes.

What Surprises People About Facebook

Asked for something surprising, Phil says internally the goal is to do the best thing for the user, with constant discussions about privacy, and that user trust is your bread and butter for any company where people share things. Phil points to a line in the S-1: “we make money to build products, we don’t build products to make money.” Phil has never stayed at a company longer than two and a half years except here, and says the technology was the draw and the company is why Phil stayed.

This episode's guests