Matty: [00:00:00] I wonder what the influence of that water bucket-driven development would be. It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m your co-host, Matt Stratton, @MattStratton on Twitter. Arrested DevOps is brought to you by 10th Magnitude, a company that figures if you’re listening to this podcast, then you must be pretty cool. 10th Magnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/10thmagnitude. This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools such as Chef, Docker, and AWS, so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arrestedevops.com/datadog. So tonight we’re going to be getting down with GitLab. I’m joined by Joop van der Woerd. Did I say that right? Or got it closer?
Joop: [00:01:20] Yeah, that’s about right.
Matty: About right, about right. And we’re going to talk a little bit about the history of GitLab and some of the fun and challenges of running a large open-source project. So you want to tell us a little bit about yourself, Joop?
Joop: Sure. I am Joop. I’m the Vice President of Product at GitLab. That means that I’m responsible for everything that goes into the releases of GitLab. For instance, making sure that every month we release something that’s interesting for both our very big community, which also includes our customers. And I have a background in— I did a little bit of engineering before this, both at GitLab and at some other companies. Before that, I actually came from neuroscience. So I left that to go into tech. I thought tech was much more interesting than the stuff I was doing. And I ended up at GitLab.
Matty: Awesome. Well, we should have had you on our cognitive neuroscience episode a while ago, or maybe when you get to that one, because I know Joop told me he’s working backwards in ADO episodes. So maybe you’re going to get to that one and say like, oh, why wasn’t I on that one?
Joop: [00:02:29] Yeah, I know. It’s actually my degree is actually cognitive neuroscience, my formal degree. So yeah, that would have been good. I haven’t listened to it yet.
Matty: So, can you tell us a little bit about the history of GitLab? So, I personally kind of have only been familiar— well, I guess I was going to say only for the last couple years, but for all I’m going to do, you’re going to tell me that that’s as long as the project’s been around, and so maybe I’ve known it for as long. But when did kind of GitLab get started, and yeah, what’s the history?
Joop: Yeah, so you’re correct, it hasn’t existed for many more than a few years. It’s quite a nice story. So in 2011, a man in Ukraine, Dmitry, he was working as a PHP developer. And this was the time that at his organization, they wanted to switch to Git, because this was the hot new thing, and he wanted to use it. They started to use it, but there was not a real good solution to put this somewhere. There was GitHub, but company didn’t allow him to put the code outside of the organization, off-premises. And there was not a good alternative to that. So he decided, well, why not do it myself? So being a PHP developer, he started in Ruby on Rails. And there’s a nice thing that I always like to mention. At the time that he was living in Ukraine, he didn’t have any streaming water. So he would come home from working and programming in PHP every day, He would come home and he would work the whole night on GitLab without streaming water. So whenever he needed water, or anyone else in his house, he had to walk at least 100 meters to a well, bring down the well, get a bucket, put the water in the bucket that he took, walk back to the house, and meanwhile he is developing GitLab. He did this for a few years, and he put the project up on GitHub, because this is where the programmers were at the time. And it started gaining some traction. And he just continued his life. And for about 2 years, he still didn’t have streaming water. But GitLab was getting more and more popular. He was just working on it in his free time. So at around 2013, Sid, a Dutch guy, or Sietse is his real name, but his nickname is Sid. He thought, you know what, that GitLab is a pretty cool project. Why don’t I make a SaaS solution out of that, gitlab.com. So he did. He sent an email to Dmitry and he said, hey Dmitry, I’m going to start this. I’m going to start gitlab.com. I’m going to ask people for money and they can host their code there. And that’s it. I’m not going to involve you. And Dmitry, being the nice guy that he is, he said, go for it. Good luck. Enjoy. And Sid started that. A few months later, GitLab.com was running, people were using it. Sid was always checking Twitter, and Dmitry tweeted, I would like to work on GitLab full-time. So Sid contacted Dmitry, he said, okay, why don’t you work for me? We start a company together. And that’s basically how GitLab as it is today was founded.
Matty: [00:05:38] That’s actually a really great story because I like how it’s kind of like, okay, I did a thing and I was working on it, and then it’s like, oh, I just wanna work on this. Cool, let’s do it, versus any kind of big plans. I was trying to think of, like, some, and not analogy, but something about the, while you’re in the middle of coding, you need to stop, and the amount of time it takes you to walk and get your water, like, is this something to do with the time between commits? I wonder what the influence of that water bucket-driven development would be in the patterns that exist in the early GitLab code.
Joop: One of the nice things about the, the story is that it was after founding GitLab, he still didn’t have streaming water. He only, I think in the current time, so by now he moved to the Netherlands where he has streaming water, but his house in Ukraine, he doesn’t have streaming water yet. I think now he has a well next to his house, so he doesn’t have to go as far anymore.
Matty: [00:06:39] Yeah. Well, it’s iterative, right? It’s continuous improvement when it comes to that. Yeah. I mean, I remember my first encounter with GitLab, was probably the way that I think a lot of people do when they’re looking to solve, or at least at the time, looking to solve for an on-premises GitHub-like implementation. And I had, again, had a customer that we couldn’t put their code on GitHub.com. They weren’t in a position for GitHub Enterprise. And it was like, we really needed to POC the concept of even getting them to use Git in the first place. It was like they were on some ancient version of Accurev that nobody— everyone was afraid of touching because nobody knew how it worked. And I was like, we need to get you on Git, we want to do it quickly and fast. And that’s how I came across GitLab. And I’ll tell you, the thing that I think, like, endeared GitLab to me at first was the fact that it uses Omnibus, the Chef packaging. Like, I went to install it, I’m like, this is— and I wasn’t working for Chef, but I’ve been using Chef for a long time. And I was like, that’s really cool, use Chef to install GitLab.
Joop: [00:07:47] So, Yeah, this, this, so Omnibus has actually been a big deal for us because GitLab wasn’t always using Omnibus. I mean, that hasn’t existed for that long. Before what we did, it was a Ruby on Rails application, so it’s, it’s kind of hard to ship to people. So what we had is just really long installation manuals that you had to walk through. And I, I remember, and you know, there’s still people that run it like this, it’s at least 12 steps and they’re quite involved. And they, you know, you’re installing dependencies of dependencies and you have to configure a whole lot of things on the way. That’s quite complex. So when we discovered— we, I mean, we didn’t make Omnibus, right? But we saw that we could do this with Omnibus. That made a very big difference for us because suddenly we went from having this very complicated method of installation, we went to this almost instant installation, like single command installation. This was very powerful. And I remember we were nowhere near the size that we are now, but I remember that downloads went up like 1,000% at the time. It was incredible. The adoption grew much, much faster since then. And I think part of the success that we have nowadays, we really have to thank to Omnibus. So, and we still put a lot of effort into developing for Omnibus. It’s the foundation of how we distribute GitLab.
Matty: [00:09:09] Yeah, it’s a powerful thing. And I know just from my experiences with Chef, you know, pre-Omnibus Chef is, you know, you talk to people who the last time they install Chef Server with Chef, you know, something pre— I’m gonna be like, oh, like, no, it’s so delightful now because it’s just all packaged up. And again, it’s a complex system that can be automated. And then you can, again, you’re focusing on getting the features and the usage because your customers and your users, they just want to use the tool, right? And that sort of time to, like, first delight, if you have to, in order to even see if you like the experience of using something like GitLab, if I have to go through, like you said, this, you know, 12-page or whatever, maybe 12 pages long, but this long step of manual things, I may just bail at some point or just sit there and say, this is hard. And we see that with a lot of stuff with Chef, and that’s why we say we wanna get you up there, that time to first delight as quick as possible so that you can see the value when you’re using something. And, you know, ’cause your core competency is not being really good at installing GitLab, it’s writing the code that goes in there.
Joop: [00:10:18] So that’s— Yeah, I think the threshold to start using a product is incredibly important. And for us especially, it’s a very technical product. It’s developer-oriented. That doesn’t mean that it has to be hard to use. It has to be extremely easy to use. And that includes the configuration and the installation. So every opportunity that we have to make this easier for our community, we take. It’s really important for us to make this as simple as possible. And what we did lately is we actually have a package repository. So It uses Omnibus, so it just gets the Omnibus package so that now you can just install GitLab by doing apt-get install gitlab.
Matty: Right. So I know that, so one of the things with GitLab that I think is interesting as a company is really the open organization, right? Being not only open source, but you’ve referred to them being an open company. So first of all, can you tell me a little bit about what do you mean by saying it’s an open organization or an open company? What does that mean at GitLab? And then we could talk about how you got there.
Joop: [00:11:26] Sure. So what it means is that everything that we think we reasonably can do in the open, and that has some benefit to the greater community to be open, that is publicly visible and open to see and contribute to, and in many cases, open source. Practically, that means that all our development, so that means from inception stage of having an idea for a feature, feature or having a bug report to the stage where someone picks up that issue, does the code review, merges it, and releases it, that is all completely open. So that is all on a— of course, it’s on a GitLab instance that is publicly accessible and where people can contribute. So this is actually gitlab.com, the server that we have running there. And this means that anyone, if you have a good idea, you can create an issue and we might start working on it. Or if you see that there’s an issue that we wrote down and you would like to contribute to, you can do that as well. You could even be involved in the code review process. So this is the part where we do open development, but it doesn’t constitute the whole open organization. What we did is that after doing this, we decided, you know, there’s much more that we can do, and there’s a lot of benefit to be gained for ourselves first, But also for the rest of the community. So what we did is we started to open up our handbook. So on our website, you go to about.gitlab.com/handbook, you have our company handbook, and it contains basically how the company works, all the rules, all the things you are allowed to do, how certain processes work. This gets updated daily, and it’s just, you know, it’s just a website with a few markdown files. And the website is open source. And on the bottom of every single page, there’s a little link to that specific page in the repository, and everyone can contribute to that. So that’s how we bring our organization in. And we started to do the same with our support. We started to do the same with our operations of our public instance, etc.
Matty: [00:13:37] Yeah, that’s interesting. And it’s something I’m curious about. So you talked about, again, that you’re opening up your company culture, the way that the company works. And I think that’s cool to have that handbook in the open like that. And you talked about, again, you’re developing in the open. From a product management perspective, you talked about from your job, you’re thinking about your roadmap, you’re thinking about which features and how you prioritize them and everything. Is that all completely done in the open, or is some of it still a little like, we’re not quite ready to be completely transparent about this thing yet because it might never happen kind of thing?
Joop: No, it’s completely open. So you have it, it consists of 2 parts. You have on our website the page Direction, so it’s about.gitlab.com/direction. That’s basically our— we don’t call it a roadmap because we don’t want to commit to shipping something on a certain date because we want to be flexible, but that’s basically our roadmap. And then on the other hand, we have the public issue tracker, the same one where all our developers are active, That is where I do all my work. So that’s where we, and with we, I mean anyone in the community, whether they are employed by GitLab Inc. or not, where anyone works on features, where people think about features. And this is also where I write down my thoughts, where I give my opinion about what we should do and when we should do certain things, why and how we should do certain things. That all happens there. We feel that there’s not a really good reason to not do this in the open. We are confident in that with the feedback that we get from the community, that we can always ship something better. So whenever I have a certain idea that is maybe interesting for new customers, which in the end is what, of course, we are interested in, if we put it out in the open, we have a good chance that we get feedback from either existing customers— this happens a lot— or potential customers. Or other members from the community, which are also really important to us. So, if we wouldn’t have done this in the open, we wouldn’t get the feedback. So, yeah, everything is in the open.
Matty: [00:15:56] So, I guess, I don’t want to say playing devil’s advocate, but when I think about transparency of this, and that’s something I think is really important, and I talk a lot, well, I talk a lot in general, but when I talk about DevOps culture, I talk about organizational change, one of the things One of the things that we talk about in general a lot is that the, you know, we’d say the list of things that you can’t automate is a much shorter list than you really think it is, right? But then similarly, I talk a lot about how mandates are a thing to be avoided, right? You don’t wanna just be coming down from upper management and saying, we’re doing this now without— and I go back to that same thing and say that list of things that you really can’t share the why to the rest of your organization is a much shorter list than you really think it is. And I think a lot of times people avoid avoid that transparency because of either A— well, it’s always, I think, because of some type of a fear, right? Either fear that I am not going to be— you know, you could go back to that maybe I’m making this decision for the wrong reason. So, if I can just tell you to do it and you’ll just do it, then I won’t have to defend my decision. So, that’s the easy thing right there, laying on the authority, or the fear of I’m not— you know, because again, you might have to— it does require some effort to know if this is the the right thing to share? Because depending on your organization, you may have things that are under confidentiality agreements, or you may have patent, you know, again, and if you have a default perspective of share nothing and only share when I think about it, versus my default position is share except for the exception, that requires a lot more work. So, what would be some things you would maybe give if I’m thinking about wanting to help my organization understand how to become more open, even knowing that I might not be able to do it all at once, But what are some good suggestions for moving into a more open type of organization?
Joop: [00:17:56] It required a lot of pushing towards being open. Internally, we had a lot of resistance from everyone, myself included. We thought a lot about, is this really a good idea? Is this something that we want to do? You talked about, for instance, confidentiality. What we do when we get a support ticket, which is private, of course. And our customers don’t want to be named in public for most cases. I think for 99% of the cases, they don’t want to be named. So what we do actually is that we do create an issue on the public issue tracker. We just sanitize their name and we put a link to our support system. So there’s a link there for anyone that has access to the support system that can see, okay, this is a customer, or this is this specific customer. We label it as being a customer, and then we just say, hey, this big customer, or hey, this medium customer that does this and this, enough for some employees of GitLab Inc. to realize, oh, that is this and this customer. And then just the issue. And other than that, the process is the same. So that, I think for us, the struggle was, are we not adding too much process, too much extra work by going to the open? Our experience is that, well, yes, of course it adds some, but the positives massively outweigh than negatives in that case. That is one side. The other side that you also mentioned is that what happens if you have to speak the truth, right? So we get a feature request from a customer and it goes through support usually. One of our engineers, they make an issue in the public issue track and they say, hey, yo, do you think this is a good idea? What do I say if I think that it’s not a good idea? Well, I have to say that I don’t think it’s a good idea. My obligation is that because we are open is that I have to say, well, I don’t think it’s a good idea because this and this and this. And I have to just be very open for feedback on that and to be able to hear people say, yeah, but in our situation, we really need this. Or, well, okay. Or sometimes, well, I think you’re wrong and you’re not listening to me. You know, you have to be very open for feedback. In the end, what we want to do is we don’t want to be right. We don’t want to be authoritative. We want to build a really good product. So listening to these people is extremely important. That doesn’t mean that we will build everything. And, you know, we cannot build everything and we have to make certain choices. And sometimes someone will be disappointed because their specific workflow requires a specific thing and we can’t ship that. But the only way to get respect from your customers and to be transparent is to just communicate that and say, okay, I understand your use case, but we can’t ship this. Or, you know, you have a problem, maybe we can solve it in another way. That requires a lot of effort, but in the end, the product will be better because you will gain another fan of your product, even though it might not fit one-on-one with what they were hoping to get.
Matty: [00:21:06] Yeah, and I think that’s, you know, the last thing that anybody wants to feel is ignored, right? Because, again, this feature that I want is the most important thing in my life. And like you said, in the larger reality, it may be a super edge case that you just have to prioritize against. But feeling like at least it was heard and heard enough to say, you know what, I actually Again, it’s not because you’re wrong to want this thing. We just can’t do it for whatever reason. And sometimes either, like you said, maybe it’s a resource prioritization perspective, which is just, this would require a ton of effort to do and it affects, impacts one customer versus things we could do to make 10,000 customers delighted, or this actually inherently would break something for 10,000 customers. And I think that that’s That’s just when it’s— I think it’s really hard because you hate to say no to people, but it’s better to say no because than to just ignore, you know, to have these things, I guess, happen more behind closed doors, and then you have no idea. At least this way, I know that I was heard, and we had a little bit of a discussion about it. And to me, in my experience, engaging with company, you know, with product, in that way is— and when it’s a discussion, so I’ll tell you, you know, and I’m not trying to pick on Slack because they actually do a really good job, I think, from communicating things, but a lot of times, you know, they’re very responsive to when you reach out to them and say, hey, it would be really cool if Slack did this, and what you get is, it doesn’t do that, it’s not on the roadmap, but we’ll tell the developers. And then you’re kind of like, Where’s the discussion about this? Because I’d like to, you know, so, but that’s a lot more work, right? So, I think that’s the gist maybe of your experience, right? As you said, it does add extra effort to be open like this. And that’s probably why it’s one of the reasons it’s not the default position that a lot of organizations take. Are you like, I know that there are some organizations that are like 100% transparent you know, they publish salaries and things like that. And I don’t know, what do you feel about things like that? To me, that I, I go back and forth on how open is too open.
Joop: [00:23:38] Yeah, so we don’t do that. Um, things like revenue information, salary information, that is something we don’t share. The reason for that is that, at least, I mean, this is, this is, this also comes from my, my, uh, my opinion about this, is that this is not immediately valuable to the community. It works really well as a marketing tool, I can imagine, and it’s interesting. I like to read about it from other companies, but it’s not immediately interesting to our community. It’s not immediately beneficial to our community in the same sense as that, you know, the information that we do disclose and everything that we are very open about We want people to contribute to the things we share, like our website, like our product, like our operations even. And when we would do that with salary information, that it wouldn’t follow the same philosophy.
Matty: [00:24:42] That makes a lot of sense to me. So, what about, like, so we talked, you know, being open. We’ve had shows in the past, we’ve talked about things like blameless postmortems or doing postmortems in the open. What are your kind of experiences with that? I know you talked about, you mentioned in our notes about an outage or some downtime a couple of years ago in 2014, and kind of was that part of the learning experience maybe a little bit, like having that actually happen?
Joop: Yeah. So, just for context, we run GitLab.com, which is a private— it’s our own GitLab instance that we run. You can put your projects there for free. And the reason that it’s completely for free is because we want to have a good place to load test GitLab. But we run it as a production server. We put our own stuff there. So we want to keep it up. It has to be backed up. It has to work 24/7. So in 2014, we had a big brownout. I don’t remember exactly what happened. But the instance was down for many hours. At the time, we were with, I think, 6 people in total in the organization. And we were all in Europe, so we were not distributed yet. At least all the engineers weren’t. And I think this happened partially while we were sleeping. It was a drama for us. So when we discovered this, this is of course really bad because there are some small companies at the time already on GitLab.com, and they were really depending on this. What we decided to do, the first thing we did is that we went offline, we opened a Google Doc to discuss this, and then we said, you know what, why not just open this up and share this? I put a link of the brownout, I put it on Hacker News, and we got a lot of positive responses. People started helping out us there, and basically people started to say, wow, this is great that you’re sharing with us what is happening. And even though the instance was down, I have so much more faith in you now that I see that, one, you’re working really hard on it, and two, you’re happy to disclose exactly what is happening. So at that point, and I think this was one of the moments where we were really realizing that being very open and being very clear in what is going on, what are we doing about it, we really don’t want this to happen either, that really struck a chord with us. And it really set us thinking. It was not exactly that time that we started opening up everything, but I think that was one of the things that led up to it. Of course, we were already an open source product. And what we do nowadays is that we opened up our operations. So we have a repository where we have issues, where we do our operations as well. And one of the things that I personally really like, and it’s, you know, we’ve had some problems with GitLab.com. Now it’s pretty stable. We had some problems in the past. It was a little bit slow. It was a It was unreliable. It was offline too often. We had too much downtime. We were really frank about it. We were really open about, okay, this is the situation. This is how it is now. We know it’s not good. We even had on our website saying about, you know, it said, this is GitLab.com. It’s slow. You should realize that we have downtimes. You know, your data is safe. Everything is backed up. But yeah, it’s just not as fast as we want it to be.
Matty: [00:28:17] It’s all safe. It just takes a little longer.
Joop: Yeah, yeah, yeah. Yeah, we were not satisfied with it ourselves, so we wanted to be really open about that. Nowadays it’s pretty fast, but I think people appreciated that, us just being really open about it. And one of my favorite things is our GitLab status Twitter account.
Matty: It’s—
Joop: I know that it’s just our engineers, our DevOps engineers that man that. So whenever something happens or they are working on the instance, then, you know, they put, they put messages there. And we have some engineers and they’re just They’re just so funny. You should just check that Twitter account out. It’s really funny. Oh, NFS is playing up again. It’s really open and it shares the emotions that our engineers are having. And I think it’s great and people respond really well to that.
Matty: Yeah, there was one I saw from— looks like it was a couple of weeks ago, but just says, woohoo, GitLab.com is loading again. Yeah, yeah. And that’s the thing.
Joop: [00:29:18] I think it humanizes it.
Matty: Too, right? So, when I think about, again, when you’re open with this stuff, people always have to remember that there are people behind these systems, right? And especially, it’s interesting to see the impatience that a lot of operational folks have with operational systems and their uptime. And you’re like, but you’ve been on the other side of that pager, right? If I’m an ops person, and Amazon is having an outage or something, that’s a place where you would think there would be a lot of empathy because, yes, it’s making my life difficult that AWS is down, but I know there’s someone who’s having a— there’s a lot of people that are having a much worse day in the same way that I do. And I think that when you see organizations be clear about this and communicative in a human way, it empowers that that empathy for it, and you get more of that HugOps reaction on Twitter, which is the, oh, wow, GitLab is having issues, HugOps to the GitLab ops team, because we know that they’re having a bad night and they could use some love, right? Versus if it’s just this outage, you’re just like, oh, I’m trying to do work, right? What’s going on with that? And I think that goes a long way. And I think part of the reason people get scared about being transparent and public about the postmortems, if you will, or even those statuses, is because there’s probably a couple of things. One is just concern about maybe there’s this thing like, oh, well, that could be taken out of context, and then I could be held liable against an SLA some way or something like that. So, maybe sometimes people wanna hold their cards close to the vest because they’re afraid of some legal action based on it, or they feel like, oh, well, this makes us look unreliable, when the reality is complex systems fail. And if you try to hide it, you actually probably just look more foolish than someone who just owns the fact that this is gonna happen. It was interesting, like, on the HangOps Slack team, there’s been a conversation today about incident response, talking about this idea about celebrating, not I’m not necessarily celebrating failure, but one of— so, Charity Majors, who previously was at Parse, Facebook, and she’s been on our show before, but she was talking about this idea of saying, you know, she had a software engineer she worked with, and she said, you know, he started, and he, you know, said he’s a backend engineer, he joined up, and 8 months later had not caused a single production-impacting event. And she said, I scolded him, and I told him to step it up. He’s like, you’re being too— he’s like, if you haven’t broken something in 8 months, then you’re probably not trying hard enough or you’re being too cautious. Again, you can only have that kind of blameless culture that’s necessary if you can be open about it, even open inside and out because as soon as you’re being transparent about it, you’re not scared of being wrong. You’re not scared of failing.
Joop: [00:32:46] Yeah, I think, I think what we have learned is that being open and especially being transparent and very communicative in the sense that we are people responding to people online, people on Twitter, it has always paid off. There has not been a single situation that we regretted being very present online. This, this, this goes for our operations and our DevOps, but it also goes for anything else. So Whenever we have a new release, for instance, or if we are in the media at all, we always make sure that we are there. And that means that I am there with my personal account, with maybe a GitLab account. CEO is there. Everyone that is interested in it, feels like commenting there, is there. And it really helps that when someone says something about your product or your service, and you are there to say to them, you know, okay, I hear you, um, this is what we are doing, maybe we can do something extra, and show— and showing that you’re doing something extra, people always appreciate that. And it’s very— it’s really, it’s really easy to be online and respond to people, you know. You do the same thing as those people, you just go to a forum and you just post something. The only thing you have to do is be open and transparent and be a little bit friendly and This has always paid off for us. I cannot stress enough how good it is to just be transparent. And once you get this into your system, once you get used to doing this, people will, you know, this will become second nature.
Matty: [00:34:27] So I’m actually kind of interested about your operations too in general. So you’ve talked about kind of, we talked a little bit about the beginning days of the company and you made reference to, oh, at this time in 2014 or whenever it was, we had about 6 people or anything. So, what’s the growth curve, first of all, been like? And where are you, you know, yeah, like, you know, how do you do ops? How do you actually run your organization internally when it comes to operations in terms to, you know, release and things like that?
Joop: Sure. So, in 2004, I joined GitLab at the beginning of, very beginning of 2014. At that time, we were with six people, of which four engineers, which included the CTO and the CEO, the founders, meeting Sid. A year later, so this was a year ago, we were with nine people, and we went to Y Combinator, and we did that whole thing. Now we are with about fifty people, and I think about half of them are engineers, and we are growing very fast. So we will keep expanding. Expanding the team like this. We do things in a very particular way, and it has a little bit to do with our history as a company, as an open source project. So, the whole story about Dmitry releasing GitLab and doing that, he started, he released the first GitLab release on the 22nd of the month. And when you are alone and you have an open source project, it’s kind of nice to have this solid release date. So what he did, he decided, you know what, I’m just gonna release every 22nd of the month. So we still do that. Every 22nd of the month, we still release GitLab. I always call it the release train. And I announce in our Slack channel, I’ll say, choo choo, the release train is going. And this is something that we try to get the whole company behind. And not just the whole company, the whole community to get behind. We want people to really look forward to the 22nd. And we feel that this is something that’s worked because we’ve done it 51 times now. And we’re going for a 52nd, I think without fault, always on the 22nd.
Matty: [00:36:49] Nice. And that gives you some, like, it makes, I like the thing, like it makes it kind of an event too, right? It’s a thing you’re looking forward to. It’s like an exciting thing. I was just chuckling a little bit because I was— I have pulled up, you know, I still have your GitLab status in the other tab. And there was a tweet from a while ago that said, because GitLab.com has 3 single points of failure, we expect the service to be unavailable at some point. And it’s just, again, as an operations person, you read that, you’re like, yep. Yeah, but you’re being open about it. You’re like, yeah, we know it’s gonna happen. And it’s like, like that would be a thing for everybody to say, but you’re actually saying it.
Joop: Yeah, yeah, we have this, we have this engineer. I won’t call him out because he wouldn’t appreciate that, but he’s, he’s just so funny with these tweets, and I— he makes me laugh every single time. And I, I’ve been following this, and I, I follow this Twitter. It’s, it’s just, it’s so funny. I mean, it, it has a bit of sadness because we use these entities ourselves, so whenever it’s down, we’re all like, ah, it’s down again, but he’s just, it’s just deliciously cynical that he can be. And I think now they’re down to one single point of failure and they’re trying to solve that as well. But yeah, it’s really great. I think it makes this situation much lighter for everyone.
Matty: [00:38:09] Yeah. Because again, it’s real, right? So when you’re— so you talked again, like, so, you know, you use, obviously use GitLab for your own, you know, projects and things. And then what’s When it comes to, like, kind of building for, you know, again, being now, you know, speaking as a fellow toolsmith, or at least, you know, working for a tools company, and thinking about how tools influence behavior, what are some of the things when you think about from a productization perspective about what are the, maybe those behaviors that are influenced by features and plans and things that you do with GitLab and that you would influence based upon the way that you as a team work as well?
Joop: That’s an interesting question. I think one of the things that— so, we always want everyone to use GitLab, and especially we mean internally. The whole dogfooding concept is extremely important to us. The way that we think about it is that, well, I’m just going to be upfront, we would try to use GitLab for everything. Everything, right? And it doesn’t always work because GitLab doesn’t do everything. So in a lot of ways, what happens is that we run into something, we run into a certain wall, and we say, okay, we want to have this, we want to be able to do this, and we see that we’re not able to do that. And what we start to do is we just try to solve it within GitLab. So we start to build features within GitLab that solve that particular problem. I think an interesting example is that quite recently— so we’ve grown a lot past year, and we moved. We had a feedback tracker where we would get all the feedback that was external. And we decided, you know what, we’re gonna do this all in GitLab. You have the ability to vote now in GitLab. We added that recently. So we said, okay, we don’t need that feedback tracker anymore. So people started putting all their feedback in our internal issue tracker, which is great, but suddenly we had the problem that we had thousands and thousands of issues where before it was maybe a few hundred. And it was extremely hard to follow up with these and to handle these. And what we do is we just look for ways of improving the product to solve that problem.
Matty: [00:40:42] I think that’s an interesting approach too. The thing that happens when you’re in this space is you want to be everything to everyone, but then thinking about how kind of the Unix philosophy influences tool vendors and people who are using tools, which is the— when you think about a tool that does one thing and then you chain them together, and obviously, you don’t wanna just do the thing because you’d like to be able to be a good solution, but also, and this again is the thing that I think about in the space as a consumer of things and someone who works for a vendor, a tool vendor, is also like, when do you kind of cut bait on it and say, okay, you know what, we tried to make this happen with GitLab, and that’s actually the wrong thing. There actually is a really good solution for that, and it doesn’t necessarily take away from people’s delight in using GitLab, and they can still use it just as awesomely, but maybe we We should just say, just integrate over to this other thing for that one piece rather than trying to build it ourselves. I think that’s a tough balance because I don’t think you wanna default to that position necessarily, but I think you maybe need to have— is having the better part of valor to know when it’s time to do that and say, this just isn’t what we do. We tried it really hard, and there’s actually a really elegant solution for that that hopefully is not very competitive for us in other ways. So, we could be very embracing of them. I know it’s hard because people kind of want this integrated landscape, right? And especially as you start talking about larger organizations that have, like, different little parts of them, the one piece that does all the things, it gets hard. But I think that that’s a thing that we still should strive for, rather than just saying, oh, well, that’s not what we do. Although, I mean, to a certain point, you probably don’t wanna GitLab to be, like, your accounting software.
Joop: [00:42:47] Now, I think there’s— I have quite a nice example for this. So we want to create this integrated environment, right, where you can do your whole workflow from start to end, up to deploying and having everything ready. But we work with Git, and we made the decision to work with Git. So that means that we don’t work with other version control software. That also means that if you are coming from a different version control software, you should work as Git is intended. And we built GitLab around Git as it is intended. So that means that sometimes we get feature requests from people from the community, from customers that say, oh, we want to do this specific thing because we’ve migrated from SVN, for instance, where we can have this very specific permission management on very specific parts of the code or maybe specific directories. And then it’s up to us to say, well, I’m sorry, but if you clone a repository, you have the entire history available. And we’re not going to change that because this is the way that Git works. The reason that so many people adopted Git is because it’s so lightweight and because you can do everything local, which is because you have everything at hand. So we’re not going to change this. And to those kind of requests, we have to say, okay, we’re not doing this. You know, we’re not even going to try this because this is simply not the way Git works. We will do our best to explain to you, we will solve the issues you are having in other ways. But this fundamental thing, no, you cannot do that. And we’re not going to do that.
Matty: [00:44:21] That’s, again, being, you know, customer-facing. Yeah. You know, I had a conversation recently where I said, you know, I feel like it’s the, I want you to take all my pain away, but your solution must still include all of the things that gave me my pain originally. You know, or can you make me have your thing, but it should have all the terrible things I have now? Too. And that’s just a lot of talking about, like, walking that line between cruel empathy and teaching the right hard thing. And I think a lot of people eventually understand that, which is, if everything was working well in your organization, then why are you looking at another solution? But then we sort of default back to this, but we have this, and we have this, and we have this. And you’re like, well, but maybe that’s your thing. Like you said, if you go back and use Git, the way it’s intended, that’s solving a problem. You’re not trying to make Git be SVN because if that was the case, why don’t you just stick with SVN? If it was so delightful for you, right? So, that’s great. What are some, just any other things that kind of in your experience, and again, working, being part of an open-source community and being community-driven, I’d like to, yeah, kind of interested to know a little bit more about the community engagement with GitLab?
Joop: [00:45:46] Sure. So, we have about 1,000 contributors, and that ranges from people contributing a single commit to people contributing hundreds of commits. And whether they are employed by GitLab Inc. or not, it varies quite a lot. I think some of the best features that we have in GitLab, and I will name some specifics, they were contributed by community members, meaning people that are not working for GitLab Inc. For instance, we now have a button that whenever your CI passes, which can take a long time, maybe you did code review already and it’s good to merge. So you want to merge it, but CI is still running, or it’s maybe still pending even. You don’t want to go back and check, like, oh, is CI already done? No. So what you can do now is you can just press a button which says merge when build succeeds. Really nice solution. We didn’t think about it. In fact, it just got contributed by someone from the community. And this person, we did give this person an internship with us, so he’s now working for us and he’s a great developer. But I think this is amazing. I think this is such a nice feature. We put it up in our release post. It’s really nice. And, you know, it just, it sort of came out of nowhere. And there are several others. There’s another one one where you have the fuzzy file finder. So you can quickly find a file if you just press on T. Also community contributed. Interestingly, so we have this philosophy when we talk about community, we include ourselves because we are the— we have stewardship of the project, but it’s an open source project. So we don’t own it. But we also talk about the community as in people that use just the open source, but also the customers. And what we see is that customers start to contribute as well, and they start to be active in the issues as well now that everything is open. A concrete example is CERN, the particle accelerator in Switzerland. They use GitLab internally. They are a customer of ours, and they actually contributed quite a lot of features to our Enterprise Edition, specifically related to authentication. So they were very open about it. They communicated in the open on our issue trackers. They send us merge requests. They work together with our developers, with developers from the community. And together we just build better software. And I think that’s really cool. I think that’s amazing. We have customers that contribute, people that are sitting in, you know, like I am sitting now at home in my little office, coding and building stuff that they like. It’s quite varied. Great.
Matty: [00:48:32] Yeah, I think it’s interesting too to be seeing kind of the shift, especially with enterprise, you know, sort of that behind-the-firewall approach with open source, with the ability that these companies are having, or even the appetite for contributing back. And I’ve thought about, you know, I think a lot of that is, I think there’s still companies that have challenges with it because of worried about, you protecting IP and other parts of their company, not understanding, not wanting to set precedents, or again, just not being used to that because they weren’t maybe a software company from the beginning. But I think over time, I think what companies are starting to see is it’s also part of attracting talent. Because if I’m a really good engineer and you want to hire me, one of the things I’m probably going to be asking you is, what’s your open source policy? Can I contribute things back to these projects that I think are important? And if you tell me no, I’m probably gonna wanna go work somewhere where I can do that, where we can share what, you know, not our core differentiator things, but, you know, and I think it’s really, it’s, again, everybody wins. And when, I think when companies understand that open sourcing, you know, that part of that is, again, it’s, they get other people working on their defects for them, you know, as well, just makes everybody go faster. You know, if I start start something, maybe there’s a contribution to GitLab that I’m like, okay, I want this thing. I can’t quite get the whole way there, but I can get it started. And then someone else, right, maybe it gets merged in as not quite there, you know, or whatever, but then someone else can pick that up and say, oh, I see where you’re going with that. I can bring it across the line, right? It’s powerful stuff. I’m really, you know, it’s— I just like the way the industry is going with the approach towards open when it comes to Yeah, I think so.
Joop: [00:50:30] This year, one of the things that we want to do is we want to really encourage people to contribute to GitLab. So we started a few initiatives. One of them is we have at least one developer right now, and maybe more in the future, that their full-time job is to coach incoming merge requests from the community. So this is not merge requests coming from within GitLab Inc., where we have, you know, developers working whole workdays, but just people from the community maybe making their first contribution. And this person makes sure that these merge requests, they get merged, they get finished, they look good. And he coaches these people in case that if the merge request gets abandoned, he can pick it up and maybe finish it. If not, the per— the person themselves, they finish it, and he makes sure that they do it in a way that we expect, and it, it all works accordingly. Another thing that we do is that— so I told you we have an open issue tracker. We have a specific label which is up for grabs. It’s not our idea, this is a public idea. And if you want to get started with contributing but you’re not sure what, you can just filter for this label. And it has issues that are small, quick wins, and kind of simple that you can simply tackle. So you So you can just find one of these issues, you can do it. It shouldn’t be too hard if you know a little bit of Ruby or a little bit of JavaScript, you should be able to do it. And this way we lower the threshold a little bit for first-time contributors and make it a little bit easier to become, you know, one of these contributors in the community of GitLab.
Matty: [00:52:05] That’s great. That reminds me a lot of when we had Phil Dibowitz from Facebook on talking about open source and about getting started in open source. Talked about those kinds of ideas about, yeah, it can seem intimidating to go to contribute and that there’s skills, maybe even besides just being a coder, where you can help contribute to a project by being really good at writing bug reports and being able to maybe go into outstanding issues and saying, like, I’m gonna reproduce this a little bit better and get some more output that makes it easier for someone else to handle the code, or again, taking the small stuff versus feature requests. But I think that’s great. So I’ll be interested to talk to you after a while and see how those initiatives are going for you. But yeah, this went by pretty quick. This is great. I’m a big GitLab fan too, so I was glad to hear a lot more history other than just what I knew from using the product.
Joop: [00:53:06] Good to hear.
Matty: So, before we get into the checkouts, just give a little update for listeners on some community and event stuff. So, remember, if you have an upcoming conference you’d like to see promoted on Arrested DevOps, fill out our handy form at arresteddevops.com/conf, like for conference. And I realized maybe I need a better URL since I always have to explain that one. Some upcoming conferences, so DevOps Days Rockies in Denver will be April 21st through the 22nd. ADO listeners can save 10% off with the discount code of ADO2016. And then we’ve got also DevOps Days Atlanta is going to be on April 26th through the 27th, and you can get 20% off with the discount code ADO2016. And you can also go to DevOps Days Seattle, which will be May 12th through the 13th, and they’ll give you 15% off with the discount code ADO2016. So at least this year, we’ve gotten the DevOps DevOpsDays folks to be consistent with the discount code name to type in, at least. Everyone’s offering a different discount. We have a lot of open calls for proposals. So, if you’ve got ideas for talks, both Rockies, DevOpsDays Rockies in Seattle, their CFP is open till February 28th, which may be in the past when I publish this podcast. I’m not sure yet. But ChefConf CFP has actually been extended, I believe, until March 15th or sometime in mid-March. March, and that’s at chefconf.chef.io. DevOps Days Atlanta is open until March 1st. DockerCon CFP is open until March 18th if you go to 2016.dockercon.com. Platform/SpringOne, their CFP is open until March 24th, so go to platformspringone.io to submit for that. Both— so DevOps Days Vancouver and Minneapolis and Abstraction, are all open until March 31st. DevOps Days DC is until April 15th, Salt Lake City till April 19th, and Amsterdam is open till May 30th. So lots of opportunities. So remember, as I always like to tell people, you don’t have to write the talk to submit it, just come up with an idea, you’ll figure it out after you get accepted. And we’ll go into our checkout. So what do you have for us?
Joop: [00:55:24] So I have 2 points in a checkout. I have one more technical one and one less technical one. Um, the— and hold on for the version information. iTerm2, the terminal emulator for Mac, it’s a really popular one, everyone uses it. They released version 3 beta. It’s really nice, it adds a bunch of cool features, one of them being tooltips, which learned me a lot about their new version. I really enjoyed this, so check that out, iTerm2.com. And I have another one, and this is less technical. If you like podcasts, and I suppose that you do when you’re listening to this, and I hope that you do, I have advice for a nice network, Relay FM. relay.fm is their website. They have amazing podcasts about tech like this one. I really recommend you check them out. They have so much stuff. It’s really nice. I’ve been listening to all their podcasts and I can’t stop. So that’s mine. What about you, Matt?
Matty: [00:56:27] So I have actually 2, one that I just realized to add. So the first one was something that I saw recently. So if you go to 10x.engineer on the web, you’ll find something kind of funny. So that’s the number 10x.engineer. And then there’s also an app that I recently started playing around with. I think it’s for Mac only. It’s called FlowState. It’s kind of a dedicated writing app. It’s been called the most dangerous app. It’s pretty expensive for what it does. So I’m not sure you want to try it out for $10. But the idea is it’s a distraction-free writing environment. But you set your buffer of time, and if you stop typing within that amount of time, it will delete everything you typed. So, it’s to sort of force you as if you’re trying to do some writing to write in a stream of consciousness way. So, if I say, for example, and like I started doing it with just like doing like a 5-minute thing and writing straight for 5 minutes was really, really hard. But it’s kind of an interesting idea. They talk about, if you go into the app, they talk about the philosophy behind why you might want to do that. So, if you’re feeling, if you got a, you know, someone gave you an iTunes gift card and you’ve got some credit to burn, and you wanna drive yourself crazy with writing, check out FlowState in the App Store. So, you also, for free and without stress, could subscribe to the Arrested DevOps newsletter at arresteddevops.com/banana-stand. It’s the best way to know about upcoming podcast episodes we have and cool news with DevOps. Thank you to our sponsors. You can check them out again, remember, at arresteddevops.com/10thmagnitude and arresteddevops.com/datadog. Thanks to Joop for joining us. This was awesome. Loyal or disloyal listeners for that matter. If you enjoy or you don’t enjoy Arrested DevOps, we still would appreciate it if you would go to arresteddevops.com/itunes and leave us a review in the iTunes store. That’s really how people find out about us. And we’d love to know what you think of this episode. So you can leave comments at arresteddevops.com/gitlab. And we are on Twitter @ArrestedDevOps, as you might guess, and feel free to email us at shows@arresteddevops.com if you’ve got input, ideas, feedback, and ideas for future episodes, especially. So, I’m Matt, @MattStratton. This is Arrested DevOps. And remember, there is always DevOps in the banana stand.
