Continuous Integration – CI Told You So!

Matty: [00:00:03] Hello, everybody, and welcome to episode 5 of Arrested DevOps, the podcast that probably won’t destroy your career with bad advice. I’m your co-host, Matt Stratton.

Trevor: [00:00:18] And I’m your co-host, Trevor Hess.

Matty: [00:00:20] This week, we’re going to be talking about continuous integration, which is a something that I really know is important, but I don’t know a whole lot about, given that I’m on the ops side of DevOps. So I’m really looking forward to this episode for my own personal knowledge gaining, and we have a pretty great panel as well. But before we go to our panelists, let’s start with the retro. I’ll go first. Mine is pretty quick. Basically, what I’ve learned since the last episode is that I hate Subversion. That’s about it. I really want to not use it anywhere that I can avoid. I’ve been trying to get Chef cookbooks to work inside Subversion, and it’s just making me really angry. So, Trevor, what about you?

Trevor: [00:01:13] Well, mine’s slightly related, not the Subversion part, but I went to a chef training course last week and so now I know how— I know what Chef is, how to use Chef. It’s really fun and really cool, and we only really got to focus on Linux stuff during the training, but I’m really interested in getting it working on Windows and doing some configuration management there. I’m also really happy that it’s built on Ruby because Ruby is fun and easy, which kind of segues into our new section. So we’re gonna introduce a new section called User Stories. It’s a tentative title. If you guys come up with something better, tweet it at us and we’ll consider it. This week we’re going to be talking about configuration management, and seeing as that’s really Matt’s wheelhouse, he’s going to briefly explain what that is.

Matty: [00:02:04] Right, thanks. And the point of these user stories or requirements gathering, whatever we’re calling it, is to introduce a topic in just a couple sentences to whet the appetite for something we’re probably going to delve into a little bit deeper in a future episode. But configuration management is really— when we talk about it this way, it’s exactly what it sounds. It’s saying, how do I manage to treat my infrastructure and control it and understand it and know exactly what it looks like at any given time and know where my changes occur? One of the great examples of why a configuration management tool is great is think about the idea if you had, you know, a fleet of 1,000 servers. And this happens to tech ops people, they know. You sit there and InfoSec comes to you and says, hey, we’ve decided we no longer want RDP listening on 3389. You need to change that all to this other random high port. And in the old days of not that long ago, you’d be looking at the case of now your sysadmins, your tech ops people have to go, yeah, you could write a script, you could do whatever, but you’re ending up going and doing a touch on 1,000 servers. With a configuration management tool where you’re defining this, you could execute this change in 30 minutes, which is great. The other thing is that you’ll also know all the places that it’s executed, and you could also easily roll it back. There’s a lot more to configuration management. We’re gonna have an episode about that in the future. If you are interested, especially in Chef, I highly recommend one of our related podcasts, I guess, which is called the Food Fight Show, and I’ll put a link in the show notes. And that is a Chef-specific podcast where they cover all sorts of DevOps-related things but primarily focus on the tool Chef. So that being said, let’s stop talking config management. Let’s start talking continuous integration with our great panel, who I’m gonna let introduce themselves. We’ll start with our first panelist, who is Mathias Meyer. Mathias, you want to introduce yourself? Tell us a bit about yourself?

Mathias: [00:04:16] Yep, sure thing. I’m Mathias Meyer. I work for Travis CI, where I’m mostly responsible for infrastructure amongst other things, and bacon and coffee as well. And Travis CI is a continuous integration platform which originally started out for open source projects, but we now evolved it to a product platform. I myself have somewhat been exposed to continuous integration for a long time now, and it’s quite a fascinating topic. And yeah, I’m afraid there’s not a lot to tell about me, but I’m very much looking forward to the talk, to talking about continuous integration.

Matty: [00:04:52] We also have with us Joe Hirn. Joe, tell us a little bit about yourself.

Joe: [00:04:57] Thanks, Matt. I’m Joe Hirn. I work for DevMind Software. We’re a boutique consulting company. We operate out of Wicker Park in Chicago. We work on mainly local projects. We do remote work as well. We try to do all of our work in-house so that we can sort of foster knowledge sharing amongst our team and kind of help grow and learn what one project is doing and help another project. So there’s a lot of cross-pollination there with that. I probably start about 3 to 5 projects a year for the past 3 years now. So getting a project up and running and getting the things that are going to keep it healthy are something that’s very important to me and something that I kind of revisit every couple of months. In terms of the topic, continuous integration, I’ve been using it since around 2007 when Jenkins was called Hudson and people paid for a thing called Bamboo, and I’m interested to have a conversation with you guys about it.

Matty: [00:05:59] Awesome.

Trevor: Great.

Matty: [00:06:00] So I think one of the things that would be great for our audience before we delve into our long list of specific questions is, at a 30,000-foot level, how would you define continuous integration? So, you know, as Reddit would say, talk to me like I’m 5.

Trevor: [00:06:17] It’s explain like I’m 5, Matt.

Matty: [00:06:19] Explain like I’m 5.

Mathias: [00:06:21] Explain like you’re 5.

Trevor: [00:06:23] Wow.

Mathias: [00:06:24] That is an interesting— Joe, do you want to have a try, or—

Joe: [00:06:28] Sure, I’ll take a stab at it. So, you know, continuous integration is essentially just a build server at its core, right? It’s just a build server that’s going to take your commits and hopefully every commit do a build, and it— the main problem that it really solves essentially, just the continuous integration part, not what you can extend a continuous integration server to do, is the works-on-my-machine dilemma where one person says, hey, worked on my machine, but, you know, then you check out their new code and it doesn’t work because they had something stateful going on, they have their machine configured a particular way, so I like to think of continuous integration in terms of, like, how you use it and how far you go into using it. It’s kind of like a dial on the responsibility of a team. You know, when you start with just curing the works-on-my-machine dilemma, that’s one level of responsibility. When you take it all the way to continuous deployment or continuous delivery, you know, that kind of is turning up the dial on how much you actually have a confidence in your team to make sure that they’re covering all the Ps and Qs and doing everything that’s necessary.

Trevor: [00:07:39] Mathias, you wanna follow on with that?

Mathias: [00:07:41] Yep, absolutely. That’s a great view from the tools perspective. I personally find continuous integration a lot more fascinating from a cultural perspective because while you have this tool that is sitting there and waiting for you to make any changes and to, well, do whatever is necessary to make sure they’re running, I don’t know, continuous integration to me is a little bit more about making changes fast, integrating them often— that’s where the integration point comes from— integrating them into master, into trunk, whatever you use, whichever version control system tickles your fancy, basically, and to iterate on them quickly and to have this system in place that is not only, you know, trust in your team, but you also have this tool, a continuous integration server, which can make sure that all changes that you commit, that you integrate into master or whatever, that they still work, you know, and that they work in the bigger picture of whatever project you’re working on, not just your local machine, because, you know, the biggest thing that we keep hearing is, it works on my machine, which is probably true, but, yeah, it’s the continuous integration server is to be, you know, the unbiased judge of whether something works or not, or at least whether it works in the context of tests.

Joe: [00:08:55] Yeah. Can I just agree strongly with— I love your use of the word culture there, because I sort of said developer responsibility, but I really do think that it’s kind of that culture of information sharing and open communication that continuous integration kind of fosters, because, you know, we’ve all got to stare at each other and wonder, why is the build broken? Who broke it? Who’s going to fix it? It really— that responsibility and that chore alone really kind of creates— sets the tone for a different culture on a team.

Mathias: [00:09:27] Yeah.

Matty: [00:09:28] I’ve heard it said that a version control system is basically a communication system for developers. It makes them communicate with each other. And I think that goes to Mathias’s point about— right, that it’s a culture. It’s not just the tool, and the tool helps drive that culture. Or it’s a way you do things. So I have a question, which— one of the things that I know with a lot of— I do a lot of work around continuous delivery, and continuous integration is a key piece of that. And what I find is when I work with clients, they tend to want to not do CI all the way. They’re like, sure, we’ll do continuous integration, but we really want our feature branch, and we still want to do all this merging and all this branching and merging. And so my question, and that’s why I put in our outline, I said, how do you handle that kind of scenario? Because my understanding of CI is that you have one source of truth, like we said. You’ve got your mainline branch, and that’s where you’re working. How do you handle branching? So if I’m coming to you and saying, hey, I’ve got my, you know, mainline code, which is current, and then I’ve got features that we’re developing, and I want to create a branch for that, and I want to just keep integrating up, how do you usually recommend that being handled, or am I misunderstanding?

Mathias: [00:10:52] I would probably ask you, why can’t you develop this feature on master? What is keeping you from doing that? That would be my follow-up question. I cannot really recommend anything because in that regard, you know, every team is a unique snowflake. But my question would boil down to that. You know, people want to— we have a lot of customers who use feature branches. It’s a very popular thing. GitHub uses a lot of feature branches, and it works for them because they continuously ship feature branches to production. They try them out, you know, they use a small percentage of their app servers that they deploy a feature branch to, so it kind of works.

Joe: [00:11:26] It will—

Mathias: [00:11:27] you will still see whether something works in production or not. But, yeah, the question really boils down to why can’t you work on this feature in master, or what could we do to enable you to work on this feature in master, which, again, comes into this culture thing. You come into the realm of, you know, maybe we need a feature flip on this. Maybe we need to be able to just enable it for a small amount of people. Maybe we need to isolate it better, so that it doesn’t affect any other code that is running in production, things like that. The feature branches, I don’t know, it’s an interesting thing. We use feature branches ourselves, and I’ve— as you said, your point is a very good one because I’ve come to think about that quite a bit, which is mostly thanks to Jez Humble, who was very— he’s very inspirational in that regard. I don’t think there’s a single truth in how you can develop that, but I think if people insist on feature branches, you come back to this culture thing of asking them, you know, what can we do? What can we do to enable you to ship this to master?

Matty: [00:12:28] Yeah, it’s funny that you mentioned Jez that way ’cause he was definitely someone that, yeah, was a few weeks ago was tweeting about that exact same rant about, okay, I want to do continuous integration, except— and he’s, like you said, he’s very adamant about the no, boom, trunk, done. And my problem is I’m the ops side of DevOps, right? So I understand at a certain point, but I start to get a little bit out of my league when I’m talking to a developer to say why this is better for them, you know, because I’m a believer, you know, if I want to introduce change, I need to be able to say, this is why it makes your life easier. And this is not, you know, it’s the right thing to do, but hey, if you’re all working on mainline and we’re not doing feature branches, for example, this is where it makes your life easier. And that’s where some of the challenge lies for me, and the same thing happens with continuous delivery, is I’m like, I know how to explain this, why it’s great for a release engineer or a tester or, you know, an ops guy.

Mathias: [00:13:32] The customer.

Joe: [00:13:34] Yeah. So if I could throw my 2 cents in here, I want to first of all really say that I like what Mathias was doing with sort of driving the question of why are we doing this and why can’t we do trunk-driven development or trunk-based merging. Because what he’s doing there is he’s challenging complexities that people just add onto a project because maybe they saw somebody else doing it and having success with it, and really kind of wondering, what exactly is this getting us, and why can’t we deliver to trunk, you know, or master, or whatever your version control decides to call the canonical mainline, right? There’s a guy, Paul Hammant, that has a whole great series of many, many blog posts on the topic about trunk-driven development that I kind of point people to, because it’s my belief that we should all be committing to master as often as possible. I believe that a feature branch should be— first of all, I don’t believe in feature branches. I think if you want to create your own branch to save your work and push it up and go home and come back the next day and make sure that you saved it somewhere, that’s fine. But you’re not integrating if you’re not putting it on master. Things that happen on a branch don’t really happen. On an open source project, you know, such as the Linux kernel or even, you know, smaller GitHub projects, when you’ve got people doing all sorts of stuff on the side without necessarily collaborating on the actual project vision, and then they want to present their work to the maintainer and say, this is what I think should go in, then I think, you know, the pull request and feature branching thing really kind of fits that model a little bit more. But for a typical team that’s working together and talking together, to me, commits don’t happen if they’re not on master. So I like to kind of think of feature branches as almost like a little mini coup, a resistance in the development lifecycle, where it’s like, hey, I’m special, and I’m going to go work on this feature branch for a little while. But the longer that resistance is there from merging it into master, the longer it’s going to be before I can give you feedback on the commits that I see, your team members can give you feedback. Maybe you wrote some helper functions that the team could have started using.

Trevor: [00:15:48] That’s just—

Joe: [00:15:49] yeah, I just really wanted to kind of expound on Mathias’s excellent example of sort of pressuring people to, why aren’t we committing to master? What are feature branches really getting us? Because simplicity isn’t easy, right? So you can introduce this concept, and then sometimes it’s very hard to pry it out of people’s hands. But I’ve taken down many Git flow posters off people’s walls and introduced them to trunk-driven development, and their lives have never been better.

Trevor: [00:16:14] A lot of the projects I’ve worked on, we kind of use Git as a— and feature branch is kind of a gated check-in kind of thing where we use continuous integration to make sure the stuff you built on your feature branch works. You know, you’re continuously merging master into your branch, your feature branch, and then merging your feature branch back into master when your feature is built. But what you guys are talking about sounds way better.

Matty: [00:16:38] Are you doing that every day though, Trevor?

Trevor: [00:16:41] Yeah. Okay, so every time we pick up a new story, we create a new feature branch, and then periodically through the day you should be merging development in. You’re keeping an eye out for what’s coming in. You’re keeping an eye on development to see if something new is there and pulling it in to make sure that what you have isn’t out of sync. But anything that’s going to be pushed out to our user acceptance or our QA doesn’t get broken.

Matty: [00:17:07] Boy, that sounds like a lot of work.

Trevor: [00:17:09] It is. It really is.

Mathias: [00:17:13] I think this is where pull requests actually fit in a little bit, you know, in this whole integrating-into-master thing, because pull requests, you know, the idea is when you send a pull request on GitHub, it creates this merge between the upstream branch and your changes, so it is sort of an integration, but yeah, if you let it live for long enough, it will obviously still fall behind master, but it’s one thing that GitHub, you know, as you push changes to the upstream master, GitHub will always update that merge commit, so you always have an integration point, but of course, on the one hand, you know that it integrates or it doesn’t, but on the other hand, it also tells you that, oh, I can wait a little bit longer shipping this feature. It integrates well with master. I don’t have any rush to ship this to production. And I think what Joe said, the open source is actually a great example. In open source, it works very well with feature branches and with pull requests because it’s just how contributions work. And, you know, that’s a very popular feature that’s being used in GitHub, and it obviously makes sense. But, yeah, it boils down to getting into people’s heads and getting them to question what they’re currently doing, and that’s sometimes a hard thing. That’s why it’s always useful to get new people on the team to actually bring up these questions as well.

Matty: [00:18:34] Where do you think you start, right? So if you’re in an organization that’s not doing continuous integration, but you’re convinced. You’ve heard this podcast, you’ve said, you guys have me sold, I need to do this. So where do you start?

Joe: [00:18:48] By doing.

Matty: [00:18:50] Set it up.

Joe: [00:18:51] It’s easy. Things like Travis make this so easy to do. And, you know, whether it’s Travis or whether it’s any of the other online things that are available, or you just download Jenkins and fire it up and just run it on your local machine and then show it to your teammates. Say, this is our continuous build. It’s been broken for the past week. You know, and if you show that to your manager and they don’t get it, then heaven help you, because it’s just such an immediate source of feedback that I don’t know what the roadblock to adopting CI in your project would be.

Mathias: [00:19:29] This brings back memories to companies I’ve worked in where we introduced continuous integration. That was like 10 years ago. That was an interesting challenge. I think back then the major issue was that big Java projects or even C++ projects, they didn’t have an automated build, you know, and that’s the crucial thing that you actually need for a continuous integration build. For something to take your source code and look at it, but it doesn’t give you any benefit. You know, the obvious first thing that you need is some way of an automated build, so that’s where tools like Ant came from. Like, my first Java builds were automated with Ant, or C++ builds with Make. But that’s becoming less and less of an issue, thankfully, because, you know, as apps are developed with scripting languages, web apps are built with Django, Rails, all these kinds of tools that all come with automated build tools out of the box. So there’s at least one barrier removed for most projects to actually start using it. You could just type any command, whatever you run in production, as a basis to start continuously integrating your project.

Matty: [00:20:32] If I’m understanding— because where I see it is that you can get really excited about this and see the potential where you say, hey, this is great. I can do soup to nuts. I can do it that I do a check-in, it does a build, it runs these tests. It runs Selenium Grid, it does all this, it automatically deploys it to QA, blah, blah, blah, and then that’s where you have this, like, boil-the-ocean problem, right? Oh my God, where do I start? And it sounds like the answer is you start by just automatically building and looking for a build error. Forget about unit tests, forget about coverage, code analysis, or whatever, just saying, did the build work? Yes?

Joe: [00:21:06] Yeah. So this is getting back to the recurring theme of, like, responsibility and the culture that the continuous integration build server fosters, because as soon as you set up your CI server and realize your code’s not automatically building, you should feel this internal shame that there’s not a single command that you can run to compile your project. And then hopefully you will build that script, check it in, and now you’re at that first step of CI. Then you’ll see that little test phase that you have maybe disabled or it says 0 tests, and it just sort of lets you know, like, hey, wouldn’t this be cool if I had, like, some passing tests? Maybe 1 or 2. We’ll start there. And then there’s that deployment checkbox that you want to check. You want to say, that would be really cool if it did this thing. So I love that the concept of CI sort of introduces this framework of things that you should probably be doing and be concerned about, because it’s not just, you know, I wrote this if statement and it compiled on my machine. It’s how am I gonna actually deliver something to a user and get customer value out of it, right?

Matty: [00:22:16] So that goes into another question that I have too that I start to see is how would you say when we talk about CI tools, like we talk about Jenkins or Travis or TeamCity or whatever, and this might be pedantic, it might be splitting hairs, but what’s the difference between saying I’m using a tool that’s called a CI tool for doing continuous integration or using a CI tool to orchestrate a workflow automation? Because it seems to me like we’re kind of growing what we want these things to do.

Mathias: [00:22:48] I kinda agree, but, you know, at the core, a continuous integration system is something that runs commands. So even at the simplest— in the simplest way, it can orchestrate whatever you want it to do. But it’s probably through things like continuous delivery, you know, that notion of continuous delivery, of always being ready, or having a build pipeline to ship something to production through automated unit tests, then automated integration tests, ship it to QA, have QA sign off on it, and then automatically deploy it to production. I think we’re seeing this more and more where we definitely see customers demand it from us, and that’s why people love Jenkins, because it’s powerful enough to allow them to do these kinds of workflows. I don’t know, it’s for me, it’s probably more of a natural evolution of things to happen. We started out like 10 years ago with these very simple tools that allowed us to do one thing, and now, you know, we want to automate more. We want to ship this to production as quickly as possible, so, you know, trying to model these workflows is a natural thing to happen, I think. And even automating them is a good thing because, you know, the more friction you can remove from shipping code to production, the better it is. If you don’t have a QA department, it’s probably— well, I’m not gonna say it’s better. I don’t know if it’s better, but you have one step less for code to go to production. And, if you see a break in production, you ship a fix. You fix it in master, you ship the change to production. And, I don’t know, this orchestration is just the natural flow of people wanting to automate more of this entire process of getting things into production, for me. That would be my interpretation of that. And that’s— I would consider that a good thing.

Matty: [00:24:22] I think so. I think it’s a matter of the vernacular catching up with what we used to call things, ’cause it’s like— I run into that with clients where I talk about, we’re going to have a CI tool, and they’re like, what? It’s doing— but, it’s more than continuous integration, like, this deal, because we don’t have a better name for what Jenkins or Travis do, so we’re gonna call it that. But really it’s this, it’s the software ALM orchestration tool, right?

Trevor: [00:24:47] Right. We have our whole workflow to get from, you know, development to QA to UAT to production. It’s all on our continuous integration machine. You know, we move our databases, we move our code, we move everything. We run our unit tests, we run our UI tests, it’s all orchestrated through our CI.

Matty: [00:25:11] And I think what this still goes back to when you start looking at what these tools were for at first, and even where the adoption was coming in, like you said, even you go back to when there wasn’t the ability to do an automated build, and then it was built around saying, okay, well, now we have— we’re using things like Java or C++ that are compiled languages, so we do have a build that has to happen. We actually have to compile it, and that’s what we’re looking for. And then you think about the things that we do today where I’m sitting there saying, I’m using my CI server to actually integrate Chef cookbooks and test them and have it spin up VMs with Vagrant to see if the cookbook compiles before it gets released. It’s like, you know, I’m running my infrastructure changes through continuous integration. It’s just a bizarre idea. I mean, it’s an amazing idea, but you know what, 3 years ago, 4 years ago, you’d be like, you’re doing what with the what now?

Joe: [00:26:07] So you’re turning up that dial on the responsibility meter because, you know, in a bigger organization there might be a committee somewhere that says, oh, we don’t trust that, and yeah, whatever case it is.

Matty: [00:26:20] But it’s—

Joe: [00:26:20] if you automate it and you prove it, and you trust each other to write the scripts that’s going to automate it, then you’re going to be able to focus on spending time on the other problems that you need to solve, which there always seems to be plenty of in the software industry.

Matty: [00:26:36] That’s how we’ve been able to sell this kind of stuff— not to go off on a tangent, but again, like the config management stuff— to organizations that are like that, where, you know, the dev side of the house totally digs it because they’re saying, great, I want the thing where I can make my change and I don’t have to put in a change control ticket and wait for a sysadmin to go do X, Y, and Z. I want to be able to just go edit the Chef cookbook. And your traditional tech ops is command and control, right? And I think the best example I saw, and this goes back to being able to trust, and I think it goes back to we don’t need more QA, you know, we need more robots, was giving a presentation to a client talking about configuration management and how it would work with infrastructure as code. And there was some, you know, muckety-muck from ops who said nothing through the whole thing and was just sort of sitting there and watching, and then said, now wait a minute, you’re telling me that I can have the developers do the work, but I still get to push the button that says it’s okay because I know it’s okay? And I said, yeah. He said, okay, sold, and he walked out of the room. And we were done. And that was it. It was just saying, that’s great, because what happens there is, like you said, you have that trust, right? The ops folks get to say, these are the rules. These are my tests. So it’s like the same thing about your end users, right? You give me tests and that’s my requirement. And when this passes, then I know it’s good. But that only works when you have this nice automation. We don’t have people testing it because people make mistakes. And that’s what I love about using these tools for the workflow is that we can say, let the robots do what robots are good at, you know.

Mathias: [00:28:18] I think it’s— one interesting word that was used earlier is confidence, you know. And this is what it’s all about to me as well. Like, shipping something, whether it’s an ops change, an infrastructure change, or just a code change, you want to have as much confidence, or you want to give your team as much confidence of shipping it to production as quickly as possible. And I think that’s a great example. Trust fits in with that as well, but I just lost my point. I guess I just wanted to bring up that confidence is actually a perfect example, no matter where you use continuous integration, and in the end, you know, this entire automation, it kind of fosters that you continue— you know, it sounds at first like you automate it once, and then you’ll never have to deal with it again, but it kind of fosters that everyone actively cares about that the build is always green, that the build’s always fast. You know, and that these things work, that you can continue to trust your CI system in whatever it does, to give everyone confidence to ship things to production.

Matty: [00:29:15] So when we talk about this, so talking about testing within the way that you guys like to do your CI workflow, how do you like to build your testing framework? Where do you test what?

Joe: [00:29:26] Sure, I can tackle that. I’m a proud member of the Rails community, for better or worse, but the main thing that I really love about the Rails community is that if you put something out that doesn’t have, like, a how-to-test-this in the README on the front page, it’s not going to be a widely used gem. The testing community in Rails, or, you know, in Ruby in general, is so great. The tools make it so easy to test, like, the corner cases that when you’re moving to a different technology and you want things like transactional fixtures, or you need to sort of stub controller params for web stuff, it’s just, like, so nice that it’s already right there. So, you know, we do testing first. We use it to drive the design as God intended, not in its diluted form. So we’re very lucky that— and also the way you run your tests is very simple. Its commands are, like, kind of built into Rake. And then the hardest part about testing in Rails is kind of figuring out the isolation level that you want in your tests in terms of do I need to, you know, use, like, Capybara for this because I want to test the JavaScript that’s going on there, or do I want to actually hit the database on this test? Can I avoid hitting the database on this test? So figuring out the levels of testing— figuring out how you get the test pipeline going is already sort of determined for you. But then you get to think more about how you actually want to structure your tests, spend time refactoring your tests, because it is a first-class citizen. If you’re using— I mean, I think all the frameworks kind of support this now, but, you know, the more traditional BDD where you have sort of a specification built in— I’m not gonna endorse Cucumber, but I’m going to say going so far as to writing your code, writing your tests in a way where I can read the blocks of code, not even look at the actual test itself, but just sort of read the structure and know what the intention of the code that I wrote was trying to do.

Mathias: [00:31:34] That’s good.

Joe: [00:31:35] I’ve worked on projects in other languages recently, Clojure specifically, where it’s kind of not necessarily— the ecosystem around frameworks is not fully baked, so there’s a lot more kind of do-it-yourself sort of stuff. Like, for instance, getting a test instance of your database that just gets created, right, you know, just the initial step of your test run, when the empty test— and then have the transactions automatically roll back to the point where it’s magical, and you didn’t even know that was happening. There’s a little more roll-it-your-own, you know, it’s kind of the mantra of Clojure with it. Striving towards simplicity and not wanting to kind of conflate the responsibilities of different things, but it’s kind of leaving the padded, cozy, warm fireplace, bear-rug testing environment of Rails has been sort of— it’s been overhead on the project, for good or for worse, I’m not sure.

Matty: [00:32:42] But it’s—

Joe: [00:32:43] it really starts, like, from step 1. I mean, from rails new, you automatically already have something that will— it doesn’t need to build, but it automatically will have something that will run the tests, and it actually emits a couple of fake tests for you. So it’s really not hard to get up and running with tests. And then segueing that into how we do continuous integration, as I mentioned, I start a number of new projects every year. If it’s an open source thing, we really love using Travis CI for that. We have a couple of projects with Travis builds on our public repository at github.com/devmind, and I love seeing that Travis CI flag when I’m going to go use a gem and know that the tests are passing so I can know that I have a little bit more confidence in this gem. We’ve actually, on a lot of our clients, used CodeShip. CodeShip, we’ve just had a good experience with it. We started off when they were pretty new, and Moritz Plassing, if he happens to listen to this, I want to thank him. He has been a very, very awesome— I mean, like, customer support down to the wire, where you just kind of raise a question and he responds to you automatically. He even followed up with an email a week later because he hadn’t heard from me to make sure one of my problems was fixed. So we got really good customer service from them. We had a lot of good support with doing things like firing up, you know, a Capybara server, which has a whole bunch of weird dependencies like PhantomJS and a lot of binary stuff that kind of needs to be in place, and they have also got the deployment aspect of it figured out pretty well. So for the customers that we build projects for, I don’t think it’s free, but I think the cost is pretty cheap. We use CodeShip on a lot of our projects when we want to kind of have some sort of hosted CI solution.

Matty: [00:34:39] So I have a question, and Trevor alluded to this, and this was sort of a thought, this idea of preflighting. I kind of dig this, and this is the idea— and I’ll tell you how I understand it, and then if I’m correct, then you can tell me your opinions, and if I’m wrong, you can tell me why I’m wrong and what it would be. But the idea where I don’t actually, as a developer, get to commit directly to trunk, I commit to some preflight, either branch or repo, and that’s where my CI tool builds against, and then if it passes, that gets committed in. So that’s how I understand preflighting, and I’m curious as to your opinions on that as a practice— pros, cons, and differences.

Joe: [00:35:24] I think it kind of comes back to that whole responsibility and environment of trust. So, I mean, locally on my machine, I run the tests as part of a Git hook, right? So I have a pre-commit hook where before I can actually commit to my local branch, it’s going to execute at least the bulk of the tests. Some of the maybe integration— by that, I mean it’s talking to external systems or something like that— that take too long, or maybe they’re, like, some Vue-based Selenium tests that are running on multiple browsers that would impede my productivity, we defer those to the CI build, but I test everything. I run all of my tests. They talk to the database. They talk to Elasticsearch. They talk to Redis. They talk to whatever they need to talk to, so I have some idea of confidence before I can even issue a commit that I would then push. I think when you’re introducing that preflighting gate it’s sort of a smell, just like any other process, where if we need to gate this, it means that there’s some other problem, and this is the stopgap solution. But I think that there’s probably a deeper problem if you can’t trust your developers to commit code to master that you should be addressing.

Mathias: [00:36:42] Yes, I agree with this. That’s definitely what it boils down to. It’s a barrier, and the question is, why do you put it up? Why do you put it up, and what can you do to avoid putting it up?

Joe: [00:36:53] It’s like—

Trevor: [00:36:55] trust versus control.

Mathias: [00:36:56] Yes. I ask a lot of questions on these kinds of things because, like, in teams, people tend to stop asking questions at some point. They just take processes for granted, and I don’t know, it’s like, what is keeping you from committing, from letting your developers commit to master? I mean, at Google, tens of thousands of developers commit to a single repository, to one branch, every day. Why can’t your company do it? That’s the question I would ask, you know?

Joe: [00:37:25] Or why— they have a GitFlow poster on the wall.

Mathias: [00:37:28] Yes, they have the GitFlow poster.

Joe: [00:37:30] Yeah, so everybody can stand around a poster and figure out how they’re supposed to develop software.

Mathias: [00:37:34] There’s a GitHub equivalent of this, of, you know, people having their own private forks of upstream repos and not being allowed to push to the main repository. Same thing. It doesn’t show a lot of trust and confidence in the people working on the product that you want to sell to your customers, and that’s what kind of worries me about preflight checks as well.

Joe: [00:37:55] Yeah, and again, you know, special-snowflake idiom, like, for an open source project, that’s a great model because I don’t know who you are and I don’t want you committing directly to it, but the fact that you can create a branch or a fork and issue me a pull request that I can review and then pull into my— that’s a very strong tool that you can leverage, but I think it’s when you’re all on the same team and sitting in the same room, I think it’s really just an inconvenience.

Mathias: [00:38:22] That’s exactly the point, you’re all on the same team.

Matty: [00:38:25] What do you think are— we’re coming close to our top, so I got a couple more things I’d like to hear. We’ve talked a lot about why it’s valuable, good ways to start. What are some mistakes, do you think? What are stumbling blocks that people have when it comes to CI? I guess maybe some horror stories without naming names, or just hypotheticals.

Joe: [00:38:56] So things that I’ve observed are not having some sort of present indicator in the room. Like, just grab, like, that 15-inch monitor that is on somebody’s desk or in a closet somewhere, and, like, plug it in and put it somewhere in the room so that everybody can see the status of the build. Because, like, you don’t want to look at the status of the build. You want the status of the build looking at you. If it’s not in the room with you, then it’s so easy to ignore, and you’ll go in, and you’ll see the build’s been broken for, like, a week, guys. What’s going on? And, you know, responsibility, responsibility, responsibility, but, like, if your team is just kind of like, eh, you know, then they’ll just ignore those emails. I mean, I’ve seen people put filters on continuous integration emails because they’re so damn annoying. A stupid build is broken again, right?

Trevor: [00:39:53] Oh, that happens all the time.

Joe: [00:39:55] Yeah.

Mathias: [00:39:56] I think this is a great example for, you know, against preflight checks because if you’re worried about people breaking the master build, you should be worried about people fixing it as quickly as possible if it is broken because it will break, and if you make it as visible as possible, you know, there’s nothing really to worry about. Then you know the team is aware of its responsibility to fix the build as quickly as they possibly can.

Joe: [00:40:19] Right. Yeah, and people gamify this all the time. Like, I think there was actually a gamification plugin for Hudson at one point that kind of ran metrics on who introduced uncovered code and stuff like that, and who’s broken the most builds and everything. I’ve seen people, you know, like, there’s the traditional build gnome where if you’re the last person to break the build, you get it, and that’s kind of nice, but, you know, that creates the sort of unfortunate— the way that gamification always does, where it kind of creates this environment of, you know, like, if you broke the build unintentionally and for whatever reason there is, you know, people just have different sensitivity levels to that type of thing. So you don’t want it to be sort of like a punishment for breaking the build. You want it to just kind of be something that’s going to foster communication and trust within the team. So as a team, we can figure out who broke the build. Doesn’t really matter. Is the build fixed matters, right? So on larger teams, I’ve seen where there’s kind of like a build master for the day where if the build breaks, it’s their responsibility to get it up and running. So it doesn’t even matter if it’s their code commit. That’s a great way to figure out what the hell some other guy is doing on your team, because you have to look at their code and see what kind of a bug they introduced. Or, you know, maybe it was something else. I mean, there’s a lot of things that go wrong in the DevOps world. It’s unfortunately not as nice as talking with all of the pieces of memory that you’re in complete control of. There’s boxes that go down, there’s processes that get hung, there’s so many things that make DevOps a more non-deterministic world of programming than you take for granted. And assigning that build master kind of alleviates the rest of the team, at least for that day or week or however you rotate it, from having to worry about taking care of those incidentals while not dropping the ball on responsibility of making sure that build server is purring.

Matty: [00:42:10] Great. Well, thank you. I think we’re getting to the— now we will move on to our next section, which is what we call the checkout. This is the part when we share with you, our listeners, something that we recommend. It can be technology, it can be a book, it can be a beer or food, anything that we think is cool or interesting and that you should check out. I will go ahead and go first. So I have 2 checkouts this week. My first one is another podcast, but it’s one of those podcasts that eventually kind of ends. It’s called Windows Azure Friday, and it’s the Windows Azure team putting them out. Each episode is every Friday, and it’s more of a how-to type podcast that’ll deep dive on a specific topic around Windows Azure. So we’ll put a link to that in the show notes. The other one that I’ve been meaning to recommend for a while is if you are of the iPhone bent, there is a calendar app that I really like called Mynd, M-Y-N-D, and I use that instead of the built-in calendar. It works with Google Apps, it works with my Gmail, works with my Exchange calendar. And one of the things I really like about it, besides the fact that it doesn’t just look like a regular calendar app, is it’s very cognizant of geolocation. So it tells me when I need— if it knows if I have an address for an appointment, it will tell me when to leave. You know, so it’ll say like, okay, you’re supposed to be at this place at this time. It’s going to take you 15 minutes to drive there. So then I’ll get a notification 15 minutes before that says you should leave. And if it notices I have not left, it then tells me I’m running late. Some people may like this. Some people may really hate it. I really like it. It’s called Mynd, M-Y-N-D. It’s in the App Store. I’ll put a link in the show notes. I don’t remember how much it cost, but it was a nominal amount. So, Trevor, cool.

Trevor: [00:44:09] Um, I just want to apologize, I’m so quiet. My blood sugar crashed during the recording here. My checkouts this week— actually, I just read this this morning. Somebody who— this guy who controlled the @N Twitter account got through some social engineering kind of got extorted out of his Twitter handle, and it’s an interesting read. It was on The Next Web. I’ll put the link in the show notes. And my second checkout is a book. It’s called Drink More Whiskey: Everything You Need to Know About Your New Favorite Drink. That’ll be arriving today, and I’m looking forward to it. Joe, you want to go next and tell us what— something that piqued your interest recently?

Joe: [00:44:54] Sure, I don’t mean to be self-serving, but I’m really excited about the fact that we’re going to be hosting a training session at DevMind next month, February 24th. It’s on Agile product design. The instructor is going to be Joel Tosi, who’s a very, very interesting person in the agile space, and I say agile with a lowercase a. The material is coming from a company called Dev Jam, which operates out of Minneapolis. They do a whole bunch of different big companies all across the country. They’re run by David Hussman, who’s known as the Agile Dude. He’s a really, really interesting guy. If you have a chance to check out some of his talks online, or if you ever see that he’s speaking in person, I really recommend that you go and listen to him talk. He’s really interesting. So I’m kind of excited to announce that we’re going to be doing a training course in something that’s very important to me and something that I think that we spend so much time sharpening our technical chops gets really overlooked by the developer community.

Trevor: [00:46:01] Great.

Matty: [00:46:01] We’ll definitely put a link to that in the notes too, so if you’re in the Chicagoland area, you know, check it out. So, Mathias, you want to bring us around?

Mathias: [00:46:10] Yep. So I’ve been thinking about a book lately that I read a while ago, and it’s called Drive: The Surprising Truth About What Motivates Us by Daniel Pink. It fits in quite well with the whole culture thing that we’ve been talking about and, you know, trust, responsibility in a team. I think it’s a lot behind the GitHub way of doing things as well. That’s where I got the recommendation from, from Tom Pressmuller from GitHub, and it’s a really great book. And the other recommendation I have is I’ve been drinking an excellent coffee that I was sent by a friend in London. I’m a bit of a coffee geek, so I was very excited about this one. It’s a Yirgacheffe from Ethiopia, which was roasted by Caravan in London, and it’s really amazing. If you can get your hands on a bag of that, it’s a pretty awesome coffee.

Matty: [00:47:00] Awesome, thank you. Well, thanks everyone for joining us. I’m Matt, I’m @MattStratton on Twitter, and I’m Trevor, @TrevorGHess on Twitter.

Trevor: [00:47:11] We’re Arrested DevOps, so @ArrestedDevOps on Twitter, and remember there’s always DevOps in the banana stand.