software deployment

Matty: [00:00:07] Welcome to Arrested DevOps, episode 13, Software Deployment. I’m your co-host, Matt Stratton. I’m @MattStratton on Twitter.

Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.

Matty: Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that believes DevOps makes their clients more awesome. You can check out their 360-degree cloud services at 10thmagnitude.com. And make sure you do because that’s a brand new website as of today, and I think Trevor actually worked on it.

Trevor: So as of Saturday, or actually it’s Friday, Friday night at 8:57 or somewhere close to there. But joining us today for this episode is PagerDuty’s very own Ranjib Dey. We’re going to talk about getting software delivered because it doesn’t count unless it’s in production.

Matty: This episode is also sponsored by PagerDuty. PagerDuty eliminates the noise, chaos, and manual processes across the entire incident lifecycle to decrease resolution time. PagerDuty is trusted by companies like Etsy, Nike, and GitHub. To sign up for a free 30-day trial, visit arresteddevops.com/pagerduty.

Trevor: [00:01:17] This episode is also brought to you by Datadog. A monitoring service that pinpoints performance issues caused by your commits, builds, and configuration runs. Datadog is available for a 14-day trial and will graph, alert, and correlate events and metrics from over 70 common tools such as Docker, Chef, Jenkins, and Git. For more information, check out arresteddevops.com/datadog.

Matty: So let’s get right into it. Ranjib, you want to introduce yourself? Tell our listeners a bit about you?

Ranjib: Hey, my name is Ranjib. I work as a system administrator at PagerDuty. Before joining PagerDuty, I have worked on lots of life science algorithm-related works. I have worked with companies like ThoughtWorks and Google and worked specifically on the area of system administration, cloud, and build and release.

Matty: Awesome. So first of all, thanks for joining us. And we want to talk about, about how do we get software out? How do we ship software? How do we release all these changes and ideas? To get them out in front of our users to get back the feedback that we want to get. So to get started, what actually does software delivery mean to you when I say those words?

Ranjib: [00:02:24] So for us, or for me, software delivery means gathering a particular requirement from end user, from some feedback process, and once you have developed, and whatever involves once a feature is developed till it hits the customers use cases or the customers are using and giving feedback on that. So that particular step to me is software delivery.

Trevor: Cool.

Matty: So now there’s a lot of different philosophies as this goes, and one of the questions is, what do you think is the best way when we talk about delivering software, or at least the way that— the method that you found the most success with?

Ranjib: Yeah, so I think though there are specific use cases, there are specific strategies and organization-specific customization, In general, we kind of tend to agree that we want software to be delivered in a stable, in a well-tested manner. And we also want the change that we are introducing to be as minimum as possible. So there are 3 consistent themes on this, that you want incremental changes, and you want regular or periodic changes, and you want the whole step to be automated or semi-automatic manner in a reliable fashion.

Trevor: [00:03:34] So how do we go about doing that? So we’re, you know, we’re talking about small, you know, I’m assuming small commits, uh, and making sure those commits— so in the strategies that you’re familiar with, are you executing on continuous delivery or— okay, cool, excellent.

Matty: Hey, what’s, what’s continuous delivery?

Ranjib: So I think continuous delivery is the ability or the attitude towards delivering your software in a well-tested and incremental fashion. In most cases, it just generally tends to be an, you know, extension of your CI environment where you have the CI and it doesn’t matter how you develop the code. Sometimes you provide, use it feature branches, sometimes you have kind of stable and different environment-specific branches. But the crucial part is when you merge them, before the merging it goes via a PR or a review process and a testing process. In cases it can be automatic, in cases it can be manual, or it can be semi-automatic. It doesn’t matter how. Like if you’re following things like XP, you push the code review right up front during the development directly, right? While in other, say, mostly free and open source software on top of GitHub that we develop, we go to a later post-commit review process in terms of PR. So nonetheless, the thing is that once you have write those features, those has to be smaller and independently tested. So every feature that you have has an independent test. So now you take this whole approach and take it another level, you say that now I should— I want the ability to introduce only this small change to my production, and I want to do it in a very, very safe manner. And probably I want to do it in a lot more sophisticated way. You may decide to release a particular feature only to a subset of your user. So you may also— you want it like, say for example, based on geography, based on age group, or some, some other parameters, or maybe some expected— some interested customers who are only want to see those features but not the rest. So those kind of abilities you add on, and that very process is continuous delivery. In due process, you also learn the risk-taking abilities. The entire organization also goes through some kind of cultural changes because it was not common in the beginning.

Trevor: [00:05:39] So along those lines of continuous delivery, so the steps you’ve kind of outlined there are not necessarily steps, but some of the pieces are small commits, lots of testing and getting it right out in front of your users and getting their feedback and being able to respond on the fly to what their wants and desires are. So let’s go through each of those steps. I think most people kind of understand the concept of a small commit, but what is a small commit to you?

Ranjib: To me, a small commit is anything, anything that is a change that has been automated and that we need to persist and that That is something the user, you know, expects. It can be a unique feature. If you cannot think of a feature and we cannot associate that with our commits, then it’s not probably a full commit. It’s not a full work. You want it to be independently verifiable. It kind of goes back to the theory of, you know, stories, agile development, right? And you can have like smaller stories or you can have large stories which later we call them epic, and then again we break them. So to me, it’s a logical extension of that. But you are not only talking, talking about development. Your commit can also represent your infrastructure changes. So if today you are doing, say, canary deployment, but you want to go from that to, say, some kind of dark launching, and that particular logic is implemented in some form of code, that is a feature, right? Even if sometimes it’s not user-centric, but for your internal programs or your infrastructure, that is a feature. And you practice it in the same way as you develop any other software. It also means that particular feature has to be tested, and it has its own independent tests. And every time we break the feature or anything else breaks that feature, it is expected that that particular test will be broken. We would like to use that test to drive the requirement or to ensure that by further changes, those particular features are not broken. Which we call also sometimes regression. If it is known error and you have added those particular, uh, tests, we call them regression. You don’t want the regressions to come up as well. So features are like this, and a commit basically embodies a feature. You can also have breakdown, you know, one feature into multiple commits for yourself, but when it goes for the merging, generally PR or a feature branch means one particular feature. Cool.

Trevor: [00:07:59] That’s what I like to hear, actually. That fits very well with how I like to do things. Again, you know, so that you’re bringing it into the next step too, where we’re talking about testing. So to you, what kind of tests should we have on our software before we’re putting it into production?

Ranjib: So I always think that more test is better, but it is not— it is also not very rational, specifically if you’re starting from greenfield. It is also important to understand that writing tests is easier in the earlier stage than later, because test kind of modern test or unit test kind of, or even functional test using Cucumber, they reads like documentation. So you can actually replace documentation, you can get rid of documentation, you can have version working documentation but in form of tests. So more tests is better, but you don’t generally get time. So depending upon your business and your exact requirement you, you design the tests. Say if you are starting from greenfield projects, I will definitely say that functional testings are the best ones because there your focus is only that whether a particular feature is working as it is expected or not. You don’t really look at the code how it is implemented. You don’t really use those tests to drive the design of the code. You are using mostly that my functionality is intact because that’s your topmost priority. Once you have those things, then you kind of realize that in due process you might have some tech debt, you might have some design issues, and you can also have unit tests. You don’t have to go through this learning process if you have already done it. So if you have already executed projects which has unit testing as well as functional testing, probably you will already start upfront unit testing in your greenfield projects, and which is just perfectly legitimate and which is really, really good. The benefit of this thing is that now if you hire someone or if a new person joins, right, I’m assuming your business will grow and new people will come. They don’t have to really familiarize themselves with the entire code base. They can look at this code and they can change just by, you know, looking at the test. Or in, in worst case, you can forget how the code works and you can just make the entire code works and you can just make the changes, uh, that is required and you see the test failing. And you do bare minimal work to fix those tests. That pretty much drives the design. But as a result, now when I write some stuff, I am pretty sure that even if I don’t understand the code fully, I have not broken existing stuff. So, these are some. And then lastly, you can have integration tests. Integration tests are kind of, you can, not functional tests. That means they are black box. You don’t know how the code is. You are not aware of it. But they also incorporate all the external services. Think of that if you are writing an MVP, say Model View Controller, MVC framework like Rails, and individual application logic can be tested using unit tests like RSpec or TestUnit. Any user is clicking and particular functionality is getting delivered, that can be tested with Cucumber. But will that feature work if my backend is down? Suppose I have set up one load balancer and 5 background unicorn and 2 databases, you know, for resiliency. And I want to test that fact whether that works or not. Uh, the one way to do is that create a simulated environment and take off one of the unicorn. But how you test that? So to do that kind of testing, you only— not only need a full environment and system automation, plus also the ability to automatically deploy it and then inject those failure and then gather the feedback in forms of test and test harnessing. That is integration testing. But this may not be, you know, absolutely necessary in the beginning. As you grow and as you hit this obstacle, you introduce these things. Yeah, in its simplest form, it’s more about, you know, getting the value that you want. And in its most sophisticated form, you actually use the test to design or derive the simplistic solution.

Matty: [00:11:55] So what we’re saying is that one of the things is we can automate our deployments and we do them a lot more quickly if we’ve got good faith in the product based upon our testing, correct?

Ranjib: Yes.

Matty: Would that be it? Yeah. And random thing, like when you talk about that integration testing, and it’s funny because I’ve heard different people use that phrase different ways, which can be at either end of the cycle where either they’re talking about, oh, this is when we want to test integrating our code, but I like it more the way that you said it, which is more of systems integration. And that’s kind of a— I think it’s in the Continuous Delivery book that the term is used as non-functional tests, which is kind of a rough word Because it’s like we’re testing to make sure things don’t work, but that’s not really it. Just the things that are outside of our normal functional tests. Like you said, like either testing for high availability or when you’re doing things like performance testing, stuff that you’re not necessarily wanting to do every single commit, you know, but, but when things make it to a certain level in the pipeline. We also, if listeners want to go back, our second ever episode was all about testing. And so you can check that out at arrestedevops.com/2. For episode 2, where we talked in depth about testing, and that might be interesting. One thing I want to think about that came to mind when you talked about starting with functional tests, and that’s a— it’s kind of interesting to me because I think the same way. And when I’m writing, when I’m quote unquote writing code, I guess I shouldn’t put that in air quotes, but I’m usually writing like a cookbook, you know, so I’m doing this with Chef and it’s 10 times— it is still code. It’s code. I shouldn’t, I shouldn’t. Minimize it that way.

Trevor: [00:13:31] Don’t knock it now that you’ve tried it.

Matty: That’s correct. So, but if I think about it, I tend to, while the functional tests may not be the first thing that gets tested in my pipeline, where I might have like unit-level stuff sooner, I tend to write those tests, the functional tests first, because they’re the ones I know first. So is that a way to kind of summarize what you were just saying? I mean, not summarize, but am I understanding your point correctly?

Ranjib: Yeah, actually the interpretation of these terms vary. Specifically for integration testing and functional testing, it varies a lot. Even in the unit testing frameworks, people argue, which is fine and debatable, but we know kind of the generic use cases. Like, though you may not write the functional test first, and though you may write the functional test first, you may not run them first. We know for a fact that we’ll always prefer to run the unit test first. If they are available. Reason being, they are the fastest one, right? So in your developer workstation, if you have a really large code base, it is expected— or the goal of unit test is also to execute fast. You know, it does not execute anything except your code. So say, example— for example, if you have a database call, you may want to mock that call because you don’t want to test the database. But data— databases, uh, or for example, if you’re writing on top of Chef and you have some searches, you can stub it out because you don’t want to test Chef server search process. But that is debatable. A lot of people have said it may be good, it may be bad, but nonetheless you get the benefit of executing your code fast and to govern the design. And integration testing also same, it comes later. As you said, a lot of people like in the Continuous Delivery book, Jay talked about non-functional testing. If for us it is, it is different because We are kind of internally SOA architecture and we have all these services come together, give a much, you know, broader or much usable interconnected pieces of software which you feel like a platform, which is kind of federated. That is also an integration testing. I also work a lot on package management, operating systems. Earlier I used to work for— work a lot on Fedora packaging. There, if you think of a standard packaging, Say you are packaging libcurl and you have the code for libcurl which is written in C, or maybe it has a lot of binding in Ruby, Python. Testing those code will be unit testing, while if you build a package and you use those packages to make a curl call, that can be called as functional testing because that time you are not really bothered about how curl is implemented. On the other hand, integration testing will be, say, Bodhi or Koji-like system where all the packages getting integrated and you are saying Ubuntu as a software is working with, or Fedora as a software is working with a particular version of libcurl, particular version of kernel, particular version of glibc, etc. Because at the end of the day, your user is not really want to only use libcurl. They want to use it in conjunction with this whole platform. So depending upon your concern, you might have different interpretation, but overall you want to add this, you want to address the same values or same benefits.

Trevor: [00:16:36] So once we’ve got our tests in place, So we’ve kind of— we’ve got our unit tests to test our code, our functional tests to test our larger scope, and our business-driven tests, our Cucumber tests, to test full functional actual implementations. How and when do we run those tests? So you mentioned, you know, as we’re developing software, we’re gonna be running our unit tests very quickly and repeatedly because it’s the fastest thing that runs. But where in our pipeline of continuous deployment delivery do we put and run our tests?

Ranjib: As you have just mentioned, you start as, as your organization grows, you write tests, and, and then the tests are broken down into multiple stages. And slowly you hit a state where state becomes bottleneck, test becomes a bottleneck. Okay, not only the time taken and the complexity involved, but also the infrastructure that needs to run the tests. So, generally during that time you introduce something like CI or continuous integration server, which is basically an automated service which runs the test for you and gives you the feedback, generally in forms of red, green, blue, whether you have all the tests passed, failed, or, you know, they’re just broken due to some error condition. And you consume this information as a radiator Because you want other people to look at that. And if a build is broken, probably you don’t want to merge something on top of it. That’s the reason. So a follow-up or, or a logical step soon after you have the test is that to have CI. And again, if you have not done it, you realize it as you learn and you implement it. But if you have done it beforehand, then you can go full, you know, frontal and you can have a CI server straight away. And it is also pretty easy now that you have a lot of hosted CI services, but none of them solves all of the problems. So depending upon your need and requirement, you can choose any of the CI servers and you can automate whatever, whatever takes a lot of time from your end to the CI service. So first you want to automate and offload to CI service and establish a workflow so that the team is comfortable with it, around it. And then you introduce the CI server as a feedback mechanism for your deployment pipelines.

Matty: [00:18:54] I have a question that’s specific around PagerDuty because you guys are very service-oriented, as you said. So you’ve got lots of products that all intersect with each other. So when we’re talking about building a continuous delivery pipeline, and I know that when I initially started learning about CD, I kind of really thought about this very much like there was literally one pipeline and all the products were in it, blah, blah, blah. And then I’ve kind of seen that that’s not maybe necessarily necessarily clear to do it that way. In your setup, do you kind of have separate pipelines for each product and then they come together at some point, or is there really just like one set of, uh, deployment pipeline for all of your products?

Ranjib: Yeah, so, uh, our internal CI infrastructure is kind of, again, represent or shows our internal evolution. Till I joined, we didn’t have actually a CI. We had a lot of tests, and, and I, I joined when we had like 14 people only. So we had tests, we had those things, but we didn’t have any infrastructure to run those. So our first thing was to set up a Jenkins server and do a feature branching and an individual pipeline for each of those. Because I have used Go and other tools, CI service, I will not say those as pipeline because pipeline is a bit complicated than that. You can have things like fan-in, fan-out stages, and all those things are pipeline. So in particular CI context, pipeline means a bit different, but I think what you mean is that per project build capabilities and feedback capabilities. And yes, we have that. Later on, we moved with a kind of a blended solution because we had, we were SOA and there are multiple people working in each, working on a different feature branch. You will have a lot of PR raised and that used to be a bottleneck. So you’ll have a high build queue. So we offloaded that bit on Travis. Travis is doing perfect job. On that. And it also works perfectly with GitHub workflows where you get the feedbacks not only in terms of, you know, your PR becomes red, green, or yellow. You can also get a post-back hook in your HipChat and stuff like that. So that is one thing that we are doing. And also, uh, we still have the internal Jenkins, but it does not give comments or give feedback on individual PR, but it does the reverse. Once you merge, we take our master and we deploy to our downstream environments. That is more like kind of we are doing integration testing with that, not really functional testing. But that is more due to the fact that Travis really is, is not foolproof in that way. You don’t have pipelines. You cannot say that, uh, that you have 10 dependencies, each— if each one of them breaks or one of them gets updated, you want to trigger downstream processes with certain other kind of thing. You cannot do those kind of— and those I have to call them as pipeline, or in CI services we call them as pipeline.

Matty: [00:21:34] Gotcha. Okay. So that certainly helps for sure, because it gets— to me, it seems like it’s very easy to build. I shouldn’t say it’s very easy, but when you’re talking about a single application that kind of works within itself, adopting continuous delivery kind of makes a lot of sense. And then you start to look and say, but like you said, when you have these integration points come in, and that’s usually when I’ve talked to customers, when the wheels start to come off, because it’s not as intuitive, I think, at that point. That you’re still philosophically, you have kind of a virtual pipeline. You say everything must go through this way, but it doesn’t necessarily mean that you’ve got a physical pipeline that everything follows the same way.

Ranjib: Yeah.

Matty: So when you talked about, we were talking about deployments, you brought up, you mentioned something in passing that I’d like to talk about a little bit more because I think it’s pretty interesting. And that’s the idea of canary deployments. So do you want to talk a little bit about what that means and how that can be useful in the real world?

Ranjib: So I think this is useful when you want— when you are doing smaller, again, a smaller feature-based development and you want to try it out to get feedback before you roll it out to across a large set of audience or end users. There are multiple ways to do that. So one is that one feature of this, or one requirement of this kind of process, is that you want to expose the feature to only subset of user. Another will be that if you have a large cluster of services, how you can roll it out that it does not cause any downtimes. So to do that, there are multiple processes which actually addresses both of these things. Canary deployment, blue-green deployment, dark launching, all of them pretty much on the similar line, but they kind of give feedback to each other. In case of canary deployment, you have a router kind of entity in front. It can be a hardware router, it can be a software router, it can be something that logically represents a router. And what you say that only some subset of— if say deploy— I develop a feature which is accessible in a particular URL, say xyz, and in my router I am saying that if you hit— say I am running nginx as a router— that if you hit this xyz, then it goes to a canary server where it— and it fetches that particular response from those set of subset of server. As a result, when you developing this, you have to release only on those. So any kind of code changes are now sandboxed to only those subset of backend services. But this was the earlier requirement, but we are addressing this problem by changing the router configurations and, and, and then designing the whole deployment and release process around that. That is canary deployment. You can do multiple other ways. Those will be on the lines of, you know, dark ones in your Bluebeam deployments.

Matty: [00:24:17] Gotcha. And then similarly, so this is, this is a thing I’d like your opinion on, is a lot of times when we talk about this type of work, one of the first things that comes up is, oh, how do we roll back? Roll back, right? Everybody gets freaked out about wanting to do rollbacks. And Mark Burgess wrote something not too long ago that in his mind was the mathematical defeat of why rollbacks are a thing. So I’m curious, not to say, hey, do you agree or disagree with Mark, but What do you find in the real world when it comes to rollbacks? Is— how often is that relevant, the ability to roll back a change? And how often does it actually help you to be able to do that? Or are you able to even ever do that appropriately?

Ranjib: I always felt that the ability to have to do rollback is always awesome nonetheless, but it’s— sometimes it is just difficult, or sometimes it is impossible to do it. But if you have, first of all, it’s really your business requirement whether you want to do it or not. And it is also driven by culture. Some people would like to— like, as an organization, we always tend to roll forward. And it is also supported by the fact that we have heavy automation and testing in place. So even if we have a small bug, we have the confidence that we have to ship only that kind of fix instead of rolling back, which might involve a lot more things. Sometimes rollbacks are just not possible. Suppose you have database migrations. But again, remember there are ways to design your database migrations that they are— that rollback is possible. Didn’t— I remember when we were adopting continuous delivery for different clients and enabling them, this was a very common concern for them. Like, if my release involved database migration, how do I roll back? So after doing some work on those things, what we have figured out that if you can split out your entire migration into 2 chunks, or you can have your release broken down into 2 chunks. So in one, first one, you probably stop using a column name. Say your database migration involved dropping a column. So in one release, you stop using the column name. In the second release, you actually do the migration. As a result, you can always go back to the rollback state. But this is like 2011, and now we know a lot more things can be done. Now we have persistent store, or the prototypes of persistent store. Which is time series aware. So you can tell directly, think of you have a Cassandra or stuff like that, that go back to 30 minutes. If you can do that, then I think a lot of current rollback issues will be automatically solved. Now these are not really production-ready system, but these persistent data stores are being developed. So I think rollback will be absolutely possible. And same with deployments and containerized applications. If you have containerized application, you can always keep a couple of old versions, you know, on the same host and you can switch over very fast. And in that case, rollback is still possible. Not necessarily it is expected or it is the intended thing to do, but what I think is that even if you don’t prefer rollback, if you have an outage, you might want to roll back for a small time till you get— it’s kind of you’re buying in time till you push in the hotfix. But as a generic purpose or as a generic rule, I think you should always optimize for rolling forward. You should never optimize for rolling back.

Matty: [00:27:30] I like that. Always optimize for rolling forward and not optimize for rolling back. So I just repeated that so I remember it.

Trevor: Always a good plan. I want to get back to CI because I think CIs are awesome. So how in continuous delivery do we get our builds out? Because there’s, you know, I’ve seen strategies where the continuous integration server automatically deploys to production, which I get a little nervous about still. I’ve seen situations where it’s a button on the CI, and then there’s, you know, the least interesting where you do it yourself. So what do you think is the best way to do it, and how do you do it?

Ranjib: I don’t think there’s a There’s a best way.

Matty: Yeah, best is a tough word. What’s your favorite?

Ranjib: Yeah, my favorite has been recently we have been deploying straight from our HipChat. We have a bot which does it. You can talk to it, we can ask, and it can do stuff for us. It’s, it’s working really neat for us, and, and I kind of like it. It is— it reminds me of the IRC days where we used to have IRC bots that used to do deploys for us. Not only deploys, to do various, uh, automated stuff, things like meeting, taking meeting logs, doing meeting agendas. But now we are having those things with HipChat, and I like it. But it is important to understand that the integration is really thin. It is the backend automated process that is doing the deployment, right? That involves orchestration, that, that involves all those complex resiliency features that embodies all those tools, are more important. And I think it is good HipChat, but not necessarily this is the best one. I have seen a lot of people seek for that button, and you have to sometimes give the button, right? So, and primarily CI is the thing that they deal with. So at that case, I’d prefer to put it in the CI server itself. In that case, again, it is simple, right? You just basically saying that there will be a Jenkins button which will trigger my same automation script. But where you’ll feel comfortable, it’s really a question for the particular organization, how much they trust, who they trust. And whom they want to trigger this thing.

Matty: [00:29:43] So you use your HipChat bot for more than just posting funny GIFs into your chat room?

Ranjib: Absolutely, yeah, because when you, when you do that, yeah, because we have the GitHub push notifications or the GitHub webhooks also enabled, so you can see merges being happened, you can see PRs getting reviewed, and you can deploy them right there. And if something goes wrong, you can point the person direct that I’m doing this. Or you can say that, hey, I want to log this kind of environments and I want to do all those things. Because that is the primary mode of communication and we have learned historically all the tools that you are using, you may start with all the awesome and the best tools, but as your organization will evolve, all of them will reflect your company’s culture and trust at the end. You can have absolutely awesome tool acting really stubborn and non-agile. You cannot do anything very quickly. You cannot mold them if you do not have the culture to back those things.

Matty: I think that’s a, that’s a really good point, is that you have to get— the people have to be comfortable with using those tools and understanding what it— that it’s, it’s okay, you know, or figuring out what are the things that freak you out. Like Trevor said, you know, it came in and he said, oh, you can have a thing where it, you know, at point it pushes it to production. He’s like, oh, I’m a little nervous about that. And I guess even though I’m an engineer, like I put on the psychiatrist hat, right? And I would talk to the customer and say, so, so why do you feel that way? What is it that’s making you nervous? Because chances are we could do something about that and not, not necessarily do something about making you feel differently, but figure out why are you nervous about it? And then, oh, I’m nervous because what if something goes wrong? Well, we wouldn’t do this if we didn’t have this particular safeguard in place or we didn’t have You know, it has to pass all these tests before that happens. It doesn’t just randomly happen, you know, and then because what you don’t want to do is you don’t want to steamroll over people’s concerns, right? If someone has a concern, you don’t want to say, well, you’re wrong, or hey, catch up with the times, buddy. You’re an old fuddy-duddy, right? You want to say, wait a minute, you’re worried about this for a reason, and I should acknowledge that and then figure out if either A, I missed something, maybe you have a reason that never occurred to me, or B, it just was natural to me to know that this was taken care of, and I just have to make sure that I communicate to you that why this is— here’s some more information that maybe will make you feel better about it.

Ranjib: [00:32:03] Exactly.

Trevor: Everyone’s comfortable.

Ranjib: Yeah, I think Trevor’s concerns are super legit, you know. From my experience, at least in, in last 10 years in this kind of activity— deployment, release, development— I have seen at least 80% to 90% outage due to deployment, as simple as it is. And I, I can understand, and I thoroughly buy in that deployments are major reason for outage. And, and actually, that, that is one of the reasons I would like to automate it first. And we probably not— cannot avoid it. Even if we automate, the failure is not avoided. It is just faster. It is just less manual. Earlier you could fail in 10 ways, uh, where— so you have the ability to stop in, in 10 intermediate steps. But you can now, you can fail with one click of button and there’s nothing you can do in between. But that also means that I am now tied to person, I am tied to documents to do those things, and I am also very prone to human error, which may not be an issue when if you have a 3-4 team, but which can be an issue if you have 30 apps developed by 300 people, 10 people in each team. That can be an issue. So to me, the concern— I think we can reverse the concern. We can say that Whatever is your concern, let’s address them and let’s test this whole thing in a sandbox environment. We know for sure that we cannot avoid failure. So let’s not try to avoid them. Let’s try to come up with a scheme where we can make failure extremely cheap, extremely affordable. And we’ll give you environments or places where you can develop things, fail, and then again reiterate over them and fix them and learn the tool or give those intelligence inside the tool. And make them make the tool better and then use it for production deployment. So absolutely, you cannot have automated deployment if you don’t have multiple environments where those things can be tested, those workflows, those known errors. There will be certain parts that cannot be deployed or that cannot be automated straight away, or that cannot be automated from CI or from HipChat. I, and I can give, give you concrete examples of those things. Like even for database migration, if you have really large-scale databases you have different kinds of migrations, very different. The tools are different. You don’t run Rails migrations, uh, you run completely different kinds of tools. You do in-memory as a single transaction migrations because you are running a high-volume traffic. So those things cannot be automated with standard CI or standard, or at least they may be done from CI, but it will involve a lot of work, or we are just— the CI systems are not mature, no way, compared to the other tools which you could use. And in those contexts Maybe using those tools for deployment makes more sense, more safer.

Trevor: [00:34:34] Right. There’s tons of different ways you can go about assembling a system, automating the assembly of a system, putting everything together. But I think the one thing you said, and it’s been said on Arrested DevOps before, and it’s going to be said 100 more times or more, is do not be afraid of failure. That is the important thing, is fail quickly and fail cheaply, like you said, and Fail fast.

Matty: Yep.

Trevor: And it’s going to come up again and again because yes, let’s—

Matty: yes, that’s the thing, right? Like that’s the, I think that people have to understand it’s okay to stumble. It’s okay to fail. We want, that’s how we learn, but we want to make it repository. Yeah. Well, we would, if there, if there’s small, small failures along the way and that’s dude, that’s where the fast comes in, right? Too, is just chunk it up a little. So we talked about some of the patterns for software deployment, you know, some of the things around continuous delivery and this automation. What are some of the anti-patterns that you see people doing that are kind of a, I guess you could say, a recipe for failure or a recipe for a bad time?

Ranjib: [00:35:35] Yeah, I think one of the biggest anti-patterns that I have seen is people becoming hardliners. They’re stuck with some nuance instead of meanings, you know. Historically, we have words in tech, we call them buzzwords. They represent some movement, some cultural shift, and then you have some people who really understand because they have gone through those pains and then they respect those things. But then overall, they then write the blogs and all those stuff and lay it down how those things are used. Later on, you see another second or third wave of people who does not have those experience, that’s why they came, but they understand what it is and they also understand this is a hip thing. So, and they’re stuck with the nuance, like what is integration testing? The point is that you want to integrate and you want to test the whole thing. But there’s no point. It’s a bike shedding discussion, you know, to that how I can do integration testing. Or similarly, that how— what should be the frequency of doing continuous delivery, right? Should I deploy 20 times a day? It does not make sense for a JVM application to do 20 restarts. If you— so if you do not have a hot deployment pipeline, doing deployment that involves restart on a JVM application 20 times a day just does not make sense, right? So I see Increasingly, people get super accustomed with the nuance of those things, not really the meaning. So we have to stay focused on what the value— and those words are given at particular time, they make sense at that particular time. There are perfect— I mean, it is absolutely possible that in your case you have to twist it in a very different way. That is one biggest anti-pattern I found. The second anti-pattern I found is that accumulating tech debt. This is particularly true for me because I predominantly work now on operations. We always work on tactical stuff. If we accumulate tech debt and you think that, hey, I’ll use the test later because this is not— this is my MVP, you know, I don’t want to do premature optimization. That’s not right. You know, working fine is not your feature. I mean, it’s not a value-added thing. It is a feature. I should be able to monitor your application. That is not a feature. That is a requirement. That is not a value-added thing. If it is running in production and you want someone who has not written the application to rightly monitor it or to, to say that it is working fine or not, we really need some insights. And monitoring is the insights. Monitoring is a uniform watching glass for us in operations, at least. So if you are thinking you are building an application that needs to be delivered continuously and there is no monitoring inside it, so those— that’s not gonna fly. And again, so remember not to confuse with the nuance. It’s not the monitoring, it is In general, the notion of making a piece of software operationally feasible, operationally flexible, that has to be there. And people tend to not, not do that. You want to isolate services in the beginning. Later on, it is way more effort. Same for the code, same for the— and same for the culture also. You don’t want to make large groups making all sharing a single repo, all stepping onto each other’s commits. You want to make smaller groups of people And you want to give a lot of independence on those. Let them choose that tool, let them choose that technology. And of course, you’ll have a peer review process. I mean, that’s— I think the recruitment itself ensures that, right? You don’t want to recruit people who does not really want to do this kind of stuff. So these are some of the anti-patterns, like fascinating over particular words and those, say, testing, missing, thinking that testing is upfront investment. Or thinking that— thinking from an operations perspective, an upfront investment, these are anti-patterns. These are not premature optimization. This is the thing you need.

Matty: [00:39:10] Cool. So before we move into our checkouts, Ranjeev, if you were to give in 1 or 2 sentences, what do you think would be the most important advice to someone when they want to think about how they can improve their software delivery practice? What might that be?

Ranjib: I think that you have to make the decision. Magic bullet. There are certain things. So always keep in mind, continuous integration is a thing. Continuous delivery is a thing. A lot of people done has some work on this. You need to read and you need to adopt it. There is no magic bullet. There is no button. Nobody’s giving you the button that you click and it will solve your problem.

Matty: I would agree. So now we’re going to go into our checkouts where we’re going to talk about something cool, something we would like our listeners to be aware of. So we’ll go to our guest first, Ranjib. Do you have for us to check out?

Ranjib: I would like you to check out this particular book called Universal Principles of Design. So Universal Principles of Design, it’s a book that alphabetically, uh, enlists most common designs, the design paradigms. And basically the context is more of a UI. So you have things like colors, you have things like tones and all those things, but there are a lot of stuff that are common. Like the first one topic talks about 80/20 rule. That probably 80% of the problem can be solved by 20% of the feature of a full-blown solution. So ensuring that 20% works fine is very important, and focusing the first 20% is important. You don’t have to make a full-blown, you know, perfect thing. There are a lot of stuff like that. It talks about symmetry, it talks about semantics, talks about visual understanding. What can be visual? What we can, can visualize? What helps us correlating? Uh, human minds are tuned for pattern. It also talks about how you create patterns, how you ensure. So if there is a common concern in my build pipeline, I would like to model it as a common pattern and I want to make it up the pipeline. So if it fails, everybody should know because that way I can ensure that a larger group of people is aware of it and they can think. There are more eyes looking into those problems. So yeah, universal principles of design.

Trevor: [00:41:15] That sounds awesome. I think I’m going to read that.

Matty: That’s half the reason we have this podcast is just to get tips from other people.

Trevor: Absolutely.

Matty: So I’ll go ahead and give mine. Mine is a project on GitHub called Homesick. And this was brought to my attention today by a friend of the podcast, Fletcher Nichols. I was trying to figure out what the ramifications would be of keeping my home directories on my iMac and my MacBook in complete sync. And I posed that question on Twitter and was concerned about files stepping on each other, et cetera. And then Fletcher pointed out this tool called Homesick, which is available on github.com/technicalpickles/homesick. And it’s a small Ruby gem that basically lets you define how you want to sync your dotfiles, which is what solved my problem. In my case, I primarily wanted to keep some of my config files in sync. So I had the same, for example, tmux experience on all my machines. And homesick seems to be a way to do that. I can’t tell you how well it works for me because I haven’t tried it yet, but I may report back on that in our next episode. And my other checkout, this one’s a little silly. So this is a Ruby gem called Hodor. It was written by John Kaiser at Chef, and it pretty much takes in a string and converts it to Hodor and then It can also take Hodor strings and convert them back to the original string. This makes no sense if you’re not a fan of Game of Thrones. The usefulness of this is potentially debatable, but it’s still kind of funny. So that would be the Hodor Ruby gem, and we’ll put a link to that in the show notes too so that you can include that in your projects for who knows what reason.

Trevor: [00:43:00] You gotta Hodor your projects.

Matty: You do, and you gotta Hodorize them. It’s like Enshepherizing them, if you guys— if anybody remembers that. Trevor, you’re too young for that, but From alt.swedish.chef.bork.bork.bork and use that.

Trevor: Alright, so I guess that leaves me. And I’ve been hacking for the past week and a half, so I’ve been not doing a whole lot of other things. But today I discovered a fun little site which I hope will be fun in the future, but it’s the first week of it, so who knows. It’s called willyouhack.me, and it is exactly what it sounds like. It’s a bunch of little Right now it’s one hacking challenge, which isn’t very hard at all. It’s, in fact, it’s quite simple. I suspect most listeners of this show will be able to solve it in less than 5 minutes. Some maybe less than 1. I hope it gets more challenging in the future, but it’s really interesting and I’ve wanted to tinker with hacking for a while. So it’d be interesting to see how long I can keep up. And secondly, check out fishing. That’s, I’m going to go fishing this weekend with my brother and I’m really excited about it.

Matty: [00:44:01] That’s fishing with a, with an F, not a PH, right?

Trevor: That’s correct. For actual fish fish.

Matty: Not passwords.

Trevor: Largemouth bass, to be specific, is the target. I hope that he’s not listening because it’s supposed to be a surprise.

Matty: That’s the first time that I’ve ever heard you say that you hope someone’s not listening to the podcast.

Trevor: Well, I would be surprised if he was. So that’s my checkouts. Matt?

Ranjib: Yeah.

Matty: So on the next Arrested DevOps, We will be talking about how to fuck up DevOps with DevOps thought leader Pete Cheslock and maybe Nathan Harvey if he decides to join us or not. So you can learn more about that at arresteddevops.com/14. We’ll be doing that at 7 PM Central on Monday, July 7th, but if you don’t remember that date, go to arresteddevops.com/14 and there’ll be all the info about that. And also don’t forget, we have a newsletter. You can subscribe by going to arresteddevops.com/bananastand. It is the best way to know about our upcoming podcast episodes and cool news with DevOps.

Trevor: [00:45:12] And you know, this is the part of the show where Ron Howard steps in and critiques Matt’s narration and really knocks it down a few pegs where it belongs. But other than that, we want to thank our sponsors PagerDuty and Datadog. Thank you guys very much for supporting the show.

Matty: And Ranjib, thank you for joining us. This was a great conversation. I hope it was super helpful to everybody. And everyone, you can check us out on the webbernet at arresteddevops.com or on the Twitterverse @ArrestedDevOps. I’m Matt, @MattStratton.

Trevor: And I’m Trevor, @TrevorGHess.

Matty: We are Arrested DevOps, and remember, there’s always DevOps in the the banana stand.