Doing Releases Right with Scott Hain

Scott: [00:00:00] They’re like, oh, goddammit, fucking people. Why can’t they make their shit work?

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton. We’re going to talk today about releasing software because that’s a thing that people do. But first, A word from our sponsors. This episode is sponsored by CircleCI. Designed for modern software teams, CircleCI’s continuous integration and delivery platform helps developers push code with confidence. Trusted by thousands of companies, from 4-person startups to Fortune 500 businesses, CircleCI helps teams take their software from idea to delivery quickly, safely, and at scale. Visit arresteddevops.com/circleci to learn why high-performing DevOps teams use CircleCI to automate and accelerate their CI/CD pipelines.

[00:01:10] The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener, there is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained, and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app. When we think about releasing software, there’s more to it than just having a clever pipeline. I’d like to welcome to our show today my good friend Scott Hain. So Scott, can you tell us just a little bit about yourself? What do you do? Tell the listeners what they’ve won.

Scott: [00:02:23] Absolutely. Uh, and yeah, thanks for having me on, Matt. This is, uh, exciting for me. Um, I am a former release engineer, uh, former quality— well, current quality engineer, former, uh, engineering services kind of person, uh, former support person, former engineer. Um, I could go on and on and on. I’ve, I’ve I’ve been on multiple sides of the releasing software train and have dealt with a lot of unique circumstances in the past. So right now I am a, like I said, a quality engineer at HashiCorp trying to make things a little bit better for our customers and our support folks and our engineers. That’s pretty much me in a nutshell at this point.

Matty: Fantastic. So we’re not gonna spend a whole lot of time today talking about kind of the basics around continuous delivery. We’ve had Jez Humble on the show a few times talking about these things, and you can check out the show notes if you wanna go back to those old ones, but it might be helpful to take just a minute to level set around when, ’cause there’s definitely people have different thoughts around what we mean when we say continuous delivery. So if we could kind of get our vocabulary in sync, um, what, what, Scott, I’ll let you kind of give yours and then we’ll see if I agree or if I have to adjust.

Scott: [00:03:51] Absolutely. Um, so in general, the CI/CD is what people mostly say. Um, CI being continuous integration and CD being either continuous deployment or continuous delivery. Typically when I see the, the, the acronym CD, I think of continuous delivery, which for me is the delivery of software. That can actually mean a couple of different things that I’ll go into now, I guess. For me, and for what we’re talking about here, everything starts with code. You have to think about what you’re actually releasing. Are you releasing a single binary that people download? Are you releasing a binary and some wrappers and some installers, things like that for Mac or Windows or other things that need a special installation mechanism? Or are you actually deploying something that people will use like a SaaS or something like that, which may use a binary that you create? So continuous delivery in my mind is that portion of the thing. You’re actually making some kind of artifact from code What happens next is really up to you. In my mind, continuous deployment is actually delivering software in terms of we are now deploying a SaaS or upgrading our service offering or something like that.

Matty: [00:05:27] I always kind of think about it, and I think the reason, and I think we’ve gotten past this a little bit because things have matured, but a lot of times when people hear continuous delivery, they assume we mean continuous deployment, which is I commit something to the repo, it passes tests. If all the tests pass, it ships and it’s live, right? Like then, so, and everything’s through the automation, whereas continuous delivery is almost all the way to that, right? It’s that at any point your software is releasable and it’s a business decision about do you actually say now present that in the case of like a SaaS, like Scott’s talking about, enabling that feature, ship— actually shipping it. Um, it’s, it’s at any given time you can ship your software. That’s kind of continuous delivery, which is like, oh, wait a minute. No, it’s not ready. Okay. If it’s not ready, then it’s not in that state. And, uh, the only reason I also bring this up is, so Ken Mugrage was talking on Twitter the other day and we were talking about code freezes and kind of his point was he’s like, you can still do continuous delivery through your holiday code freeze. Because you’re not deploying that necessarily, but it doesn’t mean you have to shut everything down during your code— if you’re saying, okay, it’s whatever reason you’re holding back your deployment, that’s really what you’re freezing is deployment, not necessarily ensuring that your software could be ready. You can actually be working on things. That being said, let people take a vacation. I’m not saying you have to work through that, but that’s actually kind of a beautiful thing about continuous delivery is it lets you do that. Right? You know, kind of if we weren’t working in that model, then yes, you’d have to kind of freeze everything because you have to freeze deployment. And this is letting us say we can get all the way right up to that and then say maybe we aren’t ready to do that. So kind of level set, we’re basically on the same page. Hopefully all you listeners are now knowing what we’re talking about. But if we go back to what you started with, you alluded to what it starts with. This all starts with code, right? And that’s what we’re talking about when we say things are moving to the left, maybe, right? It’s like we’re starting with some code.

Scott: [00:07:36] Absolutely. And to your point, just to sort of pile on that, what you said there, the actual deployment of the software is completely arbitrary. Most of the time you have a release cadence, you have a code freeze time, you have multiple ways of dealing with that. I’m not gonna get into them too deeply, because there’s so many of them and it’s so specific to, uh, to the customer. But I think the key takeaway there, and this, this, uh, this is really kind of one of the most important things, uh, for me personally being in, in a quality team right now, uh, is you want to get the best possible, uh, best possible artifact that you have that you can deploy sitting there ready at any point in time And then you have to make the human decision of, is this ready to go? And that’s where that release cadence, uh, decision is made. And that’s a human decision. Um, and if it’s not ready to go, it’s not ready to go, but it should be able to go.

Matty: [00:08:42] And I think that’s the key, right? The being ready doesn’t mean, is it tested? Is it whatever? It means you don’t have the right business decision, right? You know what I mean? Like, it’s not ready. Like, I think the word ready is important because ready doesn’t mean Um, uh, if we say if it’s ready, it doesn’t mean complete necessarily, right? It means, is it— are we ready to make that decision to, to release this to— into the world, right?

Scott: Yeah, yeah. And there’s, there’s a lot of, uh, there’s a whole dance around that that I, I think we’ll, we’ll have some time to talk about later, um, about what else goes into orchestrating a release, uh, because a release is not just a binary that’s suddenly available to the public. There are blog posts, there are announcements, there are so many other things that need to be coordinated. And most of those are, for most companies, are unfortunately human-driven and human-created. But there’s absolutely 100% the ability to automate most of those things. So hopefully we’ll get to, we’ll get to chat a little bit about that.

Matty: [00:09:47] We’ll get to that. Yeah, let’s, let’s start again back with the code, right? So what are some of the things as we’re thinking about that in that beginning of the lifecycle, the beginning of the of the process of getting to this beautiful new feature that’s going to make us all this money. We’re starting with the code. So what do we need to think about? What have you seen that are some just really key practices? Because you got to get it right here, right? You’re going to— things you do wrong at the beginning are just going to multiply and make everything harder later. So what do we start with? What’s our good hygiene to get started?

Scott: So I think one of the things that, uh, keeping that in mind, and, and I’m just gonna kind of talk about the binary use case that I mentioned earlier because, uh, a SaaS usually there’s, there’s a lot of uniqueness there, um, and just, just a lot of different options. So we’ll, we’ll leave that alone for, for now.

Matty: Um, it’s probably the same but more so. Exactly. You know, I mean, like the practices we’re going to talk about here will apply to your SaaS plus more things.

Scott: [00:10:54] Yeah. It’s binary++. So once you have that consistently versioned that you can automate, you want to actually go and make sure that it is in a releasable state after it goes through your CI system. And the CI system, we can talk a little bit more about that later. But It should basically create the binary from the code for a specific version, sign it so that you know that every single time you build that code, you are getting the same exact binary. There’s some caveats to that, of course, with signing, unfortunately, because you end up with a different GPG signature if you have to sign it from like Apple or Microsoft. That’s just a thing. But you have the same version that’s going out and moving through the pipeline with whatever that artifact or artifact bundle ends up being. As soon as you’re done with that, rather, as soon as that PR gets merged, for example, and you actually go and create that versioned artifact, you’re moving to the next version.

Matty: [00:12:10] It’s a point in time, right? You, you don’t, you don’t, we don’t get to go backwards in time. Right. So we don’t get to be able to be like, oh shit, there’s one more thing. Let me just sort of get it. Nope. They just cut a new one. Right. And it’s, and that’s why, you know, you have versioning strategies so that you’re not, you know, so you don’t have, you know, like artifact.final.back.back.back. Yeah. You know, that final, I really mean it.

Scott: It really removes the, uh, the, the, the need for RCs as well and betas and things like that because, um, Well, let me back up just a second and, and, uh, talk about versioning strategies for, for like 30 seconds, because there’s, uh, obviously the popular SemVer. Um, I’m a fan. Doesn’t always make sense to have SemVer though, because you may, you may not be in a position as a company to actually want to roll out 1.5.237 because, you know, and then 1.5. 3,000. Now, you know, it’s a bit— it’s not a, not a realistic example, but, you know, some customers are like, why is there a gap here? What is this? And depending on, on what you’re, what you’re, uh, you’re building, that can be, uh, that can be a blocker. So the other thing, uh, other popular way that I’ve seen is, uh, with, with just a, just a date timestamp, uh, the entire— the entirety of when this was, uh, this particular thing was created or the artifact bundle started its journey through CI.

Matty: [00:13:40] Perhaps popularized by Microsoft with things like Office 2003. I was going to say, maybe don’t use years as your major version number because, you know.

Scott: Yeah, let’s not do that.

Matty: That’s not exactly what we’re talking about here, just so we’re clear. There’s also a difference maybe between your branded version and the actual version of an artifact.

Scott: Absolutely. And that’s a really key point to make there as well, is that your branded version likely will be 1.7 in this example that I stated. But if you look at it, you may be, I don’t know, 1.7.23.456. I mean, consistency is the important part here. You may have a V in front of it. Just make sure that everything does. The other thing that I wanted to mention, though, is how that removes the requirement for a beta or RC is once you— going back to your original point of the software is releasable, once you actually get to that point where it is in a staged area, you can show that to the public if you’re— if you so choose. That is your beta. You have a rolling beta. The next piece of software that comes out is going to be this if everything is intact. It may not have all the features, it may not have all of the things that you need, but it is a thing that someone else, uh, one of your customers, could grab, uh, and fire up in their integration environment because your customers absolutely have a dev environment. Uh, spoiler alert, they do, uh, and they are looking for bugs in your software and they they don’t like it when they find them, and they will let you know.

Matty: [00:15:34] I think there’s— and again, we could go down this thing, but, you know, kind of thinking the thing that matters, I don’t think it— like, and there’s not one right way to have a versioning strategy, except there’s a wrong way, which is to have it be inconsistent within your organization or within your stuff. As long as whatever it is that you do, like, like Scott said, you want to put a V in front of your, you know, your your tags, okay, do that, but do it all the time, right? Or if you’re going to do some dev odd thing or whatever, where if it’s an odd or even number, that means something, it’s got to always mean that though, right? Yeah. And so what that brings into is the more complexity you add to your strategy, the more people have to understand the complexity. So, you know, keeping it somewhat simple and not so much overly optimized might be a— yeah.

Scott: Yeah, that’s ideal. That’s ideal. And, and, you know, you may have multiple artifacts as well, um, multiple artifacts that end up in a, in a bundle, and they may be different. So you, you can, you can think about meta-versioning, for example. That’s a little bit more advanced, but meta-versioning is— can be really, really cool where you have, you know, multiple services that you— or, or, you know, multiple code bases even that end up as a single deployable or releasable bundle. Consistency inside that, I’m going to disagree with you ever so slightly that your entire org needs to have the same versioning scheme, as long as each project actually has their own consistent versioning scheme, because you will need to integrate with other things at some point in time inside your company, and as long as that’s the same, then you can create automation for it. Um, consistency breeds automation.

Matty: [00:17:19] The word org consistency as makes sense to the size of your—

Scott: yeah, right, you know, product org, etc. Yeah, yeah, yeah.

Matty: All the people that are going to have to reason about this should be consistent.

Scott: Exactly.

Matty: So when you talk about this, you know, let’s talk just about the tooling. So we’ve got this, you know, we talked about having a good versioning strategy, and this then I think goes into having a very consistent workflow, right? Like we, we do things very similarly. And I think the one, the one thing I’m interested in, because we kind of touched on it a little bit about having sort of this org-wide thing. And I think that’s something that companies tend to want to do is say, this is how everybody’s going to build software here. And it’s a little bit of the dance you have to do of how much you want consistency, but then also if you don’t— if people don’t buy into, don’t want to use the thing, they will fight tooth and nail to not use the thing. Um, and especially because, again, a compromise is when nobody’s happy, right? So if you, if you’re a very large enterprise and you’re going to try to find one true ring of workflow for every single project, the one thing I can guarantee is that not a single person in your company will like it. Yep. Right? Like, it will be the wrong— it will be an okay answer for everybody, a terrible answer for some people, but it will never be a great answer for anybody versus, you know, where that optimization comes in. So like, where can you draw those lines about like, what is the thing to say like, no, you know what, squads, you all got to kind of do this part the same, but here’s where you can optimize to your own particular needs?

Scott: [00:19:02] Yeah, this is, this is a really great, really great question because it’s, uh, It’s a hard, hard question because we, you know, you and I both been in this business for a while. You and I both know that mandates don’t work and saying do whatever works for you also doesn’t work because you need to find that happy medium like you said. So consistency, giving teams the guidance again about consistency in things like versioning, tooling, et cetera, and maybe some suggestions. This is where having a cross-team, cross-functional room release engineering or engineering services or quality engineering team can be really, really valuable because they can, as long as they nail the right level of granularity, that’s a key point there, they can actually say, here are our recommendations for tooling and here are the things that you really do need to adhere to unless there’s a really great reason that, you know, you can describe to us as an exception. Honestly makes me think of, of, of like SOX and things like that, um, you know, regular regulations, because it’s like, well, if you have an exception, just let us know, uh, and, and we’ll talk about it, but otherwise, please follow this. But to your original point though, uh, this is, this is a key, and this is a really key point, um, any engineering tooling that you build for an engineer has to make them happy. It has to be easy to use. It has to be, uh, adding value. Otherwise, they won’t use it. Or they’ll find a workaround that they sort of are using it and they hate it. And I’ve seen that over and over and over again, especially in the mandate case, where it’s like, you must use tool X to do this. And while standardization and consistency is definitely the thing that I love, as I— we should have had a word count, like how many times did Scott say consistency? But if you look at that level of granularity that you need. Um, let them do what they need to do and provide them with things that make them happy. It’s just kind of as simple as that.

Matty: [00:21:17] So you talked about being on a quality team. So testing, I think, is paramount, right? Like, at least it’s front of mind to you. I mean, testing should be front of mind to everybody, but, you know, let’s be honest. Yeah. Um, so one first thing to go about— so let’s talk a little bit about that word quality, right? Yeah. What, what is it? What does that actually mean? Because people think different things. So when we think about testing and ensuring quality, what does that actually mean?

Scott: That’s also a good question because it’s going to be different for everyone. Um, there are so many measures of quality that I hesitate to even broach the subject, but since you asked me nicely, um, it’s going to be different for every company and every single, uh, metric of quality is going to be slightly different. Uh, and have different priorities. Uh, for some folks it’s, uh, mean time to, uh— oh crap, I’m spacing on it. What is it? Help me out here. It’s mean time to, uh, not failure, but MTTR, mean time to resolve. Yes, thank you.

Matty: [00:22:25] Okay, you’re talking about quality, about like number of incidents, or—

Scott: yeah, I mean, number of incidents, MTTRs, uh, number of bugs, number of Sev 1s. All of these maybe may or may not be important. So it’s going to be up to, uh, to, to your org and your company and your team to, to know for this piece of software, this is the most important thing, right? Uh, for a SaaS, it may be uptime. Uh, for a piece of software that you don’t— uh, I’m sorry, not deploy, but you sell to someone, it may be number of bugs. Um, I don’t really like that metric, but you know, yeah.

Matty: The thing about, about any of these metrics, we, we, we you know, talk about this a lot is obviously people will work to the metric you give them.

Scott: Exactly.

Matty: And that can run into a whole bunch of other things. There’s a great anecdotal story that Jez tells about an organization that said, okay, in every sprint, we’re going to add one more test, which resulted in a bunch of tests that were assert equals true. Sure did add some tests. And this again is the problem with like a code coverage metric, right? You’re like, great. So we have 90% code coverage, but the 10% we don’t have coverage on is the most critical part of the application. So all these things are— I think there’s a difference between metrics that are interesting and metrics that are used to measure, again, the word quality, because the word quality goes back to value of the people participating, right?

Scott: [00:23:47] Yeah.

Matty: You know, and especially as always, be very cautious with any kind of metric that’s tied to someone’s compensation because you will get network effects you aren’t expecting out of that. But what I want to think about too is I’ve talked about like Sev Ones. This is a really good example because, yeah, If the way that you’re measuring quality is the number of SEV 1s, what you find is the metric that your folks tend to, even subconsciously, focus on most is not mean time to resolve, but what I call mean time to innocence, which is how quickly can we prove this wasn’t a SEV 1, right? And that doesn’t help any—

Scott: or it wasn’t our fault.

Matty: Right, yeah, exactly. It’s a whole thing because if that’s how that’s being measured. When we think about things about quality, I think we need to think about metrics that are interesting and metrics that are actionable. Like, as in, we’re doing— we know, you know, they’re not hard and fast, and if this happens, then we’re doing a shitty job. It means, oh, huh. Like, a lot of times, I think when you see the needle move, what you want the feel to be is that it makes your ears perk up and go, hmm. Let’s take a look at why this happened. Oh, actually, it looks like there’s not a really big deal. This is just— it’s just a thing that happened. Or if I see this thing start to shift and then go, ooh, looks like we’re having a problem with this quality gate here that we didn’t notice. And that’s when those things can be really helpful because you don’t want to ignore them because that’s how you have a normalization of, you know, you have this thing where you don’t notice the stuff. But, you know, again, the last thing you want is for people to be incented against making it not happen to the point that then they will just make the alert doesn’t go off, right? You could— we could— we— I can make a dashboard say anything you want, but that’s not helpful.

Scott: [00:25:37] Yeah, absolutely right.

Matty: Like, if we’re not learning from it.

Scott: Yeah, 100%. And, uh, my hot take— not really a hot take, uh, to go back to one of the, one of the things that you said— is just Please, please don’t tie compensation to metrics, quality metrics, or anything like that. That is a terrible way to motivate people. And just, okay, please. What I fall back to, to a certain degree, is there are already some great research and stuff that has been done on quality metrics and what works. The Square framework, um, uh, I have it written down right here, ISO/IEC 25010. That is ISO/IEC 25010. And, uh, the, uh, the Quamocos, uh, framework, like those are a great starting place to look at those. And, um, yeah, basically like you can build your own thing on top of that pretty easily. Um, and, uh, you know, have a lot of great outcomes from that, as long as you’re talking about what your requirements are. So yeah, I don’t know if there’s really much value in going super deep into metrics other than you should have them and you should basically go and see what’s most important for you.

Matty: [00:27:09] Well, and I’ve heard you kind of talk before about customer confidence, like this idea of— I’m really interested to dig into that a little bit more about what are we talking about there, because I can tell you what I think you mean, but, you know, I’d much rather have you tell me what you actually mean.

Scott: Yeah, yeah. So customer confidence, uh, the best way to describe it is how much your customers trust you. Um, how much do they trust that when they are using your tool or your software that it will do the thing that it is advertised to do, or not even advertised. More than that, it does the thing that they need it to do. Because let’s be fair, they don’t give a shit about what you tell them it should do. They give a shit about it— if they never have to think about your software and it never causes a problem, they’re happy. As soon as they have to think about it, Like, they’re, they’re like, oh, goddamn it, fucking people, why can’t they make their shit work, right? That is, that is what customer confidence is, is how much they have to touch your software and administer it, because they don’t want to. They want to focus on their own business outcomes.

Matty: [00:28:20] There was— oh, and if I can find it, I’ll put it in the show notes, but I’m not going to promise. But I did see a tweet the other day kind of to that effect, which says, you know, we keep talking about customer delight, and they said, what I want is I want my customers to be completely unaware that my stuff exists, or something to that effect, which is basically what Scott just said, which is I— we need to focus more on like, uh, our customers not, not thinking about our experience but just doing the stuff they need to do. And, and that— and yeah, I’m gonna see if I can find that tweet. It was, it was, it was said much more, um, eloquently than I just did.

Scott: But well, it’s, it’s the old, uh, and, and, and this is the The, the, the whole ops is a thankless profession and, and, you know, that kind of thing where, uh, I’m doing my job really, really, really well. You don’t know that I exist. Right.

Matty: We could, we could even take a reference to Futurama, to the excellent, excellent Godfellas episode when the supreme being or whatever it is says to Bender, sometimes when you do your job right, nobody even knows you did it at all. You know? So if that’s good enough for God, it’s good enough for software vendors. That’s all I’m gonna say.

Scott: [00:29:34] Absolutely, absolutely. Yeah, I mean, there’s a, there’s a lot of things that’ll tell you as well, and this is kind of an important point too, is like there’s a lot of things that’ll tell you if customers have confidence in you. You know, that one of those being, and this is a thing that, that I have, I’ve seen actually happen, is a customer saying, I We’re fucking tired of being your QA. And that, oh my God, if that doesn’t sting and make you feel bad, it should. It should. Because that’s rough. And you know they’re having a bad time. So, I mean, that’s one of those things why quality and testing is important. Also, If you read Nicole Forsgren’s book Accelerate, you’ll also see why quality and testing makes your stuff go faster and you make more money and everything is way, way better. So I’m not even going to go into that because she and her co-authors basically killed it on that book.

Matty: [00:30:47] So the other thing is, I mean, I think for the most— I would say for the most part, but I know this isn’t true, but there’s kind of this this model of, you know, we had QA teams and they were really very siloed and everything, and we’ve all supposedly gotten so much more agile and all these kinds of things. Reality is, no, we haven’t. Um, I mean, many of us have, many of us have, but, you know, I, I can tell you a whole bunch of enterprises I’ve talked to that we aren’t quite there. But when we think about a more modern way of thinking about quality and testing and how do we really democratize this and, and become more cross-functional, you know, whether sort of just saying be more cross-functional, but like sort of from an actionable perspective, what are some, you know, tips and tricks, some ways to kind of turn that thing around?

Scott: Yeah. So let’s get into the nuts and bolts of this. Like how do you actually implement this? There’s a couple of things. I mean, we’ve kind of tiptoed around it, but really, how do you make this work? It starts at the very beginning. Make your PRs run tests. Make your— have as much as possible within reason run in your PRs. For the moment, like, let’s just assume that you’re using GitHub. You know, there’s, there’s, there’s a lot of other options, but assume GitHub at this point in time. Your PRs should give you the highest level of confidence that they possibly can within a, you know, reasonable time. So, you know, don’t do everything, but do, do as much as you can. The other thing about that is, Make sure that your PRs create an artifact, as long as it is reasonable. Maybe it’s just one artifact if you’re shipping, like, 12 different platforms or something, but make sure that it builds at least something that you can actually, as a dev engineer, pull down and be like, what broke here? My PR is not working and it’s a really weird thing. I need to test this locally. Let’s do that. The second thing from there is make sure that every time— I mean, this is kind of getting into the basics, but it’s worth saying again— make sure that when you merge a PR, you run exactly the same tests.

Matty: [00:33:04] Hooray!

Scott: Because main branch creep is a real thing. They have a lot of automation built into GitHub, for example, that tries and makes so that that’s not a problem, but it’s still a problem. Um, and from there we go into the meat of it, and this is my favorite part. Um, this is the RelEng person inside of me being super happy, um, is you need to make a workflow that works for you, that has tight feedback loops, that has quality gates. And what— by what I mean by that is at every stage you are moving your artifact farther to the right you are ensuring that the quality is increasing, your confidence is increasing, and the reliability of said thing is increasing. There’s a bazillion ways to store artifacts and everything like that, but moving it to a new place when you are more confident because you’ve run certain things— and we’ll talk about that in a second— uh, is, is really the key.

Matty: [00:34:14] So when we think about those things that we’re going to run, right, is this when we’re thinking about our workflow, our orchestration, all of these pieces, how they tie together?

Scott: To a certain degree, yes, um, because at its simplest— and this is unfortunately probably never a scenario that you’ll run into— but at its simplest, you’ll end up with a pipeline of sorts, a workflow of sorts, maybe multiple pipelines, totally depends on your scenario. But what, what, what’ll happen is you’ll have an arbitrary number of actions that need to take place. These actions can be anything from running tests on your software, running code coverage, which, as we mentioned, eh, take it or leave it, running a security scan. Those are some of my favorites.

Matty: Running—

Scott: you can boil it down to actions. Anything that makes you more confident that this is a releasable piece of software and a mature That should start off with the quickest, highest-value tests. Unit tests are a great example of that. From there, maybe some notifications, right? You want to know immediately that that has passed or failed your unit tests. Then move it to your next stage. Run some tests on the binaries, like make sure that your binary is the right version. In a super, super simple case. From there, make sure that you have notarized it at that point in time, your bundle or whatever you’re actually releasing. Move on from there to load testing, perhaps, a long-running instance of it, of your software that you have, you know, actually gone and thrown some, you know, sort of customer-like data at. Run through that and make sure that everything is still performing the way that you want it to. And you haven’t had creep, you know, maybe you changed a database thing and you screwed it all up. Who knows? And then also make sure that people can upgrade. Like, this is the part where you need to talk to your customers. This is the part where you need to think about what your customers are going through when they are actually using your software and maintaining your software. Because if you have a shitty experience upgrading from one version to another, guess what? They’re not gonna upgrade. And the longer they don’t upgrade, the worse it will be. And then guess what? I bet you you’ll have a SEV 1 and you’ll need 4 people on the engineering team to come and help upgrade the software, and etc., etc., etc. And you’ll have to write another custom tool to make it so that you can pull the data out. It’s gonna last 4 days. It’s gonna suck. Don’t do that.

Matty: [00:37:01] Eventually, you heard it here first, don’t do that.

Scott: Eventually though, you’ll end up with a piece of software that you have enough confidence in and you have enough— it has enough maturity, you’ve run enough tests, you’ve run enough things, security scans, there’s a ton of different things, so those are just examples. But at that point, it’s essentially a releasable artifact. You’re in a staging area, I like to call it, where you could release it at any point, going back to the very first thing we talked about where, you know, you push that button, some actions happen, and one of those actions is the software is now public. That’s my spiel. Having a pipeline or a workflow that supports this is the foundation.

Matty: We’ve talked a bunch. There’s been a bunch of practical stuff. But maybe let’s spend a little time kind of talking about, okay, how do we get— like, we’re where we are, and obviously it’s going to depend on what your role in your organization is, the size of your org, all that stuff. But I think Like, yeah, how do we get— how do we get to— we’ve talked about the there, right? The things that are important. But you’re like, great, Scott, but now how do I do that? Like, what can I actually do? Like, how do I build a plan?

Scott: [00:38:16] Yeah, that’s a super good question because most of the customers and orgs that I’ve dealt with, they’re not at a place where they can actually do that. It’s not greenfield. They’ve had some stuff that they’ve put together, it’s grown organically most likely, and they need to create a plan. So there are a number of ways to do this, depend on the org. I feel really like hand-wavy in talking about this because so many of these things are company-specific, but from a generic standpoint, you need to make a plan that is really a migration. If you think about it like a migration and you think about it in chunks, break that down and do some story mapping, you’ll be able to find that sweet spot of still being able to do the work that you need to do because you can’t just stop everything and be like, all right, we’re going to take 6 months and fix our release process. I mean, maybe you can. I actually worked at a place once where they did that for a whole year. Guess what? That was really, really painful. I wouldn’t recommend it, uh, because I’m not a fan of the sort of big bang, uh, things. Um, and hopefully you aren’t either, right?

Matty: [00:39:34] Well, right, you’re not gonna— I, I think that’s the thing is you approach this iteratively, you approach it in small chunks. It’s the same thing about any kind of transformative work, you know, and it’s, it’s, it’s very, uh appealing to think about, like, well, I want to think of all the contingencies, especially if you’re in an enterprise that is not like, as Corey Quinn would say, you know, like Twitter for pets, right, or something like that. And yeah, I, I remember, you know, working with, with, you know, huge, you know, health, uh, providers and stuff like that where they, they, you know, feel very much of the, like, we’re gonna measure 12 times before we cut once, and we have to think of all the kind of like, to paraphrase Pulp Fiction, I got to think of all the fucking ifs. Well, here’s the thing. First of all, you’re going to miss some anyway. So thinking you’re going to think of every single possibility is not going to happen anyway. So the quicker you say, you don’t try to optimize for that and say, let’s just start doing something, but do it in a small enough way, and we’ve got good feedback that if it doesn’t go the way we want it to, We can adjust, we can pivot, we can work.

Scott: [00:40:42] Yep.

Matty: And it’s just your stuff all happened, like Scott said, organically and slowly over time. So your fixes can work similarly too, right? Like you, you know, look for where you have these areas of opportunity and especially that don’t require turning your entire org around to do, but even are there things that within the constraints or the frameworks or the requirements that maybe are mandated to you, you can still make things better. Because a lot of times we’ll sit there and be like, well, I’d love this better, but we’ve got these like ivory tower architects that dictate how we do everything. And I’m kind of like, do they though? Yep. You know, there’s—

Scott: yeah, they don’t have to. Yeah, there are places you can work still with that. Absolutely. Yeah. And, and, and this is where doing a little bit of analysis can go a really long way. Um, you know, get— getting some input from, from your engineers, um, if you’re in a sort of traditional There’s a service team of some sort, releng, engserv, whatever it is. There’s a team providing services and is handling the CI/CD and stuff like that. It’s pretty common. But talk to your engineers. Find their pain points. Don’t spend too much time, don’t listen to everything they say, because they are very highly opinionated, and that’s good, because that’s their job. They should be doing that. Talk to your support folks. Listen to what they have to say. Like, they are the ones who bear the brunt of the pain when a customer is having a bad day. So buy them coffee as well, they deserve it. Um, and, and go from there. Make a, make a tactical plan to solve a small problem, move from there. Make some RFCs maybe. Like, I’m a big, big fan of RFCs. Uh, make that and be like, hey, I would like some info. Like, I would like you engineers and use support to take a look at this and see, is this actually going to help solving the right problems? Do I have the right actors involved? Where do I go from here? And then break that down into an action plan. You know, again, story mapping is great. I love it to death. Do that. The most important thing is tie it directly to company value and company goals. Like, do that because guess what? You’ll be able to. If the project that you’re working on that you want to make, the changes that you want to make to make life better, is the right one, you’ll be able to go and say, oh, you know what? This increases quality or this increases velocity or this makes customers happy. Get that, figure that out, and tie it directly to the bottom line. You will be able to if you’re doing the right work.

Matty: [00:43:24] There is a, um, a talk I’ve given a couple times called Everything’s a Product, and it’s, uh, you know, everything’s a product. It’s about, uh, applying product management practices to technology services, and I’ll put a link in the show notes. But I mean, this doesn’t mean things like, you know, put like an NPS score on would you recommend this Jenkins pipeline to a friend or colleague. I don’t know, maybe you want to do that. Please do not.

Scott: But no, there’s— please don’t.

Matty: When you’re building, when you’re building something like this, especially if you’re like Scott said, like a, you know, a rel eng or, you know, an end service team or something like that, you have customers, you know, those are those engineers. So you can run this like a product you can manage. And there’s a lot we can learn from product, uh, product ownership, product management principles. So think about things that way. I think, I think we tend to only think about managing products and things that our external customers consume. But these same principles apply because it’s about feedback. It’s about, um, having a plan. It’s about getting input. But then again, your customer is not always right. Like Scott said, your engineers will be very opinionated about what they think they want, but they might not know what they need, you know, and you can, uh, but that doesn’t mean you dictate it, but it means being able, like good managed product management is synthesizing what your customer tells you they want. And then understanding how to take that and yes, and it into something that they need. Right. So yeah, learn how to be a product manager. That’s, that’s my, my, that’s the, that’s my, uh, super important thing when you’re true. And even if you aren’t like a dedicated RelEng team or whatever, even if you’re building it internally to your squad, think about it that way. Right. Um, where those things go. What, uh, I mean, here we’re coming, coming to the end of our time. We could, we could have about 5 more episodes on some of the stuff we started to talk about.

Scott: [00:45:15] I could talk for hours.

Matty: Yeah, me too. What though, Scott, if you wanted to leave everybody with one thing to keep in mind about how to make releasing software better for them? What’s like your one key thing? I know you said the most important thing, but just the overall, what’s the one thing you want people to think about?

Scott: And when they, when they’re done tuning in here, be empathetic, be empathetic to your customers. To your engineers, to the people on your team, and really, really, really try to put yourself in their shoes. Because at the end of the day, like, and I’m gonna quote Adam Jacob here, because one of the very first, I think it was in the keynote or something before I worked at Chef, and this is one of the things that made me fall in love with Chef and want to work there, He said, happy people make happy software, which makes for happy customers. And I may be paraphrasing slightly there, uh, sorry Adam if you’re listening, but that really, really stuck with me. Um, and, and that’s really the key to it, is be empathetic, uh, understand, take the time to understand it, um, especially if you’ve just been in an engineering role or if you’ve only been in one role and you haven’t had The opportunities that I have, for example, to be in support and have that, be in a releng team, be the customer of a thing, or I’m sorry, a consumer, a customer of this product. One of the things for me where I got a lot of empathy was I actually have been on both sides of the Chef experience. The company I worked for before that was an enterprise that I was, I was a Chef customer for 3 years, um, and, you know, I got to go on the other side of that, and it was a whole different set of, of feelings. And, and that’s, yeah, that’s the thing, just, uh, be empathetic.

Matty: [00:47:24] Well, head over to arresteddevops.com/doing-releases-right for this episode’s show notes. If you go to arresteddevops.com/itunes and leave us a review in the Apple Podcasts store. Uh, I guess that helps other people find the podcast, and if we like it, we might read it on a show or something. Uh, you can also subscribe to us on Spotify or iHeartRadio or anywhere fine podcasts are not sold but given away. Uh, Scott, thanks for joining me today. This has been super fun. I’m really glad that, uh, you’re on the show. Thanks so much.

Scott: Yeah, thank you, Matt. I’m very happy to be here and love the show. First-time caller, long-time listener. So yeah, this is a great experience, and I hope that some of what we’ve talked about today can help people out.

Matty: As always, this is Arrested DevOps, and remember, there is always DevOps in the banana stand.