Charity: [00:00:00] I build Debian packages because I’m a sick fuck who really enjoys it.
Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m your co-host, Matt Stratton, @MattStratton on Twitter.
Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.
Bridget: And I’m your co-host, Bridget Kromhout, @bridgetkromhout on Twitter.
Trevor: Arrested DevOps is brought to you by 10th Magnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. 10th Magnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/tenthmagnitude.
Matty: This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 120 common infrastructure tools such as Chef, Docker, and AWS, so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arrestedevops.com/datadog.
Bridget: [00:01:24] All right, so this episode is called Who Owns Your Availability? ’Cause recent events in the npm community have rekindled that perennial favorite discussion about dependency management, single or multiple points of failure, and how you can control them, or maybe how they’re always on fire all the time. We figured we’d talk to longtime ops professionals and get some viewpoints on this. So we have here today Pete Cheslock, who’s been on the show before, Pete, you want to tell us about this 15 years of DevOps experience and tell us about yourself?
Pete: Yeah, so it’s awesome to be back on the show. My 15 years of DevOps experience is the greatest troll at ThreatStack.
Trevor: I love it.
Pete: And I, and I’ve— for those that don’t know, on the ThreatStack website in the About page for the leadership team, you can see the bios of everyone on the leadership team at ThreatStack and And clearly no one clicks on it because it’s been this way for about 2 years now. And it basically says Pete Cheslock with his 15 years of DevOps experience, which was a joke by my VP of Engineering who put that in there.
Bridget: [00:02:32] And 2 years ago, so you haven’t actually gained any DevOps experience in the last 2 years.
Pete: Absolutely no DevOps experience gained here.
Matty: In fact, maybe it’s gone down.
Pete: We should update it to at least like 10. I’m pretty sure I’ve lost it.
Bridget: Alright, so, but anyway, so Pete is in charge of— is it like ops and support?
Pete: Yeah, so right now I run our operations and support teams. And yeah, we do the cloud and the DevOps and try to be as secure as possible because as a security company, you know, we, you know, got to pay attention to that kind of stuff.
Bridget: Yeah, I think my elevator pitch for ThreatStack when I’m pitching it to people, which, you know, I think I did at re:Invent— my elevator pitch for ThreatStack is ThreatStack, it makes it It makes all of that cloud stuff for you slightly less terrifying.
Pete: I’m going to have to send that over to my marketing team. That’s good.
Matty: Great.
Bridget: And also delightfully, we have Charity Majors. Stratton, I know you were going to intro Charity. Tell us why we got her on the show.
Matty: [00:03:33] Well, basically, I think Bridget reached out to Charity and said, hey, we need someone to do some ranting. Would you like to come and rant with us? And Charity said, Yes, because whiskey and ranting. So is that about right?
Charity: That was literally it. That was literally it. I had no context in what we were going to be talking about.
Bridget: But I hear you like rants.
Charity: I have been known to rant once or twice in my career.
Bridget: Charity, for our listeners who may not be familiar with your work, can you talk a little bit about your career, ops and otherwise? Where are you at right now?
Charity: Yeah, totally. I just co-founded a company 3 months ago called Hound, which I’m really excited about. I think that we are going to make— to quote Silicon Valley— we’re going to make the world a better place for anyone who has to deal with machine data and real-time analytics and telling what the fuck is going wrong with your distributed system at any point in time. Before that, I was first infrastructure hire at Parse. We got acquired by Facebook. So I’ve spent a bunch of years dealing with some of the worst co-tenancy performance problems that you can imagine. And I’m super excited to start a company where I get to help create those co-tenancy problems instead of just dealing with other people’s. Nice.
Bridget: [00:05:00] Fantastic. So I feel like there is a website for this because there’s always a website for this. And this one has actually been around a lot longer than the npm This is whoownsmyavailability.com. If you go to it and you just keep reloading it, it tells you, spoiler alert, that you own your availability and then gives you some reading, which I think is great. I know some stuff from AllSpa and other things come up. And I feel like this is one of those things where it’s easy to kind of armchair quarterback or Monday morning quarterback when somebody else has something go wrong with their dependencies. But the reality is we’ve all had this stuff happen. And so I think it would be kind of useful and constructive for us to talk about it, talk about the stuff that can go wrong, um, when we’re building, you know, as Charity was pointing out, like complex interconnected distributed systems that have various failure modes. Like, talk about where these things go wrong and talk about how we can try to architect our code and our systems to make them go slightly less wrong. Like, we know we’re not going to make them perfect, right? But how we can avoid some of these problems, I think it’d be a cool thing to talk about.
Charity: [00:06:00] We start by saying I love that website, like, whoownsyouravailability.com, and it says you. And that’s always like a gut check, like, yes, you. But it’s also not you, right? It’s you, it’s your team, it’s your processes, it’s the teams around you. And so it’s not so much that, like, I think that we in operations tend to get this hero-martyr complex where we are very easily convinced that it is actually just us that owns it. I like to— I would like to just start by saying that we should perceive that you as, you can’t blame a vendor. You can’t blame your platform that your shit is built on top of. You can’t pass the buck, but it’s actually not up to you to be a hero or a savior or a martyr.
Pete: Yeah, so Bridget, I was gonna say, it’s definitely good to focus on what we can do to make this I don’t want to say problem go away, because it never goes away. But focusing on tips and tricks, I think, at least a lot of the stuff that personally I’ve learned the last few years. But also, applying blame to any of this stuff is just comical, because within just years, I’m sure many people can think of other communities, other projects. The only thing that we definitely know is that if it’s If it’s online and on the internet, it’s going to go down. If it’s some sort of platform, it’s going to break in some way. That’s the one guarantee we definitely know is going to happen. You definitely never want it to happen to you, but it will happen to you at some point. It’s always going to happen.
Bridget: [00:07:42] Wait, so you’re saying computers: yes.
Pete: Computers: yeah.
Matty: And business, right? It’s interesting because when I was thinking about this, some of us are thinking about this from context of, you know, the tweetstorm of late of what’s happened, or we think about, you know, outages at the service level and things like that. But inside the podcasting community, there’s actually— this sounds like to us, this is maybe going to sound like such a no-brainer, but there’s a huge thing about own your own RSS feed because like everybody loved FeedBurner, right? And then, you know, folks who knew what they were talking in the podcast community say don’t do that because you are now putting yourself you don’t own that thing, right? And because what happens when a project that you depend on, if it becomes abandonware or goes out of business or totally changes? So it’s not just about the uptime of the services that you depend on, but realizing that it’s somebody else’s business. So it’s business and computers, I think.
Bridget: And I think that this is the moment where we do have to pour one out for Parse, because Charity has some deep personal insight into exactly that.
Charity: [00:08:45] I will pour one out for Parse, but Parse did not fail for technical reasons, so I probably shouldn’t say much more about it.
Bridget: Sure, sure. I’m not even saying the technical reasons. I’m actually saying that— or I mean, I don’t know anything about those. I mean that a lot of people built stuff using Parse, and I was really impressed and happy to see that the Parse community pulled together when it was no longer going to be a, this is being run for you over here. It suddenly became, this is an open source project. This is a different kind of SaaS. These are a whole bunch of open options. Like that, that kind of proliferation of options when people realize that Maybe this one central single point of failure/service isn’t going to be something that we’re going to have, but there are other choices.
Charity: Yeah, completely. And I mean, you see this with, you know, you saw this with Parse, you saw this with Heroku, you see this with AWS, for God’s sake, the Godzilla that rarely fails, but when it does, it’s like everybody on the internet is like, uh, welp. But you still have to own it, right? You still own your availability. If an AWS availability zone or region goes down and you made the choice to be single-homed, that may have been the right choice. It may still be the right choice. But you can’t pass the buck and say, oh, well, it’s their fault. And with Parse, I’m so glad and so grateful that they’re trying to create a glide path. They’re trying to open source. They’re trying to open it up and make it as easy as possible for people to transfer off. But, you know, even the people who chose to build on Parse, they own their availability there too, you know.
Bridget: [00:10:22] So I’m guessing for people who did not spend the last day and a half reading jokes about LeftPad and LeftShark on Twitter, probably are wondering at this point, wait, what? What are we talking about? So maybe just to kind of— I don’t know if Trevor, would you like to be the voice of the dev here and explain to us exactly You know, the, the TL;DR from the npm blog about what happened?
Trevor: I’ll just read what they wrote. Earlier this week, many npm users suffered disruption when a package that many projects depend on, directly or indirectly, was unpublished by its author as part of a dispute over a package name. The event generated a lot of attention and raised many concerns because of the scale of the disruption, the circumstances that led to this dispute, and the actions npm Inc. took in response. So npm, we’ll put a link in the show notes. There’s a whole write-up about what happened and what npm did about it and the kind of effect that it had on not just some companies but a lot of companies.
Bridget: Yeah, so I think the, the TL;DR, whatever, is like that because this is a global namespace and a package that a lot of people were just importing live got unpublished and then it really was like not available for all of these sites that were trying to pull it in and use it live. So this is kinda one of those, huh, okay, people in the Go community have had this problem. Think about, you know, all of your PyPI packages, whatever, like, you know, even Chef Supermarket. Can we talk a little bit about where you vendor— you know, what were you saying, Pete? Like, vendor your deps, kids?
Matty: [00:11:59] Vendor your deps, kids.
Pete: That’s right. Yeah, I mean Rendering everything— people, I guess people— this was actually the one thing I put on Twitter a few times because I did have a fairly rage-induced moment. So actually, ThreatStack, we have a lot of Node.js, and this didn’t affect us because we used Artifactory. And, you know, we really do control our dependencies for our projects, mainly because every time we pull in a dependency, you know, we’re basically adding risk into the equation. So, you know, we do have developers who are pay attention to that, but we also do something called vendoring your dependencies. And if you don’t know what that is, it basically just means— there’s probably a lot of examples and descriptions for it. The way I usually describe vendoring dependencies or vendoring packages is really just taking a copy of that package, bringing it internally, and serving it up yourself. So instead of depending on, like, an upstream npm repository, you can use open source tools and paid closed source tools to store that stuff locally. We’re a Chef user. I vendor all my Chef dependencies. I don’t call to supermarket. I don’t talk to the outside world. If I’m using it, I bring it internally. And it’s a similar reason, is I just want to kind of own that world. More and more, we even do the same thing around vendoring full-on Debian packages. If there’s a third-party package we use for— maybe Cassandra is a good example. They have third-party packages on there. Debian repositories they run, just so that we can ensure that we always have the right version that we need at the time. You know, we again, we pull those locally and just shove them in our own repo as well. So it can seem to a lot of people like, why would you pull all this stuff locally? It’s all on the internet. Until like you get burned literally one time, and then you kind of look back and go, oh well, that’s the reason why.
Charity: [00:13:48] Cassandra is a great example of a thing where like They just stopped publishing a package that we were relying on, and suddenly we’re like, oh fuck, can’t bring up new instances without upgrading your entire Cassandra cluster. And we went, you know, part of this I want to point out is—
Pete: That was literally the exact scenario that we went through, which is they only stored the most recent version. Right, right. And you’re saying to yourself, well, I don’t want the most recent. I want the one I’m running.
Charity: No, I have not tested this recent version. Part of this is, like, it’s very much tied and targeted towards the lifecycle of your company. If you’re a startup, and I’m right now speaking from the perspective of a 3-month startup, it would be a total waste of my time to vendor every package that I’m depending on. My availability requirements are not here. They’re here. And my time is not here. It’s like here. And so it would be a total waste of time. But I’ve been through this cycle of growing up your company. And there comes a stage where these failures start to take you down. They start to affect your ability to push code. They start to affect your ability to roll back. And like, it’s a continuum. But there’s a point where you arrive where every apt package you use every gem you use, every npm package that you use, you should have a local cache.
Bridget: [00:15:16] Especially because, and I’ve brought this up on the show before, but I’ve actually talked to the fine folks at Docker about it and they were like, we’re so sorry. But sometime back, a couple of years ago, they pushed a new version. Sorry, they pushed a broken version of the Docker registry. I think it was 0.6.9 and didn’t change the version number on it. They just pushed a broken one. And we couldn’t bring up instances anymore because we were running local registry, but we were just doing the pull from Docker Hub of, you know, the official registry container. So guess what? We put the official registry container over in our own account on Docker Hub so we at least could pull from one that we knew was known good. This still had a point of failure there, but it was like, you don’t know when that’s going to happen, right?
Trevor: I mean, I’ve seen the same thing with Jenkins. Jenkins only hosts Debian packages at the latest version. You can pull down the specific versions, but you can’t just, you know, add the repository and apt-get Jenkins and get the right version. They released a version where we couldn’t use the Groovy scripts, so suddenly I had to learn how to build a package feed.
Matty: [00:16:24] I think what’s funny is all the stuff that, you know, we kind of lols at all those silly enterprise people that do this. I’m like, they didn’t get burned by any of this shit. Because all the people that are like, hell no, build systems don’t get to go to the internet. We do everything on the inside. It’s not because of availability, right? It’s because we want, you know, it’s for whatever other reason, and maybe the initial reason for it might— we can question the validity, but you’re safe, right? They’re like, it’s no problem because our build systems don’t do this. And I think the other thing that’s scary is when you think about the depths of depths of depths, right? So you might, you know, all the times when people are dependent on things that they don’t know they’re dependent on. Even knowing how many people would use a system, and I can’t think of examples that came up, but I’m sure they were, that were using some system that had a dependency on LeftPad. They didn’t know that, right? They didn’t even know that it was Node, right? It was being built in a whole different way. And so it gets really— again, the more we create these abstractions, we don’t know where our risks lie necessarily, and I think that’s what starts to become challenging. Sorry, I was gonna say, I wonder how many people look at—
Trevor: [00:17:32] like, every once in a while I’ll look, but I’ll be honest and say I don’t necessarily care— how many people look at their Burks lock files or their gem lock files and see the depth of the dependency tree that gets generated?
Matty: Especially if you’re just trying to use a tool, right? Like, I think a lot of us on this, on the panel right now, would sit there and say, well, of course I do, because I’m writing custom cookbooks with deep resources and blah blah blah blah blah. Okay, how many people are just like, hey, I just need to install Consul, and there’s a cookbook that does that, and I’m going to pull it in. Or, I just need to use this gem, and it’s done. That’s the thing, right? Like, if even—
Charity: I just need to bring up a host, and, like, my first boot script involves— I didn’t realize it involved, like, you know, half a dozen calls to, like, apt-get and update for, like, half a dozen deb repos and, like, the gem repositories and, like, all this shit. All I wanted to do was bring up a host, right? And if I want to bring up a host because I’m having scaling problems, the last thing I want to have to do is, like, detangle somebody else’s outage.
Pete: [00:18:38] Yeah, so we ship packages for our customers. We have an agent that our customers install to scan their systems and capture events for security reasons and security bits. And I’m one of— there’s a handful of people who have the ability to sign those packages because security and things. We lock access to keys and all this other stuff. So I’m one of the few people who actually sign those packages and does deploy to our customers. And that’s my biggest fear, of making some mistake as part of that process. You can automate it all these different ways, but it’s still— I like to call it, there’s a high pucker factor sometimes when you’re pushing code that thousands and thousands of systems can grab at any time. And if you were to push a release that was bad, you never want to cause an outage for another person. Being on both— I’m on both sides of it nowadays, where not only am I consuming other people’s packages, I’m producing packages for our customers. And it’s a bit scary sometimes. And you do everything you can to follow the processes. But at the same time, I kind of hope that my customers do what I do, which is vending packages we’re putting up, which are stored on S3 using repositories. They’re durable, and they’re up there, but at the same time, taking out that one possibility of upstream network connection or proxy— I think you said before, why did enterprises not get burned by this? Because there was some security person in the back room who said, no outbound internet connection on any ports at all.
Charity: [00:20:12] Yeah, or there are other ways to build resiliency into this system, right? One way is caching literally every package you ever use. Another way is not having to reinstall every single package every time a node boots or an instance boots up. If you’re building your base image with Packer and you’re aware of what things can and can’t fail, make the process only fail on a couple of things. If you’re bringing up a Cassandra node, have everything baked into the AMI maybe except your Cassandra package, which you then cache in an apt repo that you control. Right? So if your apt repo is down, you can blame yourself and you can fix that pretty quickly. But you don’t have to debug like, oh, well, is EC2’s Ubuntu apt repo messing up? Or is some fucking dependency somewhere along this path messing up?
Bridget: You know? Some third-party repo somewhere, right?
Charity: [00:21:13] Like, you should have as few things that can fail while you’re doing critical things as possible.
Bridget: Yeah, that’s, that’s kind of the, like, build as much simplicity into your system as you can because you know that everything else is going to be a shit show. Like, I, I like what you mentioned with Packer because that’s what we ended up doing at Drama Fever, is using Chef to define our, um, you know, base image of what we wanted it to look like and using Jenkins jobs to build AMIs with Packer so that, yeah, of course we were Dockerizing all the Dockers, but we also had our instances would come up in AWS exactly as expected and then do their Docker pull and get the most current software to run. But at least we eliminated a lot of potential points of failure. Not all points of failure, but—
Charity: You’re building your Packer image with a Chef cookbook, so it’s still code. Right? But if your package image generation fails, you’re just like, oh, well, that’s annoying. I’ll wait an hour. Like, this is probably not going to impact my, my ability to fix things and run things in production.
Bridget: [00:22:16] Exactly. And I think that’s a good way to look at it too, is like, when you’re iterating rapidly and developing parts of your infrastructure, you are going to be experimenting and finding things, finding all those corner cases. But you probably don’t want to be finding all those corner cases while you would prefer to be sleeping or drinking whiskey.
Pete: Yeah, Charity, something that you said earlier in the talk about, you know, at your stage of your company, you know, your, you know, availability and risk levels are different than other people at different stages of their company. I think that’s actually a really good point. When I started here at ThreatStack, you know, a couple of years ago, you know, we, we had nothing really. It was a beta product. It was a couple of servers in Amazon. We had to kind of build out the thing that customers would consume. And in the really early days, as I go back and remember, You know, there were some things that we did because of necessity. You know, we’re a big Node.js shop, and we’ve got some other programming languages, and we would build Debian packages because npm installing on all your nodes is just way too much of everything, really. So we build packages for those. But for a lot of other things, like when a new node came up and provisioned a new node, it’s like, here’s the 10-minute Chef run, right, on every single node. Because at the time, you know, there was like time and resources and all these other things of Even going down the path of, like, packerizing that wasn’t something we had the resources for. But then, of course, as we grew and our uptime and scale requirements increased, that’s when it was like, okay, now let’s throw some resources and say, hey, let’s take some time and let’s packerize the base so nodes come up in, you know, 1 minute instead of 10. But not just the speed difference, but that ability to say it’s built with Packer via the Chef Cookbook, and it has all the bits I need, all that’s in code. And the number of external calls it has to make to become available and start consuming data is significantly less. So it’s like you take your risk profile, this huge amount, and decrease it down a bit.
Bridget: [00:24:09] Onto that, I want to say not doing the Chef client run live is a lot more calm-inducing. I would much rather have the Chef client run happening not live than have it all running on the production servers, and I wonder how this is going to go. No, no, immutable, immutable that shit. Don’t wanna.
Charity: I 100% agree. I just, I feel this like, I’ve been feeling this a lot more over time as my like scope and scale has changed drastically many times over the past few years. Whenever we’re talking about best practices and trying to give people our like advice from the mountain on how they should do things, like I just like to toss in Dude, it’s contextual. It’s so contextual. Over-engineering at an early stage is just as damaging and can be just as company-killing as not cleaning that up and not making it faster and more robust when you’re later on in the cycle.
Pete: That is a huge, huge point, premature optimization. When we started and came on board to ship our product, the original plan was 6 months. We were going to ship in February or something like that. Well, that was all fine and dandy until we had the opportunity to launch at re:Invent. So it’s like, hey! The marketing team comes over one day and says, hey, actually, we’re going to ship in 2 months. Are you ready for that? You have to make choices, right? So we put all these things, and we were doing just basically what was required to get the system up and scalable so that we could ship at the time required. The most important part, though, and I think this is where companies miss out on it, is that if you don’t go back later and pay down that tech debt that you just accrued. That was one thing that helped us a lot, is to go back at a later time and say, all right, here are these things that we didn’t package, or here are these things that we didn’t build the way we wanted to build them, so let’s go back and actually do that now versus just hammering forward with more features.
Matty: [00:26:09] You want to do it in a deliberate way, right? You want to be aware of the changes that you’re making. You know, as they say, you know, you don’t have a scale problem. You’ll know when you have a scale problem, right? You don’t have to worry about that right away. But I think you do need to think about where these potential— you know, again, you need to have your eyes wide open, right? You’re making a decision about acceptable risk. You’re saying, okay, I’m not going to vendor every single debt that I have because that’s not the right decision for the business where we are right now. That being said, we’re aware of it. We’ve made the decision. As the business changes, we need to understand that because otherwise what happens is you get into this scenario where it’s 3 years later and you have an outage and, you know, the boss who wasn’t even there comes and it’s like, well, why did this happen? You’re like, you don’t know, man, you weren’t there. And it’s like, who cares? You probably made the right— every fuck-up was the right decision at the time, right?
Bridget: You know, you took out that technical subprime ballooning mortgage for a good reason.
Matty: Yeah, but, but, but then to like continue to go la la la la la la la, that’s the mistake, right? You know, when, when it’s right. And I think Charity, I really agree with you when you said, you know, we can give this like, well, this is the right thing. There’s— I mean, we all kind of have a different perspective, you know, or we do because we’re 5 different people. Nobody should do exactly what we’re telling you to do, right? Unless you are Pete Cheslock and you work at ThreatStack and you came from all these other places and you’re solving the exact problem Pete’s trying to solve for, which is always a DNS problem.
Bridget: [00:27:39] Always.
Pete: Or actually, no, it’s a security problem now. It’s always a security— that’s the new one.
Matty: Gotta stay on brand. So, but I think what might be an interesting thing to kind of go to, to think about is like, okay, putting that in context, here’s your warning label, listeners. Take all of this with what makes sense to you. What are some good ideas, right? Like, we’ve already talked about a few. Like, you know, Pete kind of made this, you know, he’s like, hey, if you’re Pete, vendor all your shit, right? Maybe that’s not appropriate for me yet. So what are some other things we can do to think about protecting ourselves from someone else’s— and again, Charity, I know it’s not someone else’s fault— but a system I don’t have control over.
Charity: Yeah, affecting me totally. I think that for those— so the things that you guys were just saying, I think, all boil down to being able to exercise good technical judgment, which is a thing that requires not playing by a set of rules, but taking, like, some hard-won experiences and making really hard calls that honestly are never 100% right. You know, you’re going for, like, 75%, 80% right, like, move the company along, prioritize reliability, stability, all these things. But I feel like as you grow as an organization, the most important things that you can do to reduce operational risk is decentralizing accountability. I have this huge rant that I’m working up on, and it’s not going to come today, but it’s like, look, your operations engineers honestly are not responsible for your reliability. Your software engineers need to understand and own their services from end to end. You need to be looping in your ops people from your very first design meeting and saying, how does this scale? How do we instrument it? How are we going to understand it? How are we, the software engineers, going to understand it in the middle of the night when it pages us? And operations engineers are amazing spirit guides and career counselors for you in leveling up your skills. But we have to get away from this whole software engineers build the thing, toss it over the wall, ops engineers get paged, and then try to bring the feedback back to the software engineers about how to make their services more resilient because it’s too late. The feedback isn’t effective if it isn’t immediate and if it isn’t somewhat painful.
Bridget: [00:30:16] They try to bring the feedback of, so this query, if you liked it, maybe you should have put an index on it.
Charity: Just saying. Yeah, yeah. I feel like tightening feedback loops and making the people who are building the things responsible for keeping the things up and then focusing as a team on not how do we keep this thing from ever going down, but how do we make it so that whenever any component fails, the system limps along in a degraded state or some things don’t work, but it’s still performs all the functions that are not related to the specific thing that is failing.
Bridget: Good circuit breakers, good continuous partial failure states, that sort of thing. Exactly.
Matty: It doesn’t go wrong because of human intervention that went in there and edited a bunch of host files to point it somewhere else or doing whatever, right? Correct.
Pete: Because that doesn’t help. Correct. Definitely thinking about and planning for various ripcords that you can pull at different stages of events that are happening, that’s where— I mean, I’d love to say that I have 100% of all my packages stored locally and my build systems are pulling everything locally. It’s so hard to do that, and one of the things that we’ve even tried to do, and this is where I’m going to put my security spin on things, and I’m going to say the dirty S-word for security and why vendoring is actually important from that perspective. When we monitor our systems to see what they’re doing, because we do the DevOps, right? We give a lot of people broad access to the systems under the classic adage of developers write better code when they’re responsible for managing the systems of that code. But at the same time, we want to verify they’re doing proper things. When we find that systems are making a lot of calls out to the internet, you know, for various packages connecting arguably all over the place. We find connections to Southeast Asia and we find connections into Russia, and they’re okay connections because they are, you know, some apt repo being served up via anycast or something like that. But trying to understand and differentiate between if that is a good connection or not can be tricky. And so the main reason we’ve actually started vendoring a lot more things is really just to reduce all those random connections out so that we can now finally say, like, this connection that happened to Russia should never have happened because, you know, there are no packages in Russia that we should be getting. Not even the Russian packages, Pete?
Bridget: [00:32:57] What if they’re great?
Pete: That’s why we bring them local. They’re so great. I want to, I want to bring them in, bring them in under my, my wonderful umbrella. You know, so that’s kind of where we’ve done it from, like, a security perspective of just saying, you know, And for us, again, the security side of it’s important as a security company, but just understanding, like, why— we saw a connection out to China, and we literally were like, why would any system talk to China? And we spent a little time and tracked it down, and it was just like an app security update, and through the magic of DNS and internet routing, you just end up in China for one random connection. You know, to solve that, it’s like, all right, well, whatever package was up there, let’s, let’s pull it inside internally. And so it basically kind of reduces the scope of where things connect to, and we can still, you know, let our developers have access to systems and basically verify, you know, what they’re doing. And just, you know, it’s a visibility thing really.
Bridget: That’s definitely a better, better solution than trying to like, you know, fix all the BGP routing on the internet.
Matty: [00:34:02] So I think, in a serious perspective, besides fixing the internet, the challenge of what you’re talking about, Pete, is that you can overcorrect for that. You have to make sure that it’s still a fairly unintrusive process to do that. And as I, in my job, work with lots of enterprise, and so it’s sort of a thing where never assume internet access, whatever, and then it’ll go through this. Well, every single line of open source code ever here has to be reviewed and approved all the way up to the CIO. Great! Okay, so I haven’t protected a whole lot there because really what I’ve done is, again, I’ve just made it so that people who need to do that stuff are going to figure out some way to do it that you never see that I did it. You know, as Sasha famously said on the Ship Show a couple of years ago, she said, if you treat your developers like children— I’d say, if you treat your employees like children, they’re going to behave like children. She said, right? And as I said, you know, if you are making it so that it’s super hard for someone to do things the right way, they’re just gonna become a subject matter expert in doing it the wrong way.
Pete: [00:35:10] So you need to make— and they’ll find it like really well too. Like, they’ll find the most amazing ways to like, oh, like only port, you know, 80 is open outbound. It’s like, all right, let me route my like every single connection over port 80.
Matty: Right, back to my computer at home.
Charity: I just changed that /20 to a slash— whoops.
Bridget: I’m sitting here thinking about the MongoDB ports a couple startups ago where I was auditing the security groups, as one does, and I was like, what are these Comcast IPs that can talk to the prod MongoDB? Oh, someone put that in from their house. Well, this is super. I’m— it’s like, okay, our PCI scans didn’t catch this, but I’m like All right, this is fantastic.
Trevor: So we’ve kind of talked a bit about that we should vendor things. Where do we vendor them? How do we vendor them?
Pete: So let’s say you’re an Amazon user, I think, and you’re looking to kind of get started with storing your packages locally. Or Azure. Or Azure, sure. I guess— I don’t know, do they have an object store? This is my obligatory S3, some object store, is a brilliant place to start. You know, just, you could put it there pretty easily. You know, I had to— admittedly, I had to learn a lot of this stuff doing ThreatStack a couple years ago on just what tools are out there to even kind of vendor things. You know, we’re a Debian Ubuntu shop, so we use Debian packages, and there is some great, like, tools out there. There’s a tool called deb-s3, which is a Ruby gem that pushes packages to S3, and it can push them to public, you know, S3 buckets, private S3 buckets. It’s basically apt repos on the cheap. There are some tools called—
Charity: [00:36:53] Go ahead. You can totally make this part of your image build process. Anytime you install a package, you just wrap it in a function that also stashes it in S3, and then that’s your source.
Bridget: If you’re doing this as code in your CI system, which hopefully that’s where you’re building your images anyway— I mean, hopefully you’re not doing it ad hoc. Once you’ve done it ad hoc enough to know where you want it to be, and then you have your CI system building these images and installing these packages, then just add that as a step in your Jenkins job or whatever and go on with your life. You don’t want the 42-item checklist that some poor ops individual is trying to work their way through, and then they forgot number 32, and then everything is on fire.
Pete: That’s just going to make everybody sad. Yeah, definitely. The other thing, too, is that I always try to mention this, which is, There’s tons of open source tools, right? The classic line there is, open source is only free if your time is worth nothing. Sometimes my time is worth nothing, so I’m going to use open source and maybe learn a little bit about it from there. But there are tools that you can buy. You can buy your way out of this problem pretty easily. There’s Package Cloud, which is more recently doing some really awesome stuff with RubyGems and Debian and Red Hat packages. Artifactory is like the— it does and connects with everything. We use Artifactory at ThreatStack. It connects to everything, you know, Nexus, and I’m sure other people have thousands of other tools out there that you can use to stash like whatever kind of packages you have. And that’s—
Matty: [00:38:22] I know a cool one tweeted at us, and maybe we’ll share it with people because yeah, it seems like every customer I’ve gone to has their own special one that they bought that they’re like, oh, you’ve never heard of And I’m like, no, what does it do? And they’re like, it’s that. I’m like, great, okay, do your thing with that thing, right? And put the, you know, put the bits in the place, put the bits in the place, and let’s move on with our lives.
Bridget: Besides just a blob store for, you know, any data whatsoever, if you are using something like Docker, then you probably want to consider not just using public Docker Hub, but at least having some other option depending on what you’re needs are. Like, there’s ones you can run yourself, there’s ones other people can run, there’s S3-backed, you know, host-local, whatever. Like, there are a lot of choices, but I gotta say, I got paged more by Docker going down than our stuff going down. So like, when we had a dependency that would lead to me getting paged because of Docker Hub going down, like, I got paged because of that. So it’s probably, again, back to the— if you’re gonna own that availability, like, have a plan.
Charity: Yeah, I was just gonna say, like, if If the context of owning availability is about caching package files, which I only kind of just realized was the point of this podcast right now, that’s seriously one of the easiest problems in computer science. There’s just, like, 50 million ways to do it, and it’s not hard. If you decide it’s important for you, like, do it. Yeah.
Pete: [00:39:48] You know, one thing that we haven’t even gone down the path of, and I will tread on this subject lightly because I am far from an expert. On it. But we haven’t even talked about how do you verify the packages that you’re even getting are the packages that you’re getting. I’m gonna tread lightly on the words PGP.
Charity: I mean, come on, GPG is the best, and we all know exactly how to use it and never forget the, like, the flags, right? Right.
Pete: And I will say this, which is one of the best things about the 3 of the tools that I have used and currently use DebS3, which is like an easy little Ruby gem to push packages to S3, Artifactory, PackageCloud, is that all 3 of those have very easy ways for you to make signing packages easy. And I know the PackageCloud team has written extensively about basically how broken everything is around verifying packages are actually signed correctly. But, you know, you can look at it almost like by vendoring your packages and owning that side of it, you know, you can kind of further, you know, increase your security around understanding that these packages that you’ve, you’ve brought in locally, even if you have to build them locally, you know, maybe you don’t even get a Debian package or an apt or a yum, whatever package, you have to build it yourself.
Charity: [00:41:09] Really, that’s really great information. I didn’t really that the security features of those— I’ve never considered using any one of those services, and now I understand why I might. Every place I’ve ever been, I’ve ended up also building a lot of packages, which is why that’s always been the seed of my local repo is, well, I have to build a package for this and this and this. Well, OK, I’m just going to just start sucking in other packages too.
Matty: And I think that’s partially where people are starting to have challenges is that requirement to build. Where the competitive differentiator of Kmart— I don’t know why I just picked on Kmart— is not building awesome packaging tools. It’s the thing they do. Speaking as a vendor of things like this, and several of us are, that’s why the companies we work for exist and the things they do is to say, you do the thing you’re great at so you don’t have to build all the shit. And I think you have to like think about where does it make sense to, to take advantage of the stuff like that because versus— but then you’re going back down this road of now I’m trusting someone else. And I think that’s where people get really— they really get torn is it’s like, well, I don’t trust someone else to do it. I’m going to do it myself because then I own it, whether or not it’s because it’s a SaaS solution or even an in-house thing. But then, are you— how wrapped around the axle do you get around doing all that stuff when that same— and Charity, it goes back to your thing about where you are in the lifecycle of your company, right? Yeah. Well, you’re totally right.
Charity: [00:42:46] I build Debian packages because I’m a sick fuck who really enjoys it.
Bridget: But, I mean, yeah, and you might enjoy it, and you also might decide at some point that using tools someone else built is probably fine, especially if they’re open source and you can actually see what you’re getting. And that’s kind of the differentiator in my mind of like, if you’re going to say, I’m not going to create widget X because widget X already exists in the world, and I’m gonna, you know, not do the resume-driven development and not build widget X, like you probably still want to see what’s actually going on with widget X. Because when you go for the closed source thing, you’re like, its APIs seem to behave. I think it’s going to do what I want it to until it doesn’t. And then you’re like, okay, this really sucks.
Charity: If you’re running any client of anyone’s on your host for anything, for gathering metrics or whatever, like, if it’s not open source, that’s just not even— that’s a non-starter.
Trevor: That’s one of my favorite things to walk people through is, you know, I— because a lot of people I work with work much more traditionally with closed-source software, when I start walking through Chef cookbooks with them and they say, well, what is— what does this action do? I say, you tell me.
Pete: [00:43:55] And they go, oh.
Matty: And I think it’s, again, going back to besides of just like from a trust perspective of I want to look at your code so I make sure you’re not doing shitty stuff in your closed source software, it also goes back to the what happens, like it makes me be able to understand how it works. So if you do go out of business, vendor that I’m using, and I have to figure out some way to replace you, I don’t have any mystical private API crap that I don’t really know what it actually was doing. Right? Charity, to your point, if it’s instrumenting on my machine, I should understand everything it’s doing so that I can figure out, even if I’m not going to write a thing to replicate it, at least I can know, here’s all the things that thing did, even though I only cared about—
Charity: You get a stack trace, you need to be able to look at the fucking code to see what is generating the stack trace. I mean, this is not actually rocket science.
Bridget: And then you get to be that really lonely person who Googles the error message and all you find on the internet is the source code that generates it. You’re like, this is super.
Charity: But it’s better than not being able to find the source code on the internet, right?
Bridget: [00:44:58] This is exactly how I ended up digging myself out of an Amazon-induced HBase sadness. I was like, oh, so what you’re saying is I just need to backport this fix that doesn’t exist yet in Amazon’s AMI for Hadoop clusters. Okay, at least it’s open source, so I can. It’s horrible, but I can.
Pete: You find the mailing list article of you asking that question from like eight years ago, and no one responded.
Trevor: As always, there’s an XKCD for this. So XKCD 979 talks of Denver Coder, and what did they learn?
Charity: Or even worse, you search for the problem, and you find you providing the question and/or the answer from a couple of years ago, and you’re like, mother. Fucker. That was— I answered my own question 2 years ago, and I killed those brain cells with whiskey, probably.
Matty: So I think too, like, what we can, you know, again, thinking from a vendor perspective as well, is the things that we can do. So software vendors who are listening or are on the show, not that we have influence, right, is figuring out how we can make it so that it’s easier for people to not get completely fucked by dependencies that we create for them, right? So things like, I know just in general, it’s something like within Chef, like it’s sort of the problem when you’re starting out as a startup, right? You are a developer at some software company that’s a, you know, that is not in the enterprise world, and you’re like, great, I have unrestricted access to the internet, so when I build my installer, of course it can go out and do all the things, but need to think, A, about the fact that your customers may not even have that ability, but also, is that even the right thing, right? Are you creating this monster? Then also thinking about how you can make it so that, yes, you may have these wonderful as-a-service things you’re providing, but it actually may be valuable for your— your customer may want to own the availability of that. Things like— again, I’m just sort of giving a chef example because I’m aware of it, right? But, like, okay, Supermarket’s open source. I have lots of customers who run their own internal Supermarket, and it’s not to mirror it. It’s not— sometimes it’ll do to that, but then they’re like, this way, we— like, Pete, to your point, I know what those packages are, you know, or I know what those cookbooks are, and we’ve vetted them. They’re okay, and I’m not— and so if Supermarket takes an outage, I don’t care, right? And then the same thing with if I’m running, you know, if I’m using package clouds or something similar, something that I can do if I need to have it inside my firewall, But I can still have all the goodness of it and not, like, have it be this clunky, you know, IBM thing or whatever.
Charity: [00:47:43] One mistake that I have repeatedly made is shifting the source of truth from some repo that I don’t trust that keeps going down to GitHub. And then you have 2 problems.
Bridget: Well, that’s a really good point, too, is, like, okay, if you’re talking about the Docker registry, or if you’re talking about, you know, GitHub, or if you’re talking about, you know, like you’re writing some Go and you’re like, okay, well, obviously I’m just gonna point to this over here. And it’s like, okay, all of— we have to have a plan for when all of those things are unavailable. Like, if you can’t deploy software because GitHub is down, it— like, you have problems.
Pete: Yeah, so one thing I think, Matt, you were kind of saying it before a little bit, or you were kind of touching on it, So I had a conversation recently with one of our engineers, and we were basically talking about monitoring plugins. We use Sensu, so Sensu plugins, plugins that can do host-based checks and do metrics for systems and fire metrics onto Graphite. Basically, the conversation we were having— I’m a proponent of, like, we pull everything local. Again, if it’s a Sensu check or metric, it lives in a cookbook somewhere, so we can drop it on the host. And he was like, oh, no. They have this new setup. You can just gem install. You know, the Sensu plugins. And I was like, that’s a little weird. And we started looking into it and I was like, yeah, I don’t like that. Just grab it and bring it locally and render it. And he was just like, well, that seems kind of against the open source model. And I was like, you know, the best way I could describe it to him, and I even look at this as a bit of like own your availability, and it’s like owning your dependencies as well because it wasn’t that I was afraid of couldn’t gem install whatever. I mean, there’s ways to solve that one. I— the ultimate fear, and Charity, you kind of said this, is like, if that script program, whatever, doesn’t work and you were just gem installing it, you’re not even looking at the code, you know, you don’t own it, so you don’t understand how it works, how to debug it, how to read error logs and stack traces and things like that. But by you bringing it in, hopefully it encourages you to read through the code and understand what it does. And how is it— like, in one example, how is it talking to Graphite? How is it grabbing data from the API? How is it working? Because, of course, what literally happened, like, a week later was that there was a whole segment of the code that just wasn’t working as expected. Turned out to be completely dead code that should go away, and it shouldn’t be even in there in the first place. But the thing is, is, like, if you were just gem installing and running it, like, you would have ran into that and probably would have been far more confused and having to dive into, you know, whatever. I kind of look at it more of even just not only owning your availability and your dependencies from just a safety world, but just from an understanding, like, it’s your code, you’re bringing it inside because it’s going to do your thing, and own it from that perspective.
Bridget: [00:50:36] I’m a really big fan of having a pretty good idea of what things look like when they’re working correctly, since God only knows that when shit’s on fire, yo, and it’s 3 in the morning, and as Charity has put it before, you know, it’s 3 AM, suddenly you’re an expert at Redis because fuck. Like, when that’s going on, you don’t want to also try to figure out, is X, Y, and Z normal? Does the process table always look like this? Like, you would like a better picture of what normal is as your baseline before you’re trying to figure out what’s going wrong. So having some amount of understanding of what you’re actually running before everything is on fire, or at least before everything is really visibly broken, is, I think, a really good place to start.
Pete: Yeah, I mean, every time you pull in a dependency, you’re basically adding risk into your world. Some organizations, that’s fine, actually. You can absorb that risk. That’s okay because of the stage where you’re at. You need to move at a faster speed than worrying about that risk level. There are other companies that are the complete opposite, which is like, And also just personal kind of world, like you said, if I get woken up at 3:00 AM, I want to know how the damn code works. Like, so I’m gonna understand this before it goes out.
Matty: [00:51:50] I wanted to think a little bit too about, for, you know, folks when you might be in a more traditional enterprise or just a larger organization where you don’t— you’re not necessarily in a position to make some of these calls yourself. And I know that applies to a lot of our listeners. When you might have management above you who loves the ability— the idea to have someone to blame, right? Who’s saying, oh, no, actually, I want to be able to say that this software vendor totally fucked it, right? And then, so what are some things we could think about to help people manage up to say, like, no, you actually don’t— because, like, we’re making— we’re saying this is why it makes sense, right? You want your shit open. You want to use open stuff so you can understand it. And then I think what I see sometimes is the, well, no, I don’t want you spending time understanding it. I just want a thing that works, and if it doesn’t work, then I can go sue the crap out of IBM because they took down my, you know, like, you know, widget factory. So what are some things we might be able to, like, again, help our listeners who are stuck with that, like, help communicate this?
Pete: [00:52:51] So I guess if you’re a person who really wants to kind of own your availability with your packaging or your dependencies, but maybe your upper management doesn’t want to for some reason, and I don’t want— you know, sometimes you got to play the game, right? You got to say the right words. So if you’re really trying to get this done, I think the easiest way is to just say it’s a security issue and maybe go and get the head of security to join you in that one because you just basically find the biggest curmudgeon possible to come over and be like, no, it’s a security issue, and start hammering on, like, you’re out of PCI scope!
Matty: How many security people in enterprises are going to be in favor of open software, Pete, actually? I think that’s the first part.
Pete: No, no, I’m saying bringing it in.
Bridget: Vendering it, sure. I mean, we also, like, OpenSSL, the gift that keeps on giving, is the joke I think I’ve seen going around Twitter. Like, it’s not that code being open source makes it automagically more secure. It’s more that there’s a better chance of somebody who isn’t that specific original vendor finding the problem, right?
Matty: [00:53:53] It’s back to Charity’s thing about saying, I want to make sure I really understand stuff, right? And so the problem is sometimes you’ll have the folks who are helping you, you know, kind of in charge of what you do with your resource, saying, no, why are you troubleshooting that? Why don’t we just buy something that just does that? And we will have an SLA with them and then F them if they screw it up, right? We know that doesn’t work because SLAs are like the lawyer’s favorite thing to ever say, right? Because they’re all made up and they never apply.
Charity: Like, again, the thing, the thing that I want to bring up here is that It’s about— how do I boil this down? It’s about different audiences. It’s about having to report about failures and successes to different audiences. Like when you’re postmorteming a developer platform to developers, the way that you build trust with your audience is giving them the dirty details and taking ownership Being like, this is what happened. This is what we’re doing to mitigate it so it doesn’t happen again. This is what we are— this is what makes you trust us. When you’re talking with enterprise leaders who are up the stack, I think that the most effective— and I am a terrible person to really ask for— I really fuck with big orgs. What helps me form empathy with them is realizing that their audience is, is different. Their audience doesn’t understand and doesn’t want to understand software. And so the reason that paying money for a thing, having an SLA for a thing, having it be intermediated by lawyers is because they have to report to higher-ups or customers, like customers who are like, you know, Kmart customers, right, who don’t give a shit about the technology. They just want the tech to work. And so if you want to get the CIOs and like the executives of big corporate environments on your side, you have to understand, you have to explain it to them in a way so that they can explain why it matters to people who have the same context as you. You have to explain it in terms of, like, you know, just like Pete was saying, just paying money for this and asking for an SLA is not going to make it better. It is not going to make us have fewer outages. It is probably going to make us have more outages. We won’t be able to debug them. We won’t be able to understand them. We will be waiting on someone in another company to track this down and and trickle the fixes back to us. So, like, you can translate that to, like, corporate-ese and legal-ese and CEO-ese or whatever, but I’m telling you this will build us a better product if we do it this way.
Bridget: [00:56:51] That’s, that’s a really good way to put it. It’s like, it’s not that you need to reinvent the universe from scratch every time you’re gonna bake an apple pie or whatever, but you do have to make thoughtful decisions about your tech and thoughtful decisions about how you’re going to compose all of the underlying, you know, substrata of where your dependencies lie because—
Charity: and empower the people that you have hired to actually fix the things you hired them to understand and fix.
Bridget: Totally. Exactly. Pete, I’m sure you have thoughts on this too. I would actually— I’m going to turn to Pete and then back to Charity for just some kind of closing wrap-up thoughts on the whole who owns your availability, just because we’re getting near the top of the hour and we’re going to have to unfortunately let folks go. So Pete, just starting with you.
Matty: Oh, go ahead, Trevor.
Trevor: I was going to say maybe this answer will be just as succinct as it was when we started. Every other word maybe?
Pete: Yeah, no, I mean, this is awesome. I mean, it’s definitely a great conversation and it still opens my eyes too to how everyone’s companies are different, everyone’s requirements are different. What I’m trying to build is different than what you’re trying to build, and my risk tolerance is different than other people’s risk tolerance, and my budget is different, too. That’s a really major part. I think that’s the one thing we have to remember for everyone is that we’re all at different stages, and a lot of times, leveraging GitHub as your source control for your packaging and everything else, We do it via necessity and we do it because purely I don’t have the money, like my business doesn’t have the money. That’s okay. I think at the end of the day, taking the steps that are right for your company, for your job, and you don’t necessarily have to do everything overnight. I mean, we’ve been working for years to try to go from building everything and pulling from every place on the internet to to bringing things locally. But every time we do it, we try to do it kind of in a rational way and do it for specific reasons instead of just blindly doing it. So I think that’s the ultimate thing is, you know, you don’t have to listen to what talking heads on the internet say about you should vendor every dependency you can find. That’s a bit more hyperbole than the reality of the situation, which is, you know, think about it smartly. And just as you are building systems and and finding kind of cracks in the mortar, use that opportunity to say, hey, would this make things a little bit more durable or reliable if we just brought this dependency in locally?
Bridget: [00:59:31] Nice. Awesome. Charity, who owns your availability?
Charity: I want to echo what Pete said about making good technical decisions is always context-dependent. It always requires good judgment. You have to know where you’re going. You have to know where you want to go, even if you’re not there yet, even if you’re compromising along the way. But when it comes to who owns your availability, it’s you, but not you personally. Like, you’ve built a successful org and a successful company and a successful culture if every single person that you work with feels and internalizes that they own it, right? That it’s not some other team’s problem or, you know, some other software engineer’s problem or the DBA’s problem or a vendor’s problem or a, you know, whatever. It’s like, yeah, we’re going to fuck things up. Cool. Accepted. But, like, this is on us. We care about it. We all care about it. And it’s the job of every single one of us to make this service reliable. And durable and as good as it can be.
Bridget: [01:00:43] That’s awesome. I love that idea of the shared responsibility. Because again, and this is a nice subtlety, I guess, in English, is you singular or you plural? And the answer is yes, all of the above. Yes, yes, totally. Nice. Okay, sweet. So community event stuff. Stratton, I know you had a bunch of DevOps Days that you signed us up for as a sponsors, so you want to mention a couple of those?
Matty: Absolutely. So DevOps Days Rockies is April 21st through the 22nd. You can save 10% off the price with a discount code of ADO2016. By the way, you can probably take a gander, and ADO2016 will probably get you a discount at almost any DevOps Days, but what discount will vary. For example, Atlanta is April 26th through the 27th, and they’ll give you 20% off. Seattle is May 12th to the 13th, and they give you 15% off. So remember, the DevOps Days discount code is ADO2016. It also works for Minneapolis, and I just realized I don’t have this on here yet.
Bridget: [01:01:45] So I don’t even know what percentage it works for.
Matty: I don’t know what it is, but it’s something.
Bridget: You should register. You should come.
Matty: It’ll be great. Next year— we’re sorry, this year we got all the DevOps Days to at least use the same code for us. Next year I’m gonna push for the same amount because it’s crazy. We’ve got a bunch of open CFPs as well. So DevOps Days Vancouver and Minneapolis and Abstractions are open till March 31st, which, you know, depending on when you’re listening to this, this may still be available or not. Washington— DevOps Days Washington, D.C. is until April 15th. Haha, I love that. Salt Lake City is until April 19th. Amsterdam until May 30th. The CFP for that conference will also be open until March 31st. We also have t-shirts now and mugs. If you go to store.arresteddevops.com, you can buy them. Right now, we just have the unisex shirts, but actually, I just ordered some proofs of fitted shirts for Bridget, so she’s going to decide which ones we’re going to have.
Bridget: [01:02:47] I like trying the shirt on and seeing if it is only for 12-year-olds or for actually me-shaped people before I recommend to people that they buy it.
Matty: Absolutely.
Charity: Absolutely. If you need any beta testing, I’m here for you. Oh, yeah. Awesome.
Matty: Absolutely. We will send you stuff. And if you are not on the show and you’re just listening, you know, buy yourself a shirt or a mug, you know, or don’t. We’re not the boss of you. So, but we do have a newsletter we’d love for you to sign up for at arresteddevops.com/banana-stand. I remember I remember to send it out at least once a month, usually, sometimes. It’s a great way to know about upcoming podcast episodes and other random Arrested DevOps news.
Trevor: Thanks again to our sponsors. Be sure to visit them at arresteddevops.com/10thmagnitude and arresteddevops.com/datadog. Pete, Charity, thank you so much for joining us on such short notice on such a great subject.
Pete: Yeah, this was awesome. Always glad to rant about packaging.
Charity: Always about basically anything. Thanks for having me.
Matty: [01:03:47] If you’ve enjoyed the rants, please go to arresteddevops.com/itunes and leave us a review in the iTunes Store. It actually really does help us. And I know Software Defined Talk totally made fun of us, even though they didn’t call us out as us for asking for reviews, but whatever. And I think all the other—
Bridget: they weren’t talking about us, I don’t think. Like, I don’t think they were talking about us. Because they actually were really excited when I left them a review and they read part of it. They mentioned it at least on air.
Matty: I think they were more making fun. I think the idea in this episode, and I’ll try to remember to put it in the show notes, it was actually very funny, is I think it’s Matt Ray that came up with the idea of saying he was going to have a podcast that was called Please Leave Me a Review on iTunes and that’s all that was going to be on the podcast episode after episode.
Bridget: They had one of their episodes was called Leave Us a 5-Star iTunes Review. Review or something.
Matty: That was this one, yeah. I like it because at the end, Cote says, I met that Matt Stratton guy, or I was hanging out with that Matt Stratton guy at DevOps Days Detroit. He’s a class act.
Bridget: But of course, Cote said it in his, like, he’s a class act, like, you know, the warm Cote voice.
Matty: [01:04:54] Yes, the stentorian tones. It gets me through every flight.
Bridget: I save up all of the Software Defined Talk and Lords of Computing and the Pivotal Conversations stuff that Kote does, and I just listen to them on flights. They’re so relaxing. Okay, so, and yes, we would, we would like to know what you thought of this episode, so please leave us comments at arresteddevops.com/availability, and Stratton will definitely read them.
Matty: Yeah, I absolutely will, and then I will share them in Slack with my co-hosts. Anyway, yeah, we’re also on Twitter @ArrestedDevOps, so like, talk to us and stuff. We love it. We have lots of followers and we engage with you. And I don’t mean that in a weird brandy way. I mean, like, we talk to you. At least I do. I’ll put it this way. If you talk to @ArrestedDevOps on Twitter, you’re probably talking to me.
Bridget: I think he shared the credentials to me, but… have not pursued. They’re all in our Slack.
Matty: It’s all ChatOps-yed up anyway, whatever.
Bridget: If you got credentials in Slack, right? That’s where you keep credentials, right?
Matty: [01:05:55] I never would do something like that.
Bridget: Chesslock, that’s where we keep credentials, right?
Trevor: You know, it’s really funny to me, Matt, you said that, uh, you said that the end of the episodes were getting stale, and this has been like the most ad-libbed episode ending I think we’ve ever had.
Matty: That was— well, I rewrote it. This is all scripted, by the way. So let us know if you have ideas for future episodes or ways in which we could improve the ends of our episodes.
Trevor: I’m Matt at Matt Stratton, and I’m Trevor at Trevor G Hess, and I’m Bridget at Bridget Kromhout.
Matty: We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand. Whatever how the music goes.
