Matty: [00:00:06] It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matty Stratton. Today we are going to dig into a topic that everybody loves to talk about, which is security. You know, without security, we are Insecure. But before we talk about my own insecurities, let’s hear from our sponsors. Got robots writing your code? Great. But you’re going to need a safe place to run it. Fly Machines are fast-launching, hardware-isolated VMs that spin up in milliseconds. Perfect for sandboxing even the sketchiest LLM-generated code. Dynamic subdomain routing and zero-cost scaling to idle? Yep, got those. Robots break things. Fly machines restart cleanly and scale effortlessly. Fly.io is built for humans and loved by robots. Check out arresteddevops.com/fly and make your robots happy. All right, joining me on the pod today, I have returning guest Kat Cosgrove. Kat, welcome back to Arrested DevOps.
Kat: [00:01:18] Howdy, Matty. It’s lovely to be back. It’s been, it’s been a hot minute since I’ve done one of these. Couple years at least.
Matty: It has. I was trying to remember all the different episodes we’ve done. And if you go to arrestedevops.com, you can figure that out. But I know we, we talked about deserted island DevOps in the past. We talked about the social network that has been renamed to the, the new name. So previously known as Twitter. There was a whole episode about that, but security, right? You know, and I think specifically maybe we’ll be thinking about vulnerabilities and how we respond to them. And, you know, maybe in the, container and kube space, but it’s a broad topic, but we can cover it broadly or less broad as it might be. I don’t know. I’m so good at this. I’m so good at this.
Kat: You’re killing it. You’re killing it. Yeah. Security is like a never not hot topic. I think the more of our lives and money and livelihoods and government functions are tied up in computers, the more anxious people are about security and the more dangerous vulnerabilities can, can get. The, the days of the hacks and worms and Trojans and vulnerabilities that we saw in the ’90s, that, that seems charming compared to how dangerous things are today. You know, it’s not just like, I don’t know, a chain email going around that crashes your computer anymore. It’s something that shuts down the entire planet nearly, like with the, with the CrowdStrike outage, which was, you know, a bug, not an actual vulnerability, but things like Meltdown and Spectre and Heartbleed seem terrifying today compared to like how goofy things were in the ’90s. The names get more terrifying too.
Matty: [00:03:07] Yeah, I was going to say even earlier, because if you think about kind of what’s generally considered the first real worm, was the RTM, Robert Morris’s virus, which was kind of an accidental virus, which I want to say was in the ’80s. But what’s fascinating about it was at the time there was like, what, 100 computers connected to the internet or whatever it was. So it was relatively straightforward, but even then it wasn’t, you know, so it’s funny, you read the stories about it and it’s like all these sysops are getting on conference calls to try to solve the problem because they’re the only people, but it was only affecting them. Then you think about the much larger scale that, that can come in. And also, what was the impact, right? Because again, it was like research institutions and schools, which are important, but it’s not that necessarily people weren’t able, you know, millions of people couldn’t conduct their day-to-day lives, which, you know, the whole, whole of the internet and everything is held together with, you know, baling wire and good intentions. You know, it doesn’t take much. So you talked also, you mentioned, you know, kind of those things and, you know, The impact is much, much bigger now. And I would imagine it also becomes challenging because of dependencies and complexity. So when I start again thinking about in the ’90s, in the time, you know, our systems weren’t as complex. And it was, you know, fond of saying like there was a time you could hold your entire stack in your head and you knew what all the things were. But we run into, and this was, you know, you know, we did an old episode with Pete Cheslock and Charity Majors called Who Owns Your availability. And this was after LeftPad, but it’s, you know, the dependency upon dependency upon dependency. So when there’s vulnerabilities in a package, you may not even be aware you’re using that package or that thing because it’s 17 layers below the thing that you’re using. You’re like, I just, you know, and that was kind of the thing. How much LeftPad’s a great example of that.
Kat: [00:05:05] It’s really hard to like peel through the layers of something to find. You have no idea. Like think about the, like the Equifax hack. Right? That was because of an outdated, wildly outdated version of Apache Struts. And like, yes, they should have updated that version of Apache Struts. But with hindsight being 20/20 and all of us being like armchair computer science doctorates, like we can imagine that we would not make that same mistake, that, you know, that would be a thing that’s easy to keep track of. But It happened to Equifax and it could happen to you because you may not know that you have a wildly outdated package chilling out 3 dependencies deep. Like, and that’s how we ended up with, you know, things like software bill of materials and 85,000 pieces of security tooling and every conference and their mother hard focusing on nothing but security for content for a couple of years because of stuff like that. It was really, really splashy. It destroyed a lot of people’s lives. I have credit monitoring for the rest of my life because of that. So that’s, that’s, that’s pretty cool. Thanks, I guess. But yeah, like back in the day, these kinds of attacks were like, they certainly were destructive, right? Like the attacks were novel. They didn’t happen as frequently. People didn’t have a good understanding of how these attacks happened at first. We were slower to respond to them. We didn’t have really ways to prevent them. And now they’re significantly scarier. We’re more used to it happening. We’re more defensive rather than just being reactive to an attack. But the amount of damage that can be done is absolutely wild these days versus 1989.
Matty: [00:07:03] Yeah. And there’s kind of 2 pieces because I was just thinking about when you’re talking about knowing what you have, because that’s how you can, you know, and And that’s not, you know, SBOMs and things like that help with that. But that’s still always going to be responsive, right? And you’re like, oh, there was a vulnerability. Now I can get to that. But we always want to be as proactive as we can, you know, which again, we’re not always going to be able to be. But I think about things like Dependabot. Dependabot are like, they’re going to let you know when there are known vulnerabilities in a thing that you have, which is actually a capability of being somewhat proactive. There may have been a zero-day on it. You may already have been compromised because of it, because it depends on how quickly that happens. But that’s a pretty proactive way to connect to that, but then also how often do we just pull in so much stuff?
Kat: Like all the time, like constantly.
Matty: Yeah, which we need almost none of, right?
Kat: Yeah. And big companies spend, I mean, any company really with any software company, but especially big companies, they spend an absolutely outrageous amount of time and money just mitigating CVEs. That’s pretty much, they’ll have entire teams that just do that. And do they mitigate all of them? No, absolutely not. They get like the super high priority ones. They get the very highs and maybe the highs. But if it’s going to take more effort than is worth it for the like potential exploit, they’re gonna just leave it there, which is maybe the decision that somebody at Equifax made about Struts. That’s, that’s speculation. And I have no idea what was going on internal to Equifax at the time, but Yeah, most software companies do not mitigate the majority of CVEs in their dependencies. They just leave them chilling there because it’s often just not really worth it versus the risk, whether that’s because it’s a vulnerability that’s particularly difficult to exploit or it’s a vulnerability that doesn’t allow somebody to do a ton of damage. Most CVEs just get left chilling there. So if you think that your software is secure, it’s, it’s probably not. And nothing, nothing is immune. Kubernetes had a pretty hot 9.7, 9.8 a couple months ago at the time of recording. Vulnerability in Ingress NGINX, which affected 38% of cloud environments. So, that was super fun to deal with from a maintainer perspective.
Matty: [00:09:38] So, one of the things that I kind of wonder about, and, like, there’s lots of problems, right? I’m not looking for all the answers, but I’ll run into this in a small place, and I’m sure this happens in my own projects and stuff, and so, I’m sure this is only exacerbated at scale, is when you’ll sit there and you have a dependency for this thing you’re using, And it’s got high, you know, there’s a CVE on one of its dependencies or something like that. And that’s some shit that hasn’t been updated in 5 years. It’s some little package that somebody did whatever with.
Kat: Oh, we’re on the XKCD comic now where it’s just like one guy. Yeah.
Matty: Yeah. Except that this guy, the guy that’s maintaining it doesn’t even do anything with anymore. And which is fine. It can happen. And then I guess like, it’s like, what do you do? I guess you need to like, at that point, do you need to fork that thing? And then like now you have to maintain it yourself and yeah, private fork.
Kat: You fix the problem yourself. It’s open source, baby. PRs welcome, except when the guy who is the one admin for the repo goes AWOL and now the PR is gonna sit there stale. So yeah, you fork it and fix it yourself and use your own forked dependency. And then if you leave that thing public, then God help you. Maybe other people see that you did that and now you are the new XKCD comic guy. And that sucks. Like, I’m a big fan of open source. I think open source is the best and most secure way to build software. But I do have to admit that it has some downsides. And that’s one of them. That’s one of them for sure. The ease with which people find vulnerabilities in open source software is also like, it’s kind of a double-edged sword. They get found more quickly, so they get addressed more quickly, but also they get found more quickly, so they get exploited more quickly, right? And that’s a pretty big downside. But as a community, we do have the benefit of having a veritable army of very dedicated weird nerds, myself included, willing to fix these things fast, right? That is something that proprietary software can’t have. You just simply cannot move as quickly with a limited number of people and limited visibility into what’s going on in your codebase. So open source is better. Don’t let the vulnerabilities and the Dependabot alarms scare you off, but you are going to get quite a lot of alerts and most of them you’re not going to be able to mitigate yourself. So that’s, that’s, that’s fun. It is what it is. We’ve got a lot of new cool security tooling on the market, a lot of new cool observability and monitoring tools on the market that make it easier to tell when something might or is or has gone wrong so that you can, you can respond more quickly. But obviously we prefer to prevent these things from happening at all rather than just responding to a big, big piece of drama that is at best publicly very embarrassing for you and your, your company and at worst destructive to your livelihood.
Matty: [00:12:50] So, you just referred to, you said, you know, there’s a bunch of tools. What are some of the things that you feel like should be in everybody’s kit? And maybe there’s multiple, you know, there might be like 3 tools that do this kind of thing. But if you’re sitting there and saying, you have to have this, if you don’t have this, hit pause on the podcast, go install this in your production environment, and come back and listen to the rest of the show.
Kat: You absolutely do need some kind of observability and monitoring tool. Like, you need to be using Something like OpenTelemetry, something like Prometheus. You need, I mean, which one of those types of tools you use depends on what kind of environment you’re working in. Obviously, I’m a cloud-native person, so I’m going to default to cloud-native tooling. But you cannot just be like raw-dogging production. You have to know what is going on in there at all times. You have to know when you’re getting weird traffic from a weird location hitting a weird endpoint. A lot, like that kind of thing can be indicative of something very wrong going on, whether it is an attack or just like a bug, just some— something farting, a container falling over. You should not be blind to what’s going on in your environments. Similarly, with alerts, you don’t want to firehose it. You need something intelligent enough to make those alerts useful rather than just bombarding your engineers or your DevOps team or whoever is getting your alerts. With a whole bunch of nonsense that could actually be nothing. So, it’s gotta be something intelligent enough to make some of those decisions for you. You do need some kind of dependency management situation. You need to know what your dependencies are and what your dependencies’ dependencies are. Cannot be just raw-dogging those either. Can I say raw-dogging on this podcast? I’ve said it 3 times.
Matty: [00:14:42] You apparently have multiple times. So, you are capable of doing it.
Kat: We’re going with it, because that CBS anchor said raw dogging on the air.
Matty: So my question is, is raw— is the use of the term raw dogging sufficient to merit the explicit tag on this episode?
Kat: I think not. I think if you can say it on CBS, you can say it on a podcast.
Matty: But if we were to say fuck, then I’m gonna have to make this tag.
Kat: Well, you just said fuck, so there you go.
Matty: So there we go. And we’ve said it twice because we violated the PG-13.
Kat: Yeah, no, it’s not PG-13 anymore.
Matty: Okay, well, now we know that this episode That won’t be the first, first time either. It won’t.
Kat: I probably cussed in the last ones too, so it’s fine.
Matty: In the pre-show recording with guests, Bridget would say, we have made our peace with the explicit tag, so feel free to speak as you would like to. Oh, which amazingly I didn’t actually say to you, which I normally would have.
Kat: So, you know, I do usually ask. So that’s, that’s my bad before I go on a podcast anyway. Don’t raw dog your, your production environment. Know what’s going on. I think that most pieces of DevOps tooling, like, do have a security angle. Your infrastructure as code tool has security implications, and you should be using an infrastructure as code tool. You should also, you should not be, like, hand rolling your AWS deployments through clicking in the UI. That’s outrageous and miserable, and I think you hate yourself if you do that. I try. Not to talk about my own employers on podcasts, but I do work for a company now that solves exactly this problem. I work for Minimus. I’m their head of developer advocacy. So here’s me advocating, I guess. We build very small container images with— most of them have zero CVEs. That is a super useful way to start your application environment with no CVEs. Obviously, we can’t do anything about vulnerabilities that you introduce through your own like maybe questionable code. My code is certainly questionable, but your base image will have 1 or 2 or none CVEs in them. And they do get automatically rebuilt when there’s an update, which is pretty cool. So that’s a good way to start from zero. You should be using some kind of like software bill of materials. You need to know. We’re all tired of hearing about SBOMs. I know we are, ’cause it was like all we talked about at KubeCon for what feels like 3 years. And it has finally cooled off, but they, they are still important, especially as more and more of our lives are sucked up by computer. I, I find myself intentionally seeking out hobbies that do not require computer because I’m so tired of computer.
Matty: [00:17:37] Personally, that’s actually like really, number one, like a really good and healthy thing. But it’s funny, I remember having this conversation with Pete Cheslock years ago where we were just like at the bar and whatever, and he— something came up about him playing piano and how he was learning to play piano, and he’d been playing piano for like a year. And I was like, I had no idea, you know, that’s really cool. And he was like, well, I don’t talk about it on social media, it’s not computer related. But that was part of— part of why he did it was to say, hey, I want to do something that’s not like involved in tech as a hobby, but also sort of peripherally was like, and that’s also not part of how I talk on social then. I’m not, I’m not talking about it or whatever. And I was like, oh, that’s, you know, I talk about it like in person, but it’s separating that and it’s challenging, but it’s a good idea. And I feel like I, I tried to do that. I think I am doing better because I ran, I had a self-discovery thing. I think we’ve all been down this path when you look at your hobbies and you’re like, all of my hobbies are just a reframing of work.
Kat: [00:18:40] Yeah.
Matty: You know, you’re like, and I think, I think I really became apparent of this years ago when I was, you know, like on the dating scene and like when people like, what do you do? What’s your hobbies? I’m like, well, I have a podcast. What’s your podcast about? Tech. Well, I build this website. I run a conference. I do whatever.
Kat: Oh, Matt, that’s embarrassing.
Matty: It was embarrassing.
Kat: Good Lord.
Matty: I don’t know how, how I got past the first date with Steph then, but it worked out. Yeah.
Kat: She’s much, she’s very patient. She’s a, She’s a saint, I think. But like security is a security and computers are a burnout factory. So if you are listening to this and you don’t have a hobby that doesn’t involve computer, I am begging you to literally not, not figuratively. I’m not trying to be snarky. Literally go outside and touch grass. Like, please go touch some grass or moss.
Matty: Even if it’s raining, you can dry your hands off.
Kat: You can, you can. I went on an 11-mile hike last weekend in the rain. It was lovely. I recommend it. My thighs hurt afterwards because it was uphill for the first half. And by uphill, I mean up mountain, but real big hill, up very big rocky hill. Yeah. Yeah. Touch some grass. Especially if you are on somebody’s security team on the mitigating CVEs treadmill, which is like the much less fun version of the loot treadmill in video games like Diablo. We like that kind of treadmill, right? That’s a good treadmill.
Matty: [00:20:19] What treadmill?
Kat: Loot treadmill.
Matty: Loot treadmill. I thought you said loop.
Kat: No, no, no, no. Loot treadmill. Games like Diablo and Borderlands get called loot treadmills because after a certain point, there’s absolutely nothing to do in this game but farm for more loot. And for whatever reason, that feels like super rewarding in both of those games. And it’s like somehow worth it to spend 30 hours grinding for a sword shaped like a cheeseburger in Diablo 3. I don’t know how they managed to make that fun, but that is fun.
Matty: But doing it, doing security looting— or not security looting, maybe security looting is fun. I don’t know.
Kat: Oh no, that’s called crime.
Matty: Matthew.
Kat: That’s crime. You invented crime there. You invented hacking, which is fine, but don’t get caught and don’t commit more than one crime at a time.
Matty: It just made me think of The Good Place. But all I hear in my head is there’s a scene where Jason is working on trying to get his dance troupe going and he had told them all, he’s like, no, we don’t do crime. We’re doing the dance troupe or whatever. And then after, you know, how many iterations of failing at it, he goes, all right, well, I guess it’s not gonna work.
Kat: [00:21:31] We’re doing crime.
Matty: Yeah, go do crime.
Kat: Go do crime. You know, I— look, I’m not going to advocate for crime. However, you know, live a little. Is that publishable? Fuck it.
Matty: To say live a little, I think that’s safe. You’re not implying necessarily that living requires the doing of illegal things. You know, we are not providing legal advice on this podcast. And if you come to this podcast for legal advice on how to live your life, then God help you. You need to touch a lot more grass.
Kat: Yeah, honestly, you need to fully log off, actually, especially if you know like anything at all about me. Do not take life advice from me.
Matty: Don’t do that. I think if you are taking your life advice from Kat, then you need to Ron Swanson that and just throw your computer in the dumpster. And pretend that they never existed.
Kat: I wish rage rooms would come back. I do. There’s not one here. I looked recently. There was one in Glasgow.
Matty: My friend Jamie, who is Glitter Burrito on Bluesky and I think other places. So she, she’s in Georgia somewhere, I think. But I remember she posted something relatively recently about being in a rage room, and it was actually very cool because you’re like, The pictures, it’s like full, not hazmat suit, but like big protective suit with the goggles and the giant sledgehammer and breaking shit, you know?
Kat: [00:22:54] So, you know how some conferences have like therapy dogs? KubeCon has done this before. I was just at RSA, unfortunately, God help me. And somebody had baby goats there. I think that there is a real opportunity for some conference or some vendor who wants to spend a lot of money on a big-ass goddamn booth to do a miniature rage room. There’s probably like some liability issues there, but I think it’s worth exploring because surely I’m not the only one at a conference burnt out on computer who wants to take a baseball bat to a rack. Like you can buy these things for cheap.
Matty: That would be the updated cloud-native version of that scene in Office Space when they’re like going full like execution mode on the printer.
Kat: Yeah, it’s that. It’s that I want like I want that, but I want like a Dell PowerFlex.
Matty: [00:23:57] Exactly. Yeah. Like, you know, you need some switches. You need some, you know. Yeah. Yeah. Because that’s the thing. Because the only reason that we don’t want to murder a printer right now is because we don’t have to print anything.
Kat: You know what, actually, I have a printer, okay? And I’ve had to print something recently and it was not that bad.
Matty: What I have had to do with my printer, because for 2 reasons. So one is the good thing about the, I can’t see the printer and you’re like, you’re literally right next to a computer. But for some reason, my HP printer or whatever continually needs to— fundamentally, anytime I want to print, If it hasn’t been in the last 5 minutes, I have to unplug the printer and plug it back in anyway. And so what I do is I just keep my printer unplugged all the time because the other thing is it has a status light on it that you cannot turn off. And this printer is in my bedroom, which means if my printer is plugged in, there’s this blue glow throughout the entire room. So my combination was, number one, I don’t like that. Number two, chances are I’m going to have to unplug and plug it back in every time I want to use it. So you might as well make the default be that it’s unplugged.
Kat: [00:25:06] Sure, that makes sense.
Matty: Some metaphor for the internet or working in tech in there somewhere, we’ll find it.
Kat: But, you know, I would actually like— I know that this is absurd, that podcasts, like famously visual form of media, so you people can’t see this, but behind me there is a big enormous gaming PC with like RGB LEDs on absolutely goddamn everything. My mouse, my keyboard, everything. Has RGB LEDs on it. But for the most part, I wish shit had less LEDs on it. I have 2 of those enormous ASUS ROG routers that look like, I don’t know, a satanic altar with like 9 antennas. And there are so many LEDs on that thing. And one of them is in my bedroom. So I can turn off the like fancy RGB LEDs on it. And I do, but the status lights, stay on. So it just like lives with a t-shirt over it. Otherwise there’s shit blinking at me while I’m trying to sleep and I have, there’s nowhere else it can go. It has to go in my bedroom because that’s where the entry point for the fiber is. So if for some reason like ASUS or whatever other hardware manufacturer or Brother printers is listening to this, can we have a button to turn that shit off? Like, I don’t want that. Turn, turn that shit off. Because I don’t want computer visible in my home in anywhere other than the computer room. You know, like in the ’90s, we had like the computer room or the family room happened to have the computer in it. I’m bringing back computer room. This is, this is computer room, my living room and my bedroom, no computer.
Matty: [00:26:53] I would, I would prefer that myself. I kind of had to make my bedroom also be computer room because I had to make computer room turn into one of my son’s rooms. So that, that negated that. But even that, because I thought about that, because I’m gonna put up like a partition behind my desk, you know, so at least because I need the like mental block between my desk and my living area. But I was like, but if it’s a bright light, it’s gonna bleed through, you know, a room divider thing or anything. So this is all a metaphor for security. That’s why we’re talking about it. So it’s fine. So if you’re still listening to the show, I appreciate that. But that also kind of brings us around to, to, to an end. So, you know, if you’d like to see the show notes, we’ll put some links to some of the things we talked about. You can go to arresteddevops.com/diggingintosecurity for this, for those show notes. If you go to arresteddevops.com/itunes and leave us a review in the Apple Podcast Store, that can help other people find the podcast. You can also listen to the show on Spotify, iHeartRadio, and Audible, anywhere that fine and less fine podcasts can be found. And I do think that I may be this close to finally updating that redirect so that it’s not /iTunes anymore because it hasn’t been called iTunes in forever.
Kat: [00:28:07] Oh my God, what’s it called now?
Matty: Well, it’s just Apple Podcasts or Apple Music or whatever. Like, so referring to it’s the Apple Podcast Store, it’s not iTunes where the podcasts are. And coincidentally, I think what kind of made me decide I’m okay with eventually finally doing this is I’ve been updating the Hugo template theme that this podcast uses amongst many others, and it all still internally refers to everything as iTunes instead of Apple Podcasts. And even, you know, and I was like, all right, fine, I’m doing a big version. We’ll call it Apple Podcasts and iTunes because there’s going to be people doing this who don’t even know what iTunes Anyway, that’s how you can listen to us. We hope you listen to us more. Kat, thanks for, for joining me today. We’ll, we’ll have to continue, continue these conversations. And this is Arrested DevOps. And remember, there is always DevOps in the banana stand.
