Dominic: [00:00:00] Because the hackers are making the same discovery with the same speed, right? Like, except they’re highly motivated to, to use them as quickly as they can.
Matty: And they don’t have change control review boards to go through, so they can actually move faster. It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton. We are digging in for a great conversation around the eternal topic of security. But before we get into that, let’s hear some words from our sponsors. Kolide sends employees important, timely, and relevant security recommendations for their Linux, Mac, and Windows devices right inside Slack. Kolide is perfect for organizations that want to move beyond a traditional lockdown model to one where employees are educated about security and device management while they’re fixing important problems. Visit kolide.com/ado to sign up today. That’s kolide.com/ado. Enter your email when prompted to receive your free Kolide gift bundle after trial activation. You can try Kolide with all its features on an unlimited number of devices for free for 14 days No credit card required. Try it out at kolide.com/ado. Do you want to improve your software team’s productivity but don’t have the information or visibility to know where to start? Zenhub helps teams build better code faster with a developer-friendly productivity management platform. Zenhub empowers startup and scale-up teams to get more done in less time wherever they are. With the best-in-class GitHub integration, team productivity insights, and AI-driven automation, Zenhub helps you plan work, track progress, and see the big picture across any team. Automate your Agile and get hours back each and every week for building features your customers want. Get started for free at go.zenhub.com/arresteddevops and enjoy 10% off your first-year subscription of ZenHub. Do you ever start a query going in your log aggregator, go get a cup of coffee while you wait, and by the time you get back it’s not the answer you needed and you’ve started to forget what you were looking for to begin with? You don’t have time to waste like that when you’ve got issues that need fixing now. Whether you need to understand your entire overall system or drill down to the individual user level with traces, Get the right answers fast when you need them with Honeycomb. Go to honeycomb.io/arresteddevops to use it for free. We’ve been talking about this idea. I don’t know who we are, but people, people are talking. Maybe it’s called DevSecOps. I don’t know, but security is a thing in case you didn’t know. I hope I’m not the first one to tell you that, but I’ve got a couple of guests with me today and we’re going to be talking a little bit about DevOps and security, but maybe not in the way you’re always used to hearing about it. But let’s find out. Joining me first, Sue, you want to introduce yourself to our listeners, please?
Sue: [00:03:19] Hi friends. My name is Sue Choi. I am co-founder and CEO of Mondoo, an infrastructure security company for DevOps.
Matty: And also joining us is Dom.
Dominic: Hi everyone. I’m Dominic. I’m also one of the founders of Mondoo, co-founder, co-creator of Inspect and a couple of other tools. Security nerd. Developer, hacker, coder, weird mix.
Matty: I am really excited to have you both on this show. Longtime listeners may remember that I spent a fair amount of time, I used to work for Chef and that’s how I met Dom and Sue back there. So we’re going to try to not have a podcast episode that’s just catching up on gossip and we’ll actually talk about stuff that you care about. You will have to tune in.
Sue: I know, I can listen to that podcast.
Matty: We do again, you know, fans of the show, I, I use the word fans loosely, know that we have our one self-indulgent episode of the year, which is our year-end wrap-up, which is pretty much just all the hosts getting together and shooting the shit. And we are always surprised about how many people make it through that entire episode, but you never know. You never know. But so yeah, uh, let’s just kind of start with a broad question, if you would. When we’re thinking about DevOps and security and InfoSec and how this fits into this idea of DevOps. Like, what the hell is going on? Right? Like, where are we at? Let’s start with that.
Sue: [00:04:46] Yeah. I feel like let’s start with like the reality out there is that if you believe that software is eating the world, then hackers are having a feast, basically, Matt. They’re going yum, yum, yum, yum, yum, yum, yum. Right. We’re leaving it to the security team to actually protect us from all of these organized threats. There’s just not enough security professionals out there. I think that DevOps teams actually do a lot of security, but a lot of that work is hidden and it’s not visible. There’s a lot of reasons why. What we need to do is to increase our security posture for any organization is we need to get as many people explicitly working on security in a way that’s easy and part of their own workflow.
Matty: Yeah, because we talk about that, right? Like, there’s a lot of tropes, there’s a lot of glib statements that people may make on Twitter and in talks, and by people I mean me. So I’ll— but, you know, like, what one of them is the, you know, security is everyone’s job, right? So, so what the hell does that actually mean, right? Like, it’s easy to say, but How do you make that everyone’s job? Or should you make that everyone’s job, right? Is that maybe not the right way to think about it?
Sue: [00:06:07] I, I think it should be everyone’s job, but I feel like people don’t know how to do it, right, in a way, or they’re not necessarily incentivized with shared metrics. And I mean, there’s a reason why there’s the security team and then there’s the DevOps team. I don’t know how many people I’ve talked to and I’m like, hey, let’s talk about security. And Maddie, I think they sort of black out because it’s not a joyful topic for them. It is really intense. It’s quite serious with really like hard ramifications if we get it wrong. Right. So how do we get us talking about security? How do we get us talking about security in a way that can help us move forward versus blackout and need lots of libations to make the pain go away?
Matty: I saw this great tweet the other day, and I will try to find it and put it in the show notes. I really apologize that I don’t remember who tweeted it. I’m gonna kick myself when I find it. So in this tweet they said, you know, you see all these tweets from DevOps folks that are like, here’s a time that I fucked up and took down production. Let me tell you a funny story. And they were like, why don’t we have that? How do we get to that level of blamelessness and safety with security? You know, and then the reply they put was, oh, but security incidents like cost money. Well, so do tech incidents, right? Like, oh yeah, like that’s that. There is this thing where we’ve— I, first of all, I don’t firmly believe that as an industry-wide we’ve completely embraced and understood blamelessness. That’s a whole other problem. But like, people do feel kind of okay now to sort of like tell that, that story of the time they screwed up and fat-fingered a command and took down production or dropped this table or did whatever. But it— there’s this, like you said, that when people, you know, you said they kind of black out because like admitting a failure that’s connected to security like just seems Like the stakes are higher, but are they, or are they not?
Dominic: [00:08:15] I think they’re all part of making sure that the application actually, or your infrastructure actually runs the way that you intend it to. And if you look at it from that perspective, like make infrastructure run the way that you intend it to, you get the operational perspective and the security perspective. And I don’t think necessarily one is higher than the other. So for example, if the service is down, it may be super secure, but I can’t use it. That sounds bad. On the other side, if the service is running perfectly, but your front door is open and you’re giving out my credit card information like Halloween candy last year, then that’s another problem too. Right. And so it’s a balancing act between both of these teams. It is hard to get them together because both have something really, really important to contribute, but their feedback, their priorities, they need to be aligned. Like you need to come to the same table. You need to have a candid conversation about what can be done and what cannot be done. Not every security issue that the security teams find will be fixed. Not every security issue that they find necessarily needs to be fixed either.
Sue: [00:09:22] Yeah. It’s this reality of— I call it paper security. And then you have real security. And I think Security teams have the mandate to make sure that we’re doing the paper security right. But the DevOps team actually don’t necessarily care about paper security. What they really care about is, one, let’s do things that are real security. To get to that answer of what’s real security is hard.
Matty: It requires kind of understanding a little bit more of the underlying stuff, right? I don’t know if I mean the underlying tech, but I know I’ve dealt with, you know, with organizations like this going back to just doing like a DevOps transformation, right? And would run into things like, oh, we can’t do that because of this regulatory reason or this compliance reason. And you’re like, but really? No, what you can’t do it because of how your organization chose to implement that control That’s not actually what PCI or SOX or whatever actually says. And it goes all the way back to the Phoenix Project and the 3-ring binder, right? You know, we’re— that’s what, that’s what I said. It feels like everything old is new again. We’re still having these similar conversations of, but why are you doing something, right? Like, Sue, like you said, that the actual— you can sit there and say, yes, we tick the boxes in this particular way, but You know, it’s sort of like The Simpsons thing, right? Where like you see Mr. Burns and Smithers going through 17 different layers of physical security and they get into the secret room and there’s a broken old screen door with a dog running through it, right? You’re like, yes, but we got all those other things. And as many, many memes and many, many conference talks have illustrated, right?
Dominic: [00:11:14] And all of that resulted in a very, very interesting end result, right? Like, so we have all these requirements Let me tell you a story. So 10 years ago, when we first started to introduce cloud systems, and that was my previous life, I was at Deutsche Telekom and in the telco business. And so our systems needed to be secure. It’s a requirement. Otherwise, you know, the internet doesn’t work. And so we were introducing cloud systems and clouds on their own, together with DevOps at that point, which we were also introducing, were very, very different. From what security knew at that time. And so I remember having my first security expert who came in, the auditor for the company, who brought all of the paper security manuals with him and wanted us to implement those. However, that was not possible. Like some of the things that were written in there were just not applicable in the way that we needed to run our infrastructure. And so the conversation didn’t go very, very far in the beginning, simply because We rejected the binders and that made us immediately distrust what security wanted from us. So that led to conflict. Conflict led us to have a much worse in terms of security at the end of the day. It only resolved itself when we started to switch our pen testers and our auditing team. And we had somebody come in who looked at it with fresh eyes. And he basically said, yeah, like everything we have in the docs so far, like some of these things are applicable and other things are not. And we need to figure out what we actually need to do to make you So in that very moment, we had a conversation that happened. We had somebody on the security side who understood, who was able to digest all of these findings that were coming out and all the requirements, was really able to pinpoint what we needed to do in order to make it as hard as possible for an attacker to get through. And so together we were able to formulate new requirements for the space, which was exciting. We were able to put them into our automation pipeline as well as, you know, in the policies that we’re rolling out. And that led us to an amazing result. But that’s like one example of where it went well. And I think there is a million examples where that is not necessarily how things go. And then you’re in a difficult state. You end up sitting there thinking, oh, you know, I don’t want to deal with security. That’s a pain in the ass. And I mean, how much can there go wrong? But indeed, there is a lot that can go wrong these days. And as we have seen recently from the more active ransomware attacks, which by the way, are huge proponents of the automation that we’re preaching. You can see how far-reaching they can be just leveraging some of these misconfigurations, vulnerabilities, and so on, and tying those beautifully together in an automation that screws you over in a big way.
Matty: [00:14:00] I think that’s interesting when you were sort of telling the story about how you had, you know, a little, you had this, this blocker and then someone came in from the outside to help put it through there. We don’t always have that luxury though, too, right? You know, like, and I think any of this, anytime we run into these like conflicts, if you will, across, you know, from, from disciplines and stuff. It is sometimes a combination of misaligned incentives, but even that comes back to the larger point of, like, sorry, we can’t talk about DevOps without talking about empathy, but 100% is that in the context of, like, understanding why people have the challenge that they do, right? Like, nobody— I shouldn’t say nobody. Almost nobody is getting in your way for a thing you’re trying to do at work because they’re a dick, right? Like, it’s because there’s something that’s important to them for some reason. And we also know, like, policies are organizational scar tissue, right? Like, there’s a reason. And if you can understand that, that’s your first step. You still may never be able to convince them, but you’re certainly not going to if you aren’t understanding where they’re coming from. That’s harder than it sounds, right? I mean, we’ve been standing on the stage saying that for 10 years.
Sue: [00:15:11] Exactly. And so, and I don’t think security is is there yet, right? When you look at just even security talks or security conferences, you don’t see empathy. You don’t see workshops on team building. You don’t see that kind of care and focus as a topic versus like war game scenarios or training. And so I feel like that it’s almost, it’s unclear why it is the way that it is, but I feel like it’s actually shifting on the security side. And as security becomes more diverse and there’s more opportunities, I think we’re going to see a lot more conversations that look like what we went through with DevOps 10 years ago and talking about empathy.
Matty: I think it’s maybe, you know, not, not to paint with a super broad stroke, but a lot of the approach, a lot of classic approach to security is very adversarial. You can’t talk that way all the time about certain things and not have that kind of unintentionally influence your entire way. If it is starting to think it’s like, it’s us against everybody. So much of this was at the beginning of DevOps. I’m an old-school ops person, and you go back in time even 15 years, and the way Matty thought about life was protect my systems from those damn developers. My colleagues were my enemy, you know? So when you think that way, you’re not gonna build bridges because you’re thinking you have to protect things from that and everything becomes a fight, right? Everything becomes a defense.
Sue: [00:16:54] Yeah, I mean, and I feel like there’s a lot of professional ramifications. So we’ve been talking to a lot of CISOs who are very interested in our opinion. And so we’re like, great, can you introduce us to your CTO, director of operations, You’re a director of cloud engineering, SREs. We have so many wonderful job titles now. I love it. And their reaction is just like, oh no, they hate me. And I’m like, why? Like what happened where there is at that level, right? A CISO versus a CTO. And I think that to, to do well in a C-level position, you have to know how to talk to your peers and your peers aren’t just technical, right? And I feel like CTOs have really sort of mastered this, is that they know that their peers are the CFO, their peers, they report to CEO, their peers are, you know, Chief Revenue Officer, Chief Marketing Officer. So they have to relate things to business value. And I think with security, it’s, it’s harder to relate that versus the, the CTO. So what you do talk about is just is risk, right? And you don’t— I don’t know if you make a lot of friends of trying to like do risk avoidance, and if you’re the most popular kid on the playground because of it. I mean, they don’t even get along. But I feel like for us to really make DevOps plus security, like DevSecOps, a thing, I think it needs to start from the top too. We can’t discount that type of messaging at that level?
Matty: [00:18:35] I think, you know, this is where like all the jokes about being the chief no officer and all this stuff for a CISO come, come from this perception. And I think ops folks and security folks should be better friends because they kind of could share the same stories over, over a pint. Right. Which is, I’ve always said, and I think this is true for security too, that, you know, being in ops is like being the corporate lawyer. Like nobody knows when you do your job. They only know when you don’t do it, right? And then security, I think, feels the same way. You know, you’re— everyone is aware of what your security team does when there’s a breach or when something bad happens, but they don’t know about the 10,000 times a day that they protect the system or do whatever. And it’s— that makes it so the only way that you know about someone is when they’re trying to stop you or something like that. The more I think about this too, it’s like, you’re right. Like we, we, we went through this with DevOps, right? Like, and it wasn’t easy and we aren’t there either, by the way, just so we’re not declaring victory for DevOps on this show right now. We still have work to do. Please invite me to speak at your conference about it.
Sue: [00:19:41] I feel like as a community, can we invite our security folks to come to a DevOps Days? Could we let them into our community and what would that look like? What would they collaborate with us on? And so I think taking it from that lens of our tent is definitely getting bigger in DevOps. And so there’s been lots of people who have tried and it’s unclear why that hasn’t really spread and resonated. Dom, do you have any thoughts on that?
Dominic: We’ve been trying our best, actually from both sides, to get the folks from the other side to come in. If you look at some of the big security conferences, you had some people from operations come up and live in that context too. And you’ve had security folks who came to DevOps conferences as well. So there were certainly some attempts. I think it’s a hard one to bridge because you are looking at a similar set of problems, but you don’t necessarily have the full picture that the other person has. So for example, from the security perspective, one of the things that we have heard was, one of the big topics in security is, can’t you just auto-remediate that issue? Can’t be that hard. Let me just push the button and it will just fix itself. I give all my issues that I have to operations, just going to auto-apply all of them and we are going to be perfect. And then of course operations sees that and is saying, oh my God, you’re going to do what to my infrastructure? And so there is context in which every single one of these security requirements fixes vulnerabilities and so on lives in. And so if the answer were as simple as push the button here in order to apply it, I think we would have done that already. I think we’ve realized that in the DevOps, in the time that we’ve been practicing DevOps, that applying change to infrastructure in a way that is safe, reproducible, that has been tested, that is not going to make us fall apart is not trivial. But we have developed a really, really good muscle to do that with the development teams together. Now it’s time to expand that to the security teams. And it’s the same conversation we have with the development teams, whatever new changes you’re trying to introduce, we need to carefully introduce them, test them, and focus on the things that actually make the most sense. So for instance, what doesn’t make sense is to give me a list of 300 findings and saying, do all of them. I actually need to see a list of 10 critical issues or important issues for our infrastructure, where we’re at, that we want to tackle together. And that’s where we want to spend our time, because none of those come for free. Every single one of these issues will take some amount of time of thinking, of digging into it, of testing, of making sure everything still works.
Sue: [00:22:23] Yeah, I feel like there’s a lot of tension or just a lot of rough edges when it comes to just negotiating that list of 300 down to 10. And a lot has to do with like classic negotiation skills, right? It’s like, what are you trying to achieve? What are we trying to achieve? What’s our decision-making framework? Like even, even that level. And we’ve had those conversations with the development team, right? But we don’t necessarily use that kind of framework with security. They’re, they’re, I mean, as you said, they’re not being jerks. They actually are just trying to do their job. But what kind of like logical decision-making model could we actually implement to keep everybody sane during this process? And then the output of that would be like, we’ve increased our security posture. And I feel like DevOps like has the biggest stake in it because we have to fix all of these things, right? It’s—
Matty: [00:23:26] I was just going to say about it, like El Marquez, it was funny when you were talking about having security people at DevOps Days. So El Marquez definitely sits strongly on the InfoSec side. And she gave this amazing talk at DevOps Days Houston. And I’ve heard her say before several times, the term I like where it says, you know, it’s It’s DevOps, the sec is silent, right? You know, which is— that’s— I, I kind of go back and forth with that portmanteau because I’m like, you know, we’ve all been doing this DevOps thing a long time. I don’t know if you agree with me, and I’m like, but this is what it’s always been. It’s just like, thanks Andrew for calling it DevOps because now everybody thinks it’s just about that, but it’s always been all of this. And so, but I’m like, if DevSecOps like helps security feel like they understand that they’re part of this, then that’s cool. If you think about shift left, right? You know, people talk about shifting left and the problem with any buzzwordy Twitter term is that there’s no room for nuance. Right. So like shift left without explaining the nuance makes you think that we’re just going to have the software developers do all the security shit too. Right. And we tried this with DevOps. It was called NoOps. And there’s a reason it didn’t work is because it turns out that like domain expertise is And specialization actually matters. So it’s like the same thing about how, you know, when we talked about DevOps, we, we learned that it wasn’t just give your software engineers root in prod and make them carry a pager. Yeah. I mean, that’s like part of it, but it’s not just that. It’s not like, here’s, here’s, here’s root.
Dominic: [00:24:55] Good luck.
Matty: Have fun. And then the same, we still have operation as a discipline, right? It would be a shame to have to learn these lessons the hard way all over again. Yeah.
Dominic: It’s subject matter expertise. Right. And at the end of the day, you’re trying to combine as much of that together and get that around the same table and get that to talk to each other to solve these problems.
Matty: But I think you just nailed it, Dov. You’re like, to get that same thing around the table, it’s not get all that stuff into one person, which is unfortunately what starts to be where that gets felt is like, oh, we’ll just move it to that team and they’ll just do everything. And yeah.
Dominic: Yeah. So this reminds me a little bit of how the whole security thing started. I’m not going to derail this for too long, but like imagine in the early days when we didn’t have security, everything was open. And then suddenly somebody realized, hey, so you know how you lock your home up when you leave the house? Like that’s probably something we should do with our computers, right? Like people going into our bank and taking out money, that’s not a good thing. Can’t be. So security came in and their goal was just like help people lock the door, but it was like very, very focused on just this sliver of a thing. For the last 20 to 30 years, that’s what we’ve been trying to realize. Like we’ve been trying to say like, let’s make sure that they’re as good as they can be. And so we build up a lot of amazing subject matter expertise in that. And that’s also a lot of the people, by the way, that are moving with the crazy new technologies that we are introducing on the DevOps side, right? Like with the rise of serverless and Kubernetes and containers and all that, they’re constantly catching up to that and trying to understand, okay, how do I make that secure? Because the hackers are making the same discovery with the same speed, right? Like, except they’re highly motivated to use them as quickly as they can.
Matty: [00:26:37] And they don’t have change control review boards to go through, so they can actually move faster.
Dominic: Yeah, a couple of less steps, right? Like, they don’t have to go through a month of, I don’t know, writing like performance reviews where you’re busy with other things. So it has become— it’s important to acknowledge that there is a lot of really, really helpful knowledge One of the things that we’ve been trying to do, just like working on this new startup and working on this new topic, has been to look into how can we facilitate that? Like, how can we get that knowledge into a form that is codified, that can easily be shared out on real systems where I can see what’s going on? And we have seen similar things in the DevOps realm with a lot of the frameworks like Terraform, Ansible, Chef, and so on. We’ve seen things like this happen with frameworks like InSpec. And I think this is like naturally where we want to build on and extend this thought because that’s where you can distill a lot of the, like, you know, it’s not about the, for instance, Inspect or Chef or Terraform, whatever profile. It’s about the knowledge that goes into that. It’s about what it’s trying to say and what it’s trying to say about your systems that makes it valuable and interesting. And so if we take that and we get the security perspective from that and we align it with the DevOps perspective, then you get to like a shared piece that describes what these systems should be like or what we think is the best for these systems, combining the knowledge from developers, operations, security, and whoever else plays a role in that.
Matty: [00:28:05] So how can we get smarter about this stuff? I want to have that empathy for my security cohorts. And, you know, basically the way that I learn about InfoSec is I follow a bunch of InfoSec people on Twitter and read their shitposts and try to distill from the joke, what’s the truth. But I feel like there might be a better way.
Dominic: A couple of points here. What I like, I’m not saying I have the perfect answer, but I think there’s a couple of hints that we can take this to. So there is no one good answer to all these security problems, similar to how there is no one good answer to the bigger DevOps story. It really depends on how your organization is structured, what your teams prefer, where their strengths and weaknesses lie, how they like to work. It’s the same thing here. So it’s not trivial. However, one way that I have found worked really, really well for me was security, just like DevOps, Pretty, pretty big problem. So it means you’re looking at a lot of technologies and a lot of parts of your stack at the same time. So try to break it down into just a couple of topics, just like pick one topic, for instance, I don’t know, like SSL/TLS, and just start to have that conversation with your security teams, what that means to implement certain guidelines around that. Like don’t take the 300 results that you have, but just focus on that one topic and what that means to your organization and what you’re trying to do. And then you will see like what they would love to do, plus what you cannot do, because there are certain things in SSL that you just cannot operate this way, or what you don’t want to do because customers will be left on the wayside if you do certain things. And that’s where you get to alignment. But yeah, pick one topic and dig into it. Yeah.
Sue: [00:29:45] I was just thinking about like, what is it like to be on the security team. And can you imagine going and having to talk to your operations team to ask about, for instance, like you’re trying to help your company get more customers by getting a SOC 2 certification. And there’s a lot of weirdness in that, right? Because there’s a lot of checkboxes. So for instance, say you’re 100% remote, but you’re supposed to have cameras monitoring the, the entrance exits, right? But you can’t put no, because then you’ll— you don’t want to fail your SOC 2 certification. Like, that’s just one, like, example. There’s hundreds of those in that. And you have to try to figure out a way to, one, like, say yes, but also, like, not stretching the truth. Or you just are so exhausted that you’re just trying to, like, just get a binary answer. And then if you talk to any DevOps person and ask them a question, it’s always— I think that’s part of our culture is like, it depends.
Matty: [00:30:56] It depends. Yeah.
Sue: So that’s what their day, weeks, months are like, aside from like if there is a security incident, which usually you don’t find out about until potentially like weeks or months after that. Right. So yeah, I just can’t imagine what it’s like. It’s an interesting field. And I find people that in InfoSec are especially creative because they’ve taken a topic that is both really threatening, right? Everything’s just like uber, just sort of yelly and they find humor. Right. And so I really look up to all of those creators.
Matty: Don’t you have to? Like, what do they call it?
Sue: Gallows humor.
Matty: Yeah.
Sue: Yeah.
Matty: Right.
Sue: Yeah.
Dominic: So the other thing I would touch upon is you want to build a relationship, of course, with your security team. Now that seems obvious, but of course that’s not trivial to execute. That is largely built on trust. And so let me give you 2 negative examples that we have— actually, I have, I was part of the team. So we have done in the past. One was where we went through a certificate to make sure that our environment is secure. And that certification was signed off on a Friday. Security had put the entire environment in a freeze state for about 3 weeks in order to conduct the deep audit. That, first of all, that is hard for anyone who works in DevOps to do. Making the environment unchangeable for such a long period of time when you’re trying to get to a launch is a nightmare. So that’s one thing. The second thing was, however, so that’s where they lost trust with us. But we also did it the other way around, which was after we had gotten the certification on the Friday, Through the power of DevOps automation and scale, we had put that environment into a different state so quickly on Monday morning, you wouldn’t even know what happened. And so that’s where you lose trust on the other side. Then security becomes overprotective. So they’re basically going to say, oh, we can’t trust you. Like you’re going to do whatever you want to. And we basically say, we can’t trust you guys because like you’re bringing your crappy tools into our environments. And You’re telling us to do a million things and not letting us launch our products, which also can’t be the right answer. So that is a very, very difficult exercise to get right. But the one thing that I can do is, you know, in your role, you may not be able to influence the other side. You may not be able to change them, but you can always change yourself. And so if you indeed play the game where you try to trick them, then, you know, that’s how the thing will play out. They won’t trust you. And it’s hard to build that relationship.
Sue: [00:33:33] I’d love to hear more stories out there from the community of when it did work or when it is healthy and some success stories there. That’s what we don’t have, right? We have these great metrics. We have these great success stories about DevOps overall. Like, look at our, you know, MTTR, look at our change fail rates, like all of this. Look how many times we’re releasing per year, per day, per minute. But we don’t have a lot of success stories overall. And I’d love to hear more of them. I think the community, I think we’d learn so much.
Matty: I feel like this is just one of those challenges with security too, is being really specific about like what you do is considered like risk, right? You know, and we kind of, we kind of, again, we felt that way with infra and with all that stuff. And we kind of eventually got over it back with the, like, don’t expose the, you know, the, your, your web server version in the header, because then people know who to attack. You’re like, maybe if that’s the easy— if that— if obscuring that is the only way you’re protecting yourself, maybe you have other problems. But I think it, it goes back to just that risk, right? I did find the tweet, which is from Cat Sweet, and it’s sort of like you said, you know, people talk about those instances, oh, but that has business impact. So does your infra outage, right? And the same thing. So like admitting, you know, people are like, oh, if, if people knew this, then we’re liable, and there’s, you know, like a lawsuit or something. Well, you know, if someone’s going to do that, they’re going to sue you because like your load balancer was down too. So, right. You know, I don’t know. I know it sounds easy for me to say, but I feel like, like maybe we need that renaissance of like transparency of telling these stories. And, and I think there are ways you can tell them without getting into like the specifics of if you say this thing, then, you know, people will know how to attack you. But we have the same problem in DevOps too. Like I know people in large enterprises. I mean, hell, before DevOps Enterprise Summit, nobody was talking about what’s happening. And, you know, as far as you knew, DevOps was only happening at Netflix, right? And Etsy. But I know that I can think of several people I know who wanted to tell a story in a talk about like how their DevOps didn’t work well, like how that, you know, like telling their failure story and like PR for this large company goes, no, you can’t say that. You can’t ever say anything that means we aren’t amazing. And you’re like, really? Really? Like, it’s gonna, like, cause material damage to this insurance company because someone’s like, guess what? We didn’t get DevOps right the first time. You know, but there’s this thing, and I, again, I feel like it’s like the stakes feel higher in security anyway. So figure out how to tell the story without giving away everything and come tell it at a DevOps Days.
Sue: [00:36:20] So let’s talk about Equifax, and we know that they had a huge breach. And everyone asks, who is the CISO during that time? No one wants to be that person. Something bad happened on your watch, right? But it’s really interesting that when you talk about like a major outage, do you blame the CTO? I, I don’t think so, right? Like, I don’t— I think a lot of CISOs live in infamy after like major breaches, but like that doesn’t necessarily happen With the CTO, so it’s just another data point on your that your part. When was the last time a security person, if you ask them what’s your ten biggest priorities right now, could could they just name it and share it with your team? Has that ever happened?
Matty: I don’t know. Let’s find out on Twitter right now. How about can we just say five? What do we make it just five?
Sue: Yeah.
Dominic: Five. So I have actually good security people. They will immediately answer. They know what they’re looking for. By the way, but there’s a difference between the technical security and the legal security, basically. So some of the compliance requirements that you have that you just have to follow. And then there’s a deeply technical security, which actually understands what’s going on. So the ones— there is the rise of the new discipline, the DevSecOps discipline, or the security developers, essentially security engineers who are deeply technical, truly understand what is going on in your infrastructure and how to best secure it. And so those types of people are actually naturally fitting into the DevOps story, into the DevOps model. And those are the ones who have these deeply technical conversations with you. If you find that you’re talking to someone who doesn’t technically understand what is going on and basically insists on certain requirements being met because they have been documented, insists on requirements being met because that’s what the document says, then find the other person. You got to find someone who actually will have the technical conversation with you and is able to outline what certain risks are and is able to work with you on the right solution.
Matty: [00:38:33] For those of you who are listening, so we’re recording this. I know you’re shocked that this wasn’t recorded moments before you’re hearing it, but we did, I did tweet and ask security folks if they could name their their team’s top 5 priorities right now. And I will put some of those tweets in the show notes, which you should go check out at arrestedevops.com. There’s a topic kind of kicking around here. Forget all this NFT crypto bullshit or whatever. The only thing anybody wants to talk about now is supply chain. And thank God, because that’s actually like something we should talk about. What, first of all, could either of you like kind of When people are sort of throwing this around, like, what are they really talking about? Right. Because I think a lot of people are talking about it. I’m like, I’m not 100% sure everybody even knows what it is, except it sounds good.
Dominic: The term of the supply chain is kind of a recurring term of everything that goes in as a dependency into our software building process. So that could be all the dependencies that you’re pulling into your apps. It could be all of the required components that you need to run your infrastructure. And how everything comes together is getting tested, validated, verified, making sure that it’s working well before you actually push it into the production environment and roll it out. So as a concept, I think it’s, it’s a much better term than some of the things that we used to have in the past, but it’s getting more important in the security domain because a couple of years back, somebody suddenly realized, hey, so we can wait until production breaks or we could start to look at how things make it into production and see if we can prevent them from coming into production if they are bad. So security started to, again, this whole term in the industry, shift left, shift towards the developers, shift towards DevOps, and try to help them identify these issues early on. The thing, however, that we also see with this is that if you apply the same rules that you have been using in your production environments, so let’s say for instance, you’ve got your 300 findings, You take the 300 findings and you move them as far left as possible where somebody is sitting, like me, for instance, trying to get their pull request through a pipeline and then sees this long list of issues, then my reaction will either be, you know, commit and push through, or I will maybe dig into a couple of those issues. It’s gotten so bad actually that we are now seeing that people only react to critical issues. They aren’t even reacting to anything else. And I think that’s a slippery slope. That we’re on, because that means that some other findings will suddenly be labeled as critical in order to make people move on them. But that’s not the answer. I think it’s important for us to understand what makes it into our software and truly understand the priority of issues that we have. I think a lot of the tools that we have right now out there aren’t necessarily helping us as much in that regard as they should.
Matty: [00:41:29] So when we think about that supply chain and the things that make it up. The theme of this episode is everything old is new again, right? Or we’re going down this, and I feel like it’s like, hey, Left Pad was 5 years ago. Why haven’t we figured this out? It just gets harder, right? When we think about all of those dependencies, like you said, and it’s, oh my God, do not look at the node_modules of my My freaking blog, right? My blog has got so many packages in it just to do this one thing, much less software that does real stuff. And I remember, like, when I was working at Chef, you know, there were scenarios where if you want in certain enterprises, and I wouldn’t be surprised if this isn’t still true, where any open source component required, like, CIO approval for it to exist on the network. And but then even then I was like, but I’m like, how are you really tracking that? Like, how do you know, right? And especially it’s one person. So the way to protect your, your supply chain, by the way, is not to require your CIO to approve every open source component you ever do, because that’s crazy.
Sue: [00:42:40] Have you seen people— are people still limiting types of licenses too that you can run inside the corp?
Matty: Probably, but I feel like that’s from a different perspective. But it wouldn’t— I’m sure that goes back to like this, you know, liability perspective, right? Like, oh, if we have to do this, you have to do this. And it’s like, but, but, but all these transitive licenses anyway. So it’s, you know, basically this is— I’m here to tell you, like, just give up. Just— it’s everything screwed. Just throw some software against the wall. Letting sand think and talk to each other was a mistake. I’m gonna go buy a ranch in Montana. Um, no, but like, okay, but how realistically— so on one end of that spectrum, you have this like You could be the like, just YOLO that shit, who cares? But then the other is you can’t, and you can’t like understand what every transitive dependency is, right, Dom?
Dominic: Yeah, that’s right. So first of all, the, uh, a lot of the tools that we currently have in this domain, they’re doing us a little bit of a disservice. So they’re helping us in identifying what issues might be, but let’s be frank here, they’re full of false positives. And the psychological effect that that has is that if I have to read into a bunch of issues for 15 minutes and the end result is they’re all false positives and I got nothing that, you know, nobody’s going to give me those 15 minutes of my life back, then that’s a bad outcome. And I think this is something that needs to change in this space. It’s not about the number of results that you find. It’s actually about finding the right issues and then fixing those. So that’s the first thing. Giving up, by the way, we’ve seen what giving up does. If you give up and you are the target because you’ve got something to protect, then there will be people who will find ways to get to that information, to get to that value. So we have to actually lead this conversation day in and day out because it’s worth protecting our customers’ data. It’s actually worth making sure that our infrastructure is safe. Now, to the supply chain itself, I’ve heard conversations that say, by the way, to your point, everything old is new again. So I’ve heard this 5 years ago, people were saying, oh, we’ll just do it in the supply chain. No need to look at production at all. Once it’s solved there, it’s solved everywhere magically. Nobody will ever touch production. Right? Wink, wink. Of course. So the same thing is happening right now where people think that we will shift left, we’ll put everything left, we’ll not need the right side anymore. And we’re going to be great. That’s not the case out there and we know it. So I think to me, the right answer has been to get the right level of controls into your pipeline, make sure that they are visible. But let’s also be real, a lot of the tools that we use in the pipelines today, we activate them in a way that isn’t going to block the pipeline. Like we get the reports, but if you’re not going to deal with the 20 false positives that are coming out of that, then you might take a look at it once. I think it’s just like, you know, you’re learning to walk. So you’re taking one step at a time. The first step that we’re currently taking is we get visibility. Great, thumbs up. Next step, prioritization. You actually need to understand what you need to prioritize in that pipeline. I think that’s the step that we need to get towards. And then the third step is actually fixing the issue. Once you understand what needs to be fixed, actually with a real priority, then you can go on to fixing it. And I think this is— sorry, small rant— where the securities industry is a little bit lost on. Because they’re now talking about, like a lot of companies, I don’t want to name names, but there’s a couple of companies that are now talking about, hey, what if we like auto-remediated all these issues? We’re going to auto-fix them for you. We’re just going to push the button for you. And I think they’re getting ahead of themselves. Because as we have seen in DevOps itself, We need to identify what changes we’re trying to introduce very, very carefully so that we don’t screw customers on the other side. Because like, for instance, we disabled the login mechanism for like 30% of our user base. That sounds bad, no matter how secure it’s got. So you find the right balance. Prioritization, I think, is one of the biggest issues that needs to be cracked in the security industry. Once you have that, then you can really tie it together with the DevOps principles of automation and scale, because that’s where the DevOps industry, like the whole DevOps space has been fantastic. You want to do something at scale, consistently everywhere, detecting drift, make sure it’s in place. Awesome. We’ve got a solution for that. We’ve got pipelines, we’ve got testing chains. We’ve got all these beautiful pieces. We need to understand what needs to truly be done and how we can get it there.
Matty: [00:47:12] That actually, I think, is a really good way to kind of summarize where, where we’ve been. I like to walk away with something actionable. So like, if people are wanting to improve this type of work in their team or their organization, what’s like the first thing or like the actionable thing you would recommend they go and do tomorrow?
Dominic: So the first thing that I would recommend is— there’s actually 2 things. Sorry, if I can mention 2, because they’re both on number one. There’s one cultural aspect and then there’s one very, very technical aspect. The cultural aspect to me is if you never talked to a security person or you haven’t talked to somebody who is helping you or supposed to help you with the security stuff, start to build that relationship. It doesn’t matter. You can go there and talk about, for instance, anime or Star Wars or something. Just build that relationship, even if you don’t necessarily talk about the tech, because the moment that you talk about the tech, you’re going to beat each other’s skulls in. You don’t want that. But start to build that relationship. That’s the first recommendation that I have. Everything else will come with time. And then, you know, you can build up. But if you have an adversarial relationship, then you’re fighting against each other, which means that security has the opposite of what they’re trying to achieve, which is like operations fighting against them, not implementing what they actually need to implement, thus being less secure than if they weren’t there. And then the second one is a very technical recommendation that I have. And this one is around visibility. I think a lot of companies have a bigger problem with visibility than we necessarily admit to ourselves. And again, I’m not going to name the company. I was at a larger company a while back without more context to this. And we were asking simple security questions, not necessarily security questions, just when like a major breach was out there and we knew we had to deal with the vulnerability. We were asking visibility questions about where certain components were, how they were set up, what the architecture is, what the relationship of all these assets are. Putting your hands in front of your eyes and saying nothing bad is going to happen is not the solution. You actually need to get as much visibility as possible, even if it’s creating too much noise at the moment. Working through that noise, step 2, but you need to get visibility first. And so if you don’t have visibility, please try to get it and try to get an understanding of your infrastructure first. Then you can actually work through the prioritization, i.e., the second step and the fixing, the third step.
Matty: [00:49:38] Sue, what’s your, what’s your actionable way to solve all of everybody’s problems tomorrow?
Sue: So yeah, I mean, there is like buying your security engineer or your security team a beverage, but in this day and time with everyone remote, I think it’s really hard to build those relationships. So it takes an extra step. Maybe that extra step is getting inspired by DevOps party games and getting everyone for a 30-minute game of something. Number one. Number 2 is there’s a lack of shared understanding of what people’s goals are. We just had a conversation with a large organization, and the DevOps team thought that security should just be in charge of policies, and they should be in charge of all the tools. They never really— and they came at it from that perspective, and no one shared that with security. Security was all over the place, and then the DevOps team had the attitude of, why are they stepping in our space? Security didn’t know that that was the understanding of some unspoken agreement. So I feel like that there’s just like, just the pure like, hey, let’s get aligned. What are your priorities? What are our priorities? And making sure that how do we tie in to make sure that we don’t impact our end customer at the end, right? That’s the overall goal. How do we do these things and make our customers’ lives easier?
Matty: [00:51:16] So fantastic. All right, you heard it, everybody. So go fix everything tomorrow. You now know how to do it. So this has been super fun. And if we had a Patreon, I would say you could subscribe for the secret episode when we just sit and talk gossip with each other, but that’s not actually happening, but you could pretend you can try to figure out how to send us money on Patreon. That being said, this has been really, really interesting and educational to me kind of going over this and rethinking some of these things. I hope it was really helpful to you as listeners. And like Sue said, We would love to hear like success stories. You could either tweet them at us or, you know, you know, maybe give a talk somewhere. You know, maybe you want to come on Arrested DevOps. Let us know. But what you can do right now is go to arresteddevops.com/stilltalkingaboutsecurity for the show notes in this episode. We’ll put in some, some links, a couple of things we talked about. If you go to arresteddevops.com/itunes, you can leave us a review in the iTunes store if you want to help other people find the podcast. And yes, I still call it the iTunes store. Even though I think it was officially changed to Apple Podcasts like 3 years ago. But I don’t want to change the redirect, as everything old is new again.
Sue: [00:52:24] So who knows, they might change their name back.
Matty: We’re bringing iTunes back, people. You can also find us on Spotify and iHeartRadio, and we’re even on Audible. We’re found everywhere that fine podcasts are given away for free. Sue and Dom, thank you for joining me today. This is It’s been a delight. We need to do more things together. So thank you for being a part of this.
Dominic: Thank you for having us.
Sue: Absolutely.
Matty: And this is the rest of DevOps. And remember, there is always DevOps in the Banana Stand.
