Cloud Native Security with Michael Isbitski

Matty: [00:00:00] And that’s how we did security in the ’90s, yo. It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matty Stratton. We are going to dig into some really interesting insights around the industry, especially around the cloud-native and cloud-native security world. But before we get into all of that, let’s have a word from our sponsors. Collecting compliance evidence shouldn’t involve spreadsheets and scavenger hunts. With automated controls and over 75 integrations, Drata automates the process without needing to be an expert. Drata supports 14 frameworks including SOC 2, GDPR, HIPAA, and ISO 27001. Companies like Notion and LemonAid have shared how Drata simplifies audits through automated evidence collection. Don’t let audits slow down your dev cycle. Request a demo today and get 10% off along with waived implementation fees at drata.com/partner/arresteddevops. Feeling like you have too many alerts, overwhelmed by vulnerabilities, and at the end of the day not deploying apps as quickly as you would like? Sysdig hears you. Security in the cloud can be overwhelming and security posture is suffering. You need a way to prioritize what matters so that you can move faster. Shift left is the right operating principle, but you must shift left the right way. Sysdig roots everything it does in runtime insights. By knowing what is running in production, you can prevent, detect, and respond to threats and do it at cloud speed. To learn more about Sysdig, visit sysdig.com/arresteddevops. Let’s talk about one of the most exciting events in the DevOps community, DevOps World 2023. If you’re someone who’s passionate about learning, networking, and staying up to date on the latest trends, then attending DevOps World is an absolute must. So what can you expect from DevOps World? The list is endless. First off, get ready to hear from some of the most inspiring and innovative speakers in the industry. The sessions will cover everything from AI automation, cloud-native architecture, security and risk management, to continuous delivery. And the best part is that DevOps World Tour 2023 is coming to 5 cities across the globe: New York City area, Chicago, Silicon Valley, Singapore, and London. Find a city near you and register today at arrestedevops.com/devopsworld. It’s time to dig in and learn a little bit more about cloud-native security, and joining me is Mike Izbitsky. Mike, I’m so excited to have you on the show. Can you let our listeners know a little bit about yourself?

Michael: [00:02:54] Thanks, Matty. Yep. Thanks, Matty. Yeah, it’s, it’s really, really awesome to be here. I am Mike Kaspitzky. I have quite a long career. Everybody can check out my LinkedIn profile. I’ve been in industry for quite a while. I started as enterprise architect for a very large telecom, Verizon, probably know, and actually shifted into the security space, latter part of my, my stint at Verizon and assessing all manner of applications by Verizon, by Verizon Mobile. I’m sure many of you have used it. I will, I will not divulge some of the issues I may or may not have found. And actually left Verizon to get into research and advisory with Gartner, specifically focusing on the application security space. But I’m sure many of your readers can attest to application design as well as application security is massive, very quickly spills into infrastructure. So quickly found myself in the world of cloud and containers and Kubernetes and infrastructure as code and all those fun DevOps things. So I was at Gartner for about 5 years, you know, researching, advising all manner of organizations. And then I’ve kind of left to go into vendor land where I kind of spread the gospel of security concepts, certainly cloud and container security. I am a director of cybersecurity strategy at Sysdig.

Matty: [00:04:16] That is exciting. Yes. And thank you for not throwing shade at how insecure some of our listeners might’ve been when they were using Right? Be like, look, I saw what you all were doing, people, with your hotspots.

Michael: Yes, I’ve heard it all and seen it all. Never any dumb questions.

Matty: Well, I want to talk a little bit. I think we’ll be kind of focusing again about what security looks like in this kind of cloud-native world and what people are doing with it. And we do have, you know, Sysdig has the Cloud-Native Security and Usage Report, which we’ll have a link to that, by the way, in the show notes, and we will be talking about it quite a bit. But I’m thinking maybe that’s a good place to start because I’ve got some questions on your thoughts around a lot of things happening. But maybe if we start with some of the things we’ve seen from this report, that might give us a place to start. What, you know, you’ve been around the block here, you’ve seen a lot of stuff. We’ll just say that way. And was there anything surprising that you’ve seen coming out of this?

Michael: Yeah, quite a few. Some surprising to see trends go in one way versus the other. I mean, Sysdig’s been doing the research for a number of years and actually helped kind of author the report with some other threat researchers within Sysdig— Crystal Moran, A1, also a former colleague Anna Bellick from my Gartner days who is also at Sysdig. So yeah, this is— there’s a lot of data in the report, and I should stress that it’s actually based on customer data that is anonymized. So this isn’t survey-based, it’s actually real-world observations. It is Sysdig customers, right? So you’re talking about a slice of overall industry and then those specific organizations that have acknowledged they have a security problem and that are using tooling like Sysdig. So that’s kind of an interesting background, right, where it kind of frames the trends, right? So when you see trends like number of vulnerabilities being as high as they are, and I— we can certainly dive deeper into, well, what is a vulnerability, because your audience might not be fully in the security space, but that’s, that’s a pretty alarming one just to see how many vulnerabilities are out there. Kind of speaks to, I guess, a little bit of state of open source or hygiene of componentry. So organizations are faced with that, right? It’s like you have to source something, you know, as you’re building your apps and infrastructure. Nobody’s building from scratch, and it’s— some of it’s pretty poor, right? There tends to be a lot of latent vulnerabilities and the things you source. So then how do you, how do you kind of navigate those waters safely, and then deploy high-quality applications and infrastructure into production? So a lot of, a lot of Sysdig customers are— that is a big use case, it’s a big problem. So we saw very, very high stats on vulnerabilities that they would see, but then what’s actually in use? And this is something I, I used to research pretty heavily in my Gartner days, right, because it’s dependencies have nested dependencies or transitive dependencies. So it gets very complex very quickly. And organizations don’t always really see that until they start scanning. Right. And then it’s like, I scan and every time I scan, I find something. So that’s kind of the reality. And the picture is usually much worse than you can imagine at the outset. So, so those, those stats were kind of alarming cost, cost statistics. Some things trended in different ways than I would have expected, like human identity increasing over machine identity, you know, because the trend tends to be automation and services communicating, less so human users. But some of that, you know, is kind of virtue of coming out of the pandemic and organizations staffing up. So it skewed the trends a little bit that way. But yeah, there’s, there’s a lot of kind of interesting numbers that we could dig into for sure.

Matty: [00:08:05] Yeah, I’ve got it pulled up over here and I just was skimming around as you were talking. Yeah, it says last year we found that 88% of identities in Sysdig customers’ cloud environments were non-human roles and that dropped to 58%.

Michael: Yeah.

Matty: So I think that’s an interesting concept again, because you hear that and you’re like, which is it’s still the majority, but just barely. Yeah. Yeah.

Michael: And I did a double take quite honestly. I was like, is our data right?

Matty: Can we check our math? Yeah. Yeah.

Michael: So that one was a little surprising. Rising. And I talk a lot about the topic of like secrets, right? Because it’s like, well, they’re hardcoded credentials typically for machine identities or services so they can communicate. So we wanted to make sure we had the data right. But yeah, that was actually true, right? We kind of saw that human identity rise up. So that’s good, right? For organization staffing and employment. Yeah, absolutely. Different tide.

Matty: When you kind of think about this, you talk kind of about the supply chain and sort of like these transitive dependencies and where these things come in. You know, I think to, you know, KubeCon in LA at the end of 2021, it’s like all anybody wanted to talk about was SBOM, software bill of materials. We’re talking, we’re talking, we’re talking, we’re talking. And sort of now this has been, what was it, a couple of years now that we’ve been supposedly talking about that. And it kind of sounds like we’re just talking.

Michael: [00:09:25] Yeah.

Matty: I guess. Are you sort of seeing that? Like I’m looking at the report and it doesn’t look like we’re doing a great job about actually implementing this stuff, it’s still a big issue.

Michael: Yes, absolutely. And a very deep topic. You know, one of the other things I’ve been talking about more recently is the US National Cybersecurity Strategy, which the current Biden administration kind of restated some of, you know, what, what was already building. And, you know, one of the core components of that, besides things like zero trust architecture, which is very big in the security space, is the concept of software bill of materials, right? SBOMs. And not new, right? We’ve been talking about SBOMs. God, I mean, I remember back in my Verizon days, was the one of the vendors that was really prominent in the space. But, you know, we need to get a handle on all of the componentry that we use, but then those components, what do they reference, right? And it becomes this giant spider web of what you’re using, right? Your, your quote unquote software, but it’s, It’s massive, right? So like that landscape, and I, I don’t think we had specific data on how nested componentry, like references can get, but it’s, you know, the reality is there’s 2 problems, right? SBOM formats aren’t necessarily settled. There’s a few kind of formats in contention for what might be the, the standard for SBOM, but then it’s, you know, what does that SBOM look like? Like, it’s gonna be quite lengthy, right? ’Cause there’s just so much. And then it needs to be very dynamic because, you know, as we think about application design and system design, right, there’s the thing that you build or design, right? But then over time as you operate it, it starts to drift from that, which can create some security problems. But then it’s also, well, your manifest, right, or your SBOM is gonna change. So it has to be very dynamic. And then it also has to kind of account for all your partners and suppliers, right? Because think about complete supply chain and everything you do as a designer or developer, but also operating as a business, you’re working with different entities to do something end-to-end, right? So everybody’s kind of working with their own applications and systems and components. Those things then get integrated and connected. So it’s just much more complex than just kind of stating that original problem of, well, let’s understand the makeup of this application that we’re installing on our Windows desktop, right? It’s kind of not nailing the complete picture of what the complete architecture looks like and all its integrations.

Matty: [00:12:04] I wonder about a couple things too, and I’m kind of looking at, and maybe there’s some things that are more endemic to a type of technology that lend itself this way. The data might tell us that. Not to throw any shade at JavaScript, but we’re— there’s, there’s some problems, right? And so a couple of things that are punching out to me, by the way, we’ll put a link to where you can register and download this report in the show notes, because I think you’re going to want to look at it. In the report, you break down image bloat based upon the type, like saying, let’s look at Go versus Java versus JavaScript versus OS packages. And you look at JavaScript and it says fewer than 1% of JavaScript packages are in use at runtime. And I have my theory on this as someone who uses JavaScript, but not in a way that would be in the runtime. I feel like there’s a lot of build mechanic stuff that’s written in JavaScript. You know, I think about things like Gulp, right? Like maybe I’m the problem, it’s me, but like I have a website, it’s the DevOpsDays website, for example. It is a static site generator. Okay? So there is no vulnerability in its code, so to speak. Like it’s minimal, right? Yes, I can load JavaScript into the frontend, but for the most part, it’s writing out static HTML. If you look at my package.json, there’s a ton of packages in there.

Michael: [00:13:20] Yes.

Matty: None of which are actually used by the web app, but they’re used in the build process. So now I don’t want to say it means we don’t have a vulnerability, but that vulnerability is a little bit different because you can’t do an attack against devopsdays.org using that thing that’s in the package because it’s not in the built artifact. But if I look at the repo and I run— I look at things like Dependabot or look at these other tools, it says you’re insecure as hell, Matty, you know, because you have all these pieces. So I wonder if JavaScript being so tied into build systems, like, like I guess when I think about it, like when I’m building a Golang app, I don’t have like a ton of— my build system in Go doesn’t suck in a lot of other packages to build the Go app and the Go binary. So I wonder if that’s part of it.

Michael: There’s a few problems, right? And we try to— I mean, we have to kind of make some educated guesses on what might be happening, because it’s, you know, we don’t necessarily know, well, what’s the business case for this specific app or system for this customer, plus it’s anonymized. But there’s a few things, right? And I’ll say, kind of touch back on some of my experience as an enterprise architect and an AppSec practitioner was, it’s somewhat what you described actually with the web app design, right? There’s, it’s funny, you kind of think when you’re looking at a website, there’s 2 things that are prominent. One is definitely the JavaScript dependency problem. The other tends to be marketing tracking cookies or analytics pieces of data. There’s a lot more there than you anticipate. Then you plug in your favorite web app debugger. Maybe if you’re in the security space, it’s something like OWASP ZAP, but you start to see all those things that get tracked, but then also all the JavaScript that gets referenced. Typically what happens is, you know, if I am developing the website, I’m going to source some components because I don’t want to create the entire wheel. I’m going to kind of pull a library. I don’t necessarily think about, well, what are the 100 libraries that that thing’s going to reference? And I probably don’t care about all the security issues with it. And then that you’re— if all things go well, you’re going to deploy that thing to production. Then your marketing team might kind of take a look at it like, this is great. We know we need to monetize this more, kind of promote search engine optimization.

Matty: [00:15:41] They’re going to start tacking on more JavaScript.

Michael: If there’s payment processing, that could be yet another JavaScript library. So it speaks to a few things, right? One is that JavaScript is very prominent. There’s a lot of ways you can enable business functionality through JavaScript. So it’s just very prominent in kind of all architectures. But that also creates that security problem, right? The nested dependencies, plus it’s just very dynamic because all those libraries could be changing, right? Particularly if you’re not pinning to a specific version, you’re going to get the latest. And who knows if that has particular problems with it because you might not have scanned it. So that happens all the time. That’s very common in front-end design. And I’m glad you kind of called out your specific scenario with the website. But there’s also kind of back-end infrastructure, and those things would be, you know, engineers are going to be sourcing from infrastructure as code and container images, likely pulling from public registries and repos, but it’s very much the same problem, right? Particularly if that container image then has some JavaScript artifacts in it and that you kind of referred to it as like a build process, but it’s absolutely correct, right? It’s your CI/CD build process. You’re going to build that thing, deploy it, and that JavaScript is now going to be in there, right? But that’s one kind of— it took me a few minutes to talk through that. The other is, is it reachable in code, right? And this is where Sysdig comes in with the whole concept of runtime insights. And we’re talking about it more as this concept of in-use or in-use exposure, but it’s kind of this big problem that’s been in IT for— I mean, man, it’s like as long as I can remember as an architect and AppSec practitioner, it’s just like, you, you don’t know all of those pieces, like who all these business groups that are tacking on things. And it’s like, if you sat there and looked at it, you’d identify all the components, but you have no idea what the actual business case for it is. And you’re not going to disable that component. So there’s kind of this massive risk acceptance that, well, somebody must have blessed this, right? Because it’s there and I’m just not going to worry about it. But it it’s kind of everywhere, right? And then if it’s in backend infrastructure, there’s very real security vulnerabilities or potential security issues that if exploited could result in things like landing and expanding in the infrastructure, moving laterally, escalating privileges, executing code remotely. So very— it’s a potentially very disastrous outcome, but organizations are essentially flying blind, right? Because you could run a scanner, you’re going to know every dependency that’s there, and you’re going to see all of the known vulnerabilities, but you really have no idea if that code is reachable in runtime. So you’re kind of taking a best guess at what the actual risk is and if something is exploitable.

Matty: [00:18:41] That actually brought— and that was, that was interesting. And it talks about in the report about being able to know, saying what matters, which percentage you’re seeing are available at runtime, and where that comes in. And And just to be clear, like I didn’t necessarily bring this up to say like, oh, I think this is a flaw in how you’re presenting the data. I was interested to me about also, I wonder how many people end up doing the, oh, well, we don’t use that much. It’s all in our build system. So I’m going to ignore this. And then you’re like, cause I could do that. Like I listen, you heard, y’all heard me do it live. I went and said, oh, there’s no JavaScript vulnerability because it’s only used in our build system. And then I’m like, no, we use Bootstrap. There’s frontend JavaScript. I, and even I sat there and said, oh no, it’s fine. Yeah.

Michael: So yeah, I mean, I used to, I would, it’s given me recall back to Verizon days. You might say a flashback. I mean, it depends on the development team I might be arguing with. I mean, I, as a security practitioner, I always tried to be more cognizant of, you know, what engineering teams are faced with, right? They’re always under the gun. You have to deliver business functionality. So it’s very easy for a security practitioner to be like, well, how didn’t you— that’s like Security 101. How’d you mess that up? Right? And it’s like, it’s just not acknowledging the reality of how systems are built and the number of vulnerabilities, dependencies that creep in, and then the vulnerabilities those dependencies might carry. But even doing things like static analysis of code, which could generate something like an SBOM, going back to that question, But it’s like, well, what do you do with that once you’ve assessed it? You know, and a lot of engineering teams would just suppress the issue because they’d say, well, it’s an open source library. I don’t own that and I’m not going to fix the code. So it’s kind of that risk acceptance. Sometimes it gets described as false positives.

Matty: [00:20:27] Normalization of deviance too, right? You know, where it’s always our security things are always red. That just happens. Yes. Yeah.

Michael: I mean, it’s noise, right? We talk about that a lot at Sysdig also, which just, there’s a lot of noise, right? And it’s like, it gets worse as you get into cloud native, because there’s just, there’s just many more things, right? And then it’s more ephemeral. That’s another thing we talked about in the report. But like, if you’re newer to cloud, I mean, a lot of cloud adoption tends to start as like, well, we’re going to consolidate data centers, we’re going to move virtual machines into cloud, and they’re consuming infrastructure as a service. But if you really want to take advantage of cloud scalability and elasticity, you start getting into cloud native. Architecture and then using things like container services and Kubernetes and serverless abstractions. And if you’re designing your app to be more microservice architecture as opposed to a monolith, it’s going to amplify yet again how many types of components you’re working with, right? And then those things get powered by containers typically. So these things kind of jockey back and forth, but the end result is there’s just a lot more— I don’t want to say crap, but it can feel like that, right? And it’s like if you’re staring at a dashboard, it could be a sea of red. For sure.

Matty: [00:21:36] Yeah, I mean, the distribution, you know, you think about, you know, we talked a little bit before we started recording about, you know, what we, we did before. And I think about, you know, my, my, my couple decades working for a living as a system engineer and a sysadmin was thankfully in like the, the client-server ASP LAMP stack days when it was like I actually could know everything about my infrastructure in my head because it wasn’t that complicated. I thought it was, By the way, now I’m like, oh, you poor summer child, you sweet summer child. You have no idea. You had just a bunch of databases. You didn’t, you know, so it’s that distributed place. And I think one of the things that’s, that I’ve learned in my years in this industry, and especially on the side of, you know, evangelism and things we’re doing here and talking about products and everything is nuance is hard. And the shorter the phrase, the more nuance is necessary and the more capability for understanding. There’s a, There’s a great little part in the Robert Heinlein book Stranger in a Strange Land when the character Michael from Mars is trying to understand why he said something that upset people. And he’s like, he noticed that in English, the longer the word, the less people got upset. And this was a very short word and the word was God, right? You know what he was like? So, so I’m going to talk about shift left. So I feel like we fight about this a lot and I think it’s because, well, right, because what does it mean? It’s a short phrase. And it’s depending on what you mean by it, it’s either incredibly frustrating or it seems like it should just be common sense. So we’ve had a couple different episodes about this before. We had Tanya Janka on years ago talking about shifting left. I gave a talk about shifting left so many years ago before when I thought I kind of thought of it myself. I didn’t, by the way, but you know, it was a talk that was— this is enough years ago that I can now say this. It was because I was frustrated with one of my customers and their sysadmins. And so I wrote a defensive talk to talk about why they were wrong. They didn’t know that, but it was talking about moving left securely. And it was the idea, again, it was fundamentally based upon the idea that just that your title does not imply infallibility unless you’re the Pope. And sysadmins are not the Pope actually, you know, and we, we sort of have this idea. Sorry, I’m going off on a little bit of a tangent, but this will connect back to what we think shift left means is that this going back, well, I can’t give you access to this in production because you’ll mess it up. And as operations folks, we tend to sort of have this approach, but I’m like, as if because I’m in TechOps, I am infallible. But I’m like, all of us go out to the pub and have a couple pints, and every single person that works in system administration will tell you about the time they completely fat-fingered something and cost the company $10 million or whatever. Like, we make mistakes like everybody else. Anyway, so when you think about shift left, kind of I’m curious to know, like, kind of your thoughts on it and what you think either people are interpreting— I’m hesitant to say correctly or incorrectly, but I feel like in this one there’s a right and there’s a wrong.

Michael: [00:24:30] So yeah, no, it’s— yeah, and I love that you kind of brought up the, like, the art of words and language, and it’s very true, right? It’s kind of hard to convey nuance in— even in dialogue, right? It just takes a lot of time, right? And if you’re writing words God help us, right? You could definitely set somebody off. So shift left. This is not going to be a concise answer to my whole point about communication, but it’s a podcast, Mike.

Matty: Nothing that happens on a podcast is concise, right?

Michael: Yes. It’s a lengthy dialogue. I mean, shift left to me, right? It’s like I have a lot of experience in industry, so I’ve kind of seen, you would say, the traditional model, which people would describe as waterfall, right? It’s kind of, we’re going to build the thing. We’re going to do as much testing as possible. Hopefully that includes QA, load testing, system integration testing, all that user acceptance testing. Does that always happen? No, right? And that could be another hour-long conversation. It’s like, it’s going to depend on the criticality of the thing you’re building, like how much effort actually gets pushed into that. And then within DevOps, there’s a lot of— or was efforts to automate that, but sometimes organizations will just forego it. And then on the security piece of it, it’s kind of like, well, you know, we know as an organization we have a lot of potential risk. We’re obligated by our internal compliance to do certain things, and then regulatory mandates also apply, right, for your industry or the types of data you’re working with. So there’s a lot of things you need to check. Traditional security might have been more compliance-focused where it was questionnaire-based and just kind of interrogate the engineering teams, which is a difficult proposition. That’s kind of the world of governance, risk, and compliance. Some organizations still operate that way, right? They don’t have modern InfoSec or like DevSecOps approach, you might say. But it’s— that is kind of the world of secure design, right? It can be very manual. It’s very hard to automate. There’s not necessarily tooling that can do it because you need a lot of human experience and expertise, then that becomes a problem in itself, right? Because I mean, you, you kind of described it in the scenario of like how things get built and operated. There is no one subject matter expert that knows all pieces, right? Some of us may say we’re full stack, but do you know everything about every piece of technology and, and every single app? Like, no, right? And then the interplay between them adds new complexities and variables you never even think of.

Matty: [00:27:00] Full stack is another one of those where the nuance happens, because if you’re a JavaScript developer, it means something. The of the world hears it literally. And as my friend Paul likes to say, oh, you’re a full-stack developer. When was the last time you wrote a device driver? But if you ask people in the JavaScript side of the world, to them full-stack just means front-end, back-end.

Michael: Yeah.

Matty: Front-end, back-end JavaScript, which is fine that that’s what it means there. But it’s sort of like, that’s where I think a lot of the argument comes in where someone says, I’m a full-stack developer. And then you ask them about networking and they don’t know. And you’re like, ah, what do you know? And they’re like, But full stack to me means front end, back end JavaScript. And so you’re not misrepresenting. So it’s hard.

Michael: Yeah.

Matty: Yeah.

Michael: I’m right. And then the network engineering spot piece is dead on. Right. And I had a lot of background in that, like network engineering. I don’t really fancy myself a network engineer, but I know enough to be dangerous to understand enterprise architecture, the security problems that can result from the network kind of engineering aspect of it. But yeah, it’s very rare to find Somebody that’s going to know all of those pieces. But yeah, full stack’s another. Zero trust is another that’s kind of, can really get people’s, get them in a tizzy. But yeah, it’s, it’s so yeah, secure design, right? It’s like, how do we make sure everything’s built securely? And then there is kind of that utopia. It’s like, well, if we did that, then we’re never going to have security problems. And that, that, that sometimes is where secure design becomes extremist. It’s just over the top. It’s not reality, right? Because we talked about nested and transitive dependencies and supply chain with partners and suppliers and how things get integrated, you’re always going to have security problems, things manifest in runtime. So you should focus on trying to design securely. And then the other piece of shift left is really automating a lot of your testing, right? Because it’s kind of, we need to push as much security as we can into early design, development, and release. So test the code early and often, test it within the IDE, test it upon code commit to Git repos, test it within CI/CD builds, and then maybe even test it again as it’s running, right? And then you get into more dynamic testing. But it’s— I mean, I just described 4 kind of pieces where you test, and it’s like each one of those is going to scan results. And then now you have a correlation problem. It’s like, am I finding the same problem That I just reported, right? Because you might’ve reported as a defect. Now you’re kind of swarming a developer or development team. Maybe it’s an outsourced contractor as well. So it’s, these are the problems that usually arise with shift left. And it’s, it’s like a lot of the problems you had in traditional waterfall. It’s just, you’ve kind of pushed it to the early stage. So yeah, it’s, it’s, it’s an interesting topic to unpack. Hopefully I did some justice to what I, what I think shift left is, but it’s, I think it’s, it became kind of a catchphrase to get people thinking like, we need to really incorporate security into our early thinking and then test early, as opposed to let’s taste— test when it’s in production, that it’s too expensive to kind of go back and fix things.

Matty: [00:30:02] Well, I think there’s a few things that, that come with that too. And so like one, I just sort of work backwards. You talked about, you know, like even testing and when it’s in production, and it’s— I always like to think about the idea of monitoring is just testing with the time dimension, right? You know, and that’s one of the things that the overall practice applies to security and other pieces of that where If you’re testing something before it’s released, you need to be monitoring for it and vice versa. Because if you don’t have parity between your monitoring— yes, I know we’re in the world of observability now, nobody monitors anymore, but yes, you do still. It’s yes and. But you need to have parity there, right? Because if you have a monitoring check for something that you don’t have a test for in your build, well, great, you’re gonna flip your monitor the second it goes to prod or vice versa. I’m like, if this is important enough that I’m testing it in pre-prod, But I don’t care enough to test it in production, then is it that important?

Michael: Yeah.

Matty: So that parity is strangely, or maybe not, like I think I’ve seen a way bigger gap than anybody wants to admit.

Michael: Oh yes.

Matty: [00:31:02] And it’s because of different responsibility. Yeah. I think with the shift left thing, and you, you, you, you spoke to this in quite a bunch of different ways, is like many other ideas, you know, at, at a certain level of the organization, these things are looked at as cool, I can get rid of people. And shifting left to me is not shifting the work to the people on the left. It is actually moving that domain expertise earlier in the conversation. So, this is why NoOps never happened, right? It tried. They tried. They said, cool, we’re going to give our software engineers everything that ops does. And then it turned out ops is actually a skill and knowledge and a domain area of expertise. But what we can do is we can bring the operations folks earlier into the part of it and it’s connected. And I think it’s the same thing. It’s like expecting your software engineers to suddenly become domain experts that your InfoSec folks are, who are super experts, is unfair. And also to be quite fair is a little bit insulting to people that work in the security industry, right? It’s, you know, like if someone comes up to a graphic designer and says, cool, can you teach me everything about Photoshop? This afternoon. Like, you know, whatever, you know, I don’t, I don’t like, I don’t like referring to Dilbert too much, but there is one where it’s, there’s a bit where it’s like, because I don’t understand it, it must be easy, right?

Michael: [00:32:19] You know, and let’s ask ChatGPT, right?

Matty: Right. It’s coming away from the waterfall piece, but if it’s thinking, again, it’s all a DevOps thing, right? It’s a model of, it’s cross-functional, it’s, we’re thinking about the problem earlier, but we’re not taking that and dumping it on the people who traditionally do the work earlier in the flow.

Michael: Yeah, and I’ve had, I’ve had far too many advisory calls while at Gartner talking through this problem, and it— you, you hit the nail on the head, right? There’s— there was definitely a desire to automate, maybe headcount reduction, but it was also, can, can we push the burden of scanning onto other teams because we can’t scale the security team? There’s not enough of that expertise even available, so let’s kind of get app dev teams or SWE to handle it. So there was a lot of that. You know, there’s other problems that pop up too, and you started to hit on this, and it was like when you’re talking about test cases. Well, yeah, you need to build proper test cases to even exercise that functionality that you might find potential security problems, but there’s also an even deeper problem, which is the efficacy of like scanning tools and even kind of best of breed, some of those dynamic scanning application security testing tools or DAST, I mean, they’re kind of glorified fuzzers, right? And it’s like without good test automation, you know, things like Selenium scripting, you just weren’t even going to hit those functions, right? And it’s like you’re time-bound if you’re thinking in like a larger release window. And then even presuming all those prerequisites are in place and you completed a scan, it’s like, well, now you have probably found some issues, and then what do you do as a release decision, right? Because most organizations, they’re like, it’s all about release velocity. That’s why they went all aboard the DevOps train. Then it’s like, somebody’s— this is kind of, it’s glorified like change management, like who’s going to decide if this is good enough, right? And then how do you automate your pass/fail build decisions? So like that, none of those problems really got addressed. We just kind of shoehorned a lot of the testing earlier into the lifecycle.

Matty: [00:34:26] Which did— I mean, let’s be fair, things are better, right? Oh yes, absolutely. So, and I even think back to a couple things, and I’m— my brain is forgetting the name of the tool, but like one of the things I talked about in that talk years ago was about democratizing your security tools. But I remember places I’ve been where even if you wanted to say, even if everybody was on board to say, let’s take our you know, security scanning tool that we have and let’s bring it earlier in the equation, the CFO would lose their mind because that was a $20,000 a seat. Well, no, because they were crazy expensive. They were like $15K a seat. How do you, how do you do that? And so I do think that as an industry in the tooling, that has changed. Things are more accessible, which lets us use them more. I mean, I can think of lots of tooling that I had in place as in TechOps that I was limited by the cost, right? I was like, hey, I wanted to do, you know, code instrumentation and it was $50,000 per frontend server. So I only ran my traces on one of my 30 boxes, you know, because it’s crazy. But you talked about the turning into change management, right? This, I do think one of the advantages of moving things earlier is the release risk goes down a little bit because First of all, you know, the earlier to introduction of a defect we find it, the easier it is to pick it. Traditionally, and I say traditionally in an agile way, but like, how did we do this stuff, right? We did QA testing all the way through, blah, blah, blah. And then at the end you had a hardening sprint. So you had this one sprint, which is when now we bring in security and they’re going to run everything and they find everything that’s wrong. And we don’t do a damn thing about it because this was the last sprint before release. And the GM’s gonna lose their mind. So we get a note from the security team that says it’s okay. And you know what? Bad— I like to say bad guys on the internet don’t care that you have a note from your parents that said it was all right. Signed Epstein’s mother. So, and those decisions rarely are made in the context of acceptable risk. It was more of, well, it’s acceptable risk, but the risk is outweighed by the fact that we have made business decisions that the, the train is too far down the track. But if we are able to catch it a little bit earlier, we can adjust to that. But just like Agile— to do Agile right requires your whole organization to change, and few did, you know— doing this type of changing how we think about security, it’s not just about the engineering group that has to change. How we think about product release has to change. How we think about sales has to change and it generally doesn’t.

Michael: [00:37:08] Yeah.

Matty: Yeah.

Michael: That’s kind of the dream, right? Yeah. It’s very true. Right.

Matty: But then you can’t be surprised when it doesn’t work. That’s— yeah.

Michael: And I remember like the security testing, it’s like, there was, you know, Verizon, there was plenty, plenty of teams and it was a mix, right? It’s a mix of waterfall and agile methodology. Some teams were more DevOps, some were getting into cloud, all different tech stacks. So it’s like, it’s a nightmare, right? When you kind of zoom out and you think about for a given organization and all their apps and systems they need to look at. There’s just too much, right? And it’s like, you have to think about your entire risk posture, not even just that one app in isolation. But yeah, it’s kind of like, it’s a very slow-moving ship traditionally, particularly on the security spectrum. And by the time you can get a security engineer engaged to look at an application you might be trying to do end-to-end, it might take them 2 weeks, and then the code changed, right? Because that’s the nature of agile methodologies and DevOps practices. So it kind of invalidates your security results, but, you know, organizations will have, they have to kind of turn a blind eye to that because it’s, I mean, you described it, right? It’s like, it’s a culture change. It’s kind of part of digital transformation. It’s not just the tech you’re working with. It’s also how you work together, people, processes, and technology, right? Not to use a clichéd saying, but it’s just very true.

Matty: [00:38:28] Right.

Michael: And I’ve seen it time and again, like in my own experience, but then also advising organizations.

Matty: One, one of the things I’ve thought about, and this was more like when I was at PagerDuty, just thinking about reliability, right? And so about thinking about reliability as a feature, and security falls into that. And that was one of the conversations we’d have when you think about product ownership and product management. Like, when you’re looking at the features of your product, security and reliability are features that— and they’re features that your user wants, but they don’t say it that way. And so, right, like, and I’m curious in what you’ve seen with working with these organizations, is that something you’re seeing? Are people considering security a product feature? And by that, I don’t mean it’s a feature that we can charge more for, like, you know, OAuth, like, but as in being a secure product, not AuthZ and AuthN, like, yeah. Or is it still considered an InfoSec problem that’s around the organization and it’s not at the product Like product science answers, yes, yes, it’s all good. Okay, as long as it’s getting better, that’s all I want to hear is that we’re getting better.

Michael: [00:39:35] I mean, there’s still the case that like, well, can we market a security feature? That definitely still happens, right? Because it’s product, everybody’s— not everybody, but certainly every business is trying to make money on the thing they’re building, right? They’re investing time and money to do it. The— I think there’s Definitely bigger emphasis on security, and I’m glad you mentioned democratization of tooling, but also information, right? So like organizations like OWASP, Open Web Application Security Project, there’s just a lot more material out there and ways to train so you can become very familiar with why these security issues do become major problems, right, and can lead to incidents and breaches. The other thing that brought a lot of focus, I’d say, in Onto InfoSec as a problem space. I mean, you could say cybersecurity. I mean, cybersecurity is kind of— it’s InfoSec kind of evolved to include the physical or real-world things, right? Like IoT, operational technology, and you get things like safety and resilience become part of the equation also. So cybersecurity is one kind of way that that’s been renewed. Which is kind of where I’ve kind of swung back to looking at the National Cybersecurity Strategy and digging in there. The other tends to be privacy, right? Because it’s— you start to touch on, as a human being, right, there’s like that expectation of privacy and the things you share and how it can be used against you. So I think that touches people more deeply, that they’re more incentivized to care, right? So if somebody leaks something about you, it’s a problem, right? It’s like, I hate getting the letters. I mean, I don’t think there’s been a month that’s gone by that I haven’t received a notice about some kind of breach. And it’s like, man, nobody can seem to hold on to data. And that’s kind of the reality of system design, right? It is that complex, right? And how do you actually protect data? But yeah, I’d say the privacy piece really brought things back to the forefront. And you could probably trace that back to GDPR. Regulation, kind of protecting privacy of EU citizens. But I mean, those things are expanding, right? There’s other legislation in the works to protect privacy of US citizens as well. So I mean, privacy is that new lens. And it’s interesting, like some security programs are evolving much more so where they have dedicated privacy officers and privacy functions. But that’s certainly brought a lot more concern from organizations that they should care about security problems because the privacy impact can be even more damaging, certainly monetarily, but also brand damage. And those things also come back into play with, I mentioned the National Cybersecurity Strategy, but SEC disclosure mandates as well that are bubbling up.

Matty: [00:42:26] So I think the other, I guess, buzzword or for lack of a better term, but just to sort of— I’d like to get your take on it, is around zero trust because I’m not a complete expert on it, even though I have a talk I’ve been giving that has zero trust in its name. But that’s because it’s called Zero Trust Is for Your Network, Not Your Teams. And it’s actually a talk all about people and team trust. And I’ve actually had a couple of people who’ve seen it, or the reviews have been like, maybe you talk more about zero trust and it will help your metaphor. And I’m like, okay, I should learn more about this. So, but also, you know, I hear a lot about it and I wonder how much of that is very similar to the shift left thing too, where it’s like, it means this, but what does it really mean and what are people really doing? And the report makes it sound like people aren’t doing much.

Michael: Yeah, and it’s true, right? So I guess let’s try to arrive at a definition of zero trust. And it’s, I mean, I typically start at the concept of least privilege, which is generally well understood amongst the security community, but you should be granting only enough access that somebody can do their job. Or get the function they need, right? If you’re thinking of like consumer use case, that’s least privilege, right? And it’s like how that plays out in reality can be incredibly difficult, right? ’Cause if you think about all the chains of access controls that you have to go through to connect to all of the pieces of a system, right? The database, the backend infrastructure that serves it all, frontend, the identity authentication and authorization mechanisms, all of that is important to enforcing proper access control. So I mean, that’s part of the least privilege discussion. Zero trust gained momentum. I think it was Forrester kind of pushed this, and I don’t know who’s even accredited with it originally, but it’s, it’s kind of least privilege on steroids. It’s, well, always assume the environment’s compromised also, right? So you need to be doing all of your authorizations continuously, right? Because you, you just don’t know. Maybe the front end was compromised, maybe pieces of the backend infrastructure are compromised. Maybe the endpoint is compromised. The endpoint could be a desktop, could be mobile, right? So it’s like, it’s a lot of those converging forces of like modern system design, but then also consumption patterns, right? The environment is just, it’s, it’s a very complex beast. We’re not all living in a data center accessing an application. So Zero Trust architecture is like building on those least privileged concepts, enforcing like dynamic access controls, and then doing assessments of client code and client endpoints to make sure things aren’t compromised. Does also include things like zero trust network access, right? And that’s, I mean, you mentioned this yourself when you asked me the question. Typically when people think zero trust, they think zero trust network access or ZTNA. It’s definitely part of the equation. And, you know, the other way, like when I was doing Gartner advisory work, sometimes clients would ask, you know, like Google BeyondCorp, Right, was kind of the thing. And it’s, I mean, eventually became a product you could buy, BeyondCorp now, but it’s a ZTNA product, right? It’s kind of VPN, if you will, but maybe more cloudified, so it’s more usable. But the other one that was in the mix was BeyondProd, which really it’s Borg, which is Kubernetes, but it’s like, how do I build my entire enterprise architecture to also include things like zero trust concepts? So There’s a lot to it. Like shift left, it does tend to be glossed over, right? People cling to one piece that they think is very actionable, like the network restriction piece of it. And actually, you were, you were talking into this a little bit earlier, and it got me thinking, like, a lot of times I get clients asking, like, well, how do I secure development, like secure the development environments, right? You start getting these discussions about isolating networks, particularly if it’s an offshore engineering team, and it gets messy, particularly when you’re talking about like DevOps and like cloud-native development. It’s like, how do you get all these moving pieces to work properly? And that does always happen. But there’s definitely, I’d say, regulation and security requirements are sometimes at odds with that, right? Because they will promote, you should actually define a very hard network boundary on this, that’s actually going to create a lot of problems for code commits, building, releasing, operating the thing. So, you know, zero trust sometimes gets overloaded to it’s going to be the cure-all for all that, but it’s a very big problem, right? And it’s actually very prominent in the National Cybersecurity Strategy also, but the government’s given themselves, I think, 10 years to get good at it. So we’ll see how that goes. Yeah, 8.

Matty: [00:47:11] Yeah, exactly. Yeah, no, it’s funny because you talk about that and I think about, yeah, again, to me, it falls into, to me, I, was the, the zero trust network access. And again, just from the context again of the grizzled old sysadmin who was like, how did we do things? It was like, you know, was very much perimeter security.

Michael: Yep.

Matty: And then it didn’t matter because how are you going to get in, right? And you fundamentally— and it was probably mostly okay, you know. But again, because it was— I had 2 data centers, so first of all I had 6 VLANs I was dealing with, that it was easy to block and just say, this is only what’s allowed through there. But also in today’s vulnerabilities, that still wouldn’t necessarily probably work today because even with what was, even with the way we would construct a DMZ, those vulnerabilities are not coming from a pure shell. You know, you can’t handle everything with ports. We used to be able to a little bit better.

Michael: Oh yeah, yeah. The firewall contingent is still strong. And I’d be remiss to say, you actually mentioned the system report, which I I thank you for. I should refer back to that, right? So there is actually a stat about the permissions just not being—

Matty: [00:48:16] 90% of granted permissions are not used.

Michael: Yes. Yeah. It’s like, wow, we’re still having this problem. And it’s like, yes, we haven’t even fixed like least privilege, right? It’s like you shouldn’t, you shouldn’t have active identities that aren’t even being used because they could be compromised. It’s a security risk.

Matty: This is a whole big conversation, but I kind of wonder like part of what causes this, and I would, I I think is onboarding is hard. Let’s just, let’s just take, take ourselves all the way away from technology and just think about in an organization, right? When you started a new company, what’s the usual thing that happens is you sit there and you’re like, okay, we’ve got Mike, the new developer. Okay. Access management, clone Jimmy and make Mike look the same because they kind of have the same job. And it makes sense. And we, we, we do this. It’s not just about people. I think about this flashback to like my chef days is people would say, You know, you go and you’re trying to build like infra code to build an Apache server, right? And you’re like, well, how do you build an Apache server? They kind of hem and haw and eventually it’s, well, we just sort of take an existing VM and copy it, right? And so you do that enough and you get to the point that nobody actually knows how that box gets built and how much junk is on it that was needed at one time, you know, and all this. And the same thing happens with permissions, right? Where you’re like, okay, Because most of the time, I shouldn’t say most of the time, I don’t have data, but 90% of granted permissions are not used. Make me feel comfortable to say this probably happens most of the time. We are doing the same thing we did with the Apache servers, but with our new hires.

Michael: [00:49:46] We’re saying, yeah, exactly.

Matty: You have a similar job to Sally. We’re just going to give you all the same rights as Sally. And versus the other way, I will, I will tell you in the company I’m at who takes security in a very serious way, which is good because we host a lot of data. It is definitely least privilege. And the way is we will give you almost nothing and you are going to be real annoyed for a little while because you’re just going to keep bumping your head. And that’s when you’re going to know you need a thing. Yeah.

Michael: Damage the user experience.

Matty: But, but, but if your company culture is such, you can do that. We also have a company culture of it takes you a minute to get started. So it’s okay that it’s going to take you 30, 60 days to get sorted as you figure it out. But otherwise we could go the way, which is like, oh, You know, Maddie, you’re, you’re doing this while the other people in DevRel, who this, this, this person who’s been here for 3 years, who probably had a different kind of a job in the beginning, or, you know, so, um, yeah.

Michael: And I love that the onboard, like thinking in terms of the onboarding, right? Because it is a very human experience, right? You start somewhere, it’s like, it could take like a month or months to get everything you need and you might even give up, right? It’s like, oh, I don’t need, I’ll ask somebody else to help me that, right? And that’s, a problem, right? Because it’s like, well, they’re doing something for you, that’s not really least privilege in itself. But yeah, it’s like, it’s exactly the problem. You start with the vision of how it should be, right? Group-based permissions or role-based access control. But then, you know, everybody’s unique, they’re doing different things in the business oftentimes, right? So they start to need more granular permissions, and then you deviate from what that original group permission or role was, and that, that’s kind of the reality, right? So it’s like least privileges seems simple in concept, and same with zero trust, but it’s like in practice it’s really difficult to, to execute on.

Matty: [00:51:33] I’m going to make a security confession and then we’re going to, we’re going to wrap up, but then maybe the way we’re going to wrap up is maybe I’ll get you to make a confession yourself too.

Michael: Confess all your sins.

Matty: So my Netflix password is still a variation of the domain admin password when I was at apartments.com almost 10 years ago. Because that was an impossible password to change. So it’s emblazoned on my head, in my brain. And my Netflix password can’t be super complicated. It’s better now, but for the longest time, if you wanted to log into Netflix in a hotel, you had to type your password in. Like now you can do the app password, the app thing. So I was like, it had to be one that I was really good at. I would never mess up and I had it memorized and—

Michael: Oh yeah.

Matty: Yeah. So the security, I guess the confession is really just, we never changed that password even when somebody left. Because it was— fortunately, I think in the entire like 4 to 5 years I ran tech ops there, we only had one person who knew it leave. So that— so part of it was it was— I was able to get away with it by I had really good retention on my team. So yeah, you can have bad password policies if people don’t quit.

Michael: [00:52:37] Yeah, yeah, password policies are true.

Matty: Not true, not true, not true.

Michael: I, I, I’ve— I mean, I could share a lot of bad things I probably shouldn’t. I’ll say this, like for myself, I certainly use some bad passwords because I also have a 3-year-old son. Doesn’t always afford me a lot of time to think of a very long, complex password or invoke password manager. So I will oftentimes revert to a simple password and one that my wife can remember also.

Matty: Yes.

Michael: So yeah.

Matty: Every now and again, people will say, I don’t understand why password aging matters to a certain point or certain policies. And I said, well, let me explain to you. In Windows NT, 4.0, before, before the Active Directory, you could say a password would expire and you could say you can’t— it can’t be the same as any number of passwords, but there was no minimum age of a password. And the way our password policy was set up in that company— it’s a different company— was passwords expired after 30 days, but you started getting warned 15 days in, which meant every 2 weeks you were being annoyed to change your password. So, you know what we had on my TechOps team? We had a little Visual Basic app that would go and you would type in your current password and it would go change your password 15 times. And the last one would be to change it back to the last one. And that’s how we did security in the ’90s, yo.

Michael: [00:53:56] Yeah, I remember those splits. Yeah, NTLM, NTLMv2, password length. Fun stuff.

Matty: It’s gotten a lot better. So, this has been great. There’s obviously security is a huge topic. We have many more things we could dig into. If you go over to arresteddevops.com/cloud-native-security, which is what I decided this episode is called just now. So arresteddevops.com/cloud-native-security, you’ll see the episode show notes. We’ll have a link to the Cloud Native Security and Usage Report and any other upcoming things that might be happening, any other materials that we think of, or maybe none. So don’t come yell at me if the only thing in the show notes is a link to that report. If you go to arresteddevops.com/itunes and leave us a review in the iTunes Store, that can help other people find the podcast. And yes, I still call it the iTunes Store even though it’s been called Apple Podcasts for like 6 years, but I’m too lazy to change the redirect. You can also find us on Spotify, iHeartRadio, Audible, all the places fine and less fine podcasts are sold and given away for free. So Mike, thanks for joining. Where might people find you coming up? Are you gonna be at any conferences, any events, any streams or things?

Michael: [00:55:06] Yeah, I’m actually going to be at a Gartner Security and Risk Management Summit in a few weeks. I think it’s a little under a month now.

Matty: Nice.

Michael: So certainly if you’re, if you’re there, come find me. But I am on LinkedIn. I try to check in there at least once a week. If you think anything, any, any questions or war stories you want to share, certainly do reach out to me there. I live and breathe this stuff.

Matty: So I don’t know if I haven’t been to a Gartner event in so long, but I used to go and they were the best sessions to me were what they called the rogue sessions. So there would be a Gartner session where it was a Gartner analyst speaking contrary to Gartner’s position on that topic. And they were actually some of the most fun ones. So, but that said, this is not Gartner. This is Arrested DevOps. And remember, there is always DevOps in the banana stand.