Ian: [00:00:00] Attackers actually generally are unconcerned with whether or not you have your compliance boxes checked.
Bridget: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout. Today we’re live at the Minneapolis-Dallas meetup. We are talking today to Ian Coldwater about their KubeCon keynote and Kubernetes security. But first, a word from our sponsors. The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener, there is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul, and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app.
[00:01:37] Ian, I am so excited to have you on talking about Kubernetes security. First question, Is Kubernetes security possible?
Ian: Bridget, I’m so excited to be here on Arrested DevOps talking about Kubernetes security. And yes, in fact, Kubernetes security is possible. That said, Kubernetes is not secure by default. So if you assume that it’s going to be secure out of the box, you may be unpleasantly surprised. It is possible to secure it, but it does take some work.
Bridget: I feel like, and this by the magic of prerecording, this podcast is gonna be published after KubeCon. Now you’re going to be keynoting at KubeCon North America about Kubernetes security. What is the over-under on how many more CVEs there are going to be between when you think you’re done writing the talk and by the time you’re giving the talk?
Ian: Oh, wow. That’s a question. Well, I don’t personally know of anything that anybody is sitting on at all. And therefore, the number is completely unknown.
Bridget: [00:02:46] And that’s a great answer because if you think about it, Yeah, there are probably things out there. I want to say some stuff came out and like there’s some Golang stuff in like the last day that I saw. And there are, you can pretty much assume there is something out there you have to care about. Assuming that you cannot know everything, where’s the best place for people to start if they’re thinking to themselves, I would like to Cooper some netties, but I don’t want to be terrified?
Ian: So what’s the best place to start if people are trying to figure out how to secure Kubernetes?
Bridget: Yeah, like say they want to have Kubernetes, but I don’t know, those people in security or risk management or whatever are saying, oh, I don’t know, I hear things about that. Is it secure? How do we make it secure?
Ian: Okay, there are a lot of really good resources about Kubernetes security out there and more all the time. Aqua Security and StackRox and Twistlock all have pretty good engineering blogs where they talk about CVEs when they do come out. And the details of those, what can be learned from them, and stuff like that. There is a Kubernetes security announce list. And if you want to find out about the CVEs and the announcements when they happen, I recommend joining it cuz it’s really important. And in general, you can find out about how to secure Kubernetes. There is information out there on the Kubernetes documentation. There are a lot of really good talks that have been given at previous KubeCons and other places about this. There are really good folks on Twitter who give a lot of good information about it, and it’s out there. I realize that I just gave a pile of stuff, and that’s sort of one of the things is that there’s a lot. Kubernetes is a lot. There’s a lot of moving parts. It’s pretty complex, and securing it can be too. So I see the fact that there’s a lot of it and I validate it, but also you can do it.
Bridget: [00:04:35] I’m happy to hear it’s possible. Say somebody is just rolling up and is like, all right, I’m definitely gonna put this cluster out in production in the world. We’re gonna use it for real because this is what somebody wants to do to make their VP level up is to say that they’re running Kubernetes in production. So given the fact that someone’s going to run it, what are some of the top N, I don’t know, 3 to 5 things that they should probably do right away, like maybe before they open it to the internet?
Ian: Admission control is really important. And not everybody uses it. And not everybody uses it to the full extent that they could. But admission control is the biggest thing that you can do to stop attackers from being able to compromise your cluster.
Bridget: I’m gonna stop you right there and just say that people who are listening to Arrested DevOps may think, admission control, yes, I want that patch. That sounds great. Is it whale-shaped? What exactly is this thing that you’re talking about?
Ian: [00:05:37] Kubernetes admission control is a set of policies that you can put into your pod specs, into your clusters to be able to dictate what kinds of, for example, privileges get run, what is allowed access, who is allowed access. You can put pod security policies are part of admission control. That kind of thing. And it allows you to have granular control over what is and is not allowed to be happening in that cluster. And the things that can largely stop attackers are largely under the banner of admission control. The UX could use a little work in a lot of cases. Pod security policies in particular are kind of notorious for it. But it’s really important and it’s worth taking the time to learn. Other things that people could do before they put their Kubernetes into production is be careful about what’s exposed to the wider internet. People can go to sites like Shodan, S-H-O-D-A-N, .io.
Bridget: [00:06:38] We’ll put a link in the show notes.
Ian: And find anything that’s exposed to the internet. It’s basically like a search engine for the internet, and it indexes every 24 hours. So if folks think that they’re, like, too small to be a target or that nobody will ever find them if their stuff is exposed, that’s not actually true. So make sure that stuff isn’t exposed that doesn’t have to be. And if there is things exposed, that you’re protecting it appropriately. So for example, if you have SSH ports, make sure you’re using SSH bastions so that people can’t just get into them.
Bridget: All right. This sounds like good advice for people who get to make the decisions about which ports are open or get to make decisions about what the admission control settings are. What if you’re like, hey, I am just a dev standing in front of a cluster wanting it to not hate me? Like, how can people who are using Kubernetes but aren’t necessarily running the cluster have a good experience with it?
Ian: Good experience is a different question than how they secure it.
Bridget: Ah, okay. So what if it is not necessarily their job to secure the cluster in general, but they’re gonna be rolling up with their app, which is hilariously configured perhaps. What kind of decisions can an app developer make in order to have their application not cause problems with the security of the cluster in general?
Ian: [00:07:55] Great question. So things that developers can do. One way that you could potentially get into a cluster is via, for example, supply chain attacks. So make sure that you are aware of your supply chain, what’s going into it, what kind of dependencies you’re running, like what you have in there. Also—
Bridget: Wait, before we move on from that, supply chain sounds like something for, I don’t know, a grocery store. What if somebody doesn’t think of their software as having a supply chain? What exactly do you mean by that?
Ian: Software has a supply chain, especially modern software. If you think of, for example, I’m going to use npm as an example because it’s an easy one. Any given Node module often has a lot of dependencies, and sometimes those dependencies have dependencies, and maybe they have whole trees. And it’s not just JavaScript that is like this, like a lot of languages are like this. So if you’re pulling in libraries, if you are, for example, pulling from a container registry or writing to one, things like that, all of those are part of the supply chain, and all of those are potential vectors for compromise. So you want to make sure that you know what you have running, that what you have running is up to date, is relatively secure, and that you’re watching out for those potential vectors. Because yeah, supply chain attacks, being able to compromise that developer’s application or perhaps the container that it’s running in are really big vectors. Another thing that you have to watch out for is your people, including yourself. Because a lot of the time developers are really smart and they know that they’re smart and they know that they know things. So they might be actually more vulnerable to things like phishing attacks than folks like marketing or sales because they’re pretty confident about themselves and their technical knowledge. And maybe just like pay less attention to those trainings by the numbers. And also a lot of the time they have more access so they can make really exciting targets. So watch out for people trying to social engineer you and get access to the things that you have access to because that can also be a big contributor to things like compromising an application or supply chain attacks.
Bridget: [00:09:58] So I’m hearing all sorts of trust no one, which is great. But there’s realistically, there’s, I feel like there’s this eternal struggle between pin all your versions of everything ever and run the most securely updated current everything ever. How do people navigate that in a Kubernetes world?
Ian: Well, a cool thing about containers is that containers actually, if you do it correctly, enable much easier patching than we’ve been able to have before. Like it used to be that like hot patching was sort of hard and a pain and people had to be good at it. And nowadays you can actually just spin down that container and spin up an updated one and it actually can make it easier for you. In a Kubernetes context specifically, what you need to watch out for is you want to have a plan for upgrading. And specifically for upgrading in place because a lot of the time when Kubernetes gets upgraded in place, some of the security features that come out with newer versions, don’t necessarily go into that cluster context to avoid breaking changes. And that’s a thing that people should be aware of and that they should make a plan around. Because if you’re assuming that you’re going to get every new security feature as you upgrade in place, you may or may not actually. And you want to make sure that you know that.
Bridget: [00:11:15] This is a lot of context for people to keep in their heads. And I want to call back to what I thought was a great concept from your KubeCon keynote. Where you said that defenders, which is to say all of us who are trying to build things that produce business value and aren’t terrible, defenders think in lists and attackers think in graphs. And I think that’s an interesting concept, and I’m not sure everyone has context for that other than the computer science classes with some O that they sort of remember. So can you give us a little more depth as it were, into what you mean by that.
Ian: So a lot of the time, people who are not attackers will think in terms of, in a security context, perhaps a list of compliance boxes that maybe they check every one of. In a developer context, maybe, you know, like you’re counting your sprint points, you know, like how many features did you get out in that period of time? And those look like lists. Did we get them all? Did we get them all completely? You know, like, are we good? Okay, we’re good now. And they’re not necessarily looking at the the layout of what is running in that application or in that cluster. Whereas attackers are looking at that. Attackers actually generally are unconcerned with whether or not you have your compliance boxes checked or what your sprint points look like. A lot of the time they want to get in and they want to figure out what is in there, what’s running in it, how those things talk to and connect with one another, if anything in there is vulnerable. And they want to get like a lay of the land. And a lot of the time people who are developers or operators don’t necessarily think that way. They’re just thinking about like the things that they’re responsible for, but not how those things connect. And a lot of the time they might not even know what’s in there or how those things connect. So that’s really important to be aware of and to be able to think toward because attackers absolutely are doing that. And if you don’t know the connections between your resources, but the attackers are looking at it, that’s a disadvantage for you.
Bridget: [00:13:14] Interesting. So if somebody wanted to build that mental model, or maybe a documented model of this graph, of the interconnectedness of their Kubernetes cluster and everything in it, specifically, which starts to sound complex and terrifying, and ever moving and changing, are there tools or methods, something that you would recommend for people to build out that graph so that they can audit it and be horrified by it and fix it?
Ian: That’s a great question. VMware recently put out a tool called Okdent. And Okdent is useful for both folks who are on the good side of administering clusters and trying to figure out what they have going on in there. It’s also really useful for pen testers because you can run it locally with the privileges that you have access to, which before that tool came out was a much more difficult thing to be able to do. To be able to visualize that with your own privileges and what you have access to. So I recommend trying that. It’s really useful. I hear that Visual Studio Code also has some pretty cool resources about being able to do cluster data visualization, although I’ve never personally tried them.
Bridget: [00:14:20] Yeah, the Kubernetes extension for VS Code is open source and out there and probably worth looking at for people who are interested in that sort of thing. Plug for my colleagues who work on it. I also am curious, it seems like a very upbeat, a very positive message. Usually, people get defensive when security comes calling because they think they’re going to be told they have to stop doing everything they want to do and everything is probably broken. I’m curious how you, and you’ve worked in security for a while, not just Kubernetes, but how you set those expectations and that perspective. Basically, like, you’re very suspiciously positive for a security person. Tell us why.
Ian: It’s kind of weird, isn’t it? You know, security people, as you said, are really notorious for being the team of no, right? You know, like, we stop people, we block them, we slow them down. And I’m not sure that it is actually helpful for interpersonal relations or cross-team communication. Because when people don’t like you, they don’t want to talk to you or listen to you. And also, I’m not sure that it’s helpful for security either. Because a lot of the time, developers and operators are the people who have their ears to the ground. They actually know what’s happening more in their stuff than you do if you are a security person who’s on the defensive side. And if you can create better relationships with people, with those engineers, with those developers, with those operators, you can work together because we’re actually all, I think, trying to work toward the same goal for the most part in a context where defenders are working with developers and operators. And as an attacker, like, I’ve also been somebody who’s worked in DevOps, and so I know what sprints are like and that people are really mostly trying to do their best and are trying to get on with their sprint and on with their day. I think most people mean well, and I think most people are good. I think if we can meet people with empathy and meet each other where we are, realize that we’re working toward the same goals, and communicate better together, I think that that can really do a lot for us in terms of being able to secure software better. And being a negative jerk doesn’t really seem like it helps with that, so I just don’t do it.
Bridget: [00:16:45] I like it. It’s very— you’re very tactical about it. You’re like, that’s not effective, I’m gonna do the effective Sounds like you have a lot of Kubernetes security and security background in general. What if people want to be you when they grow up? It’s possible that you wanna be you when you grow up, who knows? But what if people want to become this sort of pentester, maybe even red team member, Kubernetes and security expert? What would you recommend for people who want to follow this path? What should they do?
Ian: People who are interested in learning more about the practical side of security have a lot of options these days that are not just mess around with the computer in your school illegally or whatever that perhaps people had when I was a kid. There are a lot of practical ways to be able to practice offensive security skills, such as capture the flag games, which are not the same kind of capture the flag games that we played as kids. Like, the flags aren’t physical. They’re perhaps on a server that you need to compromise in order to be able to get in and get the flag. And those—
Bridget: [00:17:53] I mean, that sounds like a fun game, but it also sounds like there could be possibly litigious people who own that server. Can you kind of clarify how these games work?
Ian: So these games are things that are sanctioned. People put them on, and it’s a way for hackers and people who are interested in gaining offensive security skills to be able to kind of sharpen their skills. Sharpen their swords. It’s like a skill-building thing. And the people who run those servers do it on purpose and know that you’re going to hack them and have that be okay. There are a lot of challenge sites out there. A good CTF to start on is overthewire.org or pietcoctf.com, which are pretty beginner-friendly. There are so many more out there, and there are CTFs that happen in person and online all the time. And yeah, I would say get Getting hands-on practice, being able to attack something that you have permission to attack, it’s very important, is a really good way to like be able to make that real to yourself and to have it not be sort of abstract and academic, but like, oh, it really is that easy. And I’ve put on CTFs internally for like developers before and for operators, and it’s amazingly effective when people are like, oh, Oh no, and you’ve been telling them about this for months or years or whatever. Just doing it themselves and finding out how fast it is, like, oh, you can literally just hit that button? Oh my gosh, can be a really helpful thing. I do recommend that everybody do that if they have a chance. I will warn you, every once in a while I meet a developer who gets bit by the bug and they want to do it all the time. It happened to me, it might happen to you because it’s really fun. Warning about that.
Bridget: [00:19:30] Nice, I love it. Okay, so it sounds like there’s a lot in this space, there’s a lot going on. If people want to follow up with you or follow you on the internets or gain more wisdom in this area or whatever, where should they go? What should they look at? What should they read?
Ian: If people are interested in following me specifically, I am @IanColdwater on Twitter, which is my first name and my last I also have a LinkedIn that says this site is only good for phishing. I do not recommend adding me on there. I will not add you back. Twitter in general is a really good place to learn about security in general, what’s going on, get up-to-date security news because people who are like just luminously brilliant experts in their field just talk and share with each other and share resources. And it’s a little bit of a time suck, but very useful.
Bridget: Nice, I love it. Okay, and through the magic of time travel, what— by the time people are listening to this, it will be mid-November, and they will have hopefully already watched your livestream keynote. And I’m wondering what you want to say to them as a coda after they have watched your keynote, after they have checked out at least several of these sites. What’s the— what’s your parting thought for them in terms of what they should hear, what they should know, what they should do?
Ian: [00:20:56] You don’t have to be anybody other than who you are. We don’t all have to think alike. Diversity is strength, and that’s really good and really important. It is really important though to be able to step into other people’s shoes. Whether you are somebody who works in security stepping into the shoes of developers or operators, or whether you are a developer or operations person trying to step into the shoes of an attacker, Being able to understand how other people think and how that can play out in the way that we design software or products or operate them can be really valuable in both helping us understand and get along with other people and in helping us design and operate our products more securely. So, you know, learn how other people think, step into their shoes, lead with empathy, it’s important.
Bridget: All right. Thank you so much for joining the podcast today, Ian.
Ian: Thank you for having me. This has been great.
Bridget: Super fun. Okay. Head over to arresteddevops.com/kubernetes-security for this episode’s show notes. And the site also has our newsletter and all of the Arrested DevOps stuff you could ever want. Visit arresteddevops.com/itunes and leave us a review in the iTunes Store because I am told that apparently helps people find the podcast. I have no idea how this these things work. I’m Bridget at Bridget Kromhout. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.
