Kubernetes & the Future

Kelsey: [00:00:00] Oh, man, I’m deploying all over the world using Kubernetes, and look, I can deploy right in front of you right now. Then you’re like, oh, man, maybe I need some Kubernetes.

Bridget: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout, and I’ll introduce today’s super dope guest after a word from our sponsors. The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener, there is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app.

[00:01:30] So exciting to be chatting with Kelsey Hightower. Kelsey, you’ve been on the show before, but for new listeners, can you introduce yourself?

Kelsey: Hey, I’m Kelsey Hightower. I refer to myself as a minimalist, so I’ll keep it real short. You know, I’m the type of person who enjoys learning in public and helping other people do the same.

Bridget: I love it. When people are wanting to have conversations with you, I feel like Twitter is someplace I see you having opinions. You recently had what I thought was a pretty interesting opinion just about the future of Kubernetes. Just to set the stage for why we’re having this podcast today is you told the Twitterverse you wanted to come on podcasts and talk about where Kubernetes is going, or what comes after it, or the future. I love this prognostication thing. Lay it on us, Kelsey. What’s happening?

Kelsey: No, what’s funny is it’s not the future, it’s the now, right? Because if I could predict the future, then they need to add another zero to the paycheck. I think where Kubernetes is, is in that sweet spot. I like to think about— remember the 56K modem? You get the dial-up, you hear the modem. Actually, some people like that modem sound because you know you’re getting on the internet, right? You hear that sound. I think Kubernetes is in the same boat where people say, oh, man, look at all my nodes. I got my cluster up to date, and it makes them feel like they’re doing computing, right? Like, oh, my cluster.

Bridget: [00:03:01] They can see all their stuff.

Kelsey: I can see my stuff. And the truth is, the internet got real interesting when that went away. I remember when I got DSL for the first time and you never had to dial up. You were always connected and you didn’t think about it anymore. I didn’t think about how close my computer was to the phone jack. Wireless routers came out around the same time, and then you kind of liberated yourself from that hook in the wall. When that happened, that’s when I thought the internet got super interesting. Fast forward to 2019, where we are now, I’m streaming Netflix, I’m streaming everything, my TV is mounted to the wall with no wires showing. The internet is now just a thing. Things tend to get better when they disappear. I’m a big fan of Kubernetes, have always been. But I know that we’re just in that 56K modem era of Kubernetes, and when we hide it, then I think more people can leverage it without learning how to manage it all. That’s where I think Kubernetes has to go, whether we take it there or someone else does. I think that is the key to its long-term success.

Bridget: [00:04:07] I think that’s a really interesting point because I will say that the exposure I’ve had to Kubernetes has included things like running workshops with Jerome Pettazzoni, who I know that you’re going to be podcasting with him as well. I look forward to listening to that one. But definitely running workshops with something that is constantly changing, it’s like you’ve got this moving target. There was a recent discussion just today over on Kubernetes Slack about the number of releases that a cloud provider will support versus how quickly things come out from the release team. I feel like this is a very fast-moving space if you are paying attention to the bleeding edge. At the same time, the entire world out there is not able to install exactly what’s in the release candidate and run it in production tomorrow. What do you see as When you’re saying the future is now and this is happening, but if somebody is listening to you and they say, Kelsey, that sounds wonderful and we’re stuck on 1.12 and we aren’t getting anything new anytime soon, or Kelsey, that’s wonderful, we don’t even have any Kubernetes yet. What do you say to them?

Kelsey: [00:05:28] I think honestly that’s a good place to be in because if you don’t have this problem, you don’t really need this solution quite yet. I like to think about Linux went through the same transition. Remember when Linux came out? People were rolling their own distros, Linux from scratch, Slackware, and then we got Red Hat, Canonical, and then we stopped thinking about so much of building Linux in our userland from scratch. We got more into, well, which vendor or distro do you identify with because they will provide and we just get to leverage it. If we fast forward on that Linux analogy just a little bit, Think about people that are running Android phones. Do you think they care about Linux? But they get to harvest the benefits of Linux. They get to just use a mobile phone that they just install apps to. It’s a mobile device, and they never really have to touch the Linux kernel. I think there’s going to be a lot of people who will miss the entire cluster management phase of Kubernetes, but one day, Kubernetes just may be the thing that’s baked into the thing that they’re using. Then, they’ll just be using Kubernetes. I think, for a lot of people, it’s early for Kubernetes. We’re only 6 years into this thing. It’s not like— Golang is only, what, 10 years old? This stuff is so early that VMs still work. Some people will skip the whole container thing and go straight to serverless for some of their workloads. It’s fine. But, one day, I would say, Kubernetes-style APIs are starting to resonate with more and more people. Just like we saw from the web, web pages and web servers and browsers, we saw RESTful interfaces come out of that, where we took HTTP, we took the verbs, and now we’re making GraphQL and APIs. Some people will start to just leverage some parts of Kubernetes without ever being a cluster administrator themselves.

Bridget: [00:07:19] I think that’s a really good way to put it because a lot of people are familiar with this idea of, like, okay, you have APIs. Cool, that means you have endpoints, you have something that you can interact with, you have hooks you can hook into. They hopefully don’t have breaking changes and point releases constantly. Is that how you think most people are going to consume Kubernetes in the future? Because I feel like you mentioned serverless. There’s a lot we can talk about in that direction, but there’s also this idea of, if Kubernetes sounds like a big thing to you, maybe all you need to think about is you’re going to have an API to make your stuff work. And you don’t have to think about the substrate as much. Is that where you’re going with that?

Kelsey: Yeah, because Linux is ever-evolving. Man, Linux has a release all the time. The Linux kernel, Bash, all these userland components, they’re nonstop releasing at this point. What we’re getting, though, is checkpoints. GKE represents a checkpoint. AKS represents a checkpoint. Fargate represents a checkpoint. K3s, which is like this miniature version of Kubernetes, where They swap out a lot of the components to run on embedded or smaller platforms. These just represent checkpoints in this ever-moving project, and honestly, most people aren’t even ready to adopt all the stuff we currently have. I think there’s going to be plenty of time for you to catch up. I like to think of this as, I am very happy that the innovation is still going on, people are still excited, new ideas are always showing up, but for the average person, who I put myself in that boat now, I am a consumer of Kubernetes these days. I go to the checkpoints, and I just use it as is, and I’m fairly happy.

Bridget: [00:08:58] That says a lot coming from you, because we’ll put a link in the show notes to your Kubernetes the Hard Way. You are a Kubernetes OG. This goes way back to the 2015 era. Can you talk a little bit about what people can learn from Kubernetes the Hard Way style, either yours or anything like that? What can they learn from that now? What are your recommendations for them if they’re looking at actually productionizing any such thing?

Kelsey: That’s a good question because early on in Kubernetes, there were no docs, really. When I learned Kubernetes, before I wrote my first line of code, before I’d done my first PR to the project, I had to learn how to install it. What pieces go where? What’s this scheduler thing all about? Once I got the whole thing assembled, Then, I was able to identify issues that I had with the entire system once it was put together. I was at CoreOS at the time, so I was building on extensions to make Kubernetes work well with CoreOS nodes so we can register them, that kind of thing. When I think about Kubernetes the hard way, you can’t really fix a system that you don’t know how it works. That’s just really shooting in the dark, and if you’re going to take something to production, you probably want to know how all the pieces kind of fit together. I think, for a lot of people— and I have an ops background as well— the thing I always want to know is, well, how is it supposed to work?

Bridget: [00:10:27] You want to see it in production with some idea of how you’ll know if it is working.

Kelsey: Right. I think a lot of us can appreciate the happy path. Kubernetes the Hard Way is about going through step by step, in a very tedious, fashion, no scripts, so that people can say, oh, this is what the kubelet does. Oh, it connects to the API server in this way. Wow, that’s where the SSL certificate goes. I think once people go through that once, then they have that foundational knowledge to improve the system, troubleshoot the system, or debug the system.

Bridget: That tells me that people still get value out of looking at some of those details. I think that the container.training that Jerome and Brett and I have taught people a lot of stuff from is relevant again, but at the same time, we definitely aren’t showing people something that they should use in production. Would you say that people can start with something like Kubernetes the Hard Way and then get to where they should be? Or what would you recommend when they’re actually taking this stuff to prod?

Kelsey: [00:11:33] I think production is very tricky because There are so many layers to productionizing a system, performance tuning, security. Most clusters come out of the box with flexibility in mind, not necessarily security. You can run random images from the internet. You can run things as root. Some people aren’t using any security policies. Some people don’t even know what policies are. You kind of run in this world of convenience. I remember when SELinux came out. Almost no one knew how. I’m going to bet most people still don’t know how to use SELinux. Every system admin, even though they won’t admit it, turns it off.

Bridget: It’s like the first step, turn SELinux off, then move on from there.

Kelsey: Yeah, you turn it off. People act like this isn’t a recurring thing. A lot of times, most systems have been optimized for ease of use, and we’ve been making this security trade-off for a long time. Kubernetes, the hard way, has to make some of those trade-offs, because if I did every possible security step, the guide would be 10 times longer, right? I think there are ways that you should probably automate away those things. I think the cloud providers, the various distros that are out there, the various tools that help you get a cluster up and running, they tend to try to automate a lot of those very tedious security things that you should do to lock down your cluster. I think that’s what you need to start thinking about for production. Security probably is the number one thing you have to consider. When it’s time to go to production, because you can always add performance. You can always tweak and tune the Linux distro or image that you’re using. But once that security hole is too big, it’s a little too late to go rewind the clock on a breach.

Bridget: [00:13:10] Somebody says to you, you know, this Kubernetes thing you decided you wanted to do, turns out it’s a security problem. You’re like, but it doesn’t have to be! They’re like, not sure if we believe you at this point.

Kelsey: But the nice thing that I’m seeing on the productionizing front is now, when you go and watch a talk at KubeCon, or you read someone’s security guidance or their docs, they’re starting to say the most common things. Do exactly these 4 or 5 things, and then we raise the security profile for a lot of people at the same time, which I think is really a big game changer.

Bridget: Yeah, I think Ian Coldwater talked about that some in their keynote at KubeCon, and I think even just some of the work that Snyk Security has been doing, like Gareth Rushgrove has been doing a lot of stuff in this space that ties in with Open Policy Agent and other things to just make it easier for people to do the right thing.

Kelsey: 30% of Kubernetes the Hard Way is because of the security feedback that I’ve received from people over time. The reason why I take the time to generate SSL certificates for each component and for each The reason why I encrypt the secrets in the etcd database and show you how to do that and verify that it’s encrypted. A lot of that straight up comes from the community saying, hey, teach people about RBAC, teach people about etcd encryption, teach people about SSL certificate management. While I’m not probably going all the way that I could go, I am definitely getting you past the let’s just do nothing stage.

Bridget: [00:14:45] Yeah, and it’s kind of funny, too, because I have taught the, like, hey, let’s take a look at the dashboard. We’re just going to open everything up and take a look at it. Then there’s, like, you have to put the giant disclaimers of, like, do not actually do this. If you actually do this, we are not responsible for your terrible life choices.

Kelsey: Exactly. Well, that’s why I leave the dashboard out of Kubernetes the Hard Way. I don’t want to send someone thinking that that’s how you do it. I’m like, whoa! I won’t be the person that told you to do that.

Bridget: I mean, like, we show them the dashboard and then we say, unless you want to mine cryptocurrency for someone on the internet, you really should not do this on your systems. And if somebody is mining cryptocurrency on this right this second, we’re shutting it down right after the workshop. So, you know.

Kelsey: Exactly.

Bridget: But yeah. Okay. So when I was, I was thinking about the stuff that you’ve been talking about lately, that’s Really interesting. You have pointed out this kind of, you know, I don’t know if trade-off is the right word, but kind of there is complexity. We have to acknowledge there’s complexity choosing the right layers to interact with or choosing exactly what you should abstract away and what you should do. Obviously, in a workshop, this is going to be different. If the goal is learning versus production, this is going to be different. I would be curious to know, what are you seeing in terms of the trends right now, and what do you recommend in terms of which complexity should you incur? Is it incidental? Is it necessary? I would love to hear your thoughts around that in the Kubernetes space, specifically.

Kelsey: [00:16:27] This is why I love looking at other systems and how they evolved over time, like CDNs. I remember when we used to glorify FTP servers, and then SFTP. Do you remember that?

Bridget: Oh, not only do I remember that, but I remember horrifyingly, it was like 2012 and a major customer, I’m not even gonna say what vertical they’re in, but I’ll say a major customer for the startup I worked at, a customer you have heard of and probably given money to, you listeners in the world, they really didn’t wanna use SFTP, they wanted to use FTP, and I had to talk them into using SFTP as like the lowest common denominator of them giving us more datasets.

Kelsey: There you go.

Bridget: So horrifying.

Kelsey: So people still use these kind of protocols to transfer files. And then I remember it was such a big task of getting those files delivered as fast as possible to people in the world. And some people thought they were just going to evolve FTP to become more robust, right? Like some people are— there’s probably still an FTP conference somewhere. It’s probably like 8 people gathering doing their thing. But then CDNs came out, and CDNs took this very complex problem of how do we get these images and video files close to people in the most efficient way possible and be cost-effective over time and maintain the security. So now we don’t really think about that problem anymore because it’s gone away in many ways and gotten more powerful while most of it disappeared because the complexity grew and the number of people who could actually understand and maintain that complexity shrunk. But we kind of all benefit from it. Now, when I look at the compute world, the compute world is a little slower to doing that because there are so many more people who believe they understand the compute problem. So, every couple of years, a new person comes around and invents the new platform again. We see it over and over again, right? You and I have been part of some of those startups in the past. Now, when I look around, the serverless world is saying, look, there are about— there’s 80% of the compute use cases we understand. No one ever has to build that again. So if you want to use that, you can use that. But that doesn’t mean mainframes go away because they’re still really good at certain workloads and certain tasks. It doesn’t mean VMs go away because of the same reasons. So I think people look for this all-or-nothing that the next thing will replace all the other things. No! I really look at this as we’re going to have multiple things in parallel, and you get to sit back and choose. Which one meets your needs. If it was me these days, if I had to start from scratch, I think I would probably try to go as high as I could and focus on building my app and the product before going to play infrastructure again.

Bridget: [00:19:15] That makes so much sense, too, because the nuance that you’re pointing out here, if I can try to summarize, and you can tell me if this is right, but the nuance you’re pointing out is, If you’re starting something new, you probably don’t want to incur a lot of overhead you don’t need. However, at the same time, it’s not like it will be a good use of time or money for your bank or your airline or whatever to redevelop everything that they have and make it all serverless functions. Like, in reality, they’re not going to do that.

Kelsey: Without a doubt. And there’s a trust element here, and I get it. Because I have a lot of empathy for people who say, well, if I leverage one platform, I’ll get locked into that platform. And I think we’re trying to do a good job with various efforts like the CNCF and various other organizations trying to create open standards for a lot of this stuff. I think what we want is— the internet is so complex that you probably aren’t going to dig up a wire to your house and connect directly into the backbone of the internet. You’re not doing that. So you trust that a provider is going to give you an open interface, an IP address, so that you can jump onto the internet and participate like everyone else. I think that’s going to happen to compute. We need the providers to gain our trust so that way we can use open interfaces to participate in this big global compute set of offerings.

Bridget: [00:20:46] That actually takes us to a really interesting place of security and compliance. I mean, I work at a cloud provider, you work at a cloud provider. I’m sure you have these conversations from time to time where someone says, gosh, we would love to put workload X in the cloud, but you know, our security/compliance/auditors don’t like that idea. I’m thinking, like, because the data center you’re leasing down the road is safer? Or the basement of your building is safer, really?

Kelsey: That’s when the human elements meet technology, right? Because if I don’t understand how a thing can be secure, I might just be a little bit skeptical. We haven’t earned that trust for a lot of workloads. Now, there are some companies, their banks are 100% online.

Bridget: Oh, absolutely.

Kelsey: They see it being too much of a risk to have a single building with a door on it that I can walk in and take the server.

Bridget: Right.

Kelsey: So there’s different degrees of understanding the risk.

Bridget: Yeah. Now, it’s just, it’s a really interesting space because, all right, so you’ve got your mainframe workloads and you’ve got your VMs and you’ve got, you know, maybe some stuff that is in a container and you’ve got your Your COTS applications come from your upstream vendor that you can’t really do anything about, and you certainly can’t refactor them down to their functional components or whatever. Then you have maybe the new development people are doing. I think that our industry loves to focus on the exciting new development people are doing because, sure, it seems simple. It doesn’t come with any legacy. It doesn’t come with any burdensome customers who don’t want you to change things or move their cheese. But realistically, The customers who pay money for cloud services or anything else are going to probably, by definition, have money and have customers and have workloads that they care about. They still might be looking at Kubernetes and thinking, I would love to solve some of the multitude of problems I have with Kubernetes. How do I get there, and am I going to be burdened by too many megabytes of YAML doing it? When you’re talking to people, about that transition? How do you make Kubernetes, or what comes after Kubernetes, which is to say, using Kubernetes just as a primitive, how do you make that accessible to people?

Kelsey: [00:23:04] When I go talk to engineers with hands on keyboard, or maybe the leadership, senior leadership at these companies, one thing I do is I ask them to stop using the word legacy. Just don’t use it, because right now, it kind of has this derogatory context behind it. I say, this is classic infrastructure. It’s the stuff that actually worked cutting everyone’s paycheck. So, it’s the thing that actually works.

Bridget: I like to call it the place where all the customers and money are.

Kelsey: Yes, there you go. So, if we start from there, we say, look, you did a good job building out those systems. Now, if you look at the current systems you have, what problems do you have? They will say, well, on these platforms, we have a problem with things that you would describe as service discovery. We have a problem with failing over, so when this part of the data center goes down, we’ve written all of these scripts and we hired all of these people to bring it back up on these other nodes, and we have a big on-call rotation to do that. When I look at that, that’s the pain. That’s the opportunity for value. When Kubernetes comes along and I look at what Kubernetes can do, having those previous jobs before, I can easily say, you know what? There’s a part of Kubernetes that does solve that problem, like, very specifically. That problem doesn’t mean we have to build that part ourselves. Maybe we don’t need all of Kubernetes. We don’t necessarily need to replatform everything we’re doing onto Kubernetes, but I can tell you that there’s a much better checkpoint these days for incorporating some of these fundamentals into our infrastructure, and if we can get that by leveraging Kubernetes, Then let’s pragmatically show the value. I always ask people, always have a good reason why. We brought in Kubernetes because we were maintaining our own scheduler in something that looked like Kubernetes, but you know what? We have 15 people that are doing that. I’d rather have those 15 people working on this other problem you’re yet to tackle, or half of them contributing to Kubernetes to make sure that we can leverage it going forward.

Bridget: [00:25:05] And I love that too, because this is you having the very human-centered conversations of like, if someone thinks that their entire value is the thing they’re doing right now, they might be resistant to something exciting and new that could make their life way better because they don’t see themselves there. And if you can help them see what they can be doing that adds a ton of value, there, then there will be way less resistance. Those human factors are so huge when it comes to getting people to adopt something that seems different and, therefore, perhaps threatening.

Kelsey: Well, that’s why I like spending time on-site with people on their home turf where they’re comfortable. I’ll just ask and listen for a little bit and just say, what are all the things that you’re not doing? What are the things on your backlog? A lot of times, it’s the observability stuff. It’s the security stuff. These are just the things that no one ever gets around to doing properly because they don’t have time. Even if Kubernetes automated you out of a job, which it won’t, all it’s going to do is probably give you time back so you can start doing this other stuff.

Bridget: [00:26:11] I love that, because I’ve got to say, #OpsLife is you have your hair on fire or whatever right this second, but then there is the stuff you would love to get to, and you know you probably won’t anytime soon, and the luxury of being able to do those things that will make some Everything’s so much better. That’s huge. But I do think that the Kubernetes ecosystem, if you’re approaching it from first principles, or if you’re approaching it from, I don’t know what’s going on here, but now that I have a little bit of understanding of pods and nodes and so much YAML and everything else, how do we get people to I don’t want to go into the long tail or the future being very unevenly distributed, but I do feel like there are the— maybe they are a bank, but they jump on this stuff early. Then there’s the people who are thinking, is that stable enough? Can we use it yet? What do you think? Can people actually use these now?

Kelsey: [00:27:15] That’s what people— they’re not touching the internet. You’re not going to touch it. And that’s okay. Like, if you don’t want to touch the internet, that’s cool. But the thing is, these things will progress without you giving the green light. And I think that’s where people have to be a little bit pragmatic. The world is going to continue to move forward with or without you. If you look up one day and you see a tool that might be useful, only you can make that decision if it’s actually useful for you. And the best way of doing that is being a bit more informed, like what is the tool actually doing? Does it even solve my problem? And that’s the engineering piece where I don’t think people give themselves enough time to say, let’s bring it in, let’s put it through its paces, and let’s either say yes or no. If we say no, let’s write a doc internally. The reason why we’re not using tool X is because of reasons. If those reasons get solved before we find something better, then we’ll revisit that decision. That’s just engineering. I think that helps with the whole fashion and the vogue-ness of all this, like, oh, Kubernetes is hot! I’ve got to be on it! I’ve got to be on it! I think that’s where people kind of run into that brick wall. There’s one more thing, too, fundamentals. For a long time, I’ve worked at Puppet Labs. I’ve used Ansible before. I’ve contributed to Ansible. I’ve contributed to Terraform. That whole era, that last 10 to 15 years, that was us attempting to treat infrastructure as code. We wrote a lot of code. We have these very advanced domain-specific languages, DSLs, to allow us to describe infrastructure, and then we get into for loops, and then we inherit the problem of dealing with any codebase— code reviews, testing, and so forth. Now, with Kubernetes, we’re trying something slightly different. I’m not saying it’s radically different. I’m saying it’s slightly different, where we’re forcing ourselves to no longer do that. No more infrastructure as code. Now, we’re doing infrastructure as data, meaning if you want to be able to do some new capability, you have to do it in that control loop, the controller, that thing first. Some people call them operators. It doesn’t matter. All the if statements and for loops in any programming language you want, you have to implement it there. The state machine lives there. Then, on the front end, we restrict ourselves to data, so we call that YAML. Even though Kubernetes doesn’t actually support YAML, it only supports JSON and protocol buffers, YAML is how humans found a way to write data out in a way that we can interchange it to some API. Now, we’re in this world where we’re starting to treat infrastructure as data. Why is this powerful? The drawback is it’s a lot of YAML. The benefits, just like we saw with assembly language, you can have any programming language you want compiled down to a common format that can be executed on a large generation of CPUs. Now that we have this YAML, aka the assembly language, we can now build tools in any programming language we want, and they will all boil down into this data model. Then guess what you can do? You can take Helm, which is implemented in a real programming language, adds a ton of great features, gives you key values, all of this stuff, templating. I can run Helm as a preprocessor. It spits out some data. I can take that data, pipe it to Kustomize to patch it, and then when it goes into Kubernetes, it can go through an admission controller that can add even more to the data model before it lands into Kubernetes. This is like the dream come true that we can finally describe infrastructure with a type system and interchange the tools without throwing everything away and starting over every time.

Bridget: [00:30:55] I think that’s really powerful, just this idea that It is declarative, it’s not imperative, which I think is maybe a mental hurdle that people have to get past. But then this idea that your tools are composable, because— and that’s one of, I think, the great strengths of the open source ecosystem and why I definitely want to work in that space and resist anything that says, we’re going to go down this specific rabbit hole that’s not cross-compatible with anything else, because it’s like, Speaking as someone who does work at a vendor, I’m very well aware that not everyone uses my employer’s services for everything, but they may want to use it for some things. So I would love to be cross-compatible so that everyone who wants to use, say, more than one cloud or whatever can. But like, do you, do you think that we have— I don’t want to I don’t want to go down a giant CRD-shaped rabbit hole, but I think that there’s a lot of complexity in this space. And so does the flexibility mean that the complexity becomes unapproachable for people? That’s kind of what I worry about.

Kelsey: [00:32:12] It depends on who the user is, right? So we know that if we had no computers at all, to evolve to this point would require this kind of complexity. Sure, there’s a, there’s a total amount of complexity. Let’s just put it in a box. What parts of the system deal with that complexity? So without these tools, then the humans hold it all in our heads and spreadsheets, right? In our minds and spreadsheets, we take all that complexity and then we try to articulate it via bash scripts, right? That’s, that, that was the world we came from.

Bridget: So I mean, I’m very familiar with the world that says dump out what exists in your cloud provider and check that into GitHub, and then maybe have Jenkins break the build if someone changes something that they didn’t check in, because, like, yeah, we cobbled together a lot of stuff like that before we had a better solution.

Kelsey: I think what Kubernetes is doing is formalizing the complexity. Now, we can critique the complexity, we can talk about where it belongs, but at least now we can see more of that complexity in one place. Now that we look at that complexity, different people deal with it at different levels. If you are really great at your job, and you’re the person managing that Kubernetes environment, and you’re creating all the CRDs, your consumer, the people you work with, will know nothing about Kubernetes. They will just give you a CRD that says, you know what? Deploy my app across 50 different countries. You can create a CRD that just says that. What zones do you want? Then, my control loops do all of this heavy lifting, so you don’t have to. I think that’s a place where we can transfer the complexity from the edge of the company, the developers, the other operators, and move it into the system. But, in order to do that, then the system has to be capable of encapsulating that much complexity. I think that’s where you see this tradeoff. Kubernetes is a very powerful tool. This extension model can be very simple, or it can be very complex. The nice thing is, when it does get more complex, we don’t have to abandon ship on it and find a whole other tool to start over with.

Bridget: [00:34:26] That kind of brings me to, if people are trying to figure out approaching Kubernetes now, let’s say they don’t need Kubernetes 101, they’ve been using it a little, but they want to implement everything that they would love to do that they hear about, what should people— should people be jumping into CRDs, which, for people who are just now tuning into the Kubernetes world, are custom resource definitions? Should people be looking at operators? What would you— or something else entirely? What do you think people should be looking at if they’re like, okay, I am ready to really understand this Here’s the thing.

Kelsey: If you just want to deploy some containers, writing CRDs and operators is the equivalent of writing kernel modules. Like, oh, I want to understand Linux. Let me write my own device drivers. Hey, look. You can.

Bridget: Maybe you don’t want to do it on your employer’s time.

Kelsey: Yeah, but there’s a group of people who should know how to do those things and will need to do those things. But, for the majority of people, that is not the job. That’s not the job. So, I think you’ve got to ask yourself, am I looking to use Kubernetes as a tool to run my application that has this very convenient API that lets me declaratively say, I need a load balancer, an SSL certificate, a DNS name, and this container? You can wipe your hands of it. Now, if you’re managing the cluster, You can learn how to use it as is out of the box, and the community will provide, and you adopt the best of the breed, and you just provide that to your developers. Now, sometimes you’re going to have to go beyond that. Kubernetes is still early, meaning every integration in the world does not exist. That’s where you’re going to have to probably scratch your own itch. Now, if you want to be able to do that, you want to be able to scratch your own itch, Then, yeah, CRDs are how you do things in the Kubernetes world. Like we said earlier, CRDs are backed by control loops, where all the complexity and logic lives. Those 2 things go hand in hand. But that’s only if you need to extend the system.

Bridget: [00:36:40] That kind of comes back to the conversation that you have, and I have, and probably a lot of people have with customers, when they say, is technology X right? And you’re like, Let’s talk about the problem you’re trying to solve, because picking the technology first is like— I mean, that’s classic resume-driven development, is kind of what I’m thinking when somebody is suggesting it.

Kelsey: It’s like, but it’s so hard now because you, you know, things aren’t the best they can be with your current stack. And, and maybe you’re caught in the middle of, I don’t know what I don’t know, so I don’t even know what question to ask. And I may not be able to even articulate the problem. All I know is every time We try to deploy the app, something goes wrong, 10 years in a row.

Bridget: And it’s not always the same something. Yeah.

Kelsey: So then you go to KubeCon and people are like, oh man, I’m deploying all over the world using Kubernetes. And look, I can deploy right in front of you right now. And then you’re like, oh man, maybe I need some Kubernetes. And I think I’m partly responsible for some of this, right? I think people say, damn, all I know is that there’s this new checkpoint where some people don’t seem to have the exact same problems I have. They have a different set of problems, but maybe not the exact ones I have. All they know how to do is say, is Kubernetes right for me?

Bridget: [00:38:04] Ask your doctor if Kubernetes is right for you. Exactly. When people are figuring out if Kubernetes is right for them, realistically, we’re all humans. Some of us are going to make decisions based on what is interesting to us or what’s right for our career. Not everyone is going to start with, like, you know, well, you start at the CNCF landscape and you go, oh my God, that’s way too many. Then you go to maybe the cloud-native trail map or whatever, and you say, okay, I’ll get some CI/CD. That’s great. We’ll get some Source control, that’s great. Hey, orchestration, containers. If people are to the place where they’re moving beyond that, what are some of the exciting things that they should be looking at that maybe will help them and also help their career become a little bit more what they want it to be? What’s the new Kubernetes what that you have in front of you?

Kelsey: Here’s the thing. I was teaching my daughter. How to make a web page, you know what I mean? Like, it looked like GeoCities, you know? We got the HTML tags, we got a little bit of JavaScript, some CSS. She’s got the blinky tag thing going on.

Bridget: [00:39:17] And, uh, is this like with Scratch, or is there—

Kelsey: No, no, no, just straight up. No, no, no, straight up text editor. Oh wow, HTML tags. And then the nice thing about that deployment process is it’s just Chrome, right? She goes to her browser and she opens up the file and she sees something. OK, so we’re making progress. She’s like, how do I put a picture? OK, she got a web page. Nice. Then she asked a question. She was like, hey, I text my friend 127.0.0.1 and they can’t see anything. Right. So I’m like, oh, wow. Now we get to talk about the actual internet and networking and infrastructure. And I was like, I would be a bad parent if I taught about Docker containers and Kubernetes. Like, that’s not— what she wanted to do. She wanted to show someone her webpage. So what I ended up doing is just using like Firebase or something and just said, hey, we installed Firebase. She said Firebase deploy and it spit out a URL and she put that in her phone. It was like, oh my God, I can see my website. And then she gave it to her friend and she could see it too. Yeah, to me I was like, that’s the end game. Like that, that’s it. The goal of the people like us We’re building these platforms to enable that for as many workloads as we can possibly imagine, to a point where if you want to run your own cluster, you will always be able to do that. But, I want people to have the ability to just say, I have an idea, I have the code done, and I want to see it come to life. Why can’t that be possible? That legitimately should be possible, and I think that’s where I think the— The excitement. Between that wow moment and everything below that, that’s all the opportunity we have. Any platform that itches us closer to that moment for people, that’s the exciting thing. Kubernetes will evolve towards that way from the ground up. The serverless stuff will backtrack its way down to support different kinds of workloads outside of functions and serverless or stateless containers. Everyone knows that’s the endgame for compute. So that’s the thing that I’m really focused on. So that’s where I find excitement across the board is when people push me towards those moments.

Bridget: [00:41:35] And people love that. People love it just works.

Kelsey: That’s our goal.

Bridget: Especially, and I think that’s a good thing for us to remember as technologists too, is like, as fun and interesting as all of these toys are to play with, like, we are also Hopefully building things that produce actual value, like something that people would pay for or would vote for or would be interested in.

Kelsey: That, that’s our endgame, and we should never lose sight of that, even though I’m not saying though, if you work on those things in the middle, very important thing. I’m not saying that it doesn’t matter. I’m saying is it’s not the endgame. It’s not why we do that.

Bridget: Yeah, that’s I think that’s, that’s pretty important. And that’s probably where we should leave it because we’ve been talking for a good long while here. I really appreciate your time today, Kelsey. And where, where can our listeners catch up with you if they’re, if they’re thinking, wow, that, that Kelsey guy, I want to hear a lot more from him. Where can they do that?

Kelsey: [00:42:40] I aggregate in one place at this point. I’m online on Twitter. DMs are open, even though I promised to close them. But I actually find it very valuable learning from other people. And direct messages is one way that still works for me. So Twitter is where I’m at.

Bridget: Excellent. So people can find you on Twitter or if they, if they want to see you speak or interact somewhere, where would they get a chance for that?

Kelsey: Random meetups across the world and they’re just random. We always announce them like 2 2 weeks before they happen. There’s lots of content on YouTube. Like you, I’ve been doing this for a long time. If you’re just missing Kelsey in your living room, you can totally pull me up on YouTube, and I will talk to you about all kinds of stuff.

Bridget: I love it. I feel like that’s pretty much where I’m at, too. I run a local meetup here in Minneapolis. I’ll probably be at KubeCon EU, though I won’t be speaking. It’s mostly just you really want people to be able to connect with their local community, connect with the people in their organization, and hopefully some of the stuff we talked about today will spark some ideas for them. Thank you so much for coming and talking to us about that. I’ll tell people that they can head over to arresteddevops.com/ kubernetes-future for this episode’s show notes, and leave us a review in the iTunes Store if they want to help other people find the podcast. Note that I have no idea how that works, but apparently reviews— I don’t even know. We’re on Spotify and iHeartRadio if you’re into those. And yeah, I’m just really grateful that you were able to make the time work and come chat with us today. So, thank you, Kelsey.

Kelsey: [00:44:27] Awesome. Thanks for having me. Great.

Bridget: Um, I’m Bridget at Bridget Krumhaupt. This is Arrested DevOps, and remember, there’s always DevOps in the banana stand.