WebAssembly, Krustlet, and the Future

Brian: [00:00:00] It’s moving the direction of the foot gun so that it’s not pointed at you.

Bridget: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout with a great show for you today, but first, a word from our sponsors. This episode is sponsored by CircleCI. Designed for modern software teams, CircleCI’s continuous integration and delivery platform helps developers push code with confidence. Trusted by thousands of companies, from 4-person startups to Fortune 500 businesses, CircleCI helps teams take their software from idea to delivery quickly, safely, and at scale. Visit arresteddevops.com/circleci to learn why high-performing DevOps teams use CircleCI to automate and accelerate their CI/CD pipelines. If you are like most of your friends in DevOps, you probably prefer using open-source solutions for observability. But you also wish you didn’t have to sacrifice scalability, performance, and simplicity. With Logz.io, you get the best of both worlds for your cloud environment. You can use the tools you love at the scale you need. Logz.io is a fully managed service that offers complete cloud observability for engineers on one unified platform: log management and cloud SIEM based on ELK, and infrastructure monitoring based on Grafana. To give it a try for yourself, sign up for a free 14-day trial today at logz.io/ado and for your chance to win a free Logz.io t-shirt.

[00:01:53] The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener, there is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app. Today we have 2 guests and I’m going to ask them to introduce themselves. I’ll start with Taylor Thomas.

Taylor: [00:02:55] Hi everyone, my name is Taylor Thomas. I am an engineer in Microsoft Azure on the team that’s known as Thais Labs, and I’ve been working in a lot with cloud-native technologies, particularly around containers and Kubernetes for the last few years and recently with some of the things we’re going to talk about today.

Bridget: Thanks, Taylor. And Brian?

Brian: Brian Kettleson, and I’ve probably been most active in the Go community as an organizer of GopherCon, author of Go in Action. I don’t know, I’ll put everything else that I did in the Go community, all those things. I work at Microsoft as a Cloud Developer Advocate, leading a group of software engineers that are contributing to upstream open-source projects. I’ve been working a lot with Taylor recently on Krustlet, which is the project that we’re going to talk about today. Great.

Bridget: Thank you, Brian. I’m titling this episode WASM, Krustlet, and the Future because I think this is some really exciting, really cutting-edge stuff. Our RESTful DevOps listeners may have heard of WebAssembly. Maybe they saw all the great blog posts that you folks wrote about Krustlet last week, but they’re uncertain probably about what exactly is this WebAssembly thing and how is it relevant to my life? So I would love if you wanted to just start us off there. Whichever of you wants to jump in and say, what even is WebAssembly?

Taylor: [00:04:22] So I’ll go ahead and take this. So WebAssembly is a compiled language that was invented for the web, hence the WebAssembly. That’s where it originally came from. That operates in a complete sandbox, and you can compile modules down into these— I mean, sorry, your code down into modules, and they are able to be imported into JavaScript and other things in a browser. And so you’re able to make some pretty performant applications for the web using them. For what we use them for, there’s been a recent development from the Mozilla Foundation called WASI, or the WebAssembly System Interface, if I remember correctly. Remember that correctly. And so this WASI interface is allowing us to build WebAssembly modules that can run anywhere, as opposed to just inside of a browser.

Bridget: Yeah, and Brian, I know you’ve also been doing a lot of work. I actually, when I was deciding who should I have come talk on the podcast, I literally went and looked at the commit history on the particular project you released last week, and I was like, well, these are the people who are committing to it quite a bit, and I got some of them.

Brian: [00:05:30] So, yeah, if I could jump in on the WebAssembly promise, one of the things that intrigues me about WebAssembly is the idea that it’s a binary format that can execute anywhere that has an interpreter. So, in theory, if you’ve got a WebAssembly interpreter on your Linux AMD server, it will work there as well as your ARM32 Raspberry Pi. Same executable, same WASM file, no changes at all. So that’s one of the things that’s really interesting to me about WebAssembly is the realization of that write once, run anywhere promise Java brought us so long ago and we all laughed at.

Bridget: And wait, you’re saying that those things are actually true now? Does that mean write once, run anywhere in a horribly exploitable fashion? Like, I know people say, oh, there’s great security here. What is the security promise or the security possibilities in this WebAssembly world?

Brian: [00:06:31] So in WebAssembly, the execution is entirely sandboxed. Impossible for the process to break out of its host, and the only way to expose memory into that WebAssembly execution environment is to do it explicitly. The only way to get memory out of it is to explicitly pull it out. So Unless there’s a bug in the implementation of your WebAssembly host, it’s completely secure. And it was designed from the beginning, from day one, with security in mind.

Taylor: Yeah, in the wild web of the west, you need that sandboxing. And so it’s really, really nice to bring in that sandboxing model into an actual normal runtime on any system. And it really is down to that where Like Brian said, you cannot do anything unless you explicitly expose it to the runtime, which is very powerful. It can introduce some other things where you have to actually go and expose that, but then you’re also not having to deal with other things that you normally would, where, okay, well, do I have access to this, or do I have to lock down this thing? It’s only what you give it.

Bridget: [00:07:42] This is really interesting from the point of view of when people talk about Kubernetes, they often caution the world, just so you know, the defaults are not secure. And you have to implement all the security or turn on all of the controls that would make the open-by-default Kubernetes secure. So, am I hearing that— how does this relate to Kubernetes? If people like the idea of Kubernetes and the idea of this WebAssembly thing, however it works together, how do you— do you get to pick the more secure or the less secure? How does that work? And you’re both like, that question makes no sense. OK, so the reason I’m asking that is because, You’re talking about this Krustlet thing, so maybe start there. Explain how do we have— what does Krustlet do, and then how does that bring us, hopefully, the security promise that you just mentioned?

Brian: Yeah, that’s definitely the best place to start. Krustlet is an implementation of the kubelet specification written in Rust, but built to execute WebAssembly instead of Docker containers. Like a regular kubelet does. So it takes a specification for a pod, and it determines from that which WebAssembly file to retrieve from the internet somewhere, and then it executes that WebAssembly file. So there’s no containers involved in a Krustlet at all. It’s simply executing WebAssembly. And because of that, to answer your original question, you are running, your pods, your workloads, more securely, because the WebAssembly environment is more secure inherently than a container runtime is. But your Kubernetes installation is only going to be as secure as you made it. So, our Cresslet project isn’t going to change anything about the defaults of your Kubernetes installation.

Bridget: [00:09:35] It doesn’t magically make Kubernetes more secure. Away with you! But still, making the workloads more secure is still an improvement.

Taylor: Yeah, and I think that it makes it a little bit simpler too because Kubernetes provides a great set of abstractions over managing various infrastructure things, but you still have to know about those things. Like, you still have to worry about AppArmor and like SELinux permissions and permissions on a drive and all. You don’t have to worry about that in a Wasm pod because you’re not actually running a container. You’re not running a Linux file system inside of it. You don’t have to So, the security model— there’s still security considerations, but it takes out a lot of the complication of that, which is very nice for people who are not necessarily infrastructure engineers. I’m assuming that many people who listen to podcasts working in DevOps are building platforms or things for the engineers that they support, and the engineers now don’t have to worry about all these other security things, and also, If you’re the person writing the platform, you don’t have to worry about Linux permissions and things as you run a module. So that’s where that security benefit really comes into play. It doesn’t automatically fix everything, but it reduces the platform, like the surface area for the attack when you’re doing it.

Bridget: [00:10:59] That’s great. Now you’re speaking my language. So you’re saying we can spend less time stressing out about SELinux. That sounds great. So how is it that the— and I do pay attention to what the DevOps world is talking about and what the WebAssembly world is talking about, and I don’t necessarily see the DevOps world thinking about WebAssembly much. Is that because they think, oh, it’s something that people will write and then we’ll deploy it and it’ll be just like everything else people have written and that gets deployed and that we don’t think about? Like, what is going to change in this exciting WASI future? What is different for people who are used to just seeing the workloads that are deployed as interchangeable widgets?

Brian: I think the first thing to mention is that WebAssembly on the server side is a new concept. It’s something people haven’t been thinking about much. Most people, when they think of WebAssembly, they think of something that got compiled and you’re running it in your web browser to make you know, a game appear in your web browser instead of using JavaScript. And this is using WebAssembly on the server side. So that’s absolutely a new concept most people haven’t even considered. The second part of your question is, you know, what does that change for us? I think the idea of the write once, run anywhere makes this a game changer in terms of cost-effectiveness. So, add all of those security abilities that we had, but then put it on a 96-core ARM processor, and that ARM processor can run your WebAssembly files just as well as the more energy inefficient AMD processor can, and you’ve got a pretty big winning strategy for lower cost running your workloads.

Bridget: [00:12:55] Yes, more secure and less expensive. That sounds good.

Taylor: Yeah, that’s one of the things that we liked as we wrote this with Presslet is that because everything is just essentially running as a separate thread in a process in Presslet, you end up being able to pack things more tightly because if something isn’t doing any work, it just parks the thread. You haven’t had to allocate a memory space, because if you’re not familiar with the underlying architecture of how Kubernetes is working, it has something that’s called a shim, a containerd shim, and this thing is a separate process for every single container. It’s very lightweight. I’m not trying to complain about it. They’ve done a very good job of keeping it minimal and on top of things, but the thing is, you still have to spin up a whole other process, whereas this, everything, because it’s all within that same process, is just a thread that can be completely parked if nothing’s going on and switch between those. And so that really helps in situations like an ARM-type device, anything in that edge computing or things of that nature where you have more limited resources. You’re going to now have something that you can pack things a little bit more tightly instead of having this Docker overhead that’s running on such a small machine. When you have the big 94-core monstrosities that we use sometimes, that’s fine. Docker is a very minimal overhead in the larger scheme of that. But when you’re talking about, like, you only have a gig of RAM and a small CPU, having something that can take up less space like that is so important. And that’s what we think one of the big benefits of having WebAssembly in Kubernetes will be.

Brian: [00:14:36] WebAssembly specifically was designed to be interpreted in a stream. So you send a stream of WebAssembly bytes, the compiled WebAssembly binary, and the runtime interprets it as it’s streamed across. So you don’t have to load the entire WebAssembly file in order to start executing it, and that lowers the memory footprint that’s required on the host device drastically. You could have a really large WebAssembly executable, but the, the processing of that happens in a stream of bytes rather than all at once. That makes the overhead much lower.

Bridget: This sounds like the sort of thing people want for their Internet of Unpatched Things, right? I mean, but this hopefully keeps the Internet of Unpatched Things from being quite as terrifying.

Brian: You would hope so, but my expectation is that even something as secure as WebAssembly on the server isn’t going to prevent Internet of Things vendors from making insecure configurations.

Bridget: [00:15:38] Okay, so this is pretty interesting, but this is also— I want to dive a little bit more into the actual Krustlet project. And in the green room, Brian made a comment that I thought was hilarious that I want to hear more about. And I quote, I built 95% of an evil thing today. So start there.

Brian: All right, so there’s 2 pieces of Krustlet that are interesting right now. We’ve got 2 execution runtimes. One is for WASI, which is the WebAssembly System Interface, and WASI has a limitation currently that it does not do networking. So, we can schedule a WASI pod to run, and it will execute that WebAssembly, and it will interact with its own environment, but we can’t really make it listen on a port or return a web result yet. Because WASI hasn’t matured quite enough yet. That’s gonna happen soon, but until it does, we’re supporting another WebAssembly standard created by some smart folks over at Capital One called WebAssembly Secure Capabilities, WASCC. And WASCC uses RPC between the host and the WebAssembly module to execute requests and responses, which means that it can do network calls. So you can start a Wasm instance in WebAssembly that listens on a web port. So you can write a web server, for example. One of the interesting concepts of Wasm is that it has what they call capabilities, and those capabilities are secure. The idea behind capabilities, as far as I understand, was that as a shop building, say, microservices, for example, you’ve got a common set of services that you want to run for your hosting of these microservices, say, logging, key-value operations, database information, things that all of your microservices need, you don’t want to have to configure those all in each microservice. So instead, we configure them once on the host. The WebAssembly modules don’t know what they’re talking to. They just know they’re going to make a request to a key-value store. And those are capabilities. You can hot-swap capabilities on the fly in your host, which is really cool. You’ve got your key-value storage set to Consul, and Consul fails for whatever reason, you can hot-swap it over to Redis, and the actors don’t even know the difference. That’s really cool. This afternoon, I was hesitant to start on new code, because the podcast was coming up. So I thought, what if I built a capability that did something evil? And so I generated a new capability and I called it Shell. And this capability allows you to, from the secure WebAssembly environment, make an RPC callback to the host and execute a shell command as the process that the host runs at. So imagine, What could go wrong? Nothing could go wrong there, right? Imagine simple things like sh -c and then ls, you know, get a directory listing of the directory that you’re executing in, or something much more interesting like rm -rf root. That’s why this is obviously just a fun proof of concept, but it does prove that the capabilities are both secure and also just as smart as you make them, and security is always in the eyes of the beholder.

Bridget: [00:19:21] All right, Taylor, so what do you think of what Brian built, and what directions does that send your mind?

Taylor: I mean, the first thing is like talking about— it’s absolutely terrifying. But beyond the fact that it is terrifying, it’s showing the flexibility of what you can do. This is still very new. And that’s one of those things that we’ve kind of touched on a little bit but haven’t really addressed is that WASI especially is very new. Most languages have WASM support, but they don’t have WASI support necessarily. There’s only 2 or 3 languages with some strong support and some that are starting to add it. And one of those things is like this where there’s networking and stuff. And so one of the reasons we’ve loved using Wasmc is that it kind of fills that gap in the meantime while we’re trying to continue iterating over that as a whole WebAssembly community about how we can implement all the parts of WASI and make it so that it can do things like networking. But what Brian showed— and Brian’s done a couple interesting examples and demos and things around what you can do with inserting in a provider and then being able to hot swap it on the fly. And that’s the thing is beyond just doing that in Wasm, that’s something we envision that hopefully you could do it with other Wasm things as well. Because you could hot-swap something out without too much difficulty because it’s just a stream of bytes. If you can pause and move things around properly, it’s something that could be completely doable.

Bridget: [00:20:49] That sounds like a really interesting way to— I mean, just looking towards the future— to build really resilient infrastructure. Not only do you have failover between your various clusters of insert data store here, But even the capability, as you mentioned, of being able to swap in a completely different one. But what does that mean in terms of how people are, say, writing the data or even reading it if they have built-in expectations? Does this help put in a layer, a shim of sorts, so that they don’t accidentally do anything that’s too specific to the kind of thing that you’ve now swapped them away from? I’m struggling with how you prevent people from tripping over their own feet if you’re going to suddenly replace their folders, crystals, or whatever.

Brian: You’re definitely creating a least common denominator interface between the things that can provide your capabilities. So for example, if you’re using all the different logging providers that are available as a service out on the internet, you’re not going to be able to, to put all of their features into that interface that lowest common denominator. So instead, you’re just going to have an interface that allows you to print debug messages and warning messages and error messages, but maybe not the more complex things that each provider might operate. So yeah, the capabilities providers abstract over what that capability might be, which in turn means that you may not necessarily get all of the features of the different providers that That reminds me of how I was just actually watching a really good webinar this morning about the ASMI, Service Mesh Interface Specification.

Bridget: [00:22:36] Say that 3 times fast. My colleagues and collaborators who were speaking on there were kind of talking about exactly that. It’s a spec. Everyone implements, hopefully they meet the spec, and then they might go off a different direction with their own customizations that aren’t necessarily at the base level of what’s in the spec. But part of that whole community effort is figuring out what should be in the spec. So is that happening over in the WASM, WASI, WASGI, I’m gonna need an acronym guide, community? Like, who is the WASM police who decides whether or not something is in or is not in the lowest common denominator that things are supporting?

Taylor: Yeah, so there’s something called the Bytecode Alliance that I believe is operated by Mozilla. And so it is a conglomeration of a bunch of different people and companies and anyone who’s really interested in the WASM space. They’re driving everything around the WASI spec. WASM might be part of W3C. I can’t remember where the specification is at, but it’s a longer existing specification that’s been around for a while. And so All these new things with WASI and that future work is done through the Bytecode Alliance, which has been a great group to work with. They’re trying to define different standards and figure out how to implement them. Nice.

Bridget: [00:23:59] We’ll have a link in the show notes to the Bytecode Alliance. This does make me wonder, these are names we’ve heard of and relevant folks in this space who seem competent. Does that mean that this is all ready to be rolled out to production 3, 2, 1 now? What is the recommendation or caution around that? Is this ready for primetime?

Taylor: Well, if you go to the main Crestlet repo right now, you’ll see big warning signs with construction sign emojis that says, this is not ready for production, because it isn’t. We don’t have networking in one of the providers because it’s not there yet. We are still working on things. If you’re familiar with Kubernetes, You’re missing things like init containers and volumes and all sorts of things like that that are still not there, but what is there is the basic functionality to run pods. It’ll run, and it’ll stay running until it is done or however you set it up. That is there, but that’s definitely not a production-quality thing yet. We’re still a little bit out from that, but that’s why We’re looking for anyone who’s interested or trying it out because then that feedback helps us identify bugs and problems and what we need to implement and what they need to be able to start getting closer to an actual real production use case.

Bridget: [00:25:24] I feel like you are an engineer and you answered in a very engineering way, which is you answered for the specific use case of Crestlet. I think I was asking a very uninformed question of, WebAssembly: real? And so like, is there, um, Krustlet is obviously very early days. Is there something Wasm-ish that you can point to that’s actually being used for realsies on something that people could click and see? Like, where is the general WebAssembly space in terms of something being used not just in test cases?

Taylor: Yeah, most, uh, most major technology companies with big websites are using Wasm at this point. One of the examples I saw, we were talking about some this morning, I saw one, there’s AutoCAD from Autodesk, they have a web version and that is running in WebAssembly. And I think they spoke about that at a Google conference. And you can go and look and most big companies and big websites are using WebAssembly in the browser. And so that is completely 100% ready for production. It’s the WASI side of things, which is running it on the server away from a browser, that’s a little bit more undefined. There’s not too many— I don’t think there is a super good production example I can think of. There’s just some really good workshops and demos and things. Some of those are part of Breslet and others are outside of it, of what you can do with WASI.

Brian: [00:26:54] So I can think of 2, at least 2 production usages of Wasm on the server side right now, and that would be edge computing providers like Cloudflare and Fastly. Both of them are using different versions of Wasm runtimes to modify edge HTTP network requests. Fastly has the Lucet Wasm runtime, and Cloudflare, I don’t remember, I think they use something based on V8 isolates. But I don’t remember which runtime they’re using. But both of those allow you to upload WebAssembly and execute them on the edge. One example of that would be my website, briankettleson.com, which is running in WebAssembly right now up in Cloudflare, just because I can.

Bridget: So that’s really interesting because I feel like a lot of times people who are working in this platform-ish, DevOps-ish, whatever space, don’t necessarily get a chance to jump in and start creating something that is very new and very exciting. Maybe the only people using it in production are the big edge providers. What would you say in terms of scope down now to this Crestlet project, what would you say is the best way for people to get involved? Say they wanted— they’re like, this actually sounds awesome and I want to try it. What should people check out? What should people do?

Taylor: [00:28:25] This is something that maybe can seed some ideas in people’s minds. One of the demos we’re going to work on is actually going to have some Cresslet running the Wasm provider and a Cresslet running the Wasi provider and do HTTP-triggered job processing. And so the Wasm modules will be able to come in and receive the request and do something and then kick off some sort of batch processing that’s done using Wasi. And so that’s one that maybe could give some ideas, some basic job processing or some simple HTTP response. I also think it’d be really interesting to have a Wasm actor return, like basically serve up web pages that are also running on WebAssembly modules in the browser just for funsies. But those are like, there’s actual legitimate things for that. Like if you have a lower traffic of application, that’s something that you could try right now.

Brian: One of the use cases that I’d like to explore is IoT. I do a lot of barbecuing, and I previously built a Raspberry Pi controller for my giant barbecue. And it’s— right now it’s running Go, and the Go microcontroller turns the fan blower on and off to control the temperature of the fire underneath the barbecue pit. I’d like to turn that into a WebAssembly module and allow that to run on the same Raspberry Pi, but with a much lower memory footprint and much better security. So that’s, that’s the next project that I intend to use WebAssembly for.

Taylor: [00:30:03] There’s some interesting things you could do around— I think I had gotten past mentioning serving up a website over HTTP using that. So like, that, that’s another one. But you can do like anything that I was thinking of, like using a function as a service, like where you have a small application that’s kind of like bursty. That could just respond to like specific events. It’d be really interesting to serve up an API that way. Those are the kinds of things people can start doing now and would actually help us identify features that we’re missing or bugs that we have that are related to more quote-unquote real-world projects.

Brian: And one of the interesting things about the architecture of the Krustlet as it is, you can run the Krustlet process on on any device, you know, from your Linux server all the way down to your micro:bit controller, which means you can extend your Kubernetes cluster all across the globe, as long as you have the networking bits set up right. So, with a little bit of ingenuity, you can have a pretty massive cluster of Kubernetes nodes that are powered by really tiny little devices, if you wanted to.

Bridget: [00:31:13] Disclaimer, the devices should belong to you or you should be authorized to use them. We are not advocating any such, you know, Internet of Unpatched Things hijinks. Brian, this actually reminds me a little bit of the Virtual Kubelet project that you’ve also been involved in, in that the idea of expanding the reach of your cluster. Do you want to kind of compare contrast what the projects are like there?

Brian: Yeah, actually, Krustlet is a Virtual Kubelet in the sense that they both do exactly the same things. They both operate against the Kubernetes API and tell the API, the control plane, that they are a kubelet and they are here to operate scheduled traffic. The difference is Krustlet is written in Rust and Virtual Kubelet is written in Go.

Bridget: Wait, now I feel like this is the moment where we have to have the controversial conversation of, wait a minute, I thought Go was the language of Kubernetes. What is this Rust thing, and why are we writing this in Rust? Why?

Taylor: [00:32:21] So I wrote a blog post about this. That’s probably the easiest thing to link for those who want to check it out after. There were 2 main reasons around why we decided to go with Rust for this as opposed to the language most often used to run Kubernetes, which is Go. And the first one is that, like I had mentioned before, Wasi does not have a ton of language support yet. Rust is one of the best that’s out there right now. And so what it is able to do is compile those Wasi targets with not too much extra effort. So because of that, a lot of the other projects around WASI and WASM stuff are done in Rust, and so it makes it easier for us to import those in and embed them in the project. So that’s the first reason. The second reason is that we wanted to apply some of the strengths of such a powerful language like Rust to the Kubernetes space, and so that was something that I dug into in depth in that blog post that I wrote, but it It has some very good safety guarantees that come with the language that are caught at compile time. They result in compile-time warnings— or sorry, not warnings, compile-time errors. And so those are very, very important to us. It prevented some bugs from happening that we would have introduced thinking they were okay because it stopped us with its various protections around how long data is supposed to live and whatnot. And so the strengths of Rust are, like any other language, they’re not a silver bullet, but they solve a lot of problems and help us avoid a lot of bugs that have existed in other systems-level languages for such a long time. So that’s one of the other reasons we really wanted to do it.

Brian: [00:34:14] Yeah, there’s definitely an opportunity cost in adopting Rust. As a longtime Go programmer, one of the things I loved about Go was It was really easy to pick up Go, start writing productive code with it. It’s easy to read Go. Rust is harder to read, and it’s harder to start off with. There’s a logarithmic learning curve in Rust, and it took me more than a month to be even close to productive when I started doing this, maybe in February. I didn’t really start getting decent code written until end of March, so maybe even 2 months before I was writing useful code, and even then, it was leaning on people like Taylor, who had been doing it for a long time. So, it’s harder to pick up Rust. There’s definitely a big learning curve. The nice thing is that once you get past that initial curve, you really start to appreciate how much work the runtime, or the compiler, is doing for you. It’s moving the direction of the footgun, so that it’s not pointed at you. You’re deferring that so that you can’t really shoot yourself in the foot so well.

Taylor: [00:35:24] Yeah, and I would also add something that I slightly disagree with is that I find Go very easy to jump into and start reading with some of it, but as projects get larger, it’s much harder for me to follow along. And once you understand some of the things of Rust, which is some of those weird things you have to learn, it actually reads fairly easy and in some cases better. Especially around control flow management. There’s some powerful things. If you’re super interested in languages and stuff like match blocks and error handling are very, very interesting. You might not agree with their flavor or whatnot. That’s everybody’s personal choice, but I find it very readable once you learn how to leverage those.

Bridget: Now, as somebody who doesn’t do a ton of software development themselves, but maybe deals with all of the ephemera around software. I will say that the Go mods versus deps thing was kind of messy. Is there— are there any pitfalls in Rust that we need to watch out for that, you know, it’s a moving target and this is a specific area that’s going to be iffy for a while, anything like that?

Taylor: [00:36:34] Actually, that’s one of the big strengths that I talked about in my blog post was around Cargo is an absolute gem, which is the package manager for Rust. It is— so one of my other things I do is I’m one of the core maintainers of Helm, and if you touch Kubernetes at all, you’ve probably at least heard of Helm. It’s a very big Kubernetes project that a lot of people use, and we use a lot of Kubernetes dependencies. And from my personal point of view, every time I’ve ever had to add modify, or upgrade any of the Kubernetes libraries in Go, it has been an absolute nightmare for us. We spend hours and hours of work, and then we have to manually vet a bunch of things. It’s quite difficult. But Cargo has just a bunch of really good things, such as conditional compilations called features, that are very powerful and allow us to— One of the things that we use, it’s something super simple but very handy, is In one of our Rust packages, they’re called crates, we expose the ability to use a command line flag parsing if you want to, and you bring that in by enabling one of these features. Otherwise, the dependency is not brought in and it’s not even compiled into your code. And so not only has it been easier to manage those dependencies for us, it’s also added some other really interesting and helpful things. From a developer standpoint. And so I actually consider the dependency story for Kubernetes in Rust to be infinitely better than Go. There’s just a handful of libraries that are used, and a lot of those things have those features that allow me to bring in only what I want. The only thing that they’re missing is some of the more advanced stuff like patching, and I’m not talking about sending patches, I’m talking about generating patches and doing things like reading streams of resources and whatnot. But the main functionality is there, and it’s very nice to use.

Bridget: [00:38:36] We’re just scratching the surface of really interesting things, which means, of course, we’re running low on time. So where can our audience follow your work or learn more? Let’s start with Brian and then go to Taylor. Just, you know, where can people find out more about this stuff, about your projects, etc.?

Brian: The main repository for Krustlet is at github.com/deislabs/crustlet. I think that’ll be in the show notes, of course. And you can see the code there and the open issues. We absolutely love having people jump in and contribute and try this out. Most recently, in the last couple weeks, we’ve had quite a few different people come in and play with rustlet. So that’s, that’s a ton of fun. We’re a very welcoming group. And I have to say that the core group of committers are incredibly patient with people who are learning Rust too. So don’t be afraid to come in and try to make some, some changes. People will be very friendly for helping that.

Bridget: [00:39:46] Awesome. And we’ll have a link in the show notes to your blog as well, Brian, since you just recently published a bunch of interesting stuff in this space. I did.

Brian: One of the things I’ve been enjoying is streaming my coding efforts, several of them recently with Taylor, on Twitch. So there’s links to those blogs, links to those videos up on my website. So you can watch us live coding Krustlet. There’s 3 or 4 good videos up there. For learning how Krustlet works.

Taylor: And stumbling our way through it. It’s quite entertaining if you have a little bit of imposter syndrome to come see this in action.

Bridget: Nice. And will you be doing more of those streams coming up?

Taylor: Oh, I’m sure.

Brian: Absolutely, yes.

Bridget: Excellent. All right, Taylor, where can people find out more stuff about these projects from your point of view? So there’s some—

Taylor: those blog posts that we mentioned before. I would also recommend, and we’ll link these. I’ll make sure we have the link. One of the product managers over here at Microsoft wrote a blog post that’s a more higher-level view of Crestlet and where it fits in and kind of our views around WASI and WASM here in the future. So I would highly recommend that. So those blog posts are a great place to start. And if you want to, if you’re interested in all this, we’d love to see you over in the Crestlet repo. We’re definitely looking for anyone who has experience with doing these things in EKS, GKE, DigitalOcean, IBM, all the cloud providers, because we want to make sure that these instructions are available to everybody. This isn’t— although the primary contributor base are people from Microsoft, this is not just something we intend to be a Microsoft-focused product. It’s supposed to be something very open to the community that people can contribute to. We already have people working on some EKS instructions, and we’d love to see those kind of things. And any use cases or demos, just let us know, tag us on Twitter, whatever it might be, so we can spread that around and show people some examples of this work we’re doing.

Bridget: [00:41:49] So there’s going to be— I can see a lot in these show notes. So if folks want to head over to arresteddevops.com/crustlet, with a K, for this episode’s show notes, Visit arresteddevops.com/itunes, leave us a review in the iTunes Store if you want to help other people find the podcast. I have no idea how that works, but apparently, if people leave reviews, dot, dot, dot, profit. We’re also on Spotify and iHeartRadio, if you’re into those. Thank you so much, Taylor and Brian, for joining today. It’s exciting.

Taylor: It was a pleasure. Thank you for having us on, Bridget. It’s been fun talking about this.

Brian: Thank you, Bridget. It’s been a pleasure to be on.

Bridget: I’m Bridget at Bridget Kromhout. This is Arrested DevOps, and remember, there’s always DevOps in the banana stand.