← All episodes
EPISODE 173VIDEOAugust 4, 2021

Multicluster Service Mesh with Phillip Gibson and Annie Wang

Read the transcript

Bridget: [00:00:00] This all, I’m not gonna lie, sounds very complicated. It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout with a great show for you today, but first, a word from our sponsors. This episode is sponsored by CircleCI. Designed for modern software teams, CircleCI’s continuous integration and delivery platform helps developers push code with confidence. Trusted by thousands of companies, from 4-person startups to Fortune 500 businesses, CircleCI helps teams take their software from idea to delivery quickly, safely, and at scale. Visit arresteddevops.com/circleci to learn why high-performing DevOps teams use CircleCI to automate and accelerate their CI/CD pipelines.

[00:01:05] BridgeCrew is the all-in-one cloud security platform for developers. They automate and embed security throughout the entire development lifecycle, so you can streamline your DevSecOps toolchain into one solution. By integrating infrastructure as code, security, and compliance into your version control systems and CI/CD pipelines, BridgeCrew empowers you to find, fix, and prevent cloud misconfig faster. Get started with BridgeCrew for free at arresteddevops.com/bridgecrew. This episode is brought to you by MacStadium, leading provider of cloud solutions built on Apple Mac hardware. As more teams are working from home, having your Mac build infrastructure in the cloud can make it easy for your app devs to work more efficiently. No need to have someone in the office keeping an eye on the Macs. Let MacStadium do it for you. And if you need a fast, scalable, modern way to run Mac virtual machines, MacStadium’s virtualization platform, Orca, is purpose-built for running iOS and macOS CI. Orca takes a standard macOS VM, puts it inside a Docker container, and then uses Kubernetes to orchestrate everything, all on Mac hardware. Orca is easy to integrate into your current workflow with plugins for all the popular CI tools like Jenkins, GitHub, GitLab, and Buildkite. If you’re building apps for the Apple ecosystem, learn more about MacStadium at macstadium.com/arresteddevops. From that link, you can also get access to a free 2-hour sandbox to give Orca a try.

[00:02:47] We’ve talked before on this podcast about Kubernetes and service mesh, but we can get more complicated than that. To talk about this exciting topic of multi-cluster service mesh, comma, what even is, we have 2 guests, and I’m going to ask each of them to tell us a bit about yourself, like where you live, what you work on, what surprised you the most when you started looking at multi-cluster service mesh, starting with Philip.

Phil: Hello everyone, my name is Phil Gibson. I’m a PM here at Microsoft. I’m focused on our cloud-native security projects, OSM, Open Service Mesh, being one of them. And I live in Atlanta, Georgia. I’m originally from Chicago, so go Bears. And I’ve been collectively with Microsoft for about 13 years or coming up to 13 years.

Bridget: All right, so incorrect NFC North opinions aside, go Pack Go! But I am really excited to have Philip here giving us his insights because I think he spent a lot of time looking at how specifically how end users use things in this space. Which ends up being key for those of us who work at vendors. It’s like, hey, it’s awesome that you want to develop some software, but how is it going to get used? And I— that is actually— that specific topic is why I’m also excited to introduce our second guest today. Annie, you’re on the last week of summer internship with this team, being mentored by Phil. And I’m wondering if you can tell us a little bit about yourself, where you live, about school. What surprised you the most in this internship?

Annie: [00:04:30] I’m Annie. I’m a PM intern on the team this summer working on Multicluster. I am originally from the San Francisco Bay Area, but I’m an uprising junior at UPenn and I’m studying computer science with a focus on cloud computing. And yeah, I think the most interesting part of this internship for me was just getting to work with an open source community and seeing how being a product PM on an open source team is different from being one on a regular product team.

Bridget: For those who’ve listened to this podcast for a while, we’ve done a number of ADO episodes about Kubernetes, including one last summer explaining service mesh with Michelle Norelli and Delian Raychev. So, restodevops.com/service-mesh, you can go listen to that one if you want the intro. We’re going to be at kind of a 201 level here today. The exciting topic of multi-cluster service mesh. Phil, I’m going to toss it to you first and say, what does that even mean? And show your work.

Phil: [00:05:35] Yeah, yeah, so multi-cluster, right? I guess at a high level, probably the most accepted or popular description of a multi-cluster service mesh is the ability to be able to manage multiple Kubernetes services connected in an east-west communication flow across multiple clusters. As I’m saying that, that probably sounds super convoluted, but I think what we found, and we’ll probably get into this, just the discovery of, hey, who needs a multi-cluster service mesh? It means a lot to a lot of different people, you know, and it’s really kind of like trying to figure out what that true definition is and, and, and what problems we’re trying to solve, uh, you know, for, for people trying to consume like a multi-cluster service mesh or service mesh in general, right? So I’m, I’m a bit rambling right now on this topic, unfortunately.

Bridget: [00:06:44] That’s great because I think that’s You hit upon what I think is a really important point, that being what are people trying to do? And you also mentioned something that I’m gonna, I’m gonna toss to Annie and ask her to explain for our listeners who hear east-west and think north-south, east-west. Wait, what do those things mean in a cluster? Annie, do you want to kind of give us a little bit of insight into what exactly the directional stuff is?

Annie: Yeah, so I’ll first start off by talking a little bit about what happened before multi-cluster functionality. So what Phil talked about, something called north-south traffic, where traditionally if you have 2 clusters and you have pods inside one cluster with services, pods will have to exit that cluster to access the services of the other cluster, Cluster B. But then now with east-west traffic, We’re enabling the ability for clusters to not have to— or pods not to exit that cluster to talk to those other services so they can access the ingress of the other cluster without having to exit either cluster.

Bridget: [00:07:58] That opens up so many questions like, do these clusters have the exact same security model and how do pods know which stuff in the other cluster they should talk to? How does that work?

Phil: Typically, what we’re talking about now is, if you’re familiar with how Kubernetes works and this whole notion of a control plane, and then there’s data plane where a lot of the traffic is happening, and the control plane is where the operators are configuring configurations and everything. Being able to take, one, that control plane, functionality. So how you’re configuring your cluster, extending that to a service mesh, and then having that control plane distributed across multiple clusters as well. So that’s kind of almost step 1, is making sure you’re able to control and do all your CRUD operations for your mesh across a distributed architecture there. If one system goes down, you still have access to it. And then 2, Again, kind of what Annie was talking about, the north-south, east-west communication. Now we have essentially a method to where you have services in one cluster can communicate, again, I guess we’ll say horizontally across multiple clusters without having to go through kind of like, or enter in through a normal ingress. Wow, just as I’m talking about this, it’s like, how deep do we want to go, right? Because you got the traditional ingress controllers, which, if you’re familiar with Kubernetes, hey, that’s how you expose your service out to the world. But then with multi-cluster service mesh, we now have a concept of an ingress for the service mesh, right? So again, not going through those traditional ingresses, but going through a more east-west ingress where then we’re populating routes and, uh, and everything else.

Bridget: [00:10:14] So when you say it’s ingress but it’s different, and then you say east-west, my mind immediately goes to, gosh, coming in from the side. Is this at all, uh, side channel attacks? Like, exactly how do you connect these clusters that may have different security models, like in a multi-cluster scenario, do both clusters need to have all the same access to all the same everyones, or is there a way to set any access more granularly?

Phil: The cluster itself can be a security boundary, so you’ll definitely be able to within your cluster have its own certain RBAC controls or policy controls, etc. I guess, you know, we’re using ingress as the, the raw term, which means to kind of enter into, right? Uh, and instead of, again, that traditional— what we all know of an ingress controller, you know, the popular ones out there, uh, which are exposing services to something that’s coming into the cluster, uh, same concept is just, you know, for the service mesh, we’re saying, hey, we’re going to have an incoming ingress on an additional cluster that we’re able to populate routes through have that configuration kind of sync through the control plane that’s going across all these clusters so that service A that’s in cluster 1, that when it needs to get to service B in cluster 2, it knows that, hey, I actually take this route and that’s going to end up touching this ingress that is strictly just related to the service mesh, and it gets routed out of the cluster that way.

Bridget: [00:11:56] This all, I’m not gonna lie, sounds very complicated. Um, and I thought Kubernetes was hard. So I think, Annie, you have a great blog post you wrote, um, talking about conceptually from a high level some of the benefits of multi-cluster. And I think Phil hit on a few of those, but maybe you can kind of lay out for us, lay out for our listeners, why would someone want to do this stuff that sounds frankly kind of like a lot? What are the wins?

Annie: I think Phil talked a lot about east-west traffic and how that’s different from what was traditionally done. But some other benefits include, before, a service mesh only runs on one Kubernetes cluster, so you’re limited to the size and scale of that Kubernetes cluster. But by enabling the ability to deploy services to multiple you’re allowing your applications to be scaled more horizontally as well. I think another big benefit has to do with disaster recovery and failover. One aspect of that would be the ability to deploy the same service to multiple clusters, and that enables some sort of failover if something happens to a single cluster. Something else that is also possible would be the ability to redirect traffic away from unhealthy clusters, and this helps minimize damage and makes sure that your workloads are still running.

Bridget: [00:13:23] Okay, so wait, I’m going to pause you right there because I’m hearing things that we’ve heard before. I’m hearing, oh, load balancing, and I think, okay, load balancers, we’re familiar with that, but is load balancing against entire, like, you know, a whole bunch of different clusters are behind your conceptual load balancer? Or like canary deployments, blue-green, that sort of thing. We’ve heard that before, but now we’re hearing instead of just pods or services or whatever, you’ve got entire clusters behind these. Is that the right framing?

Annie: Yeah. The idea is if you have one unhealthy cluster, you’re able to divert traffic away from that unhealthy cluster towards your healthy clusters. That enables some sort of failover.

Bridget: Hey, and then there was a little bit more, I think, that I interrupted you, but I was like, we got to get a little more clarity here. What else you got for us?

Annie: I think the last impact could be zero downtime between Kubernetes upgrades. If you have multiple clusters running services and you need to upgrade some of those clusters to different Kubernetes versions, you can ensure that your workloads continue running. As you’re updating your clusters by updating them one cluster at a time.

Bridget: [00:14:38] That sounds great, but I have read the Kubernetes release notes a few times and quailed at all of the API deprecations and breaking changes and this, that, and the other. I’m going to look to you, Phil, and say, do we have any kind of gating or like, oh, okay, we’re going to do a zero downtime cluster upgrade, but then these APIs work over here and these ones don’t work over there. How would a multi-cluster scenario handle differences between the clusters?

Phil: Yeah, you’re getting into it now.

Bridget: This is the hard stuff, but I want to know what are you thinking?

Phil: Yeah, it’s funny when we get to this level because I think the value of what made Kubernetes so popular from the start is it’s offloaded just a lot of labor-intensive configurations. So like we’re talking about now, load balancing, scaling applications, automatically putting runtimes on distributed resources. And I think we kind of take for granted now because we’ve just grown accustomed to having this functionality as default, right? But it’s doing a lot under the covers. Yeah, so when you’re talking about a multi-cluster service mesh and specifically around APIs, within the mesh itself, obviously there’s controls and webhooks of things that it’s gonna reach out and do. If you’re talking about your application, like if I’m exposing my application, it has certain paths and APIs, You can also kind of blend that into your config as well. And those are things that we are currently working into the service mesh too. Because again, customers or users, the chemistry set is cool, right? But I think people really want something that’s simple and easy to implement. And so I know in our journey and us discussing multi-cluster inside of OSM is really, hey, how do we keep this, the experience super simplified so that we kind of shield you from kind of all the heavy lifting that goes on? So there’s still a lot to be imagined there, and we’re still kind of working on how best to make that experience.

Bridget: [00:17:21] Yeah, I think that Annie had some, some stuff she wrote about the philosophy. I would love to hear what’s your take, Annie, on coming into this project and wanting to give people the experience of, you know, reducing the complexity like Phil mentions.

Annie: And I think I talked about it briefly in my blog post, but we really wanted to align with sort of the OSM philosophy of making things really simple to use and to configure. Our philosophy for Multicluster is to enable these complex, larger use cases while still maintaining a really simple and easy-to-use experience.

Bridget: But hey, complexity is conserved, so it’s going somewhere. Apparently into a bunch of YAML, or is there still YAML here? Everything is YAML, right?

Phil: There’s YAML. There’s ConfigMaps. There’s new CRDs. All of the above.

Bridget: You get all the everything. So I want to switch gears for a minute here and say, now this was an exciting new experience for our team having an intern working with us on this. I think it was probably also kind of an interesting experience for Annie to come in and start working on this project. Annie, I want to hear a little bit from the point of view of you weren’t already steeped in the Kubernetes ecosystem and having service mesh opinions, TM, and all that. What did you think when you got this intern assignment? Starting from, what is a Kuber-what? What were you thinking when you looked at this?

Annie: [00:19:01] Yeah, I was definitely very overwhelmed. So many words. Kubernetes, service mesh, even things like, yeah, simple things like OSS. I had no idea what that meant. So yeah, I think it was a very overwhelming experience to start, especially because in university, since Kubernetes is relatively new, you don’t learn about stuff like that in school. Yeah, so I think the experience of, I think being an enterprise PM was interesting, working on more technical projects, because I, like you said, don’t have a starting opinion on a lot of these. I personally am not a service mesh user. I think getting to talk to a lot of different people who use service mesh or people who have experience in service mesh was really interesting and valuable. Just hearing what they liked about it, what they didn’t like about it, things that they think could be improved. I think that was really great. Opportunity to increase my knowledge in service mesh and really valuable to my internship too.

Bridget: [00:20:09] You touched upon that discussion. Of course, you’re talking out there to people in the community and users, enterprise customers, etc. I think Phil has spent a bunch of time looking at this line between project and product. What would you say, Phil, is your approach for when you’re figuring out this is something that needs to be a product offering, this is something that should be in the open source project? How do you approach that?

Phil: Initially, I probably approach it the same way. For us, the benefits of open source are tremendous. That project is either going to live or it’s going to die on the vine. So what better way to have some immediate feedback than to share it with the community and see if there is a following? For it and that it’s answering a need for a lot of people, right? If it’s just something that is kind of, you know, a simple add-on to something else, then maybe it won’t get that traction. But I think, you know, we really try to focus on, hey, what are the gaps out there? What are the pain points that a lot of customers are talking to us about? You know, those conversations where you say, hey, I only wish it could do this. You know, when you start to get a bunch of signal on that, you start to think like, hey, there’s probably a need for some type of functionality around this. You know, we’re getting multiple signals from different types of customers, et cetera. And then, yeah, then it’s really about us trying to figure out kind of how we’re gonna create this and expose it to the community, et cetera. So yeah, no, I think, what better way to either know, hey, this is something useful, or this is something that, uh, it’s not useful, than to share with the world and get some feedback. So definitely, uh, you got to have thick skin, uh, in the open source game, right? Because, and, and you, you really, uh, you know, a lot of the times, you know, a lot of the criticisms, you know, uh, you know, they could get you, but, you know, the fact remains there’s just maybe some truth behind know, those criticisms too, and those need to be listened to as well.

Bridget: [00:22:27] Yeah, that’s a, that’s a really important point. I think that we’re all learning when we’re working in public, right? And we’re not just releasing something that is correct and right, because why would you be working in public if you, uh, didn’t want to learn from the community? Um, so Annie, I know you’ve participated in a bunch of community meetings, as well as customer calls and written a blog post. I’m wondering if you can kind of talk a little bit about the, the side of the product work that is this open source, that is this community, that’s GitHub, etc., what you’ve learned about that.

Annie: The biggest thing that sort of was interesting was just the challenges of building features in open source versus product and being able to find that customer or the need, like Phil said, just sort of rallying the community to really throw things out there and seeing if they not stick, but if they solve an actual problem. I think that was interesting to me where I think on more mature products or just traditional products, you have a very defined group of customers that have a very defined need, and it’s easier to sort of figure that out, whereas in open source, you have this huge community and everyone’s using these projects for different use cases. I think it’s really interesting to kind of go out there, rally the community, and also sort of find the niche within the open source community that actually needs what you’re building. So yeah, I think that was the biggest thing that I learned from the open source community where, yeah, it’s just how to rally the community to figure out if the problem you’re solving is something that someone actually needs. That’s actually why we did release the blog post, just to invite discussion on multicluster and invite people to come talk about it with us and work on it with us so that we can figure out if it’s a project that solves a really big problem for people.

Bridget: [00:24:27] I know, Phil, you have an agenda item on a forthcoming service mesh interface spec community call, just about like, hey, what are some of these ways we’re evolving this spec in the public? Can you talk a little bit about the spec versus implementation and where you see the best places to push changes, Phil?

Phil: I think the spec is important. I’m assuming we’re talking about the Service Mesh Interface spec, so the SMI spec, for those that aren’t familiar with it.

Bridget: We’ll have a link in the show notes.

Phil: I think what’s cool about that project is there’s no code, like, as far as, like, the spec is concerned, right? So we always get people like, well, where do I go download this? Where do I go? And, uh, we have to clarify and say, no, this is, this is kind of this agreed-upon, uh, specification of how this, uh, APIs are to interact with things, right? And to me, that’s totally core to a lot of things, right? Because if you can’t agree at that point on, on kind of what APIs should, should make things uh, you know, available or light up features, then, uh, most likely any project tied to, uh, you know, spec is not going to fail. So, uh, it’s, it’s, it’s been a, you know, I, I wouldn’t say challenging, but because again we’re trying to get this consensus of like this whole large community to agree on a lot of different things, right? And hey, some, some of those checklists they happen quick, and a lot don’t happen as quick as we want them to, but we’re listening to the community to kind of tell us, hey, where does this spec need to go, and what are the most important things that need to be bubbled up for us to really enhance the specification? And I just think it’s just a good hygiene to have in any open source project.

Bridget: [00:26:35] When you talk about listening to the community, there’s an interesting nuance there because, as Annie alluded to earlier, not every member of the community needs multi-cluster service mesh. Annie, you want to address that a little bit more? Just like, what kind of applications or customers or end users are looking for something like a multi-cluster service mesh?

Annie: Yeah, you bring up a really interesting Because a really interesting thing that happened during my internship was, in the beginning, I was exploring users who would need it and what the problem is. Service mesh is a relatively new concept, so some people are just beginning to use it, just beginning to explore how they can use a service mesh in their applications, and finding the need for multi-cluster, where someone would benefit from the scaling, the failover, That was difficult at first because everyone, every user we talked to, had relatively simple applications. They were just exploring how to use service mesh. So yeah, I think— Phil, do you want to talk a little bit about the more specific applications of it?

Phil: [00:27:49] Yeah, let me just add to what you laid out there because I think you nailed it. Annie and I, we went into this and we were thinking, hey, everybody wants a multi-cluster service mesh, right? Like, that was our default. Like, and, and a lot of that, uh, comes— it stems from just the enterprise mind state, right? Hey, I got a very, uh, critical application, it needs to be distributed, I’m using the service mesh, then obviously I need to have a service mesh that’s distributed as well, right? So it comes from a good place, but I think What Annie and I found out in this journey is that the need for a multi-cluster service mesh was kind of directly tied to your maturity with Kubernetes. What we found out is that, hey, if you’re brand new to Kubernetes, multi-cluster service mesh is probably so far away because, hey, you’ve probably just refactored Uh, some application that was running on the VM. You finally got it into a container, right? You know, and now you just want to expose it out port 80, right? And then have people come shop at your, your website.

Bridget: [00:29:08] Um, and then port 443 as well.

Phil: Yeah, yeah, make sure things are secure, make sure you got your certs. Uh, and then I think from that, then you start building muscle, and I think probably the next step is, okay, How more apps can we onboard to the cluster? So I think, you know, again, initial step is, hey, we’re using Kubernetes. Oh, this is great, you know, all these built-in features, scaling, etc. Let’s just add to the cluster. Let’s, let’s just grow the cluster horizontally and on the cluster apps, right, right. And, and it’s funny, like, I remember, I don’t know, a year or so ago, I was reading about like the Alibaba Cloud, they have a cluster that had— I don’t know, was it like a million pods? Just some astronomical number. I’m sure someone will correct me, but I just know the number of nodes and pods was just insane. And so then you have that, but then again, now the enterprise is telling you, hey, we need to have a region here in this location. We need to have another region there, and we to have these apps, you know, distributed, you know, in case something happens, region goes offline, or whatever the case may be. And by the way, we’re using service mesh, and again, we’re going to need that in these other locations. Um, not to mention, let me, let me just back up, not to mention that some people don’t even need or know that they need mTLS, right? So again, I made that example.

Bridget: [00:30:39] Oh, do you want to I feel like we should gloss that for our listeners. Wanna clarify what is mTLS?

Phil: Yeah, yeah. So this is kind of, I won’t say it’s a source of confusion, but when we talk about it and people are like, oh, okay, yeah, maybe I do need that. So think about how the world works today. We talk about port 443, right? So TLS, hey, I go to my favorite shopping website, There’s a little lock there. I know that, hey, if I put in my credit card info, that channel is secured, right? So that’s kind of a— I wouldn’t say one way, but it’s just say one person has been basically validated of who they are by some other source, right? And that’s who gives you the certificate, and that’s why you trust them where you can put your credit card in. When we talk about mutual TLS, right, so the M is mutual. Mutual. So it’s TLS, same tech, but now what we’re saying is, hey, not only am I trusting who you are, you’re going to trust who I am too, right? So think about your favorite website that you shop at. Not only are you okay with putting your credit card and, and the details in your browser, but what if they knew, hey, you’re Phil Gibson and I trust you to even shop with me, right? And so that’s where we get the mTLS. So I’ve been through some iterations on that analogy, so hopefully that resonates with a bunch of people out there. If not, let me know, give me some feedback, and we’ll come up with a different one.

Bridget: [00:32:18] That actually brings up an interesting question that I’m gonna, I’m gonna toss to Annie, which is there’s an awful lot of acronyms, terminology. Um, the fact that Phil has very clearly pointed out not everyone needs multi-cluster service mesh. Not everyone needs service mesh. Maybe you’re just starting to need Kubernetes. Like, as you layer all those pieces in, um, which ones do you think you’re going to take, take back to school, talk to people about? Um, what I learned on my summer vacation is Kubernetes. Like, which stuff do you think is going to resonate with everyone you’re interacting with, um, in a college, you know, computer science program?

Annie: Yeah, I definitely think server mesh is so new and so far out there. I probably won’t be bringing that one up that much, especially multi-cluster specifically. But yeah, I think Kubernetes is really interesting. There’s so many use cases. At school, we still use VMs for anything. I’m a cloud computing focus, and we don’t use Kubernetes. That’s because Kubernetes is a relatively new thing. It was developed within the last 5 years. So yeah, I think I’ll definitely be bringing that one back to school, telling my classmates about it. And yeah, hopefully in the near future it’ll also get incorporated into some curriculum.

Bridget: [00:33:43] That’s so interesting to think about. Like, of course, everything in the academy, as it were, is going to both be presaging everything we’re doing in industry because it has innovative research, but then also lagging in that it’s not that everyone can redevelop a curriculum on a dime and suddenly have a curriculum that incorporates everything that Phil blogged or talked about at a conference last month. It’s not going to all make it into a curriculum for this fall. It just isn’t.

Annie: Yeah, so what’s the—

Bridget: how do we bridge the gap there? Like, do you get Phil to come in as a guest speaker for one of your classes and, you know, get him like telling some of the cloud computing students about what’s in their future? Like, how do we—

Annie: how do we bridge that? Yeah, I think, um, like you said, um, the industry changes so often, so it’s impossible to, you know, base each year’s curriculum off of whatever new thing is going on at that moment. But yeah, I do think that university does teach you the fundamentals to be able to go out there and learn about it. So yeah, I have enough experience as a CS student to be able to come here this summer and get an understanding of what Kubernetes is, what service mesh is. But yeah, I think for college students, especially in their later years, to just be aware of what else is out there in the industry, going out there and doing internships like this just really helps inform them on what’s in the industry. Then I think university is really for building those bottom-line skills that really help you learn whatever it is you need to learn later on.

Bridget: [00:35:21] Phil, I’m just curious what you think because I’m listening to Annie and thinking, you know what, we all hopefully are learning and should be learning and are trying new things all the time. Do you want to, as we wrap up here, give us your perspective on— you’ve been at Microsoft for a while, you obviously have not been focused on multi-cluster service mesh that entire time. Do you want to give us a little sense of journey so people can imagine themselves following a journey like yours, of like which texts you focused on over time?

Phil: Yeah, yeah. I mean, I was just going to touch on something. So like, I, I really commend Annie, right? Because, you know, when we got, uh, an intern, you know, I was chatting with my boss Lockie Evanson, if you know him. And so we were kind of going back and forth like, okay, where can we insert the intern to kind of help us do things, right? And we, we, we thought multi-cluster. We thought like, okay, do we want the intern to have a fun experience here at Microsoft or not, right?

Bridget: [00:36:25] Uh, we give the intern the hardest project.

Phil: Yeah, yeah, that’s the hardest thing that we’re dealing with right now. Uh, and, and so yeah, let’s, let’s give it to the intern coming in. But no, I mean, uh, Andy’s been awesome. And I think, you know, you can hear from what she’s been talking about today, and I think that’s kind of evidence of how really motivated she was to kind of get in and kind of understand what this whole ecosystem is about and what is this whole feature set about. So yeah, I hope, Annie, you kind of spread the word there when you go back to school. But yeah, I guess, you know, back to me. Yeah, as far as my journey, um, you know, I would say, you know, if you are a technologist at heart, just, just follow your passion, right? I’m using passion again. Uh, for those that don’t know, I mean, uh, I was considered a boomerang here at Microsoft, meaning I left and I came back. But I left and I went to work for Docker, and, um, and I was at Microsoft and was someone had mentioned containers. I didn’t really know. This was years ago. I was like, well, what is this whole thing about? Like, what’s wrong with my web apps, et cetera? And I went through like their tutorial, and then this light bulb came on. I was like, whoa, this is pretty slick, right? And then I kind of just devoted maybe the next portion of my life to understanding Docker and how this can solve a lot of issues for a lot of the enterprise customers at the time. That was a great experience to be at Docker and then to come back to Microsoft and bring that experience and bring what customers are— what the advantages are for them to be in a more cloud-native computing environment. Um, so yeah, I mean, I think if, if you’re passionate about something, you know, go for it. You know, let that kind of drive your avenue as far as where you want to take it. And, and then most importantly, know it’s okay for you to be passionate about something, but then again, for open source, see how you can enhance it. You know, I think that’s one of the things a lot of people don’t think of is, uh, they’ll come in, they’ll consume Kubernetes, they’ll consume consume all these other open source projects, but, uh, and they may have super intelligent ideas about things. And, and when I say contribute, that doesn’t mean, hey, you’re sitting down writing tons of lines of code, but just an issue out there and say, hey, how come it doesn’t do this or that? And you’d be surprised how we would still call that a contribution to the community. Oh yeah.

Bridget: [00:39:22] Oh, absolutely.

Phil: And then you know, people who have the time can pick something like that up and see if it’s worth it.

Bridget: I’m just— when I think of open source contributions, I think of how you, Phil, wrote some really long documentation on the Microsoft Docs site, and Annie came in and made a one-word PR that was literally just a clarification on the docs. And she thought, oh, should I put this in? This is one word. And I thought, you know what, this is an important clarification. Yes, you should absolutely put this poll request in and make this change. Like, that can sometimes be the most valuable thing of all, is removing a little bit of ambiguity or confusion.

Phil: Yeah, and I like how there’s been this, uh, pivot or this, this embrace of, uh, people now being recognized as contributors who are just doing content.

Bridget: You know, just having to be just making it possible for someone to use it. Yeah, no big deal.

Phil: [00:40:23] And not having to be, hey, I’m not this like, you know, 500-level Golang expert, or, you know, etc. But, uh, again, um, anytime that you can kind of soften the ride of someone who’s new to come in and maybe read a doc and get through it faster than someone who’s got through it prior to them is a good thing, right, for the whole community.

Bridget: Okay, so I’m gonna— I think since you’re mentioning docs, I’ll say, what things would you recommend people read, watch, look at, etc., to learn more about this? And then same question, more about you and your work. Well, we’ll start with Phil.

Phil: You know, I would say this. Just, you know, if you’re into multi-cluster service mesh, look and see what’s all out there, right? There’s other service meshes that are deploying their kind of way that they figure out multi-cluster. I would say just get familiar with the terms and the functionality in general. And then, you know, spin up every mesh that supports it and just, you know, comments on them, you know. That would be my number one, is again, just get familiar with the technology.

Bridget: [00:41:46] Annie, what would you recommend people look at so that they can learn as you’ve been learning?

Annie: I think, like Phil said, the value of being in an open source community is the fact that everything is out there for you to learn about. Definitely try all the other service meshes. Look at their documentation. Just to learn more about the ecosystem, what options people are offering, because that was definitely really valuable for me in my learning, where I didn’t just learn off the OSM docs. I also got to learn off of all the other docs of all the other service mesh offerings that are out there.

Bridget: Annie, if people want to catch up with you, keep up with you, follow your work, etc., find out what Rising Juniors actually do and what Rising is, that’s incoming, right? They changed all the words incoming since I was in school. Um, yeah, where can people keep up with you?

Annie: Um, yeah, I think my Twitter is in the, in the bottom. Um, so yeah, definitely follow me there. Um, and yeah, I got to start my Twitter, um, based off of the work I’ve done on OSM and multi-cluster this summer. So hopefully I’ll keep posting there too.

Bridget: [00:42:55] Very exciting. All right, and how about you, Phil? Where can people, uh, see more of your work or catch up with you.

Phil: Yeah, yeah, you might be able to find that on my Twitter handle. So @PhillipGibson, Phillip, 2 L’s, spelled the correct way. I’ll warn anyone going there, you’re gonna see a lot of, uh, how to smoke barbecue. So just, just be prepared, you know, there’ll be some techie stuff, but, uh, don’t be surprised when you see, uh, a couple of wrapped briskets, uh, on my Twitter handle as well.

Bridget: But surely that’s a feature, not a bug.

Phil: Yeah, yeah, that’s it.

Bridget: I want to thank both of you for joining us, and I want to tell our audience, head over to arresteddevops.com/multicluster-service-mesh for this episode’s show notes. Visit arresteddevops.com/itunes, leave us a review in the iTunes Store if you want to help other people find the podcast. Narrator, I don’t know how that works. We’re also on Spotify and iHeartRadio if you’re into those systems. Which are apparently things. And, uh, yeah, Annie, Philip, thank you so much for joining us today.

Phil: [00:44:07] Yeah, thank you for having us.

Annie: Thank you for having us.

Phil: Awesome.

Bridget: I’m Bridget at Bridget Krumhout. This is Arrested DevOps, and remember, there’s always DevOps in the banana stand.

BROUGHT TO YOU BY

Bridget chats with Phillip Gibson and Annie Wang about multicluster service mesh.

Bridget talks about multicluster service mesh, at a “201 level” after the earlier service mesh episode, with Phil Gibson, a PM at Microsoft focused on cloud-native security projects including Open Service Mesh, and Annie Wang, a PM intern on the team that summer, in the last week of the internship and studying computer science with a focus on cloud computing. The cold open is Bridget: “This all, I’m not gonna lie, sounds very complicated.”

What Multicluster Means

Phil says the most accepted description of a multicluster service mesh is being able to manage multiple Kubernetes services connected in east-west communication across clusters, and notes it means different things to different people. Annie explains that before multicluster, pods had to exit a cluster to reach services in another cluster, which is north-south traffic, and with east-west traffic, pods can reach the other cluster’s ingress without having to leave. Phil adds two pieces: a distributed control plane, so you can still configure the mesh if one system goes down, and an ingress for the service mesh, separate from the traditional ingress controller that exposes services to the world, where routes are populated and synced through the control plane so service A in cluster 1 knows the route to service B in cluster 2. On security, Phil says a cluster can itself be a security boundary with its own RBAC and policy controls.

Why Bother

Annie lists the benefits: a mesh limited to one Kubernetes cluster is limited to that cluster’s size, so multiple clusters let applications scale horizontally. Disaster recovery and failover mean you can deploy the same service to several clusters and divert traffic away from an unhealthy one, and you can get zero downtime during Kubernetes upgrades by updating one cluster at a time. Bridget asks about API deprecations across clusters on different versions. Phil says that’s the hard part, that Kubernetes became popular by offloading labor-intensive configuration, and that the team wants to keep the experience simplified and shield users from the heavy lifting, which is still being worked out. Annie says the philosophy is to align with OSM: enable complex use cases while keeping things simple. Bridget notes complexity is conserved, and Phil says to expect YAML, ConfigMaps and new CRDs.

An Intern on the Hardest Project

Annie says the internship was very overwhelming at first, with terms like Kubernetes, service mesh and even OSS new, since Kubernetes isn’t taught in school. Annie is not a service mesh user, and talking to users about what they liked and disliked was valuable. Phil says the team gave the intern multicluster, the hardest thing they’re dealing with. Annie says the most interesting part was learning how building features in open source differs from a regular product team, where you have a defined group of customers: you have to rally the community to figure out whether a problem is real, which is why Annie published a blog post to invite discussion.

Project Versus Product and the Spec

Phil says the benefit of open source is fast feedback, since a project will live or die on the vine. Phil looks for gaps and pain points that customers mention repeatedly, and says “you got to have thick skin in the open source game.” On the Service Mesh Interface spec, which has no code, Phil says people ask where to download it, but it’s an agreed-upon specification of how APIs interact, and getting consensus from a large community is slow, but listening to the community is good hygiene.

Who Needs It

Annie and Phil discovered that the need for multicluster is tied to maturity with Kubernetes. They assumed everyone wants it, from an enterprise mindset of a critical application, distributed service mesh, but early users had relatively simple applications. Phil says that if you’ve just refactored a VM application into a container and exposed it on port 80, multicluster is far away. Growth comes first by adding more apps to a cluster, and then the enterprise asks for regions and failover. Phil also explains mTLS with a shopping site: with TLS you trust the site, and with mutual TLS the site also knows who you are.

Learning and Contributing

Annie says Kubernetes is new enough that school still uses VMs, and Annie will bring Kubernetes back to classmates but not multicluster service mesh. Annie says university teaches the fundamentals to learn the rest. Phil’s career advice is to follow your passion, noting Phil left Microsoft for Docker after a container tutorial made a light bulb come on, and came back. Phil says contributing doesn’t have to mean writing lots of code, and filing an issue asking why something doesn’t work a certain way counts. Bridget points out Annie’s one-word documentation pull request, a clarification that was valuable, and Phil likes the embrace of content as contribution. Phil’s and Annie’s advice for learning is to spin up every service mesh that supports multicluster and read their docs. Phil is on Twitter as @PhillipGibson, with a lot of barbecue brisket.

art credit: “Spiral” by roland - CC0 1.0

This episode's guests