← All episodes
EPISODE 58February 26, 2016

Open Your Stack with JJ Asghar

withJJ Asghar· hosted byMatt Stratton
Read the transcript

Matty: [00:00:00] This sounds like Chicago politics to me. I feel right at home. It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m your co-host, Matt Stratton, @MattStratton on Twitter. Arrested DevOps is brought to you by TenthMagnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. TenthMagnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/tenthmagnitude. This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools such as Chef, Docker, and AWS so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a 14-day free trial at arresteddevops.com/datadog. So what’s this OpenStack thing all about? Joining me is JJ Asghar to help school me and you about what’s going on in the OpenStack world. JJ, can you tell us a bit about yourself and your background with OpenStack?

JJ: [00:01:34] Yeah, hi. My name is JJ Asghar, and I am the OpenStack Chef guy. I represent OpenStack in the Chef community, and in the OpenStack community, I represent Chef. I’m kind of like the bridge between. I own the whole work, including all the interaction— or all the integrations with the Knife plugin, the Test Kitchen plugin, and the Chef Provisioning Fog. And I’m slowly but surely getting— taking over the Fog integrations too on fog.io, which has been significantly more challenging than I thought it was going to be. I’ve been working with the guys.

Matty: How long have you been working with OpenStack?

JJ: I got involved back in the Diablo days. That was the 4th release, so basically 4 years, and OpenStack’s about 5 years old.

Matty: It seems like you’d be a good person to tell the audience a little bit about, maybe without going too much into it, kind of maybe the history of OpenStack, why OpenStack exists, why do we care?

JJ: [00:02:40] That’s actually a great question. A 50,000-foot view of OpenStack is, in short, a private cloud that you want to build. You can do anything ranging from database as a service to compute to imaging to object storage to even running containers and build Kubernetes clusters inside of it. It’s a fast-moving project that aims to build the cloud that you want.

Matty: And so that seems like that’s giving you a kind of very general, like, hey, good, I can build my own private cloud just like the big boys do within my own data center. What are some of the advantages of doing something like that versus using something that’s maybe more pre-canned, like something from VMware, or just saying the hell with it and I’ll just put it in EC2?

JJ: Oh yeah, sure. So there’s a challenge there where, first of all, if you care about owning your data in your data center, OpenStack is an open source, free way of creating your own cloud instead of having to pay a VMware tax. So there’s a certain problem at scale when you start talking about ESX and tens of thousands of machines. VMware gets prohibitively expensive around the 10,000 hypervisor mark. That’s where OpenStack learns to shine because you would rather probably invest your time in hiring someone or hiring a body who can give more than just a a software license cost and support contract to VMware, where you can actually have a body and someone who you can train and gain something and spend that time and effort in an engineer for OpenStack instead. On the flip side, OpenStack is also extremely good in dev and QA environments, where if you want to, instead of spending the money on EC2 instances, where running an EC2 instance for the whole year, let’s just say for fun, costs $1,200. Well, there are certain size companies out there that they might have dev and QA, and running a machine that does nothing for $1,200 a year, that could actually stop you from gaining the growth that you’re looking for. So OpenStack comes in there where you can take reclaimed hardware build out a dev and QA environment that uses the exact same APIs that EC2 uses and be able to do it locally and not have to spend that exorbitant amount of money inside of EC2, which helps because, well, you’re bringing everything inside in-house, and with reclaimed hardware, when it depreciates, you get to get even more life out of it, which is even better.

Matty: [00:05:49] So if I understand correctly, so OpenStack is kind of a consortium. Is that the right word? It’s— there’s a whole bunch of organizations organizations that have kind of chipped in resources and expertise to, um, kind of create this, right?

JJ: That was the way it originally started. Um, it originally started between Rackspace and NASA, where NASA created a front-end API to something called NOVA, which allowed you to spin up VMs using a Python API. And then also there was something called Swift, which was object storage, which is cloud files from Rackspace. And it’s grown over time. Different vendors have pushed in and everything. But just recently, OpenStack got its 506 status in the United States. So it’s actually a nonprofit business organization now. It’s the same tax code that the NFL, or the National Football League, or even churches have. When you work on OpenStack software and push back into the OpenStack community, you can actually go to your employer and say that you’re doing charitable work because it is considered a charity.

Matty: [00:07:13] Really, about being part of the Church of OpenStack, basically, is what that is. What I’m hearing here.

JJ: That’s exactly right.

Matty: Got it. Got it. Cool. You said that recently they’ve gotten that 503C?

JJ: 5036.

Matty: 5036. Okay.

JJ: Something like that.

Matty: Okay. They’ve gotten that. What else is kind of the state of the union with OpenStack today?

JJ: So we are right now in our Mataka release. I probably just butchered that, which will be coming out pretty soon here. OpenStack has cadences every 6 months where we try to get the features out as quickly as possible. This puts challenges down for operators, which I’ll hopefully talk about here in a little bit. But it is a fast-moving project. There’s about 19 different projects inside of OpenStack right now, and you can pick and choose from all those different 19 different products— or different projects, I’m sorry— to build out what you consider your ideal cloud to be.

Matty: [00:08:32] So you don’t necessarily have to use all the pieces, right? So like, what are some of those projects? You don’t have to name them all, but like, give me some examples of some maybe some of the projects that maybe I might want to use, maybe I won’t, or some of the ones that are obviously, you know, table stakes necessary?

JJ: Sure. So most of the projects are based off of— there’s a handful of what they call the core projects, and those range from— one’s called Keystone, which is the identity service. Another one is called Glance, which is the imaging service. I mentioned Nova earlier. There’s one called Neutron, which is up for debate if it’s either core or not. There are certain people who think it is, certain people who aren’t. Every software project has their detractors and supporters, of course. There’s also— there’s quite a few. And just for the core ones, I mean, everyone needs to be able to authenticate with your cloud. You need to have the right to get in. You need to be able to build a machine. Even the more fringe projects, I guess is a good way of putting it, would actually use Nova underneath them to do the work.

Matty: [00:09:51] Got it. What do you think— if I’m thinking about wanting to use OpenStack in my organization, what’s— you’ve been working with it for a while. I hate to say, like, what’s the learning curve, but I guess what are some of the challenges you’ve seen people have with adopting that into their organization or kind of the— maybe the state of the union of talent out there, expertise, things like that?

JJ: Sure. The biggest challenge of getting OpenStack into an organization isn’t actually OpenStack itself. 9 out of 10 times, it’s teaching a company to go to the cloud. I actually had a personal experience at one of the companies I’ve worked at before working at Chef where the idea of a cloud— a cloudy system where you— if a machine acts up, you put a bullet in its head and just rebuild it from scratch didn’t really work with that company. Even though they spent all this money and time and effort to build an OpenStack infrastructure, it was more challenging for them to understand that machines can be ephemeral. And that’s actually the longer pole in the tent with this whole project, or with this project, is trying to tell people, okay, first of all, OpenStack is not free VMware. As much as people want it to be, it’s not. People might sell it as free VMware. They might try to do that type of stuff, but it’s not. It’s— you can’t have someone who spent all their time and effort using vSphere and vSphere, all the different VMware tooling, and give them an OpenStack cloud and expect them to do the exact same thing. It’s a— you— a good analogy is if you take someone who spent their whole life working with Windows and all they know is Windows and then put them in front of a FreeBSD box, they’ll get around. They’ll be capable of doing their job, but it’s going to be— there’s going to be a long-term, a huge learning curve for them to understand that there’s not a sys— different things like that.

Matty: [00:12:01] Well, it’s a different way of thinking about things, right? You know, like we’ve said before, Windows is an API-based operating system. Nix is a file-based operating system. So again, if you’re thinking about it, VMware, you’re used to doing things in a certain way. It’s built for a certain thing, and I think that VMware’s has tooling and products that are intended towards doing the same things as OpenStack. But if you, like you said, if you’re a traditional VMware admin, you would have the same problem, I guess, just saying, okay, I’m going to move to using EC2 or Azure, right? Like, it’s, it’s, so it’s really more just the idea of utility computing service, or, you know, thinking about things as services, not as instances, and, you know, all that, that good stuff. It actually translates into just another question, which I imagine— I think I know the answer to, but fundamentally, I mean, obviously, the core of OpenStack, I’m guessing, runs on some type of variant of a Nix system, the core hypervisors and the core systems themselves, not Windows, but from a guest perspective, what’s the Microsoft footprint look like in the OpenStack world?

JJ: [00:13:11] Actually, it’s surprisingly high. The main hypervisors in OpenStack are mainly KVM and QEMU. They’re the same thing now, but you get the point. But there are Hyper-V ports to it, and also I know of many successful production boxes of Windows 2012 R2, and there’s actually one company called cloudbase.it that built their whole business off of getting specific Windows images for you for your OpenStack cloud.

Matty: That’s very cool. We’re living in this really beautiful heterogeneous world, which I’m loving. I know you told me something about something called OS Ops. Can you tell me a little more about that?

JJ: Sure. So there’s a— in the actual world of OpenStack, there are 3 main camps. There’s the developers, there’s the users, and there are also the operators. If you actually look at the OpenStack marketing data or marketing images and all that, Wait, just so I understand, you’re talking about the users of OpenStack.

Matty: [00:14:41] Now, when you say developers, do you mean developers of OpenStack or general software developers that are interacting with OpenStack, like in their world?

JJ: The answer is actually yes.

Matty: Okay.

JJ: That’s the problem, and that’s where all this stuff gets murky. Yes, so there’s 3 camps, and the devs… Normally are the people who commit code to OpenStack to make OpenStack better or to give a new feature, bug features, or whatever, bug features, features or bugs. Then there’s also users, or what I prefer the term consumers of OpenStack, that just have an API endpoint that they spin up a kitchen instance so they can run their kitchen tests or run a Redis server because they have a little Rails app or whatever. But then also, there’s a third group which are the operators. And those are the guys who— or people, I should say, who actually run OpenStack clouds. And unfortunately, they have been underrepresented in the actual OpenStack community for a really long time. The Foundation actually noticed this a while ago and started giving mid-cycle meetups that would happen usually in the US halfway through the cycle so operators could come together and talk about these things. The first one, or the second one, was in San Antonio, which was about a year and a half ago, and the next one is actually in Manchester, UK. Unfortunately, I can’t make it to that one, but it’s the first one that’s actually out of the US. The reason I’m bringing this up is this is basically the only time that operators can come together that aren’t— that aren’t the summit, which is extremely hard to get to because it’s usually in a very rather nice location. And also Operators have to pay their way to go to the summit. There’s this thing called an ATC, which is, I believe, Active Technical Contributor, that if you, within the last 12 to 18 months, maybe it’s less now, but it might be 18, you get a free ticket to the OpenStack Summit where all you have to pay for is your lodging and your travel. The actual summit is free. So that’s like a $600, $700 ticket. If you’ve given back to it. Now unfortunately, operators can’t get this ATC. Let’s admit it, operators, we write code. We write a bash script or we write a Python script or we write a Ruby script that does something for us. And unfortunately, because it’s not a Python script and doesn’t go into one of the official projects, you cannot get ATC right now. That’s where osops comes in. OSOps is our opportunity to come together as operators of OpenStack clouds and have a place for you to actually just share that tool, that bash script, that Python script, whatever, and give it back to the community and ideally in the near future get ATC. We’re not quite there yet. I’m still trying to get more people involved, but the foundation is extremely happy with the way OSOPS has taken off, and they are seriously considering letting us get to the point where we’re not even part of what is called the big tent, actually part of the core projects, because you need people to run these clouds, and this project, or this OSOPS project, is the tooling to make these— make running those clouds easier.

Matty: [00:18:31] Very cool. So if I was thinking about getting started with OpenStack, How do I do that?

JJ: That’s another great question. As someone who’s been spending a lot of time in the OpenStack community, you would imagine I would say something called DevStack. I will tell you very, very, very quickly to run away from DevStack. It has grown into a monstrosity, and it’s called DevStack. DevStack is a development OpenStack instance. It will not teach you what you need to know. What you need to first, before even starting with OpenStack, you need to figure out first what you want to build. You can’t just say, I want an OpenStack cloud. You’ll never get anywhere. You’ll never get off the ground because there’s just— you’ll get choice paralysis very quickly. Because you have so many different options to go through. And there are some projects out there, just like the OpenStack Chef project and OpenStack Model T. There’s also some Puppet and even some Ansible builds. I don’t know about Salt. I think there’s some Salt ones that you can build small, small clouds so you can learn on what you need, what you think is useful, and what you want to build. But it’s just like any software project. You can’t just go in there and say, I need a new application to do blah, and just be like, throw it against the wall and see what happens. No, there’s a lot of engineering choices you need to make. My advice to anyone starting with OpenStack is first figure out what you want to build and then move forward.

Matty: [00:20:17] What about— there’s some commercial implementations of OpenStack as well, right, if I’m remembering correctly? Or just ways that people kind of get jump-started with it. What are some of those that you’re aware of?

JJ: Yes, so there’s Mirantis out there who’s made an extremely successful business on building OpenStack clouds. There’s also what was called Blue Box, but now is IBM, who builds a business off of building OpenStack also, and those are great. Vendors make— can build you an OpenStack cloud for what they think you would want, which is useful, don’t get me wrong. But the challenge there is that you’re off-handing or handing off the responsibility of understanding the intricacies of the cloud to a third-party vendor. When you go down that path, there’s no difference in going into going to Mirantis or Bluebox than asking another MSP to come in and ask you to build a cloud for you. It’s— I come from the camp that if you’re going to build a cloud and want something that you have full control of, you should understand how the whole thing works. But maybe it’s cost prohibitive. To spend the time and engineering time on there. Yes, you can absolutely find a vendor to do that stuff, and there are multiple out there.

Matty: [00:21:51] Cool. You talked also about the ability to use some stuff like Chef or Puppet or Ansible to get your initial stuff started. How does the OpenStack world work with some of our other hot technologies that are out there if we’re thinking about things like Docker or Cloud Foundry or things like that? Where are the Venn diagram lineups of this kind of stuff?

JJ: The short of it is because OpenStack is trying to gain as much popularity as quickly as possible, there are projects for all of those. There’s actually apps.openstack.org, which there’s a one-button push now for you to pull down an image a glance image to your OpenStack cloud and be able to build out a Kubernetes cluster for you, which is pretty slick if you have to admit that. Granted, you have to have some of the more newer technologies on your cloud, but a one-button push to get a Kubernetes cluster is pretty slick. There’s also Docker support with a project called Magnum, which actually spins up using Heat, which is one of the projects for orchestration, 3 or 4— or 3 CoreOS machines, puts Fleet on them, or makes them into a cluster with etcd, and then turns around, changes your API endpoint to that CoreOS cluster, and allows you to use Docker like you would any other any other way you would use Docker at a production level for containers. There’s a link that I can give to the show notes of this where you can actually see all the 19 different projects for OpenStack where you can pick and choose what you’re looking to do.

Matty: [00:23:56] Great. You talked about OpenStack as an organization, and I’ve heard this rumor that you’re running for something called the Board of Elections. Can you tell us what that— first of all, what is that board, and then maybe tell people how they can vote for you and why they should?

JJ: That’s funny.

Matty: Get on your bully pulpit.

JJ: Yes, yes. Sorry, my notes were wrong. It’s actually Board of Directors. My copy-paste did not work. But yes, I’m actually running for the Board of Directors for OpenStack. My whole platform of me running for the board is that if you look at the people who’ve been on the board over the last couple years, they’ve been removed from OpenStack day-to-day. The things that they’ve been trying to push for are great and are a great idea for if we were in what I would call the later phases of OpenStack. I use an antidote here, or an anecdote here. I can’t say the word, a story here kind of explaining where I think the OpenStack story is right now. And there’s a— in software development, there’s the idea of pioneers, town planners, and city builders. And the idea is that in a pioneering phase, you can go do whatever you want. It’s all just wilderness. It’s the Wild West. You can go to Node Or you can change around your code to Rails the next day, or you can decide to do everything in Django. It does not matter. Then eventually you get into what is the town planning phase where you’re like, you know what, we need a bank, or we need, you know, some roads, or we need a general store. We need a place that we can kind of congregate and start getting a little bit more structure to what we’re going for. And then finally there’s the city planning phase, or city builder phase, where that’s where you spend all the time to build the skyscrapers and the highways between cities and just grow everything through it. Now if you read a lot of the CIO magazines out there, or you read a lot of the blog posts from even the Foundation, it makes OpenStack sound like it’s in city building phase. Just throw money at us, we’re gonna build everything you’re expecting. Operators have a very different view of this, and I’m one of these people where if you actually try to build an OpenStack cloud, it’s still quite challenging. Even though with— as part of the OpenStack Chef project, I try to make it as cheffy and delightful as possible, it’s still extremely challenging. And I truly believe we are just getting out of the pioneering phase and just getting into the town planning phase. A good milestone, if you will, is actually apps.openstack.org. It is a marketplace, or if you will, a general store for our town. It’s a place that we can go and all agree that yes, this is a way to build a glance image for Kubernetes cluster. Yes, this is a place that I can get a Windows 2012 R2 image from cloud-based IT and actually work on my cloud. This is extremely— it’s a milestone, it’s good, but we’re not at the point where we need skyscrapers. Yes, I admit it, we need the vendors to give us money to continue moving forward, but we need to be more realistic. There are times when features get pushed into OpenStack so quickly that you can tell that people just pushed to get them in the next release when people or operators haven’t even adopted these ideas. I mean, we’re still arguing over how to do networking on just a layer 2 and layer 3 networking inside our clouds. When we’re talking— when we have certain vendors out there giving us money to build edge routers inside of the SDNs. Now, if we can’t even figure out how to get our VMs to talk to one another in a consistent way, like with either OVS or Linux Bridge or whatever, why are we talking about edge routing on SDNs? I mean, that’s a massive disconnect. It’s great that we have the money coming in, But shouldn’t we be spending more time on how to get something consistent and supported and we all understand how it works instead of, you know, all the bells and whistles out there? So that’s kind of my whole spiel about the board of directors is that I want to go— I want to be elected to the board so I can ask these questions and I can just ask why we’ve decided all Spending all this time and effort and resources on doing these things when we have people’s careers, including myself. My career has been based around OpenStack. My career is based on this, and when I start seeing how the sausage is made, I wonder why are we doing the things we’re doing? I want us to tell everybody why this is happening.

Matty: [00:29:23] That sounds legit to me. So, I probably don’t get a vote, but you would have mine if I could.

JJ: I appreciate that. I really do. And there are—

Matty: you will—

JJ: if you have the ability to vote, between January 11th and 16th is the voting. And you’ll get an email with a link in there to allow you to vote for the speakers. Those spots.

Matty: And we’ll put a link in the show notes to where you can vote for JJ if you have the ability to do that. Who has the ability to vote for this, by the way?

JJ: So I’ve heard different stories about this. And it seems to change from what I remember what it was. I mean, I had the rights last year, but the year before I didn’t. So I don’t really know.

Matty: This sounds like Chicago politics to me. I feel right at home. Tell us a little bit about OpenStack Model T. You sent me a link to that. That looks pretty cool.

JJ: [00:30:32] Yeah. This is something I’m really proud of. First of all, if any of your listeners read SysAdvent, I wrote a little narrative about it. For the first day of SysAdvent this year. OpenStack Model T is an opinionated build of OpenStack with one Chef cookbook. My whole plan is I went through the install guide for OpenStack. If you’ve ever had a chance to read the install guide for OpenStack, you’ll know that it’s quite long and quite time-consuming. Well, I’ve automated it with Chef. I’ve made some personal choices, for instance, or opinionated choices, I should say. There is RabbitMQ as the messaging queue. The base OS is Ubuntu. But that’s more or less all of it. But the point being is that this is a way for you to be able to eventually— I’m creating a reference architecture out of this. So you can build with that reclaimed hardware that I was mentioning before with the smaller customers or the smaller companies. You can run with one recipe and expand out your OpenStack cluster by running one recipe for a controller. Whenever you need to add a new compute node, you just run one recipe on that compute node and it adds it to the OpenStack cluster and all of a sudden you have a horizontally scalable cloud. It’s eventually— or I’m hoping for this time— or I’m hoping for ChefConf 2016 to have a talk there about building out one of these reference architectures with using one cookbook called Pixie Dust. To bootstrap an OpenStack cloud with a bunch of hardware, and then actually on another machine, use Test Kitchen to do testing on that OpenStack cloud, all within the space of the 40 minutes that I have on bare metal.

Matty: [00:32:44] That’s very cool.

JJ: Yep.

Matty: And I guess, so speaking of, you know, what’s, what’s going on just with the, the, if people are really good with Chef and they know some OpenStack or vice versa, how could they help out with telling this story?

JJ: Sure. There is the OpenStack Model T, but once again, that’s my opinionated build of OpenStack. There’s another actual project inside of OpenStack called the OpenStack Chef Project. We have cookbooks for every major one of the all the major different projects to be able to build out your OpenStack cloud with Chef cookbooks. We are a very small but dedicated agile team, and we desperately need more people to help us. We’re doing the best we can with the resources we have, but if you have any Chef knowledge and have any interest in OpenStack, we would love for you to come over to our side and help us build out these to make them great. We’re in the middle of a refactor right now. We acknowledge that over the last 2 cycles, we kind of added all the features and have everything including the kitchen sink in there. And that’s gotten to the point where it’s— we think it’s actually scared some people off. So this is where we decided to pare everything down so it becomes a little bit more straightforward. We also have a Chef repo, which is called the OpenStack Chef repo. I think you can actually just Google that directly. And if you have like a 16-gig MacBook Pro, you can build out with Chef provisioning and Vagrant a multi-node OpenStack cluster on your laptop by just running a handful of commands. And then we have some documentation on taking those same commands and the same build and to build it to an all-in-one bare metal build. So you can actually build— if you have like a— for me, I had a 96-gig quad-core Xeon that got donated to me. And I was able to take those cookbooks, run the all-in-one build on there, and even using— learning how to— using Hosted Chef, and build an OpenStack cloud production ready, which is still running to this day, in about 45 minutes. That’s something worth showing.

Matty: [00:35:21] Great. Well, this has been awesome. I’ve definitely learned a bunch. Is there anything you would want to finally share with our listeners about OpenStack, or even for people who maybe have been using it for a while, maybe some things they aren’t thinking about or different ideas? Parting words?

JJ: The short of it is that the operators and the guys who are the people who are running your clouds, we’re trying to come together and trying to get going. If you have a tool for OpenStack or OS Ops, we have a tools contrib directory where if you just write a bash script And when we finally get that, um, ATC ability, all you got to do is just give it to us and you’ll get ATC. It’s— we’ve tried to create this to have the lowest barrier entry possible for operators.

Matty: Awesome. Well, so, uh, before we move into checkouts, a couple little bits of, uh, Administrivia. Uh, if you have an upcoming conference you would like to see promoted on Arrested DevOps, please visit arresteddevops.com/conf. That’s C-O-N-F, like conference, and fill out the handy-dandy form there, and we’ll get back to you and figure something out. And speaking about our website at arresteddevops.com, we are all open source now, and it’s pretty rad. So if you want to help out, we’re accepting pull requests at github.com/arresteddevops/ado-yugo. So if you notice errata in our show notes or anything like that, feel free to send us a PR. Or if you want to help out with anything else with the website, it’s all built on a cool technology called Hugo. You can check it out at gohugo.io. And feel free to take a whack at any outstanding issues we have listed on our repo. But now let’s do some checkouts. So JJ, what do you have for our listeners to check out?

JJ: [00:37:19] I got a couple things here. The first one is, I’m very impressed with This War Is Mine. It’s a Steam or iPad or iPhone game where you— it’s in essence you play as survivors for a war-torn city and you have to do resource management inside of it. And it’s a really interesting kind of take on what you would have to do in that environment. It’s a lot of fun, and I just found out recently too, it turns out they’re building— making a board game out of it, which is pretty neat too. So in theory, you could be sitting around your kitchen table with your family and you’d see who would go off and steal the food first. The second one I have is Labyrinth Black Ale. It’s probably my favorite beer right now. It’s a quadruple from Labyrinth that is about 15%, and it comes only in boomers, the 22-ouncers. It is so nice, especially with how cold it is right now. It will keep you nice and warm. My third one is, if you have the voting rights for OpenStack, please vote for me.

Matty: [00:38:37] Absolutely. I just have one. It’s a game called Asphalt 8. It’s available on iOS, I think also Android. I have been playing it because I just recently got myself a new Apple TV that supports apps. And so that was one of the games I downloaded. It’s a racing game. It’s pretty fun. It’s definitely one of those where they try to, you know, get you with micropayments and all that stuff. But even if you don’t want to spend money, it’s still a pretty low barrier to entry racing game. I’ve been having a lot of fun with it. And pretending to race a Tesla. That’s pretty rad. So I recommend that. It’s at gameloft.com/asphalt8, the number 8. So, and don’t forget that we have a newsletter. arresteddevops.com/bananastand is how you can sign up. It’s the best way to know about our upcoming podcast episodes and cool links and stuff that I find about DevOps. And thanks to our sponsors. Be sure to visit them at arresteddevops.com/10thmagnitude and arresteddevops.com/datadog. Thanks to JJ for joining me tonight. And listeners, if you enjoy Arrested DevOps, or if you don’t actually, one way or another, we would appreciate it if you’d visit arresteddevops.com/itunes and leave us a review in the iTunes Store. And we would love to know what you thought of this episode, so you can leave us comments at arresteddevops.com/openstack. You can also tweet at us @ArrestedDevOps. And we’re happy to get your input, ideas, or feedback at shows@arresteddevops.com if you dig that email thing. Please let us know any ideas you might have for future episodes. I’m Matt, @MattStratton. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.

BROUGHT TO YOU BY

JJ Asghar joins us to help school Matt about what is going on in the OpenStack world

Matty asks JJ Asghar to explain what’s going on in the OpenStack world. JJ is the self-described OpenStack Chef guy, representing OpenStack in the Chef community and Chef in the OpenStack community, and owns the Knife plugin, the Test Kitchen plugin and the Chef Provisioning Fog integration. JJ got involved back around the Diablo release, the fourth one, so about four years ago, and OpenStack is about five years old. Matty opens by saying the story sounds like Chicago politics, which makes Matty feel right at home, and comes back to it later.

What OpenStack Is

JJ’s 50,000-foot view is that OpenStack is a private cloud you build yourself, from database as a service and compute to imaging, object storage and even container clusters. It started between Rackspace and NASA, with NASA’s front end to Nova for spinning up VMs and Rackspace’s Swift for object storage. It has since grown to about 19 projects on a six-month release cadence, which JJ says is a challenge for operators, and you can pick and choose which ones you use. The core ones are Keystone for identity, Glance for images and Nova, with Neutron the subject of debate about whether it counts as core. JJ says OpenStack recently got nonprofit status in the US under the same tax code as the NFL and churches, so contributing can be described to your employer as charitable work, which Matty calls joining the Church of OpenStack.

Why Not VMware or EC2

JJ says if you care about owning your data in your data center, OpenStack is an open source way to build a cloud instead of paying a VMware tax. At scale, VMware “gets prohibitively expensive around the 10,000 hypervisor mark,” and JJ would rather spend that on hiring an engineer you can train than on a license and support contract. It is also good for dev and QA, where a machine sitting idle on EC2 for a year costs $1,200, and reclaimed hardware can run the same APIs as EC2 locally and get more life out of its depreciation.

The Hard Part Is the Mindset

According to JJ, “9 out of 10 times, it’s teaching a company to go to the cloud.” At a previous employer, they spent money and effort on an OpenStack infrastructure, but the idea that a misbehaving machine gets rebuilt from scratch never took hold, since the company couldn’t accept ephemeral machines. JJ’s warning is “OpenStack is not free VMware.” Handing someone who has spent their career on vSphere an OpenStack cloud is like putting a lifelong Windows user in front of a FreeBSD box: they’ll get by, but with a long learning curve. Matty compares it to Windows being API-based and Unix file-based.

The Microsoft footprint is surprisingly high, JJ says. The main hypervisors are KVM and QEMU, there are Hyper-V ports, JJ knows of production Windows 2012 R2 boxes, and one company, cloudbase.it, built its business on Windows images for OpenStack clouds.

Operators and OSOps

JJ says OpenStack has three camps: developers who commit code to OpenStack, users JJ prefers to call consumers who just want an API endpoint for a Test Kitchen instance or a Redis server, and operators who run the clouds. Operators have been underrepresented for a long time. The Foundation responded with mid-cycle operator meetups, the next in Manchester, UK, the first outside the US. Operators also usually can’t get an ATC, the Active Technical Contributor status that earns a free summit ticket worth $600 or $700 for anyone who has contributed in the last 12 to 18 months, because their bash, Python and Ruby scripts don’t go into official projects. OSOps is a place for operators to share those tools, with the goal of getting them ATC someday. JJ says the Foundation is happy with how it has taken off, and is considering making it a core project, since you need people to run these clouds.

Getting Started

JJ says to run away from DevStack, which “has grown into a monstrosity” and won’t teach you what you need. First figure out what you want to build, because saying you want an OpenStack cloud gets you choice paralysis. Projects like OpenStack Chef and OpenStack Model T, along with some Puppet and Ansible builds, let you build small clouds to learn. Commercial options include Mirantis and Blue Box, now IBM, which are useful, but you are handing off understanding of the cloud to a third party, no different from asking an MSP. JJ comes from the camp that if you want full control, you should understand how the whole thing works, though the engineering time may be cost prohibitive, JJ admits.

OpenStack fits with other technologies: apps.openstack.org has a one-button push to pull a Glance image and build a Kubernetes cluster, and a project called Magnum uses Heat to spin up CoreOS machines with Fleet and etcd and points your API endpoint at the cluster so you can use Docker as usual.

Running for the Board

JJ is running for the OpenStack Board of Directors, and Matty says JJ would get a vote from Matty if Matty had one. JJ’s argument is that board members have been removed from day-to-day OpenStack. JJ uses a story about pioneers, town planners and city builders. CIO magazines and Foundation blog posts make OpenStack sound like it is in the city-building phase, but JJ believes “we are just getting out of the pioneering phase and just getting into the town planning phase.” JJ sees apps.openstack.org as the general store, a milestone, but says features get pushed into releases before operators have adopted the previous ones. JJ’s example is that people are still arguing about basic layer 2 and layer 3 networking while vendors fund edge routers inside SDNs. JJ wants a seat to ask why, and to see “how the sausage is made.” Who has the right to vote isn’t clear to JJ either. JJ had it last year but not the year before, which is how Matty got to Chicago politics.

OpenStack Model T and the Chef Project

JJ is proud of OpenStack Model T, an opinionated build of OpenStack in one Chef cookbook, which automates the long OpenStack install guide, with RabbitMQ as the queue and Ubuntu as the base OS. Run one recipe on a controller and one on each compute node you add, and you have a horizontally scalable cloud on reclaimed hardware. JJ hopes to give a ChefConf 2016 talk on a reference architecture bootstrapped with a cookbook called Pixie Dust and tested with Test Kitchen on bare metal.

The OpenStack Chef project has cookbooks for all the major projects, and a small team that needs more help. It is in the middle of a refactor because the last two cycles added everything including the kitchen sink, which JJ thinks scared people off. With a 16-gig MacBook Pro you can build a multi-node OpenStack cluster with Chef Provisioning and Vagrant, and JJ used the same approach on a donated 96-gig quad-core Xeon to build a production-ready all-in-one cloud in about 45 minutes, which is still running. JJ’s parting message is that operators are trying to come together, and if you have a tool, drop it into OSOps and, once ATC is possible, you’ll get it.

Check Outs

JJ

Matt

This episode's guests