Liam: [00:00:00] We’re in this age of just kind of madness, really, quite frankly.
Matty: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton, and co-hosting with me is… Trevor Hess.
Trevor: Yay.
Matty: So yeah, it’s been a couple of years or almost 2 years since we last visited the topic of DevOps and Microsoft chops. Say that right.
Trevor: The last time we did that, I think I was in Paris and you have to drink now.
Matty: That’s true. That’s true. If you’re playing the Arrested DevOps drinking game, if Trevor talks about being in another country, take a drink. So anyway, we’re talking about Microsoft and DevOps tonight, uh, with the largest panel in Arrested DevOps history, uh, citation needed. At least on a recorded show and not a non-DevOps Days show. The show notes for this episode can be found at arresteddevops.com/microsoft2. But first, a word from our sponsors.
Glenn: [00:01:10] Arrested DevOps is brought to you by Tenth Magnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. Tenth Magnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/10thmagnitude.
Matty: This episode is sponsored by VictorOps. Built for modern incident management, VictorOps provides a unified platform for real-time alerting, collaboration, and documentation. Driven by your IT and DevOps system data, VictorOps helps you respond to incidents more effectively so you can minimize downtime and make being on call suck less. Visit arresteddevops.com/victorops to schedule a demo or start your trial. Mention that you heard about VictorOps here on Arrested DevOps, and you’ll be eligible for some sweet discounts too. This episode is also brought to you by Hired. Hired is a platform for top developer jobs, and they love DevOps people. Developers get an average of 5 to 10 offers on the platform, all with just one application. You get job offers and salary or equity upfront before you interview, so you don’t have to waste your time interviewing for jobs you might not want. And they work with over 4,000 companies from startups to large public companies all over the place. ADO listeners get double the $2,000 bonus just for signing up at arresteddevops.com/hired. As I mentioned before, the last time that we had a show dedicated to the trials and tribulations of doing the DevOps in Microsoft environments was back in February of 2015 with Jeffrey Snover and Jessica DeVita. You can check out that episode at arresteddevops.com/microsoft-devops. Apparently Trevor was in Paris. Whoop-de-doo. But this time we have a much larger panel of DevOps practitioners who live their lives doing this Microsoft stuff. So let’s go around the horn and introduce the panel. Go ahead and introduce yourselves.
Liam: [00:03:06] Hi, I’m Liam Bennett. I work for an organization called Clarinet, who are a managed hosting provider. I spend kind of all day, every day with Windows workloads on AWS, Azure, GCP. My claim to fame that you might also know me as, I previously worked at OpenTable as the infrastructure engineer there and released all of the Puppet modules that live in the community.
Brandon: This is Brandon. I’m a systems engineer at Columbia Sportswear, and I spend most of my day trying to figure out how to do DevOps on Windows. So typical systems engineer type stuff, managing the infrastructure and figuring out how to automate as much as you can with PowerShell and Chef and various other tools.
Reuben: Yeah, my name’s Reuben. I am the DevOps practice lead at a company called Freedom in New Zealand. So we’re an online platform system for payments, travel, and expense. And I don’t do as much coding or infra as what I used to, so I’m more about the coaching and and getting people inspired.
Glenn: [00:04:14] Hi, I’m Glenn Sarti. I’m a senior software developer at Puppet specializing in Windows. Been doing this about a year at Puppet, and previously I’m doing desktop infrastructure automation in Australia.
Chris: I’m Chris Hunt. I’m a Windows platform engineer at Ticketmaster, and big claim to fame at the moment is a pretty sizable DSC implementation. That is, we’re approaching 1,000 nodes on a pull server.
Matty: So this is pretty cool. We’ve got a pretty wide range of technical implementations, different roles of what people are doing. So everyone alluded a little bit to the environments where they work, but I kind of want to get a feel for what’s working well. What are you In the environment you’re at with some of the stuff you’re doing with regard to DevOps, where are you kicking ass? And there’s a follow-up question to that.
Chris: [00:05:15] I’ll say DSC is working well for us with the caveat of it’s challenging.
Brandon: I think for us what’s working well is figuring out how to use automation pipelines to get our work done. And quite a bit of work has been done in ChatOps trying to expose tasks to other groups who may not necessarily have the skills or access to do various things. That’s really working well for us and gaining quite a bit of excitement.
Glenn: PowerShell has been an absolute saver for us. So trying to bring Windows into the Puppet world. Just going from strength to strength with PowerShell.
Reuben: PowerShell and Chocolatey, how we distribute stuff all over the platform, making from users’ desktops to test environments all the same, abstracting that away. That’s been really well.
Matty: It’s interesting to think about those 2 components being so critical to be able to move with any kind of rapidity or velocity is a robust shell and package management. We didn’t have that in the Microsoft world before, and it made a lot of this stuff— a lot of what you would think of as automation was just macros as a way that I would think about it. When I think about how I would write old VB scripts and stuff like that, at the end of the day, it was the equivalent of a macro, which was like, do this thing, do this thing, but you want to actually do any thinking about that thing, it becomes a lot more challenging and you have to introspect so much. You have to make so many more assumptions. I’m super jealous of all of you ’cause I’ve spent most of my working career before I went to go hand wave at a vendor as a Windows system administrator. And it’s all so much more rad now. It’s like, I’m like, damn, man. You know, we thought, you know, and I, I, I’m pretty sure all of you have been through that too, so none of that’s like, you know, but I’m just like, yeah, I’m super jealous that like all this cool shit’s there and I’m like, I don’t get to use it except to help my customers.
Brandon: [00:07:36] We don’t really like to talk about the dark days, uh, in BBScript. I’d like to forget about it.
Matty: Resume next.
Chris: Well, I just, I mean, we’ve gone like 10 years and just recently, I guess the last year or so, have a way to distribute, package and distribute modules. So everything before that was like hacky scripts to install. And so now we actually can sort of version things and share code. That alone has also been a big help.
Matty: So I mean, thinking about that, of where the strides have been happening, I wanna think about from the, I guess I would call it, it’s not even necessarily a community of practice, and I don’t even necessarily call it a community, but it’s more of a, I can’t think of the right word, common work folks. Listeners, I’m on 2 hours sleep. Anyway, but when we say, like, okay, one thing we have in common, we’re kind of a guild of Microsoft system engineers of some kind, And as a practice, how we’ve evolved over the last decade or so, there’s things that we’ve changed, but there’s also places where Microsoft the company has changed. And so we said, okay, well, 2 of the things that were super important were PowerShell and then Chocolatey, right? Like having some type of good package management, having a robust shell, but You know, and I think, Brandon, you alluded to, again, like being able to share code and like thinking about how that mentality has maybe changed. Like, that was something I think was unheard of 10 years ago, unheard of 5 years ago, other than code. Maybe you might throw something on CodePen, but oh geez, what was that? There was like a website where people like shared, like system administration VB scripts. I bet you if I look through my old bookmarks— was it Scripts? Yeah, maybe that was—
Chris: [00:09:46] I’m gonna find it.
Matty: Yeah, and it was like you go and you’d like search and be— someone wrote a VB script for like adding users to like an Active Directory group or whatever and all this other stuff. And, and it wasn’t really open source. I mean, it had to be because it’s VB script. It’s source open, but there was no way to help make it better, right? And are you seeing with yourselves, with your teams, with people in your own communities, how are you seeing kind of that adoption of this model of open sourcing the things that you’re doing, given that we come from a very closed closed-source platform up until recently.
Liam: All right, so I spend— most of the stuff that we do is based on open-source tools, um, and we spend a lot of time in that ecosystem, um, and on the Microsoft repos that they’ve released. And we’re seeing, we’re seeing a lot more people, um, contribute to those things. But what I think is important is that for those organizations, it required Microsoft to make that first move. It was essentially 2014 when Ballmer stepped away from Microsoft. It was entirely— it’s a massive culture shift for them, and they just got permission to do all of these things that they’ve wanted to do for a while. But it required, for the rest of the Microsoft community as a whole, it required Microsoft to make that first move to give them permission to say, okay, it’s okay to do open source now because we’re doing it first.
Chris: [00:11:41] Also, Microsoft had sort of a history of building their own solution. So, you could invest in something open source, And then they just come along and create a commercial product and kind of kill you. But now it seems like you can start an open-source product, and they’ll just contribute to it instead of clobbering you and building their own.
Glenn: That said, I mean, I’ve worked for a lot of banks and military and government organizations. Getting people used to contributing to code outside of their firewall is still a big challenge. I have no idea how to get around it, but Hopefully with Microsoft being more open and more forward about their open sourcing, that they can at least show that it’s possible to do those kind of things.
Brandon: Yeah, that’s something kind of interesting with a lot of, probably I would call traditional enterprises, is one, it’s using open source is probably less of a barrier, but actually contributing back to open source is a much harder thing to get people on board with.
Matty: Yeah, I was just gonna say, you know, the comfort level with using open source software is expanding, but, and to be honest, it’s, I kind of see that it’s not really specifically always around the Microsoft stuff. It’s like behind that firewall, wherever it is, there’s kind of this resistance. And it’s a lot of times it seems to be having to do with like the company’s lawyers, like, starting to understand digital a little better, right? And a lot of times there’s, I think, this concern that either it’s going to set some precedent that’s going to release other IP that’s really important to them, that like, oh, because we open-sourced this chef cookbook, well, that’s a precedent. So now somebody else can think that like this movie that we made is in the public domain too. And I know that sounds like crazy, but that’s what lawyers are for, is to think of the the crazy things that someone might wanna do, right? Or also getting an understanding of what’s plumbing and what’s different, right? You know, and I remember when I was at apartments, we did, you know, we did, when we did our big migration from VMware to Hyper-V and Microsoft was like, cool, let’s do a whole big like article about this because nobody’s been dumb enough to do that yet. ’Cause at the time that was the thing. Not saying Hyper-V is dumb, but this was years and years ago. It was like, they’re like, oh wow, someone’s really doing this, awesome. And it was a big thing with like our PR people ’cause they’re like, whoa, you know, rent.com is gonna know what we do for virtualization. I’m like, they could give a shit. Like, I’m glad you think what I do is so important, but it really doesn’t matter, right? This is all just block and tackle of that stuff. I think, and I guess because there’s so much stuff that’s tied to line of business application stuff, that’s really usually more tied on the Microsoft side, is I think why people tend to feel a little more resistant to the open nature of it. But I also should caveat to say that I’m the first person to make sure to say that I don’t think that the only place that people are doing Microsoft stuff is line of business stuff. I mean, it’s running obviously a lot of cool, sexy frontend shit too. But there’s such a preponderance of that that I think it’s harder for people to see from a legal perspective within a company, like, why would they care about sharing that kind of thing? I don’t know. And are you seeing, like, also, is that stuff seems like it’s built a little more fit for purpose, right? You know, sometimes, like, why would it be common?
Brandon: [00:15:25] I don’t know.
Matty: I don’t know where I was going with that.
Chris: I was gonna say, it seems like the open source licensing is opened up more as well. More people are putting out MIT licenses and permissive licenses. I know years ago we looked at it and, you know, the license would say if you put this in your software, you have to open source your software. So, you know, you can’t use an open source library or you have to open source your whole line of business. That obviously doesn’t work. But now that, like you said, businesses are starting to separate sort of that plumbing code from their business code. They don’t mind being open with that, letting anybody do what they want with it commercially.
Matty: I’m curious with anyone who’s working in a shop that’s very .NET, Visual Studio, TFS Extreme, whatever they’re calling it today, but you know what I mean? Like that developer the developer stack, the Microsoft developer stack kind of piece. One of the things that I found to be true that was actually super challenging in a pre-DevOps world, because we didn’t know to call it DevOps at the time, but if we’re going to try to do that in a shop like that, was that so much stuff was built to make things really easy for the developer to package their stuff up and do all this stuff, and it was super hard to then take that stuff that came from MSBuild or WebBuild or whatever, and actually operationalize it. And I’m wondering what’s been, like I said, a lot of folks saying, hey, we’re using a lot of open-source tools, we’re using this and the other. I mean, is anyone using, and I know at Ticketmaster using DSC, but is anyone using the Microsoft stack for deployment, like Visual Studio Server or whatever? TFS is called now, and MSBuild, and all basically the prescribed Microsoft way. Is anyone doing that on the call? Not in general, I know people are doing it in general.
Reuben: [00:17:31] We’re doing some components of that, but we’re not using— we’re using TeamCity as opposed to VSTS. I have a big problem with VSTS and the fact that we talk about running our builds differently on a server than what we do locally. The whole Visual Studio experience has always been about the developer and everything just needing one pane of glass, one application, which has some really good outcomes for a developer. But that right-click publish, oh my gosh, it’s such a nightmare. People then just, their definition of done, when they’re finished is just a package. In the day of a monolith, that might be really okay because all my package and all my dependencies can be articulated at compile time. But when you’re jumping into the world of distributed computing, that can make the ops people very mad.
Matty: [00:18:36] Again, I could be showing my— it’s been a while since I’ve had to deal with it, but it was again, like you said, it was built from the perspective— a lot of it’s built from the perspective of I’m the developer, I’ve got the codebase, and I’m just spraying it somewhere. So much stuff was built on transforms. This is how I know this has happened, so I’m going to write all my configs and it’s going to have basically, at the end of the day, sed. That’s going to go and do some replacing for whatever these strings should be. It meant that you had to really understand Visual Studio to deploy an application. Then with us, we sat there and we’re like, we’re not going to buy this at the time, $8,000 to $10,000 license for every sysop who needs to deploy. So it ended up being everything ended up having to be very manual. We couldn’t even use Microsoft’s tools because it didn’t scale, right? Because they’re like, and I know they’ve gotten better about that in terms of at least making it a little easier to get at this stuff. I think that’s gotten smarter around that. That being said, so thinking about that mentality, if you will, that’s kind of in that more traditional .NET workflow, what have been some of the things you’ve gone through culturally within your organizations, maybe, to get people to kind of care a little bit more about operationalizing their code?
Liam: [00:19:59] So I think I can talk to some of this. I was a release engineer in a previous life, and I think that the cultural aspect is is actually, you win most of these places over just by making it easier. For the most part, you can kind of pull away from some of the traditional Microsoft tooling if you can make the solution that you’re proposing easier. A lot of the pain points that I’ve seen, particularly around VSTS, There’s good reasons why people are pulling away from some of the components of that. It’s all nicely integrated, and that’s great. But actually, what you get as a result of that is you get things that are kind of not best of class. But culturally, because it’s the Microsoft product, they feel bound to it. So you have to provide something that is better in order to get any traction in doing something else. But you only have to do that once or twice, and then they’ll start to question the whole thing. They’ll start to question whether there are other bits that don’t necessarily have to come from Redmond. VSTS is a really good example because TeamCity is increasingly common in what we see now. We see a bit of Jenkins as well. The source control element of VSTS is dying incredibly fast. Even Microsoft themselves are moving to Git now. Actually, it was an interesting, just a kind of sidebar, there was an interesting thing that they said recently about moving the Windows Core codebase over to Git. Which is apparently 40 billion lines of code, which takes a little bit of time to clone, apparently. But even they’re moving away from some of the bits themselves, as they— and it probably is tied into some of the open-source stuff as well, that they’re kind of moving away. So, you know, again, it’s back to what I said at the beginning. If they’re doing it first, they’re kind of giving permission to for the cultures that they’ve created to also change.
Matty: [00:22:28] I agree. And I’ve definitely been places where it’s the Microsoft is the answer, what’s the question? And to be fair, there’s places where IBM’s the answer, what’s the question? And I think it was at the— Adam Jacobs talked about this at a couple of different ChefConf keynotes where he’ll say that kind of in the old software vendor way, right, like you had your one major software vendor and Y’all went out golfing and blah, blah, blah, and then you’re like, I need a thing that does a thing, and you’d call up your vendor and he’d be like, well, we sort of have a thing that does that. And you’re like, cool, give me that thing, right? And then it sucks because that’s not really what they’re good at. So I agree that like Liam said, once you at least can open that up to say, let’s put the Redmond tool on the same playing field as the other tools, right? So if we make that decision, we’re making it with intentionality. And not just because. And I think Microsoft has made that way easier in number one, like you said, by leading by example, by saying like, hey, we don’t even do our thing. Like this was one of the things for me, uh, with running, you know, a pretty heavily trafficked e-commerce site on that was all Microsoft-based and then trying to manage it and monitor it from the front end using SCOM. And I, you know, one point was like talking to Microsoft and like, what do you guys use to manage, to monitor microsoft.com? And they’re like, not SCOM, you know, like we use Gomez. And it’s like, okay. And, but so it’s first of all, leading by example to say like, hey, maybe this isn’t the right thing. And then again, System Center is a good example of this where they made it way too much like, well, if you want this thing, you gotta have this thing. And like, this was the thing that, you know, Stover’s been saying for the last couple of years, which is the, hey, would I rather you ran .NET application on your— when you’ve got a Windows server, would I rather you’re running a .NET application? Yes, but you want to run a Java app on it?
Trevor: [00:24:28] Okay.
Matty: Okay, in Azure, would I rather it’s a Windows VM? Yes, but you want to run a Linux VM? Okay. They’re like, I’m getting paid somewhere, right? And so there’s that decoupling of the need for all of those pieces is making it simpler. I think what’s hard within an organization of the size of Microsoft, and I’ve said this before, is that I feel like up at the very top, they get it, right? And like down at the line, like the engineers get it. The middle management and the sales don’t get it yet, right? Like there’s still, it’s still, there’s a lot of stuff that gets pushed though. It’s hard if you are an organization that’s not like really driving that cultural change to like look at the other stuff, you can get a lot of messaging from— you still are comfortable with that kind of primary vendor. I’m curious, so Glenn, like you’ve been building a lot of tooling around integrating, you know, like you said, kind of— I, I’m intrigued because like I’ve got some thoughts on it from the Chef perspective. I think we’re talking about, you know, doing a lot of the same stuff, but taking tools like Chef or Puppet that were kind of never intended to work in that space. And in both cases, I think Windows is a first-class citizen in either of those frameworks now. What’s that experience been like for you to kind of help that journey along, or what have you observed?
Glenn: [00:25:57] It’s interesting. So you’ve got people that have been using Windows back in the dark days and they don’t want to ever touch it again. You’ve got new people that come into the Windows infrastructure and they’ve used things like Visual Studio and things like that and going, yeah, we can, you know, get this great IDE. So you’ve got people on both sides. I’ve even had a Linux admin who is actually absolutely loving PowerShell and hating that they have to go back to Bash. But yeah, it’s interesting trying to apply things like Puppet and Chef and DSLs and things like that. Trying to get people’s mind changed about how you’re actually going to model architecture as opposed to just deploying infrastructure. Particularly in a Windows world, we’re so used to just next, next, next, next, next, finished.
Matty: I think there’s also a thing from the vendor perspective, and it’s funny, like, the irony is this was the lesson that Microsoft learned, themselves learned the hard way with Office 6, I think it was, on Mac, right? Where they said, because Office, Word on Mac was amazing, right? Like from day one, it was there before it was there for Windows. And then it was like, and again, I may get my history wrong, but I think it was like Office version 6. They’re like, let’s make it look just like the Windows version. And it was terrible because it’s like, it’s a different operating, it’s a different thing. And that’s been one of the things that I know within Chef, when we look at that, we’re like, what we don’t wanna do is take, like there’s certain things that are commonality, right? You’re like, like you said, the way you model infrastructure in certain ways, it really doesn’t matter where you’re coming from. But when we’re looking at things like stuff that we’re doing with Habitat or things like that, we’re like, you want people to be in where their comfort zone is. Like, the tool should not get in your way. It should be like, hey, if I’m a Microsoft— if I’m a Windows sysadmin, I do have a way of looking at the world. And that’s not a wrong way, right? That’s not a dig. It’s just a different way. And if I have to completely like turn my my world on my ear for the 10 minutes a day that I use this other tool, that’s just totally— then I, as the toolsmith, completely failed, right? Like, it needs to be— it needs to feel like Windows stuff feels a certain way, like Unix stuff feels a certain way.
Glenn: [00:28:21] I think Chef and Puppet need to do both better jobs at trying to onboard Windows people into that kind of space. I mean, I did a PuppetConf talk about that, you know, how not to freak out when you start running modules for Windows. So we have a lot of work to do.
Matty: I think it’s also that Microsoft is helping us by making that ecosystem easier to address. There’s something, if you— and I’ll put a link in the show notes if I remember, but one of my favorite episodes of DevOps Cafe was years and years ago when they had Jeffrey Snover on. And it’s when, you know, first time I ever heard him make the reference to the differences. Unix and Linux are a document-based operating system and Windows is an API-based operating system. And that’s why there, you have to treat them differently. And that was sort of his story about like, you know, Snover came to Microsoft to write Unix tools for Windows and was gonna be like, oh, this is easy, no problem. And then went, holy shit, this is totally different, right? It’s hard to do. And that’s been, you know, that was again some of the early challenges with— and I was doing Puppet and Chef on Windows when Puppet and Chef on Windows was not the thing that you really were supposed to do. And it was super duper hard because except for IIS, there’s a reason IIS is also the example everybody uses because it’s document-based, so it’s super easy to manage. But as Microsoft gives us that ability, like we don’t have to hack our way around it. That’s the thing. And I imagine it’s similar in the Puppet world, but like if I think about bootstrapping in Chef, Linux has a bunch of stuff I already have for free that I could just get at. I’m like, I have SSH, I know it’s gonna be there, I know exactly how to get at it, it’s gonna work, you know, 99% of the time, it works 100% of the time, cool. Then I go to WinRM and I’m like, every single enterprise in the world is scared shitless of WinRM, the first thing they do is turn it off because they’re freaked out, blah, blah, blah, all this stuff, and so we have to do all these hacks. And it’s also like kind of hard to set up ’cause like, okay, I wanna have a common set of, I wanna secure my SSHD. That’s a file, dump this file on all my servers. Now I’m done. Now I wanna secure my WinRM. I wanna make sure that I’ve got WinRM configured properly across all my things from boot time. Well, run this command, dump this registry key, do this. So what do I do as an InfoSec person? Screw it, turn it off. No, no WinRM. So like, as Microsoft, and I think they’ve, you know, are getting better at thinking about remote management first, obviously. I think that makes it easier as a toolsmith to make the onboarding not— make the onboarding more delightful because you don’t have these yaks you have to shave just to get started.
Liam: [00:31:13] Yeah, not all of us are Matt Rock.
Matty: Few of us are.
Brandon: Yeah, onboarding these tools is really a kind of a super important concept to get right for a lot of people, especially when you have Windows admins who are not familiar with these technologies and you’re telling them this is the new way to do things. And if they keep on running into roadblock after roadblock after roadblock, what incentive do they have to keep trying? You know, it’s very easy to just pull— to throw up your hands and say, you know what, this is not going to work. So whatever the community can do and vendors can do to make this as frictionless as possible is, is the right way to do this.
Glenn: Has anybody used the Docker for Windows installer?
Matty: I’ve installed it. I also ran it to prove Glenn wrong about something once.
Glenn: I’m just bringing that up is actually a good example of how you can have the best of both worlds because the installer is graphical, it works, and you can still fall back to good old Docker commands. It’s actually a pleasure to use when it works.
Matty: [00:32:23] Yeah, and it’s funny, I actually feel like maybe it’s because I’m not on my Windows machine as much, but I’m like, I feel like it works better than it does on my Mac because Docker for Mac seems like it’s every third release is the one that goes, this uses 5,000% CPU randomly now. Sorry about that. The Windows one doesn’t seem to do that. I would agree that that experience, I think the Docker team did a really good job from that onboarding perspective. And I don’t, I’m going to ask, you know, I guess folks who’ve used it, like I’m obviously more familiar with ChefDK. Like I think we’ve come a long way. I think obviously there’s still work to do, but the ChefDK installer is pretty nice, right? In terms of it’s, it’s a graphical installer, it gets you your pieces and it gives you a shell that’s already configured the way it’s supposed to be. So it gets you most of the way you’re supposed to be there, but there’s still stuff where it’s like, and then it’s like, but oh, you want this thing to work? Well, go read the doc on the GitHub repo and run this command into your profile and do this. And it’s like, but really shouldn’t that be a choice? And so like, is there, so when I’m trying to onboard, just sort of thinking through other technology and other things like that, Like, well, let’s think like, you know, let’s talk about like DSC, for example. What’s that experience like if I’m like, I want to start doing DSC? Like, do I totally freak out? Is it a delightful onboarding experience? Is it something that I have to really, really want because I’m going to push real hard to get there? What do you think?
Chris: [00:33:56] I’d say it’s delightful for about one server, and then after that it starts to go downhill.
Brandon: Yeah, you really kind of need a management platform to deal with DSC correctly. And I think Microsoft has stated that as well, is like, well, you know, that’s where Chef and Puppet can help you out. You could, of course, roll your own, but that would be a hard chore to accomplish.
Matty: I think that’s the hard part about that, right? And that’s the quote, right, is that— again, he’s not on the show, but he might as well be because we’re going to quote Snow for like every 20 seconds. You know, is the, you know, DSC is a printer driver, you know, Chef or Puppet is Microsoft Word. You’re wrapping into that. I guess I’m thinking about when I want to get started, I’m like, hey, I want to write a thing. Again, zero to delight. I want to sit down, I want to start infracoding my infra. It’s way better than it used to be. I remember when I was at Tenth, and if we had customers that were going to be Chef customers, we’d be like, here is the 12-page document of how to set up your workstation, so that you can write some Chef code. Now it’s like, okay, cool, so at least there’s ChefDK. Now here’s like 3 more pages of document to be able to enable DSC if you want to write DSC, use the DSC provider and stuff like that. What’s your wish list for making all that better?
Liam: [00:35:27] For me, and we essentially have the model of using Puppet to wrap DSC, Actually, at this point, and I’ve used a bit of Chef as well, actually, at this point, most of the tools in terms of initial usability are actually pretty good on the whole. It’s actually now the second and third step of maturity that’s the next roadblock, because there’s a huge jump in going from, Okay, I can install Puppet or Chef, and I can create some resources, I can create some files, and I can install some packages. But then there’s generally this huge other jump of, okay, now, in the DSC example, you kind of have to have this management infrastructure. Even with Puppet these days, you kind of have a huge chunk of infrastructure to manage, to roll that out to a sizable amount of machines. And so there’s this kind of gap between, okay, I’ve got some basic stuff running, but then, you know, here’s how I manage my entire fleet. There’s a kind of a big step there that I imagine most of us on this call have been down that rocky path. But that isn’t at all smooth, and everyone seems to kind of take a different path down it, but it’s all kind of very much the same. And it’s down to things that are common across a lot of communities, which are, you know, this is how I structure my projects, this is how I, you know, this is how and why I need that management framework. This is why the audit tools and things are there. All of those components that make up config management in production is not necessarily a story that is, I think, as well told as it could be.
Glenn: [00:37:46] Have you found it kind of governed or shaped by the culture of the company you’re in?
Liam: I think so. I think Definitely, it depends on, you know, from my perspective where I am now going into kind of various different types of organizations, it is entirely dependent on the maturity of the engineers within that organization. If you’re going in trying to put in Config Management into an organization that is it’s not their line of business, it’s an internal IT-run organization, then that jump is going to feel epic to them. But if you’re going into an organization that has operational staff, that has engineers that consider IT essentially their business, even if it might not be necessarily, they consider it a core function of their business, then they have engineers that might care about that sort of stuff. So, even if they don’t know it necessarily, they’re gonna have a much smoother time.
Matty: [00:39:00] I think, too, I mean, yeah, it’s also like Conway’s Law in action, right? So, depending upon the culture of the organization is how you’re gonna feel comfortable about, you know, in a high-trust organization is where you’re gonna feel good about moving stuff to the left and saying, like, sure, I’m not scared to let or Chef run every half an hour on my nodes? Because actually, and the thing is, you know, it’s like, I don’t remember even the context, but Adam was like, sometimes when people feel some way really strongly in their gut, they argue with math, right? I can sit and I can prove to you on the whiteboard why when you tell me you want Chef to run once a month on your nodes, you should be scared shitless of what’s gonna happen when it runs, right? But if it runs every half an hour, you should feel super good about it. And I can tell you that, but if you are in a non-trust culture, it doesn’t matter. You’ll argue with the math, right? You know, and you have to kind of be able to get to that. And I think that there’s organizations where if it’s not one where, you know, like you said, failure leading to inquiry, right? Like if you don’t have that idea where instead failure leads to like punishment or failure leads to let’s put up a blocker wall to make sure this never happens, you end up with all these artificial walls that could be fixed in a different way, but you sort of have a very symptomatic fix to it, which is the, oh, well, one time somebody screwed this thing up. So now we need to make it so nobody can ever release anything ever, unless it’s the forms filled out in triplicate. And it’s like, but wait, that’s not really the problem. So I think that’s challenging. I think the other thing that I’ve started to realize is living in the DevOps vendor bubble, you kind of forget that certain concepts like test-driven infrastructure or testing infra code and everything, that is just— I’m just like, dude, that’s table stakes, man. No, it’s so not, right? I think as vendors and as partners and people who are practically trying to help people, we need to remember that What we think is table stakes is not, and we need to bring people up to some of those most basic things. It’s too bad that Trevor’s mic doesn’t work because he’s got a super good story about this. Trevor, do you want to try to talk? It’s too bad because he has this great story about teaching test-driven development to an infrastructure engineer and them having that lightbulb moment of going, Oh cool, so now I totally know that it’s gonna work and now I can feel good about that.
Brandon: [00:41:46] There unfortunately is still quite a few admins who still use .bak as their source control system. So yeah, there’s a lot of work to— and expecting someone who’s at that level to jump into Chef and Puppet is probably not realistic or very— be very difficult. So it’s a lot of baby steps to learn the fundamental, the absolute fundamentals of working in this type of way. So what is proper source control? You know, what is configuration management at a basic level? What is, how do you test your infrastructure? You know, what test frameworks are there out there? There’s a lot of stuff to get someone who’s not working in this kind of mode There’s a lot to get them up to speed. You know, there was one instance where a person I was working with, and he’s been doing IT for, I don’t know, 30 years, and he’s like, you know, I’m basically learning my job all over again with all these new tools. And I was like, yeah, you kind of are. You know, there’s, you know, source code systems, there’s testing frameworks, there’s configuration management, there’s new monitoring tools. There’s a lot for some— for people to swallow.
Chris: [00:43:03] Maybe it’s, I don’t know, is this just my perspective, but it seems like from the Linux side, most configuration management effort came from the dev side. Like a bunch of developers said, I want to build servers better. And the Windows side seems to be sort of led more from the ops side. And they’re just they don’t have that skill set of working with software the same level.
Matty: So maybe where people were piecing it, but that’s like, I mean, Luke was a sysadmin, Adam was a sysadmin, you know. The only person who’s like the, I, I wrote infra software and have never stepped foot in a data center before, is Mitchell at HashiCorp, which I always thought was funny. He wrote Faker and he’s never never set foot in a data center. I’m like, well, that’s pretty cool that you could do that. But there is sort of that, but I think it’s true though, is the approach. And this is one thing I will tell you that I learned. I sort of knew it, but I learned it a lot from doing pre-sales at Chef, is there is like that notion of Linux admins are good, they’re programmers, right? ’Cause like, that’s very Unixy, right? Like you have to, like, you write code, you do that stuff, you’re command shell junkies, You know, fuck the GUI, blah, blah, blah. Windows are all click next admins, blah, blah, blah. And I gotta tell you, man, so I’ve been out there, I’ve seen a lot of companies in the last couple of years. I’ve seen a bunch of Windows admins that can shred PowerShell like crazy, and a bunch of them are on this call. I have seen so many Linux admins that are like, you’re taking away my blade logic where I click, but wait, but no, but in this thing I get to click a button and it does a thing. I want to do that. So, first of all, I’m just trying to spread that gospel that, like, let’s make sure people understand that, like, being bad at command line knows no operating system.
Liam: [00:45:08] I think the scary thing from what we see now is that there’s this incredible— there’s such a huge spectrum of maturity now where You’ve got, and this is culturally and in terms of tooling, where you’ve got one end that is specifically around Windows, where you’ve got line-of-business applications, you’ve got really legacy stuff that people still have that just frankly isn’t automatable. It’s never been written that way. But still, people have to manage it. Then you’ve kind of got this— what I see as this middle ground, kind of arguably golden age where we’re at now, of, you know, we have config management, we have the new shiny Terraform, the HashiCorp suite of tools that we can kind of do everything with. Even the things specifically around PowerShell, the ecosystem of additional tools and the PowerShell gallery and everything is really starting to mature now. So we’re kind of in this golden age of, we have the tools to manage it. The software that’s being written is taking automation into account. And everything seems kind of happy. And like you say, more people are aware of these tools, so they’re starting to reskill. Unfortunately, the way our industry moves is just as these people are starting to reskill, we pull the rug under them, and then we bring in all the shiny new things like like the containers, doing crazy things like .NET Core on Kubernetes, all the stuff that’s kind of on the edge. What we’re doing there is saying, okay, great, you’ve just spent 4 years learning Puppet. Well, you might not need to do that as much anymore, because this is what the next shiny thing is. But there’s definitely, I think we’re definitely in a golden age of managing Windows. I do think there’s some scary things in the future, but actually, 90% of what we’re seeing now is actually the other end, is the horrible, heinous, you know, BAU apps that were never written for this. I can give you a concrete example that might terrify you. Apparently, what’s running— of the Windows workloads that are running on AWS right now, 30% of them are 32-bit operating systems. So 2003 Windows 2000 is running on AWS. That’s a fact. And so people are putting things there that were never— 100% were never meant to be there, but they’re doing it because there’s tools to allow them to do it. So, we’re in this age of just madness, really, quite frankly.
Matty: [00:48:11] It’s a tough transition. And I think that’s the hard part. I think you summarized that super well. And I guess the last thing I would think about when we look at this, the 2 bits of advice that I would think of for someone looking at this is number one is, again, is it’s like, Don’t boil the ocean and don’t get into analysis paralysis, right? This is the thing that I see when people say, like, now we’re gonna do, we’re gonna chef all the things, or we’re gonna continuously deliver all the things or whatever. They sit down and they need to think of every corner case and like, okay, so how are we gonna make the whole thing highly available and super scalable? And I’m like, you don’t have a single node under management yet. Why do you give a shit, right? Like, write some code. Do some stuff, make it better later, right? Or the same thing of, like, well, we shouldn’t build a pipeline for our Puppet modules because we haven’t run any tests yet. Fine, have a pipeline that does nothing but just shoves it through, and that’s how it deploys, and put the tests on later, and figure it out as you go. It’s like avoid the yaks. I have this problem myself. I’m going through it with— I’m like, oh, okay, I want to rewrite some frontend code. So I’m like, oh, well, you know what? I’m driving my car through there, and I’m like, yeah, it’d be really cool if, like, this thing did a Travis build job where it did a Selenium thing and it made sure that the logo looked right. The next thing I know, it’s like 4 hours later and I’m futzing around with PhantomJS. This is all for some website that I’m the only one that writes code for anyway, so who cares? It’s super easy to keep going into pulling at threads or shaving the yaks, which is, oh, well, if I’m going to do this and I have to learn this testing framework, Then I have to learn this, then I have to learn this. Wait a minute, how does Netflix do it? Oh, well, they have all this stuff, so we better have all that too. It’s like, just pick one thing, do that one thing, then move on to the next thing. I think that helps you with all that new shiny also because you’re componentizing it. You’re sitting there and if you’re saying, hey, okay, so we were using Puppet and doing all this stuff, and now it’s like, oh, well, now we’re going to do stuff with containers. So maybe I don’t have to do config management, but along the way, when I was implementing Puppet, I was implementing some continuous delivery practices, and I was thinking about writing tests and thinking about my infrastructure as code in a way. So I can take that same— because those are the harder things at the end of the day, like learning Puppet syntax or Chef code or, you know, Ansible or any of this nonsense, right? Like, that’s, that’s the easy part because you just go look that up on the internet. You’re just gonna cargo cult somebody else’s code anyway. But the thinking idea is what’s super duper hard. So I think the more that we can get the principles in place, and we are running out of time. So I want to have one wrap-up question from everybody. I’m going to maybe start to ask this question on the show, but I want to know what’s your, especially since we’re now talking a Microsoft show, What’s your daily driver? So we’ll kind of go around, I’ll ask each of you, what’s your main workstation and what’s your kit on it that you like use for your stuff? So Brandon, I’ll let you go first.
Brandon: [00:51:24] So I’m running a, like a 4-year-old Mac Pro cheese grater with Boot Camp running Windows 10 on it. So I had a former boss who left the company who had a really killer rig and I stole it from him when he left. I’ve been running it ever since.
Matty: Any especially cool tools that are indispensable to you on your kit?
Brandon: PowerShell.
Matty: PowerShell, done.
Liam: Yeah.
Matty: How about you, Chris?
Chris: Yeah, I have a corporate issue Lenovo T450s and nothing exciting hardware-wise, but Visual Studio Code is Definitely my current go-to tool because pretty much if I start messing around with something like a new language, I can just go pull down an extension and the editor is the same and it just has all the language handling sort of built in.
Matty: [00:52:28] Glenn?
Glenn: MacBook Pro with no macOS on it whatsoever. Windows 10 all the way. A lot of PowerShell, a lot of Hyper-V. Visual Studio Code is one of the only things I can do Ruby, Puppet, PowerShell, YAML, blah, blah, blah, blah, blah, all in one editor. That’s fair. And Git, of course, the integration is fantastic.
Matty: I always find it ironic that Visual Studio Code’s Git integration is better than Atom’s. Liam?
Liam: So I might be the odd one out here. So my hardware is a— I’m a I’m running Ubuntu as my desktop OS. Mostly I’m doing a bit of PowerShell, the open source PowerShell now. I write all of my stuff with Atom. Then all of my Windows stuff, I run Windows 10 in a VM. But I mostly use that to do Bit of PowerShell dev and any RDP-related issues.
Matty: [00:53:34] Ruben, what are you rocking?
Reuben: So I have a department-issued Dell, I think it is, but I don’t use it because it’s a tank. So I run around on my own Surface Pro 3 running Windows 10. Really like it. It’s really nice. I’m a console kind of guy, so I have my COM emulator. And started using a lot of— oh, the ISE plugin is an ISE plugin that was really nice. That’s— I’ve just forgotten about, but that’s it. Steroids. Yep, that’s been really good, but have been leaning more towards code now. Although I find that the terminal stuff is not quite there in VS Code for PowerShell. So I kind of stick back to, um, IAC.
Matty: Yeah, I wanted to make sure that those of you who are listening who don’t know, so Trevor’s having like massive AV problems, which is why you haven’t heard from him since maybe he said his name at the beginning. But there is a running thing when— I don’t know if he still has it or not, but Trevor at least at one point had this like one of those 17-inch laptops that weighs like 50 pounds and, you know, 32 gig of RAM and 12 petabyte hard drive and all this crap. And so, you know how like they’ve got those stickers that say, my other computer is an Azure data center. So I took that and just got rid of the other and put on his— he had that on his computer and just sort of scratched out other. So it just said, my computer is an Azure data center, ’cause it was ridiculous. I’m personally MacBook Pro 3, I have a Surface Pro 3 that is running Windows 10. I almost installed Linux on it this past weekend, but then there was like a weird mouse thing and I got bored and didn’t do it. And then that would be the— then I wouldn’t have any actual Windows hardware anymore if I did that. So I’m debating, but it might be kind of fun to do because it’s dorky. Yeah, so cool. So let’s— we sort of were alluding to them a little bit, like, let’s go into checkout. So Liam, what do you have for our listeners to check out?
Liam: [00:55:50] So the first thing is, yeah, one of the tools I use every day for building infrastructure is Terraform. James Turnbull is due to write a— or complete his new book on Terraform. You can find that at terraformbook.com. It’s expected to be exactly the same standard that you would expect from James. So, if you’ve read The Docker Book, or the Logstash book, or his recent book, Art of Monitoring, you’ll be familiar. So, yeah, that’s due apparently at the end of this year, so I look forward to that. The other thing is a non-technical pick. I’m currently binge-watching a show on Netflix called Black Mirror. It’s a satirical dark comedy series from the UK. You should go and watch it. It’s a special comedy. But yeah, I’ve been watching it since it aired in the UK, and I’ve been catching up and binge-watching that recently. So yeah, that’s my non-technical pick.
Reuben: [00:56:56] Awesome.
Matty: Brandon.
Brandon: So my pick is, I’ve been playing a lot with infrastructure testing with Pester. And kind of using Pester for what it wasn’t originally designed for. But I see more and more people using Pester to test, you know, standard infrastructure things like, is my service XYZ running, you know, can I hit a port, or, you know, things like that. So my pick is the Operation Validation Framework that Microsoft has out there on GitHub. I think it’s included now in more recent builds of Windows 10, and I think it ships with 2016 as well. But it’s a nice little framework for packaging up Pester tests into a PowerShell module so you can publish and version your tests, which kind of creates a pretty powerful platform for monitoring or doing basic monitoring with Pester. Pretty cool little tool. Microsoft hasn’t been a whole lot, hasn’t been very vocal about it, but it’s out there at the checkout if people want to start playing with Pester for testing infrastructure.
Matty: [00:58:06] Awesome. Then Glenn, I’m sorry, Reuben.
Reuben: Reuben. First one is a repo by a guy called Ian Cooper. He’s a really smart guy and someone who seems to understand DevOps. He’s got one called Brighter, which is a command dispatcher, kind of pattern for a microservice. So I use that a lot for reference because it’s got some really healthy abstractions and ways to get you to a better microservice. And I have Serilog and Seq because I think logging is way better than debugging because you can’t really debug in production. And the other one is a library by FullHack who ported— I think it’s Zach Holman’s Scientist, which allows you, if you haven’t got tests in your code, you can do parallel traces of code to test to see if your new code gives you the same as your old code, which is way cool. You can reference that assembly in PowerShell. It gives you a little bit more confidence when you go to release.
Matty: [00:59:12] Cool. Now, Glenn.
Glenn: Yes. My first pick is Neo4j, which is a graph database, and a bit of a shameless plug because I help out with the Windows side on that. They’ve got a free ebook from O’Reilly, so you can just sign up and actually learn all about graph databases. They’re pretty cool. I do enjoy them. My second one is Flow Perth. I’m originally from Perth, and what they do is they organize some 2-day hack days, and they get agile people, developers, engineers, and we work with not-for-profit organizations, And we help solve their IT problems. So using our awesomeness for the forces of good. So we’ve helped Miracle Babies, we’ve helped Fatherhood Project. So just some fantastic community outreach.
Matty: That’s awesome. Chris.
Chris: Okay, so my first one is I’m a big fan of GitKraken. It’s a GUI Git client written on, I think, Electron, but they also took it a step further and actually wrote a Git client in Node, so it doesn’t have a dependency on git.exe. It’s pretty fast, it’s good looking, helps me actually do some fun branching and merging and stuff with that. Having to spend too much time in the main pages of Git. My other one, actually Doug Fink pointed this out to me the other day, is a Go executable called Termeter, which does graphing at the console. So it does surprisingly good real-time ASCII graphs. Like, so you can, pipe pretty much any tab-delimited data to it, and you get graphs without pop-up windows or leaving your terminal.
Matty: [01:01:17] I might have to check out that GitKraken because I’m a big fan of Tower, which is Mac only. And I used to be very anti-Git GUIs till anytime you have anything more than very simple trees to look at. But Getting any of the merge tools working right in Tower seems to be an exercise in futility. And so I was looking at that on the pro version of the GitKraken. It’s like it, it has an in-app merge tool and I’m like, that might be—
Chris: it’s a, it’s a pretty good tool too. I mean, it’s not like cumbersome.
Matty: So let me check that out, but I don’t know, messing with my workflow.
Chris: It is cross-platform too. Yeah.
Glenn: Git Tower just got released for Windows today.
Matty: Oh, cool. Well, that’s sort of cross-platform. I’m just so used to doing things a certain way. This is terrible. I’m like, yes, DevOps, and then I’m like, don’t change any of my stuff. So Trevor, are you going to try your checkouts?
Glenn: [01:02:17] Can you hear me?
Matty: Yes.
Trevor: Can you hear me now?
Chris: Yes.
Trevor: Really? Really? Now it starts?
Matty: Jesus Christ.
Trevor: So I’m looking at the, uh, the Blue Raspberry portable microphone. I’m not 100% yet, but it looks small and not as heavy as my Yeti. Um, so much like my 17-inch laptop that I abandoned, I’m going to abandon the time. Um, also Factorio, which is a really fun collaboration game where you’re building a— you’ve like crash landed with a bunch of people and you have to basically reinvent everything you need to get off the planet, which is kind of neat. I’ve also been playing with PvPGen, which is an open source implementation of the classic Battle.net and Westwood servers. So if you’ve played like the original Diablo, Diablo 2, Red Alert, those games, that was the system. It’s an open source implementation of the the servers for that. I finally got it running, which took me Linux and how to use CMake. And now that I got it running, just running, and I can connect to it and play games, I’m trying to stand it up inside a Habitat because shave all the yaks.
Matty: [01:03:38] So those of you who couldn’t understand half of what Trevor just said, there will be links and descriptions of all of his checkouts in the show notes at Arrested DevOps dot com slash Microsoft too. But we did want him to be here in spirit with his checkouts. My checkouts, I’ve just got a couple. So first, maybe it’s weird to talk about an app that’s only available on Apple devices on the Microsoft show, but lately I’ve become enamored with an app called Bear Writer. That’s bear-writer.com, and it’s basically just like a notes app with like cool markdown stuff and I’ll probably love it for like 2 months and then never open it again and move on to some other note thing. But right now I’m really enjoying it and it’s nice. It’s just very, very good across my iPad and phone and Mac and everything. Um, I’ve also decided to become obsessed with Go. So I’ve been writing a bunch of Go stuff just because why not lately? And I recommend Nigel Poulton’s, uh, Go Fundamentals course on Pluralsight was really, really good. So it’s not free. You have to have a Pluralsight subscription, but if you do, it’s just really good, and I look forward to checking out some more of his other stuff. And so Franklin Weber just released a plugin for the Atom text editor for Inspect, and you can get that at github.com/chef-training/language-inspect. He’s also working on an extension for Visual Studio Code for Inspect as well. Check those out. I’m sort of trying to write an Atom plugin for linter for Cookstyle instead of RuboCop. We’ll see if that ever actually gets released. If you have an upcoming conference you’d like to see promoted on ADO, you can fill out the handy-dandy form at arrestedevops.com/con. So, guests, where can people find you on the internet or maybe in real life? Are you going to any cool conferences in the near future? Brandon?
Brandon: [01:05:40] Yeah, I’m— so I’m DevBlackOps on Twitter and devblackops.io is where you can find me on my blog. And the next conference I’m going to is the PowerShell and DevOps Global Summit up in Bellevue in April.
Matty: Liam?
Liam: The best place to find me is on Twitter @LiamJBennett. My next big event is I’m at re:Invent in Vegas at the end of the month, which should be fun. Cool.
Matty: Chris?
Chris: I’m @LogicalDiagram on Twitter and will also be at the PowerShell DevOps Global Summit in April.
Trevor: Glenn?
Glenn: @GlennSardi on Twitter and glennsardi.github.io if you want to check out my blog. Hoping to be at the PowerShell Summit in April in Bellevue and hoping to be at DevOps Days Seattle in April as well.
Reuben: Awesome.
Glenn: And once I put my talk in, I hopefully might be speaking. We’ll see how that goes.
Matty: Sweet. Reuben, where can people find you? That sounded ominous.
Reuben: [01:06:44] So, @DefSol, D-E-F-S-O-L, on Twitter is primarily where I am. And down in the Southern Hemisphere, it’s getting Christmas summertime. So we’re all talking about taking breaks and going to the beach as opposed to going to any conferences next year.
Matty: Awesome. So yeah, so head over to arrestedevops.com/Microsoft2, that’s the number 2, for this episode’s show notes. You know, our website has links to sign up for our newsletter. You can get merchandise. We got cool shirts. You can support us on Patreon so we can do this more often. Basically, all the rest of DevOps stuff you could ever want, and maybe even some stuff you don’t want. Leave us a review in the iTunes Store. It helps people find the show. Yeah, so thank you, panel. This was great. I’m really excited. We had a lot of really smart conversations, I thought, and we didn’t break Google Hangouts. That was awesome. That was my goal. With that, I’m Matt, @MattStratton. I’m Trevor, @TrevorGHess. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand. That was my terrible Trevor impression.
