Enterprises with Bryan Liles

Bridget: [00:00:02] Greetings from the editing booth, folks, with a quick disclaimer. The internet gods did not smile on us during the recording of this episode. We had some technical difficulties that led to some audio issues. I’ve done my best, but I apologize in advance if the audio quality lags a bit here and there. Okay, disclaimer over. Let’s start the show.

Bryan: When you have more than 2 lawyers, everything becomes harder.

Bridget: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout, and with me today, I’m Matt Stratton.

Matty: Uh, today we’re going to be talking about DevOps in large enterprises with Bryan Liles. The show notes for this episode can be found at arresteddevops.com/enterprise. But first, A word from our sponsors.

Bridget: Arrested DevOps is brought to you by 10th Magnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. 10th Magnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/10thmagnitude.

Matty: [00:01:23] This episode is sponsored by VictorOps. Built for modern incident management, VictorOps provides a unified platform for real-time alerting, collaboration, and documentation. Driven by your IT and DevOps system data, VictorOps helps you respond to incidents more effectively so you can minimize downtime and make being on call suck less. Visit arresteddevops.com/victorops to schedule a demo or start your trial. Mention that you heard about VictorOps here on Arrested DevOps and you’ll be eligible for some sweet discounts too.

Bridget: This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools such as Chef, Docker, and AWS, so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. This is actually exciting and we shouldn’t cut this out because I went back and looked. The first episode where I was an actual co-host was one where we were talking about DevOps in large enterprises, but the URL was different.

Matty: [00:02:29] It was also when the Goat Farm podcast was launched.

Bridget: Exactly. So, yeah, it’s very exciting. So, today’s guest, I am really excited. Bryan, do you want to tell our listeners a little bit about yourself?

Bryan: All right. Well, I’m Bryan Liles. I’m in the DC area. I wouldn’t consider my job a DevOps job, but I have a lot of what we might consider DevOps components. I’ve been— I started out over 20 years ago as a sysadmin. I’ve done security, I’ve done networking, I’ve done development, a whole bunch of different languages, and now I’ve ended up here as a director in engineering at a large-ish bank, not a large one, but a much larger one.

Bridget: No, what I think is really great and really funny about that is I think when I met you, you were at DigitalOcean, which is probably the opposite of a giant enterprise.

Bryan: Yeah, so I’ve always wanted to do a whole bunch of different things. And really the appeal of DigitalOcean to me wasn’t really the hot startup. It was more of how can I get down to the basics of where I started? My first job was at an ISP. I worked at one of the biggest hosted providers. I worked at the company that created the term application service provider, which is the precursor to SaaS. Nice. This has always been where I’ve been. I actually heard about DigitalOcean on Hacker News and I’m like, oh, this is cute, and it kept on moving. Then it just got to be a position where they’re like, you should come work for us. I interviewed there and I stayed there for a few years. And overall it was hugely possible. It was hugely positive. And I’m really thankful for the time, but it isn’t where I wanted to end up. And so I did move on because I really wanted to do something that was much bigger than anything I could do by myself. And working at a bank is one of those things.

Bridget: [00:04:30] And that, I think that that’s huge. And that’s really key to a lot of the people who listen to this podcast and come talk to Stratton and myself and Trevor at conferences. They work at, you know, someplace that they didn’t find out about on Hacker News, and maybe it has a lot of legacy, and they think, well, those, those places sound fun and cool, and they’re doing great things. How do we do this stuff at our giant enterprise? And that bridging that gap, like helping people see how they can bring all of the exciting changes that they hear about, you know, on the interwebs into their giant enterprise, is why I’m so excited to talk to you today.

Matty: You know what’s interesting that I’ve seen over the last couple of years is how that question has changed. So a couple of years ago, the question, I don’t even wanna say it wasn’t a question, it was a statement. The statement was DevOps won’t work at big corp, period. Now what’s happening is we’re hearing, how can we do that at big corp? And that is such a fundamental foundational, philosophical shift that’s super important, right? Like we, we, to be able to go from the, ’cause once we believe that we can do a thing, we just don’t know how to do it, we’ll do it.

Bryan: [00:05:44] But as soon as we find out it doesn’t work, then— We’re still missing something though. I still think we’re missing a big part of it. So how is easy. You go read a book, you can go learn how. The problem that people realize is they see what Google is doing with their SREs. They see what Dropbox is doing with their SREs. Hear about DevOps at such and such company, but you know what they always miss?

Matty: Why?

Bryan: Why did they have that solution? Why do they need that solution? I think that a lot of teams need to actually sit back and look at is, why? What are the issues we have? Then figure out why does DevOps work for me? I’ll say in a lot of cases that there’s a lot of DevOps principles that most likely will work for you, but you need to understand why, because if you don’t, you’ll never understand what success looks like, and a lot of teams seem to skip out on that. And I’ll tell you, even at big orgs, we have DevOps teams at Capital One, and they do, and we’re a big org. We are, there’s a lot of people. I can’t even tell you how many people work here, and I’m not talking about how many people work at the bank. I’m talking about how many people work in tech. I will say that it’s the biggest chat platform I’ve ever seen. When I look in our source repository, it’s pretty big. There’s a lot of code going on here every day, so there’s just, so much, but we need to understand even internally what DevOps means. And whenever you have an old-school team where you have a whole bunch of developers who hack, hack, hack, hack, make great products, and then they toss them over the wall to ops, that didn’t work. So, now we have brought in these teams of DevOps people who are really focusing more on CI/CD, but that’s not even, you know, that’s only like a like a tiny smattering of what DevOps is. There’s so much more.

Matty: [00:07:34] So I think that’s— and there’s to unpack that. So what I’m seeing a lot is that it’s getting— and even the outcomes are maybe even being understood at the very top level, you know what I mean? So like, you know, some big muckety-muck at Chef or Pivotal or whatever goes and talks to the CEO of big corp and the CEO goes, or the CIO goes, yes, you have sold me. We talked to these big broad strokes about how transformational this all will be. We are gonna DevOps all the shit, right? And then that message never makes it all the way down, right? Like, and then you’ve got the boots on the ground who are like, oh, well, we have a DevOps initiative. So I guess that means we need to like install Puppet, right? You know, it’s, they’re back to the, like on all of Reddit where DevOps means, system automation, right?

Bryan: They’re like, I—

Matty: then we need to do those things. And like you said, we’re missing— and the good thing, maybe not the good thing, but I, I, it reminds me of something, uh, Jez Humble was speaking at a meetup in Chicago a while ago, and he said the good and bad thing is I’m gonna have a job for life because nobody ever gets this. He’s like, I will always be able to be telling this because there’s so many people that just don’t hear it. But it’s like the The kind of the, the good thing is we’re starting to understand that. And also what I’m starting to see, and maybe it’s been going on for a while, but it’s specifically happening to me, is those higher leaders in the organizations are bringing in people like Bridget or myself or Jess or anybody, right? To say, you know what, come in. I don’t want you to come and talk about Chef. Come in and talk about these principles of DevOps to our town hall so people understand the why, right?

Bryan: [00:09:22] So that’s the most important piece, and we talk about this, or we can get down into the Chef versus Puppet versus Salt versus Ansible discussion. We can get into the Jenkins versus I’m using GitLab CI thing or CircleCI or CodeShip or whoever else, or Travis or whoever else. And that’s like, those are fun, and I think we talk about those because we like cool products. And I know you had the founder of Honeycomb on last week and talking about observability. You know, those are all super fun things to talk about, but really we need to back up, and especially from like a CIO’s position going down to their smattering of VPs and the VPs that work for those VPs, senior directors and the directors down to the actual workers, what we really need to do is work on focusing on those ideas of CI and CD and logging and monitoring and all these other great things as principles. And then once we understand them as principles, then bringing in tools like, we think that declarative infrastructure is a great idea. We shouldn’t— we should start with that conversation rather than bringing in the, hey, Terraform’s cool, or we’re on AWS, we’re going to use CloudFormation. So those are actually, that’s like the encapsulation things to developers. We hide all those things, but we really should be talking more about these fundamental techniques like CI/CD, declarative infrastructure monitoring and logging.

Matty: [00:10:50] How do you pick your CI tool if you don’t know why you’re doing CI, right?

Bryan: Oh, you know what?

Bridget: It’s right back to the why.

Bryan: Well, you know what? At big corp, they’ll just tell you that, as an organization, we use Jenkins. We’ve decided to use Jenkins, yeah.

Matty: And then you figure out how to hammer around it, or you— yeah, that’s—

Bryan: And we have a team that does, and we have multiple teams. We might have 20 or 30 people working on Jenkins as a shared thing, but what gets lost in that conversation is, first of all, why are we using Jenkins, and are we actually using Jenkins for the process of using Jenkins, or are we using Jenkins to help ship code? And I think sometimes, especially, and this is no fault of our developers, and I hate to, I mean, I am a developer too, so I hate to ever talk bad about developers, but sometimes developers are just given the problem of solve this thing with these constraints. And when it comes to running it in ops, they’ll say, well, nope, someone else handles that. And they actually are just brought up that way and trained that way. So what happens is they write these things And they just want to use these tools so they can work on the next feature. They don’t even care about anything else. And I think that’s a big loss, and I definitely see it in these big companies.

Bridget: [00:12:03] And I think when you’re, when you’re talking about a large company like that, isn’t that one of the risks of trying to scale up this sort of transformation, is people end up having their little territory or their little fiefdom or their little thing they want to make decisions about? And fitting that into the larger picture is significantly more work than it might be in a much smaller org.

Bryan: So what you see is that really at a larger org, and here’s my disclaimer, I am a Director of Engineering at Capital One, but I am not speaking for Capital One. I’m only working at Capital One. So what I’m getting ready to say here is not Capital One’s position, but really what I’m— and what we like to see or what I— people that think like me like to see is that it’s more of from the strategic level. So, you know, your directors, your VPs, your CXOs, they should be talking about strategy and they should be talking about the, this is where we need to be. And you let your practitioners, so, you know, everyone who isn’t one of those people, actually figure out how to get there. And as long as you have that feedback loop, things should work. So what happens in these orgs is that These products are expensive. Terraform in the enterprise probably costs a lot of money. GitHub in the enterprise costs a lot of money. What happens is because of the dollar amounts, these higher-level people need to get involved, but they need to realize that they’re only there for the financial support and for the where we need to be support. Whenever you learn how to divorce those 2 things and you establish that trust, then you actually start seeing that synergy work. I use synergy non-ironically. I just noticed that.

Matty: [00:13:47] No, it’s totally true. I want to think back to something Andrew Clay Shafer said at his— last week was DevOps Days Denver, and in his keynote, he talked about something that coincidentally I was talking about later, and it made me feel really validated that I wasn’t completely full of horseshit. But it was like all backed, and something I preach to my customers all the time is outcomes are all that matter. Right? I don’t care how you got— and that can happen at a macro level and at a micro level, right? So the outcome is— and I think, Bryan, that’s a lot of what you’re saying— what are the values? What are the things we’re trying to accomplish? What’s the outcome? The tool we use to get there—

Bryan: I’ll add one more thing. It’s not just outcomes, it’s repeatable outcomes, right? If I was able to do this on Tuesday, I better be able to do this on Wednesday.

Matty: Correct, correct. Not that you need me to tell me that you’re correct, but I mean, that’s a very, yes, that does not conflict with what I was saying. But like, I’ll give the example. This was what my talk was about a little bit is we think about all these ways we’re gonna keep people from doing something to be secure. And the reality is we’re gonna get into this arms race of trying to think of like how to keep them from doing A, B, and C, and they’re just gonna do X, Y, and Z because they can’t run A, B, and C. But really what we wanna do is make sure that the outcome of A, B, C, and X, Y, or X, Y, Z is not something— is what we want. And as long as we’re always thinking about the thing we want, and is the outcome that we’re— the thing— because outcomes represent the values of our company and the needs of our company, right? Is the outcome that we want— and we can take this all the way back to the AllSpot talk at Velocity in 2009, right, where everybody missed the point, which was everybody just looked at the— they didn’t understand. I shouldn’t say everybody because obviously a lot of people got it, but many people went back to, well, how do I do that? Oh, okay, where’s the button for— oh, that’s cool, you do 10 deploys a day. How did you script that? Which was— the outcome wasn’t that they were able to do a deploy quickly. The outcome was that because there was understanding, they had high trust and they were able to do it safely. But we still get wound up in— because we are, whether we’re developers or sysadmins, like you said, we have that engineering mindset of, I have to solve a problem within what I believe are the constraints that I have, whether they’re real or imagined.

Bryan: [00:16:05] So that’s an interesting thing. I, first of all, I will say this, that constraints are an amazing thing. And for everyone who’s working at a startup that says, you know, these problems would go away if I just had more money, that is the most naive thing. Because I’ll tell you what, Um, and, and when you work at a bank, uh, money is actually a different conversation. Um, think about it. And I don’t even work at a large bank. I work at a, like the 10th largest bank. It’s something, we’re like a Fortune 150 company. We’re not even, we’re not Goldman. We’re not Bank of America or anything like that. Yeah. But think about it.

Matty: When the largest bank in the world at one point.

Bryan: So, so if you think about when you take, so you’re always looking for this next thing and people say, what if I just had more money? No, actually, that’s not what you want. And because I’ll tell you, I can see projects that have lots of money, and that doesn’t guarantee success. The real success comes from the constraint. You always got to have some kind of constraint. So if you have all the money in the world, you got to think about, well, can we reduce the time? Can we set up a hard deadline? Constraints are a great thing. That’s, I think, all the greatest problems that have been solved, and at least in the history that I can understand, only exist because of— all these great solutions only exist because of constraint. And you look at the big examples in our community, like let’s look at Google. We can look at Google as a huge success. Google succeeded because they were trying to do this whole little ad thing, but they were trying to do it with smart people, so they were trying to do less people, more money, less people, more money. I don’t think Google So, imagine all the money that Google has right now, they couldn’t start Google again. And think about the same thing with— so, if you take, like, the Microsoft to the Google to the Facebook, they all needed each other. The constraints were that big behemoth that were before them. So, we need to take this down into what we’re doing. So, when you have a project, you need to actually start thinking about your constraints, and you need to start optimizing around those constraints. Because those are where the best solutions come from.

Bridget: [00:18:10] And I want to ask— oh, and perhaps you were about to give me, but I’m just jumping in because I want to ask, because I’ve seen, for example, a Capital One recruiter at DevOps Days Philadelphia singing Philk to the tune of R.E.M.’s It’s the End of the World as We Know It. They’re saying, this is Capital One 360, and giving this whole pitch about all of the great stuff they’re doing in public cloud, like just to bring it to concrete. If you’re defining like we are going to change in this way, we’re going to use public cloud in this way, we’re going to transform our org in this way, how do those constraints help you get to your goals?

Bryan: So I’ll give you a good example of our constraint specifics here because I’ve been told not to do that. But bank, and we publicly said that we’re moving into AWS and we have a fair-sized installation here, but One thing that we find is that we run into all sorts of account limits. Like, I know how many VPCs can be peered. I know how many images you can copy across region at any given time. I know how many security groups you can have, and not because I read the docs, because we actually ran into it. So, what it’s forced us to do is actually become way more creative in our solutions around that, and because we’ve become— A good example would be copying images across accounts. Because whenever you’re thinking of your small org and you need to copy an image from your— I’m talking about an AMI or an AMI. You need to copy your AMI from dev account to production account because you tested it, that’s not a big deal. But imagine if you had, and this is not our number, but imagine if you had hundreds of accounts and you can copy 5 at a time, and you need to copy 40 images, and you start doing the math, and you start realizing that your deploy of images, not even deploy of your software, is going to take more time and more hours than there is in one week, then you actually start realizing, oh, I need to get way more creative here. I’m not going to talk about it yet. I’m still talking to the powers that be to allow me to talk about this, but there are way more creative solutions around that. And that solution would have never have come had I been able to do whatever I wanted.

Bridget: [00:20:29] Yeah, and I think that’s a great example of the kind of interesting work that’s happening at scale in enterprises that you and perhaps some of your teams will be able to talk about at conferences in coming years is like, hey, we had to work around the either logistic or IaaS or business-level constraints that we had, these are the solutions we came up with. I find those to be infinitely fascinating talks. We’ve—

Bryan: and I’ll give you one. Okay, I want to give you one that I want to give you one that I can talk about because it’s open source. And, you know, big companies and open source, believe me, it’s when you have more than 2 lawyers, everything becomes harder. So if you have a team of lawyers, just imagine how big our team of lawyers is. I haven’t met them all, but I know there’s quite a few. Open source is hard for us just because of, we’re a bank, we’re regulated by, you know, a whole bunch of 3-letter organizations and we can’t lose money because, you know, that’d be bad because we’re a bank. So we have lots of lawyers, but we do have this tool. So one big thing for us, and you find enterprises, is governance. We need to make sure that we are providing service and doing things at a certain level because government auditors are going to come in. So one thing that we did is we created this thing called Cloud Custodian. It’s like, it’s, um, it’s— I will give you the URL at the end. I don’t want to give you the wrong URL, but it’s called Cloud Custodian. We’ll put it in the show notes. Yes, we’ll put it in the show notes. But one thing that allows us to do, and we’ve given this back to the community, is it allows us to— we believe in encryption at rest. And not only do we believe in encryption at rest, we believe in 100% encryption at rest. So we actually have built tools to allow for this, and we find that we’re actually forcing clouds to be more secure, just in general. And we built tools that if you— I did this earlier today, I booted up an instance and something wasn’t encrypted, got a nice email 30 seconds later saying sorry, and then because, you know, I’m smarter than computers, I did it again and I got another email. But this is the kind of tooling that only can come apart because, you know, a large organization had this problem, but this kind of tooling helps everyone because now we provided tools that allow you to say that it will actually watch your AWS and determine if things are not running in the way that you want, or you’re not encrypted, or you’re booting, you know, how many P2 instances can you boot, or, you know, that big one, the X1, how many of those? You could probably just shut that down and make sure it never happens. So these are the kind of neat things that enterprises can bring to the table. Go ahead.

Matty: [00:23:06] I was going to say you bring it as the things that you’re able to create and release back, but then also for your vendors, like we start to hit customers like you. And again, you like, you have to deal with the constraint because you can sit there as an engineer and think like, well, here’s how I think I would solve the scale problem. Right. Right. I don’t actually have the scale problem you have, but then when I start working with you, I understand that. And like, and understanding like the ways that we might have approached helping enforce compliance in an organization until we’re actually working with a customer who has real compliance issues, it’s all just like on a whiteboard, right? It’s all just like, well, in my theory, this is what this is, but then you’re like, and even based on my own personal experience from the past, kind of having that feedback loop both through the open source community and through your vendors is something that’s really powerful. To be able to say what happens when that goes, right? Like, ’cause a lot of this tooling we built around scale, that was a different kind of scale. You know what I mean? Like you could say, well, that worked totally fine for Facebook because they can, or Netflix, ’cause they can destroy things every 5 seconds and whatever. And you’re like, I have, maybe I don’t have their exact scale problem, but I have a different kind of scale problem, right? I have to distribute to plants. You know what I mean? Or I have stores, that are shipping medical data, right? And so I can’t transmit data like that. There’s just all sorts of things that happen in larger organizations that don’t happen in the software-only world. And we have to think of creative ways around those constraints, both as the implementer, the customer, but also as a person offering the product to say, well, we can’t just be like, well, sorry, shruggy. That is how it works. I guess you better think of something clever, because what is going to happen is someone else is going to think of the clever way to offer that solution, because you are not the only financially regulated company out there. This healthcare company is not the only one. This retail company is not the only one. Someone is going to be patient zero for Pivotal or Chef or whatever, and we are going to figure that one out. Then we are going to be able to say, oh, cool, we did this for these guys and this company that works that way. So we kind of know that works because you do have similar needs when you kind of group that way, right?

Bridget: [00:25:31] So I’ll tell you something else. Somebody’s the disruptor.

Bryan: Yeah. Yeah. I’ll, I’ll tell you something else why we need enterprises. And I, you know what, I’ve worked for 20 years before I actually put this together. So, so think about all where all those neat features come from. You do need a source of money for these neat features, and I’ll give you a good example, Terraform. Terraform and HashiCorp has done a great job with creating these products, but for them to do some of the amazing things, they’re going to need a money source because you can’t live off VC dollars forever. They need to work with bigger companies to get in there so they can actually bring out some of these more innovative features. And come to find out, you go to any large enterprise, they’re running everything. So they’re actually great places to think about whenever you have these types of new products coming out on the market. And that’s why people go to enterprises for these kinds of things.

Matty: And it’s also kind of a place where I think about it in, you know, from my perspective of, you know, having spent most of my career as a sysadmin working for the big banks, the big insurance companies, all the kinds of places you usually work in tech in Chicago. And, leaving and saying, I’m never, ever, ever coming back. Um, my experiences over the last couple of years have kind of changed my thought on that. And no, this is not me saying I’m leaving Chef to go, you know, work for, you know, Aon or something like that. But like, I had washed my hands of it, right? I was like, that’s the place that innovation goes to die because that had been kind of a little bit the case the last time I was there. And I, but being on the inside now of it, seeing it, you’re like, no, It really is. It just takes— it’s a little harder, but the payoffs are so much better too, right?

Bryan: [00:27:18] Yeah, we’re a big, slow-moving ship, but guess what? When we turn around and we actually point in a direction, you get all the force of that ship. You realize a lot of water when you turn, you know? Yes, we do. So people think, oh, I’m better, I’m this, I’m that, in a smaller agile organization. Yeah, you know, it takes a small to get there because it’s not because of this and that. The reason is, is because of the people problem. When you only have 10 people, the amount of conversations you can have is, let’s say, you know, let’s do the math, it’s like 100 conversations. Imagine when you have 10,000 people and you have multiple lines of business, the amount of conversations you have gets very large, and that’s what people are actually referring to when they talk about the enterprise. Problem is, is that there’s just more people. And what we need to do— isn’t that also—

Matty: that’s also a benefit. I was just going to say the same thing, which is you can be in a small organization and have zero ability to affect change because it’s a small thing and you’re doing one thing. And if you’re in a large organization, yeah, you’re right, like, shuffling that change through the organization takes a long time because it’s 10,000 conversations. But man, that’s 10,000 conversations where good ideas can come out of.

Bryan: [00:28:34] I would tell you the number one thing, the number one reason I really like working at a big company is we have internal tech conferences that are bigger than some conferences that I’ve been to. And they’re not high level. I would say that we have some crazy machine learning talent here. Maybe, you know, we’re not, we haven’t created TensorFlow or anything like that, but we got people who can use it and they will talk about it and they will, we have an auditorium here in Northern Virginia and they will fill that auditorium up with Capital One employees And the neat thing about these kinds of talks is they can talk about whatever. And you know how people are like, I’m being kind of guarded right now when I’m talking about— they don’t have to be guarded. They can actually talk about the nitty-gritty. We use this and such and such model and we’re doing this and getting this from X and Y customer. And it’s, and it’s pretty neat that, that we have that, but you can only have that in big orgs.

Matty: Oh, you know how much I’m dying to get into those talks? Like, I have a customer, a healthcare customer in the Midwest who, again, they’re Their tech conference is bigger than every single DevOps Days that’s ever existed combined, probably, and all this stuff. And so, like, we’ll sponsor it and they’ll be like, great, you can go sit at this table. And then they’re like, oh, hey, my one buddy over here is giving a talk on CI, can I go see it? Nope. And you’re like, ah, because it’s, it’s a really powerful thing. And it’s when the organization has made that choice around the community that they’re doing internally that you’re trying to build that brings that ability to do that. And I think that’s a really spot-on observation that you have that scale of bringing all those smart people together and not having to play that where it’s like, well, can I get approval to go to velocity? And now I have to come back and do this and knowing doesn’t necessarily apply. I imagine also, without talking like politics, it’s probably a lot easier to get a lot of your employees able to participate in something like that than to send 10,000 tech people to Velocity.

Bryan: [00:30:35] Yes, that is correct. And I mean, I will be speaking at Velocity this year, so you should go. We’ll talk more about that later.

Matty: I’ll get to Velocity eventually. I don’t— I just feel like maybe now it’s become kind of a joke that I haven’t been there and it would be weird. Hey, that’s cool. I was super excited. Like, I live my life talking to the enterprise, you know, like, because all the cool startups, like, they don’t want to talk to me. Actually, I’m sure they would, but to your point about someone’s got to fund this, right? So think about Chef. Like, I mean, we love our open source community and everything like that, but the open source community doesn’t, you know, put a DJ on stage at ChefCon, you know, or more to the fact, doesn’t fund our ability to have growth and investment in features and things like that. So just so that we’re clear, we love all the people because we’re Chef and we hug people, right? But one of the things that I’m really passionate about right now, and it speaks to something you were talking about, about the needs for governance, is what I’m seeing, and also when we talked about how we’re seeing these shifts, right, of how the enterprise is looking at DevOps today versus a few years ago, So, the first thing is, let’s all temporarily mourn the term enterprise DevOps and be glad it died, and let’s hope it never happens again. I don’t know how familiar you are with that nonsense, but there was a fleeting moment right before DOSE was a thing where there were some folks who thought that there was a whole different way of doing DevOps just for the enterprise. That’s silly. Yeah, basically what it was was DevOps without the culture. Was the idea.

Bryan: [00:32:12] Oh, no, no, no. Oh, no, no.

Matty: So I’m just saying, so let’s just all agree that we’re never going to do that again. But what I’ve seen over the last few years is the entry point of DevOps into an organization has shifted or can shift. So like when I started doing this, it was very ops-focused. Like usually when you’re kind of coming in, it was usually thinking around operations or dev. We come in one way or the other, right? And what I’m seeing is where the vast amount of new entry point into an organization around these, whether it’s the tooling or even just the ideas, is actually around security and compliance. Yes. Because when InfoSec people get this, they adore it because it’s so, you know, it’s like exactly, it’s sort of this, okay, this is this thing I’ve wanted all along and I just didn’t know how to like say the words. Yeah.

Bryan: And there’s a few people in that space and we hopped in that space. I think, I personally think that our open source is better than a lot of of these commercial software, which, you know, shows that we really do care about open source. But, you know, doing governance and compliance is hard. Sure. When you have one server. When you have, you know, many more orders of magnitude more than that server, it becomes very, very, very hard. And it’s not just the actual process of remediating whatever you decide is against your, your, your— rolling that up, showing it, providing the visibility, and then being able to trust your process that it actually will scale with you as a business. And we ran into every single one of those problems, and that’s where Cloud Custodian came out and actually showed that, hey, there is a viable marketplace for some of this. It’s interesting that this piece of open source that we created inside of our company could actually— and I think there is at least one company that’s actually out there trying to make a market about it, around it. Great for them, because guess what? You know, we do compliance because we have to, but it’s not our business. You know, our business is making money with financial products. And, and here’s the best part about being in tech in a financial company: I know nothing about money. I know how to spend it, and then I know that it comes in my bank account, and I know that My Capital One credit card works, but you know, past that I can add and subtract. But past that, I know nothing about how we make money. And that’s another great place part about working for a company like this one.

Matty: [00:34:43] It’s interesting because I had the same thing when I was at Chase, where the only time I actually understood what the line of business that I supported for the last 3 years did is when I was updating my resume because I wanted to leave. And I went and looked on our like little internal web page. I’m like, oh, $1.3 trillion a day in wires go through these systems that I manage. I’m really glad I only learned this now, because that would have freaked me out, right?

Bridget: I would love to chat, Bryan, a little bit about that really interesting Twitter thread that you had with Alice and Kelsey and a number of people about the changing role of ops. Like, what does, especially in the context of, you know, a large org and all of the— it’s 2017, things are different than they were when we all started DevOpsing or whatever. Can you talk a little bit about your thoughts on the changing role of ops? In this brave new world?

Bryan: Yeah, I’ll preface it by saying one thing that I tell everyone that I work with and who works for me is there’s 2 Brians. There’s Bryan the worker and there’s Bryan the person. Bryan the worker can do things wrong, but that does not mean that Bryan the person is necessarily a bad person. I use this when I approach ops because what I’ll say is that there are some bad ops teams out there. Teams that refuse to get any better, teams that refuse to change with the times, teams that only want to look at the problems and focus on those problems rather than focusing on the solutions. And those are the teams that are going to actually go extinct and who need to go extinct. And that’s actually how I approach that conversation is that, you know, there are bad ops teams. And I know that’s not the same as being a team that is overloaded. Bad ops teams are teams that don’t want to get better. And I’m— and, you know, I also understand that a rebuttal can be, well, we want to get better, but we can’t get better. So I will actually bring into the whole Bryan’s responsibility principle is that, you know what, you are responsible for yourself. You are responsible for your output. If you cannot affect change in your organization, unfortunately, I’m going to tell you, you need to quit, or you just need to go find something else to do in life. And those are tough words, and they’re But they’re not hurtful words, they’re words of love and endearment. Really what I’m saying is that you shouldn’t be in a place where you can’t make positive change. At the point where you can’t make positive change, you should leave. And really what I’m looking for from ops teams is that realizing that, hey, you know what, velocity is increasing. Dev teams are developing bigger, faster, more complex things that need to get out there, that have more users, that have more security vulnerabilities. And if you’re just sitting on the sidelines saying, well, um, this is too hard, guess what? You’re going to get passed and someone’s going to come behind you who’s probably going to be younger and they’re going to take your job. And then you’re going to complain that, oh, well, you know, now young people are taking my jobs. I’m over 40. And you know what? I can see myself doing this for a long time. And the reason why I can see it is because I realize that times are changing. When I first got into this kind of market, Um, we had physical servers and they ran Sun OS or BSD Sun, not even, not even that crazy Solaris stuff that we just finally got rid of, but BSD Sun.

Bridget: [00:37:59] Hey, pour one out for Sun OS 4.13, man. I’ve got some good memories of those days.

Bryan: Hey, I, I will go, I will, I will one-up nerd you and say that Sun actually built us a 4.14 when I worked at Digix.

Bridget: I know, I know 4.14 was theoretically a thing, but you know.

Bryan: Yes. So, you know, this is like that nerd nostalgia. But really what I’m saying is that we’ve come from there, you know, these old Sun pizza boxes, to now where a lot of places aren’t even running their own servers. They’re running on someone else’s custom hardware in data centers somewhere. You know, is it underwater? I don’t care. Does it work? But we got to think. We got to think that this is how it’s changing. And what we need to do is if you can’t solve the problem in front of you, Ops is hard. Guess what? You need to do— you can fail and run away, or you can change the problem. I tell ops teams this, is that you need to change the solution. If your dev team is not working with you well and they’re just throwing things over the bridge, you need to go read the SRE book. I’m not really advocating for the SRE book. I think it’s good in parts, but really what I’m saying is there’s parts in there where they talk about— what you can use is use those tenets of production readiness, and you tell your dev team that with your requirements, you have— I mean, you have your business requirements, but you also have your production requirements. So if you can do these things and then tell us how you do these things, it’ll be a much better relationship. And in a lot of cases, these dev teams are reasonable. They just don’t like hearing— they don’t like hearing that your stuff is broken all the time. Yeah. And that’s what we need to fix. And you know what, this is just one solution. And what I’m trying to really tell people is that We’re looking for the harder solutions. We’re looking for new ways to look at the problems. We’re looking at 2 plus 3 is 5, but we’re also looking at 3 plus 2 is 5. And then we’re jumping into multiplication, and we’re moving variables to different sides of the equation to try to make it work right. And that’s what you’re doing as an ops person. And really, at the end of the day, you could just say that I was the adult, and I’m the person who’s going to be owning the change. I want to own the idea of change for the positive. Not always all the results, but the idea of it being better. And that’s all that I— that’s everything that I try to contribute. That’s all it’s about, just about being better. Yeah, I mean, that’s—

Bridget: [00:40:22] I think that’s a really good point. And I think people who are trying to make this leap— and I know we’re coming up on time here, so I gotta watch the time and not go down this rabbit hole too far— but I want to affirm and agree with you that people who are trying inside their organization to get their ops teams to modernize, to get everybody moving towards the exciting future they want to have, I think it’s really important. It’s really necessary to get everyone to realize they can be empowered to change. That’s something I think that leadership really— and you’re in a position of leadership, yay— leadership has to buy in. To making change happen.

Matty: I think that actually probably brings us to a good— Yeah, I think that you’re right.

Bridget: I think that is a good place to wrap up. I would just love to give Bryan a chance to give his concluding thoughts. Concluding thoughts, people who are at large enterprises and they want to change, they want to be as great as Capital One is these days, what’s your best advice to them? Where should they start, Bryan?

Bryan: My advice to anyone to solving any problem is actually to distill that problem down to the basics. So You’re not saving the world through curing cancer, so what am I going to solve right now to make one little change? And from there, you can start combining those together to make bigger changes, to making great change. So realizing that you are only solving small problems. So really, from startups to enterprises, they’re very similar. I mean, believe it or not, it’s similar. I mean, we don’t have all— actually, no, we have great offices here. So I can actually say that our office is as good as any startup’s. The only, the only difference is, is that, you know, there are a little bit more rules. Things are more rigid because guess what? More money’s on the line. But if you’re looking for a difference in work, enterprises are interesting. And I will leave you with this because, you know, I have to say this every time I come somewhere is that, I will say that working at a big enterprise is better for diversity. And I’m not saying that they are more diverse. What I’m saying is it’s just more opportunity for women and people of color to move up. We have quite a few, and not a lot, but more Black VPs of something or another than I’ve ever seen before in my entire life. It’s actually kind of nice. We have more women who are, like, at the SVP level. Who are actually looking at women in tech, and you don’t get that in a small company. And yet, you know, it’s a small number, but guess what? We have it. Yeah, that’s a good point. It’s inspiring to me. And also, you know what, I like getting paid every 2 weeks and not worrying where my paycheck’s gonna come from. And also, I just like meeting all these new people. So it’s— or no, try— this is where I’m gonna be for actually for the more than foreseeable future. And I just like to leave everyone with one thought is, what did you do yesterday? Try to make it— try to make today better. And it’s only a little change, and that’s all we can do. And then also try to do something for someone that you didn’t do something for yesterday. And if we all did that, life would be so much better. So can I— I’m gonna leave you with, where can I find Bryan over the next couple of months? Well, I’ll tell you. So I will be at Monitorama in Portland sometime in May. I don’t know the dates.

Bridget: [00:43:48] I think it’s May 20-something.

Bryan: Yes. And then I will be at Velocity.

Matty: If you’d like to know, look and see when ChefConf is because it’s the same week. Oh. That’s why I will not be at Monitorama.

Bryan: I will be at Velocity. I’ll be at Velocity in San Jose. Sometime in June. I don’t know when it is. And I’m going to GopherCon. I’m not going to speak, but I will get a soapbox somewhere where I can talk about Go. So that’s where if you want to find me. And then also, I’m on Twitter with Bryan, spelled the right way, B-R-Y-A-N-F-L. Don’t follow me. I tweet. I’m all over the place. I’m a person, so I tweet about tech. And I tweet about other things as well. My Twitter, I’ll tweet about what I want. So thank you for this. This was actually pretty fun.

Bridget: We should do it again. Thank you. This was super fun. And I know we had some audio issues here. I assume most of it’s probably my connectivity. And now there’s some sort of, I don’t know, lawn mowing or something going on down there. I don’t even know. I really apologize for the noise there. I’m going to mute and let Matt give a few of our wrap-up announcements just so that my noise here isn’t such a problem.

Matty: [00:45:06] Sure thing. A couple checkouts. Bridget’s checkouts were, so we have GOTO Chicago is coming up the first week of May. I don’t remember the exact days, but Bridget’s the DevOps chair. There’s a bunch of awesome speakers. And we will be recording like half a dozen Arrested DevOps episodes in one day because—

Bryan: We might die.

Bridget: It’s possible.

Matty: Yeah. I said to Bridget, I’m like, you want to trade off? You do some, I do some? She’s like, no, this will be great. I’m like, we’ll see what happens at lunch and how that goes. Bridget will also be at Velocity San Jose whenever the hell that is, but you guys all have Google, so you can figure it out. I’ve got a couple of checkouts that are not conference-related, but a couple of things that I’ve learned about over the last— since last time I was on the show, which was painfully a long time ago for a host. But I discovered through someone telling me that GitHub-flavored Markdown supports disclosure triangles for details. So if you’re posting in an issue and you’ve got a long stack trace, if you surround it in details flags, and I’ll post a link in the show notes, it lets that all collapse down, so it doesn’t take up the whole darn page. I was like, that’s super useful. Self-serving, so you may know that I really love the static site generator Hugo. It’s what drives devopsdays.org, it’s what drives Arrested DevOps, bunch of stuff. I wrote a Hugo plugin— sorry, Hugo plugin. I wrote an Atom plugin to help make Hugo work a little easier, so it styles, because you write all this Golang pseudo Go template stuff, So, regular Go stuff doesn’t work right. So, check that out. It’s called Language Hugo. And I know, again, I’m always behind on what’s cool, but GitHub has a tool called Hub, which wraps around Git and lets you do a lot of GitHub-related stuff at the command line, like submit PRs from the command line. So, if you’re looking to automate things, check that out. I’ll put a link in the show notes. Oh, okay. So, hey, GOTO Chicago is May 1st through the 2nd. If you want to register for that, the discount code ArrestedDevOps will discount it some unknown amount. I don’t know because it’s not in my notes. Then if you want to go to Velocity—

Bridget: [00:47:28] I think it’s $75 off. Oh, nice.

Matty: Well, hey, it’s better than— it’s a discount. Velocity San Jose, the discount code ADO2017 will give you 20% off of Gold, Silver, or bronze passes. We got some open CFPs. If you go to, as always, devopsdays.org/speaking, you’ll see all the DevOps Days that have open CFPs. One of those is Chicago. We just opened it. Go submit talks, and I will judge you, literally will judge you, because that’s my job is to judge your talks. If you’ve got a conference you’d like to see promoted on this show, we have this form at arresteddevops.com/conf. I promise you, if you fill it out, it will come into my email box. Beyond that, we’ll see what happens. Uh, go to arresteddevops.com. I didn’t even promise I would read it. I just said it would come into my email box. Uh, if you go to arresteddevops.com/enterprise, that will be the show notes for this episode. We will have a bunch of links in there. Um, you may or may not find that URL right away. You should. Uh, we also on our site, we have the links to sign up for our newsletter. Support us on Patreon. If you go to arresteddevops.com/stickers, we have some new stickers you can get that I didn’t have in our marketplace before.

Bridget: [00:48:47] What? Stickers? I didn’t even know this. I’m going to have to go to arresteddevops.com/stickers and get some myself.

Matty: Yeah, you have to buy them yourself, but we don’t make a dime on it. It’s basically just to make it easier for you to get a couple of stickers. Leave us a review in the iTunes Store if you want to help other people find this podcast. It’s one of the best ways that new people who are looking for DevOps podcasts find us is based on those reviews. So that’s just podcast behind-the-scenes crap that nobody cares about but matters. So, uh, yeah, cool.

Bridget: But yeah, but thank you so much, Bryan, for joining us today despite my technical difficulties and deficits here. It was really great having you on the show. I’m Bridget at Bridget Kromhout.

Matty: I’m Matt at Matt Stratton.

Bridget: We’re Arrested DevOps, and remember There’s always DevOps in the banana stand.