Matty: [00:00:03] It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m your co-host, Matt Stratton, @MattStratton on Twitter. Arrested DevOps is brought to you by 10th Magnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. TenthMagnitude empowers businesses to better collaborate across teams and achieve IT transformation using the cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/tenthmagnitude. This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools such as Chef, Docker, and AWS so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. So last October, I sat down with Chef’s Dr. Nicole Forsgren at the Chef Community Summit And we talked about the, the last 2 often neglected parts of CAMS, measurement and sharing. And now here for your listening enjoyment is that interview.
[00:01:34] I’m here with Nicole Forsgren of Chef, and we’re going to be talking tonight about parts of the CAMS theory that we haven’t talked about on the show before. So in DevOps, we talk about culture, automation, measurement, and sharing, and tonight we’re going to talk a lot about measurement and sharing. So Nicole, thanks for being on the show.
Nicole: Absolutely. Thanks for having me.
Matty: Can you give our listeners a little background on yourself? I know that Adam Jacob likes to say you have a doctorate in charts and graphs, but maybe you can talk a little bit about what you’ve done.
Nicole: That’s always a fun one. So my background, like back in the day, I actually started out doing some programming, some system administration. I actually started on an AS/400. So I was in green screen managing those systems, laying cable in 4-inch heels is one of my favorite jokes, which is actually true. And then I kind of took a sidetrack and went and got a PhD in management information systems, which is technology and business, because I really liked to take a look at how technology use plays into business outcomes and business measures. And also, what happens when you look at technology in context? So how do organizational and cultural things impact knowledge use, knowledge adoption? So really CAMS, honestly. And the interesting thing is that my focus since day one, since my dissertation, has always been among tech professionals. It was always system administrators, sometimes developers, but it’s always been that subset. And it was actually really interesting because when I came to Chef and when that press release kind of hit, I got this really excited phone call from my dissertation advisor saying like, you were right this whole time. You were right. Because back in ’07, I was trying to convince my dissertation committee that, you know, sysadmins were a thing and not a thing, but like that it was a, you know, a population whose work was different and was interesting and was relevant and was not just, you know, this movement that we’re calling DevOps now was a little different. They weren’t just programmers like anyone else. And it wasn’t just like other people using computers to check their internet, you know, check email on the internet. It really was this interesting intersection. And so, you know, it was this great conversation of her saying, you were right, you You were totally right. So, it’s been an exciting journey.
Matty: [00:04:11] Awesome. So, one of the things when we think, I guess for me, when I think about measurement and why measurement matters when we talk about DevOps and about the work that we do is a matter of, it’s beyond just like what you might think of as monitoring, right? Which is, are my things working? Like, to me, measurement is not, I’ve got Nagios running and I get paged if, you know, my disk is 80% full. I mean, that is measuring, but to me, I think about it as a factor of how do I gauge literally that the needle’s moving, right? Because I’m making change. I’m trying to— part of the point of DevOps, right, is to shorten our feedback loop and determine, be able to shorten cycle time and understand that. So, what are some of the things that you think are, if I’m getting started, I’m saying I’m doing this transformation, I’m trying to think about things, What are some beginning points to think about measuring, like what are things that matter if I’m coming out from nowhere?
Nicole: So there are a lot of things to think about when we think about measurement. And you’re right, it’s not just monitoring. Although, you know what I mean, that’s good, right? You kinda wanna know when the world’s crashing down and everything’s on fire. But when you’re starting on your transformation, measurements are critical and they’re critical for a few reasons. One is that if you’re going to be undergoing this transformation, it’s going to be important for you to understand where you’ve been and where you’re going. So it’s interesting, we talked about CAMS, right? Measurement and sharing. Sharing is, is a big piece of this in terms of measurement because you want to be able to communicate to yourself and to your team and to outside of your team where it is you are, where it is you’ve been, and where it is you want to go. So, you know, sometimes you want to measure where you are and where you’ve been and where you want to go so that you can communicate that. It helps with alignment. It helps with communication. It helps you clearly communicate and understand where you are and where you’ve been because sometimes we’re limited in our vocabulary. You know, we’ll talk about even lead time, and lead time can mean 2 or 3 different things. Is it code commit to code Code commit to code deploy? Is it ideation to code deploy? Is it deploy to deliver? Every team will have something that’s a little bit different for lead time, or some teams use the word cycle time. And sometimes a dev team uses one word like lead time and an ops team uses something called lead time or a QA team uses that, basically the same term. But at least if you’re measuring it, it— well, first of all, if you’re starting the transformation, measure it, because then you’ve got a solid baseline. You know where you’re starting. And I tell all teams this that I’ve worked with, because again, I came from industry and then I started doing research. And so for some background, a lot of the research that I did, I tried to structure basically as almost a consulting arrangement because I figured if it looked like a consulting arrangement, I could convince myself that it was relevant to industry and I could still publish it. So it looked like I did a lot of consulting and then I just also published it. I designed it as a rigorous study. But if you’ve got a baseline, even if it’s bad, capture it. Even if we think it’s ugly, even if we think it’s bad, you may have to start over. If you’ve been gaming the system forever, go, you know, maybe rename it so it’s something new. But if you know where you’ve been, or if you know where you are, then you can improve and you can move on, you know, in the future. Um, and that’s a fantastic thing. And if you’re measuring it accurately, and if you can write down what it is and ex— and explain how you’re capturing it and how you’re measuring it You should name it something that’s meaningful to you, that’s as close to what it is. But if I want to, I can call it Steve, or I can call it Sue. But if I’m calling it lead time and you’re measuring something a little bit different and you call it lead time, and if our measurements aren’t quite lining up, but if I tell you that I’m measuring a thing and this is how I measure it, and you tell me you’re measuring a thing and this is how you measure it, suddenly we can compare notes on what we’re measuring and how we’re measuring, and then I can now understand if our numbers are a little bit different, where they’re different and how they’re different and why they’re different. And it’s no longer a fight. We can have a disagreement about why I should get to call mine lead time and you should have to call yours cycle time. But measurement’s important because it increases visibility, it increases communication, it increases alignment, it helps you understand baselining and progress. It helps you understand if the changes that you’re making in anything, in tooling, in practice, in process, in culture, in team structure, if they’re making a difference for good or for ill.
Matty: [00:09:22] And so you can react to it.
Nicole: So you can react. So you can make changes. And then as your measurement gets more sophisticated, as you start measuring more value— sorry, more variables, more items. You know, you might just start with a handful and you’re capturing a few of them. You might put them in some basic spreadsheet applications so you can track them over time. You might get more sophisticated and start running exploratory analyses, just basic correlations. You know, you, you might start with visualizations, but then you might start running correlations, but you can move on to predictive analyses and saying, based on previous history, if I turn this knob, it will likely result in this getting more efficient. It will likely result in these types of things moving along the chain. And some of the best, most transformative, innovative companies are doing this.
Matty: [00:10:23] So would you say— to me, it seems like kind of taking a lot of the same approach that I would take if I was a product owner. And I’m trying to evaluate, right? Again, I’m looking at this and I’m doing, like you said, exploratory, I’m doing testing to say, did I get a result out of that? And so being able to identify the change that I’m making, what is, I mean, I’m gonna be kind of silly, like what’s a conversion? Like if I wanted to make that analogy and say like, okay, I’m trying to release software in a better way, how do I know that that’s better? And I’m gonna try this thing, but I need to be able to look back and say, did I move that needle? You know, what’s important to me? And I think one of the things that’s, that’s potentially challenging too is, is you talked about alignment, is understanding what is actually really important. You know, coming back to that, you can, you know, we talk about like the different groups having their different drivers, but really the ultimate driver is selling shoes, right? You know, so it’s like getting the understanding of what I’m measuring is how does that roll up into Yeah, because does it, you know, the job of the sysadmin is really not uptime, right? You know, I mean, the job of the sysadmin is to facilitate the business to sell the shoes. And so who cares if I’ve got 100% uptime if that’s not actually driving the thing? So I think that can be hard for people to figure out, like, what are the things that are interesting to measure that drive the business goal?
Nicole: [00:11:49] It’s interesting you say that because One thing that can be really, really helpful is to treat your work as a real value driver for the business, even if it’s not what that business does. So we think about, you know, managing customer churn and capturing those metrics and understanding what you do. Think about what you do in your own company and managing that as if it was the product. For the company and then communicating that to your customer. You might be providing infrastructure and your customer is someone else in the company. I think it was Amazon that had, you know, they were providing their own infrastructure and they needed to do it in such a way that it was so competitive for themselves. And now we have Amazon EC2. I don’t think that was necessarily the very original first game plan, but But when you treat your internal services as if they were a product, then you start optimizing and you start— like I said, training is a product. Training is a true value proposition for a company. But how? So then take a small step back. If something’s a true value proposition and you need to communicate the value of that, you need to have the metrics to be able to do that. You need to be able to talk about churn or uptime or something, think about creating your own little marketing deck, which I realize is painful, right? That’s like smarmy and marketingy and stuff. But even if you don’t want to have like massive polish and marketing around it, having basic metrics about how you will— your goals are to improve the value, what you’re doing now quarter after quarter or year after year, and showing how you’re providing that to your customers, to the business, That’s huge. That’s absolutely huge. So, so where to start? Think of who your customers are. For some companies, this is easy because they are in the business of, you know, if you’re a web ops shop, I mean, clear, this is pretty straightforward. This is a straightforward exercise because you deliver that. But so place yourself in your company. And first of all, I would think about what your company does. What does your company do? How do they make money? What is the top of your executive’s mind, your marketing team’s mind? How do— what, you know, what are their goals? And then think about how you directly support those goals. So if you sell shoes and you’re on the infrastructure team, yeah, I mean, your goal is to limit downtime, but your goal is to limit downtime because you need to provide the infrastructure to sell shoes. Maybe you don’t sell them through the internet. Maybe you only sell them business to business, but it’s because you have to provide the infrastructure for EDI. You have to provide some sort of infrastructure for at least the FedEx order to make it through. So how are you going to provide some type of service to enable your core business function? Think about that, and then baseline— I honestly, I would start with a handful of metrics. And it’s interesting. So, I’ve worked on— I was on a team of people that started with DevOps Enterprise Summit. Gene Kim chairs that conference. And he pulled together a DevOps Enterprise Forum where he pulled together a handful of some of the experts in the space and he let us self-select into groups. I ended up on the metrics group. And we really came together and the very first day it was interesting because we’re like, we’re going to come together and we’re going to think of the most essential metrics for everyone. And I’m pretty sure we pulled in at 155 and we’re like, yeah, what? No, we were going to come up with top 10. We’re like, no. Okay. So we, we finally boiled it down. We tried to come up with 3 areas: internal facing, external facing, and maybe culture, because culture ends up being a big indication of when things are going well or when things are going to break in big ways. And by the way, that white paper was released at DevOps Enterprise Summit, so.
Matty: [00:16:02] We’ll put a link to that in the show notes.
Nicole: Awesome.
Matty: So you can check it out.
Nicole: And again, so some of these, as I’ve mentioned, you know, are going to be context-specific. What’s important to your company? But those are 3 good categories. What’s external? And it’s funny, we dance around this in the white paper because so much of it’s gonna be external. Are you a small startup? It’s gonna be totally different than if you’re large and publicly traded and you deal with PCI and SOX compliance and all the things. So we basically whitewashed that and we’re like, just deal with External metrics. But internal metrics. What’s important to you? Are you primarily a software development team? Are you primarily a B2B business? Are you a retailer? Think about what is totally relevant for your business. Read the messaging. Look at the mission statement. And figure out how you contribute to that and try to come up with Start with 3. Honestly, start with 3. And think about how you can measure them in honest ways. And then, you know, maybe there are ways to game it, but start, right? MVP. Minimum viable product.
Matty: [00:17:06] You can always, you can always add.
Nicole: Exactly. You can always iterate. You can always iterate. You can always provide something because measurements provide insight. Always also remember that that measurements also drive some behavioral change. So at some point they might be gained, but let everyone know because they’ll drive some behavioral change from your own team, but they’ll also drive some behavioral change at the business. If you show them that you’re providing value, if you’re communicating your value to your customers, and you’re showing improvement with the resources you have been entrusted with, you may then be entrusted with more resources later, or you may be entrusted with a seat at the table when those resource decisions are being made. So these types of metrics have a lot of power in communication, in alignment, in a lot of different things.
Matty: [00:18:06] I think one thing that I find interesting is, and we don’t have to go too deep on it, if you have an opinion on this, I’d love to hear it, but, you know, Gartner’s idea of this bimodal IT, I see a lot of stuff with that. If you’re not familiar, if you’re listening, so bimodal IT is this idea that there are 2 parts of IT. I’m going to, in kind of a snarky way, summarize what I interpret bimodal IT as. Part of IT is doing all the cool stuff and they’re the cool kids. They’re the part of the company that gets to do the DevOps and write the stuff and do the Docker and all this stuff. Then the other half of IT is the legacy IT that’s running the mail servers and the file systems and the desktop. And doing all the old ITIL and doing the old way. I could be misrepresenting this, but my understanding of bimodal IT from Gartner is they’re saying that that is a thing that was an observation and is a necessity, that not everybody can move this way. What I’ve found, thinking to your point about identifying the metrics and the things that matter, I feel like So if we were to presume, A, that bimodal IT is truly true, and we have these 2 parts of IT in our company, all the stuff we’ve been talking about is going to absolutely resonate with the cool kids, right? And then the people who feel like they’re on the pure, like, keep the lights on for the internal users, we’re not supporting applications, we’re supporting mail or printers or desktops, they’re going to say, well, none of this matters. That’s not the thing. But I do believe that that’s untrue. I think you can take the exact thing you said and say the metrics of your internal, quote unquote, traditional IT folks should do that. And I think because of not thinking about that in that alignment is why there’s a very sometimes antagonistic feel to legacy IT towards the business, which is the whole, you know, you go on to, we love to like, you know, kind of, make fun of, uh, the, the subreddit, the sysadmin subreddit, because there’s a lot of people complaining. But it’s a lot of complaining about like, oh, head of marketing needs a MacBook and what the hell am I supposed to do with that, blah blah blah blah blah. And I have standards, damn it. And we have standards for a reason and everything. But the, the thing is, looking at what these quote measurements are that these, these shops are talking about, they’re really not about driving the value to the business. They’re about saying, well, we’re going to make sure that we don’t have You know, that it’s basically making it easier for me to do my job and do my thing. And I think it’s— I think you can be on the end user, internal end user support side of IT and still look at how what you’re doing is really driving the business’s ability to sell shoes and understand that’s why you’re there. Oh, I don’t know. That’s so—
Nicole: [00:20:55] yeah. So There’s a lot there. So in terms of bimodal IT, I think it’s a thing. Right now it’s definitely around, particularly as companies are starting to undergo the transformation because it’s just not really feasible to undergo mass transformation all at once. I don’t really think it’s sustainable though. I mean, it’s all the data, that we tend to look at as I’ve collected data over, as lead on the 2014 State of DevOps, I’ve collected more data. I’ve got data from 20,000 respondents. ITIL suggests that you can take trade-offs as you do trade-offs in throughput and stability, which might apply to a bimodal IT type of a context. That just, it never shows up anywhere in the data. You just don’t see it showing up in significant ways. It’s like you’re either doing it or you’re not. Like you’re kind of all fast or all slow or all in the middle. So I just don’t see it being, I don’t see it as in bimodal IT being a long-term sustainable plan. If it is, The industry I would be watching, the industry that I am watching in super, super interesting ways is like stock trading, stock market, because they’ve got those legacy systems that like you just don’t touch. Or maybe air, because SABRE and old air traffic control, I’m keeping my eye on those because some of those systems everyone’s just terrified to touch in any kind of way.
Matty: [00:22:41] I sit and I remember when I worked for a very large bank that rhymes with A.P. Schmorgon Mace. But one time I went to one of our, downtown Chicago, and they said to see one of the computer operators down there, and we’re down in the basement, and pointed to a machine that was sitting there, and he said, you ever seen one of those before? And I was like, no. He goes, nobody has. We have no idea what that does. But we’re not gonna turn it off. And again, it’s a bank, who knows? And we’ve all been there, and I think that’s, And I also think that a lot of the places, the pieces that tend to be, when you think bimodal, that’s also the places that are ripe for disruption to not necessarily needing to be there. You know what I mean? Like, when we think about those services that tend to be part of that, besides some of the stuff, as you’re talking about the legacy systems, like you said, that are running those that can’t move fast. But those are, I think, edge cases, but the places where people are not moving in that speed, where we are talking about that again are the things like managing end-user services. And that’s just becoming more and more of just commoditized. And why are we— absolutely, why am I doing that for my company anymore? You know, and again, there are plenty of people that would listen and say, no, you have no idea, we’re, you know, regulated, and we have to do this and blah, blah, blah.
Nicole: [00:24:00] And I think that I’m still seeing those transformations happen. You know, I’m seeing those transformations happen still. And in terms of metrics and communicating it to the business, if there’s a solid reason why that’s the way you’re doing the thing, then actually that’s a stronger reason to collect your metrics and communicate it up to the business.
Matty: Because otherwise they’re just saying, why aren’t we just using Gmail?
Nicole: Yeah, exactly. Exactly. And the really interesting thing is, you know, you say that for traditional IT and they don’t want to hear this. It’s been pretty fascinating and really interesting the last 3 or 4 years. Carolyn Rowland and I have actually done a workshop, a half-day workshop at the Leesa conference on understanding strategy and how to communicate your value to the business, how to be a strategic partner with the business. And there are some people that have sat in on that workshop several times, and it’s sort of understanding how to communicate your value, how to capture brief metrics So that you can communicate what it is that you do so that they understand. Because otherwise, you know, they’re going to hear about this amazing new Gmail thing.
Matty: [00:25:13] Or they’re going to say, why aren’t you doing the DevOps?
Nicole: Why aren’t you doing the DevOps? Why aren’t we doing the Gmail? Why aren’t we doing the thing? You know, there’s this great new service. Well, if there is this great new service, then you should probably be the one architecting it because you understand best and not, not somebody who got a degree in like philosophy or marketing. Quite honestly, unless they got their degree in philosophy or marketing and have been doing IT for the last several years and running it in the company. So do you want to be the one helping them, you know, orchestrate that solution and architect it and navigate that transformation and then run it? Or do you want the rug just ripped out from under you? I mean, that’s the perfect time to be capturing metrics and communicating that value and understanding how to do that.
Matty: There’s a— when I think about the sharing, so there’s a talk that I’ve given called The 5 Love Languages of DevOps. And it’s about— and it applies, I think, to here too. I think one of the things with sharing that’s really important, and I think can be really challenging, is understanding that what matters to you and the language you speak, so your DevOps love language or whatever, is going to be different than other people’s. I think that’s a big challenge. Your example was a great one there when you’re thinking about that, because if I’m a sysadmin and I’m running Exchange, like, I know in my heart of it, like, I think about whatever the security piece of it is, or I know I have all this information, so it seems logical to me. Then I go talk to the CFO and I’m like, we could do blah blah blah blah, and he’s like, but I don’t understand. And it’s not because it’s not technical, but the things that are important to that person, both because of their role, their personality, the way they communicate, it will fall potentially on deaf ears and it just looks like you’re just complaining. And I, I think about challenges. I remember a specific story when I was at Apartments.com. We had a scenario where one of our web services was throwing something crazy like 14,000 errors a minute. And they were fake errors. They weren’t real. It was just like there was a bug and it was flooding our logs. And the sysadmin on my team who was running that was just continually trying to, you know, get through the— get this bug fixed, saying, you know, get it into the sprint, getting the, you know, we’re like, we have to fix this. It was, you know, at the time, now I think about it, I was kind of like, oh geez, why didn’t product understand? You know, they only care about features and blah, blah, blah. But the reality, as I started to understand because of how it got resolved, which I’ll tell you in a second, was it just wasn’t being communicated well. And it wasn’t— it was because it was still sounding like, oh, the sysadmin’s complaining because of this and whatever. And actually the way that it potentially got, really got solved, and this goes back to sharing, is when we started putting some dashboards up on the wall. We put some Splunk dashboards up and there was one that had a dial all the way in the red, showed 14,000 errors a minute, and the GM walked past it and said, what the hell is this? And but the thing is, that’s kind of a funny story. That is not a good way to sell, right? Like, that’s the thing. It shouldn’t be because the big boss comes down and, yeah, that didn’t really— I mean, it solved the problem, but it may have helped the product owner as well see that and say, oh, that’s really a big thing. But being able to communicate and say, actually, you know, because this is admin, I’m like, well, there’s a problem because my logs are flooded, so I don’t know when something’s real or not. And Product owner might be like, well, but have we had an outage? Have we had a problem? So, who cares if you know that? You know, it’s like we have to figure out, like, what’s important to them and say, how do we communicate that? And so, that’s a really hard part of sharing is I think when we did our episode on DevOps culture change, which you can listen to at arresteddevops.com/devops-culture-change, And one of the things Bill Joyce said was, he said, if you ever find yourself saying, this is crystal clear to me, why aren’t they seeing it? Then it’s more about you than it’s about them. And I think, like you said, you know, if you’re wanting to do that, you want to be involved, you have to have the metrics, but you have to know. So the measurement is having the metrics, then the sharing is how do I communicate those metrics to the decision maker or the person I’m talking to, because I will read those metrics. They’ll matter to me differently than they might matter to you for lots of different reasons.
Nicole: [00:29:23] And I love your example of having a dashboard. And so, you know, think back to a minute where I said think of, think of 3 measures that are meaningful for you or meaningful to your team. And 3 is a nice number because it’s 3 or 4, right? It’s manageable. You can grasp it. You can keep it in your head. It also fits really, really, really well on a reasonably priced monitor that you can throw up on a wall and you can keep it up on a wall. You can do it in a spreadsheet program, right? Throw it in Excel. By the time you have metrics over 3 periods, whether it’s 3 months or 3 weeks or something that’s meaningful, The first one, let’s say your baseline in one of them was just horrendous, but you got better. You keep that up on the wall and your GM or your product manager or that CFO walks by and he sees progress. He may have gotten that kick in the gut from that first one that just hurt, right? What has that team been doing? I know we put them down in the corner in the basement because that’s where we all hang out, right? Because we always have offices in the basement. But what are they do down there? Apparently nothing, because that first time they captured metrics, it was horrendous, and then they stopped listening. But then you’ve got pictures, and you can show, you know, like whether it’s a bar chart or a line graph or something, and you show progress going up.
Matty: [00:30:53] The middle— the arrow is going up and to the right.
Nicole: The arrow starts going up, right? Or if it’s errors or downtime and it’s something going down, right, use like big letters and not very many words, so it’s easy to grasp from across a room. Suddenly you’ve got something happening. Or you can show a big chart that you send out in like your quarterly email of status.
Matty: I really like the idea of when you’re talking about, about limiting it, because I think about I used to have to generate a report for, you know, our website performance and apartments, and it was— there was so much data on this thing that, and it was, you know, and it was sent out to, you know, the board, you know, all these people. It’s like, no one’s, no one cares about most of this. Well, I think there’s like 2 or 3 things, right?
Nicole: Your team is gonna want 37 metrics.
Matty: Exactly right.
Nicole: Someday when you get there. Yeah, but your board or exec, your executive team, give them 3, give them 4. Give them the good ones, or give them the ones that you know they really, really care about, or the ones that are very important to you. Give them the 2 or 3 that you know they care about that show improvement, and then also maybe include the one that does not have improvement. And why doesn’t it have improvement? I really need this resource. You’re right, we’ve been trying, we’ve been doing our best, we’ve optimized as much as we can with tooling, but I really have to have this headcount.
Matty: [00:32:23] And then you have the data to back it up.
Nicole: And I have all of the data to back it up. And you’re right, we do have improvement in these other areas. You’re right, we were able to squeeze as much as we can, but I cannot get this other metric to improve. I guess this would be the case. I still did as much as I could without, as you asked. And you see some improvement in this other metric, but I really think I’m limited by bandwidth because I refuse to work my team Right. Insane hours. And here’s my evidence to show that I really think I’m limited. And here supports my business case.
Matty: So that actually ties into with the, you know, I refuse to work my team so hard and everything. So you talked earlier about when you said, you know, you kind of grouped it into the 3, your internal metrics or external metrics, and you said the culture metrics. So I’m— because that’s one of the things that I’ve run into when I’ve talked to folks about enterprise DevOps or DevOps in the enterprise, and a lot of people will sit there and say, like, okay, well, if you’re doing enterprise DevOps, it’s CAMS. There’s no culture. There’s no— and because he said it, I’ll go ahead and make this comment. Bernard Golden, who was a wonderful guy, but this is one thing he said often that that bothers me says culture is for yogurt. And but one of the things is they say you can’t measure culture. So it’s not a metric, it’s, it’s a squishy thing. So I’m very curious as to— I have some— we talked again with that episode with Bill, we talked about ways of, of measuring culture. But I’m curious about from what, what y’all came up with, like, so how do we do that?
Nicole: [00:34:03] There are several things that you can capture here. Culture is sort of squishy, and it’s It’s funny because sometimes people accuse me of just being the squishy culture girl. Although I do have a background in log analysis. I came out of hardware. But I care about culture and I get defensive about culture because it matters. Quite often when we measure it, if it— so when to measure it, measure it periodically. If it tanks for no expected reason, go check out your team. If you don’t, by the way, it tends to be a leading indicator for like tooling is— your tooling is about to fall apart. So culture is important. So AMS, not a great idea. Um, so if you want to stay completely away from any squishiness at all, you can proxy. So proxy is a fancy word for like how to measure a thing you can’t measure directly, because it’s not like temperature. You can’t take temperature of culture, right? So we need to proxy it. We need to measure it with something else. You can sometimes go to like HR databases and you can measure indirectly proxy with things like attrition rates. If you want to get real creative, sometimes you can look at things like HR literature has looked at things like access to natural light, which tends to screw us over again because of the aforementioned corner in the basement. Work hours. So, overwork tends to not be a good thing. Traditional work hours, so first shift, tends to be a stronger indicator of good culture. But that’s not— those are like second or third removed. They aren’t great. However, we did, in the 2014 State of DevOps Report, we did come up with a measure that is based on some research done by Ron Westrum. And it’s covered pretty extensively in that report. It was also used in the 2015 report. So, and it’s included in that white paper that I mentioned before. So if anyone wants to use that, it’s a 6-item measure. So 6 items, 6 questions. You measure it on a Likert scale, 1 to 7, where 1 is strongly disagree, 7 is strongly agree, 4 is neutral.
Matty: [00:36:25] And this is your— you’re measuring this by collecting information from your team and your employees?
Nicole: Yes, so it’s a survey, and people roll their eyes at surveys, and I’m with you, right? A bad survey is bad. This is, uh, so I also have kind of a background in psychometrics, so this is science, right? So this has been run through at this point over, you know, a sample of 9,000 and then a sample of 5,000. This has been shown to be statistically valid, statistically reliable. It shows good measures of statistical validity, reliability. This has good— exhibits good measures of statistical validity through convergent validity, discriminant validity, reliability. This is a very, very good measure. So you can run this through your teams periodically. It’s 6 questions, and the questions are, again, strongly disagree to strongly agree. On my team, information is actively sought. On my team, failures are learning opportunities, and messengers of them are not punished. On my team, responsibilities are shared. On my team, cross-functional collaboration is encouraged and rewarded. On my team, failure causes inquiry. On my team, new ideas are welcomed.
Matty: [00:37:43] This sounds familiar. I think I took this survey at Chef.
Nicole: Yes. Yes. Jez was hired shortly thereafter and he used this. So, and if you needed to tweak it just a teeny bit instead of like, on my team, I mean, you could pull that off. But it’s interesting because it pulls into so many of the DevOps principles that we want. The really cool thing about this is once you have those measures, because of all the background statistical things, the things, stuff and things, you can, it’s a latent construct, which is another way of saying, like, you can’t just take someone’s temperature. This gives you someone’s Westrum organizational performance temperature. You can average these together, they can be combined, in a statistical sense. So you can come up with a person’s one cultural measure. You can combine those from a team. And I know stats people out there are going to lose their mind over about an average of an average, but it’s because combining them is basically creating a construct. So you can create that and then look at how the team is doing overall. So you can combine that and come up with a pretty good measure to see how people are doing. And if everyone’s doing pretty well overall on those items, that’s good. If you happen to notice that one in particular across the team tends to suck, that you can take a look at that also. So for example, one of them is on my team, failure causes inquiry. If everyone scores really low on that, take a look at your practices on your team. Are you doing blameless postmortems? Are failures okay? And if they’re not okay, is it because like your backup resilient systems suck? Like have a hack day around those things. Come up with stupid, silly little prizes for making a thing fail so that it’s a funny thing and then pull everyone together to build in like the resiliency. Try to find ways to combat that and then see what happens the next time you take that instrument.
Matty: [00:39:51] Absolutely.
Nicole: And it’s that one in particular has been shown to be very, like a very, very good indicator predictive of IT performance and predictive of organizational performance. So IT performance delivery, both in throughput speed and reliable ways and organizational performance contributing to the bottom line. And it’s 6 questions. That’s fast. And it’s good stats, right? Those are good questions. That’s a good psychometric.
Matty: That’s awesome. That’s great. And I can see that being very useful. So I’m happy that we’ll be sharing that.
Nicole: And it’s open now.
Matty: Yeah.
Nicole: That’s all open.
Matty: It’s all there. So links in the show notes for that. So I think, yeah, that we’re wrapping up. But thank you very much for joining us. Is there anything, any final thoughts you want to give folks around measurement sharing? The DevOps, the how to be awesome?
Nicole: No, just keep on keeping on and measurement is good. And don’t— if one measurement doesn’t really work, iterate, right? It’s always just an improvement kata. So if something just doesn’t really work or if it’s working for a while and it stops working, it’s because your processes have also grown and improved. So just keep working at it.
Matty: [00:41:05] Great. Well, thanks so much for taking the time to be on our show. We really appreciate it.
Nicole: Great. Thanks so much.
Matty: All right. I really enjoyed doing that interview with Nicole, and listeners, I apologize that it’s taken me several months to get around to editing it and publishing it, but I think you’ll all agree that it was worth the wait. So a few housekeeping things. Don’t forget, we have a newsletter, arresteddevops.com/bananastand. It’s the best way to know about our upcoming podcast episodes and cool news with DevOps. Thanks to our sponsors. Be sure to visit them at arresteddevops.com/10thmagnitude. And arresteddevops.com/datadog. And special thanks to Nicole for joining me for that interview. And loyal listeners, or disloyal listeners for that matter, if you enjoy Arrested DevOps or just listen to it sometimes, one way or another, we sure would appreciate it if you would visit arresteddevops.com/itunes and leave us a review in the iTunes Store. And we would love to know what you thought of this episode. Please leave us comments at arresteddevops.com/measurementandsharing. You can check us out @ArrestedDevOps on Twitter, and we’re always happy to get your input, ideas, or feedback via that email thing at shows@arresteddevops.com. Let us know any ideas you might have for future episodes. I’m Matt, @MattStratton. This is Arrested DevOps, and remember, there is always DevOps in the banana stand.
