ITIL Eye for the DevOps Folks with Steven Boyd

Trevor: [00:00:00] I mean, DevOps can be a swear word depending on who you talk to.

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and business for maximum DevOps awesomeness. In this episode, we’ll be talking about the ITIL I for the DevOps folks. What is this ITIL thing all about, and can it complement DevOps? I’m your co-host, Matt Stratton, @MattStratton on Twitter. Twitter.

Trevor: And I’m your co-host Trevor Hess, @TrevorGHess on Twitter.

Matty: This episode is sponsored by VictorOps. From initial alarm to final retrospective, the mission at VictorOps is to make on-call suck less. Easily integrate with your existing monitoring systems and manage on-call schedules with rules for intelligent routing. In the live infrastructure timeline, get real-time context and see annotated alarms with resolution documentation. And when you’re in the firefight, collaboratively troubleshoot using native chat or bidirectional integrations with your favorite chat clients. Visit arresteddevops.com/victorops and sign up for a 14-day free trial to see how they’re making on-call suck less.

Trevor: [00:01:12] This episode is also brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools, such as Chef, Docker, and AWS, so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a 14-day trial at arresteddevops.com/datadog. Joining us tonight for our chat about ITIL is Steven Boyd. Thanks for making the time to chat with us, Steven. Can you tell us a little bit about your background and what you do?

Steven: I’ll— I’m, uh, Steven Boyd. I’m a certified ITIL expert and a federal employee with the U.S. Patent and Trademark Office. I’m the service desk problem manager as well as I manage the major incident process, as well as help the organization find alignment of our processes and procedures with the ITIL framework.

Matty: So, a couple of the reasons we wanted to have this episode is there’s definitely, I think, a lot of confusion or misunderstanding within the DevOps community about what ITIL is. And I think the first thing that we really need to be clear about is how to pronounce it. So, is it ITIL or ITIL? I usually actually say ITIL, but then nobody knows what I’m talking about.

Steven: [00:02:33] ITIL is pretty much the common, but you’ll hear, you’ll hear people actually say idle.

Matty: Yeah, sounds like idle. Yeah, idle. I’ve never heard idle.

Steven: I hear that a lot within the DoD community.

Trevor: Idle. It’s very clear. So let me be honest and start first of all by saying, well, what is ITIL? I haven’t been exposed to it yet.

Steven: ITIL is essentially decades of best practices related to IT service management, originally formed within the UK. It was part of, part of a government organization to document the best practices that they had found over time to reduce costs and improve efficiencies. And since then, the multiple different versions and the latest 2011 version is essentially that, a reorganization or repackaging of best practices that help you align your service management by providing a generalized framework. And it’s a descriptive framework, not necessarily a prescriptive.

Matty: [00:03:36] Gotcha, absolutely. And then a little background. So Trevor pointed out that he’s never even heard of it before. Well, that’s not true, but he has no experience with it, which is fine, he’s a dev. I’ve got a little background myself, so I am ITIL V3 Foundation Certified whatever, which means I took a test and I went to a workshop. But I actually do have experience implementing ITIL and using it. Actually, what I found interesting was when I went to actually learn it, I sat there and said, oh, that’s what this is. That’s that thing we’ve been doing at Bank One the whole time I worked there. I just didn’t know that we didn’t invent it. So I have a lot of exposure with it, and I consider myself pretty open-minded towards it. I know there are a lot of folks, and this is true, I think, of any type of practice and any side. Geez, look what I just did, and I said any side. Any type of area. Folks within certain cultural movements say, oh, this way was awful. Whatever came before ITIL was terrible to people who do ITIL, and DevOps hates this, and everybody hates the Agile people. I actually have a theory which we may expound upon later, which is that DevOps is really the natural evolution or merging of Agile and ITIL, which the theory that I have on that, and people can correct me if I’m wrong, is we did ITIL, we look at what ITIL has, and it doesn’t really accommodate very fast software-based organizations. It’s really good for service management, and that’s what it’s built for, because at the time when this stuff was happening, we weren’t in this world where every company is a software company. Where we’re in this, and we’ll dig into that in a sec. Let’s assume that I’m right about that for a second.

Trevor: [00:05:13] Assume.

Matty: Then we did Agile, and in Agile we said, okay, this is rad, so let’s get the product people and the devs and the testers and all these people working cross-functional. What about the ops people? Oh, well, they’re doing their thing over there. When I think about DevOps, I’m like, we actually really, in a way, it is kind of culturally a lot of the same things, and I’ve seen a lot of stuff that comes out of the DevOps community. I’m like, that sounds a lot like what we have tried to do in ITIL. Or, this is a lot— a lot of times people say, that sounds a lot like Agile. I think one of the things that I’m hoping is going to be really illuminating to listeners who don’t have an experience or a background on what ITIL is all about is to see how there are ways that we can build on the shoulders of giants and understand where it fits, what we can learn from it, and where we can apply it. That being said, when you think about it, Steven, I know this is a very generalized statement, but what are some of the benefits that utilizing this framework would provide for an organization? What do we get out of it?

Steven: [00:06:16] Well, that’s an excellent question, and it’s something that we started to discuss at DevOps DC, where I broached the topic. Now, one of the good things to understand to begin with how ITIL and the ITIL framework is going to benefit the organization is to actually understand the certification process. ITIL Foundations provides you the vocabulary as well as the overview of the framework. Now, the certifications beyond that are intermediate certificates, and either you have the lifecycle track or the capabilities track, and they’re focused on the role in which the person being educated and certified is going to be able to apply that information. So the lifecycle is pretty much focused, you know, within the lifecycle, which is service strategy, service design, service transition, service operations, and continual service improvement. And in the capability stream, you have 4 different groupings that take the natural alignment of different processes within the life cycles and looks at it at a managerial standpoint on how to accomplish those type of alignments. We have to step back a little and define when we’re discussing the ITIL framework, really, what are we talking about? Many times I find that when someone refers to ITIL, they’re really specifically referring to the change management process within ITIL. They’re not discussing service strategy. They are familiar with service design and service transition, and it’s within service transition where change management resides in those portions.

Matty: [00:08:00] I think a big part of the reason for that— not to interrupt, but to your point, I’ve seen that for sure.

Steven: Because that’s how it was sold.

Matty: Well, it’s how it was sold, but also, for a lot of practitioners, that’s what they really interacted with, right? So when I talk to people, especially folks who are in an operational role, or people who see it and they’re like, oh, ITIL’s about CCRBs, and about, you know, or, you know, about, oh, I gotta have this change management board, and I have to submit a change, you know, my whole life, you know, has been about submitting change control requests that get rejected, nobody knows what’s going on. And there’s all these other pieces of it that, depending upon your role in the organization, you may or may not directly interact with. With, but everybody seems to, in one way or another, especially in ops, interact with change management. So I could see how that is believed to be part and parcel of the whole thing.

Steven: Exactly. And then that’s where I think the misconception begins, because a lot of times when a vendor or someone is focused on implementing ITIL, really what they’re talking about is getting control of their change management process and the documentation of the records. So a lot of people spend a great deal of time fixating— or fixating is a negative word— but focusing on the change management process within service transition and trying to get that correct. A lot of— if we get to the point where we discuss service operations and the incident management, problem management process, we’ll come back because a great many of the contributing causes to your incidents and major incidents within the service desk are actually procedural mistakes or procedural misalignments or siloing of information or improper documentation of records or using the wrong record type or having an incorrect process. When we go back to talking about like the ITIL Foundations, the Intermediates, and the Expert level, because the Expert level is the Managing Across the Lifecycle certification, which looks at all of the processes and ensures that the certification of the individual encompasses the ability to actually create organizational internal processes based on the ITIL framework. The Intermediate is more about identifying which process or something is involved with that type of scenario or situation. So there are different Bloom levels as you go from foundation to intermediate and expert. The expert is actually the ability to create. Now, there’s nothing wrong with, with those people that have intermediate certifications going back and implementing ITIL, but a lot of times when there is confusion within an organization or there’s hate and discontent related to ITIL, it was normally because the application of what they were trying to do with the idol at the organization was framed as being idol, and they failed to align the different pieces. And that mismanagement of the alignment reflected negatively on idol as a whole because that’s how it was branded or labeled at the organization. And a lot of people, when they talk about mis— their misgivings about idol, they’re really talking about a specific implementation of maybe a procedure or tasks or a process at a single organization.

Matty: [00:11:16] I think you talked about how the framework is descriptive versus prescriptive. Unfortunately, if I’m someone who’s selling you a thing, it’s very hard for me to sell you something that’s not prescriptive. Sometimes I think it’s okay to be prescriptive within certain places, but there’s also— the hard part of doing anything is understanding your business. One idol does not fit all, right?

Steven: Exactly.

Matty: If you think you’re going to buy— we’ve talked before, too, about you don’t buy DevOps in a box, right? DevOps is a set of practices that have been created by web innovators and are being adopted across multiple enterprises, but there’s no— Adam Jacob from Chef gives a great talk about DevOps kung fu, which says, if you look at kung fu, There are many schools of it, but you can recognize it’s all kung fu, but they’re all slightly different. We believe the same thing about DevOps, which is, you may do it a little different than I do it, but we agree that it’s all that. I would think the same thing is that my ITIL versus your ITIL versus Trevor’s, because we’re different organizations, will be implemented maybe in different ways, but it should still be recognizable as a— I guess, as— I like to think about being outcome-focused, and would you say that’s a fair assessment of the point of a framework like this?

Steven: [00:12:36] Well, actually, that’s exactly correct. ITIL looks at it as triggers that are your inputs and looking at your outputs, and then based on your approach to accomplish your output, you’re ensuring that you’re having customer satisfaction, you’re achieving your service level agreements, you’re managing your operational level agreements to be able to enable successful outcomes.

Matty: So I want to think a little bit about the— and again, what we’ve done is we reached out. We’ve gotten quite a bit of feedback from our listeners, both whether it’s been through Reddit or through Twitter or just sort of me chatting with some folks on Slack, just to get some— because I wanted to get a pulse of what people were thinking about. So some of these questions may— and we talked during the greenrooming time already a little bit about how some of them were maybe not quite from that understanding. That being said, one of the questions was, how would an organization reconcile what— and I’m just saying is listed— the risk aversion tendencies of ITIL with the idea of fail fast, fail small approach of Agile or DevOps? Is ITIL actually even about risk aversion in the first place, first of all?

Steven: [00:13:49] No. Actually, ITIL is more about risk management and classifying the types of changes that you’re intending to implement. Once you understand the risk and you’re willing to accept it, then that’s when you’d actually create a standard change, which means it has the approval and it doesn’t need to go to your CAB authority every single time to receive approval. It’s been a standardized change. It’s not necessarily risk-averse. ITIL framework is more about ensuring that you’re documenting everything that you’re doing so that you have traceability to come back and identify which change led to the manifestation of the symptoms within service operations. With and resulted in, in an incident or maybe an exception type event. So you can have that type of correlation so you can look at contributing causes when you’re investigating a problem. So most of what you’re going to do within ITIL for the service transition and within the change process is about proper documentation so that risk can be assessed, not necessarily averted. Now The context of the question seems to be that ITIL takes too long, ITIL prevents me from doing something, but really what they mean is the change process that the organization is following is not aligned, or the procedures and activities are not effective or efficient to allow them to quickly assess the risk, approve the risk, and implement what they’re trying to do. But that’s the organization’s implementation. ITIL doesn’t say delay the approval of a request until it takes you too long to be able to implement something quickly. ITIL is about looking at your efficiencies, determining where you need your decision points, assuring your decisions are occurring and they’re being documented properly.

Matty: [00:15:44] Right, yeah, and I think that that’s the, you know, sort of being able to assess risk and where that process comes in is when you’re thinking about a culture or a software-driven organization that’s wanting to be able to automate through a lot of this testing. And that may be a thing that kind of works a little bit into creating it in the terms of a standard change, which may be, hey, it’s actually considered a standard change provided that it passes our automated compliance and security and standardized test piece. ’Cause anything, when we’re talking about being able to decrease our cycle time within moving from idea to value, it’s every place we put a human gate, you know, ’cause humans are slow. And it’s not to say that human gates aren’t required. It’s a matter of, I always look at it to say, I wanna put a human where it’s something that requires an actual human to do it, right? It’s requiring human judgment or it’s requiring an actual thing. It’s not something we can test for automatically in that case. And maybe that’s a way, if I’m getting that right, you know, it could be saying a standard change does not necessarily have to be defined as it’s standard because we wrote down that every time you do this, you’re going to push these 2 buttons and type these 5 commands and copy this file from there, it could be defined as its standard because all of these lights are green that we have all agreed mean that its risk is minimal.

Steven: [00:17:10] Exactly. Okay, the risk is acceptable.

Matty: Yeah, so unless I should have said not minimal, acceptable, because it could still be high risk but low impact, and that may be considered an acceptable risk. We talk about that a lot, right? Like the difference of just because something, you know, something could have a high chance of getting messed up, but its impact could be incredibly low. Like, you’d be like, well, it’s probable that this thing’s gonna break, but if it does, it’s gonna affect like 1/1,000th of our users for 5 seconds. Okay, maybe we’ll take that. Or there’s a very small percentage of a chance that this is gonna happen, this is gonna break, but if it does, it’s going to set the whole building on fire. So the risk on that second one is actually potentially possibly a less acceptable risk because its impact is so much higher, even though its probability is so much lower.

Steven: Yes, and the risk tolerance is specific to the organization about which level of risks are they willing to accept. Now, the process, though, has to be aligned with your incident and problem management processes so that if you start to see an increase in incidents, you can make that correlation and potentially identify the causation right, to a specific change potentially, and then communicate that information back to the cab for improved decision-making. So if you don’t have those flows, you have silos of information, and now you’re not making a real risk assessment, right?

Matty: [00:18:33] Where we talk a lot when we talk about, you know, things like blameless postmortems— I know this is getting into an area of your expertise, which is great— is what we don’t want to do, at least, you know, kind of the thought on it, and, you know, from you read Sidney Decker and everything, is people are gonna make mistakes. You’re not gonna cure people of making mistakes. What we want to do is say, how can we continue to enhance our system to improve that. If you’ve said, okay, we define that if all of these automated tests are passed, then it’s acceptable risk, but, like you said, we tie it back and say, these changes are actually causing these problems that are happening, then it means we have to reevaluate the system. It’s not even necessarily going back and saying, the code you wrote, Trevor, was awful, and you broke the thing. It’s actually saying, our system was messed up because it should have caught it. Now we know that, and we can make the standard change even safer.

Trevor: What is CAB in this context?

Steven: That’s your Change Approval Board.

Trevor: Okay, that makes sense.

Steven: Going back to the blameless postmortem, actually within the problem management process and the major incident processes that I’ve run in the past, at the end of the major incident process, we have what we’ve called in the DoD was an after-action report. That’s a pretty common term after something occurred, documenting what happened. So that’s akin to a blameless postmortem, which is your trigger if you need to into your problem management process, because that’s the first place you’re going to look. And within the problem management processes that I run and I design for the organization, it involves a preliminary analysis where you go in and you look in scope exactly what type of problem investigation are you intending to begin. And then you’d have your decision whether there’s a viable business reason to actually pursue it. So in this case, this goes back to what we were talking about, the blameless postmortem. It’s more about learning. How do, how do we make our process better so that the system does catch it the next time? Like you’re saying, Matt, how do we learn from what we’ve seen so that we don’t make the same mistake again? Hopefully we’re making different mistakes and we’re avoiding them altogether. But within ITIL, especially problem management, you’re not looking just to prevent issues. That obviously, that’s the goal, but problem management activities are also to minimize, mitigate, or otherwise reduce the impact to the users to improve the value that’s being delivered. And most times with incident management, they’re only looking at the restoration of the services or implementing a workaround or something like that. Not too many organizations actually go into developing their problem management. They tend to focus on change management because most vendors will tell them that’s where the majority of their incidents are going to be created from, is either an improperly documented change, an undocumented change, or a change was approved that shouldn’t have been. So if you have stronger change management controls, you’re able to reduce the number of incidents you’re seeing. But in a fail small, fail fast type of environment, you’re going to have those incidents. So it’s how quickly can you take that information from what you’re seeing, what’s being manifested within service operations, and feed it back into your development and design process to be able to overcome whatever you’ve just introduced.

Matty: [00:22:01] Which I think makes a lot more sense because, again, I guess if you’re focusing, like you said, the way it’s been pitched to the, well, if you add more rigor to your CAB and to your change management process, you’re going to reduce incidents, you’re actually putting in even more human requirements into that system, right? Because I think about my experiences. With cabs and stuff. And to be honest, the vast majority of them were making sure that nobody was trying to change a thing at the same time as another thing. You know, because you have people who could— we’re in a world where you can’t understand all of the ramifications of all the things that could happen. But I’d much rather be able to make my decision based upon the results of something automated that sits there and says, rather than me looking at it and you saying, hey Matt, I’m gonna do This is what I’m going to do, and I can say, I can read that and it sounds legit or sounds logical or not. If you can point me to and say, this is the change we made to the system and we did it using this automated way, we had an automated verification that went in that shows you the before and the after, whatever, if it’s something like that, without trying to dig into a tool thing, again, it’s to the point of, I can then sit there and say, actually, even though I may not understand all of the very specific internals because our systems are so complex, I can see that before you did a thing, stuff wasn’t broken. After you did the thing, stuff continues to not be broken, and you’re doing this in a repeatable way, not a, I’m gonna sit down and bash at the keyboard kind of way. So I have a great level of, you know, because otherwise all we’re doing is just make— it’s just depending on our people to be smarter and know more about even more complex, more and more complex systems.

Steven: [00:23:39] And Trevor, I misspoke earlier. The CAB is the Change Advisory Board. Not the change approval board. The advisory portion is back to what you were saying, Matt, is how much of the information that you can obtain throughout the normal operations can you feed back in to make better decisions? Not necessarily looking to stop or prevent the implementation of something, but to be able to say, this implementation caused this type of situation. We live in the environment where we don’t always get to control which security patches we’re going to have to implement. So you’ve designed something, and now they’ve gone and they’re going to address a security concern. Now that might cause you to have to perform other types of actions or go back and redesign how something is communicating with something else. So to be able to understand and track it back to you’ve implemented or pushed something, and now you need to address what it’s changed in your environment because, when you did perform the testing, for whatever reason, the testing didn’t show that you had the issue. That’s really tough within problem management, is to identify the contributing causes so that you can look at the actual correlation and causations of what you’re seeing in service operations.

Matty: [00:25:08] I think that’s where there are a lot of places where— in a second, I’m going to ask a question that talks a lot about how things in ITIL can fill some of the gaps in the, I hate to say, traditional DevOps approach, since it’s not that, but it’s old enough that maybe we can say traditional. But then likewise, when we think about where we’re at when we talk about automation and truly treating our systems in a codified way, those differences in being able to say, like you said, that problem resolution is really challenging when you’re like, well, wait a minute. So what happened? I always like to say, I’ve been in ops my whole life, so it’s how much time I’ve spent on the, it worked in QA, why didn’t it work in prod? Every time I have a huge human doing anything, um, manually with a system, that’s when I know I can’t trust it anymore. You know, as the great Mark Burgess quote of the, every time someone logs interactively into a system, they compromise everybody’s understanding of that system. So anyway, my point being is that’s a thing that the growth around automation and auto and being able to track those things I could see as being something to make it easier because there’s less manual steps to try to track down that a human being did a thing and being able to compare versions of infrastructure. And say, okay, this is what it did look like at this time. But that going into one of the things, so our good friend Dustin Collins, who was on the show recently, he said, you know, he said again, coming from his admittedly shallow understanding of ITIL, but he says he sees it, or his understanding is it helps clearly define roles. And one of the problems that sometimes we see in cross-functional teams that have adopted DevOps is that sometimes, depending upon how the culture is built in the organization, a lack of responsibility structure can cause problems. And the quote is, if everyone owns it, no one owns it. Again, I’ve found a lot depends on the organization. There was kind of an infamous or apocryphal talk or event where someone at Etsy was giving a talk about blameless postmortems, and someone said, well, how do you make sure that the people go back and fix the things that you told them to do? The speaker kind of said, I don’t understand the question. We just do. That’s how we work. But that’s definitely not something that scales. So the question is sort of thinking about being, are there parts of Agile we can help to bring in to help alleviate the issue, um, with that? And, you know, kind of he’s mentioned responsibility matrices, which I think about races. Yeah, which that was one of those BankOne things that I thought BankOne invented till I learned more and I was like, oh, okay. So what do you think about, and again, you also, I wanted to mention, you know, so you were at DevOps Days DC, so you’ve had some exposure to the movement and you’ve been working with stuff. So where do you see that the things you know about ITSM and ITIL, where maybe the things that the DevOps-oriented organizations are doing could be enhanced by some of these practices, especially around responsibility maybe and communication.

Steven: [00:28:09] Well, exactly. I think ITIL does have a place within DevOps. They obviously can coexist, but they’re really supportive of one another. Or ITIL really supports DevOps approach. A proper implementation of ITIL would facilitate DevOps. Now, when you were discussing about the identification of roles, that is something that ITIL addresses and says should be specified, and that’s part of the RACI matrix is the roles. But it also looks at functions. A lot of times within organizations, people seem to want to see themselves within a certain silo, a certain specific role. And ITIL says just because you’re an analyst doesn’t mean that when you’re in part of the change management process, you don’t also additionally have this type of role. And you can be assigned that role as part of, you know, being grouped into a function that does these types of activities. I saw a lot in the past within the incident management type of approach where people believed that’s all they did was incident management. Problem management was something somebody else did. Change management was a different group that was responsible for documenting that. The designers, dev had to do the documentation. That was not part of what you did within operations. And ITIL doesn’t approach it that way. ITIL says you are an individual that can be assigned any type of role within any type of the process within the organization, and you’re going to perform these types of tasks. Or activities in support of that process.

Matty: [00:29:57] I think I was gonna say, that’s, that’s kind of, kind of interesting because I think, um, you know, we talk a lot, you know, again about breaking down silos and things like that, and I don’t think there’s anything inherently within the framework that prescribes siloed behavior. And, and that was one of the things I, I gave a talk a few years ago about, about merging ITIL and Agile when I was at Apartments.com, and we’ll put, uh, the link in the show notes. I haven’t actually looked at that talk in a while, so I can’t speak to whether it’s— whether I’m going to, you know, completely embarrass myself reading it again. But one of the things was, how do we sit there and cross over? And while we may have, from a design perspective, you’ve talked about this, you know, being able to own the process maybe, but even then, one of the things that I remember from working with it, and one of the things I actually really liked about the framework was actually how interdependent all of the different practices really are. It’s almost impossible to say I am, and it’s funny ’cause I do go, I talk to what you’ve talked about. I see this with my customers all the time when we come in in a presales engagement on something and people introduce themselves and say, I run configuration management from an ITIL perspective at this organization. And I’m like, well, do you do anything with— no, I do the configuration management. And it’s like, but how does configuration management live in a void. Because for people who are listening that aren’t as familiar, all these things flow together, right? So like, if you have an incident, which is something is causing an interruption in service, right? So you’re gonna look back to— and I’m gonna get it all wrong, but just thinking at a high-level pseudocode, if you will— we’re gonna look at that and basically say, can we resolve it using a workaround that we know about? If not, does it turn into a problem? But in order to solve this, I may have to create a change, and the Making a change to configure. I mean, I’m making it really high level.

Steven: [00:31:58] Yeah, so let me stop you. So that’s a misconception with an idle. So let me clear that real quick. So one type of record cannot turn into another type of record.

Matty: Oh, I didn’t mean that. Okay, thank you for clarifying that.

Steven: Exactly, I know that. So one does trigger, but like you said with the change, one can trigger another. An incident can trigger, a single incident or multiple incidents can trigger a problem. To address that problem may require you to execute a change. Which are tied together.

Matty: That was the thing I was kind of getting to, is that these things all flow together. But thank you for making that. That’s a very good clarification that it’s not that this becomes that, but if I’m making a change, what am I making a change to? I’m making a change to a configuration item, to a thing in my environment that I care about, right? And why am I making this change? I’m making this change now. And this is the one thing that I think is tough to wrap the brain around, is that all that flow makes sense when it’s regarding thinking about the operational nature of IT as service management. When we think about it from a software development perspective, that’s where, to me, it always feels like software development, like feature development, kind of shoehorns into there because— and this was the challenge we had with apartments— was how did a backlog or feature requests feed into— and we ended up kind of having to build a bridge between our Agile tooling around backlog and feature to say that releases were tied into then change requests. Then that’s where that kind of hit off there. But, we also could have problems. If something was a problem, that would then hook back into the Agile tool that was managing the backlog to say, hey, product owner, you have something you need to work on now. You need to actually fix this.

Steven: [00:33:47] Yeah, you have to address that defect that was discovered.

Matty: Correct. A defect was discovered. I think that’s one of the things where it gets really kind of a— is where you end up having to build that bridge. You, having more experience in this, obviously, than I do, is where is the way that those kind of fit in, or is that something that it’s just the framework itself is not really necessarily intended around that stuff, so you do it somewhere else, but you build a really nice connection? So that they’re— they are talking to each other in your process.

Steven: Well, exactly. ITIL is not going to dictate how you do your approach as long as you’re looking at your triggers so you understand your inputs, and then of course your outputs. ITIL can be aligned to accomplish what you need to if you determine where those alignment points must be. And that was something that I brought up at DevOps DC. If no one’s ever sat down with the organization or helped an organization understand ITIL from ideally an expert approach, you’ve probably never really accomplished true ITIL or, or seen the alignment points. The difficulties that, that the organization may be having, having is they’re trying to shoehorn something that is really not conducive overall to the entire process. So it’s manifesting itself Because you can’t get that alignment. So someone has to sit down and say, this is how we do it where we are today, this is what we’re trying to accomplish. ITIL in the framework says that this is the best alignment points for those type of actions or activities or tasks. So how do we make that work better? How do we make that more efficient and more effective? And ITIL is going to give you some of the industry best practices of when somebody’s approached it. Now, defect management is a big portion of what you’re going to need within service transition, because if you don’t have good control of your defects, then there’s no way for you to make any type of even possible correlations between what you’re seeing in service operations to trace it back to something that you may have seen in testing and you believed you addressed and resolved prior to transitioning it into production. A lot of times you’ll see a problem, it’ll be a manifestation of something that appeared to have been addressed.

Matty: [00:36:21] I would agree and understand that. So I think that goes into kind of the idea that despite— and this is true of Nomad, and we touched on this earlier— is that zealotry in any form is bad. I’m saying that this is the right way to do it. You get this in Agile a lot of times. People say, well, Agile prescribes that you should not do things that way. People would even say, well, that’s not DevOps to do that, and kind of say that jokingly a lot of times. They’d say, that’s very non-DevOps. Or even the same thing with ITIL. I think there are probably some areas where that’s relatively true, but probably not in the ones that people expect. In fact, one of the ones that people would probably— tell me if this is an appropriate statement— is a thing that would be a thing to say is absolutely true about ITIL, and if you’re not doing this, you’re doing it wrong, is to believe that there is the ITIL way to do it, and if you don’t do it this way, you’re doing ITIL wrong. Exactly. Treating it prescriptively is actually— that’s probably the one way that you could be wrong. I think what that means, if I’m kind of summarizing it, is there’s nothing wrong with building on the shoulders of giants, taking the practices of Agile, taking the practices of ITIL, the practices that have been built around the DevOps community, and saying, how does my company actually do business? What are my problem sets that we need to solve for, and how can I take these best practices that may come in And there may be parts I, I think you can pick and choose a little bit, but I think you need to think about it in a holistic and a smart way, not just say, well, I think I had a bad experience with a change control board 3 jobs ago, so damn it, I’m never gonna work somewhere that has one, so I’m not gonna put one in. Just think about the problem you’re trying to solve, right?

Steven: [00:38:04] Well, exactly. And you really don’t want to subjectively pick and choose which portions of IDLE you are going to implement. You should look at following the framework as much as possible. If you’re going to deviate from the best practice, it’s because you’re piloting or you internally have a process that is better than the basic approach that IDLE is laying out in the framework. It’s more of a guide. If you build around this framework and you build your own internal improvements or your own little different way to do it, as long as you’re consistent and as long as you’re approaching it in in the same way you’re gonna implement the rest of your IDIL, and your consistency, and your documentation of your records, and looking at your inputs and your outputs, there’s no reason not to modify your IDIL approach.

Matty: To extend it. I think that was more what I was thinking, was not necessarily saying, hey, you know what, I don’t wanna do this part, but saying, I can build, I can extend, I can build bridges between them, and then figure out, or if there’s something that swaps in. But again, to your point, it’s gotta be because you’re replacing that input, the stuff that’s happening between the input and the output, something that is more effective for your organization.

Steven: [00:39:22] Exactly, because ITIL is going to give you— I don’t want to use the word rudimentary, but it’s going to give you a bare-bones best practice way of achieving that alignment. If you’re able to build a better, more improved process, then all it’s going to do is create additional value for the users.

Trevor: This just— this reminds me so much of when, you know, 2 years ago when I was sitting down and talking to Matt after a meetup, and Matt gave me this, this new word that I’d never heard before that kind of just aligned all the stars that I’d been seeing.

Matty: Was it a swear word?

Trevor: Well, I would hope that I didn’t need to pick up a new swear word.

Matty: I should learn them from you.

Trevor: But that, that, that’s what this feels like to me, is that these— all these different component practices that I seen elements of that kind of, like you said, build up this framework.

Matty: So one of the questions that came up, and I don’t think we need to go super deep in it, but it’s just kind of a thought, and it goes back to the idea that it’s implementation versus concept. But someone asked, and they said, how can you fit things like a CMDB in environments when your infrastructure is volatile? What’s worth documenting? And so the first thing, for listeners who don’t know, your CMDB is your configuration management database, or your, you know, Basically, it’s your big thing that says, here are all the things I care about and all the information about them that I care about. Those configuration items can be computers. They can be people. They can be files. They can be all the things. The thing that’s challenging, and this has been my experience, is any— I’m going to make a very global statement, but I believe it’s true— any CMDB that requires manual updates is about as valuable as the bits that it’s written onto because it’s It’s like updating a— we talk about this a lot. Let’s say I even keep a wiki updated with how I configure my servers. That’s only as good as the way those systems looked exactly when I did that. That being said, I think it’s— not only do I think, I know from experience that it’s absolutely possible if you’re using some— so again, if you’re in a DevOps kind of a world and you’re saying, okay, we’re treating our infrastructure as code, I’m using something like Chef or Puppet or something like that to manage my systems, that can help populate your CMD with what’s actually accurate. It’s not a discovery scenario because heaven knows, I know when I have anything that has to reach out and do a whole bunch of discovery, it’s going to take 6 days to go and crawl my whole network and then that’s out of date. But I think there are ways that you can populate that by information you already have. I think the challenge, and this is, you know, Steven, I just wanted to see what you think about this, is realizing, and this is just true of a CMDB in general, is I think it’s very possible, very easy to go way too deep into the amount of information and the you’re capturing about items, when the reality may be you got to think about what are the things I actually even care about, right? And especially, I guess, in an ephemeral world, certain things— I mean, again, I’ll give one kind of an example. When I was at the bank, and this was in kind of when we were coming into virtualization in the first place, we had— and it was part of us trying to understand how VMs were gonna work— when you would create in our database, which I didn’t know was called a CMDB at the time, but it was, was, you’d have an item for each operating system instance. We already knew about things like LPARS and WPARS from AIX and zones and stuff. One of the things would be the parent of it. It would say, oh, well, this LPAR is on this actual AIX server, or whatnot. But the problem is, then we went in there and we started using VMware. We said, okay, so this VM is hosted on this ESX server. Well, the problem is within VMware, if you’re using vCenter at the time, I mean, those VMs could move around all day long. You know, they would be on one host in the morning and another one in the afternoon potentially. And what was happening is anytime we wanted to go and do a VM migration from one host to another to balance, it was actually an entire change control process because it required touching the CMDB. And it made it to the point that you could not do automatic compute balancing. Balancing within VMware, where VMware does that— again, I’m not— this dynamic resource scaling where it would say, whoa, whoa, we’ve got a bunch over here. I think it was a learning experience of saying, at the time that we built this database of information we cared about, this was stuff that made sense to us. Our world has changed. Instead of saying, let’s keep trying to shoehorn the fields that we care about, what actually do I care about with my items? Do I care about the operating system on it? Probably. Do I care about the serial number of the hard drive? Maybe not. I don’t know. What do you, what do you think? What’s been your thoughts or experience on that?

Steven: [00:44:19] From an ITIL standpoint, it’s very easy. You document whatever is going to create you value, whatever is going to allow you to deliver value. Now, from a problem management standpoint, when you’re doing investigations and you’re trying to scope it, the more information you can have to identify why you’re having a specific symptom or issue to a subset of your enterprise or your product, your services, or your users, the more information you have to look at the deltas between who is, who is being affected or who’s not, or which, which component’s affected and which component’s not, you’re able to appropriately scope it. If you’re unable to appropriately scope the issue that you’re investigating, then you really don’t know how to properly prioritize it because you can’t assess the impact and the urgency of it. So your CMDB should help you do a lot of things. One of the most vital pieces is allowing you to scope your, your issues within your enterprise, to know where you need to be devoting your resources. And ITIL framework looks at managing your resources to a way to allow you to ensure that you’re delivering value to the users.

Matty: [00:45:32] And I think that’s a key point you made right there, which is what’s the point of the CMDB existing other than there’s someone who owns it in then it’s their job. That being said, if the point of capturing this data or of having this data available is it’s invaluable in your, like you said, in your scoping and your understanding, then you have to be able to trust that data, right? Because if it’s saying, okay, the way that I know this gets updated is it’s some intern’s job, goes and does this, I’m like, eh. But, if I know it’s coming from a trusted source that is actually Again, you’ll see I’m a believer in there are certainly lots of things that only human beings can do. I think that’s the thing I want people to bear in mind is that treating your infrastructure as code and treating it dynamically does not replace the need for being able to capture that information, and the things complement each other substantially. I think that goes back to your thing. ITIL does not necessarily replace DevOps, nor vice versa. They can be very complementary practices. When truly understood, I think. Now, the tooling that the vendors want to sell you on either side of the equation may add up and look differently, but hey, I’m a vendor and I’ll even tell you that, that that can be hard when we’re trying very hard to not be that vendor ourselves, to not be the vendor says, hey, come and buy the DevOps, because that doesn’t work so well.

Steven: [00:46:52] So that’s what I brought up at DevOps DC, was would you accept DevOps in a box? If not, why would you accept idle in a box?

Matty: Yeah, I couldn’t agree more. That was a great quote, and that’s gonna get— that’s a pull quote for this episode for sure. So we’re running close on time. I wanted to talk about 2 things. One was, you know, we’ve kind of danced around talking about incident and problem and root cause. I wanted to think if there is anything, you know, thinking about a lot of taking those things that you’ve got good practices around and what’s been either a your experience, maybe you’ve seen or from people you’ve talked to, or just your thoughts about how, you know, we already alluded to blameless postmortems, but being able to take some things that become common patterns like chat ops or blameless postmortems and layering them into the, you know, kind of descriptive things and your own particular experience around doing problem and service restoration type of activities.

Steven: [00:47:56] We don’t have time for that today.

Matty: Okay, well, we always— this is how every episode ends. We were like, we’ll have to have a follow-up episode.

Steven: Yeah, we would have to. Now on that, a lot of the organizations now, because we’re trying to provide services to a greater number of users and we’re trying to facilitate the value that the customer, the bill payer, is asking us to make available as a service to deliver to the users, we have to be more efficient and more effective on how we do do that. Incident management is one of the areas where we see an indication or record or documentation of our either inability to provide the service that we should be or some type of difficulty in the user being able to experience the service that we’re providing. And I’ll roll this back up into the ITIL framework. Is continuous service improvement is a big portion of ITIL as well as DevOps. Continuing to look at improving the service so you can increase the value that you’re providing. So how you go about achieving that within DevOps and ITIL is going to be a continual approach of integrating best practices and piloting concepts and looking at those process improvement areas. And like you said, that, that’s where they— where ITIL and DevOps should coexist together, and one should support the other, and they’re pretty much symbiotic.

Matty: [00:49:35] You know, you know what, I couldn’t think of a better way to wrap up the conversation, because 2 of the things that are actually really the most important to both of these approaches is that idea of wrapping the whole thing around continuous improvement. And that’s You know, we’re more alike than we’re different is the lesson of the night. So we’re gonna move into our checkouts, of which we only have Trevor because I forgot to tell Steven we do that and I forgot to do one. So Trevor, what’s your checkout that you were so excited to share with our listeners?

Trevor: Well, it’s goddamn adorable, that’s what it is. Check out Yoshi’s Woolly World. It’s a new game for the Wii U. And it is goddamn adorable.

Matty: Gotcha. Thank you for that. As you may recall, we now have a new section on the show when we give a podcast suggestion. We were certainly not the first to come up with having a DevOps podcast. In fact, as far as I know, the first is our friends John Willis and Damon Edwards at the DevOps Cafe, which you can find them on iTunes, but you can also go check them out at devopscafe.org. This is a show that has a really special place in my heart. Not be doing any of the things I’m doing today. I would not be doing this show. I would not be working at Chef. I would not be— I would not be knowing Trevor at all. So, well, 2 out of 3 ain’t bad of good things that DevOps Cafe has brought to my life. And I’m a little wound up on Mountain Dew. But when I wanted to learn, when I first was wanting to learn about DevOps, I started listening to this show, and I understood about 10% of what John and Damon were talking about, but I forced myself through to just let their stuff— maybe I figured by context I’d eventually start to understand it. In a lot of ways, we started this show because of DevOps Cafe, because we wanted to be able to be that show that I wanted years ago, and knowing that not everybody is as bullheaded and stubborn as I am to push through that stuff. They are both super busy folks. The episodes come out when they do, but every time they do, it’s some of the best DevOps podcasting that’s out there. Check out DevOps Cafe.

Trevor: [00:51:56] If you haven’t checked out our new website, you should come check it out at arresteddevops.com. We’d love if you’d tell us what you think about it. Any issues with the site can be logged using the Issues tab across the top.

Matty: I just added that today. Amazing.

Trevor: I know.

Matty: Continuous delivery of Arrested DevOps!

Trevor: Oh, the eggs365 just came in. Yes, chimes through there.

Matty: You can also just—

Trevor: or you can email us at shows@arresteddevops.com with your feedback.

Matty: We’d definitely like to know what your experience is like with that. We’ve rebuilt the whole thing. We’re hoping it’s faster and easier to navigate. And also, we wanted to really prove to the world, if you didn’t know already, that Matt is not a front-end web designer. But what we do have is a newsletter. You can subscribe to it at arresteddevops.com/bananastand. It’s the best way to know about upcoming podcast episodes and cool news with DevOps. We also have an iPhone app if you dig that kind of thing, and you can download it for free at arresteddevops.com/iphone.

Trevor: [00:52:59] Which reminds me, we’re signed up to become one of the first podcasts in Google Play Music when Google Play Music launches podcasts, which I think is supposed to be like months from now actually. But I thought it was sooner than that, but we’ll find out. It’s exciting for me.

Matty: We just got approved. So when you’re looking for— when that comes out and you’re looking for new new podcasts in Google Play Music. We’ll be there, but we won’t be a new podcast for you because you’re already listening to us.

Trevor: So exactly.

Matty: All right, thank our sponsors, won’t you, Trevor?

Trevor: I was about to. Thanks to our sponsors. Be sure to visit them at arresteddevops.com/victorops and arresteddevops.com/datadog. Thanks, Steven, for joining us.

Steven: It was a pleasure.

Trevor: And thank you to our loyal listeners. If you enjoy Arrested DevOps, we’d appreciate it if you would visit arresteddevops.com/itunes and leave us a review in the iTunes Store or in Google Play if they let you do that later.

Matty: However that stuff works, we’ll find out.

Trevor: [00:53:59] We would love to know what you thought of this episode. Please leave us comments at arresteddevops.com/itil.

Matty: Like we said, you know, we’re on Twitter @ArrestedDevOps. We love your tweets. We love to talk to our listeners, and if you ever happen to run into us at a conference, we’ll take a picture with you because we’re dorks like that. But we are especially happy to get your input, ideas, and feedback, and you can send that to us at shows@arresteddevops.com. Let us know any ideas you might have for future episodes.

Trevor: I’m Matt @MattStratton, and I’m Trevor @TrevorGHess. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.