The Database Calls Are Coming From Inside The DevOps

Jessica: [00:00:00] to be present with your data in all its persistence.

Baron: And mindful of your queries.

Matty: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matty Stratton, and my co-host today is…

Jessica: Hello, I’m Jessica Kerr, better known as Jessitron, and I’m very excited to have my first time hosting Arrested DevOps because I think DevOps is one of the 3 most important movements in software today. I’m very excited to have Baron on the show today because the other day, I was hosting a track at QCon San Francisco about DevOps, and Baron was our first speaker, and his talk was super awesome. And now you get to hear from him too. But first, a word from our sponsors. Chef is a community of professionals practicing DevOps every day. We are making, proving, learning, and shaping the future. We are known for welcoming, encouraging, and liberating others to do the same. We do not talk about change, we do change. Join the community and learn about our solutions at chef.io.

Matty: [00:01:22] This episode is brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 120 common infrastructure tools such as Chef, Docker, and AWS so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. The worst time to learn about incident response is during an incident. Don’t wait for an outage to strike before getting started. The PagerDuty Incident Response Training Course is now open source and free for everyone at response pagerduty.com. Based on the same training that PagerDuty employees go through, this course will show you how to streamline your incident response process, turn chaos into calm, and demonstrate the role of an incident commander. So what are you waiting for? Go to response.pagerduty.com today and check it out.

[00:02:24] The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener, there is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app.

Jessica: Now it’s time to talk to Baron Schwartz about DevOps. FYI, show notes for this episode can be found at arresteddevops.com/devops-database because we are talking about the database calls are coming from inside the DevOps.

Matty: [00:03:36] I love that.

Jessica: Baron, can you tell us a bit about yourself?

Baron: Yeah. Hi everybody, I’m Baron, Baron Schwartz. I am now a company founder and CTO, but back in the day I was a software developer and really into extreme programming in the beginning. And then I got into databases, got into consulting, mostly around performance, became really interested in measuring and monitoring, and then eventually founded VividCortex, which is where I am today. So that’s kind of like a zigzag version of my history.

Matty: Super cool, super cool. So we’re talking about DevOps for the database, right? This is this thing we kind of always talk about. We’ve had an episode a few years ago, which we’ll put a link to in the show notes because I don’t remember the URL off the top of my head, where we talked to some of the Redgate folks about what DevOps in the database is about. But these things always kind of come up. We’re always like, okay, great, we’ve got this piece, this piece, this piece, and then some database thing happens. So I think, but one of the things we might wanna talk about before we get into DevOps and the database is, Baron, we were talking kind of before we started recording about what is DevOps. And I think I know, granted the show is called Arrested DevOps, so we talk about DevOps a lot, but we don’t, we’ve sort of always been obliquely avoiding this particular definition. So maybe we can talk about that a little bit. What does DevOps mean to you, Baron?

Baron: [00:05:05] I honestly don’t know exactly how to put it into words. I feel like it’s something that you sort of live, and then when somebody says, what is it? You kind of stumble around and say things about culture and its practices, and it’s similar to maybe what Agile is in some ways. But I think the question of what is DevOps and why is it so hard itself is actually really interesting. And like you mentioned, we were talking about this before we started recording, and I had a little bit of a history with it. At some point, I was thinking, what is DevOps? I sort of know it organically, but I find it hard to explain to people. And at that point, I think this was a few years back, a lot of the big vendors were marketing really heavily around DevOps. And at the same time, there was a lot of pushback from within the DevOps community itself. About, you know, you can’t say it’s a DevOps tool. The only DevOps tool is a person who says something is a DevOps tool. And, you know, sort of like DevOps is not a job title and you can’t have a DevOps team, right? So it was all these sort of no, no, no, but there wasn’t really a yes. And I thought that was kind of problematic because I saw the vendors seeing this as something that needed to be done and doing what I thought was not a bad job at it. I did a little bit of Googling just before we started talking, and I remembered that AWS had what I thought was not a bad definition of DevOps. I found that Microsoft has one, New Relic has one, there’s a handful of others. But from within what I would say is sort of the core DevOps community, we haven’t really produced something that I think is succinct and gets people in the right neighborhood and helps them to understand what we are talking about. So it doesn’t sound like kind of— what is the term for when you’re inside a community and you use language that people inside the community know and sort of identify with, like kind of an echo chamber? Yeah, there’s some technical term for it that I can’t quite think of off the top of my head. I mean, you’re saying the same thing I am, but there’s also this kind of language that people will use to signify that they’re part of a group and people outside of the group feel excluded by it. I felt like that was what the DevOps community, we were unintentionally doing because we didn’t want to really define DevOps because we felt like that would be limiting and prescriptive, and that’s kind of not at the heart of what the DevOps movement is. But in trying not to do that, we were unintentionally, I felt, gatekeeping, and the vendors were seeing this clear need for somebody to say what DevOps is so that people new to the movement can get started and understand and not feel like such lonely outsiders. So, you know, I thought that was not a real healthy situation. I wrote a little thing on O’Reilly, maybe we can put the URL into the show notes, where I said maybe we’d be better off with a manifesto, and there were kind of a couple of pieces that were written at the same time that responded a little bit to that and presented some different points of view that maybe we’re better off without manifestos so that people can, you know, sort of discover and learn their own. Anyway, it never really went anywhere, and I was content to put it aside. I thought the discussion was useful. So we started off a couple of minutes saying, what is DevOps? And we have just done what is sort of like very classical and typical in the DevOps community where We talk all about the question, but we don’t answer it.

Jessica: [00:08:38] Oh, whoa. Oh, I mean, maybe that’s one of the things that’s beautiful about the DevOps community, because usually the insights are in the question, not in a particular answer.

Matty: That’s definitely true. Asking questions is harder than answering questions. I know that’s especially when we think about the ideas of learning, and I’ve been spending a little bit of time just from other reasons, watching a couple of John Allspaw talks. And he also was on a little bit of a Twitter rant the other day about the importance of post-incident review versus actual incident management. But the whole idea was when he talks about things to do post-incident, we talk about, it’s about asking questions, not answering questions. But when we do our postmortems, we’re mostly focused on answering questions. And the idea is that our learning comes more from asking. So I, I think it’s good to continually ask the questions, but I, I do see to Baron’s point that at some point it’s okay to try to answer them because just asking them into a vacuum doesn’t bring the conversation forward.

Baron: [00:09:46] I wonder if we can ask and then at least answer, or maybe like ask counter questions as a form of answer.

Jessica: Oh, like Socratic dialogue.

Baron: Yeah. Yeah. You know, something that would be helpful, but that would also address the reasons that people who’ve had experience with, you know, all caps manifesto kinds of definitions are trying to avoid an all caps manifesto and where that leads, which is things like co-opting and so forth and so on, but not having a manifesto or some kind of a definition or guidelines, I think also leads to co-opting. And so far the co-opting has been I think pretty good. You know, I think New Relic and AWS and Microsoft have done a pretty good job of defining DevOps in the absence of an official definition.

Matty: Well, I think we do have an official definition. We just don’t like it because it’s still too vague. I mean, if you were going to sit down and say what the DevOps community considers DevOps to be, they’re going to tell you it’s CALMS, right? But that’s culture, automation, lean, measurement, and sharing.

Jessica: [00:10:49] But that doesn’t say anything.

Matty: No, I was just going to say it’s also incredibly vague.

Baron: And also, by the way, there’s, there’s been multiple quote unquote pillars of DevOps over the years. It seems like there was first CAMS, then there was CALMS, then there was ICE, then—

Matty: I was just going to say ICE sort of was around for a little bit. No, no shade on— was it— it was Feedback. Feedback was the one. Yeah, I’m trying to remember who I was talking to. It was a good attempt. But I think it’s, the words were bigger, so it was harder to remember. Yeah, I can’t remember them. That’s the beauty of CAMs is that those are simple words. So I think that the thing with CAMs and columns is they’re good starting points for conversations, but they’re not definitions.

Jessica: Which is kind of the whole point of DevOps. It’s a starting point for a conversation. I have a definition to contribute.

Matty: Please.

Jessica: I want to say that DevOps is not a thing, capital T. It’s a situation. It’s where you put the responsibility for operations and developers on the same people because that gives them more options. And then there’s all kinds of things you can do to support that situation, including tools, including a DevOps team. But then it’s not really a DevOps team, it’s a development experience team or something like that.

Baron: [00:12:16] Right, developer productivity or a team that is supporting DevOps outcomes.

Matty: Yeah, I think that’s the hard thing about that. I mean, without kind of bike shedding over DevOps team forever and ever, is I’ve had the same thing where I always say that DevOps team or the term DevOps engineer are not inherently bad. But they can be an organizational smell, like a code smell, you know, because oftentimes when someone is referred to as a DevOps engineer, for example, what they really are is an automation engineer. And that’s fine, but that means you think DevOps is just about doing some automation, you know, or the similar thing. If your DevOps team is, is usually, again, this is what I see a lot of, is an automation team, right? This is the team that writes all the Ansible playbooks.

Jessica: So it’s a DevOps support team, but the actual DevOps teams are the development slash operations teams, right?

Baron: I think this gets to what Charity Majors talks about, that there was a first age of DevOps and now there’s a second age coming. And so what she says about the first stage is that it’s when operations people stopped typing things directly into the command line to administer servers and they started doing infrastructure as code. And I think this is very true. Looking back on the first handful of years of DevOps, it was mostly ops people doing dev. But what I’m seeing now, and what’s really important for us and our customers at VividCortex, which is why I got sort of renewed interest in this topic, is what Charity is referring to as the second age of DevOps, when developers own things into production. Developers are doing operations in some form or fashion, and they’re responsible and accountable for performance, operability, instrumentation, observability, all of these kinds of things that it takes to run things well in production and learn from them and close the feedback loop all the way back into development again. And I think that’s incredibly helpful to have that kind of nuanced view that DevOps now is not what DevOps was a few years ago.

Jessica: [00:14:26] So, Barret, I have to ask, have the DBAs stopped typing things in terminals?

Baron: Some of them have. I’m definitely seeing some of them do that. And you mentioned earlier about an earlier episode with Redgate. I would love to go back and listen to that now that I know it exists. So maybe we can put the link for that into the show notes. And I would love to see what the Redgate community or the folks who were your guests think about DevOps in the database because The database is typically seen as the realm where experienced DBAs do most of the things that developers consider that somebody needs specialized skills to do. And I’m seeing that getting more automated now, but I’m not seeing broadly that a lot of developers think of DevOps as owning their impact and their application’s involvement with the database, which is what’s been most helpful for me the last couple of years. It’s where I’ve seen what I think of as database DevOps superstars really focusing their efforts on getting the developers involved with and competent with the database so that they can stop being sort of human automation around the database and they can start being more strategic thinking about things like, how do we turn this into a platform? How do we handle forward-looking architectural and capacity planning kinds of needs and stuff like that. But DBAs can’t do that if they’re saddled with operational things that are essentially human automation, like running a schema change or going and fetching performance information about what’s happening with particular queries in the database. Now, this is not to say that developers are going to have all of the superpowers that somebody who really understands the innermost workings of the database has. You still need, in my view, you still need people with specialized skills and experience and knowledge to help developers be great with the database. But I do think that developers need to end up owning performance in the database. And now tying this back into Redgate, I tweeted a little while ago a survey. I just threw together a survey. I didn’t actually put a lot of thought into it, just wrote down in a Google Form all of the questions that I could think of related to DevOps in the database and tweeted it out. I think it got maybe into a newsletter or something like that, and I saw what looked to me like a lot of folks from the SQL Server community joining in there. For folks who don’t know about Redgate, they’re more focused on SQL Server than, for example, the open-source databases. When I looked at the responses coming back, I saw something that I considered to be a little bit odd. I had taken for granted that doing DevOps required owning your code all the way into production, sort of this second age of DevOps. And I saw that people ranked it kind of low on the survey. There was a question that I asked, how important are each of the following in your view of database DevOps? And one of the options was developers own the database performance of their code. And when you ranked it by how many people thought it was very important, somewhat important, or not important, it was kind of pretty far down the list, only about halfway up. I would put it all the way at the top. And that for me was kind of a moment of aha, I think maybe where this is coming from is the view that doing DevOps in the database is more about DBAs automating and not doing manual labor anymore, rather than developers really being— Exactly. Yeah, yeah. So, I just packed a lot in there, but what do you think?

Jessica: [00:18:17] Okay, so, yeah, I think your answer was, we’re totally still in the first stage of DevOps with the database, and yet the developers are still afraid of owning the performance of their code in the database, and hope— they’re probably waiting for tools to fix that for them. Does VividCortex do that?

Baron: No, if only. No, but, but some of our customers— we have— we’ve been getting really close to understanding how our customers succeed with VividCortex. And just quick aside for, you know, footnote, VividCortex monitors everything about what the database does. So we, we say that we help people understand what their databases are really doing in production.

Jessica: Are we talking— quick, quick definition. Which databases are we talking about?

Baron: MySQL, Postgres, Redis, MongoDB, and the various versions of Aurora and RDS and so forth.

Jessica: Oh, okay.

Baron: So focused on, yeah, focused on both SQL and non-SQL, primarily open-source databases at this point. We have support for a couple of other technologies that I won’t mention here because it’s not really commercially interesting. But what we do is we measure a few things and put it together. One is we measure the signals coming from the database itself, the things that you would typically think about measuring with a monitoring platform from statistics tables and counters and those kinds of things. We also measure what the operating system is doing, again, using the things that you already know and love like the proc file system. And then the last thing is we measure every query entering and leaving the database. Primarily through sniffing the network traffic. So we get these 3 sets of data and we put them together. And then as we’ve been looking at what makes customers succeed or what eventually leads to, for example, a lost account with customers, as I got really close to that over the last year or so, it became obvious to me that the customers who are succeeding both the best with VividCortex and succeeding the best in general are what I would now consider to be solidly in the second age of DevOps, where the developers are really responsible and accountable for how the database is doing. And the DBAs have switched out of kind of guarding the walled garden and into running a platform which is as self-service for the development teams as they can get. So I think this is a very early adopter kind of a mindset among companies. And when we see this, which is not in all of our customers, we see the bulk of the engineering team actually very engaged with VividCortex and thus the database on a daily basis as part of the regular workflow. And when we don’t see that, we see the database being considered the DBA’s job and developers are often not very curious or very interested. Why would I wanna take on yet another responsibility? The DBA’s got that, the DBA, understands the database, I want to stay in my code. So I’m not trying to say that those folks who don’t want to get involved with the database are bad or lazy or anything negative about them. I’m just seeing that there is this shift where some companies are actually saying, you know what, we lost our last database, we’re not going to replace that person, we’re not hiring another DBA. Everybody who’s writing code, you gotta own your code into the database. And a lot more stuff has to happen there, including tools. You know, you can’t tell people that they’re responsible for database and query performance and things like that if they don’t have good tools to do that with. So that’s partly why I see it so closely. Um, but to answer your question, Jessica, no, VividCortex unfortunately can’t make people bring their databases into the second age of DevOps, but we do get pulled along very nicely when a company is going there anyway, and we help grease the skids for that.

Matty: [00:22:26] I have a question, kind of thinking about what you’re talking about brought up something I’ve been thinking about lately, which is this, we’re asking software engineers to do more than they’ve done before, but on the same token, we have to remember that there’s still specific skill sets, right? And so, there’s kind of a little bit of a hubris around a lot of software engineering, which is, well, if it has to do with technology, I can figure it out, right? And maybe this is the 20 years of an ops person speaking to me, which is to remember that just because we can’t expect everybody to be a 10x rockstar engineer, right? So, how do we weigh those 2 things where we’re saying we’re asking you to do more, but we still understand that, no, you’re not going to be as— there’s still specialization. I think in the Effective DevOps book, the metaphor I like that Jennifer and Rin uses, instead of silos, they talk about islands because you can build bridges between islands. You still have to have some island. There’s still this need, and Charity will talk about this too. Ops is a skill. DBA is a skill. And so that’s what I wonder is one of the things, like you said, where people are like, well, we’re gonna have developers own the performance of their code all the way through to the database, but you still need people who have strong understanding. You can’t expect somebody to be deep on every single thing, right? So how do we weigh those things? How do we make sure that while we’re asking software engineers to do more, we’re not contributing to this, like, Well, you’re a software engineer, so you must be able to know everything.

Jessica: [00:24:07] It’s kind of that, but that is the ultimate question of DevOps, right? At least from my perspective as a developer, it’s, you want me to understand what? And somehow we as developers have been like, oh, I can handle Linux. Yeah, I can run my program. I can do Docker. And Kuber the netties and stuff. But the database, we like, well, at least, I mean, I feel like this understanding that the database, especially SQL databases, are big and important. And somehow I, like, have a deep respect for DBAs that I didn’t always have about operations people. Speaking as a stereotypical developer, actually. So, the database has continued to be frightening in, like, a respectful kind of way. And I want someone else to look at it. Well, okay, personally, I don’t. I’ve always loved digging through Precise back when we were on Oracle and getting into the queries. But But I observed that the other developers around me did not. So, you’re right, there’s like this scare factor about the database still. But if we can accept this about operations, we have to accept it about query performance. The code and the schema are like, they’re tightly coupled. You can’t draw lines there.

Baron: [00:25:43] Yeah. And you’re right that the database is scary because it’s a comp— there’s a few reasons that it’s scary. It’s complicated. Databases are doing really hard work. As developers, we like to push the statefulness down the layers of the stack and keep the upper layers of the stack stateless, because that means that if a node goes bad, we can just replace it and no harm done, right?

Jessica: Oh, that’s a good point. We’ve like crammed all the scary bits down, down, down into the database.

Baron: Exactly. It’s terrifying down there. And bits meet the disk, you know, it’s a lot of gnarly stuff that has to do with hardware and drivers and file systems and several layers of things that are persistent. You know, there’s the database and its log, and then the file system and its log, and the hardware and the RAID controller. There’s all of this complicated stuff that if you’re going to understand how your data gets stored persistently and safely, it’s really complicated. And it turns into distributed systems very quickly, even on a single server, because you end up thinking through concurrency control and how different threads are working together to make sure that everything is consistent and transactions, all this kind of stuff. And it is a lot to ask developers. And that’s why we need people who love it and get really deep into the internals of the database and can help somebody bridge that gap, build the bridges between the islands. Without sort of enabling them not to set foot on the island. And I think that’s really important.

Jessica: [00:27:18] You know, that sharing of skills, that dispersal of knowledge.

Baron: Yeah, absolutely. Because, yes, understanding how the file system interacts with the RAID controller or whatever, not everybody needs to do that. But I do think that everybody needs to understand how to model effectively. And that’s, as developers, that’s what we do.

Jessica: Model writers create the schema?

Baron: Right. I mean, schema is basically another layer of creating the data model, which is representing the world in our programs, data structures, and so forth. And there are some basics like indexing and things like that that are often overcomplicated. And I’m always trying to simplify, and I love when people are able to simplify for me as well. So some of those things can be pretty accessible, but then there is all of this gnarly stuff about how databases plan and execute queries, for example, because the theory of, let’s just say, a relational SQL database is that you’re supposed to just kind of declare how you want to get your data, and then the database is going to figure out the most effective way to do that by evaluating lots of permutations of different approaches and combinations. Picking the best one, and sometimes it does that wrong, and then something is slow, and you’ve gotta get in there and figure out, okay, so what did it think about doing, and what did it think was gonna be slow or fast about each of those choices, and what did it end up doing, and what might be a better way? And that’s where somebody who really understands databases can make a huge difference. Oh, if you just reorder this or, you know, add an index there, then optimizer will have a better index to use and it’ll reorder the join and it’ll all work magically and something goes from multiple seconds to milliseconds. It’s like, but you have to make friends with the optimizer, right?

Jessica: [00:29:09] Speak its language, kind of, right? Exactly.

Baron: On its good side and understand indexing and schema design and normalization and some of those kinds of things. But those, I think, are at least the first 80% of those things are things that developers can do that don’t need to be considered territory of a rock star or something like that. And, you know, when you run into the hard sticky parts, then you need somebody who— or you ask somebody who understands all of the intricacies.

Jessica: Yeah, I agree with you because in my first 2 years as a programmer right out of college, I was understanding the queries to the level of being able to write decent ones and model decently and look at the performance results because I made friends with the DBAs, but they had to give me permissions for that. Once they talked to me for a bit and understood that I actually cared, they were happy to give me permissions. But yeah, not everybody does that, but we can. We totally can.

Baron: [00:30:15] Yes. Yeah, we can.

Jessica: Just like with operations, when you’re trying to figure out what’s going on in production, it helps tremendously if you know what the code is doing and why, and if you have the ability to add log statements or event outputs from the code, that makes operations tremendously easier. It’s the same with the database. If you know what that query is serving, what purpose it’s serving, and what you could tweak about it, that’s tremendously more powerful than only seeing queries coming in.

Baron: Absolutely. When I was a database consultant, a lot of times people would bring me into an application that was struggling to scale, and I would immediately find the problematic tables, problematic SQL, things like that. But because of this abstract declarative nature of SQL, where it’s kind of an algebra for describing a solution to a problem, it doesn’t describe the problem very well. Intent is lost in even a very simple query. So if you’re asking, what is this query trying to do? It’s almost impossible to go from the SQL back to, this is what the query is trying to do. That’s very different from most programming languages that are declarative. You’re looking at a for loop and so you go, okay, so we’re looping over the array. The intent there is very simple. For each item, we’re going to blah, blah, blah. But when you’re in the SQL, the intent is obscured way, way, way more. And so I would go to people and I would say, I see what this is. And my first question for you is, what is the intent of this query? Because this could either be something that I can see us rewriting in a different way, Or perhaps there is a bug in this SQL because it’s not translating the programmer’s intent faithfully, but I need to get into the programmer’s head and then kind of reverse engineer my way back into the SQL to understand what’s supposed to be going on here. Can this be broken into 2 queries? Should 2 queries be put into one, etc., etc.?

Jessica: [00:32:20] Yeah, yeah, which gets back to the, the whole purpose of DevOps is if you know the why, if you have the pull then you can complete the fulfillment of that need in a much more effective way than if you are breaking it down into tasks and then pushing those tasks onto people or databases.

Baron: Yeah, you tweeted about this the other day, didn’t you?

Jessica: I did, I did. I tweet a lot.

Baron: And it’s all good.

Jessica: Baron, I wanted to bring you on Arrested DevOps in particular because in your talk at QCon, you talked a lot about DevOps, dare I say, transformations in general. And you talked about things that you’d seen work, things that you’d seen not work. And I found a lot of these, like, really inspiring. Can you give us some general advice about how to DevOps?

Baron: [00:33:28] How to DevOps, how to be more DevOpsy, which I think is an anti-DevOps thing. Yeah, you know, I think there’s a handful of core elements, and there’s a handful of things that seem to be tried repeatedly that don’t work as well.

Jessica: Let’s hear about those, because in DevOps, we’re totally allowed to talk about what is not DevOps.

Baron: Exactly. All right. So, the way I arrived at this was some of it was deep thinking and reflecting over the years of seeing customers trying things and my own experiences because I’ve had a handful of scars over the last 10 years or so as well. And many of them were things that advisors had tried to tell me about, but I couldn’t understand, et cetera, et cetera. So, I went and talked some more with customers. This survey that I tweeted out was really helpful. People put a lot of interesting answers in there. So a couple of things— oh, the most important, um, and the first thing that occurred to me was that you can’t bring in a vendor to solve a culture problem. And in fact, I think culture is not something that you define and then the organization’s performance and behavior works from it. In fact, I think it works exactly the opposite way. I was just reading, you know, culture is emergent from the ways that things are done. It’s emergent from incentives, you know, what gets praised and rewarded, what is easy or what is hard, those kinds of things. And so, when you look at culture, you’re looking at an artifact, you’re looking at the outcome, not what actually created that situation. And so, when I’ve heard people talk about, you know, we need culture change, for DevOps to work here, I think that’s kind of a little scary. I think if you’re trying to change culture by changing the incentives or making something easier, then people will naturally gravitate towards what they’re rewarded for, what has high status, those kinds of things, and then the culture changes as a result of that. So, Jessica, when you asked a little bit earlier, Again, to reference, can VividCortex create DevOps in the database? No. But when people decide that they’re going to try and include better DevOps practices involving the database, and as a part of that, they’re going to make it easier for people by giving them more visibility with less friction, then it helps. But when somebody says, hey, VividCortex, I want to bring you in here and Um, you know, for this to work, for us to be a strong customer who renews, we’re going to need you to create a culture change. I’m like, that’s not, you know, you can never put that expectation on a vendor, right?

Matty: [00:36:18] You can, you can. I think you can have, I mean, I think a good vendor will understand the things you’re just talking about, which is to say, how can we help you understand the changes you have to make, but Right, you’re not just gonna rub some DevOps on it. And I think it’s, again, it goes back to behaviors, right? Behaviors is what influences culture. You can’t just snap your fingers and say, we’re gonna be, culture, like you said, it’s emergent. And it’s hard, right? Because we want the easy stuff, right? We wanna be able to say, well, we’re just gonna start doing this thing. But it does have to do with aligning incentives. And I also think a place where, Tooling can help. I’m not saying that it solves the problem, but the way to think about it is that I’m a big believer in making the right thing the easy thing, right? So make it the path of least resistance is to do things in the manner that will reflect the change that you want to see. There’s a great book that I recommend to everybody when they’re talking about wanting to change, which is the book Switch: How to Do Change When Change Is Hard. We’ll put a link to it in the show notes, but there’s a great story in there about a manufacturing company, and they said they had this problem where they had a machine where people kept getting hurt because there was a blade, and they kept cutting themselves on this blade. They kept cutting their hands. And this, you know, the sort of the anecdotal thing is, well, they could have done this big campaign around here’s how to be safe and be careful and blah, blah, blah. But instead, they redesigned the machine so that it required you to put both hands on a switch that was away from the blade to turn it on. So, made the right way the easy way. So, I always think about that mechanism or that idea when I’m trying to help people make these changes, say, what’s the thing you want? How can we make that the glide path? That’s how a tool can help, I think.

Baron: [00:38:07] Yeah, great point.

Jessica: Absolutely. I was thinking earlier as we were talking about how AWS and Microsoft have a definition of DevOps, and it’s not bad. And that’s because they are selling tools that you can use to support yourself in your DevOps process thinginess.

Baron: Transformation.

Jessica: As opposed to Agile, which, while it has a manifesto, has been completely co-opted by vendors selling culture change, which is garbage. They can’t give it to you, but they can sell it to you. It just won’t work.

Baron: Right, yeah. But, you know, when you look in the DevOps, especially at the cloud providers, they’re also selling a new way of life. And there’s a venture capitalist, Tom Tungas at Redpoint Capital, who wrote back a couple of months or something like that, that when you’re in an early market, professional services can be really important. And what that means in our market is probably customer success. If you look at— let’s look at Google, for example. They’ve built their entire technical ecosystem around some of the things like SRE, which I think is very closely related to DevOps as I practice it. Now they have customer reliability engineering, where they have an entire team who’s trying to go and help customers make this journey from the old data center way to the new Google Cloud way. Where SRE principles are a big part of being successful with the products that Google is trying to sell. So they’ve got this team that’s trying to help people, trying to help companies adopt SRE principles and practices. And that’s professional services. I don’t know whether they’re actually charging people for it or not, but that’s customer success. That’s making sure that your customers are successful in the world that you’re offering to sell them. And so I think the two can go together. I’m currently thinking a lot about that. How can we, for example, help our customers learn from each other? Not necessarily teach them how to apply DevOps to the database, but help them understand what has worked well with other customers that have succeeded in this. Because we’ve got some customers who are just poster, whatever, not children, but poster cases of DevOps succeeding with the database. And the benefits are almost too good to be true. You know, it feels like, oh, come on, surely you’re exaggerating that a little bit, but no, they’re not. They really are moving faster, moving more cheaply, delivering better product to software, and completely outrunning their competition. We had one customer in a particular industry, a very sort of very specific kind of a company. And, they had one direct competitor that they were battling for absolute control over the space. We went to their competitor and said, VividCortex has been a really big help with Company A. We think it could be a help here too. And, the response was, yeah, we don’t really take our cues from what our main competitor is. We’re kind of in a battle to the death with them. Very shortly after that, they were acquired by them.

Jessica: [00:41:25] Ha! They’ll take cues from them now.

Baron: That’s right. The last laugh.

Jessica: That’s super interesting, Baron. And thank you, because that, that point about for a new market, you need to help the customers see the practices that will make your tools valuable. That’s really relevant to us at Atomist right now. Okay, now that we know all the things that don’t work, okay, we know at least 2 things that don’t work. What does work?

Baron: I have a few things that seem correlated and maybe causative, but it’s hard to tell the difference between them. I kind of outline it for discussion’s sake in 4 big buckets of people, culture, structure and process, and tooling, which are roughly more or less aligned with some of the core principles that people have laid out in those acronyms. Things like that, things like CAMs and CALMs. Yeah, yeah. I sort of arrived there without thinking about CALMs. So, I think that’s a good sign that if I look at what’s succeeding with some companies and it seems to be well aligned with what some people have said is the core of DevOps, then that’s probably a good sign. There’s a few things in no particular order. Let me kind of blow through the headlines from some of the slides that I gave at that talk, if that’s okay. There’s tooling, especially deploy and release tooling. It needs to, you know, deploying, releasing, and the tooling around it is really important for people to actually do these things. Monitoring and observability. You know, one of the things that I think is super helpful about being able to inspect and understand your applications, your code’s impact on the database in terms of performance, availability, things like that, is that it makes you a better programmer. You will write better programs if you can understand the set of signals that your database is trying to give you that are hard to absorb and hard to understand because it takes tooling to interpret them. Shared knowledge and process is a big one. There’s a bunch of our customers that I feel like are way out on the advanced guard of DevOps in the database that have very robust sets of processes around things like deploy confidence dashboards and deploy procedures that are linked directly from their deploy tooling, things like that. I think a big part of it is something that’s maybe not DevOps per se, but is how I’ve been thinking about DevOps without knowing it, which is what Netflix calls full-cycle developers. You know, when I talk about developers owning things in production, I noticed that there was a presentation from KubeCon, the same KubeCon in November. Let me scroll up and get the person’s name.

Jessica: [00:44:16] Is this Greg Burrell?

Baron: It is, yeah.

Jessica: See, he was also in our track.

Baron: Right, but I had to leave, and so I didn’t see it, and now I haven’t watched his talk yet. So I’m gonna do that. But, you know, some ex-Netflix people used to be advisors to me a while back, and they were talking about microservices and aligning the teams end-to-end, sort of cradle-to-gradle to cradle-to-grave ownership of your code and things like that, developers being on call, you know, all of those kinds of things. I’m thinking that maybe it’s more of the full-cycle developer than it is DevOps, but maybe we’re talking about the same thing. I’m not sure.

Jessica: Matty, is full-cycle developer the same thing as DevOps?

Matty: I think it’s a little— I think it does ease itself into sort of that second age idea. I like the idea of full-cycle developer rather than talking about full-stack. Paul Reed has a great comment where he said, you call yourself a full-stack developer, when was the last time you wrote a device driver? And so, we don’t necessarily— but I think thinking more about the cycle, about being part of— and it doesn’t— Not to get into sort of like corporate speak around like these ideas of RACI and all that, because that starts to trigger off my ITSM. RACI is like the responsible, accountable, consulted, and informed, you know, right? But you can be part of the whole cycle of development, which doesn’t necessarily mean you’re responsible for the whole thing, but you’re involved. I think being involved in through the full cycle is, that’s DevOps, right? To me.

Baron: [00:45:57] Yeah. And I think it’s much more important to be present with what you have built and shipped and your customers’ experience of it through time than rather through, for example, technologies or layers in the stack or something like that. I think the lifecycle is—

Jessica: be present.

Matty: I think that seems like probably a good place to wrap up on that very insightful piece, sort of be present through.

Jessica: Yeah, to be present with your customers and with them, with your, your software in production and your software in the database, to be present with your data in all its persistence and mindful of your queries.

Matty: Always, always be mindful of your queries. Great. So, Baron, what do you have for our listeners to check out?

Baron: Well, for myself to check out, I need to watch Greg Burrell’s talk, and I think it’s going to be relevant to listeners as well. And at some point, my own talk at QCon in Jessica’s track is also going to be recorded and made public, or the recording that’s already been made will be publicized. I think there’s a couple of interesting What Is DevOps links that I can send over to you, and maybe we can put all 3 of them in there because I think if you look at 3 vendors’ definitions together, I think you get more out of them. It’s sort of like reading several translations of a work of literature or something like that. And then the last thing is something that just came to my mind from somewhere, and it’s totally unrelated, except that it’s not at all unrelated because it’s very much about our culture and society. So it’s not about DevOps, but it’s a set of podcasts that I’ve been listening to from Scene on Radio. That’s S-C-E-N-E on Radio. And they’ve had 2 series called Seeing White and Men. And I think Men might have just wrapped up now, but it’s been over the last couple of years. And it’s about, for myself as a straight white male, you know, I am sort of in the dominant power position of privilege where I don’t actually have to examine my privilege. I don’t have to examine what it means to be white. It’s sort of like the fish swimming in water. Like, what water? I don’t feel any water. And, the first season of It’s Seeing White was really helpful for me to understand the effects and benefits of just being a white guy. And, the second season has been extremely helpful for me to understand what it’s like to be a guy in the patriarchy. So, I think those are really good listening. I’ve been recommending them to everybody and it popped into my mind. So, There’s a non-DevOps-related checkout.

Matty: [00:48:41] We will put a link to that in the show notes. I have just one that I alluded to a little bit earlier in the episode, but I think it’s a great one to check out. I had been tasked with writing a review and a summary of this talk, and so I’ve been focusing on it, but John Ausbaugh gave a great talk at PagerDuty Summit this year called Incidents as We Imagine Them versus How They Actually Are. And it has a lot to think about the reality of we want things to fit into nice little boxes and understand them. But the reality is these things are all very messy and thinking about the questioning and the learning. So we’ll put a link to that in the show notes as well.

Jessica: I should have a checkout too. So I’ll just tell you the last paper that I read and I found really fascinating, which was Karl Popper on quantum mechanics without the observer. It’s from like the ’50s or something. And that was super interesting. About propensity theory and distributions and how we really can test those. But the interesting bit is defining what is a reproduction, is defining the significant parameters to an experiment, which means defining a situation, which gets back to the DevOps is a situation thing, because I’ve got situations on the brain now. Because the question is, can you find the essential elements of a situation that mean you’re doing DevOps and therefore have a propensity for better delivery. Okay. To wrap up the show, please head over to arresteddevops.com/devops-database for this episode’s show notes, and the site also has our newsletter. Find all the Arrested DevOps stuff you could ever want. Visit arresteddevops.com/itunes and leave us a review in the iTunes Store if you want to help other people find this amazing podcast.

Matty: [00:50:27] Thank you very much, Baron, for joining us today. And, you know, and thank you for being such a great first-time host, Jessica. I really, really enjoyed having you be the host. You showed me up and showed us how it’s done.

Jessica: I appreciate you putting that in the script so I knew you were going to tell me I was great before we started the show.

Matty: Right, right. Yep, because sometimes we act professional. So thank you again, Baron.

Jessica: Baron, tell us where people can find you.

Matty: Oh yeah.

Baron: My online handle is @xaprb, which if you’re curious is what happens when you type Baron on a Dvorak keyboard. That’s me on Twitter and my website and blog are xaprb.com. Awesome.

Matty: I’m Matty @MattStratton.

Jessica: I’m Jessica @Jessitron. This is Arrested DevOps and remember, There’s always DevOps in the banana stand.