The Database: The Elephant in the Room

Matty: [00:00:07] Welcome to Arrested DevOps, episode 26, Database: The Elephant in the Room. I’m your co-host, Matt Stratton, @MattStratton on Twitter.

Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter. Arrested DevOps is brought to you by TenthMagnitude, a cloud services company that figures if you are listening to this podcast, then you’re Pretty cool. You can find out more about joining their cloud services team at tenthmagnitude.com.

Matty: This episode is sponsored by PagerDuty. PagerDuty eliminates the noise, chaos, and manual processes across the entire incident lifecycle to decrease resolution time. PagerDuty is trusted by companies like Etsy, Nike, and GitHub. To sign up for your own free 14-day trial, visit arresteddevops.com/pagerduty.

Trevor: This episode is also sponsored by Red Gate Software. Redgate makes the tools that bring the benefits of continuous delivery, safe releases, efficient development, and fast feedback to your database. Find out more about database lifecycle management and download free trials at arresteddevops.com/redgate.

Matty: [00:01:11] If you’re listening or watching this live, please feel free to share your questions and input about databases and DevOps and all the things we’re going to talk about. You can hit us up on Twitter @ArrestedDevOps, or we’re always on IRC on freenode.org. In #ArrestedDevOps.

Trevor: Data. We can’t have applications or services without it. If software is eating the world, then the data is the maître d’. However, it can be challenging to incorporate these database design and releases into the world of continuous delivery and DevOps. We’re joined today by Grant Fritchey and Jonathan Hickford for a discussion of DevOps and databases. Grant, you want to introduce yourself?

Grant: Sure thing. My name is Grant Fritchey. I am the Scary DBA. I work for Redgate Software. I’m the product evangelist there. My background is I’ve been a full-time, large-scale enterprise-level DBA for about 20 years. Before that, I was in development. I think the biggest shock I ever had when I became a DBA was the fact that they didn’t put databases in source control. That really freaked me out. Thank God we’ve got it fixed.

Matty: [00:02:14] Awesome. Thanks, Grant. And then, John, can you introduce yourself and tell us a little bit about your background with databases and the DevOps?

Jonathan: Hi, yeah, I’m John Hickford, and I’m a product manager at Redgate, and I work on the tools that we produce to help people bring the benefits of continuous delivery to the database, source control and integrations and deployments. Before my time at Redgate, I worked in a development team that was, well, quite literally a DevOps team. We wrote code, we were responsible for that code and looking after it when it was in production as well. So I’ve had a reasonable amount of experience both in terms of trying to write code, deliver features fast, and also to look after what’s happening in production as well.

Matty: Let’s get started thinking about— I want to talk both in terms of continuous delivery about databases, but just in terms of a DevOps environment. What do you think when we think about releasing database changes, DevOps or no, just in general? What are some of the major issues that come up to people when they’re trying to do a database release?

Grant: [00:03:17] Number one, it’s data persistence. The fact of the matter is, is that releasing software, while not easy, don’t get me wrong, is very straightforward by and large. You take an old piece of software, you throw it away, put the new piece of software in, you’re done. Databases, we could do the same thing, but then the phone starts ringing because the business is freaking out because there’s no more data left in the system. I would say our number one issue and the thing that causes all of the problems is the fact that we have to persist the data. It cannot go away. And that just— It changes what we can do. It changes how we can do it. And it’s the elephant in the room, as you guys already pointed out.

Jonathan: There are also challenges, I think, around the fact that the data comes from multiple places. So it might be that some data comes from the production system, so data that’s perhaps generated by users. But there’s also configuration data, data that’s coming in from development. And quite often, all of this data is flowing in the opposite direction to the way your code changes are flowing in a pipeline. So whether as code flows from development to production, data is quite often flowing in both directions. That makes it really hard when you’re making a deployment, or even if you’re undoing a deployment, you’ve really got to think about where that data’s coming from.

Matty: [00:04:28] That’s true. It’s more than just schema. I mean, I think that’s, that’s a challenge I know I’ve encountered helping with release applications and work like that, whereas this, this idea that a database change is not just as simple as, okay, well, now I’m putting a new schema in place or adding some SPROCs, right? You know, it’s the information in it, you know, and how do you version that, right? Exactly. It’s kind of always, it’s always rolling, right? The parts are always moving.

Jonathan: It’s always changing, and it’s not always at the same place as trying to read from it as well. Quite often, your database might be being accessed by multiple applications. You might have BI jobs, and if you make a change to your database, you’ve got to be, you’ve got to be really confident that you’re not going to break something that that depends on how that database used to look or the way the data used to be stored.

Grant: Or even the fact that things are on the fly. I mean, stuff’s running actively and you’ve got data changes coming in and you’re, say, adding a not null column and you’ve got to do data migrations as part of the deployment. That also has an impact on the production system, can cause downtime, slowness, all kinds of other issues. It just keeps boiling back around to the fact that because of persistence, data is a difficult thing to deal with. I mean, I think at least part of the whole push for NoSQL and, you know, these ID-value pair non-structured storage, while there’s some excellent, excellent, excellent reasons for doing those things and I’m in favor of them, one driving reason is this difficulty around database deployments. If you have an unstructured system, heck, just put in whatever you want and off you go. It just— but then you lead into other issues that where you may be putting in an unstructured system in a place where you really kind of need structured data.

Jonathan: [00:06:09] It’s very true, and lots of the same challenges apply regardless of whether it’s a NoSQL or a SQL database in terms of making sure that those changes are, you know, being tested on the way and that you’re not just finding out about problems in this kind of last mile.

Grant: Right. I mean, that’s a pretty common thing. You’ll see a lot of DBAs prefer this method, and it makes me insane as a DBA, is that, oh, well, you know, I’ll just look at the script the day before it goes to production. And it’ll be fine. You’re telling me the first time you’re deploying the script, the very first time, is in your production environment. Good job. I’m sure nothing will go wrong.

Matty: This is the thing, like, when I think back to the olden days of, like, 5 years ago, before I had started to learn DevOps and knew what it was like, that was one of the things that I’d run into is, you know, the DBAs, their job was— so when we would do releases and they would execute these scripts that were part of the release, it would be like, okay, here’s our document of how do we release this version of the application. Okay, TechOps, copy these 3,000 files here and there and change this registry key and do this thing. And the DBAs are going to run the— here’s the scripts. And those scripts would be put into version control, so yay that, right? But then things would happen and we’d kind of talk to the DBAs and they’d be like, they’re like, well, the developers gave me the script, so my job is to run it. And I’m like, if you didn’t look at them and understand what they did, then why was there any value in you pushing the button other than the dev, right? Like, where’s that? And I think that’s kind of the beginnings of like, how do we get our DB ops folks like integrated into this more cross-functional thing? Because otherwise it doesn’t— I don’t want to say they’re not providing value, but in the release part, it’s actually not providing a lot of value if you’re just saying, well, because I have a different title, I push the button.

Grant: [00:07:58] Well, I mean, I would say they’re not providing value. I mean, and I’m saying this as a DBA. I’m not throwing— I mean, if I’m throwing rocks, I’m throwing rocks at myself. And my people, so it’s OK. It, it, the thing that I found, and, and I found this out the hard way, right, is we had the setup where we would like, oh, you know, I’ll do the review before it goes to production. And so it would be Thursday, I’ll get the script; Friday, we’re going to production. And Thursday, I’m going, guys, this is not going to work. Here’s why. Here’s all the things we have to change. So we’re not going to go to production tomorrow.

Matty: And they say, too late, Grant!

Grant: Yes, exactly! You know, and it’s, You literally can’t stop a deployment. You can’t stop it the day before it’s going to go, right? It’s going. You’re— get on board, baby, because we’re going. So what you have to stop is bad development, which means getting into the whole DevOps thing, making that step into the development system, going to the scrums, going to the standups, you know, know what’s happening, you know, 6 months before you’re going to production, 6 weeks before, 6 days before, not 6 hours.

Jonathan: [00:09:05] Especially, I think, in a continuous delivery system, the sooner you can get that feedback, the better. What you— you know, the core principle of continuous delivery is getting that fast feedback, right? And the last thing you want is to find out about a data problem, as you say, before it goes into production, especially if that change could be made in development and tested and run multiple times before it even gets to production. That way you can be really quite confident that Your production deployment, I mean, your production deployment should be boring. There should be no surprises. It should just happen. You should be really confident that everything that you have, you know, can possibly have done has happened beforehand. And it is hard. It’s really difficult doing that, especially with the data.

Grant: I’m going to steal that and use it in my presentations from now on. Everyone always asks me, you know, how was your plane flight?

Matty: Boring.

Grant: You know, because you don’t want an exciting plane flight. How was your production deployment? Boring? Because you don’t want us in exciting production deployment.

Matty: You know, it goes back to, right, the same thing. When something’s painful, do it more often, because that’ll make it stop— make us want to make it stop being painful. The thing is, again, and I’ll kind of, you know, going back to, like, you know, what Grant was saying before about how in some ways, like, releasing applications is simple or simpler. Than releasing database. So then I think what we do is we shy away from this. And I mean, this even came up— we did an episode 15, which was our continuous delivery episode with, with Jez Humble. And he even said, he’s like, this is the thing that we kind of don’t talk about a lot. And he’s like— and we’re talking about like, even Facebook doesn’t really tell you how they do it. And that’s— they’ll tell you everything else, but then it’s kind of like Facebook DBAs are markedly quiet, you know. So there’s not a lot as much conversation around this, and that I think is why people want to hear what we are are looking to talk about.

Trevor: [00:10:57] Grant, when you started introducing yourself, you talked about getting your database into source control, and when we talk about continuous delivery and source control, when we’re pushing code changes to the database, you know, data is special. What are we forgetting? How do we get there?

Grant: Well, I mean, step 1, and this sounds bad coming from a tool vendor, you’ve got to get tooling around it. Now, there’s a lot of other tools. You don’t, you know, you don’t need ours. There’s other ways to do this, but you’ve got to put tooling in place. There’s not a— I cannot fathom a manual method for getting a database under source control. You’ve got to have some tooling in place. The main reason being there has to be the ability to generate differential scripts, scripts that will make changes to your objects in a way that’s intelligent. It’s not You can’t simply have create scripts and then rerun those over and over again. There’s no way to make that happen. You’ve got to have an alter script. You’ve got to have, you know, migration scripts that move the data around, and then you’ve got to have a mechanism to automate putting all these together. And the main way you’re going to get that done is tooling. So yeah, you’re going to look at, you know, us. You’re going to look at other vendors, but you’ve got to have that ability not only to get the database into source control in some fashion, whatever that might be, but then to get it out of source control in an automated fashion to your multiple environments so you can go from version 1.0 to 2.0 in a meaningful way, not constantly creating 1.0 over and over and over again. I mean, one of the things I— my main pushback on ORM tools, for example, is not the code they generate, although there is some instances where that code is heinous and evil. My main pushback on them is the fact that they’re only able to release 1.0 style code. Most of the time their DDL, their database definition language stuff, is create table. Cool, but if we’ve got data already in existence, then we have to persist that. We can’t let it go. You’ve got to give me an alter table or, and/or, you know, create a new table, migrate the data over, recreate the data, and you get the ORM guys going, well, here, Right? Our DBAs handle it, right? That’s the answer I’ve heard so often. And so that’s my big pushback there is that we’re not then using the tools we have available and we’re not then following repeatable processes that are going to allow us to automate things. And it’s capturing those changes through source control and then having that tool which allows you to generate that migration script that makes all of this possible, allows you to move into the direction of automated deployments. I mean, and there’s a whole slew of detail around that we can talk about if you want. You know, pre-deployment, post-deployment scripts, dealing with security, you know, and then the testing automation to ensure that you’ve got, you know, you think you had 100 rows, you still have 100 rows, that kind of thing. I mean, it just goes on and on and on.

Jonathan: [00:14:05] Yeah, I think, I mean, I completely agree with Grant there in that the foundation for all of this is getting, getting your database into source control, because once you’re in source control, you can then start to repeat things. And once— if you can automate that process, then you, then you can really focus on making those changes in a way which is going to have the least impact into production. You can start thinking about writing changes in the same way that you deploy them into production, in the same way that you deploy them into your local development environment and then to a test environment. And that way you get to test these changes, these these quite complicated data changes, you might be— say you’re refactoring a database. You want to make sure that when you get to production, that’s not going to take 3 hours to run if you’re running over millions of rows of data. And that’s where the continuous delivery patterns and practices and really thinking about how you’re going to make that data change, how can you do it in a way that’s going to keep downtime to minimum or maybe even zero.

Grant: Well, or even knowing that, hey, if I do this change, it’s going to take 3 hours, and I just need to know that upfront so that I can schedule and deal with it accordingly. I mean, it comes back down to knowledge. I mean, coming from a known state, source control, going to, you know, the only other known state you have, your production system, you’ve got to have a mechanism for making those migrations smooth, right? You’ve got to reduce the pain. You can’t eliminate it, but it has to be smoothed over.

Matty: [00:15:32] And I think especially when we think about that, because you kind of get into this idea of the myth of rollback, right? Like, it’s happening. You know, and we don’t have this magic, like, push the DeLorean 88 miles an hour, and we can— that’s the only way you can roll back a change in my mind, right?

Grant: Yes.

Matty: You know, get back around. And so all the more reason that knowing you can’t go back and you’re always going forward, you know, so we think about, you know, Grant, you think about saying, like, we need to be continually testing our deployment so that it’s boring. And then part of that is testing, like, back before I, you know, when I believed that I could go back in time, I would say, well, we need to test our backout, right? Because that’s part of it. That’s part of it saying, well, what happens if shit’s on fire? What do we do? And of course, we never did that. We just talked about it but ran out of time, as one does. But part of that— but that’s the beauty of CD, right? Things are going to be on fire while you’re releasing them, and then you’re going to find them early. You’re going to say, oh crap, when we push this, it blows up. But the answer isn’t, okay, well, let me just go recreate the QA environment all over again. I’ll reimage it and start it over, because you’re not going to be able to do that with prod. I just kind of want to think about that, about how do we test our deployments, and then also— and then that kind of goes probably part and parcel into automated testing of data changes too. You know, if different pieces of my app are being released at different times, how can I focus on data testing in an automated way?

Grant: [00:17:10] The biggest thing about, the biggest thing about like the rollbacks is that you’re 100% correct. You’ve got 2 methods of rollback that are actually going to work, and one is to either, you know, do a restore or undo a snapshot. You know, of stuff that immediately after you’ve done the release, oh my gosh, everything’s died, let’s undo what we did. Or the myth is, is that you have this script that can, you know, target bits and pieces to roll them back, and that’s just such a lie because the data comes in over time and ain’t nobody wants to get rid of it. So you’ve got to have the ability to just redeploy again. So that’s what— that’s why DevOps automated deployment continuous delivery is such a huge when is because the change is going to happen anyway. You’re going to make those mistakes. So you just incorporate that same process that got you to where you’re going to fix the mistakes if there were any. Talking about your QA and stuff, the, the one thing that you’ve got to do, and, and this is hard, is you’ve got to make your test environments as close to prod as you can. Now, I’ll grant you, you’ve got 15 terabyte prod system. You’re not going to have 7 or 8 different copies of that scattered across your dev environment for testing. Ain’t gonna happen. But you make it as close as possible. You, you simulate it. If it’s not exactly a copy, it’s a close copy. It’s got clean data in it, right? Get rid of the emails, get rid of the personal stuff, you know, so you’re not gonna— HIPAA is not gonna come bring the cops down on you. But you make a copy of production, you test it as close as you can to production, and then you do that over and over and over again in your test environments. Yeah, I think sometimes it’s okay to reset QA and go, wow, that burned down. Let’s hit the button that automatically refreshes our simulation of production back down to our QA environment and then figure out what the heck we did for that deployment to make it deploy correctly this time.

Matty: [00:19:05] Yeah, I think it just can’t be that that’s our only way that we solve it because then we’ll be screwed in prod. I also wanted to talk about this idea of replicating the data and production looking like every other stage. So like, again, you know, going back to the, the humble Farley Bible, right, it’s every stage of your pipeline is exactly like production except for scale. And then that makes sense for data too, right? Like, so it should look the same, but scale comes into play. And I think about a former life of mine where we had, you know, a 3.5 terabyte data warehouse when I was first getting started. And then we had, because of explosion of environments, had like 9 product teams that each had a dev and a QA, you know, so I had like 20-some environments I had to manage. And then it was like, well, we can’t possibly do this because we can’t test everything everywhere because we can’t put this whole data warehouse everywhere. And I was like, well, isn’t there a way you can trim it down? Or even the transactional database was fairly large, you know, and multiplied times however. And I kept saying, you guys don’t have to have the entire transactional database to get the functional point of that. And then it was, you know, my favorite meme lately is the whole, like, 2 Lego dudes pulling the cart with the square wheel and someone has a round wheel and says, this is better. And they say, we’re too busy, you know? And that was sort of the thing. It’s like, well, we don’t have any time to, like, figure out how to do test data because we’re so busy, like, releasing stuff. And I’m like, yeah, and it’s all breaking. So, but I think that’s the problem is that we think, well, I need to make it look exactly like prod and you’re never going to get it exact. Maybe your load testing environment probably has a size that’s similar? Because again, you could run into that where like, hey, my dev database was like 30 gig or like, like, you know, 800 meg, and then somehow it gets into this, you know, 7 terabyte transactional database in prod and it’s, it’s not performant. I don’t know why.

Jonathan: [00:20:52] I think the key with your, the key with your pipeline is to try and test. You don’t necessarily have to test everything in the same place, so you don’t need, uh, you don’t need every environment to be an exact copy of production, but I think your load testing example is great. You want somewhere where you test for size, and you’re testing for the quantity of data. You want somewhere where you’re testing for the complexity of data. Maybe that’s a QA system. And then in systems that you need to make fast iterative changes on, you probably need less data there. But you want enough to try and catch things fast. There’s some great— there’s actually some great diagrams. If you search for visualizations of continuous delivery, there’s a set of drawings that an engineer from Spotify did around Jez Humble’s book. And there’s a really fantastic one around managing data and what size, you know, what types of data you might want to have in each environment. And there, again, there are tools out there that can help to some extent in terms of either subsetting data or generating data that looks like, looks like your production data or masking data. But you want to try and— you’re absolutely right, you may not be able to have that perfect copy of production everywhere you want it. But there are things you can do to try and simulate that. I suppose as well as data, there’s, there’s a tendency perhaps with the database that there are some things that you only try when you get to production. So things like, you know, sharding. If you’re sharding your database or if you’re replicating your database, you want to try and test those somewhere in the pipeline so that it’s not a surprise when you get to production and find that there’s a problem when you try and replicate the data or shard it. And there are ways that you can try and do little pieces of the puzzle as you go. I think it’s the short, fat pipeline rather than the long, thin pipeline is better.

Matty: [00:22:39] How do you— this question came up in IRC, which was— and I think it goes right into what you were just saying, you know, about there’s things you have to try in production just because you can’t emulate certain things, right? So in AppCode, we talk about blue-green, right? You know, we’re like, oh, cool, okay. I’ll bring up a little bit over here and a little bit— and I’m going to send 10% of my traffic to this one web server that’s running the new code and everything. But data changes can kind of squirrel this all up, both in terms of letting— yeah, how do you kind of blue-green a database change?

Jonathan: So I think it is possible to do this. And it is thinking a bit about the patterns by which you make a change. So I know an example might be trying to split a column or split a table when you’ve got terabytes of data, you’re not going to be able to do that in a single go. That’s going to take a long time. But if you can build applications or data access layers or abstraction that’s going to let you deal with both versions of this data, then you’ve got a chance to run this, test this in a few servers, test this in a few different places. Maybe you synchronize that data slowly out of band at quiet times, or maybe you slowly roll it out to more and more people. But it is about— I suppose you’re right. It’s around thinking about how you can make a change perhaps in a different way to how you might make the change if you weren’t doing continuous delivery or an automated deployment system?

Matty: [00:24:06] I think it’s hard to sometimes convince— or I guess it’s changing of thinking— but convince folks that, like, because some of this comes off, like, as tech stories, right? Like, hey, I’m going to build this shim web service that helps me get this change out there, and then we’re going to throw it away. And that’s an agile thing, too, right? It’s like, hey, you may do work that gets refactored, or that only exists to get you over a hump. And I know sometimes within product groups that can be really hard. They’re like, wait a minute, you know, you’re in there, you’re doing your Scrum, what’d you do today? Well, I, you know, was working on blah blah blah web service which abstracts data layers so that we can do this. Well, wait a minute, why are you doing that? We have this user story that has to be done. And so I think it’s getting this— and that’s getting into that whole like cross-functional thing, that it’s okay to have work that gets thrown away if it’s accomplishing something better. And like, shims are like that, right? They don’t— they’re not an ongoing thing. But you have to think, you have to think that it’s okay to build a bridge that eventually the stream’s not going to be there anymore, so you don’t care about the bridge.

Jonathan: [00:25:06] I guess it’s a— it is one of those kind of continuous delivery and DevOps views where, you know, you’re— where you are embracing a view where it is more important to be able to release. And that’s, you know, that at any time is more important than what you are able to release. So it’s more important that you can make that change then perhaps you make, you know, 10 changes really, really quickly. But if that leaves you in a state where you’re not able to deploy, then you’ve kind of strayed a little bit from, from that core principle of being able to release at any moment and every change is good enough to go.

Trevor: So speaking of releases, as a developer, we’ve got the kind of the concept of the blue-green release of application code. How can I accomplish that with database? Schema changes?

Grant: Badly.

Trevor: That’s a good answer.

Grant: I’ve tried doing that. I’ll be honest, I have never really successfully done that outside of a— I’ve done it for data marts and for reporting systems where I can control the data as it comes in directly. It’s actually pretty easy to do. But in OLTP systems where the, where the data is live, the data is messy, and you’re going to try to maintain 2 copies of it either in tables or in different databases, you’re either setting up replication, you’re setting up triggers, and you’re digging a deep dark hole. And best of luck to you. Honestly, I don’t, I don’t have a good answer on that one. I have just found the blue-green AB stuff to be problematic, and problematic to the point of serious pain, not problematic of like, oh golly, I have to work. I don’t know, Jonathan, unless Jonathan’s got some knowledge that I don’t, I’d be more than interested to learn.

Jonathan: [00:27:03] No, no, Grant, this is definitely where I think if your DBA is saying that the database is perhaps not the area to try and tackle that problem, they’re probably right. So if— I guess there are— there may be— blue-green deployments are obviously possible, and storing the data during them is obviously possible as well. But trying to tackle that specifically on your database might not be the easiest battle to try and work.

Grant: It is hard. I mean, I’ve done it— again, I’ve done it with triggers where you set up triggers so that it maintains 2 copies of the data, maintains it in 2 separate tables while the rollout’s going on. And so the new stuff is going against the newer table structure. Anything that comes into the old stuff gets triggered over to the new. Same thing for the new stuff. Gets triggered over the old. It’s just a, it’s a massive undertaking and fraught with horrific danger. I mean, it’s just really, it’s Indiana Jones, man. And most of us would not survive in the fridge when the nuclear bomb went off. I’m telling you.

Jonathan: And there are probably ways to achieve the same thing. Like, if your business goal is to be able to switch between 2 versions through this blue-green process, then there are probably ways to do that whilst you’re still using the same database at the backend. So it could be that version schemas or some way of switching between 2 different schemas without physically switching your data storage and your data persistence layer behind the scenes, as Grant says, which is phenomenally difficult.

Matty: [00:28:27] So I know that you 2 obviously spend a lot of time with DBAs, data architects, people who are into all this data stuff. What’s that side of the industry or what’s that area? What do they feel with this whole DevOps thing? Are they buying into all this hippie-dippy bullshit?

Grant: It is a seriously mixed bag. I think that there’s a number of camps coming out of it. There’s one camp that— one that upsets me the most is the camp that like, oh man, no, nothing’s wrong. Everything’s fine. We do our deployments this way and it’s all— we’ve always done our deployments this way and they just work and it’s all great.

Matty: We have always been at war with East Asia.

Grant: Yes, absolutely. And, you know, I think that that group of DBAs and data architects, because I’ve seen it from both, their heads are in the sand and they’re going to get burned and they may get burned over and over again. They may be getting actively burned and they’re just, they’re so used to the pain that they’re not doing anything about it. Those guys are, it’s just hard. You’ve got to convince them that they’re in pain and you’ve got to document their pain and show it to them and, you know, and then say, and by the way, You don’t need to live with this pain. Here’s a nice pill that will fix it for you. But other than that, there’s a lot of, there’s a lot of people coming on board. A lot of people are starting to be very aware of it. It is growing. There’s a lot of cognizance of, you know, wow, yeah, we’re really doing this really badly, and the developers hate us, and I’m not enjoying my life, and every time we have to do a deployment, it becomes this, you know, 15-hour nightmare that I’m in the office all day Friday and into Saturday. And it’s like, you know, is there a pill that could help me with this? And it’s catching on. It’s just also the other hard part is you say, okay, cool, now you need to go talk to your dev team. And people, I’ve seen a lot of people just freak out because it’s like, I don’t want to talk to those guys, they’re crazy. Or worse yet, the dev team’s kind of going, I don’t want to talk to the DBA because he’s a jerk.

Trevor: [00:30:29] And usually the DBA is, but So what kind of documentation are you talking about here to kind of convince them to take the red pill and not the blue pill?

Grant: You’ve got to document how long did a deployment take, was there downtime, was there data loss, how long did recovery from the downtime take, how long did recovery from the data loss take, how much time was spent preparing for this deployment specifically in and around your production server? How much time was spent recovering from this deployment in and around your production server? Not even getting into the dev, QA, and all the other servers and whether or not you’ve done manual processes through those, which frequently is occurring anyway. But, but just let’s ignore all that for a moment and just talk about the pain in production and, and document all that. Then On top of that, to make the dev team push for this more, document how much pain the dev team has gone through to get, you know, to hop through whatever hoops and, you know, flaming circles that the DBAs have erected prior to their production environment and get that documentation on both sides. And so you can go up to management and go, look, here’s how much time you’re losing and here’s how much pain you’re losing. Here’s the amount of data we’ve lost. Here’s the productivity we’ve lost. We probably should be exploring some solutions around this area and looking at continuously improving.

Jonathan: [00:31:57] There’s a definite interest as well. If you— I’m thinking about the DevOps Days and the conferences I’ve been to over the last year. If I kind of count how many times in the Q&A session, you know, Q&A after a session, someone will ask a question about the data. Maybe it’s someone talking about how they’re using Docker or how they’ve set up infrastructure as a service, and then there’ll be someone asking, what are you doing about the data? Which usually sparks a whole discussion. So I think there is a real interest in both directions, you know, with a dose perhaps of healthy skepticism. There’s a real interest both from, I think, people involved in DevOps and from DBAs into making this work. And as Grant says, there’s a huge amount of pain, and that pain’s getting worse as businesses become more agile. So, you know, maybe once you’re expected to release once a year or once every 6 months or once a quarter. But so many institutions, so many places now, it’s becoming important to be able to release monthly, maybe even weekly. And if you can— maybe you can take that pain once a month, but can you take it every week? It’s unlikely. Most people are really, really— the need to change fast now is so important. And if the data is the bit that’s the bottleneck, now it’s where people are starting to look and starting to focus. What are the patterns? What are the practices? What are the tools that we that we can use here to make this easier. It is still a new world. This is a world of innovators and early adopters still.

Grant: [00:33:27] The one interesting thing is that, speaking as a DBA and speaking as a completely lazy bastard, I’m taking advantage of all the work that development teams have done. You guys have figured out all kinds of cool stuff. I mean, this whole— your management of the source control systems, labeling, branching, You know, the way the releases are controlled. This is all stuff that I would have to go out and try to invent if I wanted to release my database faster, but I don’t have to because all I do is I, you know, I pick up Jez Humble’s book, I pick up some of the other books, I read through them, I understand what it is you’re doing, why you’re doing it, and then I say, okay, well, how can I do this inside the database? And then I start working that way, and it’s just, it’s down to more to labor than thought. I don’t have to think about what to do or how to do it. You guys have already done that for me. I’m just, I’m just gonna go in and do the labor to make it happen. It’s actually a pretty cool situation to be in. You guys have blazed the trail. You guys have laid out the path. I mean, I just got to walk down it and make sure my stuff works too, which isn’t that hard.

Jonathan: [00:34:31] Not easy, but it’s not—

Matty: So the other side of that, right? So thinking about from the DevOps, you know, like we kind of talked a lot about that, you know, there’s automate, you know, a lot of this is going to help, these are the magic pills, right? You’re like, okay, well, the things that suck and then automate, you know, there’s automation and things like CD and things can help with that. But then if we kind of do talk a little more about kind of the reaction and thinking about, like, incorporating the data side of the house into that squishy side, right, and thinking about the, you know, the cross-functional pieces of that, because I think there’s a lot of similarity between operations and kind of the data team, right? In that there is probably not enough people. There are definitely more app devs in your organization than there are DBAs, than there are tech ops. So one of the classic challenges, I think, that comes up with the idea of being cross-functional is that you can’t sort of say, like, okay, in this product team, this is your dedicated DBA, you know, so are you seeing folks being more involving these DataOps teams the same way that they’re involving system administration? Because that’s— we kind of say, well, it’s called DevOps, but really it’s Dev/QA, DataSec, BizOps, right? You know, how are organizations bringing the data people to the table?

Grant: [00:35:53] Well, what I’ve seen— I mean, because I started doing this before there was tooling for it. I mean, we had a— I’ve been working in this— I don’t want to say DevOps, but I’ve been working across this sort of, let’s automate our build processes for about 10 years now because we were, we were in really severe pain and, and were quite aware of it. And so what we had to do was automate the heck out of everything in order for me to be able to support 10 different development teams, which, which was a nightmare, I will grant you. But by automating stuff so that we had, you know, automated builds, automated deployments, most of the testing could get done without my direct involvement. I let my dev teams do a lot of T-SQL coding all on their own. I let them use Entity Framework, I let them use NHibernate, and I just had a review process where I could look at stuff as it was under development. So I only had to spend 15-20 minutes a day per team checking to see what they were doing. Then other than that, I spent my time making sure I got backups and making sure I’ve got protection on my production servers. And then, and then, you know, and then I would only be called in at, you know, on, oh my God, this— we ran this query in dev against one row. It took 5 minutes. What the heck? You know, please help us. And that would be the only time I get directly involved except for, you know, scrums and standups so that I knew what was happening with the team, you know, ahead of time. But it is doable, but it’s all about— it all comes back down to automation. And management and a management of expectations, right? Because you’re not going to get the, you know, here’s your DBA full-time, talk to you later. That ain’t going to happen. Jonathan, you’ve probably seen other stuff that I haven’t.

Jonathan: [00:37:38] Yeah, you see some. I’ve been really lucky in that this year I’ve been out and about quite a lot visiting people who are trying to set up processes. It’s the visits that I’ve been on where you end up with all the right people in the room. You’ve got a DBA, you’ve got a developer, and you’ve got someone from the systems administration side of things, and either they’re trying to solve a problem or they’re showing you an amazing solution they’ve come up with. And I’m always blown away by the way that the things that people have set up when a team of people have got together and really tried to solve this, phenomenally clever solutions and people’s stories about how painful things used to be and how they all— this problem has been solved now. But it needs all of those viewpoints. You need people who are trying to get changes out quickly, you need people who are trying to safeguard the data, and you need people who are trying to build a process that can do all of this. And it is a combination of— it’s a combination of those 3 things that’s going to create a good, reliable process and let your business ship quickly and safely.

Matty: [00:38:45] I think something that can help with this, too, because I just sort of had this thought that In a lot of places, or places I’ve been, I can see that in some ways your data team, they’re almost like doing DevOps before there was DevOps in terms of what we think about if we talk about DevOps being the, like, hey, dev needs to give a shit about production and know about systems a little, and sysadmins have to understand a little how the application works. Well, probably a lot of DBAs are going, yeah, whatever. I’ve been doing that for my whole life. I know how to— I’ve got to talk like hardware to the sysadmins, but I also have to understand how the app works. So some of it might be nothing new to them. But what I was thinking about, and this question kind of came up a little bit in IRC, the question, you know, was being able to come up with some kind of common tooling, because I think it’s hard if you have a whole dev team that’s using— and I’m just throwing it— that’s using Git for their source control, you know, whatever. They have a great thing around that, you And then you go and the data team says, I want to do this thing, but I can’t use our data tooling in that source control, so we’re going to put all the data stuff over here in Perforce because that works better for me. And now you’ve got these silos. And even within a tool, I remember there was a point, and I don’t— there was something with internal things about which group at Microsoft owned what, but it was a weird thing where, like, you couldn’t use, like, the BI Studio tool. Like, it had its own TFS piece. But it wouldn’t work with newer— it was a weird thing. We had this problem where our data developers couldn’t use the same source control that our app dev did. Then you really have— I mean, you have a silo that’s created by the tool. Is that getting better, do you think? It’s been a while since I’ve looked at this.

Grant: [00:40:30] I think it’s gotten a lot better, but I’m going to let Jonathan handle this one because he creates a lot more of these things than I do.

Jonathan: I think it is improving and it’s improving everywhere. I think the vendors who are tools vendors like Redgate are trying to be more agnostic. Regardless whether you’re— take Microsoft, for example, whose TFS system offers you perhaps a traditional linear source control system in TFS, but also supports Git. There’s a real drive, I think, for people to let people build these heterogeneous stacks that are best of breed and It’s more important that things kind of play with each other nicely now than it is that you just have, oh, you know, all of the database code has to be in this system. It’s got its own proprietary source control system. There’s, you know, the open standards are making a difference here in terms of letting people work in the same way. And the last thing you want to do is to introduce another silo, another division in this already quite sort of siloed and divisioned world. So if you can get everyone on the same source control system, that is by far, you know, that’s going to break down problems much quicker. You can start to make atomic commits that contain both database and application changes. This is, it’s probably a prerequisite for being able to do continuous delivery effectively. Other tools, I guess, like being able to see what’s happened is probably an important one as well. So regardless of whether you’re a DBA or a developer, being able to see you know, where changes are, what state your database is in, what, you know, what application version is deployed where. So this whole kind of category of monitoring and release management is really important. Getting people using the same tools gets people talking to each other.

Grant: [00:42:21] Yeah, I mean, until we had everything in, inside of a, a common folder, um, inside of source control, you know, in TFS, whatever, it, it, um, we ran into lots and lots of problems because it was, it required human interactivity to say, now which version are you on? And which version of the database does that hook to? And okay, let’s deploy. Oh wait, no, I’m sorry, that didn’t work. Let’s— it’s the other version. Try this one. Yeah, that didn’t work. Go see what else you got. You know, it’s— if they’re all branching, they’re all labeling, they’re all versioning together, then, then it makes this, you know, again, not easy, but certainly easier.

Trevor: So we’ve kind of talked a bit about, about using source control, about using different ORM tools, migrations, and now monitoring. Are there any other kind of pieces to, to solving DevOps in the database that we haven’t discussed?

Jonathan: I think testing is a really important part of this in the same way that, you know, testing is important within your application, making sure you can test your database as well. So be that manual testing or integration testing, unit testing, static analysis tests as well. There’s— it’s really important to get to get as much automation in there as possible. And that takes time. So that’s, that’s not going to be the benefit that you get on day 1 of implementing an automated pipeline. But once you’ve got an automated pipeline, you can start to improve it and adding some testing in there. And there’s some great open source tools. So SQL Server, there’s T-SQL-T. There’s DBUnit in Java. There are ways of kind of testing your database. And I’ll let Grant talk about, you know, the the benefits, you know, there are huge benefits you get like that from a, I guess, from a sort of a product point of view. There are huge benefits in that you know that you’re not introducing regressions, but then you can also build, you know, incredibly clever tests to ensure that your data is being looked after as well.

Grant: [00:44:16] Well, and the trick there is that you can’t look at the database the same way you look at code, right, where you do the, you know, test-first development. You’re not going to create a test for your table.

Jonathan: Right?

Grant: It ain’t happening. You put a table in, that’s cool. You might create unit tests for stored procedures, but the more important tests you’re going to be looking at are when I’m making changes that are potentially destructive to data, I need to have a test built that’s going to validate that stuff and ensure that whenever I make the migration that it moved the same amount of rows I expected it to move, that the updated data looks like the updated data I expected to see. Having the ability to automate that stuff, again, T-SQLT, Sebastian Meinies’ massive work in T-SQLT, that’s a huge plus right there. It’s the benefit of assurance, it’s the benefit of confidence so that again, boring deployments to production. But that’s only going to come around if you’re making those choices to test those things that are potentially dangerous. If you don’t do the testing, and you’re just gonna— you’re crossing your fingers and running. So it, you know, it’s not, it’s not a good approach. And, and it doesn’t reflect well back to the, you know, just the standard stuff around DevOps, the standard stuff around lifecycle management. It all requires testing to be built in. So you need to make sure that that’s a part of what you do. Is that extra labor on top of what you do? Yes. But is that how you ensure that you’ve got the ability to deliver smoothly to production? Yes. So if you’re, you know, if you’re a good DBA, your main focus is protection of my production environment. Well, one of the best things you can do to ensure you’ve got good protection is make sure you’ve got decent testing of the scripts and processes that are coming in.

Jonathan: [00:46:08] On top of that as well, I think there’s monitoring. So your production database once you’ve made that change, you need to see what happens next. So does, you know, the changes that I introduced this morning, are they improving performance? Are they making it worse? Have, have we forgotten something? Have we lost something during this time?

Grant: Right.

Jonathan: So once that deployment’s gone, you can’t just leave it. And that in particular, I guess, is an aspect of being a DBA or working in operations that perhaps when you’re a developer you forget about. And in a DevOps team Continuing to monitor that performance and knowing, is my application getting more and more performant? Obviously, the presence of data can change the way that your application performs. It might work fantastically with a small quantity of data, but as it grows, things can really change. Just knowing how your database is performing can be a real indicator of how your application is healthy.

Grant: I mean, also, it does come back down to developer— I mean, not developers. DBAs have got to start expanding some of their skill sets. You do need to be looking at stuff like TeamCity, Jenkins, you know, some of the control tools for doing continuous integration because you don’t want to build that by hand. You want to start looking at stuff like Octopus where you’re going to be able to move stuff between environments and not only move between environments but have environment-specific settings to allow you to have security settings. You’ve also, as a DBA, you’ve got to start learning PowerShell. It is the— well, I’m sorry, at least for SQL Server DBAs. That is the tool for taking control over what you’re doing. Although I know some Oracle DBAs that don’t know shell scripts either, and they should also be picking up those kind of skill sets. In the DevOps world, I mean, and we’re all doing it whether you want to say you are or not, things are moving faster. You’ve got to have an expanded skill set. The fact that, you know, I know how to do backups, I know how to set up security, I know how to manage disk space. That’s great. We can only automate all of that. So what is your job again? We, we need, you know, expanded knowledge and expanded skill sets into further tools, into development, into all these other kinds of processes so that we can, you know, get together with you guys and meet in the middle, not, not have this, you know, we’re standing athwart the bridge stopping you from going to production. That doesn’t work.

Matty: [00:48:35] Yeah, I think that’s the core of DevOps right there. You know, every— you know, we think about cliché people, we think about whatever, but this comes up on Reddit, comes up in all kinds of places all the time, which is, what’s the thing I need to learn? Like, I want to be a DevOps, what do I need to learn? And it’s like, well, first of all, you’re not going to be a DevOps, but like, you need to learn a little bit of all the stuff, you know. So I think that was a really nice Nice wrap-up, Grant. So good timing, because it’s about time for us to move into our checkouts. I just wanted, John, I want to give you a last— anything you want to build on that about what’s the future of DevOps for data and DBAs?

Jonathan: I think for DBAs and developers, it’s the same. It is, as we say, it’s expanding what you care about and what you know about so that you cover more of this lifecycle. And it’s by understanding what other people are doing alongside the work that you’re perhaps more used to doing or more comfortable comfortable doing. As you gain that understanding, you’ll know who to talk to, how people can help you. And as you become more and more aware of the process, you can start to tailor what you’re doing to become— as a team, you become more efficient. And that’s the DevOps change. And it’s coming from all directions— developers, DBAs, sysadmins, from product management, for the business as a whole. People are learning to react to this much more agile world.

Matty: [00:49:55] Absolutely. I just want to add one thought, which is, as much as DBAs and data architects need to learn about the other stuff, I do think that’s something, and like you said, for developers, I think that’s the thing for sysadmins, like we really could get better and smarter about how data works. And it’s a lot of like everything else, I feel like we know more than we think we do when it comes to that. We’re aware of it, we just need to be able to understand and continually grow. So let’s move into our checkouts, thinking about continually growing with some new things. So Grant, what What type of— what do you have for our listeners to check out this week?

Grant: Okay, I’ve got 2 things. One, I’m digging Forza Horizon 2. It’s not quite as sophisticated in terms of simulation as the other Forzas are, but, you know, there’s nothing like launching a Lamborghini into the air, you know.

Matty: That’s pretty cool.

Grant: It’s a good way to relax.

Matty: Seriously delivering a V-flow into the atmosphere, right?

Trevor: Grant, did you ever play the Burnout series?

Grant: [00:50:56] Yes.

Jonathan: Yeah, yeah, yeah.

Grant: Burnout was a great game. The other thing I’ll point out is this, uh, is this novel called The Phoenix Project. It’s all about DevOps and IT. So yeah, it’s a novel. Relax. Don’t worry about work anymore.

Trevor: John, you want to go into your checkouts?

Jonathan: So, um, for me, it’s another, it’s another podcast, but don’t worry, it’s not in the DevOps space. Um, so I’m bummed.

Matty: I want more DevOps podcasts.

Jonathan: I’ve been listening to 99% Invisible. It’s a design podcast about kind of architecture and design. But I found it absolutely fascinating. So I definitely recommend that. And then in terms of the more database side of things, I’ve actually been rereading the book Database Refactorings, which is by Pramod Sadalage. And that’s in the Fowler series of books. And I strongly recommend that if you’re sort of thinking about how to make database changes in a in a continuous fashion. It’s an excellent read.

Matty: So I’ve got a couple, and I’m sure this first one is not anything new, but we all know that especially when it comes to anything related to games or gaming, I’m way behind. But I’ve just recently purchased Civilization Beyond Earth. I think it just recently came out for Mac. That might be why.

Trevor: [00:52:07] I think it was September it came out.

Matty: Came out, period? Like it was new in September?

Trevor: Yeah.

Matty: Like it’s a thing? Oh, okay.

Trevor: I still haven’t gotten it.

Matty: I’m not super behind the times. So I’ve been playing that. If you’re— it’s very much like, if we’re gonna kind of go old school, it’s— I’ve been playing, I’m kind of like, so this is Alpha Centauri.

Trevor: Well, that’s the whole idea.

Matty: Okay, so it’s the Civ V version of Alpha Centauri, basically.

Trevor: Alpha Centauri is also a Sid Meier game.

Matty: I know, I know. I didn’t mean he was ripping himself off. I just meant like, I was like, oh, it’s like Alpha Centauri, but it’s like updated for like Civ V gameplay. I’ve been liking it a lot. I also went back and because of this, re-bought Brave New World expansion for Civ V and was really kind of chuckling at apparently there’s a unit you can buy which is the XCOM Squad, which is a crossover to the XCOM franchise. I don’t know. But yeah, I’m really liking Beyond Earth. I haven’t beaten it yet. It’s pretty taxing on my 13-inch MacBook. I got to tell you. They’re pushing the boundaries there, but it’s been fun. Also, those of you who have little kids who are obnoxious this time of year, there’s an app on iOS. I’m sure there’s equivalent ones on every platform. This one is called Video Santa. Basically lets you quote-unquote FaceTime with Santa when they’re not behaving. And I used this last night and my 3 kids just sat there and stared and they all went right to bed. So I’m not sure how I feel about this, but lying— but telling your kids about Santa in the first place is kind of lying anyway, so that’s cool. And I have one final podcast suggestion, which people on this podcast— listeners may not care, but my latest podcast fascination is not Serial. I don’t have to tell everybody about Serial, you know about it. But it’s a podcast called The Gilmore Guys, and as a fan of Gilmore Girls, I’ve been enjoying this. It’s 2 guys, one of whom has seen the show, the other one is watching it for the first time. They’re both very, very funny. But if you don’t like the show, don’t bother listening to it. That’s also— they’re super young, and I sit there and I’m like, well, the reason you didn’t get that reference is because you’re 10. But that’s okay. Speaking of Trevor—

Trevor: [00:54:13] oh, oh, you’re gonna find a way to work that into every episode now, aren’t you? You’re 40 now, so— I was trying to figure out a way to work your birthday into this, but I figured nobody’s gonna listen to this close enough to your birthday for it to matter.

Matty: But hey, wait a minute, I don’t know what that’s supposed to mean.

Trevor: I mean that your birthday— there’s 5 people listening live right now, so, so everybody who’s listening should wish Matt a happy birthday on Twitter because it was yesterday. Anyway, my checkouts are over, over the Turkey time. I purchased a Wii U and it came with a few awesome games and I am super Super happy that I picked that up. I’ve been playing Smash and Mario 3D World, and it’s just been a delight. If anybody wants to play with me, tweet me and I’ll send you my Nintendo ID. We can play Smash together. Secondly, ReSharper 9 came out, and I’m excited because apparently they added autocomplete— not autocomplete— IntelliSense for regular expressions, which I hate. So the fact that there’s a tool to make it so that I don’t need to know regular expressions makes me very happy.

Matty: [00:55:36] Regex testers are the shit and super necessary. We have a newsletter, arresteddevops.com/bananastand. My New Year’s resolution is going to be to actually send out that newsletter on a regular basis. And when that happens, it will be the best way to know about upcoming podcast episodes and cool news with DevOps.

Trevor: Thanks again to our sponsors PagerDuty and Redgate, and to our loyal listeners. If you enjoy listening to Arrested DevOps, we’d appreciate it if you’d visit arresteddevops.com/itunes and leave us a review in the iTunes store. Thanks to Grant and Jonathan for joining us. Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter. Or in IRC at #ArrestedDevOps on Freenode.

Matty: Also, speaking of loyal listeners on Twitter, I want to give a shout-out to Matthew Skelton, who is @MatthewPSkelton on Twitter. He’s been providing some pretty great real-time feedback to this episode on Twitter as we’ve been going. He’s also given us lots of feedback on the show in general, which we really appreciate. Thanks, Matthew, for all of that feedback and input. And we’re always happy to get your input, ideas, or feedback at shows@arresteddevops.com.

Trevor: [00:56:46] I’m Matt @MattStratton, and I’m Trevor @TrevorGHess.

Matty: We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.