Does Testing Keep you from making a huge mistake?

Matty: [00:00:08] Welcome to another episode of Arrested DevOps, the podcast that probably won’t destroy your career with bad advice. I’m your co-host Matt Stratton, @MattStratton on Twitter.

Trevor: And I’m your co-host Trevor Hess, @TrevorGHess on Twitter.

Matty: So here we go, Trevor. We’re at episode number 2 already.

Trevor: It’s, uh, I know it’s the big 2, 2 weeks later. Lots of 2s. 2000— well, it’s not 12 anymore. It’s 2013. But we can pretend for the 2s’ sake.

Matty: Yeah, our first episode was received way more positively than I had expected, which I like to think is more of a factor of how much everybody liked it than my low expectations. But thanks to everybody who listened the first time around and for all the great feedback we’ve gotten. And we’re really excited to go into this episode. So we’re gonna start with our retro. Trevor, why don’t you tell us what you’ve learned since our last podcast?

Trevor: Well, one thing, as we’ll get to in a minute, one of the— I work with one of the panelists we have today, Nate. This past week we actually got our Sahi web tests running in our TeamCity continuous integration environment, which was lots of fun to figure out and get working. I also discovered that if you have custom USB 3 drivers, Windows 8.1 can get really upset about it and your USB 3 drives won’t work until you fix that. So that was a fun 30 minutes. Since Matt seems to be mute at the moment, why don’t we go ahead and introduce our panelists. Oh, looks like Matt may be back. Matt, can we hear you?

Matty: [00:01:56] I’m back. I hope you guys were continuing to go while I fall in and off.

Nate: I don’t know if anyone else can hear it, but Matt sounds like a Decepticon to me.

Trevor: Yes, Matt is very Decepticon sound right now. Matt, you want to talk about—

Matty: We’ll continue on.

Trevor: We’ll go on and start introducing our panelists. We’ll see if Matt comes back into the realm of the humans and not the Bionics.

Matty: I’m going to try changing my— I’m going to change out my headset, so. While you guys introduce. Luckily, this will all get edited out on the audio.

Trevor: I think we should keep it. It’s better. So we’re going to go ahead and introduce our panelists this week. We’ve got John Sheehan. I hope I’m pronouncing that right. I should have asked that earlier.

John: Close enough.

Trevor: He’s the CEO of Runscope. John, if you want to go ahead and introduce yourself.

John: Yeah, sure. So as you said, I’m the CEO of Runscope. We make API debugging and testing tools. I previously worked at Twilio. I was the 6th employee at Twilio, and then after that went to If This Then That. So I’ve done a ton of different API stuff, you know, working with developers at all different kinds of companies, helping them with their API problems.

Trevor: [00:03:06] Cool. Thanks, John. We’ve got Lynette Creamer, who is a testing guru from the— what I’ve dug up from her blog and her Twitter. Lynette, if you want to go ahead and introduce yourself. Hi.

Lanette: Yeah, I did testing for a long time at Adobe across the Creative Suites when they first started and all the way through Creative Suite 5. I did some, I guess you could say, corporate software at a large coffee company in the Seattle area. And from there, I now am the director of testing at Silicon Publishing, which is a 20-employee, all remote. We do web technology in Flash and HTML5 for companies where you want to design your own media and then print it. You might have seen some of our work on Shutterfly or hallmark.com. And we also have some custom plugins that we do and some web services. So, I’ve really more recently been involved with getting us tools to enable better testing. And I’ve learned a lot more about builds and what allows us to do good testing so that it can happen in the first place than I probably ever wanted to know in the last few years.

Matty: [00:04:46] Sweet.

Trevor: We’ve also got our 3rd and final panelist, Nate Brengel. Nate, I actually work with Nate, as I mentioned earlier, at Pathfinder. He’s our, one of our QA gurus as well. Nate, do you want to go ahead and say a few words?

Nate: Sure. So I am both independent and contract game designer. Most recently I’ve been doing mostly games in meatspace as well as some projects at Stanford. And now I guess I spend most of my day testing and developing automation testing at Pathfinder with Trevor.

Trevor: Matt, let’s see if you can come back from the silence.

Matty: Okay, so do I no longer sound like a Dalek?

Trevor: You are no longer threatening us with extermination or deletion.

Matty: Sweet, sweet. Okay, well great, well let’s get into it. I’m really excited.

Trevor: Let’s hit Matt’s retro real quick.

Matty: [00:05:47] Oh, my retro. My retro is don’t use this headset. That’s what I learned since the last time. No, actually, what I primarily learned was that— so the testing suite that I like to use for Chef Test Kitchen is now officially 1.0, which is super exciting. However, I’ve been having quite a few bit of challenges trying to get it to work with Windows guests, and I do know that’s something that Fletcher Nichols is planning to get into. They’re really targeting for a 2.0 type of a piece. And then really what I’ve learned is just that people really are excited about what we’re talking about here.

Trevor: So everybody, what is testing and what does it mean to you guys? Why is it important?

Matty: Maybe we can narrow that down a little bit because that’s a pretty big question.

Trevor: Well, we started with what is DevOps last time, so—

Matty: Yeah, that’s true, and that was a whole episode. Here’s a question that I just want to get some definitions, because some people might not be familiar, and so we talk about the same vocabulary. So when we define testing, like Trevor said, what is testing? What do you think about as the difference? Because people talk about a unit test, or a functional test, or a nonfunctional test. What do these terms mean to regular non-testers?

John: [00:07:05] I think it generally means more work than they want to do.

Lanette: I’ve noticed that it usually means stop talking. If it involves me, I’ve really overestimated the interest in listening from developers about the topic of testing. It’s a lot better to demonstrate something than to explain anything conceptually, and I think that’s just because we don’t share the same vocabulary, and it’s a good thing for me to know that and realize that it’s really not personal. It’s just human nature to talk about something conceptually doesn’t really convey value, where if they can see it and it gives them useful information, it’s not threatening work to be done in the future. It’s something quantifiable. And so, to me, testing is any action taken to give you information about the actual state of your software versus your assumptions. So anyone who takes some sort of action and runs it against the current software, whether that’s can it compile, that’s a piece of quality information you can use to make a better decision. So I just see that as testing.

Nate: [00:08:32] I really liked Lynette’s definition of that, especially because it has the opportunity there to encompass unit testing as well as user experience testing. All we’re looking there is gradations of the same sort of basic structure. I have a lot of fondness for the way Lynette thinks of that.

John: I think one of the ways it tends to get thought about, which would actually run counter to how people wanna think about it, is that I think a lot of developers look at it like insurance, right? It’s not going to prevent a disaster, but it’s going to help you mitigate those problems once they arrive. And I think if you ask somebody who does a lot of TDD, they’re actually looking for the exact opposite, right? They want to use testing as a preventative measure to prevent quality defects down the road. But most developers probably mentally have a framework of, well, this will just sort of help us in the future sort of reduce the spread of a problem instead of preventing it in the first place?

Lanette: I think that’s a false application of testing. I think you can’t really replace good software engineering practices and planning with testing. It doesn’t work. You have to have some design and some science in the first place. You can’t just write a test and have that. I mean—

Nate: [00:09:57] So, it does work. It just works—

Lanette: Poorly.

Nate: Perhaps the most cost-inefficient way that can be done. Yeah, relying entirely on testers.

Lanette: I think it can be— Testing can be a great part of software development, but it can’t really replace design. And I also think testing thought can be included in the design phase, which we try to do with pairing during the requirements.

Nate: Well, and test-driven design and behavior-driven design both are built entirely around that whole idea. Can you prove that it works when you make it rather than just making it without having to prove it?

Matty: I think that John, you know, when he was saying about, you know, a lot of the times that testing is seen as insurance, That’s a fallacy that I know I’ve thought about before. And for people that are kind of new, I come mostly from a tech ops background and not from a design or a development background. And I’ve always sort of been of the mindset that developers need good testing and good tech ops to save them from themselves. And again, it’s this idea that there’s this safety net or insurance. And then I’ve come to agree with Lynette’s point, which is to say, like, that’s not That’s— and again, it’s— that’s an expensive way to do it, right? Because we’re going to do this at the end. We’re going to catch you if you fall. And my thought before was, well, if the engineers know that someone’s going to keep them from falling off a cliff, it’ll make them push the envelope. But maybe the envelope doesn’t always need to be pushed in, in a way that testing saves you, right? Like, the envelope should be pushed in terms of ideas, not in terms of clever refactoring, or I don’t know.

Nate: [00:11:40] Sort of running home the point of this, I’ve had an opportunity to be involved in a project that uses this brick-and-mortar engineering paradigm called Design Failure Modes and Effects Analysis, DFMEA. And the particular application of it we had, instead of applying the DFMEA relatively early, so the DFMEA is a pen and paper test. You sit down and you say, what could go wrong here and how are we going to prevent it? Going forward. And we felt that we needed to apply it, but instead of applying it really early in the process and having it participate effectively in our design, it became our insurance at the end. We invested in it very late in the process and began to use it descriptively rather than prescriptively. So instead of saying, here’s how we should Here’s how we should do it to prevent ourselves from causing problems. Here’s how we did do it, and hopefully that’s good enough. As insurance, testing is the least effective and perhaps most expensive kind of insurance you’re going to get. While necessary, it comes at a very high premium.

Trevor: [00:12:56] Yeah, I would agree with that. I mean, we see, you know, in some of the stuff we’ve, you know, that I’ve worked on, getting to the point where we feel the testing is covering enough or meaningful enough to not just kind of be like, okay, yes, the call got made, but what does that call mean? Sometimes going deeper and, you know, not to plug you, John, but I’m hoping to take a look at Runscope and see if that’ll solve some of our problems with managing our MVC API endpoints in .NET. And actually seeing what they return, because that’s one of the things that, you know, sure, we can check to see if a 200 came back, but did it actually return the right data?

John: Yeah, and that’s sort of an interesting thing about API testing specifically, which we can talk more about, but we’re in this weird spot where we can do early design verification testing, so you can say, this is the contract our app is expecting very early on, even before the data for that is available. But then ongoing also test in the more insurance-type scenario that I was describing, where you could say, on an ongoing basis, are we still meeting that contract? You can build those tests as early or as late as you want. I think the value will come for companies and developers that build those tests as early in the process as possible when that contract— defining it sooner rather than later, instead of coming in later and sort of tacking on a layer of testing, hoping to get the insurance I mentioned.

Matty: [00:14:21] That brings up a question that I had too, which is when we talk about DevOps and we talk about flattening, we always talk, we say DevOps, and sort of my joke, right, is why is it not DevQA Ops?

Trevor: DevQuOps.

Matty: DevQuOps. DevQuOps. Why not DevQuOps? So, but seriously though, but to your point about when does it come in and you want testing to come in early, if we’re thinking about in a continuous delivery kind of a thought, or anywhere, where does QA fit into this more flattened organization? I think so we can just fire all our testers. That’s what you heard it here first. There’s no room for QA in the DevOps organization. That’s it. We have, we have spoken.

Nate: I think that actually more what, what the point we’re making instead of, instead of that is that if we’re integrating our development and our operations, we also really need to integrate our quality assurance. And that the days of tacked-on, just hand this to a tester who doesn’t know what they’re doing, they can just push buttons and read tests, it’ll be great, we promise they’ll catch everything, are becoming increasingly a thing of the past. They’re becoming clunky and inefficient, less reliable, and that there are options for continuous integration, best practices of testing, an increase in automation, an increase in intelligent risk-based testing.

Lanette: [00:15:50] I would say that there is this idea that developers understanding their code will write more meaningful tests, and that hasn’t happened. While there’s a lot of testers who’ve learned how to code, There are not a lot of developers who have learned how to test well, and I find that really unfortunate. I had hoped to be sharing just some basic vocabulary and types of tests besides the happy path with people, and I found a really low level of interest, and that’s been disappointing. So I’m trying to explore ways to make it exciting, throw confetti, glitter, sing songs, bring in cats, you know, whatever I can do to help testing ideas, like validating results, not just doing crap and making sure it doesn’t crash.

Trevor: If I got a cat every time.

Lanette: [00:16:52] You know, just no cats were killed in the running of this code, that kind of thing.

Trevor: Schrödinger’s test? Anything.

Lanette: Anything at all. I thought it would just be so easy to mention a few ideas of tests we could do, and they would start to happen. And I also had this idea that suddenly people would see how cool it was that we had these test results, and then we’d automatically have budget for a bunch more testers, and I could hire the smartest people I know and get them out of really crappy jobs. That would be wonderful. Those things have happened a lot more slowly than I expected. So, I guess my enthusiasm for testing has been a little bit blinding, and the developer disinterest has been a little bit surprising to me. However, I’ve noticed that developers are really interested in not doing the testing and just having the results. So, maybe If I can help more with that, that would be good. I have noticed that our continuous integration, once it works at all, is a huge hit in time savings to everyone. But we don’t, like, have DevOps people. So, we have one developer with a book and then I have my Kindle book. And he’s a lot better than I am. And I use the most insane variety of tools, and we have the most sporadic automation. But we use 6 different types of programming languages. So, I’m not really surprised that we don’t have automation everywhere. But some automation and some automated verification has been a huge time saver despite not having all of our tests integrated in our Jenkins system. We are hugely successful with running a sanity test and having a status that we push a button on. That’s hugely helpful compared to having nothing and emailed code. So, sometimes a little bit of progress in testing is enough to make a big difference.

Matty: [00:19:18] Do you feel like you’re going to start to see maybe a little bit of a snowball effect there? Is that once the value— like you said, the developers, they want to see it. You can explain it to them, but they want to see it. Are you starting to see more of those epiphanies, more of those lightbulb moments with people when they can see it and they embrace it a little more?

Lanette: I’m finding it really depends on the individual. What their focus is. And that’s always been the case with testing. It cracks me up. I’ve been to conferences in testing and in development for so many years, and every year they have this topic, which is the future of testing. I’ve never worked at 2 companies where the testing was the same. So, to think the future of testing would all be the same, or the future of development, when it’s not even at 2 teams at the same company, just cracks me up. It’s insane. To think that it would all be going the same place when we’re coming from such diverse needs and technologies. So, I think the way that testing can become more useful is by working more closely with developers, trying to figure out what can they do better than a tester can do, and where is a one-time exploratory test more helpful. That we don’t need to have around and maintain? What can we run, get the data from, and kick it out to not have it smelling up the place anymore? I really love the idea of differentiating a useful test to rerun and what do we want to just use a script to get information out of once and then move on? Because we’ve had useful results from both approaches.

Matty: [00:21:06] So, a question for the panel too. I mean, a lot of this stuff that we’re talking about, we think about rapid development, about the increase, we want to make things move faster, and I know I’ve worked with organizations who want to have these very short cycle times, but then still, you know, you’re still doing that Water Scrum thing, right, where you say, okay, and then we have our hardening sprint and that’s when we do all this testing. What are some of the skills, do you think, or the things that if someone wants to be a tester, and it takes a certain mindset to be really into this, and it’s applause all around for great bug hunters, right? What are the specific skills that you’re seeing that are necessary today, and this is for the panel, that are necessary in today’s DevOps-y, Cloud-y, Product-y, continuous delivery-y world? World that are different than maybe kind of your more standard enterprise waterfall project type of testing that we’ve had in the past?

Nate: [00:22:07] So for me, I think the big difference is having worked in a company that was ostensibly agile but mostly just waterfall, the big difference, and then having also gone the other road, is that because the shorter loops matter so much, it matters to have testers who are simultaneously invested in the process, but more importantly, invested in the product, which means that it looks like real agile development. When you find a bug, you get up out of your seat and you walk over and you show it to a developer right then. And then if the developer doesn’t have time to get to it right then, you sit down and you log a bug, rather than there being a sort of Chinese wall between developers and testers where you don’t want to talk, where the developers can’t be bothered, and the testers are scared that the developers will bite. Because, I mean, I love curmudgeonly developers, and I know a lot of testers who would be terrified to ever say, I found a bug. They just want to file it quietly and have no one notice that they did that. They barely want their name assigned to it. But it means a lot to be an upfront, product advocate as a tester to show up and say, no, this is stupid. I appreciate you, developer. You’re great. You develop code really well, and this is dumb. Because QA is closer to having a user experience than almost anyone other than the user. And it matters a lot to be a user advocate, and quickly, and loudly.

Trevor: [00:23:48] What can we do to be better at that? On both ends. This is again open to everybody, but, you know, as someone on the developer side, how can we be more receptive to the input of QA? How can we be, you know, how do we make people feel that we’re not gonna bite their heads off? And conversely, you know, how can we encourage QA to not be frightened of us? I mean, I’m more afraid of Nate, honestly. You know, I send my code up to our our QA branch, and I wait for Nate to tell me how I’m wrong.

Nate: I cultivate that aura of fear. That’s why I think it’s a valuable skill.

Lanette: I’m a big fan of pairing. We don’t have an office, so it really helps if we have new code, if somebody will demonstrate what’s new and let me know that they’re ready for some feedback. And they show me what it can do, and I can ask them some questions. And if they let me know, if I say, okay, I’m gonna test this for the next hour, can you be available and just do some other things? We can turn around a lot of bugs very quickly in that time, or I can just show them the bug, they can fix it. I used to do this in person with people when I worked at Adobe on the Creative Suite. It’s how I first started kind of getting involved earlier in the process. Where somebody would have code on their machine, and I would just go test on their system, like, in their office. Then they would help me with my Python scripting. So, it’s kind of like an exchange thing. This works really well if you have something new, and you just want to know, you know, have you missed something major? Is it working at all? Are you ready to even check it in? It’s great to have a tester just go in and ask you questions about it, and maybe there’s something you haven’t thought of. Even just walking them through your code and what you’ve made, they might have some questions about how it works with other features that you haven’t thought of. So, I would really recommend, if you’re a developer, letting a tester know that you’re ready for some feedback and that you’re open to learning something new and answering questions about your code is a great way to maybe get some extra attention and some feedback earlier so you can fix the bugs and move on to doing something you like better.

Matty: [00:26:18] So our friend Brian Barry of the Food Fight Show has just pointed out to me that we have a Q&A thing on this podcast, and people have been posting questions and I haven’t noticed them until now. So I’m going to ask some of them. And I think one of these actually goes to something that we were talking about a little bit earlier, but the question was, does having a multitude of languages make it harder to add tests?

Trevor: I can talk to this a little bit. One of the projects I’m on actually now, we’re splitting between— I mean, it’s not a super diverse set of languages, but we’re working with .NET and Angular, so JavaScript tests. And, you know, we have— you know, the tests don’t necessarily run at the same time, but we cover the individual pieces, and we do get the— we have— I think we’re— some people— we’re having the same issues with getting everybody to play with test-driven, so we’re back up to, I think, 83% overall, but our JavaScript tests are, like, 98%, and our API is at— I want to say like 85 to 90% coverage, line and method coverage. So, it is doable. It’s just, you know, they don’t run all at once. You know, if you look at our continuous integration server now, we’ve got our— we’ve got JavaScript tests running, C# tests running, and now we have our Sahi tests running, and they can run in succession as much or as many times as we want.

Lanette: [00:28:04] I would like to say, as far as code coverage goes, I give no rat rodent parts whatsoever about chasing the green bar of code coverage. I am not the least bit concerned about percentages of code coverage. I believe the real risk is what is not in your code. That’s gonna really kill you, not what is there.

Matty: And I would think, John, I’d like to know what you think about this because you’ve got a much very service-oriented, and I think this is the new way of thinking about testing in my mind, or at least what I think about this as far as sysadmin. I don’t like sysadmin, I don’t care about servers, I care about services, right? And would that be a fair statement, do you think, when it comes to testing? Back to Lynette’s point, we care about what’s happening, not lines of code?

John: Yeah, I mean, it’s built into what we’re talking about when we talk about APIs, right? The interface is essentially the entire part of it that matters. And so we’re a lot luckier because we’re defining these machine-to-machine communication methods, and those are a lot more rigid than something like a UI test or even a client-side test. You have a very specific explicit and implicit contract that needs to be upheld. Those things are usually well-defined on both sides of the conversation. Really, what you just need to— you’re more verifying that that contract is being upheld. It’s a lot easier, I think, test domain than some of the other areas we’re talking about, but you’re right that it’s not what’s in the code that you care about. It’s the inputs that you never saw coming or the combination of requests that causes deadlocks and that behavior that really is the tricky part, or a certain set of inputs that causes a chain reaction down a workflow. Things like that that code coverage doesn’t really help you with. I don’t want to knock it as much as Lynette did, but I can see where she’s coming from on that.

Lanette: [00:29:59] No, I think it’s important. I’m just afraid that that green bar has made people too comfortable. They see that percentage and they’re so focused on it. It’s like a video game. It’s mesmerizing. And it’s terrifying to me as a tester because I can run a little app on my machine that just goes through your webpage and takes pictures of shit, and people love it so much, and it’s not validating anything. And they ask me about it for days, and it doesn’t mean anything. And they don’t know why it’s not important, and I can’t get them to understand. And this scares me because they’re smart people. And I worry for the future of testing and for security and for performance and for the software that I care about. And I can’t get them to understand that all this motion with no validation, it doesn’t— it’s not valuable. And I just want to make the point that there’s more than code coverage. Not that it’s not important, because it is, but it’s a small part of testing. It’s an important part, and it needs to be there, but don’t feel comfortable because your lines of code are covered. You need to test more than that. You may need more information than that, especially if you have a user experience that you care about.

Trevor: [00:31:20] Absolutely. I mean, it’s really easy to make, you know, the green bar go up. Making that meaningful is what’s important.

Matty: I can give you all kinds of server uptime. It’s called, I’ll block port 80 on my systems and nobody will hit the website. That server will be up rock solid.

Nate: The code coverage, the big advantage of code coverage is that you know that as code is getting changed and being modified, that from a machine interface perspective, things are staying solid, which is great. I worked at a company where unit test was a bad word, and I don’t know how we survived as long as we did, where when our vice president arrived and insisted that people turn in unit tests with everything they turned in, there were, small panics and sphincter clenching, and it was very distracting and frightening that this was going on. Nonetheless, so I agree with you that it doesn’t cover everything. What really matters is that it helps you stay solid, especially as new things are getting added, and that matters a lot from a testing perspective, ’cause it prevents, what happened at this other company where we spent so much time in panic regression over and over and over again from a testing perspective that our testing resources were just jammed and we never had an opportunity to explore new areas of the software because we were so scared that at any point in time, something that was there could have broken. I am a huge fan of code coverage. And I agree with this.

Lanette: [00:33:01] I think it has a lot of fans. My concern is that overall testing by a whole team has, like, one fan, and they’re very sad and lonely. And so I’m not— totally not knocking the value of having well-designed code with great tests, because it’s only an advantage. It’s a way where you can do valuable testing when you get a build, which there can’t even be testing if you don’t have something testable. So, I definitely value having something that I can test when I get it, and that’s one way to make it happen that’s pretty effective.

Matty: Okay, another—

Lanette: I am agreeing with you.

Nate: And I’m agreeing with you.

Matty: Violent agreement. That’s what we do here.

Trevor: It’s the best kind of agreement.

Matty: A question that we also have that came in from the Q&A is, are there any kinds of tests that you can run in production?

John: [00:34:01] If you’re testing your APIs, then you can test all your APIs in production. In fact, it’s a lot easier to test them in production than it is on your local machine, since you can test them from external services. Now, obviously, I have a huge bias in this area. We sell tools to do this, and so I’m probably not the best one to speak objectively about this, but I will say one interesting thing about it. If you are testing in production, you actually run this fine line between testing and monitoring. Testing is a point in time. Did this work, and did I get the correct data back? That sort of thing. Monitoring is just adding a time dimension to that. Did I get the correct thing over time, and what was the response? For APIs, a lot of people actually want to sit right there. For other interfaces, you probably actually don’t want to get into monitoring, but for APIs, it’s actually a really good place to make sure that your systems are still working the way you intend.

Lanette: I can’t really think of a test you couldn’t run in production, except for stress tests. Where you’re actively trying to break the app, you’re gonna harm your users. So, as long as you’re not harming the users or exposing information you shouldn’t be, I think testing in production is a great idea and should be done regularly. Anyone have any cases where— that I haven’t mentioned that shouldn’t be tested in production? Because I think I could be missing something there.

Matty: [00:35:21] There’s one that I can, actually, it’s kind of the opposite, but it’s more about when sometimes you have to test in production, because there’s certain things you just can’t mimic. I know at one web company I was at, we were running in test for forever, and we were just even running A/B testing for forever, but for business-related reasons that I don’t remember the specifics, we did not allow search engines to get the beta site. It had something to do with SEO. It’s always got something to do with SEO. And so what happened though is the very first time that the Googlebot ever saw our new site was when it went live. And it wasn’t so much that it— and it caused all sorts of problems. And it wasn’t so much that it didn’t work for the Googlebot, but when Google came crawling, it just trashed some of our APIs. That our pages called because they just weren’t really prepared for the exact method in which Google crawled us. And that’s not something you can really emulate in a test environment, not because you’re not a good scripter, but because Google doesn’t tell you how they’re going to come a-knockin’, right? And it’s going to be different every time. So I’m still trying to think to your point about things that you couldn’t do. I would think stress testing is not prudent unless you’re really confident.

Lanette: [00:36:49] I love the idea of a dress rehearsal. I always try to push this thing, like you have to have a go-live dress rehearsal, just like you have to have a Black Friday dress rehearsal or whatever your big scenario is. You’ve got to practice going live, or how do you know you can do it, right? You have to test the whole process of doing it. And that includes coordinating people and how you know something goes wrong and everything at once. I tried to get that going when we automated media validation for all the Creative Suites. And that was like over 50 languages. And it was pretty crazy. And without dress rehearsals, we would have absolutely failed. Our unit tests didn’t even use real files. I mean, there were some real testing no-nos that I think people didn’t realize. I think not testing in production and not using real data is a far bigger risk than testing in production is, as long as you’re being conscious of your users and your security.

John: [00:38:03] The tricky part is that if you’re testing in production, you might be creating data that’s not real data, right? It’s test data. If that test fails in the middle, now you have all these artifacts laying around of things that are not meant to be there, are not real customer data, and whatever else. Then you have to go through and clean those up, or you just don’t have the ability to isolate maybe your data store like you would if you were running locally in a staging server, where it doesn’t matter if it fails halfway through. Or if you’re running it locally, you just throw away the whole database every time and you just don’t care what’s ever in there. With your production database, we don’t have the luxury of throwing away our production database on every test, thankfully. So it’s trickier to test workflows in which you need to create the thing, test the thing, and then clean up the thing without messing up all of your other data that’s in there or affecting customer experience. So those types of things—

Lanette: That’s a DevOps question, don’t you think? How can we make a safe, production sandbox that we can play in, you know, that has permissions and environment that’s like production, but a little bit safer. I think that’s one problem that’s not entirely solved for all companies. Maybe for some it is.

John: [00:39:20] We run a mirror of our production environment on the same— everything’s the same. It’s not like it’s just a test thing. It’s the same types of instances, same deployment system, same everything, and it’s as mirrored to production as possible. Not everyone, unfortunately, not every company I’ve worked for has had that luxury either, but that’s our safe area to do things with data that act like production but are not production data. But that’s where we sort of— any test that falls in that category, we put that there. Even still, we still monitor and test our production APIs for correctness. This, but anything that they do tends to clean up after itself. We just try to isolate that data without introducing additional variables that are changing the nature of the test. I think that’s really the tricky thing, is how do you do test data in production under a production environment without adding extra reasons why it failed or why that test is not working right?

Matty: Yeah, and there’s a couple of things that I was thinking about. To your point, John, where you’re saying, okay, I’m going to test in production, and I’m going to have this fake data. We’ve had that problem, especially if you care about logging activity. So you’re doing a website, and you’re doing all this activity logging. How do you make sure that you’re really truly getting the right thing? That’s when you kind of run into this, what I consider the fallacy of the synthetic transaction, because it’s truly synthetic. It’s not real. So you’re putting that monitoring equals 0 or whatever, some parameter on it, but that’s not exactly the same. But I’m really intrigued, you know, Lynette, with your point about saying, like, okay, I want something that looks just like production. Isn’t that the DevOps thing? And that’s where, like, and, you know, all of our Chef and Puppet people that are listening are going, yes, yes, yes, this is what we do! Because the whole continuous— there are 2 things that you said that to me just scream continuous delivery, and one was you talked about having a dress rehearsal for a a release, right? And that’s like one of the key principles of continuous delivery is releases should be boring because you’ve done it 50 billion times by the time it goes to production the exact same way. And then likewise, every single stage in your pipeline needs to look exactly like production except for scale, and then sometimes data, right? Because if you’ve got PI, you’ve got PCI stuff, you’ve got HIPAA stuff, maybe you can’t have real data, but it should look like real data. And the problem is, without having good automation, you can never do this. And that’s why, you know, we’re seeing— I’m personally, I’m seeing a lot of customers that this is what they want, right? They want to be able to say, I want to be able to have on my laptop, I’m doing my work, I want it to look like production so I could do this same kind of a test like it was in production. But I’m not, to John’s point, I’m not mucking up production. Or maybe I don’t have clearance to see credit card numbers if it’s a PCI thing or whatever. But for all practical purposes, it’s the same. And it’s, it’s, it’s, it’s a big chicken and the egg problem because it’s a, it’s a huge time saver and efficiency once it’s going, but it can be a very big pill to swallow to get it to go. Like you said, you know, you’re starting to work on this automation. You’ve got your one developer who’s got his Jenkins book, and you’ve got your Kindle. But you’re doing something. I mean, that’s the thing. You’ve got to chip away at it.

Lanette: [00:42:46] Sometimes the technology that helps us is so dead simple. I mean, we just use Jing, and attaching videos instead of bugs has helped us so much, because people don’t do the same thing with their mouse when you give them a set of steps. If you tell them to do it and they’re reading it, those words aren’t the same as showing them you doing something on the screen visually. And I don’t know why it shouldn’t be different because we all speak English, and it doesn’t make sense, but it is different, and it helps us to have a video. And that’s not a continuous integration tool. But we need it.

Trevor: Yeah, having video bug reports, or video alongside a bug report, is like— it just changes everything. It makes it so much easier to tell what and where the problem is and to actually see it happen. You know, you give me steps to reproduce, chances are I’ll find a way because I know how not to touch it. That I’ll make sure that I have, you know, subconsciously, I’ll make sure I don’t break it. And then when you hand me a Jing and I look at that and I see, oh, you clicked this and then you did that and then, oh, that’s where it breaks.

Lanette: [00:44:10] And it’s almost always you’re more patient. Almost, it’s 90% of the time. The developer is more patient than I am. Everyone’s more patient than I am. And that’s why I have found software bugs in my own car that I work around. I just have no patience. And I think that’s part of why I notice small things that other people might not bring up that lead to problems.

Matty: I think the fact that Lynette does QA for the whole world is all the reason why you guys have to follow her on Twitter. It’s one of my favorite things. Because there’s quite a few tweets that are just like that.

Trevor: Yeah, so I had to say, I’m curious, what bug did you find in your car?

Lanette: Um, there’s a transmi— well, there’s 2, but, um, they replaced the transmission with the same error it had before that they replaced it for. Um, but anyways, one is there’s a— in automatic vehicles You can’t put your car in reverse unless the brake is on, but there’s an electric failure that sometimes shorts out the switch, so you can put your key in it. When it starts to happen, you get false positives on the brake light, and it comes on periodically, and eventually it burns out the sensor. Well, the electrical problem is so hard to chase down that I just have an extra key really close to that if I ever need to move my car. And I just bought a bulk pack of the sensors to get them replaced every 2 to 3 years when the electrical problem shorts them out.

Trevor: [00:45:52] That’s fantastic!

Lanette: I figured out there’s an electrical problem in my Honda. It’s paid for every year. It’s paid for and I’m driving. It’s money in the bank. I’m really geeky. I know it sucks that I have an old car and it’s not cool, but The fact that I go on vacation because I have that old car and I can do something cool, I like so much more than a new car. So yeah, extra electrical switches to put in every 3 years for $100 versus a whole, you know, work troubleshooting of the electric system in the car is a lot cheaper.

Trevor: That’s awesome.

Matty: So So, one thing that I’ve— and I know we kind of have gone down a path of coverage and what does this mean and everything. So, also, by the way, I am so not asking the question I wrote down, which is how do you make sure that you’ve got the right code coverage, because that’s just a recipe for disaster today. But a lot of times, where does the line— when we talk about writing tests, right, and I know there’s a lot of We talked last week, or I’m sorry, the last episode, about how DevOps means not saying not it. But I find, especially when it comes to writing tests, a lot of not it. It becomes really clear at a certain level, like user testing, like UAT-level testing, or functional testing, that’s what QA does. And then when we get into kind of more of the unit-type piece, the line can get fuzzy and it can be a lot of, well, that’s not for me to do, or that’s for me to do and you should have nothing to do with it. And I’m interested to know what you guys think about that.

Lanette: [00:47:42] Whoever can tolerate it and can do it well, that’s my opinion.

John: I think if you’re going to do it, everyone should do it. So I get away with it sometimes because I’m the CEO and I don’t always have to follow all the rules, but I know that it hurts my engineers’ feelings when I don’t write a test for some new functionality and something that they’ve spent copious amount of time making sure is not brittle. I want to be better at that, but it’s really not going to work as we grow as a company if everyone isn’t writing them and keeping up that same level, because the ones that do are going to burn out on trying to make up for everybody else, and the ones that don’t are really just not pulling their weight. I’m just going to read my— the solution for me is I’m just not going to write code anymore, and that way I won’t have to write any unit tests. Plus, all my engineers will like me a lot more.

Lanette: See, I plan to use unit tests as bribery. I’m going to say, hey, you teach me how to write these unit tests, I’ll write these for you, and then you help me with something else I want to do that I can’t do. So basically, I think trade-in-kind is hugely underrated and needs to be utilized because you don’t always need your most creative architects writing a unit test that’s repetitive. So, I think we have a lot of exploration about division of labor. And this idea that everyone does everything is absolutely unsustainable and insane. It doesn’t scale. And I’m interested to see what really happens versus what we tell people happens. Because testing is the worst example. I mean, people will tell you that they’re an SDET and they write tests all day, and it’s not true. They just can’t really tell you what they do because of their job title.

Matty: [00:49:33] Like you said, I believe I agree with you with the whole notion that everyone does everything is rough. I also don’t think that’s what DevOps is. I always used to tell my team, I said, it’s not that everyone can do everything, It’s that everyone should be able to do most things.

Lanette: Everyone should have some understanding of what it takes to— you can’t be totally ignorant of what your team does, I think.

Matty: Absolutely, yeah. I mean, it’s a matter of you should understand every piece and what goes through the chain, but It’s a misnomer to think that, like, okay, your most awesome Ruby guy, you know, or Rails guy, is going to be just as good at configuring sysadmin as your most senior sysadmin, right? That’s just silly. But he should be— they should each be able to understand what the other one does, and then likewise be able to, you know, pull a task off the board even if it’s not color-coded for their particular field. Function, right? And then why would a— and I understand that sometimes you have compliance things and you have stuff where like this has to be for legal reasons be tested by someone who didn’t write it, but I really believe that more often than not those are fictions. I mean, I’m sure there— I know there are SOX things that have to do with that, but most companies, that’s just the way that their company works to say I want someone else to have tested it. And that someone else, do they have to have QA in their title? Could they also be a developer? Could they be a sysadmin? Could they be a Scrum Master?

Lanette: [00:51:12] Product owners can do a great job of exploratory testing. I’ve tried to train some of them. And I say can because it totally depends on the person.

Matty: Okay. I’ve got one more question that we alluded to earlier, but I want to bring it back because we’ve talked about a couple different things. And having had the conversation that we’ve had, I want to wrap it up with this, which is, what are the skills and talents that are necessary to make that transition from your typical QA to this more DevOps world? How can someone who’s been doing QA the old-school way for 20 years, what, what do you think they need to do to do it in the DevOps world and be able to provide extra value? That’s a question for anybody. Sorry, I was throwing that out for the panel. This is our— this is my attempt at my wrap-up kind of question because we’re coming up to the top of the hour and I want to have time for our checkouts.

Lanette: [00:52:18] I think testers have to have a willingness to go beyond what they think their job is, just like everyone else does. And learn how to be more useful and learn new things. I think that applies to everyone in every role because our jobs tend to overlap. And that’s also how we can help each other to learn more of what’s efficient and basically have some teamwork.

John: You know, I think when you start looking at systems, you have to not just test things in isolation but understand that the system as a whole has a series of inputs that might cross individual components of it and that the system as a whole has a set of outputs and that what you really are trying to verify is that if given the right or wrong data on the input side, that you’re getting the right or wrong data on the output side and sort of taking a more holistic approach to to the system instead of any one individual component of it.

Nate: [00:53:23] Yeah, I went out of— I had the be an aggressive tester, cultivate a culture of fear, make your developer— but really it’s relate to your developers. It’s spend time with them, be invested in what they’re doing and in the product that they’re building with you.

Trevor: Absolutely. Great. Before we hit the checkout, I just want to make sure we— I wanted to give John a second to kind of directly answer Matt Shelton’s question, which is kind of how John got on the panel to begin with. So his question was, SOAP web service testing can be a pain because SOAP UI is a pain. Are there better tools out there? And John kind of pinged us all and said he thinks he has a better tool, so I just wanted to give him a minute to directly address that question.

John: I’m always trying to be a better tool. I mean, so, you know, for SOAP services, you know, SoapUI is appropriately named and probably maybe the best tool for doing that across your different environments, whether that’s local, dev, production. If you’re doing something, you know, that lives more in the cloud, or you’re working on more modern RESTful services, I think the market is really ready for a more modern tool that lives in the cloud along with your APIs themselves. We’re trying to modernize service testing. To me, that means maybe not paying as much attention to SOAP as a traditional tool like SoapUI, but I do think there are new approaches to take on this. The idea of service-driven organizations, or service-driven apps, and distributed apps has really taken off. The last few years as mobile has pushed more and more services out onto the web. And so we’re just trying to modernize that. If you want to learn about it, go to runscope.com. I don’t want to bore you with a super long pitch, but, you know, like I said, we’re just trying to modernize service testing.

Trevor: [00:55:18] Great. Thanks, John. Thanks, everybody. We’ll move to our checkouts. We’ll start with Matt so that we can get— so the panel can get a feel for what a checkout sounds like when they announce theirs.

Matty: Absolutely. So as a reminder, this is the part of the podcast we call the checkouts, where your hosts and panelists provide something we would like you to check out. It can be technology-related, it can be not technology-related, just something that we’ve been digging lately that we want to share with you. So I have 2. One of them is actually on topic, and it is a tool called Pester. And it is a PowerShell module for writing test-driven development tests, kind of Cucumber style, for PowerShell. Could raise the question, why are you writing tests for scripts? Many people have asked that question, but I also like to tell people there’s a reason it’s called PowerShell and not PowerScript. But I’ve been doing a bunch with Pester with a lot of the automation we’ve been doing, and it’s really made it nice for me to make the transition from using Cucumber. Into something else, because I was going crazy trying to figure out how I was going to use Cucumber and PowerShell at the same time. My other checkout is a game, a video game called XCOM, which I’ve been absolutely adoring. It’s got a new expansion pack called Enemy Within. It’s available on Mac, PC, Xbox, PS3, iPad as well, I guess. I played it on the iPad for a while. I played it on my Mac. It’s a tactical enemy alien invasion game. The expansion pack made it like super way harder than it was before. And there’s been a lot of cursing at my computer in my apartment lately, but in good fun. So those are my 2. Trevor?

Trevor: [00:57:04] All right. Well, I’ll start with my on-topic one as well that you made me remember. So there’s a tool for Windows now called Clumsy. It’s only a 0.1, but it’s on GitHub. What it does is it lets you simulate a bad network connection. In an interactive way. So you can like make it drop packets every so often, you can make it have lag, all kinds of fun stuff. And you should— it’s interesting, you should check it out. And also, as you know, as we’ve been gathering with friends for the holidays, the board games become more plentiful, and I’ve been playing a lot of Ticket to Ride. But as well, there’s always a great digital board game, Civilization V, which I just got the new expansion, Brave New World. Which is lots of fun. It adds trading and a world congress to the game, which is actually really interesting and fun, and I would recommend checking it out. We’ll start with John. What do you have? What was interesting for you this week?

John: [00:58:11] So as long as we’re talking about games and board games, so Carcassonne is probably my favorite iOS game, and they just released the 3rd expansion pack, which is the Princess and Dre. I don’t know what it’s called, but it really adds an amazing level of manicness to Carcassonne. It actually makes the games a lot less predictable. If you play Carcassonne on iOS, you should definitely get that. Then the other 2 things I would say is lately I’ve been really obsessed with email summaries. I don’t like individual email alerts, but I really like getting information via email once a day. There’s 2 services I’ve just signed up for lately that are doing this really well. One is briefmetrics.com, which sort of sums up your Google Analytics every week and gives you like a really high-level overview compared— this week compared to last week. And then the other one is hookfeed.com, which is a way to do like daily and weekly summaries of your Stripe account activity. So really simple services but super useful for me.

Trevor: Great. Lynette, what do you got?

Lanette: Well, I wrote a blog this week about the Butler Wars between Jenkins and Hudson. I did a little research, so, and it happens to have a hilarious icon of English butlers fighting that I made. So, if you want to check that out, it should be up, I think, tomorrow. So, what I learned mostly is there’s more plugins for Jenkins, more open source community support, and there’s a newer build for Hudson. But they’re two brothers from the same mother. No real significant functional difference there. I’m looking for a couple people to play Words with Friends on my iPhone. It’s old and moldy, but what else can I say? I’m on Twitter @lynettecreame, and I talk about a lot of testing stuff if you want to talk about testing things. And I just learned yesterday that the PyCon upcoming convention has the most female speakers they’ve ever had in all of history. So, if you go to that and you see a speaker who’s good, let them know what they’re doing well, what they can improve at. As a female who goes to tech conferences, I really would like to see a more balanced audience there. So, if you would too, go support some Diverse speakers, give them feedback on what they’re doing well, and hopefully I’ll get back out there this year too on the road. I’m gonna apply for Agile 2014, maybe a couple. I wanna go to Europe, so maybe I’ll apply for a few there. See what happens.

Trevor: [01:00:55] And Nate, if you wanna wrap us up. Yeah.

Nate: So my check out is— so I do games mostly in meatspace. I am all about recommending Jeep Form, Jeep like the truck. Jeep Form is a completely unique approach to both theater games and live-action games of all kinds. They’re wonderful, almost all of them are published free online. I have one called Gunfight that’s in production right now. And the other thing that I would recommend taking a look at I know that I’ve been looking at recently is Perfecto Mobile. That’s probably more relevant. They’re an automation suite frontend for platform-agnostic mobile work. And they actually plug devices into their harnesses and do business-driven testing. It’s a pretty interesting thing. Definitely a direction that I have been considering pushing as we go.

Trevor: [01:01:59] Great. Well, thank you guys for joining us this week. You’ve all been wonderful, and I think this has been a great discussion. Matt, if you wanna close things off.

Matty: Yeah, absolutely. Special, absolute super special thanks to our great testing panel: John Sheehan, Nate Brengel, and Matt Kramer. And as always, you can follow us on Twitter @ArrestedDevOps or visit us at on your interwebs at arresteddevops.com.

Trevor: You can find us on the tubes, on the tubes, on the tubes.

Matty: If you have ideas or comments for future shows, please tweet them to us, or we accept pull requests at github.com/arresteddevops. And everybody have a great end of the year, and we will see you guys after the first of the year.