all together now

Matty: [00:00:04] Welcome to Arrested DevOps, episode 7, All Together Now. I’m your co-host Matt Stratton, @MattStratton on Twitter.

Trevor: And I’m your co-host Trevor Hess, @TrevorGHess on Twitter.

Matty: This episode of Arrested DevOps is brought to you by TenthMagnitude, helping businesses realize true agility through DevOps and cloud-enabled innovation.

Trevor: Today on ADO, we’re going to be talking about collaboration, but As always, we will start with our retrospective. We’re going to talk about what we learned since the last episode.

Matty: Do you want to go first, Trevor?

Trevor: Sure, I can do that.

Matty: It’s called throwing your co-host under the bus.

Trevor: Yeah, the wheels feel real nice. So I actually, in the past couple weeks, I picked the Clean Code book back up. And it’s been, I want to say, 4 years since I read it last. And it’s just, it’s amazing that not only the things I’ve forgotten, but the things I understand way better now. Because last time I read it probably would have been my second or third year in college. And so did not know too much then. Not that I know that much more now. But yeah, it’s been great. And it’s really helped restructure what I’m focusing on with my code. Matt, what about you?

Matty: [00:01:23] Actually, my retro ties pretty closely into our topic tonight. My team has been working with some collaboration tools. We’ve been trying to work on different ways to talk, and so we’ve been piloting— we’ve been using a tool called FlowDoc to communicate across our team, especially with consultants that are on-site and not necessarily in the office. And we just installed Hubot into Flowdock, and Hubot is an IRC-type bot that GitHub wrote for some automation. And mostly we’ve been using it for posting memes and Breaking Bad quotes in our chat, but apparently there’s actually productive things you can do with Hubot as well, but we have yet to utilize them yet.

Trevor: Sounds a lot like our HipChat client.

Matty: Yeah, and I think you can— Hubot, I know, can tie into Campfire, probably can tie into HipChat as well. So I think that does tie us pretty— brings us real close into our topic of the night, which is talking about collaboration, and we’ve got a pretty great panel. So we’d like to introduce our panel. First we have Angela Dugan. Angela, you want to tell us a little bit about yourself and your background?

Angela: [00:02:38] Great, thanks, Matt. So my name is Angela Dugan. I currently work for Polaris Solutions, which is based here in Chicago. But we have offices in St. Louis and Denver, real small consulting firm, so I won’t be offended if people are like, who, who do you work for? Just to give you a little background on what I’ve done in my past, previous to that I worked for a small little startup called Microsoft. I was there for about 5 years as an evangelist. Prior to that I actually worked at another really small consulting firm in Chicago called Software Architects that people may actually have heard of, so I’ve been in software for about 15 years. I’ve done a lot of different things, right? I’ve been architect, BA, developer, QA, you name it, right? You’re a consultant, you kind of do everything. But, but now what I’ve kind of landed in is kind of my dream role at Polaris Solutions, which is managing their ALM practice. So Application Lifecycle Management. So it’s all about collaboration and delivering better software.

Trevor: [00:03:38] Great. Thanks, Angela. We’ve also got Todd Vernon with us tonight. Todd, you want to introduce yourself? And maybe your beer as well.

Todd: Yeah, absolutely. Tecate is the beer of choice tonight that I decided to go with. I wasn’t sure, but it seems to be working out fine. Yeah, thanks for having me. It’s great to be on the show. I have about— so when I got out of college, I started at NASA, right, writing software for X-planes, so the experimental airplanes. And I did that for several years, and that’s about as un-agile as you can get. You know, you make one constant change in the code, and if you have to relink the code, then everything in the whole world gets tested again, and 6 months go by. Fun experience, but, you know, had kind of the entrepreneurial bug. So 7 years after that, I started my first company with my co-founders called Raindance Communications. We, we competed with WebEx, did an IPO, did all that great stuff. Built software, deployed software, did it totally old school, you know, and all the pain that went along with that. Sold that company. My second company was Lijit Networks, which sold to Federated Media a couple years ago. That company was an ad tech company, so we built software, we deployed software, we made revenue by the fraction of a cent, and all the pains that went with that. And we were a little more agile, but not— we actually called it fragile. But I’ve heard that before. Yeah, so my name— that joke’s an old joke. My— so my, my, my new company that’s about a year old, I actually decided to kind of look back on my career because I was the CTO of my first company, the CEO of my second, and I actually am in the collaborative software space specifically around the ops and DevOps mission. And that is simply— it was a selfish kind of maneuver having had these companies my whole career and, and, and could see kind of the, the, the pretty raw tools that were available to teams that do this really incredibly tough job 24/7, I asked myself, why is it this way? Right? It’s a, it’s really hard and it ought to be better. And so I built VictorOps, which is my current company. Um, that’s exactly right in the kill zone of this, uh, this broadcast.

Matty: [00:06:05] Awesome. So yeah, I think we’ve got some folks that know what they’re talking about. One of the things that, as we’ve gone through our episodes, we’ve talked about a bunch of different things here. We’ve talked through our long history of 2 and a half months of existence at this point. We’ve talked about some myths. We’ve talked about challenges that people have in DevOps and Agile and things like that, but it all still comes back to collaboration, and that’s why we wanted to talk about this, is there’s the challenge. If we say one of the core precepts of DevOps is we want to break down silos, and we talk about not wanting to get wrapped around the axle with tools, but this is probably an episode where tools can come into play because tools can really help as far as that goes. And I guess something I’m curious about Because I’ve faced this challenge even before I knew what DevOps was or cared about DevOps or knew to call it DevOps. But in a previous incarnation, you know, I hate email. I hate email like with the fury of a thousand suns. I think email sucks and should not exist. And IM is like right next to that. You know, I’m not a fan, but I really, really hate email. And I’m— because there’s got to be a better way, you know, and I’m curious to see how do you— what are your thoughts on how you can kind of get people to make that shift, that especially folks that are used to saying email is how I’m used to working, that’s how I want to communicate, and we want to use some new type of collaboration tool, something like HipChat or FlowDoc or Slack or whatever, IRC, just something that’s a little different. How do we get people out of their comfort zone in that way?

Todd: [00:07:59] I’ll go first. I think it has to be substantially better. I mean, that’s kind of the bottom line, you know, and I’ll point at some of the things you just said. You know, one of the things I learned in building this company, but also before that at my first company, which was also a collaborative software company, is collaboration works best when it’s vertical rather than horizontal. So like HipChat was a great example of something you mentioned, that was a horizontal collaboration tool. It’s fantastic, it’s used in tons of organizations, our platform interfaces with it, it’s very ubiquitous, but at the same time it’s also super horizontal. So you know, a salesperson can use it and a DevOps person can use it and a software developer can use it, and that’s fine, but you don’t get any of the special sauce of a vertical collaboration platform where it can combine the data and the dialogue and kind of present that in a way that you can expand it and drill into it and, you know, blow in and out of it. And that’s where the real value in collaboration comes, when it goes vertical, right? And there’s countless examples of that I can give you, but It’s not really for the show, but I think it all kind of goes back to it has to actually provide a better value than kind of what was next to it. And I think a lot of people look at email and go, well, you know, or they look at basically a horizontal chat and go, well, it’s a broken email because like if I’m not there, I don’t see it, you know. So I think the margins are very small on those kind of horizontal platforms, but if you can really harness a vertical platform where all of a sudden, you know, wow, this is like a substantially different experience. I think that goes a long way. So it has to be better.

Angela: [00:09:45] Yeah, and from, from my perspective, I also see it needing to be kind of a natural fit to what people do anyway, right? So while people may be used to email, and believe me, I definitely had a lot of email when I was at Microsoft, but it got to a point where email reminders would kind of make you angry, right? Because you’re off doing something else And now you’ve got this thing popping up in your face, oh, new email, and now I have to leave the thing I was doing and go open this other application. And so if you can find something that naturally kind of plugs into the different tools that they tend to live in, you know, day in, day out, whatever those tools are, and there’s, there’s so many of them on the market these days that it’s gonna require a little bit of research, but finding the thing that’s more natural for people, right? If you have a team where everyone on the team loves Twitter, they love that type of broadcast mechanism, right? Find a tool that kind of fits their personalities, and ideally there’ll be kind of other channels into that tool for people who don’t, right, because not everybody is going to like that style of communication. But yeah, to me it also just needs to be really natural, like it needs to be low friction to use and to just have it be part of what you’re doing, right? And ideally, if you can use it both on your computer and your phone, right, that’s even going to be better because Almost everyone always has a phone on them, on the L, in the car, whatever.

Todd: [00:11:06] Yeah, I think, I mean, just if I could just kind of add on to that. I mean, I think what has really, you know, if you look across the disciplines in an organization, you know, DevOps is definitely leading the charge in kind of a new way of thinking about this particular problem domain. But if you look at other parts of the company, their toolsets have historically been around collaboration have been much more rich. Than the ops or DevOps or even engineering crew. And I think largely that goes back to something that was just said, which is it has to inherently be mobile, at least equally mobile in a DevOps world, because you’re likely going to have to communicate at any time of the day, on weekends, in the middle of the night. If you’re skiing in the mountains, I mean, that’s just the nature of this job, is to be able to kind of always be somewhat on. And if the tool isn’t inherently mobile, almost mobile-first, I would say, it’s not really a collaborative environment for this particular application. And that’s a little different than disciplines that largely happen at your desk during working hours. And that even goes to development teams. They tend to do most of their work during working hours. They’ll develop at home at night, but you know, you can get your mind around a collaboration tool that’s more computer-centric in that world, but not around DevOps when there’s so many times they’re just everywhere. And because the discipline is so diverse, you have a network guy next to an app-level guy next to a database guy, and they all have to come together to solve a problem. Inherently, it has to be mobile.

Matty: [00:12:53] Yeah, I was going to say, because I think a lot of it is being able to replicate the concept of physical proximity, right? Sometimes you have that luxury of having your team sitting 2 feet away from each other the whole time, and that, like you said, either it’s because you can— sometimes you can’t do that because you’re actually geographically dispersed. Sometimes you can’t do that because it’s after hours. And how do you replicate the equivalent of being able to shout over there and say, you know, hey Betty, you know, what’s going on with that index over there? And I guess, and then to step away from tooling a little bit is what are those kinds of— because again, collaboration is like a big word and it’s kind of a buzzword and it’s kind of a, you know, everybody goes, oh, we’re going to talk about SharePoint and collaboration and blah, blah, blah. And really, what are we trying to accomplish, and I think how in a DevOps team, what type of collaboration needs to occur, because I guess we have to figure that out first before we say how we’re going to do it. We kind of are coming at it from the opposite direction.

Todd: [00:14:02] Yeah, so again, I’ll jump in. Sorry, I get really passionate about this subject, so I have a lot of thoughts on it. The, you know, I think an interesting, you know, interesting thing we talk about at our company that directly goes to this is, and we had a lot of internal debate about it, is what’s the correct metaphor, right? Because there’s 2 very distinct metaphors. There’s sort of the Twitter-like stream of information metaphor, and then there’s the chat-based metaphor, right? And they’re actually very different behavioral patterns. And which one is kind of appropriate to solve this problem. You know, we adopted the chat— or not chat, but the Twitter metaphor, where the idea is if you can kind of see what’s going on in the organization all the time, not just what people are saying but also, you know, the alert stream, things that are happening, you tend to be more engaged with the platform so you can solve a problem faster. Kind of another way to think of it is sort of the chat metaphor, which is not the one we went with, Which is more of kind of a direct dialogue. And even when you present the UI that way to people, what they do is if it’s a chat-based metaphor, they tend to just kind of talk about the problem, which is useful when you want to do that, but it sort of excludes the ability to kind of put the idea of updates and data into it, which we think is a valuable piece of it too. So it actually is a very interesting thing because chat is a very pervasive metaphor among technical people. But, and less so the Twitter metaphor, but it’s actually very powerful. People don’t think of it that way, but the two are very kind of different and they’re good for different things when you actually drill into it.

Angela: [00:15:46] And I think too on the collaboration front, because like you said, it’s a big word, right? Everyone’s using it, and I think it’s one of those things where sometimes companies see collaboration and information, even if it’s kind of a deluge of information, is the same thing, right? So, it’s making sure that the information is getting to the right people and that it’s the right information, right? Just spewing forth all the information possible is, is not really helpful to the DevOps guys if they’re having to filter through it and, and make sure that they’re seeing what’s important. So, there certainly needs to be some expectations up front of, hey, what’s really valuable? What do we need you to focus on rather than just tell me as much as you can and make me do all the work to really get the value out of it.

Todd: Yeah, filtering is important.

Matty: And also, I think, you know, we’ve been talking about tools that are team-based or talking about ideas to me that’s within the team, that’s, that’s like the delivery team that’s delivering the actual technology. But then, then I think about within the organization and how do you communicate and bring the rest of the business into that.

Trevor: [00:16:54] As far as that goes, separate that information out so that it’s only the useful information, right?

Matty: That’s— I think that’s, that’s super key because it’s really easy. Again, I mean, I’m— this is, this is the problem with, you know, the tech ops guy sits here and says you get so much noise you ignore everything, right? So, so you want to provide that filtering, and then the same thing can happen to somebody quote-unquote on the business side if you’re getting updates every, you know, every commit or every whatever, you just sort of go, I don’t really care. But I think it’s really necessary to provide the transparency to the rest of the organization, warts and all, right? You know, this is what we’re doing that’s right and that’s not. I know one thing I was really, really— I guess a good success story about this was We had a challenge in a company I was at where one of our applications was spewing errors in the logs. They weren’t necessarily— the problem was we didn’t know if they were really errors or not. We didn’t know if it was really a problem or something in the code that was just generating logs, but there was so much noise that it was really impossible to provide the analysis. The delivery team is going to the product owner and saying, hey, we need to put in a story. Someone has to look at this and figure this out. Someone on the product side doesn’t understand what this means. They say, well, is there something wrong? Well, we don’t know. Shortly after this, we implemented Splunk and we created dashboards that presented, provided sort of like, this is how many errors. When they saw that our application was throwing 14,000 errors a minute, and that was on a big screen that the entire company walked past, very quickly it got looked at. And I think it’s not necessarily a bad thing when you have statistics, but people want to be able to know and, and should be able to find that information on their own. Like Angela said, people should be able to consume the way that they want to. If I want to use IRC, I want to use my email client, I want to use a feed reader, that— I think that’s another key component of it, is people like to consume information the way they want to consume it.

Todd: [00:19:07] It’s one of those things, there’s limitations to it though. It’s a great example of, you know, like, I’m a big believer that in a perfect world everything goes into the same timeline. You know, it’s kind of like the black box recorder of the whole, the whole business, right? I want to know what the technical problems were. I want to know what the problems were that were coming from tech support. I want to know kind of everything in sort of one timeline, but then I want infinite filterability on it rather than, rather than a room metaphor where all these things go into different rooms. That’s actually not so useful in a forensic sense, or in the ability to kind of later go back and then, you know, reduce all the data and figure what’s going on. Instead, it’s much more useful. It’s all in one stream, but infinitely and easily filterable. And that kind of leads back to the horizontal versus vertical collaboration platform problem. When you pump all of that into a horizontal platform, it just becomes an unmanageable It’s all noise. It’s just noise, and that irritates people. So what they do is they pull data out of the stream, but that actually doesn’t help the organization in the long term because then when you actually want to do a postmortem on what’s happening, you have to pull it from 7 different sources, and it’s actually, you know, really hard to recreate sort of what went down. So the metaphor of kind of the single timeline infinitely filterable, I think, is is super powerful, but it’s a very vertical concept in the DevOps space. And historically, it’s been kind of, you know, DevOps space has been driven by more horizontal tools. So I think that’s a maturity thing that we have the ability to kind of re-engineer over the next, you know, 5 or 10 years to say, how do we really get all of those things together, right? And kind of get to the point I think you were kind of making, Matt, is how do we get— I think to a certain degree we have to get a little bit out of the separate tools mentality and sort of get into the platform mentality that exists probably in other areas of the business that you can really dig into the data and see what’s going on.

Angela: [00:21:13] Something else that, that I just wanted to bring up, and I’m sure it’s, it’s obvious, but so far we’ve been doing a lot of focusing on, on tools, which tools do a great job of kind of providing some of the stuff we’re talking about, right? Ways to kind of automate data feeds, ways to be able to bring people together that are remote, they’re not able to sit together But at the end of the day, you also, you’re just going to have to do some of that collaboration face-to-face because, you know, you can provide the best dashboards in the world, but without the context, you can have people that take data and do very evil things with data, right? They start looking at bug numbers and severities, and then before you know it, they’re using that to kind of figure out like how valuable is a team or what are they producing based on numbers. And so I think it’s still really important to make sure that that teams are still meeting face-to-face and talking about things, and whether it’s, you know, inviting product owners and stakeholders to retrospectives or whatever you need to do to make sure that you’re always going to have that face-to-face, because at the end of the day, you still need to make sure that you have some kind of connection with people and it’s not just kind of a tool acting as the interface.

Todd: [00:22:23] I think, you know, you bring up a great point, and I had a question I kind of wanted to ask you guys is, I’ve been in this, I’ve been doing this a while obviously in different, you know, pretty much through my whole career, and it seems to me that the act of writing software and deploying software and now operating it has inherently become a more social thing than it was a decade ago, right? And you don’t have to look any further than things like GitHub Where in a sense part of it is an exhibition sport to see, you know, who’s got the coolest stuff out there, but that fosters a communication. And then I think the other interesting thing, because that’s sort of thing one, I think it’s more social, I’m curious what you guys think. I think thing two is, I think with the DevOps movement and the codification of deployment, that’s actually created a common language between devs and the ops people that probably didn’t exist before because it used to be kind of an independent cowboy deal, and now it’s much more of a common language of writing code and making things happen. I’m curious what you guys think about that social element and how that’s affected things. Do you have that same viewpoint, or has it just kind of always been that way based on your generation? Because I’m a little older.

Matty: [00:23:45] Oh, absolutely. I agree, and especially with your second point because that’s a passion of mine. I’m Mr. Infrastructure as Code. I mean, that’s what I do for a living, is go to organizations and help them learn that. It’s very true. Again, kind of the joke, and Angela has known me a long time, and so I came up as the grizzled old sysadmin that sat in my silo and bitched about those cowboy developers that wanted access to production and all this stuff. I’ve done my 180, but the reason that that can happen is just what you said, is because it’s the fact that doing infrastructure work has become more like development. I guess the irony of it is that that’s really where it started. If we really want to go back and way back before any of our time, you too, Todd, right? You know what I mean? The early, early sysadmins, you had to code. Because you had to write your tools because they didn’t exist. We’re getting back to that, I think, which is what’s very cool because now we’re getting away from— there was a time when you ran ops and there were tools you used. If you wanted a tool, you bought it or you found it or you had someone do it. What I see a lot of now is people that I really look up to or that I work with that I still consider infrastructure people. Are sitting and they’re writing code to solve their problem. I think that helps ops and dev, like you said, speak a common language. In some ways, I think it’s a little easier— the sort of DevOps meld, I feel, is easier for developers than it is for ops because a lot of what we’re talking about doing is adopting practices that developers have been doing for years and that are foreign to ops. Things like test-driven development, you know, hell, testing at all. Version control. As an ops person, I knew what it was because that’s how I deploy code, but I didn’t work in that versioning metaphor unless I had to. Ops people should be learning from dev folks, but by the same token, I think dev people, as we move to this more service-oriented type of software, and people are, you know, more creating things like your Netflixes and where we think about things as a service and not as packaged software. And there’s still plenty of packaged software, but if something’s a service, you have to care a lot more about it operationally than you ever had to as a developer before. And Angela, I know you’ve, you know, you’ve spent a lot of time with a lot of different folks on the ALM side and the dev side. I’m curious to see if you would agree that you’re seeing that shift from folks who are handling the application side of the world.

Angela: [00:26:34] No, absolutely. Of course, in the beginning when I was a consultant the first time around, I didn’t work for a lot of companies that had DevOps teams. A lot of times, scary as it is, we were the ones pushing things to production. Kind of as we got farther and farther As I guess the software world got more mature, there would start to be DevOps groups, but they didn’t call them DevOps back then. It was, oh, those are the networking guys, or those are the IT guys. There was definitely a wall. You knew that you dropped a deliverable in some folder and then they picked it up and did their thing, and there wasn’t really communication. Of course, again, this was a while ago. When I was with Microsoft, so this was like 2005 through 2011, I definitely started seeing a shift, right? We started noticing that, you know, we’d have meetings, right? I was an evangelist on the ALM side, so I primarily was talking to developers. But, you know, we started noticing that infrastructure folks were showing up to the meetings. They were very interested in learning about what source control was being used, what deployment tools were being used, and it was really interesting to see kind of that that change in dynamic, right? It started to become one big team instead of a lot of little teams. I’m not going to say that’s not still true. I certainly work with customers today where, you know, they kind of do the hand waving, oh, there’s a DevOps group that does that stuff, and they don’t pull them in. And so that’s usually my challenge, you know, how to make sure that they do pull them in and they do get them involved in the process. Now, the other thing that you were talking about that was kind of interesting was was the social side, right? Writing software has become a much more social thing than it was in the past. That’s absolutely true. I’m certainly more on the software side than the DevOps side. Speaking to the social aspect of it is probably where I have a little bit more experience. It does seem like the companies that I work for, where they do treat creating software as a social activity, they’re far more successful. They’re, you know, they’re far less dysfunctional, right? They don’t have a lot of the same issues about, you know, contention between teams or, you know, people trying to kind of sandbag on estimates and stuff. They don’t necessarily have those same issues because, you know, it’s very social. They kind of, they all rely on each other, they all trust each other. And I don’t think there’s quite enough stress on that in some of the larger organizations because I feel like the larger the organization gets, of course, it’s harder. Right, to have kind of social groups because now you have, you know, dozens of developers, maybe hundreds instead of small groups. But it’s still possible, right? And I think, I think there almost needs to be more evangelism of that as an aspect, right? Don’t treat software groups like each team needs to be kind of in its own little world. Make sure that everyone is talking because that’s how they learn from each other.

Trevor: [00:29:25] Right. I mean, I’ve Obviously probably been— have the least experience of everybody in the room presently. But even, you know, I will say both jobs I’ve had now have been heavily socially involved. The first one maybe a little bit less. We were kind of siloed in our own areas, but there was always some kind of collaboration when you got stuck. It wasn’t pairing per se, but it was more of a let me pick your brain for a minute to see if we can get around the issue. And to kind of talk to what you were talking about, Angela, earlier, yeah, it’s hard to get— I’m finding it’s harder for devs to understand sometimes the value of owning the— owning or understanding that they do have some responsibility in assisting with ops. I just had a conversation with somebody the other day and we were talking about, you know, maintaining our development environments and preparing them to discuss with the client about how they’re going to— how it needs to fit into their production environments when the product is ready. And, you know, their response was a flat, oh, that’s up to them. They can figure that out, as opposed to, you know, we should be working together to collaborate and make sure everything’s going to be smooth at all times.

Todd: [00:30:46] So, so I think an interesting question, another interesting question, I want to hear what Angela thinks about this, because I agree. I mean, I think that the whole idea of collaboration is so intertwined with the social aspects of this that it’s, it’s kind of, you almost have to talk about the two simultaneously. And I’ve asked this question in other forums, and I’m curious what the people around the table here think about it, is, is, is having a separate DevOps group just kind of a temporal situation. So it does— if you fast forward 10 years, is there such thing as a DevOps group, or is everyone in the DevOps group? Does the group consume everyone in dev and ops eventually? Because it’s kind of a shift in behavior, you know. I mean, you know, we in my company, you know, in the software we build and the product we’re offering, We sort of believe it’s heading that way, where everyone sort of has responsibility on the technical side for the execution of the system, and the lines are blurring partly because of the codification and the common language. And yes, there’ll be different people who have expertise within that group, but everyone sort of has responsibility. Everyone can, at any time, look at their phone and know the health of the organization, you know, the technical health of the system. This is the world we believe in, but I’m curious what, you know, what you guys believe in, in terms of, is it always a separate function, or does it consume all of those technical parts eventually?

Matty: [00:32:21] I’ll answer first, because this is something I feel pretty passionately about, and Trevor’s probably chuckling to himself because—

Trevor: Oh, I am.

Matty: I’m a big believer in that DevOps is a It’s not a tool, title, or team. It’s a philosophy. In our last episode, that was actually one of the myths that we talked about is that you need to have a team called DevOps. We actually have found that that works, but it depends on what you’re calling it. A lot of times, like Sasha last week— or last week— last episode mentioned that a lot of times when people talk about a DevOps team, they’re really talking about a developer tools team. Trevor, you and I were just IMing about this yesterday. I think this is your new job, right?

Angela: Yep.

Trevor: I was just elected stack lead of DevOps, which the subset was CIs, Azure, AWS, etc.

Matty: I believe to your point that where we’re going, though, is that the idea is within a delivery team, a team who’s responsible for delivering a product, delivering software, is a— can and should be a cross-functional team. And like you said, everyone has their core skills. You don’t expect everybody to be able to do everything. And I’ve told teams that I managed in the past that were just sysadmins, but still was that I said, I don’t believe that everybody should be able to do everything, but that anybody should be able to do most things. And that you— and primarily, you shouldn’t— you should never sit there and say, you know, oh well, that’s not— that’s a dev thing, that’s a sysadmin, that’s a QA thing. We did an episode where we talked to developers, and I’m a little peeved that this never played out, but we laid down a challenge at the end of the episode where we said, hey, if you’re a developer and your next stand-up pull a task off the board that’s not a dev task, help somebody else out, and then let us know how that worked out for you.

Trevor: [00:34:26] And I think we actually went to— did it.

Matty: Yeah, I think you’re the only one who did it.

Trevor: You did it too, because you challenged ops to pick up dev stuff. That’s true, I challenged them to do so, but I’m saying you did that challenge. You said that that week you wound up having to do some code.

Todd: Oh, maybe.

Trevor: Well, see, yeah, wonderful. Sorry, bringing up history and you don’t even remember how it went down.

Matty: That’s true. That was ancient history. It was at least a month and a half ago. So that’s sort of my thought, is that it’s really about a cross-functional team. But the reality is, I think that’s more aspirational than executed in a lot of places.

Todd: I would take a little bit of exception to that, because we’re in the position where we see the behaviors of lots of teams. Right, so we have kind of a unique position, and we actually label them kind of old-school and new-school companies. You know, old-school companies will deploy our stuff into just kind of an ops functionality group, and it’s, you know, 20 or 30 seats. But then you’ll find a new-school company that everyone in dev and ops gets our stuff, and that’s, you know, in hundreds of seats. And to me, that’s very indicative of there’s a change afoot here, right? And it’s just Like everything that’s socially based, it’s going to take 5 or 6 years or 10 years to play it out. But, but there are amazingly progressive companies out there that everyone has pager duty. Everyone responds to those stuff. And I think that’s pretty encouraging. It’s actually more— I think it’s very positive. I think it’s much more progressive than I thought it would be when I started the company. Um, so it’s, it’s just an interesting piece of data.

Matty: [00:36:11] I don’t think we disagree. I think that you were— maybe I wasn’t saying it clearly, but that was my point.

Todd: I think we’re both right. You’re right, there’s a lot of companies that aren’t behaving that way, and I’m surprised how many there are that are, but it’s still a mix.

Matty: Yeah, I think we’re coming to an inflection point where you saw it, you’re seeing it where more and more places are seeing the potential value But they don’t necessarily know how to get there. And that’s why people like me have a job.

Trevor: That’s kind of what I’m hoping to facilitate. That’s my goal with this new— it’s not really a title. It’s an honorary thing.

Matty: It’s more work for no more pay.

Todd: Exactly.

Matty: Oh, Trevor, you’re learning.

Todd: So let me poke at the last element of the social change part, because I find this stuff super interesting, which is why I’ve taken over this questioning line. Is there a generational aspect to this? Are the younger guys more likely to be codifying the operations stuff, and the older guys are more likely to hold on to their— and I use guys, but it’s obviously everyone. Is there a generational component? Because I get about mixed answers on that from people.

Angela: [00:37:35] You know, I’ve just— since I’m in consulting, I work with a lot of different companies. I’ve actually seen a mix. You know, I feel like kind of the knee-jerk is, oh, the people who’ve been at the company for 30 years are not going to want to change. And that certainly happens, but I’ve also been really surprised where sometimes, you know, they’re the ones who’ve been doing it for the longest, and so they’re the most fed up. With how things have been going, and so they’re ready for something new. They’re ready for a change. And I think, sometimes I think that fear of changing the way you’re doing things isn’t, it’s not necessarily related to age or how long you’ve been there, but it’s more of a personality thing.

Trevor: That’s exactly what I was thinking.

Angela: Yeah, like, do you feel secure enough that you could change the way you do your job and you’re not going to be seen differently, or you’re not going to lose faith, or it seems like it has a lot more to do with kind of confidence than anything, right? Am I confident that if we completely turn the way I do things upside down, that I’ll still be successful? Or, you know, on the flip side of that too is also, how are those things handled by management, right? Is management willing to kind of take that hit of, we’re going to change things, and it might be ugly for a few months? And if the people that work for them are nervous about their jobs, if they’re afraid they’re going to get yelled at because now things take longer, those are generally the places where I see them more hesitant to change, is where management doesn’t really buy into the concept that change is hard, but it’ll be worth it in the long run.

Trevor: [00:39:00] I’ve seen some of that, but I also think it’s a lot of the people who seem to want to push against it are also pushing against it because they’re more afraid of additional responsibility and having, having more work to do. They think that, you know, so, oh well, this is another set of things that I’m responsible for. I’m going to be here an extra 10 hours a week because now I’m a DevOps person. You know, people, I would say kind of more people who don’t get it are kind of, you know, who have, it hasn’t really been explained to them in a way that it clicks.

Todd: Is it, is it fair to say they, maybe that particular individual, at least some percentage of them actually kind of enjoy the fact they’re a little disconnected from the business as a whole. They kind of like doing what they’re doing. They like, you know, the automation, they like the stuff, but kind of the more— the closer they get to the actual business, they get less interested in it. And this represents sort of something that’s, you know, irreversibly sucking them toward the actual business and what’s going on. Is that fair? Do you think that that goes on?

Trevor: [00:40:06] I think that’s a good way of describing the people I’m thinking of, much more interested in writing software than wanting to be in meetings or being involved in the business.

Todd: Right, and there’s all kinds of people. It’s not that it’s good or bad. It’s just that I think DevOps definitely brings you as a technical person closer to the health and the livelihood and the heartbeat of the business. It’s probably okay not to want to be there, but kind of an unstoppable force in some way, I think.

Matty: What was the— Trevor, I’m trying to remember exactly how Damon Edwards said it last time, but he said something about the biggest blocker or the biggest sacred cow when you talk about DevOps against it is, well, that’s the way we’ve always done it.

Trevor: Yep.

Matty: You know, and I think that’s where you get into— and I do agree, Angela, your point that if someone’s been with an organization for a long time, they can sometimes be really open to this stuff because, like you said, they’re the most fed up. But also, I think it’s not necessarily generational, but it’s experience. Depends on how much experience you’ve had. If you’re someone who’s been working at the same firm for the last 30 years and you know one way to do it and you know the big bank way of doing it, or you know this particular organization’s way of doing it, then you’re probably out of your comfort zone to think about doing something in a different way, especially because generally Stuff works, right? I mean, things are not catching on fire. We’re just trying to make them better. We’re not trying to keep a ship from sinking, especially if you’ve been somewhere for 20 years. The ship probably isn’t sinking, right? So it’s hard to argue against that, but it’s, you know, I don’t remember the rest of what I was going to say.

Todd: [00:41:53] I do have an empathy for kind of old-school ops people, right, in this world. Because I think it’s much easier for a dev to come over to this and go, yeah, you codify it, and it’s like, why wouldn’t you do that? It’s so much easier. And so you have to learn kind of some new things, but if you’re kind of an old-school ops person, you have to actually learn to code. I mean, because some of them don’t know that, right? I’ve had a lot of these teams, and that’s probably pretty scary because, you know, you sort of feel the force, you feel what’s happening, You’re, you know, it’s like anybody in any career, like when you start to see this, a sea change occur, it can be a little scary. So I definitely have empathy, empathy for those people because they’re super smart people. They’re historically like the most passionate. They have this undying knowledge that they have to keep the machine running and they get absolutely no credit for it at all. The CEO probably doesn’t know unless it goes to total shit. And so you have, you know, it’s kind of unfair, but that’s kind of what needs to happen, I think, to maintain your career, you know. And it’s much easier on the dev side, I think.

Matty: [00:43:09] Right. And like I said, most of my— up until recently, I’ve spent my entire career in ops. And it kind of makes me laugh when you said that about, you know, getting— you know, nobody knows what you do. And my last The dot-com I was in at the— well, I guess we’re a dot-com— the e-commerce company I worked for, on our Yammer site, you know, you’d have to have your little profile explain what you did, and mine said, you have no idea what my team does, and that’s a good thing. You know, because if you knew what we did, you know, if you know what TechOps is doing, we’ve done something wrong. But the downside is, and also, you know, you take it very personally, and ops people are used to being, you know, Sort of my joke has always been that ops doesn’t have a dog to kick. If you think about the whole thing about so-and-so yells at so-and-so, it goes down, and somebody kicks the dog. Well, the dog doesn’t get to kick anybody. The other downside, the other way of saying that is that shit flows downhill to tech ops, and that’s where it stops. They’re the ones who feel, rightly or wrongly, and the thing is sometimes this is incorrect, but tech ops feels like they’re the ones who are on the hook for stability. Things that look like they threaten stability, and the irony is a lot of this stuff we’re talking about, when we talk about continuous delivery, small batch size, quick release, DevOps, all this stuff is all built really around more stability. I think that’s where it becomes a really hard sell to get people who are used to saying, I want my command and control because I trust my team and that’s it. To see that there’s really a more stable way of doing it.

Todd: [00:44:46] I think there’s an argument to be made, too, that if you really adopt continuous deployment and you’re true to it, it doesn’t necessarily mean you’re less busy as a traditional ops person. In fact, I would make the case you actually have kind of a harder job, but when you do run into problems, they tend to be much smaller. And easier to solve, but the frequency is higher because you’re essentially destabilizing the system to some degree potentially every single day. But when the problem does occur, it’s like if you just have access to the right person, they’ll go, oh, that’s that one thing. Okay, great, and then it’s fixed. It’s not like it was, you know, 10 years ago when you do a deployment every 6 months and then everything’s, you know, rat-fucked for 6 months or for a whole week. Sorry, I have to get wrapped up here. It was getting late.

Matty: I think the problems become more interesting too when we get into that. When I go work with clients to understand, we say we’re doing infrastructure automation, we’re doing this infrastructure as code and more automated release and all this stuff. We say we’re not trying to automate people out of work. Right. None of this is reduction in headcount. We never—

Todd: [00:46:10] I’ve never known one of these teams to get smaller.

Matty: No, we’re just saying we’re giving you something more interesting to do. I had a team before when we were doing this when I was in industry, and when we kind of started the project, they sat there and said, well, what are we going to do? I said, you’re going to do what I’m paying you to do, because I’m paying you an awful lot of money to copy files around. I’d rather you innovate and do something good. Helpful and do what you’re good at. I think that’s a huge one. When we think about, again, how we talk, we’ve talked about how we can communicate more. We’ve talked about some tools. Now, what about just even within a discipline? There’s an interesting thing that I think it was Jez Humble that said when he was talking about continuous integration and about how CI is, and just source control in general, and commits basically are forcing developers to talk to each other. And, you know, Angela, I know you’re very passionate around ALM, and I’m curious to what you, what you see as good practices around the lifecycle of an application and making sure all the right people are knowing the things they need to know.

Angela: [00:47:24] Right. The thing that I always tell everybody, regardless of what their process is, right, like, I think everybody should be looking at What are considered the best practices of Agile, which if you boil it down are just best practices in software development. They have nothing to do with Agile, right? Talking to your team every day, making sure that you’re constantly prioritizing based on the right things. Those are things you should be doing regardless of what your methodology is. But specifically having those standups. I talked to some of the most waterfall customers you’ve ever met, but that’s just how they have to operate because of what they do, right? That’s just their industry. But I’ve, I’ve gotten, I’ve convinced a lot of them to start doing stand-ups every morning. Like, I don’t care if you only deliver once every 4 months, you should still be meeting every day for at least 15 minutes, right? Talk about what issues you have, discuss what features are going out, you know, in this, you know, small release, because people tend to think, well, we’re only releasing every 6 months. No, you’re only releasing to the public every 6 months, right? That doesn’t mean that internally you can’t actually be very cyclical and work in very small chunks and then just you know, kick it out to the public once every 6 months. But I know it kind of goes back to what I was saying earlier about just getting face-to-face with people and, and just talking, right? I think people, it’s been a while since this was true, but people did get way too used to just sitting in their cube all day and working in a silo and their problems were their problems. And occasionally they might go, you know, hey, could you help me with this? But there wasn’t a whole lot of that, right? And I think there’s starting to be more and more of that being socialized within companies. It’s You know, it’s okay to sit next to somebody for a couple hours and bang it out because it’s easier than you sitting at your desk for a day and a half trying to do the same thing. So I just encourage people to do that and even within other groups. A lot of times people think that’s just a developer thing, but I encourage testers, BAs, PMs, everybody, you know, get together with not only your whole team, but get together with each other, right? There’s no reason developers across different products can’t learn from each other or QA people that work in different teams can’t talk to each other.

Todd: [00:49:24] Totally agree.

Matty: So as we’re, as we’re coming up to our end, I want to go around the horn and have everybody give just a one, one, one or 2 sentences. If you had to give one tip that would be how you can improve collaboration or communication within your team, what would be your one thing that you want someone to come away from this podcast with?

Trevor: And I’m going to throw you under the bus this time. You’re going first.

Matty: Oh, okay. Sweet. No email. Stop using email. Find something else.

Angela: So I’ll go next. My biggest piece of advice is use your feet. Go talk to people, right? I feel like email, IM just becomes a crutch. And there’s, there’s way too much context that can be lost. You can misconstrue someone’s tone in an email. Just, you know, just move and talk to people if you can, right. If you can’t, get on Google Hangout or Lync or Skype or whatever, right. But face-to-face is always best if you can do it.

Todd: [00:50:28] Yeah, I think mine would be, you know, agile. Every technology or every kind of movement we’ve talked about today is driving development and operations closer to the heartbeat of the business. So make sure what you’re doing is going there, right? And one of the key, the key things that we didn’t talk about, but it’s one of my things that I wanted people to take away was figure out how to quantify your job in terms of value to the business, because it’s the single biggest thing I see when I talk to teams is they actually don’t understand the value of downtime in their business. And, and because of that, you know, they don’t actually have a number. They don’t know what that is. And that causes them problems all the way through the system because they can’t advocate for equipment they know they need. They can’t advocate for tools they know they should buy. And it’s because the business speaks, you know, the business. It doesn’t speak because it’s broken, right? And I think that DevOps gives the ability to quantify things because you’re starting to write code, you’re keeping statistics. You just have to go that one more step to make it into real money, and then every— it will unlock the universe for you in your business. And I think that it’s a whole other topic in and of itself, but I think it’s kind of where this is all going.

Trevor: [00:51:58] And for me, I would just say Make sure you know how to talk to everybody on your team. Not everybody’s the same. Make sure you can express yourself well to the other members of your team and the rest of your company. You know, if the CEO walks by and he asks you a question about your project, make sure you’re able to describe it in a way that makes sense to them. Or, you know, just be mindful of who you’re talking to. You know, it’s always going to help when you collaborate.

Matty: Agree.

Trevor: Great.

Matty: Alright, so it’s time for checkouts and where everybody will suggest something for the audience to check out. Angela, you want to jump in?

Angela: Sure. I’ve got 2. I wasn’t exactly sure where to go with this, so I did one that was on topic and one that’s just kind of a fun, silly thing that I like. So I hope that’s alright.

Matty: We like the fun, silly things the best.

Angela: So I’ll save that for last. So the first thing is actually a book, and this is, this is so not a software book, but it’s one that I keep referring people to, and everyone who reads it thanks me for it. So it’s a book called Drive, and it’s by Daniel Pink, and I’m sure, yeah, yeah, I see heads nodding. Yeah, I’ve been to so many conferences, Agile and ALM related, where they’re like, you must read this book. It’s short, it’s like $10, but the great thing about the book is it helps you to understand what motivates people and what drives people to act and react the way they do. Because, you know, I often joke that ever since I kind of moved out of software development and more into ALM and process, I feel like I need to go back to school and get a degree in psychology to understand, you know, exactly what Trevor was saying. How do you communicate with people in a way that’s effective? And how do you make sure that you kind of encourage the right behavior and respond in ways that are effective for each person because we’re not all the same? So I always recommend that book because it’s short, doesn’t matter what you do for a living, it’s super effective at helping you to at least understand what motivates people, right? They don’t disagree with you because they’re a bad person, they disagree with you because, you know, one of anything that could be kind of affecting where they’re coming from. So that’s my kind of podcast related. The non-podcast related, because we’re dealing with so much cold weather, there’s a store that I love, a site that you should go to called Sock Dreams, so literally sockdreams.com. I have tons of leg warmers and wool socks that are keeping me from freezing to death right now. So it’s a fun store, and if you’re also in the Chicagoland area freezing from the polar vortex, check them out.

Todd: [00:54:30] Awesome. I have 2 things, but the first thing— and I took the same kind of deal— so the first thing here is— can’t really see it, but you’ll have it up there— is a research brief by Aberdeen Group about data center downtime and how much does it really cost. And this goes directly— it’s like 4 pages, right, because it’s kind of a leader for another thing. So easy to consume, easy to look at. But the reason I encourage especially DevOps people to read it is this notion of being able to quantify what you’re doing in the language of the business. And when you read this, you’ll think in a different light when you ask to buy a specific piece of software or a specific piece of service. Piece of hardware, or even a consultant, or whatever you need, what you’re spending is trivial compared to what you’re saving the business, and you inherently need to be able to quantify that. I think it’s super interesting. It’ll kind of get you in the mindset and get you going there. The second thing is my previous business was all about the independent online publishers. We had thousands and thousands and thousands of them, so I got to know a lot of them. And there’s a great one that is a cooking blog, and it has the best barbecue chicken recipe I have ever found, and I encourage you to try it.

Matty: [00:55:52] Great. Trevor?

Trevor: So my serious one this time is an article by— I’m going to totally butcher this person’s name, and I apologize if they listen to this— by Emmanuel, I want to say, Goussart. It’s called Coding for SSDs, and basically it’s a deep dive on how SSDs work and how best to optimize your code to take advantage of the things that SSDs can do that hard drives can’t do. And it’s really interesting and really detailed, and I will post a link to it in the notes. Secondly, I’ve been listening to the Vitamin String Quartet. A lot lately, which is a group I’ve liked for a long time. They take contemporary popular-ish music and compose it for string quartet and play it. There’s 96 albums of them on Google Play covering everything from the Beatles to Metallica to the Decemberists to Skrillex. Daft Punk. Mostly you name it, they’ve done something from a band, and it’s all really fun to hear music you love in quartet form. Yeah, Spider-Man String Quartet. Matt?

Matty: [00:57:14] Great. Okay, I have a couple, and as my hat would indicate, they’re, they’re all chef-oriented. So the first one is for a tool called Mise, as in mise en place, I think is how you say it, which is what’s the French word for, like, basically setting up your cooking station. And it’s a Ruby gem that puts everything in place that you need with Berkshelf and Vagrant and all the assorted things to get started. It’s a project that’s a work in progress. We’ve been following it pretty closely. I’m real excited about it. I’ll put a link to it in the show notes. It’s pretty awesome if you’re a chef-type person and want to get things going. Also wanted to talk about— so ChefConf is coming up in April, and I’ve just confirmed that I will be there. I’m not speaking or anything. I’m just going. So— and I think the early bird pricing is ending soon. So if you’re a chef-type person, you should come because I’ll be there. And I do believe that Food Fight and Ship Show are going to be there, too. We may be doing a joint podcast. Then one other thing, a local thing to check out is this Thursday, as in 2 days. Well, it’s the 27th. Who knows when you’re listening to it? You may have missed it, but the Downtown Chicago Azure Meetup is having our Speed Dating in the Cloud event at CloudBakers where we’ll be doing a showdown between Google’s cloud technologies and Microsoft’s. I will be representing Microsoft Azure Infrastructure as a Service. Go ahead and check that out. I’ll put the link in the show notes. My final little shout— 2 little shoutouts that I want to give is my company, 10th Magnitude. We’re hiring for a .NET developer. If you’re in the Chicago area and you’re a .NET rock star and would like to join a pretty awesome place, you can come to arresteddevops.com/jobs, and there’s a link to the job. We also are doing a contest right now at Arrested DevOps. And so if you come to arresteddevops.com/giveaway, you can enter our contest and you might win yourself an Amazon.com gift card. So I just want to make sure to thank everybody for joining us. Thank you to Angela and Todd. It’s been a great conversation. Really appreciate your time and your input. Everybody make sure you check us out at— see, I told you I was going to mess up the name of the damn podcast. Geez, one of these days. You can check us out at arresteddevops.com or @ArrestedDevOps on Twitter.

Trevor: [00:59:55] I’m Matt, @MattStratton, and I’m Trevor, @TrevorGHess.

Matty: We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.