agile and devops

Matty: [00:00:07] Welcome to Arrested DevOps, the podcast that probably won’t destroy your career with bad advice. I’m your host, Matt Stratton.

Trevor: [00:00:15] And I’m your co-host, Trevor Hess.

Matty: [00:00:17] Actually, yeah, that’s true. I’m your co-host. We are equal. We don’t have rankings here. Nobody is better or smarter than anybody else. So, welcome to— this is episode 4. Of Arrested DevOps, and we’re going to be talking about Agile and how it can be used in a DevOps organization and maybe some tricks and tips or things where Agile might work differently if you’re being DevOpsy or not. But speaking of Agile, we’re going to start with the retro as we always do. So mine’s going to be pretty quick. I sat down and tried to think about what I’ve done in the last 2 weeks since our last podcast. And I couldn’t really think of anything that I did. But what I did realize is in the last podcast, we kind of threw down a challenge— again, an Agile-related challenge— where we challenged developers who listen to the podcast in their next stand-up to pull a sticky off the board that wasn’t a development task. And we said, hey, send us a tweet, let us know how that worked for you. And the response we got was basically a big Nothing. So either nobody did it because you’re a bunch of chickens, or nobody told us how it went. So I’m going to reinstitute the challenge, and I’m especially throwing this out at our guest from last week, Dan D’Angelo, who told me and promised me and assured me he was going to try it. So Dan, let us know. Also, on the note of tweeting, don’t forget, if you have questions for our panelists as we’re going, please feel free to tweet us live at @ArrestedDevOps, or if you’re watching the livestream, there is this time a little Q&A box, and there really is, so you can ask her questions. Trevor, what have you been up to?

Trevor: [00:01:58] I also accepted the challenge. One of the things I had to do in the past week was set up some automated migrations for our database, and so I figured out how to, with TeamCity, use remote PowerShell scripts to run a SQL backup and SQL restore between servers, which was a little challenging and very fun.

Matty: [00:02:24] Let’s let the panel introduce themselves. We’ll start with Len. Len, tell us a little bit about yourself.

Len: [00:02:30] Hey, thanks, Matt. Thanks, Trevor. It’s really cool to be part of Arrested DevOps this week. My name is Len Lagestee. I’m an Agile coach for Redpoint Technologies, and I’m also a pretty active blogger on illustratedagile.com. I’ve been really involved with Agile starting in 2004 and have really been either teaching it or practicing it ever since. It’s really great to be here and part of this podcast.

Trevor: [00:02:56] Great. We’re glad to have you. Patrick, you want to go ahead and introduce yourself as well?

Patrick: [00:03:02] Thank you. Thank you, Matt. Thank you, Trevor. My name is Patrick O’Brien. I am a lifelong consultant and project manager. Much more recent to the world of Agile than Len is. I spent my career being a Waterfall expert and fighting Agile toward the recent years as it’s become a flavor or practice to be implemented or be used for implementations, fighting it tooth and nail until finally the last few reasons for hating it fell away, and the next thing I know, I’m a convert myself, without letting go of my waterfall past, because every tool has its use. So you can look at me as a sergeant in the trenches.

Trevor: [00:03:51] Well, welcome to Agile, and welcome to Arrested DevOps, both of you.

Patrick: [00:03:56] Thank you. Thanks.

Trevor: [00:03:57] Well, let me ask, who would like to give kind of a cursory overview of just what is Agile, you know, the 2-sentence overview of what it is, just in case for some reason anybody listening has managed to not hear about Agile.

Len: [00:04:15] I can give it a go at first, and Patrick, feel free to chime in at any point. You know, you could read the Agile Manifesto online and kind of figure out what the values are and the principles, but really what we like to say is it comes down to 4 key tenets, and one is it’s about people. It’s about putting people in a position to do their best work and quite frankly to have fun while they are doing it. It’s about collaboration, it’s about putting those people together and to start co-creating as quick as possible as opposed to handing things off as they are working. It’s about results. Ultimately most of us are in businesses that ultimately need to deliver and have some sort of profit, so it’s about delivering results. It’s nice to have all the cool culture stuff, and have fun places, but ultimately, you know, unless we’re making money, we’re going to go out of business. And then lastly, it’s kind of just about responsiveness. We need to be able to respond to changing markets and changing customers and changing technology very, very quickly. So as opposed to having these long, lengthy projects, we have short iterations that allow us to change very fluidly and in a dynamic fashion. So that’s it. Agile is kind of an umbrella term for other practices. But ultimately, so there’s events and ceremonies that we can use to live into those tenets, but that’s it at a high level. Patrick, feel free to add on if I missed anything.

Patrick: [00:05:43] You bet. I don’t know if I’ll add anything, maybe just my own particular take on it. I don’t know where I got it in my mind just now, but somehow the number 4 different reasons behind Agile. I’m not sure if Trevor said it or not, or I just invented it, but communication, transparency, teamwork, and evolution. I’m not sure if you can see my hand with all the 4 fingers there, but I found in my experience that Agile is very effective at making that— surfacing that, but also doing it very naturally so you’re not feeling like you’re forced, whereas in the waterfall environment, you have to make a meeting, you have to sign the requirements, you have to wrote, wrote, wrote, wrote.

Trevor: [00:06:27] Absolutely. That’s my experience with, you know, a lot of the clients we work with are coming from waterfall, and you see them struggling sometimes to fight that want to have a meeting for everything. It’s clear when you see the projects here that are really, really sticking agile and the ones that are fighting to stay agile, and you just see kind of the difference in how smooth things go.

Len: [00:06:53] Oh yeah, definitely. Well, if you think about it, there’s probably maybe 10 to 15% of organizations out there that are— and that’s just kind of throwing that number out there— it’s a small number that are really doing agile well, and the rest of them are all needing to go through some major transformations in order to make this work, and cultural transformations to make that work. So it’s often more painful than it needs to be, but it’s ultimately a journey that organizations need to go through.

Trevor: [00:07:21] Is it true that in Agile, anyone outside the delivery team should, you know, not care how the sausage gets made, so to speak? Matt has always believed that a team is a black box to a product owner, and the way it should work, everything should be feature request in and the team outputs a feature on the other end. Do you guys agree with that?

Patrick: [00:07:44] I disagree with the word should in that question. It is up to— I’m sorry if I’m stepping on you there, Len— it is up to the culture of the organization as to whether people outside of the dev team should or can see how the sausage is made. In my experience, and full disclosure, I’d been doing something that is now called Scrum since long before Agile was a formulated manifesto because I’d always felt like the right thing to do. I Scrum every day, or I used to call them what’s hots or stand-ups, right? So not too far to get there. But back to the point, it’s up to the culture of the organization as to whether they’re really ready to do it, because if they’re not, they’re either blinded by the transparency or intimidated by it, and they try to get their hooks into, well, what about this, what about this, what about this? And then you drag yourself into the weeds. If you are ready, then people will find them— will appreciate the transparency. So there’s a point A and a point B there, and it depends on where on that temperature they are ready to doing that. So I don’t agree with the word should in that question, being should anyone outside the dev team see it. It just depends on the maturity of the organization.

Len: [00:09:09] Yeah, I think that’s well said, and I believe the same things. Generally, when I’m coaching agile teams, from the very beginning. I like to coach, build kind of transparency into it, so they’re used to it from the very beginning. But I think exactly as Patrick had mentioned, when we’re first getting started with perhaps a product owner that’s a little nervous and maybe unsure of themselves and just getting used to the role and just beginning to understand what that role is all about, I think having external stakeholders in there earlier certainly might cause a little bit of tension and perhaps keep them from really living into that role. So ultimately, once a team begins to mature, I think they almost want to be transparent. And so what we usually do is we just offer up to anybody, any stakeholder in the organization, if you want to stop by a review session, if you want to stop by a stand-up meeting or a planning session, feel free to stop by, kind of open invitations. The one thing that we do keep pretty sacred when we coach is the retrospective. So what we’d like to try to do is keep kind of the retrospective a place for the family to talk about family business. And so we’ve often found when folks will kind of just interject themselves in the retrospective, the team will shut down a bit and maybe not share bad news. We see this particularly with managers. If people have direct reports and a manager will go into retrospectives, we’ll often see folks shut down. So what we like to do is just keep the retrospectives kind of sacred for the team and exclusive to the team. And if anybody’s got any issues with how the team’s working then to either talk to the product owner or to the Scrum Master directly. But we try to keep it pretty transparent, and Agile teams honestly are pretty fun to watch. If you’ve got a really good team that’s really humming, there’s an energy and a buzz about it. And so I think it is actually exciting for the team just to open it up and see what they’re doing.

Patrick: [00:10:58] Excuse me, Len, I have a question for you then. In your retrospectives in this particular context, are you also doing demos of what has been built in the last sprint, or is that a separate gathering of individuals? I’m sorry, Trevor.

Trevor: [00:11:15] No, that’s fine. I was going to ask a similar question.

Patrick: [00:11:18] Got you then.

Len: [00:11:20] Yeah, yeah, 2 very distinct ceremonies, for lack of a better word. So we’ll have a review session where stakeholders will attend and the team will actually demonstrate the work. Typically we’ll have the product owner or a tester or somebody actually read the user story and the acceptance criteria for the story. And the team will demonstrate that. And then once they’re finished with all the stories that they want to demonstrate, we’ll excuse everybody else in the room. And after maybe the first time, that might be a little bit awkward to say, hey, we’re going to kind of talk about some things ourselves. But then after that, people get used to it, and then the team can just gather for the retrospective separately. Okay.

Trevor: [00:11:53] Yeah, that’s kind of— that’s the format I’m more familiar with. We don’t necessarily call it a review session. For us, it’s part of our iteration planning meeting.

Len: [00:12:02] Cool.

Trevor: [00:12:02] But yes, the concept is the same, and we roll right into retro afterward, and anybody who’s not a core member of the team is out of there.

Len: [00:12:11] Perfect, yeah, good, good, good stuff.

Trevor: [00:12:15] And I’m forgetting as I’m talking with my hands that my hands are down here and nobody can see them. The issues with talking with your hands. It’s so true. So on kind of evolving more about the team, should there be isolated tasks? So for example, should we say that, you know, there are dev tasks and there are QA tasks, there are UX tasks, or should the team be collaborating more as they’re going forward?

Len: [00:12:44] You want to take that one first, Patrick, and I can jump in?

Patrick: [00:12:46] Oh, oh, you’re going to— the Agile Coach is tossing it to the Sergeant. Okay, I see how it is.

Len: [00:12:50] Alright.

Patrick: [00:12:52] I do have an opinion on this. And it— I have found it just in my experience, there’s the disclaimer, that it’s most effective when you do have specific tasks to specific teams, because then you at least have the illusion of ownership. Whether that illusion is accurate or just an illusion is the next question, of course. But if you just throw it out there, it’s like a steak to a pack of dogs, and you’re not going to get anywhere. So if it is a dev task, it is a dev task. The dev team fulfills it. And sometimes you have multiple layers of a dev team. Like, you could have database, you could have GUI, you could have design, all within dev, right? And so if you don’t assign it to one of those streams, then it’s just disorganized and you defeat yourself. Also, and this is true with waterfall, and this is a philosophical thing. It’s important to assign this because in my experience, you do not want your testers also have been your developers. And that is incredibly important, especially if you’re getting all the way down to UAT. Yes, you may be tempted for capacity reasons to reemploy your dev folks or your designers to do UAT, but that’s an important safety tip for myself is don’t do that, because then you’re literally letting the fox into the henhouse. So that answers my question and my philosophy of do you have specific tasks to specific individuals is yes.

Trevor: [00:14:25] So what about as we approach some tasks that are kind of— need some more input? So for example, during our baking process, sometimes we really need some strong dev input. Or during some more, you know, kind of tying back to the DevOps task subject, you know, there’s a kind of an opsie, DevOpsie kind of task. And, you know, recently we had to, you know, set up our automation server to run our automated tests, not unit tests, but actual click tests. And because I wasn’t— I as a developer wasn’t really familiar with how the particular tool worked, it was more beneficial to pair with our QA guy to put together the automation.

Len: [00:15:17] Yeah, so there’s probably kind of multiple ways to kind of handle this. What we’ve done before— so in this type of situation where you’ve got kind of a— it’s almost a DevOps story, so it’s not a user story. So, you know, an end user will not get a benefit typically from kind of this technical work that needs to happen. So it’s not like as say, DevOps engineer, IDEs. So we wouldn’t really have that. So we’d have other user stories that we’d have in place for the actual feature work, but we can also have what we often have, kind of architecture stories or technical debt stories. And that might be specific to things that the team needs. So there’s some things that the team needs to be much more efficient around their development and around automation or how to do continuous integration, all that kind of stuff to build up the team. Hopefully we can get some of that stuff done before the project actually starts or the product starts. And so within a sprint, we can focus on delivering value to the users. But what I would typically do there is if somebody came to me, if a developer came to me and said, hey, here’s some things that we need to do in addition to our features this sprint, just to make things easier for us or better for us, we’d include that in the planning session of the sprint, and we’d size it with the other stories. So is this something that’s really easy to do or really hard to do? If it’s going to be half an hour or it’s going to be multiple days to do, and then we’ll adjust how many features we do accordingly. So the team will say, hey, we’re going to take a quarter of our time and spend it on these technical stories, and we’re going to spend 75% of our time and actually spend it on feature work. And then that story itself would have tasks, and those tasks would be things that you can actually write on, you know, I need to install something on the server, or whatever the case may be, whatever that technical story might be. So again, does that make sense?

Patrick: [00:16:57] So I have a question for Len on that. Do you mind if I ask?

Trevor: [00:17:00] Go right ahead.

Patrick: [00:17:02] Len, on something you just mentioned right there, you said technical debt stories or production support stories.

Trevor: [00:17:07] See, he’s asking my questions again.

Patrick: [00:17:10] Should I just shut up and just wait for the answer?

Matty: [00:17:12] No, go right ahead.

Trevor: [00:17:13] Yeah, yeah, it’s a discussion for a reason.

Patrick: [00:17:17] Do you find it effective to actually create an epic to encapsulate those kinds of user stories?

Len: [00:17:25] Only if necessary. So if it’s really big stuff. So ultimately what we like to see is stories that are hopefully aligned to some sort of architectural vision or architectural direction. So if there’s a place where the team needs to go architecturally, we need to upgrade something, or we need to install something, or we need to get better at something, or we need to remove technical debt, hopefully that’s part of an overall architectural vision that we can align to an architectural roadmap that says, hey, here’s when we need to actually do those upgrades. Or we need to do whatever that technical debt is. And at what point, what’s cool then is what we can do is once we have that, we can integrate it with the product owner’s roadmap. So the product owner now understands, hey, there’s some technical debt here, and historically there’s always been this big divide between business folks and technology. And so when we go to business and say, hey, we’ve got to go do this upgrade and you’re going to get a whole bunch of benefit from this, or we’re going to go remove some technical debt, they put, you know, deer in the headlights stare and say, well, I just want you guys to build features. But now, as we start getting some of these roadmaps intertwined, that hey, there are some things we need to do technically to improve, but we also know we need to deliver value to our customers. It’s nice to build that all into one single roadmap, and the team just consumes that, you know, as one single roadmap as opposed to single initiatives. So ultimately, you could have an epic, you could have kind of features, I suppose, but ultimately it just comes down to how do you want to get things in a granular format for the team to consume.

Trevor: [00:18:52] So you actually, in your experience, you actually call them technical stories, because that’s a heated discussion.

Len: [00:19:00] Yeah, it absolutely is. And so typically our first smell when you write user stories is as soon as you write a role on the team as part of the as a user story, so as a developer I need this, that’s a first sign that you’re probably writing a pretty bad user story. But ultimately, though, there is work that needs to occur within the team, and so we want to kind of put that somewhere. If we try to just do it on the side and really not take account for some of this work that we know needs to happen— and the cool thing about doing that is just bringing that stuff to light, the product owner starts really understanding what it takes to deliver their work. And after a while, they start saying, hey— I’ve had product owners start asking architects and starting asking technical folks, hey, is there anything you guys need to do to make our work more efficient? And hopefully the team velocity starts to improve because you’re doing some of these things. If you’re just doing them for the sake of doing them, obviously your velocity will start getting lower and lower. So again, you could debate it, but ultimately that work’s got to go somewhere, and if it’s all going to fall on one team, I prefer one roadmap. It’s just so much easier to consume, it’s easier to size it, it’s easier to plan sprints. So why not just make it easy for the whole team and say, hey, this is one— our one system of record for all the work that we do is going to be in one roadmap.

Trevor: [00:20:15] That makes a lot of sense to me. I mean, the way I’m familiar with doing it is a lot of those get thrown into chores.

Len: [00:20:21] Yeah.

Trevor: [00:20:22] And, you know, it’s hard to size a chore.

Len: [00:20:24] Yeah, exactly. And it’s hard to test a chore, you know. That’s true too. And so it’s nice to go through the process of building acceptance criteria. So if we do this, if we remove this technical debt, what’s some of the acceptance criteria? How do we know this is actually making an improvement to what we do? So it’s nice to get to go through the formalities. You don’t need to get too rigorous with it, but it’s nice to go through the actual ceremonies of Agile and Scrum to make that happen.

Matty: [00:20:49] So, by the way, I’m back. Those of you who are listening or whatever may have noticed that I wasn’t here, and it was pointed out to me that this is Arrested DevOps, not Abandoned DevOps. I am multitasking, and we had a staff meeting.

Patrick: [00:21:04] It’s all good.

Matty: [00:21:05] I’m in DevOps.

Patrick: [00:21:06] They have forgotten.

Matty: [00:21:09] I’m looking forward to editing this so I can hear everything you guys talked about during the time I was gone. It’s scary. So at this point, we had some technical difficulties with our recording and there was a dropout. To give you some context, the question was raised, what is the role of the Scrum Master?

Len: [00:21:28] Governing teams. And so they’re no longer kind of sitting in their cube taking work breakdown structures and building a chart, you know, a Gantt chart, and then assigning their tasks and then going getting status of that. What we’re really looking for them to do is one, is to focus on the process itself. So they’re kind of neutral, they take a neutral stance when it comes to process. That allows them to observe the team and see if those conversations we talked earlier about, kind of the key values around people and collaboration and results and responsiveness, is that really happening? So they’re really on the lookout for any of those team dysfunctions that are going to keep them from really being productive. The second thing, and this is probably the most crucial thing, is to remove impediments. Now this doesn’t mean that they’re the ones actually doing the removal of the impediment, but they’re shepherding that removal. So if there’s something that’s blocking a team from being productive and from delivering value, we would expect that Scrum Master to escalate as necessary, to go find the right people to get together as quickly as possible to clear that runway for the team. Oftentimes what we’re seeing, and I found this in my own experience as well, is the Scrum Master ends up being a bit of a psychologist, and so as there’s some dysfunctions kind of emerge from the teams, it’s amazing how people start opening up to the Scrum Master. And so it’s kind of a multifaceted role. I do believe that it’s a role that is absolutely necessary, especially for those folks who are undergoing some sort of kind of Agile transformation or cultural transformation that’s so used to working in a different way. It’s nice to have somebody there to kind of take that neutral stance. Taking an observation of the team and see if they’re really being agile. So oftentimes what I’ll call Scrum Masters are kind of agile coaches themselves. In fact, that’s my exit strategy. My exit strategy as an agile coach when I go to clients is the Scrum Masters. So I know if I can get the Scrum Masters in a state where they can do the same observations that I’m doing of teams, whether they’re functional or dysfunctional, and they can do that on their own, then I know they’re in good shape and I can actually exit an organization. Long-winded answer. Patrick, I want you to fire away, my friend.

Patrick: [00:23:36] Well, I’ll cherry-pick for a second. One of the things— I agree, in the broad term, that is what a Scrum Master is. And of course, me being in the trenches, I’d like to talk about what that really means to the folks that are in the trenches with me, also grabbing the bullets that are flying through the air. And one of the first things that I’d like to bring up is the fact that the Scrum Master is there to enforce whatever processes have been agreed to. A key point would be the Scrum Board, right? And there are a billion different methods that you can use and apply to the Scrum Board. I have a particular website that I like to peruse every now and then. I think it’s called agileboardhacks.com that I get to see, oh, this is how this guy does this. Maybe I can fold that into my thinking when it comes to trying to figure out how to manage my dependencies, which is— it was the last, by the way, it was the last blocker for me to adopting Agile myself is how do I manage dependencies. But the point is, the Agile Scrum Master is there to manage the Scrum, go around your team as you’re doing it. What did you do yesterday? What are you doing tomorrow? What are your blockers? How can I help? Who’s got the ball? When are you going to get it done? And then moving the pieces on the board. I actually prefer to let the team members move the pieces on the Scrum board because that’s cathartic, isn’t it? It’s a really good feeling when you go from dev done to QA ready, or QA ready to QA in process, right? And I like the team to do that because then that gets them all jazzed and keeps them— keeps the morale up, which is a great segue to another part of the Scrum Master, and that is to be a coach. Not just an Agile coach, but a personnel coach. You’re doing a good job. Thank you for coming in today. No, you like that? You like what I did there, huh? So it’s, it’s, it’s the person that not only enforces the rules that everybody has agreed to, but does it in a way without being overt about it. They’re an effective Agile coach is someone who is a coach instead of a manager.

Matty: [00:25:51] That was, to me, I love that you said that because that was one thing that I found that’s my understanding of a Scrum Master. What I would look at out of a Scrum Master that I was looking for is, again, I earlier today was talking about questions I wanted to ask and I said, isn’t an Agile or Scrum Master like the Agile cop? And then I was like, oh, that’s such a non-Agile thing to say, right? Unless you’re crazy about it. But I’ve had the case where people tend, I’ve seen that people tend to say, I want my Scrum Master to be what I think a project manager is. So the Scrum Master is a person who writes the reports and explains why things didn’t happen and things like that. To me, it’s like there’s some of the things a PM does, but really it’s more about the process, right? More about getting people through it, not like writing up reports for senior management or, you know, explaining that we’re delayed or why we’re delayed and whose fault it is and all that kind of stuff.

Patrick: [00:26:49] Yeah, yeah, hold on a second, Len. May I interject for just a second? It can be. Again, it depends on the culture. I will say this as a Scrum Master, coach, project manager myself: if you’re gonna be late with something you said you’re gonna do, I will come after you, but I’ll be nice about it. And so that when I do have to come after you again, you’re not going to be trying to hide under your desk. So it’s a very careful— and I try to do it as a waterfall PM as well, but the Scrum Master, I think, is more important to be a good coach than even just a PM, because a PM, they— it’s so established, the waterfall is so established, you’re expected to be the guy with the hatchet, right? Scrum Master, you have a little more freedom to be— help me out with the term, Len. A little sneakier with your motivations.

Len: [00:27:48] Yeah, I think that’s a good point, because ultimately what we need to remember is kind of some key tenets of Agile. It’s all about kind of self-organization and self-accountability. So to have a Scrum Master become that project manager and start pounding people for tasks isn’t really what we’re after. So I think when that does happen, when somebody is late with their task and somebody’s maybe perhaps not delivering, somebody says, hey, I got a 2-hour task here and you’re on day 3 of your daily standup and that task hasn’t moved out of in progress. I think what we would really like to do is start coaching the team to start questioning themselves. So if there’s somebody there who’s— as opposed to a Scrum Master calling somebody out, we might say to a developer or another somebody at a standup, hey, just whisper in their ear, hey, why don’t you just ask about that one task that’s been in progress for 3 days? So it’s not a Scrum Master calling them out, but the team starts holding themselves accountable. And then, hey, they notice that it’s not a Scrum Master that’s watching them. It’s the entire team, and if this person doesn’t get their work done, the entire team suffers. It’s not a Scrum Master thing. Honestly, the Scrum Master doesn’t have much vested interest in the team itself, in the actual work itself, in the product itself. It’s the team that’s responsible, the collective that’s much more responsible for it. So I’d like to see much more of kind of some, to your point, kind of that stealthiness, that sneakiness. Not so much around kind of saying, hey, I’m going to go and hound somebody to get this task done, but to really coach that team on what it really means to be together and to move together. And that’s when it gets really powerful, when somebody else on the team— I’ve said this to multiple Scrum Masters as I’ve been coaching them— you know, the best Scrum Masters are the ones that have to say the fewest words. If you can get a Scrum Master to sit in the corner during a stand-up meeting and you just watch the team kind of go through their stand-up meeting, and if a task, a 2-hour task, is in progress for the 4th day, they call themselves out. And they start saying, hey, why is this here? We need to get this done before we can get this next story, start working on this next story. That’s when it gets really powerful, and that’s when it gets really fun to watch these teams start humming.

Patrick: [00:29:51] You know what, Len, if I may come back with it, absolutely. You said 2 things there that really spoke to me, and I’d like to point out. One of them is the tipping point. There are several tipping points, and one of the big ones is when the team itself starts to scrum itself. You have the QA guy saying to the dev guy, I didn’t get that yesterday, what’s going on? I’m just sitting back going, you know, that’s great, good for you for bringing that up. I don’t have to shuttle. I’m just up there with a Post-it note ready to move it into the next phase or move it backwards, God forbid. The other point that you did was let’s back the truck up to a time before that that tipping point occurs, that Agile also provides public humiliation. It does, and that’s the way people encounter it when you’re first starting. I call it adulation. It’s the process of adulating a team. They’re coming from Waterfall or, God forbid, heroics, project management by heroics, and they’re going to become Agile. They first encounter it as possible public humiliation. I don’t deny that to them. I say, yeah, it can be.

Matty: [00:31:03] Do you want to give an idea? I have a question about that though, just to clarify. The public humiliation should be within the delivery team, right? Absolutely. So to speak, it shouldn’t be a, we’re at the demo and oh yeah, we didn’t get this feature done and it’s because Joe didn’t do his story.

Patrick: [00:31:21] That’s correct. You weren’t here for when Len was talking about that at the beginning.

Matty: [00:31:25] Okay, okay, well good. That’s something that is near and dear to my heart.

Patrick: [00:31:29] Public humiliation within a microcosm.

Len: [00:31:32] Yes.

Patrick: [00:31:33] I’m done.

Len: [00:31:35] No, no, it’s not so much public humiliation, that’s kind of a strong word. It’s really more just a sense of, hey, I’ve got a brother in a foxhole and I’m not going anywhere without him, and if I need to help with that task, it’s more of— usually what we coach is if there’s a task sitting there, we’ll just say, hey, do you need any help with that? So that’s all we’d ask. I’d coach another developer to say, instead of saying, hey, you’re really late, why has that been there for 3 days? Hey, listen, just go ask if they need any help. And usually that just opens up the dialogue. So much of what Agile is and so much of what I do all day long is I just get people together so they can start talking to each other. And that’s all I would do in this case here is, hey, there’s a task here. This person, there’s probably a backstory to it. Either this person is brand new, he’s probably stuck, or he doesn’t want to admit that he doesn’t know what he doesn’t know, and so he doesn’t want to bring anything up. He’s a little fearful to raise an impediment. So then we just, you know, just little prompts and little nudging is all it takes. So it’s not so much of a humiliation, it’s more of, hey, what can I do to help you get out of this foxhole? Let’s go and move forward together.

Patrick: [00:32:39] So when we talk— I’m sorry, Matt. The nuance of what I was talking about, about the public humiliation, is that is how it’s viewed initially, and that can be leveraged. And leveraging the coaching mentality of we can get past this into a team effort. So I just wanted to clarify that public humiliation, you strip away all the niceties, that is it toward the beginning, and it can be encountered that way. Until people get used to the idea.

Matty: [00:33:04] I think it’s— and to me, I think the accountability, the internal accountability, is super powerful if you— because a lot of the time, one of the things that we talk about a lot in DevOps when we talk about the culture of it is the idea of having a blameless culture. And to me, it seems like, again, we talk about within that team and again, sort of the way I posed the question, the way I’d written it, and I don’t know if this— again, bear with me if Trevor didn’t use my term, but saying about how outside the team we don’t care how the sausage gets made, right? So it’s sort of like you said, this delivery team to me is this family who is going to stick up for each other. And yeah, inside our standup, maybe we’re going to hold each other accountable because if Joe doesn’t do his tasks, the team is going to suffer.

Patrick: [00:33:53] Right?

Matty: [00:33:53] Because then we’re all going to be the ones who fail at that review. So we want to— it’s 2 things. One is, right, like we said, there’s sort of the positive side is, hey, let me pick you up and help you along. The other part is, hey, jackass, do your job, you know, because you’re going to make us look bad, you know. So it’s self-policing.

Len: [00:34:10] Exactly.

Trevor: [00:34:11] Yeah. And I guess I did mention your sausage.

Len: [00:34:14] Okay. Oh my.

Trevor: [00:34:17] Okay.

Matty: [00:34:19] Oh, okay, let’s not get political on this podcast. The part, so then I think about that, and so like Len, you’re saying so much of what you do is about communication and teaching collaboration, and one of the questions I had when I think about the Scrum Master is if we say, okay, and our philosophy here when we talk about DevOps is it’s, at the end of the day, DevOps, you know, we totally stole this from Don Vincent, but we believe that DevOps means never saying that’s not my job. And you heard at the beginning, at the top of the podcast, you know, we kind of had laid down this challenge saying pull something off the board that maybe isn’t normally yours. And is that something— so like you had kind of alluded to, you know, kind of talking in the Scrum Master’s ear.

Trevor: [00:35:03] Can you—

Matty: [00:35:04] if I’m an organization that either A, has already become Agile and I say I want to be more DevOpsy, or I want to do them together, does that seem like a natural fit for a Scrum Master to be able to help that type of coaching as well?

Len: [00:35:18] Yeah, I think anytime that there is a place for departments or groups or different teams to work together across the organization, I think a Scrum Master can absolutely play that role. I think when it comes to introducing DevOps thinking, I think also an architect or somebody who is a little bit more technical, because I think there are some technical things that come into play when it comes to DevOps, would be important as well. So I think if you’ve got— I mentioned this earlier— if you’ve got some sort of architectural vision to become more DevOpsy, you’ve got some sort of roadmap that says, here’s the things we need to do to get better at continuous integration and monitoring, all these kinds of things, and learning from what we’ve deployed and getting a little more kind of lean startup thinking in place. I think having that architecture in place there as well is really going to be beneficial. But as far as a Scrum Master can certainly help with getting the right people involved with talking to each other and introducing new things, I think it’s definitely a win if you can get the Scrum Master involved.

Trevor: [00:36:17] So that makes me think of another one of the things we try to enforce a lot here is the notion of a self-organizing team. Yeah. How do you guys think that fits in with this notion of the Scrum Master and DevOps and kind of not being afraid or not saying you can’t pick up a task because it’s not yours?

Len: [00:36:37] Yeah, it’s so much just mindset coaching, because I think so often, you know, when you think about the way organizations have been set up for so long, it is all about kind of the individual. You know, we’re rated typically as an individual, we get our performance reviews as individuals, and to start shifting some of that thinking into much more of a collaborative thinking and to be self-organizing around a team dynamic, it takes some real coaching and it takes some real work, and it takes the team to actually go through some trials and tribulations together to actually get to that point. So there’s been no team that I’ve worked with that have— I’m trying to think if there’s been one, but I can’t even think of one that’s gotten together immediately and gelled in a heartbeat. They’ve had to go through some trials and tribulations. They’ve had to figure out how to make things work together. So the key thing that we do is it’s very easy for leadership or management to kind of come into these types of situations and say, hey, you guys need to be more DevOpsy, and kind of force that. But what we like to try to do is say, hey, what are the pain points you guys are really solving? Team here, let’s talk through this. What are the pain points that you guys have about deploying and release management and trying to get stuff out to production? Where are the things that you guys are really struggling with? And work with operations folks to start figuring out how to solve some of those things. Then you build kind of what we call communities of practice, but you can call them whatever, centers of excellence, whatever, where you get people with different roles across all of your product teams. To start to get together to solve these problems at a holistic level. So you’re not just solving them at a team level, you’re solving them for the entire organization. So I think that’s really some of the mindset that we try to bring in culture from the very beginning, collaboration, so whoever needs to be involved, if it’s operations folks in your planning sessions, we absolutely invite them in there. But then start talking about what’s painful in your releases. You know, it’s not uncommon for me to go into organizations and they’ll come back to me and say, hey, we’re going to be agile, but just so you know, it takes us 4 weeks to get something out into production. We have to go through these CAB boards and approvals, and we have to go through all this rigmarole just to get something out to production. It’s like, you know what, you’re not going to be very agile unless we start figuring some of that stuff out. So let’s start talking and looking at some of those pain points and addressing those collectively. So often people think we can just flip a switch on agile and all of a sudden we can get stuff out to production in a heartbeat. Well, you know, many of the folks that I work with have legacy systems and archaic architecture and legacy plumbing in place that doesn’t allow that to happen.

Patrick: [00:39:04] So you’ve got to start—

Len: [00:39:07] this is kind of a little bit about that technical debt stuff that we talked about earlier— you’ve got to start capturing some of those pain points and pulling them into the teams and into the communities to start solving them together. If it’s a mandate from on high, if it’s a mandate from leadership, I very rarely see that actually work. You know, sometimes it will work, but boy, when you can get that to emerge and bubble up from the actual practitioners who are actually doing the work, that’s when it gets really powerful and it actually starts being pretty sticky as well.

Patrick: [00:39:31] You know what, Len, you just said something there. I’m sorry if I’m interrupting.

Len: [00:39:36] All good.

Patrick: [00:39:37] Okay, and that is— and I think we’re at the 4th level of our agenda here. We talked about whose fault it is, and you guys, I think we can expand upon it. But there’s an elephant in the room here, and it’s one that I have experienced firsthand a number of times, and that is what is Agile? How far into Agile an organization is ready to go? And in my experience there are some degrees here, and depending on what is necessary to be completed or rolled out, you can choose what you need. You can go for pure waterfall if it has a beginning, middle, and end, a fixed point. Agile is not that great at fixed points or budget management. Let’s be perfectly honest there. It’s good at continual. That’s what it was designed for. However, you can do it. There’s also a middle ground between waterfall and agile, and that is a sort of a water Agile fall or Scrum fall, which I’ve implemented as well. But the point is I get—

Trevor: [00:40:40] We called it fragile.

Patrick: [00:40:41] Fragile, yes. Well said. But I get concerned, and this is my point, when somebody says we’re going to go Agile. What do you mean by that is my next question.

Matty: [00:40:54] The same thing happens with DevOps. People say, like you said, we want to be more DevOpsy, download the DevOps, hire me some DevOps. And you’re like, what do you mean?

Patrick: [00:41:03] As if it’s a tool they can install, turn the knob, and everything’s wonderful, like a panacea. And that is not at all the case, especially if you do it halfway. If you want to say, we’re going to do scrums, we’re going to do sprints, my next question is, how? Right? And I know I’m stepping in Len’s sandbox when I do that, but, you know, again, I was the guy that fought it for years. I did, even when I didn’t believe in Agile, I believed in the fact that if you’re going to do it, do it all the way. Don’t do it halfway. Don’t have scrums if you’re not going to have a scrum board to manage the items in your scrums so you can have an informed conversation in front of the whole team. Items like that, and I’m just curious, I have my own personal opinions about that, and I’m curious to know what Len sees about that because the Agilation process is not an easy one. Teams can tend to resist certain aspects of Agile that if you leave out sabotages the entire Agile concept. You can probably leave out certain aspects like backlog meetings, right, meetings about your backlog. Okay, we can let that one slide, but Scrum? You gotta do it every day. You can’t have it every other day or every week. That’s not a Scrum, that’s a status meeting. So yeah, that’s— those are my— I’m just on a soapbox right now.

Matty: [00:42:33] This podcast exists for people to go on soapboxes. That’s pretty much why Trevor and I are here every 2 weeks. And I actually, if I can push you to the side of your soapbox, there was a point we had an— those of you who are listening to this on the audio later will not know that this happened because I’m going to fix it. But we did have some technical difficulties, and I apparently was talking to myself for a few minutes at one point. And I started to bring this up, and none of you heard it. But I’m curious, like, so we talked about— and again, everybody kind of does it differently, but I think traditionally when we think about a delivery team, there’s kind of that, like, delivery triangle, right? You’ve got, like, your developer and your tester, and I don’t remember who the third one is. I know it sure as hell wasn’t ops. And that’s sort of a question I’m curious about. I’ve seen this sort of like model in the past where the idea is you have this team that sort of runs along underneath all of the delivery teams, and that’s your infrastructure team, and they give you your infrastructure. And I was in an organization where we actually invented a role in the delivery team that we called a system engineer because we felt like we needed somebody in there to do that. Yeah, and I can’t claim that, yeah, because we knew about DevOps and we’re being really clever, because we didn’t necessarily do an awesome job of it. But my question is, like, Len, you alluded to, you said, okay, bring the ops people in early in the planning. And I would posit and say, why shouldn’t there be somebody operationally involved in delivery all the time?

Len: [00:44:04] Yeah, for the most part, I would actually prefer whoever really has a vested interest in delivering software or the product out to the customers should be a part of the team. Oftentimes what happens is teams can’t afford that, or organizations can’t afford that. I think it’s, you know, depending on the size of the organization, if you’ve got— and some folks I’m working with, you know, 30 teams flying around, you know, to have a DevOps person on each one of those might get a little bit tricky or expensive or whatever the case may be. Oftentimes what happens is we don’t have enough. We’ll just maybe say that. In which case then, you need to have somebody at least representing what the DevOps is all about. So hopefully you’ve got either like a lead engineer or an architect on the team that really gets what DevOps is all about, and then pull you guys in as needed. So it is tricky. For me, any stakeholder that has a vested interest in helping get a product successfully out the door should be on the team, exactly to your point, Matt. But what we’re finding is it’s kind of tough to do that all the time. We’re often finding it tough to even just get architects on all the teams. We’re short of architects, so it’s really tricky.

Matty: [00:45:15] And then that’s to me, like when you go back to the philosophy of DevOps, is that if we still go back to this idea that, you know, DevOps is about breaking down silos and not saying this is my delivery team, now I delivered this thing, now I hand it over to the release team to get it out there. And so it’s this really weird dissonance that I run into, because I’ve lived in the organization where we said, okay, I don’t have enough— because fundamentally we said that the ops person is a sysadmin. And yeah, you should— you really should not have enough sysadmins to have one on all of your teams, because that means you’ve got some really complex infrastructure that requires that many people. Like, you should have way more developers than you have sysadmins. And I’m this 20-year sysadmin, and I’m telling you that you should have more developers. Yeah, so it’s true. But so my thing is then, how— one of the ways that we can do that is through some of the practices that we espouse in DevOps, which is to say that you don’t have this— again, it’s like I hate to always harp on this whole like, that’s a developer thing, that’s a tester thing, that’s a TechOps thing. It’s like I understand there’s going to always be places, right, like I can’t— I’m a developer, I can’t pull off a task from the board that’s test the software that I wrote. But why can’t it sometimes be test the software that Trevor wrote? And sometimes I can’t pull a task off the board that says build a server in production because I don’t have access for it, but maybe I can help. And what are— I guess so this is the one thing I’m looking for, is to continue to go back to this. We’re trying to encourage people to look within their organization and their team to stop thinking in those boxes. And what are, you know, maybe some tips in this? And then we’re kind of coming to the top of our hour, so we’ll kind of probably end up with this some ways that you’ve either seen or you would recommend to someone who’s a practitioner to say, stop thinking about things as a developer thing or a tester thing or an architect thing and think about your team?

Len: [00:47:15] Yeah, wow, that’s a great, fascinating question to certainly end this on.

Patrick: [00:47:21] Well, that was definitely profound, I’ll give it that. You wrapped it well.

Trevor: [00:47:25] Yeah.

Len: [00:47:28] Wow, where do you even start with that? So, hopefully what we can get, if we can get some real kind of architectural leadership and technical leadership on a team early on in the process, they can at the very least represent DevOps, as I mentioned earlier. But ultimately, we need to get even some DevOps folks even probably into more of the coaching role with the teams as well earlier to start working with them and to kind of start eating their own dog food, for lack of a better word. What we used to love to do is, and I can’t do this with all the teams because there are actually most organizations that I work with actually have a separate support group that actually supports the applications. So exactly to your point, some of you will be developing a project or a product and actually hand it over to a support group to maintain it. Is for at least for a period, we’d actually have the team itself support the application. So if you have an agile team and they are struggling with kind of either kind of doing bad things, poor craftsmanship, throwing it over the wall and it’s breaking, having releases and it’s not going very successfully, you have to roll it back. What we do is actually say, hey listen, for a while, while you guys are developing features, you are also going to be supporting this application. And what would happen then is if something were to emerge that would typically a support group would handle, we’d interrupt the sprint and make it actually a little painful to say, listen, hey, we’re going to stop what we’re doing. In some cases, we’ll abort this sprint. We’re going to have the whole team rally around this issue. Let’s get it fixed, and let’s get it resolved, and get it out the door. And then we’ll pick up a quick little planning session and start the sprint over again. So until people actually can kind of experience the pain of release and the pain of having somebody else support their work, it gets really, really kind of tough to build up that empathy naturally. You know, some people just aren’t naturally empathetic to what other folks are up against. But ultimately—

Matty: [00:49:25] I was going to say, that’s kind of like the classic Amazon example, right, where all the developers carry a pager.

Len: [00:49:30] Exactly.

Trevor: [00:49:31] I’d just like to tack on to that, that, you know, like I said, this week I was picking up a couple more opsy tasks than my traditional dev tasks, and it was sort of a cascading effect. I had people, you know, more than one person saying, hey, can I pair with you on that?

Matty: [00:49:47] I thought you were going to say cascading, that suddenly everybody was giving you even more ops, and then that’s not helping because you already do too much.

Patrick: [00:49:54] I just remembered something. So what I was thinking was, I’m actually involved in this in a project I’m running right now, and that is— and Len, you said it, the key word— toss it over the fence. And that is, it is easy for the dev team to toss it over the fence to QA, and the QA team to toss it over the fence to the UAT, or whoever follows in that particular process. And right now we’re trying to— we, notice the word we— are trying to instill the value of the team being interested in that particular user story all the way to deployment, and then maybe back again if it shows up as a user request to change in the future. So we’re breaking down the barriers between the key points in the Scrum board by having people have ownership, the team having ownership of that user story all the way through. Oh yeah, I remember that, that’s a report function, that’s a little bit lower than this particular contract required thing, so let’s all get behind that. And it’s a part of building the team, so you’re going horizontally across. If you were to look at it on the board, horizontally across on the board than anything else. So it’s about team ownership, and almost like we are reaching back to the very beginning of this podcast, that team ownership is really what this is all about, and Agile does get us there.

Len: [00:51:17] Yeah, that’s a great point. Just real quick, Trevor, I think what you said is so crucial. Anytime, and we see this quite a bit, anytime that you would have traditionally had a handoff, that is now what we would have— it’s a co-creation point. So what traditionally would have been a dev to tester handoff, now we want dev and testers to be working together. What would have traditionally been a dev and DevOps or ops handoff is now your opportunity to co-create. So the more opportunities that we can kind of intermingle amongst the groups, the better.

Trevor: [00:51:48] Absolutely.

Matty: [00:51:49] Great, I love it.

Trevor: [00:51:50] So we are reaching the top of the hour now. Matt, I think that was what you were about to say.

Matty: [00:51:56] That was, yeah. This is a co-creation point.

Len: [00:52:00] There we go. Well played, sir.

Trevor: [00:52:04] So I guess let’s move to our checkouts. Matt, if you want to start us off.

Matty: [00:52:09] Sure. I got 2 checkouts this week. The first is something I’ve known for a while, and if you are anyone who uses— who is considered a developer— if you’re a developer or power user that uses Windows as your operating system, I highly recommend you check out Scott Hanselman’s Ultimate Developer and Power Users Tool List for Windows. We’ll obviously post a link in the show notes. Scott Hanselman, he updates this thing about once a year, give or take. Well, every few years, actually. But it’s pretty much every little power tool. Think about it as a whole bunch of the types of checkouts we do, but it’s like one giant page of it. This is where I learned about things like Chocolatey and BoxStarter and things like that. Again, if you’re a power user or you do development or ops on the Windows side with your client, really recommend it. And then my non-IT related one is, I think last week, last time I may have mentioned that I just picked up the graphic novel of Game of Thrones, and it’s really amazing because it’s not drawn based on the TV show or anything, so it’s a different visioning of the characters, and it’s just— there’s something to me, I don’t know, the way that that story manages across multiple mediums I think is pretty pretty cool. So what you got, Trevor?

Trevor: [00:53:28] Well, I don’t have anything IT-related this week because I’m just— I’m a little fried from writing all those PowerShell scripts. So I don’t know if I want to call it a recommendation yet, but I’m interested so far in what I’ve seen of the new show Helix. It’s being produced by Ronald D. Moore, who did Battlestar and wrote a bunch of stuff for the later Star Trek series. So I’m hoping that it continues to hold my interest. If not, Archer’s back tonight, so I’ll have that to enjoy. Len, you want to go ahead and tell us what piqued your interest recently?

Matty: [00:54:07] Yeah, yeah.

Len: [00:54:08] I’ll probably get too serious now, but I’ll have to mind my—

Patrick: [00:54:13] You don’t say.

Len: [00:54:16] It has to be something kind of Agile-related, of course. But really my mentor, somebody I’ve worked with for the past 3 years or so, his name is Cy Elhir. You can check out his blog, just really some great thinking around some new, even expanding Agile in ways that we never thought really possible. Him and a couple of my colleagues are putting together a little bit of a new kind of transformation framework, and it’s called Conscious Agility. And it brings in some elements of antifragility, brings in some elements from business agility and conscious capitalism, and it’s just some really cool reading and some cool stuff. So check out his blog, which is salhir.wordpress.com, and also consciousagility.com, and you’ll be happy you did.

Trevor: [00:55:04] Cool, thanks, Len. Patrick, you want to swing us around?

Patrick: [00:55:09] Sure, I may have 1 or 10. When it comes to Agile, I will highly recommend my favorite site, and that is agileboardhacks.com. You can put a www in front of it or not. agileboardhacks.com. If you were to get a book, I would recommend Disciplined Agile. It’s on Amazon. Also, Writing Effective Use Cases is another good book to pick up. That is project management methodology agnostic. As for fun things to see, I’m a big fan of Castle, which is tonight in about 2 hours. Nathan Fillion, you gotta love the guy. Chicago Fire, Chicago PD, which premiered last week, I’m watching that very carefully. They just shot an episode in my neighborhood, so I’m curious to see when that shows up. And then my constant and my fiancée’s constant guilty pleasure, in fact it’s our crack, and that is investigation discovery. We just cannot change the channel. So those are my checkouts.

Matty: [00:56:20] Awesome.

Trevor: [00:56:21] Thank both you guys for joining us this week. Len, everything you had to say was really insightful and interesting. And Patrick, I don’t know if I’ve heard anybody with such strong opinions besides Matt.

Matty: [00:56:32] We’re gonna work well together. Thanks to our panel. This has been great, and I very much look forward to listening to the beginning that I missed when you guys were all insightful without me. Everybody, thanks for listening. We will be back in a couple weeks. I believe we will be talking about continuous integration. If we’re not, we’re going to be talking about collaboration, or— yeah, we’re going to be talking about continuous integration. This is why I edit the audio in post. But thank you for listening. As always, you can reach out to us with show ideas or questions on Twitter at @ArrestedDevOps. You can find us on the web at www.arresteddevops.com. Why I’m saying www instead of whatever, because I think it’s 2002, I guess. I don’t know. And also, we take show ideas at our GitHub repo, which is also linked to at arresteddevops.com. So this is Matt Stratton saying thanks for listening, and we’ll— you’ll hear from us in a couple weeks.