Click a paragraph to jump to that point in the audio.
Bridget: [00:00:00] Fuck it, we’ll do it live. This is Canada, I probably shouldn’t swear, right? I have to be super polite. I’m sorry. Wait, I’m sorry there, I can say it like that. Welcome to Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and something, something your team maximum DevOps awesomeness. You can tell I usually read that from a script. I have no script here. We’re doing this live. This is your host, Bridget Kromhout. I’m @bridgetkromhout on Twitter. And Arrested DevOps is brought to you by delightful sponsors, some of whom are at this conference today. Let’s do a quick pause. Arrested DevOps is brought to you by TenthMagnitude, a company that figures if you’re listening to this podcast, you must be pretty cool. TenthMagnitude empowers businesses to better collaborate across teams and achieve IT transformation using cloud. They enable customers to innovate, automate, and accelerate by leveraging the power of Microsoft Azure. You can find out more at arresteddevops.com/10thmagnitude. This episode is brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 120 common infrastructure tools such as Chef, Docker, and AWS. So that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. This episode is sponsored by VictorOps. Built for modern incident management, VictorOps provides a unified platform for real-time alerting, collaboration, and documentation driven by your IT and DevOps system data. VictorOps helps you to respond to incidents more effectively so you can minimize downtime and make being on call suck less. Visit arresteddevops.com/victorops to schedule a demo or start your trial. Mention you heard about VictorOps on Arrested DevOps and you’ll be eligible for some sweet discounts too. Let’s talk to our fine guests here today. Just starting from my right, if you could introduce yourselves, say your name, maybe where you work, and what you’re doing in this conference.
Amy: [00:02:22] Like, presumably you have a reason for being Hi guys, my name is Amy Mansell, and I’m actually an organizer of DevOps Days Toronto. Full disclosure, this is my first time talking on a podcast, as well as my first time organizing the conference. In my day job, I work full-time as an events and special projects manager at Lighthouse Labs, which is a developer bootcamp in Toronto and Vancouver.
Bridget: Awesome, thank you so much, Amy. Very excited to have you here. Did I forget to say we’re recording live at DevOps Days Toronto? It’s certainly possible I did. Because, you know, whatevs. Anyway, thank you, Amy. Let’s hear from our other guests.
Aaron: Scripts are overrated anyway. I’m Aaron Aldrich. I am here, I was speaking here at DevOps Days Toronto on managing fires. So, my slides, I tweeted out, and I’m sure they’ll have recordings and everything later. So, I work as a DevOps consultant and community builder for Cage Data in Connecticut, which is in the United States. Yeah, I’m also organizing DevOps Days Hartford later this year. So, I’m also secretly here for reconnaissance and stealing ideas from other organizers. Oh, and you can find me on Twitter @Crayzeigh, which is spelled C-R-A-Y-Z-E-I-G-H.
Bridget: [00:03:40] That’s entirely difficult/confusing. Searching the internet helps with that for sure. But we do plan on having show notes. So, I’ll be getting everybody’s Twitter handles and what have you. And we’ll just have little bio pages for you on our website.
Aaron: On the upside, if you search that spelling, you’ll only find me. Yeah, Zed. Is that not what I said?
Roderick: No, yeah, okay.
Bridget: We are in Canada. It is spelled with a Zed. All right.
Arthur: Hi, I’m Arthur Maltson. I’m also here as a speaker, or co-speaker, I should say. We did a talk on deep work and structuring your DevOps team for awesomeness. I work at Capital One, and you can follow me on Twitter also @amaltzon.
Bridget: All right.
Roderick: Hello, my name is Roderick Randolph. I am also a co-speaker with Arthur. I’m a lead engineer, lead DevOps engineer at Capital One. So, I lead the DevOps practice for Capital One Canada. Arthur and I are part of what we’re calling a software studio. Up in North York where we’re bringing the technology team closer to product and able to produce great products for our customers.
Bridget: [00:04:55] All right. Thank you so much for joining us, Roderick. We’ll get you on Twitter one of these days. Thank you so much for joining us. So, I’m excited to talk to this crew in particular, and there’s so many wonderful people at this conference that I could be talking to, and it’s really tough narrowing it down. But But I thought when I listened to your talks, and then I watched Amy running around organizing, that there is something to be said about ops people running conferences and about the confluence between production launches and production events. And I would love to hear, just starting with Amy, since you’re new to organizing DevOps Days in general, Can you kind of give us a picture of how did they trick you into this anyway? And what was either surprising or delightful? Like, what, for our listeners who are thinking in their city, they might want to organize a DevOps Days, what are the pro tips that you can tell them about?
Amy: Oh, I don’t know how many pro tips I have to give. But about a year ago, I was working in the tech space, going to a bunch of meetups, including a lot of developer meetups. The DevOps Toronto meetup, and I happened to meet Steve Pereira, who is one of the organizers of DevOps Day Toronto. Shout out to Steve, who’s not here today, because he’s not feeling well. So, we wish he was here. He would probably be talking instead of me.
Bridget: [00:06:20] We had Steve on this podcast last year. Okay.
Amy: So, I met Steve, and we happened to connect actually a little while after that, and he reached out to me to see if I would be interested in joining the organizing team. So, me being a super paranoid person and wanting to know the scope of everything before I said yes.
Bridget: Hashtag ops life.
Amy: Yeah, had several meetings with him to ask him a million questions and flesh out every little detail of what he envisioned my role being. So after that, I got to meet some of our— the other organizers, and we had a meeting, and I just thought it was a really awesome event. It was my first time being part of an event that has such a worldwide presence, that’s in so many countries, so many cities, and is so grassroots-driven, I guess, that it really appealed to me. It’s something I like to do in my daily work, is a lot of community building and community focus and not necessarily profit-driven projects. So I was really interested in this concept of a bunch of people coming together who all work in the DevOps space to put together this conference. And I don’t know if any of you have planned a conference, but it’s not exactly Easy. It’s a full-time job in and of itself. So it was pretty cool to see a bunch of people who all work really hard in their daily life coming together to put on this conference. Pro tips, talk a lot on Slack. We basically have nonstop communication over Slack at all times. I think sometimes they get a little annoyed with me at the number of messages and direct messages I send if someone doesn’t respond to me.
Bridget: [00:07:59] So that’s actually— that’s a really good point. And I think that ties directly into the stuff that Roderick and Arthur were talking about earlier, this idea of there’s interrupt-driven communication that’s super necessary for live-fire production events, whether it’s a product launch or a conference launch. But how do you— and I’m curious, and I want to hear specifically from Roderick and Arthur, how do you balance when You talked a little in your talk about how you have the people who are on interrupt and the people who are not, but we all know that the future is here, but it’s not evenly distributed in people’s heads. How do you deal with that crisis when Amy is DMing the one person who needs to answer her, or you’ve got the one person who’s supposed to be heads down, but someone really needs them?
Roderick: How do you deal with that? So part of it is, you know, finding time to actually focus, right? And not being afraid to say, hey, you know, I need this time to actually work. Because often, I mean, we are interrupted constantly. We are working on some project, we’re trying to build a great new tool for our devs, and every time someone comes up and bothers us and says, hey, I have this issue I need to work on, it takes us away. And some of the research has shown that there’s like every time you’re interrupted, it takes 25 minutes to get back to what you were trying to do earlier. And so you have to kind of make time for being able to have deep work. And if you’re working in a team environment, you’re able to kind of share that with the rest of the team. And so the model that Arthur and I talked about was splitting the team into 2, into a kind of a sub-team where part of the team is working on foreground and they’re handling the firefighting while the folks in the background are kind of helping to prevent some of those firefighting— preventing some of those fires from actually occurring in the first place.
Bridget: [00:10:01] So, do you— can you maybe— I think Arthur’s going to maybe address a little more how exactly you decide where to put that split because it sounds like if everybody who understands Project X is in the background right now, good luck.
Arthur: I think that’s what I wanted to say. I think your initial point of You know, if Amy’s DMing just that one person and only they can answer, I think that’s the problem to begin with. And that’s really a challenge that we still have and we’re still working through, trying to spread that knowledge to a point where everybody can answer. So it’s not just that one person. And I’m hoping at some point we can evolve into a pair programming or pair opsing model where people— and that’s really the best way to distribute that information and to really do that knowledge sharing.
Bridget: Yeah, I know, that’s an interesting point just because there is a difference, like you were saying, there’s a difference between the interrupt-driven reactive work and the deep contemplative work. And I would say, I would push back on the idea of pairing and saying that it’s hard to do deep contemplation and have a conversation at the same time. And Pivotal is a big fan of pairing, but that—
Arthur: [00:11:13] Right. That’s a good point. Yeah, for sure. I think definitely on the foreground, when 2 people are together kind of doing that interruption, they can hopefully learn from each other. And then again, depending on if you need that deep contemplation, maybe you separate at that point. But we’re not there yet as a team, and I’m just hoping to start experimenting with that at some point.
Bridget: Now, I’m going to turn the metaphorical spotlight. The lights are up. We’re on the stage of this conference room, this conference venue, but the lights are up, we don’t have spotlights, but I’m going to turn the metaphorical spotlight to Aaron for just a minute and say, in your talk, you were talking about managing disasters, managing communication in disasters. What if your incident commander is a single point of failure in and of themselves? Like, how do you deal with that?
Aaron: That’s a great question. That’s actually an interesting question. So, I think it would depend on what you’re talking about in an incident. I think they would be a single point of failure, but the odds of that single point failing while they’re active like we have an incident right now and in this however many hours block, you know, they are obviously readily available. Yeah, stuff happens, right? So, at the same time, yeah, I think the best practice is probably to have someone who’s backup or someone who can fill in if they’re not available. You know, best practices is a great word we all like to throw around and then pretend they’re not real when we actually put them in place. But yeah, that’s the best practice is to make sure you have someone who can back up. So, if that incident commander becomes unavailable, you have someone else who can step into that role and assume leadership of that group.
Bridget: [00:12:47] It comes to mind specifically because of what Amy was saying earlier about Steve Pereira is a DevOps Days Toronto organizer who unfortunately happened to be ill overlapping the exact days of the conference, which we all missed you, Steve. But I’m wondering if you can kind of talk about what the process is like inside the conference organizing team for dealing with the fact that a major person helping with this conference suddenly became unavailable? And keep in mind that I will be taking this advice to heart because I’m definitely a single point of failure for DevOpsDays Minneapolis.
Amy: For the most part, we tried to make sure, even before, without foreseeing that this was going to happen, we tried to make sure that all information was disseminated, that everything went through our organizer group email, that we tried to communicate as much as possible in a group chat. Even if it was, I just needed someone, something directly from Eric or Abir, it was, it went into the group chat so 4 other people could see that at the same time and making sure that even if I dealt with a lot of stuff one-on-one in terms of sponsors and speakers, but making sure all that information was constantly disseminated back to the team. We had regular team meetings so that even at the point when it came to Steve saying like, guys, I’m just, I’m not gonna make it, I can’t even get out of bed today, that we just were able to, we knew what was going on, we were super organized, we had everything in in shared files, everyone had access to everything so that at that point, there was almost just like a playlist that you could go through and check off that was there. So it could run pretty seamlessly with or without him. And I think that’s a good way to work on all projects. Even if someone doesn’t need to know something at that point, it can get a little annoying. But in my experience with events, anything can happen. And having people who sort of know what kind of should go on, in the moment, they’ll be able to just take it over and have a framework to work within.
Arthur: [00:14:38] I was going to say, actually, like, this is one of the big challenges that we’re running into is how do you redirect conversations that would normally happen in a private chat or in person into a group chat where everybody can see it? And kind of building up that culture is something that takes a lot of work, but I think it’s worth it in the end. And from a previous employer, my experience has been if everybody goes to that group chat, then maybe that one person who knows the answer isn’t there, but other people have seen them answer this before, and they can answer it for other people. And that might be somebody that’s not even on the DevOps team. It could be other engineers that are lurking in the channel. They can be answering those questions.
Aaron: Yeah, that public chat is a huge thing to make sure, especially happening in the public channels. This has been a great thing to actually watch shift in our own team Slack, is I’ve watched it shift from the quantity of private messages versus public messages. And we’ve had to be very deliberate reinforcing that message of, like, please do this in public chat. And I’ll even— I’ve literally copied and pasted messages people have private messaged me. I’m like, this is not a private thing. Like, you can just ask me in the public channel. And now everybody knows.
Arthur: [00:15:50] The ones that were safe for work, right?
Aaron: Yeah, yeah. Well, it was all default open is what we said, like default to the public channel. And if it needs to be private, it can be private. But by default, it just goes to the main channel.
Bridget: And I should point out that a lot of this does have to do with what kind of culture your organization has and what kind of safety people feel around asking questions. Because if people are going to feel targeted, or laughed at, or made fun of, or whatever, for having asked that question, they might ask it in private because they feel bad about asking it. So that’s something that you can kind of try to hack just by making sure it’s safe, and demonstrated to be safe, to ask questions.
Arthur: Yeah, and that’s where we can use our privilege to kind of show other people that, you know what, even somebody who’s in a senior position doesn’t really know everything, and that’s totally cool.
Amy: I mean, I came into this year’s conference, A, only kind of knowing Steve and not knowing any of the other organizers and sort of not knowing what I was getting myself into, but they were very welcoming right away, and I definitely asked a ton of questions and sort of went in with hoping it would be that way, and then seeing how they all interacted and how welcoming they were and also getting to meet a lot of the alumni organizers who were part of conferences in the past and how all the positive things they had to say, it made me feel really comfortable with sort of how they all communicated with one another.
Bridget: [00:17:11] That’s a good point about how they onboarded a new organizer, because we have Dave Cliff sitting in the front row here, who is a DevOps Days Toronto organizer historically, and has taken time off at this point. Perhaps he will be a once-in-future organizer, who knows? But just that idea of passing the baton, I actually am going to put Roderick on the spot and ask, because you are a senior leader at Capital One, can you talk a little about onboarding new people to the process that you described in your team, or, you know, in your talk? Like, how do you get new people understanding, like, where they need to be in terms of how you want to practice and how you want to work?
Roderick: Yeah, so, I mean, for me and for us, it was more around communicating and being able to have an idea and spreading that idea around and realize— well, first recognizing that you have an issue or you have a situation that you need to kind of deal with. And for us, like, we were a smaller team and kind of Arthur came in and brought some of his experience from his previous employer. And we took that idea and said, hey, this is something we should try out. And so we started to experiment with it. We actually said, let’s timebox it for a month and see where this goes. And then as we as a team started to kind of follow this new model, discovered that it actually works. Like, there’s still— we still have some improvements to make. We’re still iterating and trying to improve it, but it actually works. And so, I mean, for me, the lesson learned is even though, you know, I’m in a senior position, there’s always opportunities to learn from other groups and other venues. And that’s one of the reasons why Capital One is sponsoring this event. We want to be part of the community and learn from one another because I think we all have similar challenges.
Arthur: [00:19:05] And actually, I want to take an opportunity to give a shout out to Piyush Chugh, who I worked with formerly, who actually created this idea of foreground/background when we worked together at a previous employer. And fortunately, we’re working together again at Capital One Canada. So I came in, I took the credit for coming up with the idea, but it was actually his idea to begin with.
Bridget: I mean, we all remix and share each other’s ideas. This is a collaborative culture. And that’s why I also, just having the experience of watching your talk, your talk, and you running around doing all this, I was like, these all have— these 3 things have something in common. I’m wondering, Aaron, if you can talk a little bit about socializing and onboarding people to the way that you were describing handling incidents, because I imagine that was also not perfectly seamless.
Aaron: Yeah, that’s— we’ve sort of developed that process and it’s— or process now, I guess, Toronto. But we developed that over time and it’s a lot about setting the tone as leadership. And it’s why I made such a big point about that. Being deliberate in how you function as those moving through the incident and saying, hey, I really need you to get rest and not letting someone work past their capacity when they say, no, I’m fine. Like, I appreciate that you think you’re fine right now, but I really need you to go home and get rest. It’s been a lot about making sure that clearly is established, that role of leadership, and that they have control over that incident. So, it’s this actually for— we were really fortunate on our team. The reason why I gave this talk was we had an incident and this sort of coalesced out of the culture that we’ve been building in our company. And I was like, wow, that went really well. Like, these were, these were, this all went really well and things worked out fantastically. We need to capture that and, you know, capture these points of what went well, what do people think went well, why do they think that happened. And, uh, it kind of built this talk. I think there’s a lot of talk about what we can do individually as contributors in incidents and how we should build tooling around that and automation. Um, but the, the company culture aspect of that incident response, I think, is big or more important.
Bridget: [00:21:17] Well, and I think you bring up a good point too, this idea of also talking about the things that go well. Like, we often in our culture of, you know, site reliability operations, DevOps, whatever you want to call it, we spend a lot of time focusing on the things that broke because I don’t know about you, but I guess I’m just kind of a professional paranoid. So, I spend a lot of time worrying about the things that went wrong and dissecting them in detail and trying to figure out every single thing that went wrong. Single hedge we can put in place against that ever happening again. And that is important, but celebrating the things that go well is also important. And I think this is something that we actually do in events, like thinking, because you gotta look at just what happened, what do you want happening again, what do you not want happening again? And I’m wondering if Amy, from your point of view, can kind of tell us like what your approach to that is.
Amy: I mean, I also spend a lot of time trying to prevent things from going wrong because in the day of an event, a number of things can go wrong. So it’s kind of— you kind of prepare for that in the lead-up. But during the day while things are going well, you actually get to see a lot of attendee feedback. And you get that feedback from people almost instantaneously because you can see, are people enjoying my event? Is it running well? Are we on time? Are people smiling? Are people talking? Did everyone eat their lunch on time? That kind of stuff, so you get a lot of instantaneous feedback.
Bridget: [00:22:39] And that’s all, folks. I know you’re thinking, shouldn’t an episode done during an open space session go a little bit longer? And, well, it would have, but we had some recording issues.
Aaron: And by recording issues, I mean the batteries in our little Tascam recorder ran out before about 10 minutes left to go in the episode. But we thought it was, you know, quality enough to put out even in an abbreviated form.
Bridget: Yeah, and thank you so much to our guests.
Arthur: This was—
Bridget: it was a great discussion. Thank you all so much for listening, and we’ll catch you next time.
