Devopsdays Chicago 2018

Matty: [00:00:00] This is all stuff that could be on the show.

Jeff: Yeah.

Matty: Okay.

Trevor: But I was waiting for you to, you know, finish taking over. It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Trevor Hess, and I have a great panel of guests here with me today. But first, a word from our darling sponsors. Your application sits on layers of dynamic infrastructure and supporting services. Datadog brings you visibility into every part of your infrastructure, plus APM for monitoring your application’s performance. Dashboarding, collaboration tools, and alerts let you develop your own workflow for observability and incident response. Datadog integrates seamlessly with all of your apps and systems, from Slack to Amazon Web Services, so you can get visibility in minutes. Go to arresteddevops.com/datadog to get started with Datadog and get a free t-shirt. With full observability, distributed tracing, and customizable visualizations, Datadog is loved and trusted by thousands of enterprises, including Salesforce, PagerDuty, and Zendesk. If you haven’t tried Datadog at your company or on your side project, go to arresteddevops.com/datadog to get a free t-shirt and support Arrested DevOps.

[00:01:31] Chef is a community of professionals practicing DevOps every day. We are making, proving, learning, and shaping the future. We are known for welcoming, encouraging, and liberating others to do the same. We do not talk about change, we do change. Join the community and learn about our solutions at chef.io. The worst time to learn about incident response is during an incident. Don’t wait for an outage to strike before getting started. The PagerDuty Incident Response Training Course is now open source and free for everyone at response pagerduty.com. Based on the same training that PagerDuty employees go through, this course will show you how to streamline your incident response process, turn chaos into calm, and demonstrate the role of an incident commander. So what are you waiting for? Go to response pagerduty.com today and check it out. Today I’m just going to ask the panel to introduce themselves. And so we’re out here at DevOps Days Chicago 2018 because I, I also am tired, Matt, so I can’t quite remember what year it is, but it is 2018, right? Okay, we’re good. I didn’t forget. Excellent.

[00:02:37] Cool. So let’s go around and introduce everybody. So let’s start with our first guest to my right.

Aaron: Hi there, I’m Aaron Kalin. I spoke earlier today on From Farm to Table and Code to Customer, so kind of drawing parallels between DevOps and the restaurant industry and kind of what DevOps can learn from it. But I’m also with DNSimple as well, so fun to be here. Finally, Matt’s been threatening this for so long. He’s like, hey, I need you to do an 80— and I think it was like 1 or 2 years ago, actually, at a DevOps Day Chicago. He’s like, hey, can you come down and do the recording? And then he never got me. So then I hear the episode later, and it’s like, oh, all right, I was supposed to be on that. All right, but here I am.

Trevor: Well, welcome.

Aaron: Yes, thank you.

Katie: I’m Katie Prizy. I gave my first Ignite talk today, or yesterday rather, which was super exciting. And it was on 5 rules for becoming a better DevOps leader.

Trevor: That’s fantastic.

Katie: Awesome.

Jeff: I am Jeff Smith with a company called Centro, and I gave an Ignite today on— what was my Ignite on? Making pages more— making on-call more humane and human. So just a sort of walkthrough on how to make pages suck less.

Matty: [00:03:46] So playing the role of a guest and not a host, so I’m usually one of your hosts.

Trevor: There’s a buzzer attached to Matt’s chair, so every time he takes over, it’s gonna electrocute him.

Matty: We’ve had guests take over before. We had Shafer on the show a whole bunch of times. There’s an episode I could point you to with the old Geeks Yell at Cloud. We might as well, me and Bridget might as well not even been on that show. It was Bryan Cantrill and Andrew Shafer. They were just going. I got really folksy there for a second. It was weird. So, but just background for this particular episode and why why this matters. So, I’m one of the organizers of DevOps Days Chicago. I was one of the founding organizers of DevOps Days Chicago. So, this is the 5th year we’ve done it, and I don’t even live in Chicago anymore, technically, and I still cannot bring myself to not be part of this event. And I think you’re right. I think we’ve done this at every— I don’t think we’ve missed a year of recording at DevOps Days Chicago.

Trevor: Not yet. It’s taken various forms, but I think we’ve gotten there. Every time.

Matty: [00:04:47] Last year’s took us quite a bit of time to actually release. I think we released it like about 6 months after the event. You know, it was a lot of post-production on it for, I don’t know why.

Trevor: It’s kind of, it’s the lost bits of Arrested DevOps. There’s a few missing episodes out there. So anyway, so as is tradition for me and DevOps Days Chicago, 100 other things happened and I could only attend like a quarter of the event. So I would love it if y’all could tell me about your experiences this week. Or these couple days, and some of what you learned, some of what you’ve taught the folks you spoke with this, or you spoke about to this week?

Katie: I learned so much these last couple of days, and it was really well-balanced with super, super technical stuff. I didn’t know what a service mesh was until yesterday, and now I feel like I could probably write a paper on it. That’s how great— it was in an open source. And that’s how great the knowledge was in that room. And the people in that room were really good at explaining it to some of the people in that room who did not know what it was. So that’s kind of one side of the spectrum. And then on the other side of the spectrum, we talked a ton about bias and ethics in our industry. And so And so we got really emotional and talked a lot about our feelings and people’s perspectives as they go through their lives and their jobs. And that was a really interesting juxtaposition for me to have so much learning in such totally different ways in 48 hours.

Trevor: [00:06:32] That’s super interesting. Could you share some of the things that folks talked about in that session?

Katie: The bias one?

Trevor: Yeah, the bias session.

Katie: Yes, for sure. So we started off with a video that most people have probably seen. If not, it’s easily Googleable. So they start people at a line, right? And they’re running a race. And so then the organizer says, okay, so everybody starts at the same place and they say, okay, take a step forward if your parents are still together. Take a step forward if you had a father figure growing up. Take a step forward if you’ve never had to worry about where your next meal is coming from. And so, by the end of that little thing, right, a ton of people are at the back. They were never able to take that step forward. And a ton of people are at the front, right, very, very close to the finish line. So, the point is, is that we all have to run this race. Everybody’s going to try to get to the finish line. But the people at the front are almost certainly going to get there first. And they would win a $100 bill. And so the point was, one, obviously, look at what’s around you. People in the front, turn around and see how far back people are. And it’s not because of anything that they did. It’s not because of decisions they’ve made. Due to things that happen to them. But so then take that $100 bill, whoever wins it, and do something with it, right? Use that to not just be aware of what’s going on, but to help the people in the back and to further their journey in life. And so that’s kind of how it started. And that took, I guess a little bit of a turn during the open space. What I thought was really interesting, typically when you have an open space about bias or something like that, it’s a ton of people who are mostly like, yes, bias exists. I’ve experienced bias. It’s a very self-selecting group. And so you don’t expect to hear many different opinions. What I thought was really interesting in this one is that there were so many different opinions. There were some, one person on one end of the spectrum who says, you know, I graduated from college in a class of, let’s say, 100, and only 3 people actually got jobs. It was during the recession, and those 3 people were women, right? So he was impacted because the companies he was interviewing for were hiring women and actively not hiring men. And then you have someone on the other side of the spectrum, right, woman who was born in Eastern Europe who came to the US a year ago. So an immigrant, a woman, and a person of color who is having a very difficult time navigating life and her job. And to have those 2 people in the room, I think, is very important. I think in that scenario, we tend to say to the guy, you don’t understand what’s going on, right? We need to listen to these people who are experiencing all of those. And I think most of the time that is correct, right? We need to listen to the voices that have not been able to be spoken aloud. But it’s also interesting that we have this other voice, right? And at the end of the day, we’re all humans and we’re all trying to get along and, and move forward together in, in life. And, and so how do we bring those 2 perspectives together to create harmony?

Jeff: [00:10:27] We had, we had a conversation about this at the table because, you know, uh, the speaker whose name is suddenly escaping me, um, Sonia, she really, she really set out a, you know, a challenge in her talk, you know, uh, directly challenging white supremacy. And, and that, uh, definitely generated a reaction for people at my table. And we got into a, you know, a very intense discussion about it. And I think the common thing is that people have a problem accepting a narrative that runs counter to their experience, regardless of, you know, the numbers of people, right? It always boils down to, well, my experience says that this isn’t true. But that doesn’t necessarily mean in whole it’s untrue. But, you know, it’s difficult to sort of move the conversation forward because you have these people that are exceptions or outliers, and you can’t really deny their truth. But at the same time, we have to accept that, well, you know, it sucks that, you know, you’re not fault— you’re not part of this, you know, privileged group, but it doesn’t necessarily mean that privilege doesn’t exist and it’s not impacting people in negative ways. And it doesn’t necessarily rob you of your ability to use the privilege you do have to, you know, further the conversation. So, you know, I, I know there was definitely a strong reaction during that talk, but I think it was positive because, uh, one, I would suspect if it wasn’t challenged quite that way, that open space wouldn’t have been nearly as full with diverse opinions, right? Uh, usually when I see an open space like that, it’s a lot of people that agree with each other, and we’re all virtual virtue signaling and high-fiving each other. Um, so it was interesting to see that there were so many people that showed up. Like, I tried to get to it and it was full. I couldn’t get in. Um, so it was— it’s great to hear that there were, like, differing opinions and, and people were comfortable voicing their perspectives.

Trevor: [00:12:14] Absolutely. And it sounds like it’s come a long way. I mean, I remember the first conversation I was a part of at an open space kind of along those lines was at ChefConf in 2015. And even back then, the majority of the circle was white dudes. And to hear that there were these different opinions, these different backgrounds coming together and sharing those stories and Kind of opening each other’s eyes to the other perspectives is really powerful.

Jeff: And going back to Sonja’s talk, even backing up from that portion of the conversation, I thought the ethics part of the conversation was really interesting too. And it started this sort of thought like, is software development a profession, right? Because that’s really what it boils down to when we’re talking about ethics and responsibility and the scope of responsibility and accountability. Is that something that we should be entertaining as an industry where, you know, software becomes a profession and we have a much more enforced, rigorous body around it? I’m not advocating for that per se, but I think it’s something that we should consider because as these systems become more and more integrated into safety-type systems and environments, you know, we’re talking about real lives. I mean, we’ve got Teslas that are parking themselves, probably running outdated OpenVPN software and, you know, they’re susceptible to being SSH’d into for the infotainment platform and all that stuff. So it’s like, at what point do we, do we force the people that are making these changes to take accountability and responsibility regardless of what the organization they work for is? Because that was the other point that always came up was, you know, we’ve got ethics from a development perspective, business goals sort of don’t align. Is it my responsibility as the person contributing to this code to take an ethical stand for the organization? And without some sort of governing body, right, with a lawyer, that’s cut and dry, right? It doesn’t matter if you are violating some ethical responsibility, regardless of the firm that you work for, you’re still held accountable. I don’t know that that exists in today’s software world, and do we need something that, you know, sort of enforces that?

Katie: [00:14:22] Right. I think that conversation is super interesting these days. I went to the DevOps Enterprise Summit last year in San Francisco, and there was a whole panel of a pilot, a doctor, and then Gene Kim, probably. And they were talking about ethics and maybe taking it even a step further of, are engineers responsible for how their software runs in production, right? If a self-driving car, if that software that runs that self-driving car is buggy and it causes someone to— it could, well, it could cause someone to die, right? Cause that car to crash. Who is responsible for that software? And the other example that was given with the self-driving plane, right? Same idea. And so I think the level of ethics could really be inserted at every single point. We talked a lot about the ethics in giving away user information, right? And Sonia talked a little bit about that, which is kind of on one side, but then the ethics of what does your software do in production? How does it impact users? And are you responsible if something terrible happens? And that’s Sooner or later, there has to be a decision made on that, or else we’re really just a bunch of cowboys pressing buttons and then saying, well, I didn’t do it, not my fault.

Trevor: [00:16:00] Absolutely. And then what do you look to as a kind of a governing body for that sort of ethics, right? Because even if you look—

Matty: just one of the— I just want to point out that was one of the large points that Sonia made in her talk, which is there is a consortium that has published some of this stuff. And she said, you go and you look at the makeup of this consortium, and they are, I believe, all men, one person of color. And she’s like, how do you— you know, again, this is not a diverse group. This is, this is one set of experiences. Absolutely. And so when you, when you look at this, how can you talk about enforcing ethics? How can you talk about doing this when it is something that is not going to necessarily be applied equally or be understood equally or in that perspective.

Jeff: And the messaging around that, I think, is incredibly important because I think one of the things that someone at our table took issue with was the idea or notion that because they were white men, they were incapable of being ethical. And that’s not really what’s being conveyed at all. It’s that there’s a spectrum there, and there’s no way that they can represent all of those possible perspectives, right? So how we actually message these things because we’re dealing with such an emotionally charged issue, I think we have to make sure that, you know, our messaging is on point so that we’re, you know, crystal clear on what we’re conveying.

Trevor: [00:17:20] Did any of the conversation come back to who uses our software?

Aaron: Well, there’s the example, so in talking about the whole incident with Microsoft and ICE, like ICE using the whole Microsoft platform to effectively deliver There’s software that allowed ICE agents to locate, track down all that stuff. That gets into a very big morality conundrum because that was a very big thing that just went everywhere. I didn’t hear too many murmurings about that in particular, but that was a very pointed example of this is what happens when— imagine if you’re an engineer working at Microsoft and you find out the software you’re helping write or manage or maintain is maybe helping some doctors reach folks, but at the same time it’s also enabling ICE agents to do horrible things. So like—

Matty: And without having some kind of governance that has backing to be able to help you when you have to make that ethical stand, ’cause this is a thing that has come up recently with people saying, well, if you work for Microsoft, you should quit your job because of this happening. Being able to say, I am going to just quit my job on this ethical ground as an engineer, as a technical engineer is actually a place of privilege. Not everyone can do that because you’re all alone when you do that, right? Like, that’s again, to a point, you know, and Tony kind of talked through that in different professions, in the legal profession, the medical profession, these are things that are understood, right? If you violate your ethics as a lawyer, you will be disbarred, right? It doesn’t matter who you work for.

Trevor: [00:18:54] The Hippocratic oath.

Matty: It’s so you don’t— not that it necessarily makes it easy to be ethical, but it’s more of a, as a lawyer, you can do the right thing because you’re not left out in the cold, because you’re not going to be quitting your job, because the firm understands they cannot do that, right? They cannot ask you to be unethical because by doing so, you will be disbarred and you will no longer be able to work there, and whatever other reasons. I’m sure there’s much more to it than that, but that’s a— that those are luxuries and privileges that are not afforded in our industry because Yes, it’s, it’s very easy for, you know, someone, a privileged individual living in the Bay or whatever to be able to say, well, that’s fine, I’m gonna just walk away from my job. But being able to do that again is something that not everyone can do.

Jeff: And I think we also have the problem of that there’s so much of our industry that is, for all intents and purposes, grounded in unethical behavior when it comes to user data, right? So if we start putting in these ethical practices and guidelines, what happens to— like, if we were to explain to a common person how advertising works, how data collection from Facebook works, if you explain that to an average person, they’re going to say, wow, this is insane. This is completely unethical. So we’re talking about wiping out Google. We’re talking about wiping out Facebook. And how do we have a serious ethical conversation without also balancing the reality that those consequences could exist?

Aaron: [00:20:18] Yeah, or that is, the governing body is sort of created for you with things like GDPR, ’cause there was a whole Open Space coming on that, and that was a very important thing to me, at least in the DNS and domain world. That caught a lot of registries and registrars by surprise, because the general way you exchange information across registries and stuff when you’re transferring domains is you need to have the WHOIS information. Even if you pay to protect it, that still has to get transferred, and under GDPR, you’re conveying private information. So already you’ve hit that brick wall. So you need to now do a separate thing to the domain owner of like, by the way, are you allowing VeriSign to transfer your information to this other party? So before that was never really there, it was just implicit. So now something like that is forcing companies to look that way. But then there’s also countries that, or even certain top-level domains that are like, oh, we don’t care about GDPR. They’ll either publish your information or they’ll still allow WHOIS privacy and all that stuff too. And we’ll see if GDPR reaches over to them in some way, shape, or form.

Jeff: [00:21:22] And there’s a lot of practical considerations with GDPR too, because while its goal is noble, technologically it’s sort of a nightmare, right? And as you learn more about it, you realize just more and more how much you’re not capable of it. So in the open space, we were talking about people making these requests. Really the tactic is sort of delay. They say, hey, give me all of my information that you have on me. Oh, well, I don’t know that you are who you say you are, so what I need you to do is I need you to go to the American consulate, right? And I need you to prove that you’re a citizen of the EU. I need you to provide, you know, a passport. So if you don’t leave the country, I guess you’re screwed, right? So, you know, we sort of put up all these hurdles. So it’s one of those things where when we’re designing this We can’t just design a rule for privacy, we have to design a system of privacy so that all of those components are sort of taken into account. Because in reality, what we’re gonna end up with is a bunch of websites that ask us to click okay, we’re not gonna read it, and we’re gonna have like a EULA 2.0.

Matty: ’Cause again, when you know when making the right thing the hard thing, then everybody will clearly do the right thing. I mean, no, you’re gonna end up half-assing your way through it in some way to be able to get the letter of the law and implement it very, very poorly without—

Aaron: [00:22:34] Yeah, it’s very much like the honor system, but there’s definitely already been some bad actors and dishonorable things that are done.

Trevor: Totally.

Aaron: So I don’t know at what point it will trigger enough response from this community as a whole to where something like that gets created. But generally, like, at least from past examples of that stuff, reactionary things like that tend to create the wrong things or something much worse. So doing it more voluntarily might be a better idea before something becomes really, really bad. Like when companies like Target lost millions and millions and millions of dollars worth of credit card data and all that stuff, it was a huge wake-up call for everyone of like, oh wow, just mishandling one thing and you lost all of this in one shot. And while they did react really well to it, and part of that probably was because of things like PCI, or banks are coming in like, hey, you lost a lot of financial data. And we’ve had data breaches, and now there’s sites like Have I Been Pwned, which very publicly publishes your information going out there on the internet. And there’s nothing really regulating that other than GDPR, but that’s only covering EU citizens. There’s nothing for citizens of the world necessarily.

Jeff: [00:23:43] And things like the Target breach, it makes you wonder, like, who— I fear a world where we’ve sort of given a pass on a lot of companies because there is no real public outcry, right? Like, this was one of the largest data breaches in history, and, you know, people are still shopping at Target. I still shop at Target, right? I’m still using my credit card at Target.

Trevor: It’s very simple.

Matty: It’s a concept called normalization of deviance, right? When we sit there and we just shrug and go, oh, I guess Equifax lost all my shit again.

Katie: Whoopsies.

Jeff: And the Equifax problem, though, was particularly perverse because we’re not really Equifax customers, right? So I really don’t have any mechanism to punish Equifax anyways, other than when I get my credit report, I go, no, not you. So that’s an extra $2.

Trevor: That’s not true either entirely. We do have— we do control our politicians to some degree, who ultimately are the ones who let Equifax off the hook.

Jeff: Unfortunately, most of us are one-issue voters, right? So it’s like, oh yeah, my privacy’s gone, but as long as you’re pro-life, I’m good.

Trevor: [00:24:45] All right, well, Aaron, I just wanna ask you if you had anything that you wanted to share from your experience quickly.

Aaron: Yeah, so that’s like on a—

Trevor: ’Cause that was a great conversation. Oh, it was, yeah.

Aaron: Yeah, there’s a really good one about remote workers, because being a remote worker myself, everyone at DNSimple is completely remote. My last gig before that was also remote too, so I’m very, very used to that whole setup. Hearing some different companies where they were trying to go full remote, but they figured out a way to meet halfway. Some companies are doing like, all right, initially they started off with, so you get one day that’s remote. And then it turned into, I think, at one company, it was like 3 days remote. And it was a super beneficial thing because a lot of them had families, so they can spend more time with their kids. Some of them had, they said it was like an hour and a half plus commute, so they’re not spending all that time dealing with gridlock traffic and all that stuff, and they get to spend more time at home with their family, wife and kids, whatever. And some more interesting conversations around, Once you now have that remote space, now you’re not in front of your coworkers. When I spoke earlier, it’s like, make sure you communicate with each other. And there’s one person who’s talking about how they basically forced someone to communicate with the rest of the company by initially— part of it was, because I came into it just as he was starting to explain it, was that they were trying to do a thing where they were using software to keep their webcam going so they would have a sort of virtual glass of like you see all of your coworkers, like their faces all the time. So while they’re working, they’ve got a camera going, and this one person refused to do it. And that’s fine. To me, it’d be like, all right, well, if you don’t want to be on camera, that’s cool. But at some point, they forced this person to be like, no, we’re turning on your camera. You’re going to be part of this conversation. They said, well, it turned out well for that person. I’m thinking like, that would be terrifying if my job told me like I need to have my camera on all the time. There’s sometimes you don’t want to see me on camera. Especially like if you make me roll out of bed at like 2 or 3 in the morning on a page or something, you do not want me on camera. Like, I’ll be on audio, we’ll talk, it’s cool. You don’t want to see me right now.

Jeff: [00:26:51] Yeah, there’s all types of subconscious stuff that you do when you’re not thinking anyone’s paying attention. Totally.

Aaron: Camera’s on, you’re like, my hair looks good now. You don’t want to see me at 3 in the morning. No, no, no, no, no.

Trevor: For those who can’t see what just happened, Jeff is picking his teeth.

Matty: There’s reasons that this episode is Not video.

Trevor: Very much so. But I just want to ask everybody one last question. Did you enjoy your DevOps Days Chicago 2018?

Katie: Yeah, absolutely.

Trevor: I was going to ask you next. So, let’s ask again. Did everybody out here enjoy their DevOps Days Chicago 2018? Awesome.

Aaron: That’s a much better response.

Trevor: Yeah, definitely. Well, this has been Arrested DevOps. Thank you all for joining the panel today.

Katie: Thank you.

Trevor: And until next time, there’s always DevOps in the banana stand.