devopsdays Philadelphia 2019

Tim: [00:00:00] Oh, weird, it turns out that that algorithm is completely racist, right?

Bridget: It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Bridget Kromhout. Today, we’re live at DevOps Days Philly. Applause sign, make some noise. And we’re going to talk to a great panel in a minute here, but first, a word from our sponsors. Arrested DevOps is brought to you by PagerDuty. In an always-on world, teams trust PagerDuty to help them deliver a perfect digital experience to their customers every time. With PagerDuty, teams spend less time reacting to incidents and more time building for the future. From digital disruptors to Fortune 500 companies, over 12,000 businesses rely on PagerDuty to identify issues and opportunities in real time and bring together the right people to fix problems faster and prevent them from happening again. PagerDuty, solutions before problems.

[00:01:15] The worst thing about the Arrested DevOps podcast is when it ends. You’re left wondering what to do next. What are you going to listen to on your commute home? How do you occupy your time when walking the dog? What are you going to listen to during the quarterly all-hands meeting? But fear not, dear listener, there is a solution. You need to subscribe to Software Defined Talk right now. It’s a weekly podcast that recaps all the news in cloud computing, DevOps, and enterprise software. The hosts, Cote, Matt Ray, and Brandon Wichard, will keep you up to date on all things cloud while offering tips on how to optimize your Costco haul, and how to PowerPoint. It’s a fun, free-flowing conversation that will keep you entertained and informed. What are you waiting for? Subscribe to the podcast today by visiting softwaredefinedtalk.com or by searching for Software Defined Talk in your favorite podcast app. DevOps Days Philly. We have a great panel. I’m going to ask them to introduce themselves, starting to my right.

Tim: [00:02:17] My name is Tim Gross. I’m a software engineer with HashiCorp working on the Nomad project.

Jocelyn: Hi, I’m Jocelyn Harper. I am a senior associate software engineer at Capital One.

Peter: Hey, everyone. My name is Peter Shannon. I’m a senior software engineer at Instacart and also the head organizer of DevOps Days Philadelphia.

Bridget: Okay. Activate applause, turn down. Thank you. All right. Fantastic. So, I wanna start with you, Peter, and you are the head organizer of DevOpsDays Philly. You brought this program to us. Can you talk a little bit? Because I wanna dive into what some of our current and past speakers have talked about, but can you talk a little bit about the curation process? Like, how do you decide that not every single talk should be Kubernetes?

Peter: It’s a good question, because really they should all be about Kubernetes, right? So, it’s a single-track conference, right? So, that’s one constraint. And so, with that, we have to try and find a balance of culture talks, technical talks, talks that we think might be culturally relevant or timely, and fit that all into the schedule. And it can be challenging because This is our 4th year doing this, and I’ve heard from folks, you know, I feel like the talks are too cultural. I’d really like it if, you know, you had more tech talks. I’ve heard the inverse of that. And so what we try to do is we try and take all those components and we try and mix them in as best we can to have, you know, a culture talk, to have a technical talk, to have talks that are timely. And I think for me in particular, I always try and have the opening talks be something that does tend to lean more towards cultural, and in particular something that I think is culturally relevant or timely. And from there, we might, if we can, try and frame other talks throughout the conference around that to sort of kick it off as a theme, but that can be challenging as well.

Bridget: [00:04:34] Yeah, absolutely. So, Jocelyn gave a wonderful opening keynote here, and I’m wondering, Jocelyn, if you can kind of summarize for our podcast audience and for the people who may have not been quite caffeinated enough yesterday morning, what are some of the high points about your talk?

Jocelyn: So, my talk was essentially about ethics within technology and that responsibility that we as technologists need to understand that we have. I highlighted primarily that we as technologists, we can seem to be indifferent to things that are happening in the world that larger tech companies that we either use in some way or that we work for, we just tend to ignore the issues that they present to other human beings that are perhaps not technologists. So I found that it was really interesting and apt that I, would talk to y’all about that very early in the morning and make you understand and have that introspection to see that where your ethics lie and how to apply that to situations that have happened just this year, and the year isn’t even over yet. And it was a lot, a lot of them.

Bridget: [00:05:47] It is amazing when we look at kind of year in review things, and we’re talking here in mid-October, but close to the end of October. It’s all very upsetting. You know, we’re close to the end of a decade too, and it’s just amazing to think about. But when we’re having these discussions, when we’re considering this stuff, I think sometimes it’s useful to realize that these aren’t even new discussions and they’re not unique to us.

Jocelyn: No, not at all. They’re not new in any sort of the way. So it’s very interesting, and I said this in my talk as well, how social media has been a vehicle for people that may have not been heard in companies. And that’s not to say that people within companies haven’t been bringing these issues to the forefront, but it’s a lot easier for people in higher places to push them aside when nobody else knows about the talks that are happening. And so I specifically mentioned Twitter because I’m very active there, that Twitter is definitely the vehicle for the general public and more technologists to know about these things that are going on. And it’s been really helpful in inhibiting change there, so.

Bridget: [00:06:53] Yeah, absolutely. And several, like in past years, Tim, you’ve keynoted here at DevOpsDays Philly and made an argument about software defining culture. I’m wondering if you want to give us, take a stab at what have you seen in that space?

Peter: Sure.

Tim: So, the talk was kind of like turning, I think John referred to Conway’s Law earlier today, And we were talking about where the organizational structure defines the systems that you build. And the idea around the talk was, like, well, the systems that we build and the choices that we make around those systems can also have greater impacts on both our organization’s culture and the culture that it’s within a context of. And some of those are about building for reliability because it makes sure people trust each other better, but it ripples all the way down to which technical communities do you decide to be part of? Are you part of a language community that has a lot of toxic behavior? And kind of like, and if I may, I kind of continued that theme in another talk I gave at DevOps Days Minneapolis last year about kind of like the much larger cultural impact that we have, where, you know, are we building technologies that like don’t have good purpose, or can we build technologies that like maybe they have a neutral purpose but can be used in bad ways, and can we make technical decisions that like drive those things to a better endpoint than kind of, like, the open-ended evils that can be done with them.

Bridget: [00:08:26] And so, if that sounds too abstract, I wanna kind of give one of the specific examples you talked about. As technologists, we’re making decisions every day about, you know, state, like that thing where the customer data lives. So, any production system probably has it. And how do you deal with state and versioning? And maybe you have to think about GDPR, maybe you just have to think about what are we storing that somebody could use for evil? Like, a lot of times, I feel like as technologists, we sometimes look at the happy path, and we’re like, in ideal circumstances, this will be used exactly this way and never abused. Well, how realistic is that? I mean, it’s like, what have you seen, Pete?

Peter: Yeah, I think as technologists, we tend to look at things more from, at least in my experience, is more from like almost like a variable perspective. So, you know, we need our service to do X and these are the inputs, right? So, and those inputs could end up being extremely sensitive data. Maybe some of that data could be used in ways that you’re not even thinking of because you’re thinking, how do I get this service to work, right? And so, it definitely starts to beg the question about, is there something we can do more? Is there some type of training? Is there some, you know, a different way of thinking that when we look at this data, we should assume to have a responsibility and say, you know, this data is really critical and the people who own this data, like the people who the data is of, you know, they would be potentially, you know, harmed or, you know, could end up being financially ruined or who knows what it is if the data were to get into the wrong hands and start thinking about it more like that instead of just a bunch of bits stored in our database. But it’s tough.

Bridget: [00:10:23] Yeah, well, and if we’re thinking about threat models, I feel like this is where as technologists who might look around our organizations and see a homogeneity of people who have gone to similar educational institutions and are from similar backgrounds and, you know, don’t have the same threat models. And that means we are going to build better products if we make sure that we have people in the entire product chain who maybe are people of color, maybe women of color, maybe have actually experienced some of the threats that the product designers who weren’t listening to them might not have even ever seen.

Jocelyn: No, I think that’s very true. And I also think that as far as just data overall is such a large discussion right now. It’s in the general populace, not just with technologists. And I think that we need to start analyzing our systems and the way that we design things a bit better. I think we also need to start asking ourselves the question, do we really need this data in order to perform the function that we have? And I think that we have just gotten to the point where as technologists, we just give all of the data just because it comes with it, and that’s fine, and we’ll just put in the database, and we have security, and it’s fine. And it’s just not like that. We need to start having more thoughtful discussions about when it comes to the architecture of these services, do we need all of this data in order to perform the function that we have? And that is something that we all need to do moving forward. Yeah.

Tim: [00:11:55] I can think of a specific example at an organization I worked with a couple of jobs ago where It was a sensor device that was designed to count people and detect movement through a space. The idea behind it was that it was designed around giving large facility owners the ability to make decisions about how their space was being used. But the designers recognized early on what abuses something like that kind of data could be used for if it was used to identify individuals. You see there’s a bunch— there was an article recently that came out about this horrible badge that you can give people and track all their movements individually, like how much time are they spending in the bathroom? Um, and this device was designed kind of like upfront to make it impossible for us to get that data from, you know, into the data about individuals from the thing. And that was what was interesting about that. What they found was that it turned out to be a competitive advantage because it was something that none of the other providers in that space could do. Um, everybody else wanted to be able to identify individuals, and there were many companies who said, you know, many organizations who said, you know, we don’t want to be able to do that because our workers aren’t going to like that.

Bridget: [00:12:57] So Though, didn’t you tell me about a funny corner case?

Tim: Yeah, because it detected the shape of human beings, if you moved mannequins through a doorway, it would detect that as a person. And so it was very confusing.

Peter: Like people carrying mannequins?

Tim: Yes, people carrying mannequins.

Bridget: Like in retail?

Jocelyn: Yes.

Tim: Yeah, it was kind of a weird corner case.

Peter: Yeah, it’s definitely interesting, though. And you hadn’t told me that story before. And I think that’s really neat that the customers sort of policed themselves and said, like, that’s a really neat feature that some of your competitors are offering, but you know what, we don’t even want to go into that right now. We don’t want to know who’s doing what. We just want to know if somebody’s in this space, right? Um, and it’s interesting that you— that the company actually came at that from, from a product decision, but kind of what you’re saying is, is we’re storing everything in databases, and I, I think I think that is the general consensus for most companies. It’s that I remember Etsy years ago, and this was more towards storing metrics. They said if it moved, graft it. And I think when it comes to storing customer data, it’s if there’s a data point, if there’s something to get, you grab it because who knows, maybe we’ll need it. Storing bits is cheap.

Bridget: [00:14:14] I mean, it’s cheap until the lawsuit, right? Yeah. Because like several jobs back, Well, let’s put it this way. When the Cambridge Analytica stuff happened, I wasn’t surprised because I too used to be evil. And the idea of harvesting people’s, like, say, you used to be able to get people’s friends’ likes from Facebook and then target things to them based on that. They turned that off in 2015, but there’s still a lot of dark patterns out there. And I think that So, if you are in a space where, whether it’s ad tech or something else, where it seems relevant to your business interests to grab all of that data, I would say, but think about where you’re going to have someone upset about how you’re using, how you’re storing it, what you’re leaking, when there is a data breach, because it seems like every other day there’s a new data breach. I mean, I just kind of assume that Shrug if something happens with my credit card or my data. Like, at this point, it just seems like it’s being breached constantly. But if you’re the person making decisions about what to store, I forget who it was who said, think of your— think of the PII and the customer data as being kind of like toxic waste. It’s like, this isn’t necessarily an asset. It’s something you have to think about how you’re gonna carefully corral and store it.

Peter: [00:15:38] And what I find interesting, I’ve only been at 2 organizations that had HIPAA data. And in both cases, it was amazing at how much care went into that data, right? So, like, you know, legal would have their say. We had to be extremely careful with who had access. You had to go through training. There was all sorts of hurdles that you had to go through to even be able to work on the systems that had access to HIPAA data. But then for some reason with almost everything else, it’s kind of like, eh. And then, you know, kind of back to your point with the IoT device, it’s interesting how maybe sometimes you’re capturing a whole bunch of data points that seem meaningless, but they can be combined in a way where they can be extremely harmful. And you might not even realize it until years later. You might not even realize it until you’re breached and then the data is combined by somebody else.

Tim: I think there’s been some examples of where that’s been, you know, the— we talk about this kind of like, there was this discussion yesterday about MLOps, right? And this idea that we’re having, or was it AIOps, right? And it’s this idea that we’re applying machine learning to, you know, to data metrics that we’re getting off our devices now, but we’re applying that to all kinds of data that we collect. And what I find kind of, frightening about that is the data is coming in as kind of this like sea of, you know, people like use the term data lake, which is very ridiculous. But it kind of like, you know, reveals the problem, right? Which is that it’s just kind of like a big blob of data that we’re operating on. And then the models that are applied to it, you know, they’re not actually magic, right? They’re applied using, you know, the preconditions that the person who’s developing those are setting. And so there’s this weird thing that people say, oh, well, it’s the algorithm deciding this, but we’re really encoding like, all the biases of the people who are doing that work into those models. So you see things like machine learning models being used as part of parole processes to say, oh, is this person likely to be— is their recidivism rates likely to be— are they likely to reoffend? And then, oh, weird, it turns out that that algorithm is completely racist.

Peter: [00:17:53] And they blame the algorithm, not the people who wrote it.

Tim: Right, right, right. Well, it’s just an algorithm. Just math, right? Yeah.

Jocelyn: Yeah, I just find data collection very problematic. And that’s probably not something, as a software engineer, I should say. I think it’s beyond just collecting and storing the data. It’s more so also, why are we collecting that data? And for what purpose are we collecting that data? Because it goes back to, is it good or is it bad? And for me, just collecting data just to have it and to store it for something that may be relevant to you in the future, isn’t a good enough reason to collect it and to store it. So that also comes into the point where I think that we as individuals need to start having more insight and more control into the data that these companies are collecting because it’s gotten to the point where it’s ridiculous. I know you mentioned earlier about breaches. For me at this point, from all of the breaches that have happened over like the past 5-ish years, I just assume if something hits my credit card, I already have my credit reports frozen from something like 3 years ago. But it’s just the fact is the matter is that there’s no real repercussions for companies or anything that happens that way. So I think we just need to start taking more control from a consumer standpoint.

Tim: [00:19:11] You were talking a little bit about our responsibility as technologists as part of that process and what decisions we’re making. And I think that extends also to the general, what kind of software are we working on? I think there’s been a really interesting discussion right now about— and I admit I’m changing the subject a little bit here. There’s been a really interesting discussion going on about to what uses is our software being applied in the general world. And there are some technologies that are inherently neutral. So Kubernetes is effectively neutral. It’s not an inherently nefarious— I mean, it’s kind of nefarious. But I work on Nomad. I have to say that. Anyway, but so it’s a neutral product in and of itself. Now, if we as technologists really believe the story that we say that Oh, what we do really helps accelerate— you were saying this yesterday— accelerate and innovate. If the technology we do really actually helps the organizations we’re doing, then all technologies can be turned to some bad end. But there is a question of there are certain technologies though that have an inherently bad end where you say, okay, well, machine learning is pretty neutral, but are there actually any really good uses for facial recognition? If you’re working on that, Are there any non-nefarious uses for that? And I kind of question if you’re a technologist and you’re saying, you know, it’s kind of like making bombs. It’s like, well, you know, like, at the end of the day, like, you know, maybe the right people get bombed, which is kind of like, but like, somebody gets bombed and that’s kind of like, yeah.

Jocelyn: [00:20:50] And, you know, it’s just people, they, I mean, a lot of people, they may not feel as passionately about this as a lot of other software engineers. And for them, it’s just a job and they go there to code and then they go home and they don’t think about the impact outside of that. And there’s a lot of people there and hopefully from this and from my talk, people will understand that, like, no, it’s your responsibility to think about that. Just because you’re not on the executive board or a CEO doesn’t mean that you’re not allowed to have an opinion about what you’re working on is being used for.

Bridget: And I think that we’ve seen a lot of examples of, in the last year or so, of people working at various companies having opinions you know, collectively or individually speaking out. And I think that sometimes we can look at that and think, well, that seems dramatic and risky or whatever. And maybe you don’t have to make a dramatic splash. Maybe the best thing to do is just to think about this data collection, think about your threat models. Like, think about— sure, think of it this way. If you are somebody who is thinking about software operability, then you’re also thinking about the ways that it can break. And that doesn’t just extend to downtime and getting paged, that could extend to data breaches, or your customer’s privacy being invaded, or unintentional side effects that maybe you need to model when you’re building your systems. Like, our deployments work this way, everything goes into this available bucket, and then we have no protections on it. I recently got an email from a provider that I had used that was like, and then everything is in this S3 bucket that’s open to the world. We’re not gonna do that anymore. And I’m like, why did you?

Jocelyn: [00:22:29] Okay, yeah.

Bridget: I mean, I know why, right? Because ship it was the answer.

Peter: I mean, it’s really challenging, right? Because all these companies wanna move fast, right? They wanna deliver. And so, in order to, I mean, if we’re talking specifically about, security related to, um, like the jump from security to ethics to say like, well, if, if, you know, this, this data is taken, um, you know, who works somewhere where everything is secure, right? And who works somewhere where security is really prioritized as heavily as it should be, or where, you know, they have a team that actually works, you know, cooperatively in conjunction, and everything just flows naturally with other teams to make it work because there’s so many different facets and layers to security. Um, and I think ultimately it comes down to probably money, to be honest. I mean, what is the financial impact if you get breached? If it’s, if it’s, if it’s, if it’s company-ending, then there’s a good chance that that area has been bolted down. But if it’s okay, you know, we have to pay some money here, we have some PR issues we have to deal with, then maybe the cost of fixing things is more than the cost of dealing with the damage. And that’s something that I think has to change.

Tim: [00:23:58] I think that, I mean, some of that’s kind of like a bigger social question than what we as individual technologists can do, though I think we can advocate for it, which is that, like, Companies have been, much like pollution, companies have been allowed to externalize those costs, right? To say, well, the society will deal with that cost, not us. It’s like putting out smog, right? They don’t get charged for it. We probably need to change that on a bigger social level, but I think that if we as technologists operating within our orgs or deciding which orgs to work for can have influence over that and say, okay, We should maybe not— because eventually, if things get to a certain point, at a certain point, the idiots in Congress will decide that they’re going to be the ones to decide how technology should work because they’re sick of— because the American folks, we’re here in America, are tired of data breaches. And they’ll act. They’ll do something. This is something. Let’s do it. Nobody thinks that that’s probably going to be a really good technical solution because they’re not particularly good at that kind of thing. So it’s probably something that we should fix ourselves before it comes to that point.

Jocelyn: [00:25:14] I’m glad that you brought up Congress because I do think that at some point, as far as making sure that all data is handled appropriately and equally, I do think at some point it’s going to take some federal law to make these companies realize— because you mentioned the money. And absolutely, I mean, when businesses go to the books, it’s like, well, is it going to cost X or cost Y? I’m going to go with X. So, well, let’s make that penalty be damaging to the company because it should be. You should be thinking about security. You should be thinking about how you’re using that data. And I’m not sure if it’s going to happen within this decade or perhaps even when I’m alive, but it’s going to.

Bridget: I mean, there are actual penalties with teeth in GDPR. So maybe one of the initial conversations is, Maybe we should be looking at that for the US because the solution that some US-based companies have, if you go to Europe and you try to go to some websites and they’re like, we just don’t have this website in Europe anymore because we don’t want to fix our systems. And you’re like, excellent, I don’t want to use your site.

Tim: [00:26:16] Well, but that’s, I mean, and I know Pete and I ran into this when I was working with him that California has now enacted a very similar law and it does have some pretty nasty teeth. Most US companies are not going to cut off California in the same way.

Peter: Um, so I think, yeah, and it’s, it’s to have that follow-through with the government to actually say, no, we are going to fine you, right? Because if it’s, if it’s a, if it’s a toothless, if it’s a toothless law, if it’s a toothless policy, it’s not going to work. The companies have to feel the pressure in order to do it. It has to be, you know, it has to be a multiple, right? It has to be like it’s X numbers of dollars for every infraction, not just like a one-time fee, right?

Bridget: Yeah, and I feel like this is a conversation that we probably, maybe, perhaps people will continue in open space because we are short on time here. So, I would love to have our panel just kind of give their summary. What strikes you as something that individual technologists can do, should do, you know, might already be doing that we should encourage in order to have this data protection and privacy, security and accountability be something that we carry forward?

Tim: [00:27:34] Sure. I mean, I think the first thing for me is that we should all remember that relative to a lot of folks, we have a lot of privilege. Many of us do very well for ourselves. We have a lot of— I mean, if I were to ask who here is hiring, everybody would raise their hand. Which means that we have a lot of power. That gives us as technologists a lot of power. And not just the act of writing code, but in our ability to decide who we’re going to work with, what projects we’re going to work with, who our customers are going to be. And I think it’s become time for us to start exercising that power. And we should probably do it together. I think I’ll stop there.

Jocelyn: Sure. I think at first, it will start with how you interact with people at work. Because perhaps maybe some of you don’t want to use your privilege do things like be on a podcast or go give talks about it. But I mean, be that squeaky wheel at work. When you see something that you don’t agree with, keep speaking up. And maybe something may not happen with that because that happens to me, it happens to everybody. But as long as you keep vocalizing that at every turn, at every point that’s brought up, you can feel better with yourself because you’re voicing your concerns about it. However, if you are somebody that wants to exercise that privilege that you have outside of work, I would definitely say start with Twitter. That’s where I started. And now I’m on podcasts and giving talks because you, you do have power within your voice and Twitter does work. I know a lot of people are like, Twitter activism doesn’t work. And we’ve seen— I’ve seen very real just in the past 48 hours, actually, that Twitter activism does work and it does have real consequences. So I would suggest you start there.

Peter: [00:29:20] Yeah, I think for me, I’d probably echo very similar to what both of you have said. Um, because, you know, I’ve been thinking about this a lot recently too. It seems that every time something happens with like a product launch, right, and this seems to happen a lot in the gaming community and things like this, everyone gets up in arms and they say, vote with your wallet, vote with your wallet, right? Like, don’t, you know, boycott this product, don’t give money to that. And, you know, a lot of these really, really big tech companies have been in the news recently, um, and folks are upset about it, but people still go to work there every day. And I think we can make choices about where we want to work. And, you know, sort of back to your talk, Jocelyn, I know it can be extremely difficult for some folks to, to be able to make those choices, right? To have the option to say like, oh, I’m going to leave this, this job here out of principles, right? But I think if, if as a community, if we banded together and showed some of these companies that, you know what, we don’t think it’s okay and we’re just not going to work there anymore. We’ll find another company that has, you know, a better code of conduct, a better way of doing things. I think it would really, really send ripples through the industry because, as Tim said, everybody’s hiring and they, you know, what are they going to do if they start losing people, if they don’t have people doing the work, right?

Bridget: [00:30:42] Yeah.

Tim: All right.

Bridget: I think that’s powerful sentiments. We’ll leave it at that. And thank you so much to our panel. I think this has been Very thought-provoking and informative. So thank you.

Tim: Thank you.

Jocelyn: Thank you.