inner source to open source with aaron rinehart

Matty: [00:00:00] I just wanted to talk about Stranger Things, but you know, that’s all right. It’s time for Arrested DevOps, the podcast where we help you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton, and I have a great guest with me today. But first, a word from our sponsors. This episode is sponsored by VictorOps. Built for modern incident management, VictorOps provides a unified platform for real-time alerting, collaboration, and documentation. Driven by your IT and DevOps system data, VictorOps helps you to respond to incidents more effectively so you can minimize downtime and make being on call suck less. Visit arresteddevops.com/victorops to schedule a demo or start your trial. Mention you heard about VictorOps on Arrested DevOps, and you’ll be eligible for some sweet discounts too. GoCD is the on-premise open-source continuous delivery server created by ThoughtWorks. With GoCD’s comprehensive pipeline modeling, you can model complex workflows for multiple teams with ease. And GoCD’s value stream map lets you track a change from commit to deploy at a glance. GoCD’s real power is in the visibility it provides over your end-to-end workflow. So you get complete control of and visibility into your deployments across multiple teams. Say goodbye to deployment panic and hello to consistent, predictable deliveries. To learn more about GoCD, visit gocd.org/arrested to download. It’s completely free to use. Commercial support and enterprise add-ons, including disaster recovery, are available. This episode is brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 120 common infrastructure tools such as Chef, Docker, and AWS so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog is available for a free 14-day trial at arresteddevops.com/datadog. ChefConf will be held May 23rd through the 26th in Chicago. Chef has been a longtime supporter of the DevOps movement and of this podcast. ChefConf will have talks on infrastructure automation with Chef, compliance automation with Inspect, application automation with Habitat, and a ton of other relevant content. Register with the discount code ADO2018 to save 10%.

Aaron: [00:02:41] Visit chefconf.com for all the details.

Matty: And remember, code ADO2018 gets you 10% off the ticket price at chefconf.com. So open source projects have a lot of benefits, as we all know, but it can really be challenging to take tools and projects that you develop internally and open source them, especially if you’re in a very traditional enterprise. So my guest today, Aaron Rinehart, is going to be sharing with us his journey and story with experiencing going from inner source to open source in 7 weeks. The show notes for this episode can be found at arresteddevops.com/inner-source-to-open-source. So yeah, Aaron, do you want to first let the folks at home know a little bit about yourself, a little bit about your background, and then we can talk about your project?

Aaron: Sure. Thanks for having me on the show, by the way. Yeah. So I am currently the Chief Enterprise Security Architect at UnitedHealth Group. And what that means is I mostly focus on innovation, engineering, building things in general, challenging sort of technology projects, I guess. And my background is I come from a wide array of industries, from higher education to the government, to aerospace, did a short stint at NASA. But that’s kind of my background.

Matty: [00:04:18] Excellent. So yeah, so the story we’re talking about is a tool you built called Chaoslingr. And can you tell us first a little bit about what Chaoslingr is for? Like, what was the problem set you were solving internally in your organization with the project?

Aaron: Well, Chaoslingr actually came about through Optum. Optum, one of UnitedHealth Group’s brands, hired its first SRE, system reliability engineer, earlier in the year. And on his second day on the job, I was put together with him because they thought that we’d be a good pair to meet. We started talking about what he— his name is Patrick Bergstrom, and he started talking to me about his background at bestbuy.com. He started talking about chaos engineering and just bringing down entire portions of the application and the system to test how reliable things like failover, how the application responded, how resilient it was to failure. And I started— it just got me, it sort of started pulling at my strings, right? And later that day, later that day, I sort of woke up in the middle of the night and I started thinking to myself, you know, I was like, wow, that kind of makes sense from a security perspective, right? There’s so much, you know, not to mention the amount of chaos in the security business. But I started thinking about how does that apply to information security? I started thinking about, security controls, the concept of a control, right? And where that came from and the assumptions we put upon that, right? And that, so security controls in general, like we put a lot of assumptions on it. We do next to nothing to instrument security controls, right? Outside of maybe an annual pen test or quarterly pen test or some sort of regulatory requirement for a pen test or a, you know, some sort of, audit of that nature. But what we don’t do is we don’t constantly instrument the assumptions we’ve made about these controls. And one of the examples I could talk about is, and if you go to Chaoslingr on GitHub, what you’ll see is there’s an example chaos experiment, security chaos experiment that we use in the form of unauthorized port change, right? Unauthorized port change, the reason why we use that experiment is because it’s a widely understood indicator of compromise, right? So to test a firewall, for example, we assume firewalls can catch unauthorized port changes. We assume it can block it, can stop it, it reports on it, sends good data to a log system, that it sends data to a SIEM, that that data can be correlated, you know. And so what we’re doing at Chaoslingr is that we’re instrumenting the security control plane, right? So we are actually— so we assume the firewall can do certain things. We assume the configuration management tool can do certain things. Well, stop assuming, start instrumenting, right? And that’s really where we’re coming from with Chaoslingr, right, is that we are instrumenting those assumptions, right? And what— because what we typically do when we design security controls is we build them, we put them out there, and we assume they’re doing their job. We do next to nothing to actually test that.

Matty: [00:07:47] Got it. Okay. So, so you started, you started building that. I’m, I’m interested to know the process when you decided, uh, to go from inner source to open source. So what, what drove you to, to make that decision that that was, that was, uh, something you wanted to do?

Aaron: Well, you know, it’s funny at the time we didn’t really, I mean, inner source, it was a thing that we’ve been trying to build for a while, the company, and just cuz we’re such a massive company, there’s a, there’s a massive amount of engineering talent at UnitedHealth Group. We rarely get to see the light of day because of the challenges that were put in front of us. I feel that’s kind of the— that’s kind of part of the challenge. But so, well, some of the challenges were around, you know, well, I guess so you didn’t know the InnerSource question is where you’re coming from. Well, it kind of just happened, to be honest, is that I participated in a number of forums and discussion groups, you know, talking about DevOps and and sort of cross-community sort of efforts. And I started talking about, hey, well, actually, you know, this is an important thing to talk about, is the name was not originally Chaoslingr, right? And I don’t know how far I want to go into actually what the real name is, but it’s an important concept to me, is that it’s important to make inner source and open source projects fun, right? Making them fun is a very important part of the success of the engineering. Now, I will give you hints about what the name might have been. Um, the name might have been— so, so the original chaos engineering tool was Chaos Monkey, right? In the Simian Army from Netflix. Well, those are monkeys, right? Chaos Monkey, uh, Chaos Kong, Chaos Gorilla— those are all monkeys, right? So we figured, right, so we’re going to create a security tool that, you know, what do monkeys throw, right? Monkeys tend to throw their, you know, so we decided to call it their waste, shall we say, their waste. So we started calling it MapSlinger, right? And I’ll tell you what, you know, I think that was a core component really, if we’re talking about success, and I don’t think a lot of people talk about this, right, is that having fun with the name. Like, you would not believe the amount of, uh, jokes around throwing— I’ll just call it, I’ll come out there with it, call them throwing poo, right?

Matty: [00:10:20] Hey, I mean, we, we, as Bridget says, we have made our peace with the explicit tag. It’s fine. You know, go ahead, you can say it. It was called Shitslinger.

Aaron: Okay, it’s fine. Well, it was actually It was actually— it was cleaner than that. It was poo slinger.

Matty: It was poo slinger.

Aaron: Monkey’s still poo.

Matty: That’s true. Yeah.

Aaron: But, uh, uh, no, um, that— but the jokes were there, right? They’re the sweet gifts, you know, people were throwing around, but kind of kept things fun when things got challenging, right? Is that people were able to make light of something and make it— make a joke here and there.

Matty: Um, it reminds me of something that, um, When we announced Chef Delivery, the workflow tool, that was one of the things that Adam Jacobs said, that he said, I have one requirement for this whole tool and it has to support animated GIFs in comments. It’s like, that’s all that matters. And it’s funny because he’s talked about that before, about its stuff again in Chef. It’s like, hey, what’s the system profiler tool? It’s called Ohai. Ohai. It’s like, hey, it’s fun. You know, it just makes you have a little bit of fun when you’re working. And there’s, I think you’re right. There’s nothing wrong with that. You know, we don’t need to take ourselves so seriously.

Aaron: [00:11:33] Oh, definitely. I mean, and then that’s, you know, I don’t know why that made me really sort of bring that up, but like, that’s something I really sort of noticed was, you know, after the fact, noticing that was a key component, you know, during it, it was just a blast, right? I mean, like, I mean, here I am, right? I’m asking people to apply a very difficult engineering concept to a field that most of the people in the company that got involved in the project, I did not know. They did not know each other, right? So I had that challenge, right? I asked them to come together to take on a very ambitious project, to apply a very sort of complex engineering discipline to a different field, right? And, you know, it was very challenging, and especially when you ask these people, And these people are some of the smartest people at Health Group. I mean, they range from everything from an intern all the way to a vice president, right? Just getting involved because they wanted to, right? That’s different, right? You know, it’s different than, you know, having an officially funded project and working through their managers. And, you know, we drove more carrot than the stick, right? People were very interested. And I actually have a great couple points on the carrot versus the stick when it comes to this topic. Um, you know, people were just excited to work with— to work on a project called Poo Slinger. It started to develop sort of a persona of its own, you know. But, um, taking on such a challenging thing, it doesn’t have to be— I really think when you break, break down the social pieces and the, you know, um, you know, it’s— the project is generally fun. These people come out of the woodworks when they’re willing to help you. Right? And that’s, you know, all— especially when you’re asking for people’s extra time. I mean, this was not— some people could not work on this until all the other stuff was done. I mean, people, uh, we got this, we released this in 7 weeks. Now, for the unicorns out there, um, that’s probably normal, right? Or that’s— that doesn’t seem, uh, difficult, right?

Matty: [00:13:46] Well, I don’t know. I mean, it’s, uh, I’ll tell you one thing is that creating something new, it takes the time that it takes. I think it’s just the iterative thing that’s a little faster. But as John Cowie at Etsy— well, now he’s at Chef, but at the time he was at Etsy when he said this, he’s like, we’re not a unicorn, we’re just a sparkly horse. It’s all similar things. I mean, I think, yeah, getting a whole new thing. 7 weeks, that’s from, you know, from scratch. That’s pretty impressive for anybody.

Aaron: Well, you know, it just, it just, it just got to be so much fun. I mean, I was, I, to be honest, I spent the weekends on it reading, reading like the Netflix during the time we, uh, we were working on this, uh, the initial release. Um, oh yeah, I didn’t add to the fact that inner source to open source in 7 weeks. So it was when we came together, We started building, so taking a very iterative nature to how we were building from an engineering perspective was very important. You know, taking a strictly Kanban-based approach to the way we were, you know, taking on the engineering. But it wasn’t just the inner source, it wasn’t just the open source piece of it. What needs to be known about that is that this was the very first open source project officially for the entire company, right? So UnitedHealth Group, right? And if anybody knows anything about, you know, large insurance companies or healthcare companies, there are a lot of attorneys, right? So the legal part’s very important. And, you know, we were able to get through in that timeline, right? That was— I mean, I have no idea how we did that. We just did it. We did everything they asked and And, uh, we were able to release it. You know, uh, that, that’s another piece that makes it really challenging. I don’t talk about very often, there’s a legal piece.

Matty: [00:15:45] Well, I was going to say, and, and I understand that there’s probably some specific things that you might not be able to talk about, but is there any maybe advice? Because I think, uh, people have these challenges, and, and I, I see this a lot with, uh, you know, in my time at Chef when, you know, we obviously encourage people, encourage our customers to say, hey, contribute back, and being able to give that guidance. I think a lot of folks don’t really know where to start and how to be able to have those conversations with their legal departments. Do you have any advice to people that are going through this to make that process easier or have a little less friction?

Aaron: Sure. I’ll tell you what one of the key things was, was that I guess so the key thing to do if you’re a large enterprise, right, would be to find the open source attorney or find an IT attorney that you can talk to, right? Find out who that person is, schedule time with them, and explain to them what you want to do and why it’s important and the value of open source, what it brings to the company. Don’t just explain the engineering, explain it from the business, how it adds, because open source Here’s another key nugget though. Find the open source attorney or an IT attorney to talk to. That’s where you start. Then you have to develop a policy. You have to talk about what types of— how you plan to manage the community, how you plan to manage contributions. If you have an open source attorney, it’ll be probably easier. If you have an IT attorney, you’re going to have to work through those challenges with them. But feel free to reach out to me or Kevin Nelson. He’s on our repo and we’d be be more than happy to sort of explain what our journey was there and, um, to anyone.

Matty: [00:17:38] And I think this is a good time to mention too, we’ll, we’ll have a link obviously to the repo for Chaoslingr in, in the show notes. So make sure to check that out to, to look at that. And, um, of course, uh, Aaron’s bio will be up on, on the show notes. And I don’t know if you’re a Twitter—

Aaron: I’m a Twitter type person. I am.

Matty: You am? So, well, yeah, so People can get some convos going there for sure.

Aaron: So when—

Matty: how long ago was the tool open source and how long has it been kind of, if you will, out in the wild?

Aaron: It’s been— oh, it’s probably been a couple months now. Maybe not. Maybe about 6 weeks, 6, 7 weeks. How about 6, 7 weeks? I mean, I’ve been traveling in Asia for the last few weeks, so that adds to it. But yeah, no, so that has been— so another key component of it was that, you know, the managing of community too when you release an open source project, right? It’s very important. You know, it’s my experience thus far has been beat the street, right? What I mean by that is that it’s if you build it, they will come is not a truth, right? You have to get it out there. You have to talk about it. You have to You know, go to social media. You have to reach out to people who are doing similar things. I reached out to Netflix, you know, just before we released it. And I said, hey, we— so one of the key things as part of Chaoslingrs, we were very— we stayed very true to the nature of chaos engineering, right? We did not venture out on our own. We just applied the same concepts, right, to the field of information security. We kept the chaos principles. Um, we— our tool operates within the same paradigms as Chaos Monkey. Um, we actually started off with Chaos Monkey and wanted to write it in Go until we realized only a couple people knew Go, right? So we’re like, well, you know, um, you know, I think—

Matty: [00:19:41] well, you know, I mean, I, I think you should have written it in Go. You know, I’m a Go fanboy right now.

Aaron: But— oh, me too.

Matty: If I, if I could— if I can learn it, then, you know. But again, it’s doing the right thing. I mean, that’s actually a really good point. And we talked about this with Jeffrey Snover from Microsoft a couple years ago where he was saying that’s one of the downsides of kind of everybody kind of doing whatever they want, where you get this thing where someone’s like, this is their pet thing they wanna do. And they’re like, and he’s like, oh yeah, I’m gonna write this tool in Ada because I’m taking a night class about it. And then everyone’s like, but nobody knows how to support this thing anymore. And I was like, well, that’s what got us Ruby on Rails, you know? There’s that.

Aaron: Yeah, no, it actually— what’s funny is it was a key— I think that was another key thing as part of the project was that we, you know, everybody got on board. Everybody’s like, let’s do it in Go. You know, the Go tutorial started flying around and I started thinking to myself, man, you know, hmm, you know, people’s part-time here. We have no funding for this project, right? We’re working out of— we’re bootstrapping this baby. And we have a bunch of people who don’t know the main programming language, right? So I was like, well, okay. So I got folks together. This is after the first couple days of the project. Everybody’s getting excited. And I started saying, well, what is the common language that we all know, right? I mean, what is a common language we all know, right? And then it sort of came out that there was Python, there was Ruby, there was Node. We decided, right, that we started having a conversation about the architecture, right? We started thinking about— if we started looking at Chaos Monkey in general, sort of picking it apart, right? And Chaos Monkey was written in an era, right? And I said era, it was like 2008, feels like an era ago. Um, it was written in an era, the old days, um, um, medieval times. Um, uh, it was written in an era where Um, uh, there wasn’t really, uh, Lambda, Amazon’s Lambda, right? AWS Lambda, right? And, and there wasn’t as many native services, right, um, that they could utilize in Amazon, right? So like they had to write their own pipeline tool, uh, I mean, they were there, I mean, they didn’t write their own, they modified it, customized it, right? Spinnaker. And, um, and, uh, anyway, so they had to write a lot of their own components. That’s why Chaos Monkey is so heavy. So what we decided to do is, well, maybe we don’t need to write all that code, right? Maybe what we really need to do is focus on what we’re trying to do, right? So that’s the great part about native sort of serverless design, was allowed us to not focus so much on the IT components or the system-level components, more on the engineering and the value we’re trying to generate with the tool. So that was a key component. So, oh yeah, but what is the language we all chose? We chose Python, right? Python made the most sense of what you’re talking about, a bunch of people that are from all truly cross-functional, everybody from, there are people in marketing that could code that came out and started working on it, right? You know, I wish we had a little more marketing, but we actually had code.

Matty: [00:23:07] Yeah, and you know what? I was just poking around and looking at the repo, and you’re talking about the community. And so I was looking at first because I was kind of interested to kind of look at like PR activity and contributors. But what I noticed is some— excuse me— I noticed some great things like, you know, you have a code of conduct in there. You’ve got a really great contributing.md file, which are all super important things if you want people to help you, right? And like you said, building that community is super essential. I think that’s something that everyone should be thinking about, even if it’s a small thing you’re starting, because I know I have projects I do that maybe I’ll be the only one that ever really contributes to them, and I feel silly sometimes putting in a code of conduct and a contributing file, but you never know who’s going to want to help, and you want to make sure that stuff is set. Also, I have a project on Morgan that I’m one of the only people that works on it. My friend Ken said, you’re the only person I know who argues with themselves in GitHub issues.

Aaron: [00:24:14] I caught myself doing that before.

Matty: It’s like a way of keeping notes. If you think about it, it’s just a communication tool and maybe you want to remember what you were thinking at a time. I have a little bit of question. This is getting into the weeds a little bit, but since I was poking around in your repo, I saw you’re using GitHub, well, maybe using, at least you have a GitHub project enabled. Are you using that for tracking? I’m just curious what your thoughts are on GitHub projects.

Aaron: Yeah. We use GitHub Enterprise internally. I will tell you, GitHub Enterprise is a transformation acceleration tool. It really is, at least for us. We have some internal Agile tools that are tough, that make things tough, right? I have— I’m not gonna say what those are, but I— there are some, I’m just gonna come out and say there are some Agile tools out there that make Agile difficult, right? And it’s not supposed to be difficult, right? It’s supposed to be simple, right? And GitHub Enterprise’s Kanban, literally I’ve taught people the concepts of Agile, I mean, rapidly. They immediately get it when they see Kanban, how fast it is, so intuitive, how, oh, I’m just writing down what I need to do and prioritizing it, and I’m moving it, and I’m just working on piece by piece. Really, I’ve gotten people who knew nothing about it, just accelerated. I love GitHub products. Everything you need is in one little package. Your wiki’s there, you start documenting, start putting out your issues. Those issues, you put them in the Kanban. You know, um, I literally— you could be 20% into a project, right, in a couple hours. You know what I’m saying? I love it.

Matty: [00:26:08] Yeah. And it’s funny, I think the reason— because I started playing with it when it came out, and I’m a little, a little less enthusiastic about it than you are. But I think it’s because, uh, so the tool I used for that— so like, I’ve used, uh, ZenHub, which is a little more agile stuff, and it’s a little heavy, but I really like Waffle. And Waffle is very Kanban, but it has a lot of neat stuff that I really wish GitHub Projects had, like auto-closing cards. So, you know, I can sit there and I can reference, you know, the issue is sitting there, the issue is a card. It also, they automatically happen. That’s one of the things I don’t really love about GitHub Projects is you have to kind of consciously make the issues into there. Whereas the issues just in Waffle, they just end up in the backlog automatically. One of the things that’s neat is with Waffle, if I create a branch and the name of the branch includes number sign and then the issue number, that connects the PR for that branch automatically to the issue and then they all magically close themselves when everything gets merged. There’s a a lot of just neat little things.

Aaron: [00:27:23] That is kind of neat. That is kind of neat. Yeah, I’ve never used it. That’s kind of cool.

Matty: The thing that some people don’t like is it just, you know, this all goes back to just GitHub’s terrible authentication API. Whereas like when you have an app, it has to like have access to all the things. Like you can’t just say, I only want the, you know, like you put it in there and if you’re in like 30 billion GitHub orgs, it wants to grab them all. So there’s, you know, people get freaked out about that sometimes, but it’s really kind of a good tool. And it’s, you know, if you’ve got open source stuff, it’s free. So like I said, I think it’s great that GitHub has added this though, because it’s right there in front of you. Like you said, it encourages people into that. And I’m sure they’re going to continue to iterate on it. You know, they’re going to keep making it better. And there’s no reason, I mean, they’ve got all the backend to be able to make these things magically work together. So it’s pretty cool in that regard.

Aaron: [00:28:27] Well, actually there’s an interesting, so I explained the enterprise side of it, right? But like there is actually a, we don’t really on the public side, we really don’t heavily, so we are using it, for the public project, right? But like for the collaboration pieces, like we really focused on using other collaboration tools, right? Like Slack, right? Because they’re more community-based, like, you know, so the project Chaoslingr is being worked on. We have a, I guess, a co-project with Netflix on this to develop the product and develop, we’re sort of taking an interesting spin to it for fun, to have fun. You know, and, but we communicate with Trello and Slack for those community type pieces. But anyone feel free to join in. We’re, you know, we’re actively looking. There’s no, it’s not an exclusive project, right? Just something we’re having fun with, you know, and if you like challenging engineering projects and you’re interested in the project, just reach out.

Matty: [00:29:34] Yeah, it’s, I’ll have to take a look, especially because, For me, what I’ve found, and I gave a talk at ChefConf about, the talk was called The Hand Waver’s Guide to Contributing to Open Source. And it was for people that are architects and stuff who don’t really do a lot of engineering, how do you do things? And one of the parts in my talk was saying, when I decided that I wanted to learn Go, this is what I did wrong. What I wanted to do is I wanted to learn Go because I wanted to contribute back to a project that I really appreciate, which is Hugo, which is a static site generator that I use for all of the things. But I felt like I had to get really good at Go before I could do that. So what happened is I did this by creating my own things in a vacuum versus actually trying to learn and help with something that was already there. So in this case, because Python is a thing that I’m not super skilled at, right? I’m a pretty decent Rubyist and got going on, stuff like that. But I look at this and I’m like, hey, you know what? This is a good way to learn a new language is to help out with a project. You may be seeing some stuff coming from me.

Aaron: [00:30:42] I may check it out. Well, actually, what we really need more than anything is that we’ve been working on—

Matty: Documentation, right?

Aaron: Well, documentation. Yeah, there’s that too. That’s always— you can insert that into that statement all the time. But what we’re really working on is the experiments, right? The value of the tool gets more and more valuable as we develop experiments. You know, like we just released— really, we’ve sort of been taking this track working with the Netflix team on— I don’t know if we want to— I want to directly say what we’re doing with it, but we’re working on a combined effort where Netflix has a tool, we have a tool, Develop a common orchestration for both tools. And it’s sort of interesting, it’s turning into being an interesting project, but we’re really, for Chaoslingr though, which the focus of Chaoslingr is instrumenting the control plane, like things like an S3 bucket, right? Like you assume when you put an insecure S3 bucket out there, what the most common cause of data breaches for Amazon, right? For someone building at Amazon, right? That, you know, you can catch stuff like that in insecure S3 buckets. Well, don’t assume. Test it, right? Test everything, right? The— I am— you’ll find that I am a security person who does not believe in the concept of risk assessments, right? We do them, right? But to me, there is— well, here comes explicit tag— there’s so much bullshit in that process, right? And I am on a mission myself to drive engineering to architecture and engineering back into my field in general, right? Information security began as an engineering problem. We became auditors and assessors, and because we, we were trying to get the rest of the industry to understand the importance of security, we end up doing is, is driving the importance of compliance, not security. And that, that is, that has hurt the field in my opinion. And I’m gonna I’m on a mission, a personal mission to drive it back, to drive it back.

Matty: [00:32:51] I’ve got this vision of you in the black suit and the fedora and the sunglasses like Belushi and Aykroyd. You’re like, we’re on a mission from God.

Aaron: I feel like it’s actually, you know, we had a great conversation at DevOps Enterprise a couple of weeks ago about this. People were just, we were obviously after James Wickett and I were talking, It’s just like people were asking, like, you know, it’s interesting you say that. Like, well, I feel that way, right? I mean, it’s like I’m sure there’s a reason why they talk about a shortage of engineering talent and security. I mean, not talent. They talk about the shortage of talented security professionals, right? Well, that’s because they’re all auditors, right? Like, I mean, like, I’m not saying they’re all auditors. I’m not discounting the value of compliance. Just so we need to be clear about this, right? It is very valuable to have compliance. Compliance brings documentation. It keeps people in check. There are lots of people that are out there that don’t see it as important, wouldn’t care if they did. But the importance— everything comes back to good engineering. Matt, I’m a big— I am a big believer in that good sound engineering begets good compliance, right? Because what I mean by that is that you can actually be ahead of compliance, right? I mean, if you’re doing good engineering, right? I remember before, like, when Docker came out, right? Security people were freaking out about it, right? Well, when, um, if you actually were just looking at the engineering and, and, and trying to focus on making, uh, things better from a quality perspective, um, it was possible to be ahead of any kind of guidance that comes out. And when it does, all you’re really doing is documenting what you’re already doing.

Matty: [00:34:36] Yeah.

Aaron: You know what I mean?

Matty: Security and compliance are just another aspect of quality.

Aaron: Precisely.

Matty: I can tell you something I think is really funny. So I have a talk that I’ve given a few times called Shifting Left Securely, and it’s about moving your security testing further left in your pipeline and all that kind of good stuff. And so my wife, who is not in the technical field, she runs marketing for a pretty large company, but I was practicing the talk. And, you know, making her sit and watch as an audience. And at one point she said, okay, she’s like, I know that I don’t really know this stuff, but is this really that innovative that you’re talking about? Because this just seems like a thing you would do. And I’m like, I know, right? You know, it’s exactly— it just, it’s, it’s, it’s really funny that when you take, you know, because when we’re inside it, we kind of have these— because like you said, you know, the way that it’s, it’s been done We have these presuppositions around it. And then if you take the step back and say, okay, what is this real outcome? What is this actual thing from a business perspective? It just becomes almost ludicrous to think about doing it the way that we’ve been doing it. But it’s hard, right? And especially when you do have people who have made their career on not being an engineer, but being a PDF jockey, right? Those are some challenges to kind of shift that because sometimes they’re in a position to make that more challenging for an organization. But I think things are on the upswing in a good way. There’s a lot of— as much as I kind of laugh about— I always say that every year at the DevOps events, there’s the theme of all the talks, right? In 2014, it was all about empathy. And whatever. And so the theme of 2017 was it was all security this year. Like every event I went to, there were just tons of security talks, which was great. But it’s also all with this DevSecOps thing. And it’s like, I’m just like, whatever, it’s all DevOps.

Aaron: [00:36:50] It’s all DevOps. I actually, you know, I feel the exact same way, right? Like, you know, we have this big— so we’re having this big discussion right now, James Wickett, myself, Several other people in the security business, even Shannon, Shannon Leets added to it. You’ve got Jeff Williams, Sam Buchenheimer, Kunal Anand. There’s quite a few people, but what’s his name? Josh Corman. We’re talking about bringing back rugged, right? And like in a new way, right? Is that, you know, what I want to say about this is that Rugged software. So there’s this conversation about where rugged was not effective versus where DevSecOps was effective. Well, that’s the way I see it is, is that I love DevSecOps, by the way. Right. But like DevSecOps is just what we should be doing. We should, it’s just DevOps. Right. And at some point, security will just be DevOps. Right.

Matty: [00:37:51] Well, it’s, it’s the whole thing is that it’s DevOps in a lot of ways is just an unfortunate name because it’s naming semantics. Yeah, it implies that it’s only those 2 groups, right? You know, when it’s the whole DevSecDB, QA, BizSecOps, whatever. But, uh, but the thing is, I, I, again, as I— and I don’t want to go back to your point, but like, sure, as I’ve made my peace with calling something a DevOps team or whatever, I’m like, whatever you have to call it to make the thing work, I’m fine with it.

Aaron: Right?

Matty: So if we want to call it that, and that will make people think that way and will be more awesome, great. Call it that. Who cares? You know?

Aaron: So, well, I guess so. I guess I would. I guess I, yeah, the whole naming convention thing got confusing for a long time, right? For the last couple of years, right? The whole rugged DevSecOps, SecDevOps, you know, it’s just DevOps. All right. Leave it alone. Right. But rugged software is something different. Right? And this is something that I, I guess I want to put this out there, right, is that, is that rugged software, we’ve, we all came together on this concept in the past, like, few months, right? The group of folks I mentioned a few minutes ago is that we started thinking about, you know, what is the future, right? You know, at some point, security will just be a part of what’s being done, right, in DevOps, right? It’s slowly making its way. It’s going to take a while, But it’s— but that is its core component, we feel, of something else, right? And we feel that, like, you know, what does it mean to develop rugged software, right? And we started— there’s this huge movement right now in software design, right, around antifragile, right? Antifragility, right? And we started thinking about what does it mean to build antifragile software from a security perspective? And we started thinking about that concept of rugged, driving in chaos, right, with Chaoslingr, right? That was one of the core focuses there, but, you know, taking some of the concepts from DevOps, like, so it’s just something we’re currently working on. It’s been fun, you know, but I do want to, I just want to sort of reiterate that, like, you know, the concept, like we do, like you and I were talking about, like, it’s just DevOps, right? The concepts are, security is just sort of late to that game, in my opinion, but at some point we’ll just all be on the same team. I hope so.

Matty: [00:40:09] Absolutely. And, um, if you’re listening, I, uh, so if you want to learn more about Rugged Software, you can go to ruggedsoftware.org. There’s a link to it in the show notes if you forget. It’s pretty easy to remember because it’s the name of the thing. And if you don’t remember the thing, you can find it the way I did, which was by Googling Rugged Software. So the good news is it’s easy to find. Uh, so, but you know, we like to have show notes, so hey, we got some, got a link for the show notes. Um, in so much as that goes. Uh, so I have a question. Um, I think these are, these are some, some really— I think we’re— there’s a lot of really forward thinking that’s, that’s happening in this, uh, in that space. And I just want to kind of, uh, kind of come back, uh, to kind of as we kind of wrap up a little bit to, uh, back to Chaoslingr. Is there anything that you would have done differently in your, in your process? Like if you said, wow, man, if I had known what I know now, this is what I would have done. Now, not saying, I mean, you did this in 7 weeks, so I mean, you crushed it, but hindsight’s always 20/20. Curious if there’s something you would’ve maybe approached in a different way.

Aaron: [00:41:14] I’m not so sure. So I’m not so sure if this is one of those, but I’m gonna throw it out there anyway because it’s important content to bring up either way, was that, so we truly released this in inner source startup unicorn-type fashion. I mean, it was all on video chat, right? Well, yes, they would, you know, some people would do a video chat. We’d do a video chat in our pajamas cracking beers, right, when we posted to GitHub, right? Um, but like, uh, it was, uh, uh, but when we released it on, on, uh, we said that it would do an initial tweet. We started going out to social media We started sending it on, we did Twitter and LinkedIn and a few other places, right? But like, that was a, this is like a Friday night we did this, right? Remember, these are all people giving up their spare time on their Friday evenings to do this. It was all voluntary, right? But like on Monday morning, I got an alert from the marketing office, a brand watch. Brand watch is like a, software, right, that keeps track of your brands, right, on social media and the open internet, right? Well, I got an alert came in from the marketing officer like, Aaron, then I ended up getting a phone call like, Aaron, what is this GitHubby thing, right? And I’m like, what do you mean, right? And they’re like, well, you’re trending number 2 for the entire brand, right? And I said, oh, okay, that’s awesome, right? Well, we don’t know what it is. What is this thing? I was like, well, this is an innovative engineering product that we came together. It’s approved through legal and all this stuff. I explained all that. They’re like, wow, this is interesting. I said, well, since we’re trending so high, can we get a retweet? Can we get some support? The response was that there’s an enterprise program for you to fill out this Volgon form and this Volgon form. And it was not beating up the marketing team. I mean, we have a very talented marketing team. It’s just that they just weren’t familiar with the concepts, right. And what we were trying to do. And I think hindsight-wise, I would have tried to spend more time with them and understand and teach them the value of it. Right. That way, cause we missed some golden opportunities, right. For retweets from the brand.

Matty: [00:43:38] Yeah. I think that’s, yeah, getting on top of that. And that’s the other thing too, is We, and I think it even goes back, right, to, um, to your thing about being able to engage with legal and thinking about how there’s, there’s other parts of your organization that are good at things that you’re not, right? Um, like, that’s one of the things, you know, even though it’s not necessarily from the organization standpoint, but, you know, with, with the podcast, it’s like, I’m not a marketer. I’m not even a social media genius. I mean, I thought I was till I met my wife, who this is, you know, who runs a whole department of that for, you know, a giant company and tells me everything that I’m doing wrong and, you know, shudders every time she sees me post on Facebook. But there’s been a lot of stuff where to be able to say, hey, take advantage, right, of, of those things where you don’t necessarily have that, that skill set or that experience, but how to, how to promote it, how to do those things, because it’s a— it is a product, right? Like, you, you want to get it out there because you want that the advantage of open sourcing it is to have people using it, which is just going to help make it better. So I think, again, making the tent big and it’s not just big within tech, I think would be probably a lesson to learn from that.

Aaron: [00:44:58] Definitely. It’s definitely been the experience too in promoting it. And I do recommend, I mean, social media alone will not get you there, right? You have to be aggressive and reach out to people. And remember, like, the engineering community is a HugOps community, right? And people will help you. This is not the engineering of the ’80s, right? And the ’90s, right? This is, this is open source has driven such a culture in who we are, right? And the willingness and desire to contribute and work together. When I reached out to Netflix, it was open arms. Just so everyone knows that. But that’s how everyone is now, right? Oh, I feel not everyone is that way, right? People are people, right? But like, don’t be afraid to reach out to somebody that you feel is interesting and you feel would be interested in what you’re doing.

Matty: Fantastic. So we haven’t really officially done checkouts and stuff like that on the show anymore, and I realize, but I’m gonna throw you on the spot just to see if there’s anything anything cool that you’ve been playing with and that you’d like to share with the audience that you think they should check out? If not, it’s okay.

Aaron: [00:46:18] Really, the focus has been really on Chaoslingr and trying to build out because it’s really when we released this tool, what we really did was release a new branch of engineering. That is also people explaining, okay, why, what is important about that? You know, how do I do that? Um, you know, what is this? What are the principles? What, you know, like all the other things that we’re like, oh, we weren’t thinking about all that really, but we probably should. So, so, uh, you know, we’re documenting all of that, you know.

Matty: So I just wanted to talk about Stranger Things, but you know, I have not seen that yet.

Aaron: You know, James was telling me all about that. He said it’s so worth it.

Matty: Like, I, we, my, my wife and I, we didn’t watch the first season when it came out, but we started watching, like, so with the second season was on, you know, was released like a couple weeks ago and we started watching the first, you know, we binged it in like 2 weeks. Um, yeah, it’s, I’m gonna have to see it now.

Aaron: [00:47:19] Yeah. Yeah.

Matty: It’s, uh, it’s like, it’s especially, you know, if you, you know, grew up in the ’80s, you know, there’s so many references and stuff and it’s, it’s, it’s nice and creepy and You know, it’s a lot of fun. So yeah, so, you know, if you’re listening and you haven’t watched Stranger Things yet, you know, get with it.

Aaron: I will definitely take that to note. Cool.

Matty: Are you going to any conferences or anything in the near future?

Aaron: Oh, I’ll definitely, definitely be at RSA and maybe the O’Reilly Software Architecture.

Matty: Awesome.

Aaron: Cool.

Matty: I don’t know where I’m going to be. This is— by the time this gets released, this will be kind of a known thing, but I am changing roles. I’m leaving Chef and I’m joining PagerDuty as a DevOps evangelist. So I’m going to probably be at a lot more events, but I don’t know what I’m going to be doing. So look for me, I’ll be somewhere. Thanks. Yeah, it’s mixed feelings. I’ve loved being at Chef and I’ve really grown from being there, but this is going to be an awesome role. So speaking of events, we got some open CFPs. There’s a whole bunch of DevOps Days have opened up their proposals, and you can find them all if you go to devopsdays.org/speaking. ChefConf has their CFP open, which is at chefconf.chef.io/cfp. And ChefConf this year is in Chicago, so you should come to it because I will that’s where I live and I’ll be there, or at least I’ll be in the city. So yeah, so head over to arresteddevops.com/inner-source-to-open-source for this episode’s show notes. Our website also has where you can sign up for our newsletter, The Banana Stand, and any kind of crazy Arrested DevOps stuff you could ever want. By the way, if you want Arrested DevOps stickers, You can go to arresteddevops.com/stickers. You can buy them. Trust me, we don’t make any money on them. You buy them from Sticker Mule for cost, or you can find me at an event and I’ll probably have some. Also, if you go to arresteddevops.com/itunes, that’ll shoot you over so you can leave us a review in the iTunes Store. It really helps when people— the more reviews we get helps people find our podcast. I know it sounds like I’m shilling for reviews, but guess what? I super am. Uh, we also, if you use Google Play, we’re in the Google Play Store. I don’t know exactly how to find us there, um, to leave us a review, but you know, if you’re into that, do it. That’d be rad. No one’s left us a review in the Google Play Store and it’ll make me excited. So yeah, uh, Aaron, thank you for joining me today. I know it’s, uh, the middle of the night or early morning. It’s like, what now, 3 AM your time? You’re, uh, on the other side of the planet.

Aaron: [00:50:08] So it’s 3:30.

Matty: Yeah, I appreciate you getting up in the middle of the night for this. That’s fantastic.

Aaron: Thanks for having me, Matt.

Matty: Yeah, and this, I know we had a, for listeners, we had a little bit of a challenge getting this episode scheduled, but I’m glad we got it done. So yeah, I’m Matt, @MattStratton on Twitter. This is Arrested DevOps. And remember, there’s always DevOps in the banana stand.