Something About Security With Ben Hughes

Matty: [00:00:07] Welcome to Arrested DevOps, episode 20, something about security. I’m your co-host, Matt Stratton, @MattStratton on Twitter.

Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.

Matty: Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that figures if you’re listening to this podcast, then you must be pretty cool. And you can find out about joining their cool cloud services team at 10thmagnitude.com.

Trevor: Hey everybody, the Chef Community Summit is coming up October 2nd and 3rd in Seattle or October 15th and 16th in London. You can use the discount code ArrestedDevOps for 10% off your registration at arresteddevops.com/chefcommunity.

Matty: Also, remember that registration is still available for the first ever DevOps Days Chicago. Arrested DevOps listeners can get 10% off of your registration with the code ADO10. Visit arresteddevops.com/devopsdayschicago for all the details.

Trevor: And don’t forget, Matt and I will both be there so you can meet us if that’s your interest.

Matty: [00:01:10] That’s a, that’s a draw. We will also be broadcasting a special episode of Arrested DevOps at the conference. This episode is sponsored by PagerDuty. PagerDuty eliminates the noise, chaos, and manual processes across the entire incident lifecycle to decrease resolution time. PagerDuty is trusted by companies like Etsy, Nike, and GitHub. To sign up for a free 30-day trial, visit arrestofdevops.com/pagerduty.

Trevor: Are you looking to change jobs? Skip the phone interview and take a technical interview online through TrueAbility. TrueAbility is a marketplace that matches tech pros and their proven skills with the best career opportunities. No certs, no degrees, no worries. They spin up a live server for you to troubleshoot and configure. Your results are delivered to the recruiters directly. What’s even better? You can take one technical interview online and apply it to several companies. Visit them today at arresteddevops.com/truability.

Matty: So, uh, what you been up to, Trevor?

Trevor: Oh, you know, this, that. I just got back from San Francisco. Last week I attended Flowcon. Which was a lot of fun. There was a lot of good discussions. One of the things I really took away from it was there was a discussion about how to kind of look at stories from a way that I hadn’t done before. And that was kind of to document— just lack of timeframe to properly explain— kind of document everything as opposed to document the, as a user, I’m going to, so I can, that kind of flow, which was an interesting take. Matt, what have you been up to?

Matty: [00:02:42] So I’ve been traveling a bunch. I’m coming to you today from Austin, Texas, where I’m down here giving some chef training to some folks. I was floating around in Ohio and California last week and just kind of flying and traveling all over the country, fueling the love of chef and trying all the different food that different cities have to offer, which is kind of fun. So yeah, it’s been some good times. Also, We, like we said, we’re planning DevOps Days Chicago and it’s getting to the home stretch and getting just a little nerve-wracking, but I think it’s going to be awesome. I’m really excited about it. I want to thank iTunes user OregonGuy3 for the review he gave us on iTunes. And as promised, we’ll read them, read it aloud. He says, it is a must-listen for sysadmins. This show is a peer of The Ship Show, DevOps Cafe, et cetera. You don’t need to buy into DevOps as a fad to benefit from it. Just be interested in tech and infrastructure in a serious way. Really great show. I think that was complimentary. He gave us 5 stars. So thanks, Oregon Guy 3.

Trevor: [00:03:47] So tonight we’ve got our special guest, Ben Hughes from Etsy. Ben, will you tell us a little bit about yourself and how you started learning about DevOps security and all that nonsense?

Ben: Sure. Hi everyone. I’m one of the senior network engineers at Etsy. Despite having network in the name, we don’t really deal with the networking stuff. We leave that to our NetOps team. So we mostly look after infrastructure. So we have some thousand machines and some hundred laptops, and we kind of look after those, working closely with operations and our help desk team to make sure that’s all good. How I fell into security, much to the dismay of, I’m sure, my high school careers advisor, I kind of dropped out of school after discovering 2600 and Linux, like Linux 1 something, and then just kind of started exploring on Linux boxes cobbled out of skips and then started exploring perhaps other people’s Linux systems. And that’s, that was a really good way to learn about security back in the ’90s. Since then, I’ve had a bunch of systems networking jobs. My last place was Puppet Labs. I worked for a number of startups in the UK. And recently I’ve just been pushing real hard to try and bridge the gap between security and other parts of the organization. Which is— Puppet Labs do a lot of stuff with DevOps, with their DevOps survey. And so I kind of found out from DevOps Days events from that, and then I’ve spoken at a few of them trying to get people on board from both the security side and the development and operations side.

Matty: [00:05:15] Awesome.

Trevor: Why don’t we jump right into things? I actually, I want to interject with a question. I’m curious, when you say other people’s Linux machines, do you mean other people’s Linux machines you broke their security and were exploring? Or other people’s machines you had access to?

Ben: It was the ’90s. It was a crazy mixed-up time. Yeah, I think at last count I had, I don’t know, 20 years ago I had like 1,500 accounts on my high school’s Unix system. And that’s— I did a lot of learning and I didn’t let it get in the way of my education, which is I think the important thing. They were less pumped by it, but that was the first time around Red Hat, was at Red Hat 6.

Matty: You’re saying that, you know, you don’t necessarily deal with kind of network layer security, not firewalls and VPNs and all that kind of stuff. So what does your team do all day at Etsy?

Ben: Yeah, so our days really thankfully kind of vary. Like today I’ve been— well, this week actually I’ve been doing things on working on Python with our Midas project, which is our Mac host-based IDS system that we released with Facebook. That’s like your traditional host-based IDS, such as, say, Rootkit Hunter on Linux. But that’s specifically designed for the Mac. And we’re developing pretty heavy on that. It was kind of revolutionary, and not many people or organizations have something like Carbon Black or Bit9 for Mac. But they’re starting— commercial products are starting to appear. So this is a good open-source project that we’re hoping to push more to after we develop it in-house. Last week I was writing Golang to try and trick Logstash into doing different TLS ciphers. I spend not enough time as I’d like in Wireshark and tcpdump, so why we don’t do a lot of the networking side of things— stuff hits networks. Next week I’m at OpenDNS’s S4 conference, which is their incident response conference, meeting up with a lot of people there to talk about how to deal, or how we deal, and how other people deal with incident response for the eventual when there is a breach of some kind or an incident of some kind, not the if, because the days of it being an if are long gone, I think. Just ask Home Depot.

Matty: [00:07:23] So, I mean, so there was one of the things I remember in a talk I saw, you talked about, you know, you gave the example, you just sort of said it’s a when, it’s not an if. And you talked about, you know, with your example you were giving at the time was about phishing. And it was the question, it wasn’t the, you know, it was basically you’re going to get phished. So what are you going to do about it?

Ben: Yeah.

Matty: So that to me, it seems pretty logical way to approach security. But do you think that’s common, you know, for an organization to think in the— I don’t want to say proactive because proactive almost feels like preventative.

Ben: Yeah. But just one of the things that you do and we’re somewhat famed for is kind of focusing on people first. So like there’s no point shouting at someone for clicking a phishing link because some people like recruiters, they’re meant to get links. They’re meant to click on them. It says PDF at the end. Someone’s sending you a resume, that’s their job. It’s just that Adobe Reader has not always had the best track record. PDFs are a great way to hide malicious content. So you can’t really be angry at someone for doing that. That doesn’t get you anything. Blameless postmortems work for blameless phishing campaigns. The more important thing to do is when someone does click on something and like talc appears or something screwy happens is to come and tell the security team so then you can start incident response, and actually doing something about it. The more you chastise people, the less they will tell you when these things happen. There’s companies you can outsource to do phishing attacks on you and do phishing education and this kind of thing. And we do some of that internally. And it’s not trying to catch people out. It’s, you got phished. Yeah, that happens. You told us about it. Thank you for telling us about it. And then I think we gave everyone Etsy gift cards, which are pretty much our libre franca currency internally. And like the more people that report phishing, the better. We try and leap on and be grateful every time anyone sends anything that looks weird. I know some organizations are like, you should stop clicking on phishing links. And you can buy— like large companies have things in their email to nerf links so that they go through an internal thing so that you can stop subsequent click-throughs on that. But for people who just use Google Apps in a small business, there’s not really anything. I mean, Google’s phishing protection is top-notch, but it’s not 100%. Nothing’s 100% in security, despite what some people will try and sell you. So being able to respond is much more important than thinking you have 100% coverage.

Matty: [00:09:39] So one of the things too, and especially topical, I guess, right now, is this idea of this conflation, I guess, of like cloud equals insecure, or, you know, kind of said there’s OMG, oh my God, scary clouds. I mean, so what is kind of your— I’ll leave that as kind of an open-ended thought around the statement of, well, I can secure my stuff way better than I trust a cloud provider to do. Do you think that’s a fair statement just to make?

Ben: I think if you go into cloud with the same assumptions as with the armadillo security model of the past 20 or 30 years, then you’re going to have a bad time. I just saw a talk at Pinterest by Werner, the CTO of Amazon. Which I agree with, at the point that the idea of something being internal, so it’s on a trusted network, is not necessarily the case anymore. And especially in cloud, unless you use VPCs and you’re using, say, some of the older Amazon stuff, there is no internal trusted network. It’s just like you’re on the internet. And some things I think Redis suffered from the, we don’t really do user credentials and access control. You should put this on a trusted internal network. Cool. Yeah, the cloud doesn’t, unless you go and build one, necessarily have a trusted internal network. And for some small shops, there is no trusted internal network. They have a bunch of machines on cloud provider of choice, and they’re across the public internet, or they’re like doing multiple clouds and trying to talk between them. So the idea of your security model of, oh, it’s fine, it’s just on a trusted network, really doesn’t exist. And is, I think, unless you spend lots of time building up your own pretend layer 2 private IP space in your cloud provider, then you’re going to have to use a different security paradigm than it’s inside, it’s trusted.

Matty: [00:11:21] I think, like, I remember I toured a Microsoft data center a year or two ago out in Chicago area. And one of the things, this was when Azure was just kind of taking off a little bit. So it wasn’t necessarily in that context. But the data center manager was saying that, you know, a lot of times, you know, we get the statement being like, oh, I can secure way better than you can. And he’s like, you know, after the— and this was true a few years ago, I don’t know if this stat is true now, but he said at the time when he made the statement, he said, after the US government, Microsoft is the most attacked entity on the internet. And his point was, so you think that you can, you know, there’s no better subject matter expert than someone who’s being attacked constantly. Again, like I’m imagining Windows Update has got to be one of the most juicy possible targets you could get, right?

Ben: So, yeah.

Matty: What do you think? I mean, for a lot of organizations, do you think that trust level is almost out of their core competency and they’re better, actually maybe better in a provider that has expertise?

Ben: I know maybe I’m biased, but I think as soon as you start trying to outsource your security, trusting someone else to do it for you, you’re going to be hit by a surprise. And if you don’t think about as many aspects as you can when mapping out where and how you could be attacked, then It’s going to be like, if you trust your service provider with all this stuff, yet you hand out free laptops on the street that have logins to it, then securing the servers all you want is not going to do a thing. And unless you have that complete overview of how your organization works and what your threat model is, then the person designing the security defenses will be in the dark about how it’s used. I think security vendors get a hard time, and that’s quite rightly so, as they’re selling silver bullets more than many other avenues of the technology space and have failed kind of more times and use fear way more than— like when you buy a Dell server, it’s not because you’re scared the IBM server is going to fall on you or anything. All the security products are sold with like, you will be hacked, there are people on the internet who are bad. So I think the benefit of Etsy’s size and scale and our security team being so big is we do have the time to go and build things ourselves. Netflix are building great tools. Square are building great tools just to go and fix their particular security problems that they’re facing. Um, and all those tools are ones that you can’t buy commercially.

Matty: [00:13:32] So that actually leads into kind of a DevOps-y thought. So you’re saying being able to do the stuff internally. So, and I’ve thought about this in pre-DevOps engagements or pre-DevOps places I’ve been when, again, where the idea of, well, we can do it better ourselves, even though it’s not our core competency, but then thinking about when you have a subject matter expert on this threat, right? But who’s maybe not like the tightest developer in the world. But when you can combine those kinds of things together, how does that work? I guess within, within your experience too, is like, cause you have the, you have development resource, right? You have people who are awesome coders working within your organization that are maybe not necessarily on the security team. Like how do you interact with that? Is that a resource that you can, they can help you with your tooling or is it someone that you just interact with?

Ben: I mean, I think that’s kind of like if you replace the word security with operations and you talk about just developers, it’s the same deal. I think everyone who’s had a bad time with security will turn around and go, yeah, we probably should have talked to security sooner in the development of this thing. Like the chipping it over the wall to operations is just as bad when you chip it over the wall to security at the last minute and you’re like, cool, it’s riddled with XSS or Why is the password on the front page of the website? Surely that’s a bad idea. Yeah, but it’s really convenient. Again, I’m clearly biased, but if you have money to allocate on security, spending it on a security team sooner is way better than spending it on products or say pen testing. Because until you have it mapped out what you’re defending and from whom, you can’t try and build any defenses.

Matty: [00:15:10] So I agree with, I was thinking about it even in the opposite direction, which was being able to, I mean, so obviously your development team being able to leverage the expertise of security within the applications and services they’re providing that are the business services or that are what we consider the classic business services. But then I was thinking when you’re talking about building your own security tooling within your competencies and stuff, have you reached into those resources into maybe the development folks who are not part of the quote unquote security team, but to be able to help with just some of that internal tooling?

Ben: Yeah, so the first person in the Etsy security team came out of being one of the toolsmiths for Etsy, and they took on security on the side while they were building the tools to do all our internal user administration. I think you can always find people in your operations team or your development team or your QA team who are interested in security and kind of go, hey, I’m working on this thing. If you’d like to work on this bit, Let’s team up. You can write good code. I can work out what we need to write. Let’s make lots of money.

Trevor: [00:16:15] So I’m a developer. So what are some— I’m going to ask the stupid question. What are some of the things I should start— if I’m not already thinking of, what are the things that I should be thinking of, you know, so that if I’m going to go to talk to the security team, what are the things I should be aware of going into that conversation?

Ben: And then, you know, I think as for developers, please stop turning off TLS verification. That’s pretty much my number one thing in the world. My one-man quest. It’s finding broken TLS implementations that just go, yeah, I’ve managed to talk TLS, I don’t need to check for certs. That’s such an easy win in pretty much every language now. And getting a cert store on there— like, if your operating system isn’t providing a list of up-to-date certs that are updated by system packages, switch distribution. As for, um, developers, the age-old thing of never trusting user input still. Users will find new and interesting ways to take something that you think is perfectly reasonable and turn it into an attack. There was a recent JSONPP vulnerability that involved encoding Flash objects, I think with— I’m going to get some of this wrong from memory, but something like zlib. So it came out all as valid lower kind of Base64 ASCII. So you could encode a Flash object to be reflected back, and then it would get decompressed back. And it just looks like plain text because it’s all 0 to 9, A to F, but it’s a valid Flash object. And you would never expect someone to do that. So not trusting user input still remains in web programming anyway. No, in all programming, because all programming has user input. Otherwise, it’s a really boring Hello World example.

Trevor: [00:17:47] Yeah. That is one of the things I do think of actively is not trusting user input.

Ben: Yeah. Never trust— in the nicest possible way, never trust users.

Trevor: Yes.

Matty: So I was thinking when you’re talking about not disabling TLS verification, so this is, this is gonna be my little security rant and not rant about security people. Actually, it is a little bit about some security teams. So there’s a common practice. So transparent HTTPS proxies that interrogate all SSL traffic, they do terrible, terrible things to cert verification in that all cert verification would fail when they go through these proxies. So it is like one of the biggest pains in the ass when we’re doing like a lot of this chef work, because we’ll go into enterprises and you just want to do a gem install and it’ll fail when it’s trying to verify rubygems.org because this Blue Coat or whatever thing that’s in between it is decrypting those packets and it’s blowing up the SSL cert. So what do you do? You turn those things off. I guess somewhat of a trade-off scenario, but there’s It seems to me there’s got to be a better way to handle that. And I’m guessing that decrypting SSL traffic through your proxy is not because you’re trying to protect from a security perspective. You’re trying to make sure your users are not doing something you don’t want them— your employees are not doing something you don’t want them to do. Am I understanding the purpose of that correctly?

Ben: [00:19:10] To give whomever you’ve been working with the utter benefit of the kindest doubt. They could be inspecting that traffic to make sure you’re not accidentally downloading something malicious through a link that you were sent. They could be— there are many good reasons to use web inspection. There are many more reasons to configure it correctly, which it sounds like that isn’t done. If you’re doing that, you could even just do the horrible thing of trusting the CA cert that they have on the Blue Coat in the RubyGems config so that it uses that CA cert to trust the Blue Coat doing TLS. Which then, yeah, like the Blue Coat can basically man-in-the-middle all of your TLS traffic, but it’s doing that anyway and everything is breaking.

Matty: So, and that’s why it’s breaking because it’s looking like it’s a man-in-the-middle attack.

Ben: Yeah. If you trust the cert of like, to play in this organization, I need to trust this cert so that they can inspect things, then things will not break. And you just have to hope that they take enough good care of that CA cert that you’re using because they can spy on all TLS traffic. Then like you don’t want to be running Chef client off the same thing you’re browsing your own personal bank on. Perhaps in that environment would be the argument forward there. Yes, I mean, there are plenty of ways to break TLS, but if you break it at the application level, then it’s not going to get better from there on. If you break it at the network layer, then nature always finds a way to tunnel out of an enterprise. I’ve pretty much learned.

Matty: [00:20:33] So I was bringing it up because it seems like a scenario where there’s something in place that’s intended to provide security, but it’s causing people’s response to do something that’s potentially less secure. Although I, you know, which is the— because I’ll tell you, that’s exactly what we all do. We go, okay, well then I disable SSL cert verification on all these tools because when I try to use it, it breaks.

Ben: Yeah. Or you just fire up an SSH tunnel or an OpenVPN tunnel or use Corkscrew and just bounce out the HTTP proxy or like, what’s it called, Iodine, and just tunnel your IP over DNS. However you need to get out of a network, you will. But yeah, that’s one of the things we try to do here is mix the security trade-off against the actually making people secure and make it so that they don’t have to circumvent our things to get their jobs done. Because then that’s what you’ll do. That’s what I would do. That’s what every sane person would do. You’re employed to do a job. I can’t do my job. I will find a way to do my job. So we can either work with these people or kind of be avoided by these people. And working with them seems like everyone wins. And letting people try and circumvent the things designed to protect everyone doesn’t really—

Trevor: [00:21:42] Yeah, it happens everywhere. I remember in high school, I had to figure out how to change— I had to figure out a backdoor way into the registry editor so that I could re-enable Control-Alt-Delete.

Ben: Yeah.

Trevor: Because I couldn’t access the Task Manager because it was Ctrl+Alt+Delete was disabled as a function.

Ben: Yeah. And you just find a way to get around these things. So I’d say having SSL not be able to verify it is broken by design, but I’m sure their security model took that into account and just taught people to click through HTTPS warnings, which is a really good thing to educate your users to do.

Matty: Well, and then the thing that I’ve found when these frags, just because again, having been at companies where this comes up, is that it’s one of those things that the browser can handle, like transparently somehow, because these proxies are built for that. But then when you’re using a tool like Ruby, or you’re doing some command line, you know, those are the things that where the stuff ends up breaking. And it’s, it’s interesting, because these practices are kind of not built, you know, to me, I’ve also, that’s just now getting off into a whole tangent about starting to do work in the enterprise with tooling and culture that was built around a more startup-y, open-source-y world where you expect to be able to talk to the internet all the time. You know, those tools don’t know what to do when they can’t because you thought you could, because that’s reasonable.

Ben: [00:23:05] Like, trust famously doesn’t scale, and in a company of 50 people, you pretty much know everyone’s names. In a company of 100 people, everyone knows everyone else by one degree of separation. In a company of 10,000, Like, you don’t trust other departments or entire other buildings because you have no idea who these people are. So there becomes more barriers to trust. And thankfully, we work very hard to try and continue to default trust everyone at Etsy and just monitor absolutely everything, log far more than we should in terms of like system logs, Splunk and Logstash logs, and be very transparent about what we’re doing. Um, because that way you get trust back and people trust your security team and will talk to you about things. Not having a trusted security team means, again, people will just try and avoid the security team. They won’t tell you when things happen. They won’t talk to you about this weird thing they saw that one time. And yeah, I think transparency in a security team is key in doing that and being able to justify why you’re doing these changes. Not because like, we want to run Blue Coat and man-in-the-middle everyone’s HTTPS traffic. So that we can show you’re doing nothing bad. It’s a terrible justification, but I’m sure if you were to sit people down and go, so we’ve had these things happen, they’re quite bad, this is what we’re thinking of doing, here’s how you can see what we’re doing, then people would be more inclined to trust you.

Trevor: [00:24:24] I mean, we keep coming back to— in various ways, we keep coming back to this concept that communication is key in getting anything done in a company and building trust and building relationships among your coworkers.

Ben: I think like a lot of DevOps is basically about, hey, depending on who’s on stage, it’s probably not about the tools. It’s why don’t we work together? Because we’re kind of employed by the same company. And ultimately we kind of want similar things, like the company to do well, us to ship code, it be secure, me be able to use libraries I want, not insane requirements and communication transparency. Talking to people, I still don’t see it as rocket surgery, but it seems to be quite a powerful tool in getting people to work together on things.

Trevor: So I believe what Matt was going to say is we had a question from Matt Rock who’s asking, what practices do you use for storing secrets?

Matty: So it’s probably within what, what you’re able to share. I don’t know that some of that might—

Ben: yeah, yeah, yeah. So Chef encrypted data bags are a lovely idea, but they turn your encryption problem into a key management problem, which is pretty much what all— as soon as you add encryption, you’re like, cool, I no longer have to worry about this data being secret. Oh, other than the keys, which I now need to get everywhere. And if I could get the keys there, then I could have just got the data there in the first place. So we’re looking to move past that. And Chef Vault by Nordstrom, which I think Chef actually use because I’ve seen Joshua commit and talk about it.

Matty: [00:25:57] It’s part of ChefDK actually too. So if you download ChefDK, you get Chef Vault. So we love Chef Vault.

Ben: Yeah, that’s a really awesome project. I was super happy when that appeared. So that, I mean, you still have bootstrapping problems, but like in cloud, you still have bootstrapping problems. You have to, you can’t go from nothing to everything in one go. There has to be some compromises there. Square, I don’t know if they’ll want me talking about this too much, but they did something cool involving kind of resident-only in-memory file systems using FUSE and sharing that way, which I keep kind of trying to hope them to release, but I’m not sure they’re there yet, but that sounds really cool. But yeah, sharing secrets is a hard problem. I’d say that the more important thing with sharing secrets is sharing them is great, being in a place where you know you can change them and nothing breaks. You could say, cool, you’ve had that root password for 5 years, and that’s cheffed out really, really well. If we change that password, what happens? Oh, we can’t change that password. No, no, no, we have to change that password. So, like that, I think once you get the sharing thing even slightly down, just go and randomly change some passwords. It’ll break a ton of stuff, and you’ll have a terrible day, but you’ll know next time when, say, those passwords leak, that you can do that and have faith in it. I think that’s like— it’s like game daying your infrastructure. Like, if I take out this database, what happens? Or if I randomly change the passwords in this database, what happens? Because creds leak. The information wants to be free, and nothing more so than your passwords. Pastebin is a treasure trove of other people’s mistakes, and GitHub Gists is just a shopping cart full of logins to things. So yeah.

Matty: [00:27:36] How do you, Ben Hughes, manage your secrets? You know, because this is just my own personal, like, now selfishness of just thinking about things like password managers and 1Password and things like that. So from a, not so much from an enterprisey company way, but just a personal consumer person who’s doing things, you know.

Ben: Yeah, that’s a really good question. I think we have a license for a password manager. We’re evaluating other ones going forward. I’ve been a big user of 1Password for a while. Due to having an iOS phone, I’m not a huge fan of LastPass, though LastPass actually has a bunch of better features. Not using a password manager in this day and age seems insane. I have like 600 different credentials in my password manager. They all have different passwords, most of which I no longer know, and I’m pretty happy with that. I two-factor pretty much anything I can find. Apparently that’s been in the news recently. So two-factor everything, use a password manager, have a decent password in your password manager and rely on the strength of the password manager because all the encryption systems in them are decent. So yeah, there’s not really much more advice than that.

Matty: [00:28:51] I would think like my advice or my request to front-end developers is name your password fields password.

Ben: Yes.

Matty: And don’t put them behind some weird chrome that a password manager can’t figure out how to populate. And that’s even somewhat to me, it’s even less about being able to autofill the password. But when I go to generate a password, like I’ve got applications that I’ll be honest, I don’t have my, I don’t have a super random password on because I couldn’t figure out any way to get 1Password to be able to handle it. Yep. You know, and fortunately they tend to not be something super important, but yeah, it’s the most aggravating thing in the world. Or when you’ll run into, this is my other aggravation, is like apps that are not super, like that don’t let you store your password in the iOS app because I’m like, I want to create this super random password that I’m never going to remember. And I guess it’ll be better in iOS 8 when password can talk to other apps, then I won’t care anymore. But now I’m just like, and now I have to go back and copy paste it out and paste it back into there. And it’s, I guess if I’m willing to say, I put your app behind my own firewall. So I trust that if you got on my phone, you’re me, you know, I should be able to, to say that, or don’t randomly sign me out.

Trevor: [00:30:03] You know, I’m constantly surprised that, you know, I have one of my bank accounts. You can only have, I think, 10 or it’s either 8 or 10 characters in the password.

Ben: It’s probably 8 because of the old Unix crypt length. I’m just gonna throw that out there. That’ll be why it’s 8, because anything beyond 8 gets cut off.

Trevor: It drives me crazy because they have these really weird requirements, and then on top of that I can only have 8 characters. So I personally like to— I don’t do the XKCD thing, but I kind of, I tend to do like phrases.

Ben: Yeah, DiceWare is a nice website where you get 6 playing die, uh, you roll them 6 times and you get a password that’s just made of words that will take longer than the heat death of the universe to crack.

Trevor: Right.

Matty: Yeah. I was going to say my 1Password password is, yeah, it’s like a 6-word diceware phrase that, and that’s because of that. I’m like, I don’t care about my other stuff. And I actually, I think it was one of the, the CEO of 1Password or something basically was like, here’s how you can handle it. And like, cause it was hard to learn, but the way I learned it, which was from like one of his blog posts is I set 1Password for a couple days to prompt me for my 1Password every single time.

Ben: [00:31:12] Wow.

Matty: You know, so it’s locked. I carried it in my wallet for those couple days because for the times I forgot while I was learning it, but it was like, now today, I’ll never forget that phrase, you know.

Ben: As someone who’s forgotten their own GPG passphrase and has had to write a script to brute force a chunk of it, I know that pain. There needs to be, I need to use GPG sufficiently frequently or the useful information drops out of my brain. Yeah. I don’t know.

Matty: Yeah, I actually had to set my 1Password to prompt me more frequently now just by virtue of it prompting me so infrequently, I’m starting to get scared that I’m going to forget it. So I’ve gone back to that it’ll only cache it. Like, I don’t know, for a while I was having it every time I came back from screensaver, it would prompt, but then I was like, Anyway, do not emulate me in your security practices. What do you think are some, along those lines, what are some common misconceptions about security, both maybe from a consumer perspective or just like someone working in technology? What do people think they know, but they don’t?

Ben: [00:32:17] Yeah, I’d say there’s the whole thing of trying to HTTPS everything. If anyone says that HTTPS isn’t fast enough, or like it’ll slow down my website, they should go to istlsfastyet.com, which is by— oh, I’m going to mispronounce his name— Ilya Grigorik, who’s at Google, who’s been doing a lot of the TLS engineering on their infrastructure. It’s one of the fastest loading pages I’ve seen on the internet. It’s over TLS. There’s all kinds of features in TLS to stop the fears of it being slow, so session cookie use session tickets, like sending data in the first packet in the handshake, like fitting your certificate records into the record size of the HTTPS transaction. The arguments for why not to use HTTPS are pretty much done, and I think everyone should just use HTTPS by default. All the big popular sites are now. It’s starting to seem out of fashion to have an HTTP site, which I think is a good thing. What else frustrates me? I think this common, certainly in the media to do with security, there’s been lots of alarming posts about zero days, which is security vulnerabilities that have just been announced and there is no patch for. So there are zero days between when this bug is patched and when the exploit is out there in the wild. And they’re all kind of scary and attention-grabbing headlines. And they The price for iOS ones is in the hundreds of thousands of dollars if you’re a nation-state looking to attack someone’s iPhone. But like 99.99999% of organizations do not need to worry about those. They need to first worry about the 300-day. Like the fact you haven’t updated your Linux kernel since you installed the machine is more of a worry than worrying about some zero-day because there’s all kinds of Are there exploits that will work that you haven’t patched? And the fact that most users’ passwords are like 4 characters long, way more bigger thing to worry about than like this brand new hot off the press exploit developed by a whole company whose job it is to write exploits. So as usual, media fears over things that aren’t real, story at 11. Yeah.

Trevor: [00:34:29] Speaking of the fear and not trying to lead with fear, how can we get security started? In a company if there’s no security team? How do we bring that to the foreground without being like, oh, if we don’t have a security team, everything’s going to blow up, we’re going to lose all your passwords, and the database is going to get deleted?

Ben: Yeah, that’s a good question. I think Etsy has been super lucky as our CEO, who used to be CTO of Clicker, Chad Dickerson, is very aware that security is a thing that needs to be at the forefront. We have a security team with more attack capabilities than a whole bunch of boutique pen testing firms. So that’s a pretty nice place to be. But I think in smaller companies, you can find there’s always someone who has spent too much time staying up late on IRC, talking to people in sketchy channels and like has an interest in security. And I think just, just allocating them time and realizing from high up in the company that security is an important thing. You need to pay as much heed as you would other important things. Like, yeah, backups are important. Get your backups in order. Security is important. So allocate some time and some people and some resources, some money to security. And like day one, you’re not going to make it secure. And despite what numerous white papers and angry people on Twitter will say, it’s all or nothing. Like, some security better than no security. Having a password better than no password, having 2-factor better than having just the password. So smaller incremental gains is always the way to do it. You’re not going to move from having no security people and no security to being like Google Zero insane squad of hackers and former hackers doing incredible things. So setting realistic expectations in both directions of we need a security team, And it’s important, but there’s no way we’re going to have that instantly. Because otherwise you’re just like, it’s boiling the oceans impossible. There’s no way we can ever do this. So we should not do it at all.

Trevor: [00:36:31] I’ll ask the more personal question is, how do you sell that to customers? Do you have any thoughts there about, you know, when it comes to selling software development, most companies want as cheap as possible. And, you know, security is, you know, often one of the things that gets pushed aside because, oh, something else will take care of it. No one’s going to bother attacking us.

Ben: Yeah, no one bothers until you read about it in some journalism of some kind, do you? Or you don’t find out about it until you read it on Reddit or where have you. How do you sell it to people? I think there’s enough of it in the news, certainly at the moment, with the recent iCloud break-ins saying we should probably care about this. This affects real people. I imagine people like Target and Home Depot are suddenly getting the last remaining coffers of their company allocated to their security teams. And I’m sure just explaining you have the option of there’s this big breach, we’re in the news, we have no good story versus, yeah, we started our security team 6 months ago. We feel it’s really important. This is a key part of what we do. I think especially for cloud venues, cloud vendors, and something made up as a service vendors, that having security in there is part of the due diligence companies would do when choosing the platform. Much the same way Amazon had that kind of scary period where US East was running off of like batteries for a bit and no one would put anything there. I think finding— I’m sure when Linode had their big break-in they lost some customers. And people were like, yeah, we’re going to go somewhere else. So I think it does affect real bottom lines, and definitely of real companies. Although Target stock price actually went up after the breach. So I could be talking utter rubbish. But I don’t know, I think you can do it not out of fear, but out of this is the responsible thing to do. And customers, we are more likely to get customers if we are more secure than our rival company. Like you wouldn’t go to a bank that had HTTPS online banking. At least I hope no one in tech would, no matter how slick the interface. You would demand like, there should be a padlock somewhere on this page so I know it’s trustworthy. Simple wouldn’t be doing as well as they are if they just had HTTP-only banking.

Matty: [00:38:46] That just made me think of something. So I have no way of knowing on a mobile app whether my communication to my bank’s web service is encrypted. All the people who’ve been trained to look for the little lock Right? We all seem to have like— there’s no padlock on my iOS app.

Ben: So there’s been a recent report about the number of Android apps that don’t actually do TLS verification, to the extent they made an Excel spreadsheet listing all the apps so you could conveniently go through it. I think genuinely the platform developers and Android and iOS need to make it a— if you are using secured channels, something is displayed in the status bar. And that is only displayed if you’re doing like perfect forward secrecy, if you’re validating certificates, if you’re using the correct sandboxes, if you’re using the secure storage on the device. And only then will this padlock icon come up in the system bar or whatever it’s called on those things. I’m clearly not a mobile developer. Just so people start getting some confidence in those problems in that way.

Matty: [00:39:53] I think what just scared me is it’s taken me this long to realize that’s not a thing.

Ben: Yeah.

Trevor: You know, so surprisingly for me, I thought I had seen the padlock in the— I only really use one secure application that’s not Google on my phone. And there is a padlock. I mean, it’s, it’s an application. So it could be, you know, it could be bullshit. But the padlock is there. I mean, it’s, it’s Bank of America. So I would hope, but I’m really hoping you’re going to send us that list.

Ben: Yeah, I’ll do that. So I recently tried to play The Settlers online and I took a picture of their, this is secure, here’s a picture of a padlock next to the URL, which was HTTP. And just like everything about this is UX fail, it’s UI fail, it’s security fail.

Matty: Right.

Ben: This is amazing. And like the submit might have been to an HTTPS endpoint, but you had no way of knowing that. It was using Flash as well. So unless you downloaded the SWF, decompiled it, looked at all the URLs it addressed, you had no way of knowing that this was legitimately secure. Suffice to say, I did not play Settlers that day. And like, my banking details does not go over Settlers. I just was so offended by the lack of security that I just walked away.

Matty: [00:41:07] So I have a silly question, something I realized as I was starting to write questions together and things. So I tend to refer to like the security department of an organization as InfoSec, and I wonder if people still do that. And if not, there is the problem that it reminds people too much of the band Information Society. So I don’t know, is that the right term? Should I still say InfoSec?

Ben: It’s still called InfoSec. You still get CISOs. I hadn’t heard of that band until you mentioned them, but they look pretty amazing. So I mean, all of our titles here which are completely made up and pointless are all like security something. I think my LinkedIn said Security Monkey for the longest time. Yeah, CSO or CISO, depending on whether you want information in there or not, they’re all valid. Does anyone care about titles in like DevOps other than the title DevOps?

Matty: Yeah, other than don’t be called DevOps and having the word— I was just, I was wondering if it shows a certain, not age, but just certain thought of an era when that was Because I feel like I don’t hear— I mean, definitely people talk about information security, or like you said, a CISO or something like that, but sort of that—

Ben: [00:42:14] I think it’s still a term. Do you say sysadmin, site reliability engineer, or operations engineer? This is how you gauge your age.

Matty: Yeah.

Ben: Okay. That puts you in a certain age bracket. Do you remember the hacker versus cracker arguments?

Matty: Oh, yeah.

Ben: Yeah. And that one’s done. Like no one still says cracker. It’s like, that’s horrible. You lost. It was way more clear-cut than the GIF versus GIF.

Matty: Yeah, that one. I have, I have feels about that.

Ben: Yeah, exactly. Whereas no one cares about the hacker cracker ones.

Matty: I have one more question just along those lines. I’ve thought about this too. So, cause we’re talking DevOps, so it’s kind of funny. I was going to say, since we’re talking DevOps, I have to ask about this, even though this is going to be one of the first times this has been brought up on the whole podcast. So The Phoenix Project, did you read The Phoenix Project?

Ben: I did, yes.

Matty: Okay, so what are your kind of thoughts on the— I’m rereading it now, so it’s fresh in my mind. The character John, so the security manager who’s like, has a nervous breakdown. I’m sorry, spoiler alert if anybody hasn’t read it, but yeah. I mean, what were your— because I know I kind of, as an ops person, went through the kind of PTSD in the beginning of that book of like, oh my God, this is too much like a life I’ve lived. But someone who, you know, is part of the security part of an organization or done those things, how did you kind of feel about the fact that, to me, that character was kind of put on as almost a villain for a while in the book? I mean—

Ben: [00:43:36] The first part of the book is very stressful. Not one to read if you’re like having a hard time at work because you’ll just, I think, close down. Should definitely come with a trigger warning on the front. I think a lot of security organizations are seen as this kind of Big Brother or authoritarian, unfriendly, unapproachable team. And I’ve certainly spoken with other companies where they’ve gone, I wish we had your security team. That sounds great. A few months ago, our director of security put on a hackers party where we watched the classic film Hackers. We dressed in a ’90s style. We spray-painted some floppy disks that we managed to find, though not the Doom ones. They were too awesome. And generally, and then ate Chinese food out of cartons. And everyone bonded over the film Hackers. And that’s how our security team is. We’re not overarching people. We’ve gone and taught various members of our organization how to pick locks. We have lunch and learns on, and we had a whole security hack week teaching people like different parts of hacking, different parts of security, all this other stuff. And yeah, the really horrible, you did this bad thing, so I’m shutting down all your access. Security teams exist. And I think For a lot of people, that is a real-world portrayal of them. And it does sound a horrible place to work. But there are many parts of The Phoenix Project where you’re like, this is a horrible place to work. I think it was Gene Kim getting all those issues he had working at many places off his chest and just be like, this one bad time at this bad company. So yeah, if you work somewhere like that, then there’s never been a better time to look at the computing job market and go and find somewhere that doesn’t do that. If you are that person, then start being nicer to people because you work with them and they have to spend 8 hours a day there too. So, but yeah, but that’s a definite thing. One I’ve not had to deal with in a long, long time, thankfully.

Matty: [00:45:25] So before we move into our checkouts, are there any thoughts you want to share? You know, things that you think people should keep in mind that we haven’t covered around the ideas of InfoSec, as I’m still going to call it.

Ben: No, that, yeah, that’s, I would say that security side of it needs to buck up their ideas too. I think they’re very— because a lot of the good incident response stuff comes out of military research and security has come far behind that. There are lots of terms borrowed from military jargon, which does make it seem very aggressive. People playing soldier at the weekend boys club, which I think is harmful for getting people involved. The phrase threat intelligence. Just brings me out in a rage of like, no, it’s just some bad stuff you know about some people. Like, one of those terms just says, I want to wear camo and run around a forest. The other one is, yeah, I’d quite like to do my job. And being less preachy about there is only 100% security or there is no security, it’s definitely not that clickbait. It’s not in any other part of IT. I don’t see why it should be in security. You can never be absolutely secure. CPUs have a list of bugs. If you’ve ever seen a tech sheet of an Intel or AMD CPU that lists all the errata and all the things broken on the CPU, computers are broken from the ground upwards. And that means that there are security vulnerabilities from the ground upwards. And you’re not going to fab your own silicon. So defense is in some ways harder than offense. And the offensive people are super crazy smart, but it doesn’t have to all be doom and gloom.

Trevor: [00:46:56] I think that’s a really important nail. I mean, so I’m a diabetic and I’m on the insulin pump that has a wireless device on it. And sure, there is absolutely a way somebody could get in there and figure out a way to give me insulin, but the benefits of using the device far outweigh the likelihood that I’m going to stumble across the person who’s going to hack the device.

Ben: The research done by the late Barnaby Jack now means the FDA are forcing people to try and do security in some of these devices and at least have some authentication so that in some horrible syndicate dystopian future, people aren’t just having terrible times with pacemakers and insulin pumps because of this. But a more worrying thing is now Windows XP has been end-of-lifed. If you’ve ever been to a hospital, look where all those Windows XP boxes are. Running things like MRI machines. Yeah, it’s a scary feature in that regard, but that’s just Windows XP.

Trevor: So awesome. That was great. We’re gonna start with our checkouts. Matt, do you want to switch things up and go first this time?

Matty: [00:48:04] Sure, why not? So my first checkout is a total I’m late to the party, whatever. I’ve finally been dragged— not dragged, but finally got around to switching my shell from Bash to Zsh, or Zsh, or however you pronounce it. At because all my coworkers do it and they have prettier shells. But also, the one thing that was really quick to me that I thought was super interesting was the first thing that I did after I got it going and after I installed my second checkout, which is Oh My Zsh, which is a whole bunch of customizations for, for Zsh. I was like, well, I have to go back and add all of my aliasing that I have in my bash profile and discovering that out of the, you know, 20+ aliases that I had, I only had to keep one of them because everything else I had already customized in my bash profile was something that either— that was basically already in Oh My Zsh. So I’ve been digging it. I’m still gonna, you know, spending way more time than I need to coming up with the most absolute perfect theme that I want to use. And there’s also an Oh My Zsh plugin that I’m going to put a check out to. I’ll put a link in the show notes, but my coworker Sean Carroll wrote a plugin to add emoji into Zsh. Which is super necessary, of course. Then also, I’m going to provide— my other checkout is Chef 12 release candidate. So this is my Chef plug. Lots and lots and lots of Chef news this week. Chef 12 has gone into release candidate. One of the big bits of news with this too is, as Adam Jacobs said, there is one Chef Server and it is open source. So Chef Server is completely open source now. It is the exact same product and we just are selling some premium features on top of that, which we think you’ll love. There’s a lot of good advantages to this. It’s being not a separate product, we’ll have new features faster, hopefully. And if you want to switch these premium features on and off, you don’t have to completely reinstall your Chef server, which is great. And the latest Chef client also now has Windows DSC, Desired State Configuration, capability baked right in. So it’s an exciting time for the Chef. And finally, this was just a little thing that I found on Etsy that I thought was really cool. Amusing, which is a prayer candle featuring the Saint Tom Waits, who is one of my favorites. And as my friend Evan says, I don’t trust anyone who hasn’t gone through a Tom Waits phase in their life. So Ben, what do you got?

Ben: [00:50:27] I’m worried now. I don’t think I’ve gone through a Tom Waits phase of my life.

Matty: Well, you have time.

Ben: No reason. Yeah, yeah, that’s true. Hopefully not over yet. In New York, coming up next month, there’s a cyber symposium done by NYU at NYU Poly, which is trying to help and introduce women of all ages and at all points of their career to security. The URL for that is a mouthful of cybersymposium.isis.poly.edu/symposium. If I was a cleverer person, I would have used Bitly.

Matty: We’ll put a link to it in the show notes too, so you’ll be able to find it.

Ben: And that’s a really good event of trying to bring both college-level and midpoint career people to all the different areas of InfoSec. There’s a lot of great speakers and it’s proving to be a really cool event. On top of Oh My Zeesh, I’m going to say Prezto, P-R-E-Z or Z depending on your locale, T-O, which is an Oh My Zeesh alternative that probably comes with— I don’t know, it looks kind of flashier, but Oh My Zeesh people have moved on to that. So you’re still behind the times, Matt. I’m sorry. Another cool tool is Gauntlet. Which is by James Wickett of Signal Science and Matt J of White Hat Security. And it’s a set of CI/CD things that you can throw your code and your infrastructure through per build to try and do horrible things like check SSL, run CSS attacks, all that against your code. So from the development and operations side, every angle of DevOps is covered, trying to make your code and releases more secure. And I would say 2factorauth.org. So you can go and find all the places you can enable 2-factor authentication, including iCloud if you’re willing to wait 3 days. That’s my real short list. Cool.

Trevor: [00:52:11] All right. I had to figure out how to unmute my microphone. I made a special keybind on my fancy gaming keyboard, and of course it didn’t work because I didn’t have the window focused. So this week I’m going to recommend this.

Matty: How do we even computer at all?

Trevor: I know, it’s, it’s amazing. So I’m going to recommend this thing called StrengthsFinder. It’s kind of— it’s like a better Myers-Briggs test. It was recommended to me by Kay Johansson at FlowCon. And once you take this test, it gives you a really interesting analysis of your strengths as a leader and like how you can use those strengths to lead more effectively, which was very interesting to kind of go through. Then Amazon Instant Video is finally available on Android. There’s no Chromecast support yet, but if you’re on a Nexus 5, screencast works just fine and looks beautiful. So Amazon, I don’t have to switch to Google Play Music or Google Play Video or whatever the hell it is because you’re pissing me off that I can’t use my Chromecast. And I can finally retire my PlayStation 3.

Ben: [00:53:20] Um, it’s a terrifying place as well.

Trevor: And finally, I want to recommend— there’s, there’s a— I don’t know how new it is, but to me it was new— new board game. It’s called Rampage. If anybody’s familiar with the classic arcade brawler game where you destroy buildings and eat people, it’s now a board game and it’s a lot of fun, and I recommend you check it out. We’ll put a link in the show notes.

Matty: Sweet.

Trevor: I don’t know who’s supposed to talk next. I think it’s Matt.

Matty: I’m so professional. So anyway, we have a newsletter, arresteddevops.com/bananastand. It is the best way to know about upcoming podcast episodes. And sometimes we even talk about cool news with DevOps. And speaking of cool news with DevOps, O’Reilly Velocity Conference is next week, September 15th through the 17th in New York City. It is the place where DevOps, WebOps, and performance professionals Gather for a legendary learning and networking experience that explores why a faster, stronger web is no longer an option, but a necessity. You can hear from the best speakers in the industry, and if you attend Velocity, you will view your work, the technologies you use, and your organization in completely new ways. Use the code ARDEV20 to save 20% at arresteddevops.com/velocitynyc.

Trevor: [00:54:38] And thanks again to our sponsors, PagerDuty and TrueAbility. And to our loyal listeners, if you enjoy Arrested DevOps, we’d appreciate it if you visit arresteddevops.com/itunes and leave us a review in the iTunes Store on your fancy new iPhone 6s and 6 way too bigs. And we’ll read our favorite ones on an upcoming episode. I also want to remind everybody that the Chef Community Summit is coming up and that DevOps Day Chicago. Check both of those out. If you check the show notes, there’s the coupon. Codes available for those conferences as well. Thanks, Ben, for joining us today. It was a really awesome talk. And be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter.

Matty: We are always happy to get your input, ideas, or feedback at shows@arresteddevops.com. Amusingly, we actually just got an email from someone who asked if we were hiring. That was kind of fun. I, I am matt@mattstratton.com.

Trevor: And I’m Trevor @TrevorGHess.

Matty: We are Arrested DevOps, and remember, there’s always DevOps in the banana stand.