Matt: [00:00:00] The last release and the first release of the year are always like the 2 worst.
Bridget: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and business for maximum DevOps awesomeness. I’m your host, Bridget Kromhout, @bridgetkromhout on Twitter. This episode is brought to you by Datadog, a monitoring tool that helps bridge the gap between operations and dev teams. Datadog brings together system metrics, changes, alerts, and events from over 70 common infrastructure tools such as Chef, Docker, and AWS so that dev and ops teams share their key data and alerts in a single place and collaborate on issues in real time. Datadog’s available for a free 14-day trial at arresteddevops.com/datadog. All right, we have 2 great guests today. First, we’re going to welcome Rob from Nordstrom. Rob, please tell us a little about yourself, your role there, Yeah, sure.
Rob: [00:01:00] Rob Cummings. I’m at Nordstrom. I’ve been here 10 years now, which is a long time, but they let me move around every couple years, mostly system admin roles. Recently moved into leadership, and today I support— since January, I support the operations teams for nordstrom.com.
Bridget: Nordstrom.com does sound relevant to people’s scaling for the holidays shopping interests.
Rob: Yes.
Bridget: All right, awesome. And then we also are speaking today with Matt Curry. Matt currently works for Allstate, but he spent a great deal of time scaling infrastructure at PayPal. So Matt, do you want to tell us a little bit about your background, what you’re doing now, and why you think that this is a— scaling for holidays is a good topic for you to talk to us about?
Matt: Sure, absolutely. So my name is Matt Curry. I work at Allstate as the Director of Platform Engineering. At Allstate, we’re responsible for for deployment and management of platform as a service, as well as a number of other products and services that we market internally to developers. Before I did that, I started here in March, before I did that, I spent 8 years at PayPal, so 8 delightful holiday seasons of scaling. Definitely some lessons from the trenches that I look forward to talking about during the episode.
Bridget: [00:02:24] All right, that’s fantastic. So I like that you say delightful, because, I mean, we all think that holiday is a time of merriment, and also possibly terror, your pager going off a lot. But it’s a little bit different from the unexpected disaster, right? Because you know that there’s a spike coming. You probably know when it’s going to be. Can either of you, both of you talk together, separately about how you deal with a giant surge of traffic that is a known quantity? You know it’s coming, you have some idea of when. How is that different from just dealing with the giant disaster type scenario?
Rob: Yeah, so we know we have 2 major spikes in the year. One is our anniversary sale, which is in July, and the other is Cyber Monday, and then elevated load, you know, throughout the holidays. What we do for our website is make sure the teams know that, by the way, the holiday traffic this year is going to be way more than it was last year, so make sure you test. We, throughout the year, will test for our next peak, although we’re adjusting that this year. Traditionally, it had been, it had been our next peak. So, um, you know, after our anniversary sale, we would start testing for Cyber Monday peak. This year, we kind of realized that, well, since Cyber Monday is actually a bigger peak than our anniversary sale, we’re not giving ourselves a lot of time to really get all the kinks out of the system. Um, so now going forward, we’re going to be testing for the next peak of the year, um, which will be Cyber Monday traffic.
Bridget: [00:03:57] But so how do you even know When you’re telling people, oh, our traffic is going to be this much, like, how are you even predicting that number? It’s like a Magic 8 Ball says people are going to want to shop this percentage more.
Rob: It’s more magic than I would like to admit to. So we have forecasts that we work with our product management with, but typically it’s pretty, it’s pretty predictable now. So it’s a percentage above last year, and then we put in a kind of safety margin on top of that. So for the last last couple of years, we’ve been pretty successful doing that.
Bridget: So Matt, if you were making those kinds of predictions over 8 years at PayPal, like how did the predictions change over time? Did it become pretty obvious when the peaks were going to be, or how did you work on that there?
Matt: Yeah, it did. It was very obvious. We could see always, you know, Cyber Monday was a big peak and then we would see the second Monday in December, which eBay would call Green Monday, is also a peak. Started to be out as roughly equal size, although as time evolved and Cyber Monday became more of like a cultural icon, so to speak, it started to see bigger and bigger volumes in comparison. Generally, those 2 days were by far PayPal’s biggest days of the year. Then you would see a smaller spike in March from people listing like all the stuff they got during Christmas that they didn’t want on eBay.
Bridget: [00:05:29] It took them until March to figure out they didn’t want the stuff?
Matt: People are procrastinators. I can share the, you know, I can sympathize. And so we could really see those. And what was interesting, I think, at PayPal that was different is the capacity team was really in charge of kind of the payments forecast. And they had it nailed down through a number of statistical processes using R and using different algorithms for forecasting. We would basically know, you know, within a 5% margin or so exactly what our volume was going to be for every week of the year. And so we would really start, we would have a pretty good understanding of where we’re gonna be going into holiday around the March timeframe. It really, once we hit March, April, May in that range, it really didn’t move much from there. So that’s when we started talking about it. And then of course, execution and decision-making takes a little longer, but we were always kind of out in front and planning And it basically was all we thought about all year long.
Bridget: [00:06:32] So, Rob, you were mentioning for the anniversary sale that would sometimes inform things for you, and that’s the middle of the summer. So does that mean that you had to start the holiday planning before the anniversary sale for the holiday after the anniversary sale? Like, how did the rhythm of that work?
Rob: We used to not do that. We used to just plan for anniversary in the summer, and then immediately after that start planning for Cyber Monday.
Bridget: That’s what you were saying, but then that didn’t work so well.
Rob: It doesn’t work so well. So this year we found ourselves not behind, but not as ahead as we wanted to be. And we know our forecast for Cyber Monday, but we don’t know how changes to our site are gonna impact the customer behavior, or necessarily what promotions we’re gonna put on the site until closer to the last minute. And we’ve made a lot of changes architecture. So now we’re, you know, we have an on-premise portion of our site, and then we have— we’re moving to public cloud, and that’s, that’s taking time. So it really kind of complicated our planning a little bit.
Bridget: [00:07:40] Yeah, sure, absolutely. So given enough warning, you’re saying that you can have some amount of planning for this, you know, on-prem plus cloud plus everything else that you’re integrating. Do you find that Like, is this the sort of traffic that you can actually model, accurately model your infrastructure’s performance for ahead of time? I mean, do you end up needing to pipe some production traffic, like mirror your production traffic into your test environments to see how it’s going to behave? Can you simulate that kind of traffic? How do you do that?
Rob: That’s a really good question. So we have 2 methods. We do model the traffic, and the models are different for Anniversary versus Cyber Monday. Just for example, Cyber Monday is more anonymous. Missed shoppers, whereas Anniversary, it’s registered checkout, so it hits different systems. So those models we have, we will generate the traffic in our internal lab. What we also have been doing is testing in production, because what could go wrong? And what we do is we, we will throw that production load straight at our production systems during a non-peak hour, carefully watch the, you know, ramp up the load, watch our systems, and then immediately stop the test if we hit a breaking point, but it gives us a really high degree of confidence that production is going to behave the way we think it will.
Bridget: [00:09:03] But you can only— I mean, can you do this for things that affect the actual data store? I’m guessing this sounds like you can’t do writes, right?
Rob: Yeah, so the thing we can’t check is register checkout. We can flag a test order, but register checkout or signing up for new accounts, that kind of thing, we don’t have that capability yet in production.
Bridget: So Matt was at PayPal and then of course at Allstate. Do you find that simulating traffic works? Do you end up needing to send production traffic through infrastructure in order to test it? Like, how does that work in either of the environments that you’ve discussed here?
Matt: So what we would do at PayPal is we had a program that we instrumented about halfway through my tenure. We called it kind of the Holiday Canary Program, which is that we would do something very similar in that we would identify basically applications that we saw as high risk, and we would flag those based on how their response time changed and based on the throughput. So if your response time increased very dramatically as throughput changed by very small amounts, then you would get flagged as like the bad list. And so then we would run these canary tests where we would basically edit the send more load either by taking nodes out of traffic or by changing load balancer settings to send more load to a particular host and see how that host responded. But as you said, testing backend systems or shared giant backend shared resources is always the bigger challenge.
Bridget: [00:10:44] Yeah, all I can think of when you’re talking about these canaries for holiday testing or holiday canary, I’m like, this would make a really dismal, terrifying, all the birds of Christmas. It’s like I’m imagining all these canaries just like, partridges in pear trees attacking you. So we’ve talked a little bit about the technical prep, um, but what about the cultural prep? Like, how do you get people on board with the fact that this is a major thing? Like, does everybody already just understand that this is something that you have to care about, or is that something that you have to do any kind of selling internally to say, hey, there’s this giant peak coming up, I’m waving a flag, we should care?
Rob: Yeah, everyone knows about Anniversary at Nordstrom, and they care about that greatly. Cyber Monday, there had been a culture of, oh, we just got through Anniversary, so Cyber Monday will be fine. And this year, we had to do a little bit of waving the flag that, you know, we should really make sure we’re actually gonna be fine. And so we’ve done that and are looking forward to it, but it did require a little bit of selling and kind of looking at how we plan for next year. But yeah, for the most part, it’s gotten better. We made a change this year. We used to do performance testing at the— horrible. We did it at the very end of a dev sprint right before we released, which is not good. Terrifying. Yeah. And so we would find all these problems, and is it a problem with the perf environment? Is it a problem with the new code? Who knows? Well, we’re going to release anyway and find out. So we stopped that because that’s a horrible thing to do and now do not have performance testing as a gate and have put the expectation on the engineering teams that as you’re writing this code, make use of the Perf Lab if you feel it’s high risk and test it. If it’s a really complex kind of big feature that’s going in, we’ll still do a full-on performance test, but we’re putting more of that on the devs and they’re taking that ownership and making it better.
Bridget: [00:12:47] Indexes on queries, totally a thing still.
Rob: Totally a thing still, I know, right?
Bridget: Nice. Okay, so when you’re talking about not wanting to gate, like, I’m seeing Matt kind of smiling here, and I’m guessing that this is something that has happened or is relevant to your life. Like, how do you prevent people’s desire to move quickly from, you know, also like allowing them to let important things like, say, putting an index on that query fall by the wayside?
Matt: So from the database perspective, the database team at PayPal had pretty good discipline around being able to see new queries and changes to the schemas in the staging environments and being out in front of that and understanding who was doing it and what the implications of that might be on production through running SQL explains. And I probably can’t fully articulate the magic that went on there, but they were really good at kind of knowing what was gonna happen kind of before it hit production and getting out in front of it and being sure that they were providing the right level of consulting. The other thing is we had a lot of discipline on being able to wire features off. So at PayPal, a feature almost never went in like live. It would go in off and then be turned on over time.
Bridget: [00:14:17] So feature flags.
Matt: Yes. Yeah, feature flags are awesome, and they can be terrible if you don’t manage them well.
Bridget: I gotta ask about that because I’ve definitely heard before the, oh, we shouldn’t do feature flags because night capital, dark code is terrible, think of the risks. Like, how do you deal with people saying that?
Matt: Um, I don’t know. I think, uh, clean up your code, be a responsible software developer. Ethics in software development is a real thing. So Bob C. Martin has a bunch of really interesting talks, just more about the true ethics, but still, cleaning up after yourself, be a good software developer. And then you get into weird situations with feature flags every once in a while. We had an issue one time where a feature actually started corrupting cookies. Which, turning that feature off doesn’t actually put things back to normal. What? So it’s not a magic solve. You can still have issues, but for the most part, when you’re dealing with large distributed stateless systems, just turning the thing off is one way to deal with the problem. And the faster that you can restore service to the customer, even if something’s broken, the better off you are. Ultimately, failure is always going to happen. You just want to make sure the customer doesn’t know it’s happening.
Rob: [00:15:47] The other thing we started doing in addition to feature flags is, as we’re moving more and more services to public cloud, is we will just send a portion of traffic there to test it. We have for sure caught performance issues. We routed 10% of our product page traffic to the new infrastructure. Everyone else went to legacy, compared the metrics there, and discovered, oh, performance is not as good. Good, and add-to-bag rates were lower. So we were able to scale it back to 1%, enough traffic to be able to do testing. They quickly diagnosed their problems, shipped a new fix, ramped the traffic back up to 50%. They were really confident they had fixed it. And suddenly, we were able to show the performance was better and add-to-bag rates were what we wanted. Having that variable, that slider is handy.
Bridget: And you, Rob, you’re bringing up, I think you’re alluding to at least, I think what’s a really important practice, which is that you have metrics around your business logic. You have metrics around the actual value that something is providing, not necessarily just how many instances are running in public cloud or what is the, I don’t know, what is the milliseconds that something is taking? Like what is the response time on this API? Endpoint. Those are certainly valuable pieces of information, but they aren’t necessarily something you can make decisions based on without having to dereference things in your head with unwritten knowledge. Can you talk a little bit about building out that kind of metrics or monitoring environment?
Rob: [00:17:30] Yep. This has been a culture change for us over the last couple of years. It’s around, like, hey, it’s the customer experience that’s most important, not how fast our backend server time is necessarily. I’m sure there’s a link there, but And we’ve put a lot of effort into real user monitoring. So what does the client view of our site look like? And we have definitely uncovered problems and anomalies in our system that we would have never caught any other way by watching that. So we can see the performance of all our browser sessions, all our client browser sessions near real time and make decisions based on that. But yeah, knowing, knowing if we’re making our customers happy is what’s most important.
Matt: So Rob, I know at PayPal when we would look at, you know, client time, basically like the user experience, we could actually see measurable differences in conversion ratio or like cart abandonment based on those times. And we had a pretty good understanding of what that looked like. Do you guys see the same thing on your end?
Rob: [00:18:37] We do, for sure. If it’s a bad enough prop performance issue, then we for sure see it. Otherwise, it’s pretty small, but definitely there. What’s kind of interesting when we’re doing client-side monitoring is, or I found interesting, is we’ll see our performance degrade at night when people go home to their slower internet connections because we’re getting it from the client view, and then it’ll improve again in the morning when people get to work. When they start shopping from their better connected internet at work.
Bridget: The shopping doesn’t cease day, night.
Rob: To Matt’s point, there is definitely a link between performance and conversion rate.
Bridget: All right. So, when you’re talking about— obviously, you have to make the cultural shifts to get people to understand and respond to and then make changes in their code based on things like, hey, this is actually affecting the bottom line. What about the much-ballyhooed code freeze? I have worked in— I worked a couple of years ago at an ad tech startup that did the dreaded third-party cookie stuff on retailer sites. Some of the retailers we worked with were really big on this whole, oh, it’s— my watch tells me it’s September. You’re not going to ship changes to your SaaS, are you? We’re like, yes. They’re like, no, no, no. You shouldn’t change anything. Can you address at all how you deal with that, say, in Nordstrom’s environment?
Rob: [00:20:09] Yeah, so the answer is going to be, it depends. On our legacy systems, our classic systems that were not built with continuous delivery or the idea of being able to ship a change whenever you want, that was not— they did not have that in mind. Shipping code to those systems is problematic. Those, for sure, have a freeze still, and there’s a little more rigor around if we do have to ship a fix, approving that. Obviously, if product management needs a fix to go in for holiday, it’s going to go in, but we’ll put some rigor there. Our new systems that we’ve built out on, say, public cloud, not subject to those same freezes. Those systems, we can deploy code whenever. Obviously, we want the engineering teams to know that, hey, this is a really important time for us. But we don’t freeze those systems.
Matt: [00:21:14] At PayPal, we definitely did it, and we called it moratorium. We had moratorium day, which is the day all of operations threw a giant party because nobody was going to be shipping code for the rest of the year, which was pretty awesome as an operations person. It was a day to be celebrated because the last release and the first release of the year are always the 2 worst because the last release, everybody’s trying to get their feature in before code freeze. And then the first release, you have like all these pent-up features that have been accumulating, you know, since Thanksgiving that like go out right in the beginning of the year. And it’s like, oh my gosh, crap is blowing up everywhere.
Bridget: Which kind of says something about small batches.
Rob: We have the exact same behavior. Yeah.
Matt: But I think as a payment processor, PayPal has, you know, merchants that rely on them to provide a certain level of predictability, especially going into the holidays. Since, you know, we experienced the holiday peak because all of the merchants experienced the holiday peak. And so there’s some expectation on the merchant level that we’re not gonna go change things underneath them unless it’s like a critical fix.
Bridget: [00:22:26] So you’re not gonna make breaking changes to the API, that sort of thing?
Matt: Correct. That would be bad during holiday.
Bridget: But there is also a difference, right, between releasing code and making a breaking change. I mean, at least in theory.
Matt: Yes, there is. But when you deal with— I mean, yes, there is. And when you deal with complex, you know, distributed systems, I think a PayPal checkout last time I was there touched like 80 services or something. You never know how those things are gonna interact with sometimes the most minor change, especially when you put it in production. Because no matter like how much testing you do or fake load you throw at something in stage, there’s always, oh, we didn’t realize this person was gonna have 10,000 addresses, and apparently like that’s a bad thing. Or like some obscure anomaly.
Bridget: Wait, someone had 10,000 addresses?
Matt: Yeah, there was like a weird thing that would occur when someone was like an eBay seller and they became a buyer where every address they had ever shipped to became like one of their addresses. And then we were like deduplicating in memory, and that—
Rob: [00:23:42] that’s—
Matt: you’re not supposed to do that. That sounds—
Rob: it was—
Matt: it didn’t work very well. I told you we would tell war stories.
Bridget: It’s good. But you actually, Matt, you brought up an interesting point, this idea of operations engineers celebrating the idea of there being no changes being pushed by development. But realistically, I’ve put in plenty of time on call. I’m actually super happy that I’m not on call currently, whether for holiday or otherwise, but I have been on call on Cyber Monday, on Black Friday. Ooh, Cyber Monday a couple of years ago is when I found out the fun way exactly how the global write lock in MongoDB works. It’s like, that was, that was a good time. Um, but this idea of if the devs aren’t pushing code, then that means nothing is changing and that means nothing will break. I kind of feel like that’s ignoring all of the other things, you know, bitrate, entropy, sunspots, your infrastructure providers, everything else that can cause problems. I mean, can you, can either of you address that? Like, Matt, you want to address that a little bit?
Matt: [00:24:50] Yeah, I’d love to. Um, It’s not so much that stuff doesn’t break, it’s just, it narrows the things you have to go look at that changed in the environment. So when you’re dealing with a large environment, you know, as, I mean, we would do basically all-day bridges for every peak day. Like, we have every single person in the command center or NOC looking at the screen, making sure that all the services are healthy, doing hourly checks. Yes, everything looks good, everything’s green, we don’t see anything that’s about to break. So narrowing the scope of, not having to account for, you know, code deployment within the context of those exercises, it just makes it a little bit more manageable.
Rob: Yeah, and from our standpoint, if we look at our, you know, ITIL incident data during a freeze for those systems, we have fewer breaking incidents. But yeah, it narrows— it narrows where to look, and if we’re being, you know, super honest with each other, it It’s a lot easier to explain to our business partners when a third-party device fails than when we touch something and it broke.
Bridget: [00:26:06] Well, that’s kind of an interesting question. I mean, so say we all try to practice blamelessness, but when you start getting third parties in, or if you’re the third party or whatever, how do those conversations go when people are upset about something going wrong and also it’s the biggest shopping day of the year?
Rob: Our third-party and partners have been really good. They will usually staff up and be ready for us on our peak days, including anniversary and our peak holidays. They’ll open up proactive incidents. They’ll review our systems. They’re on it, so usually it’s fine. We will not hesitate to push the panic button with them, though, and escalate as needed, but it, you know, our— all of our third parties are really good to us.
Matt: So yeah, I would say we had a pretty good relationship as well. I mean, the worst thing that would happen is a merchant would call and basically threaten to wire us off. So we’ll just like remove you from checkout, which is always bad. We don’t want to have those conversations. But, you know, generally That was kind of the conversation. It didn’t really extend much beyond that. Like, either it was— it’s kind of a binary conversation. Like, either it’s fixed and you’re good, or it’s not fixed and we don’t send business your way. So as a third party, you know, that’s— it drives a lot of behavior. So yeah, that’s kind of how it went on our end.
Bridget: [00:27:39] And you mentioned, Matt, you mentioned this idea of like really long conference bridges, and I know that I mean, we’re all trying to move to more of like a chat ops and managing incidents in some way that doesn’t involve like really long phone calls and makes for better written records. Can you folks go into a little bit more detail about, in terms of like, how does on-call change? This idea of around-the-clock humans looking at blinking lights, but maybe you have some sort of different automation around the blinking lights as well.
Matt: What we would do is generally groups that wouldn’t always have on-call, we would put on call. So maybe like a capacity— I mean, capacity around the holiday is always a sensitive topic because everybody wants to fix every problem by adding hardware, and that doesn’t always make the problem better. In fact, many times it can make the problem significantly worse, and you can have cascading failure.
Bridget: [00:28:40] I know this one, classic back pressure.
Matt: So, you know, that’s kind of how we handled it. I would say a lot of the work we were doing was kind of before ChatOps was as much of a thing. There’s also this soft, like Rob was talking about, there’s like a soft organizational piece of like, it makes the CTO very like warm and fuzzy that there’s a bunch of people sitting in a room, like looking at all the monitors, having a discussion, and he can like call in and hear status updates And yep, we’re still good. This is the number we hit. Feel very comfortable. Everything is boring. You know, I don’t know. There’s like a people, there’s a people component to this of where, you know, reason and logic kind of don’t always prevail. And it’s more about that like emotional comfort of, you know, yes, everything’s going to be okay. We are going to make it this year. No major issues.
Rob: I mean, we’re, we’re on a journey, right? So, In a 1,600-person technology org, some people are further along into ChatOps and automated monitoring and don’t necessarily need that all-day conference call, but other teams aren’t, and for a variety of reasons. A lot of times, it’s just the systems they support haven’t been rewritten to do all the things, and we need that call still.
Bridget: [00:30:09] So if that change— I mean, if you do have these like big conference bridges or whatever, and maybe different people are on call, does that change like the incident handling? So do you have like different handoffs or triaging? Like if you’re gonna look at what your normal incident handling is like, and then your, we know everything is gonna be on fire, so, or could, has a potential of being on fire worse than usual incident handling. Like, Rob, do you wanna go into a little more detail about that?
Rob: Yep, so for our peaks, we’ll get the on-calls into a room and just be ready for it, much to what Matt was talking about. People are all together watching the systems, and so incident escalation is much faster because everyone is right there to talk to each other. For teams that are generally not on-call but are for peaks, which we do the same thing, we will make sure we escalate much faster and get conference bridges set up much, much faster than we normally would. It’s kind of the see something, say something mentality of like, let’s just overreact to everything, at least on these couple days.
Matt: [00:31:17] Do you ever—
Bridget: oh, go ahead.
Matt: The same thing really on our side, exactly what Rob said. I mean, who plays what role doesn’t really change too much from an incident handling perspective. It’s just more overreacting to everything. You know, if you see any indication that something might be going wrong, maybe you page out to SRE and say Hey, can you proactively go look at this and be sure there’s not something weird going on here? But it is very much the same. People are still involved. It’s just they’re overly sensitive to any kind of derivation of what might be considered normal.
Bridget: So I’m wondering if there’s any lessons that you take away from this, like after you’ve been through a couple of those, that just help you improve your normal processes. Just because I have this idea that If there is a better or faster or more effective way to work and you only do it sometimes, but then you’ve tried it and you like it, what kind of changes, or maybe the other direction, you’ve done this now and you know it’s terrible and you never want to do it?
Rob: [00:32:22] Yeah, so we, over the past couple of weeks, I’ve been maybe pushing the panic button a little sooner than I normally would just to kind of get the teams practiced on troubleshooting, because to your point, we only do this a couple times a year. You get rusty. So it’s kind of like game days, I guess, but more like, hey, what’s this anomaly I just saw? Normally, people would be like, eh, it’s probably fine. And then they’re like, no, let’s track that down, please. And it’s giving the teams probably more practice than they would prefer, but it’s going to make holidays a little much more calm. But to your point, You know, we used to even, you know, back when I first started here 10 years ago, didn’t even do performance testing. So yeah, and that would lead to a horror story actually. But the, you know, we’re improving that every year. Continuous improvement has been a thing in this area. People care about it a lot more than they used to.
Matt: [00:33:25] Yeah, I would agree. I mean, the one thing to take away is like heroism isn’t scalable. So don’t rely on that all the time because you’ll burn your people out. But I think there is a lot to take away, you know, especially around performance and load testing and trying to put that onus on the team. Like, you’ve developed a process to figure out, you know, where your risk is. So can you just like assess that risk all the time instead of just right before holiday? You know, that’s the big one. But pulling people into bridges and, and doing all of that stuff all the time probably is isn’t the right answer over the long haul. But again, like Bridget, you mentioned backpressure handling. So, you know, do you really care if you have to shed load if you can do it elegantly? And starting to think about those things so that the system kind of takes care of itself and you don’t have to worry. The other thing we started doing at PayPal is we ran through a few holiday seasons where we knew we were up against like a software architectural constraint where no matter how much infrastructure we threw at something, it just, it wasn’t gonna scale even if the system was running at like 20% CPU. And so what we ended up doing is setting the target like double what we knew we were gonna hit for the year to basically force the hand of the engineering teams to go and fix those architectural deficiencies so that we could push that scale out a little farther because we just were never comfortable of, you know, we think we’re gonna hit this number and we’ve always been within 5%, but if for some reason— because it’s a pretty sharp spike, it’s a pretty sharp spike, and, you know, you want to know that you’re— you know, history always says we’ve been right on the mark, but the one year we’re not, and then that, that would be bad times for everybody. So we want to be way out in front Pretty sharp spike.
Bridget: [00:35:30] You don’t want to trip and fall on it.
Rob: Exactly.
Bridget: Rob, you look like you wanted to say something about that.
Rob: Uh, I don’t know. Yeah, I mean, I just think like anniversary, it is a spike. It goes from midnight to, you know, 12:01. And it’s just boom on the website. You don’t want to trip on that for sure. We have once. Don’t ever want to do that again.
Bridget: Okay. Are we to horror stories? Do you want to tell us what it’s like when you do trip on it?
Rob: Sure. It was my first year at Nordstrom, and the anniversary sale has always been kind of a very happy time. The ops teams would be excited because no one’s changing anything. They just kind of sit back and watch the systems and get food, and it’d be kind of a party. I was like, all right, this is cool. I can do this. I’d been here, I don’t know, maybe 3 months and still learning the systems. The whole site came crashing down for the entire weekend under load. They had never done any performance testing. We were still on bare metal servers, so like scaling was not really a thing. It was pretty— it was, it was a brutal, brutal weekend for the ops team. It initially looked like a denial of service attack, and then no, no, it’s just our customers trying to buy things. So we call it anniversary. We learned a lot of lessons.
Bridget: [00:37:22] Were some of those lessons you might want to get you some of that cloud you’ve been hearing all these good things about?
Rob: Um, I think, uh, well, that is for sure a thing today. Um, back then it was maybe we should have a performance environment, maybe, you know, we should virtualize so we can scale easier. Um, and now it’s for sure cloud.
Bridget: Yeah, yeah, because I, I hear tell that the cloud thing allows you to you know, with the hit of an API or the press of a button on a console, obviate an awful lot of those procurement meetings.
Rob: So here’s the thing though that’s interesting about cloud when you’re making that transition to cloud is a lot of those services for the short term rely on services in our data center. And I’ll tell you what you can’t provision on demand, and that’s bandwidth. So yeah, so now you’re dealing with telcos and, and sometimes construction equipment And that is kind of where we’ve had to pay a lot more attention this year. And if there was going to be a disaster, that might be it. But I think we’ll be good.
Bridget: [00:38:26] I’m guessing that Boto does not yet have any hooks into construction equipment.
Rob: Not that I know of.
Bridget: Or whatever your cloud provider of choice is, their API might not have that.
Rob: No.
Bridget: It’s kind of funny. I’ve talked to people who are trying to bridge that gap between their data center and public cloud. In the case of the individuals I was speaking with, their employer was looking at AWS, and AWS wanted to sell them some Direct Connect, which sounds— if you remember that, uh, like joke thing that was going around translating all of AWS’s service names into what they should actually be called, I think that one was called something like give us all your money. They’re happy to give you a leased line between like your data center and, I don’t know, us-east-1 or something, but it’s for, for the, the small sum of all of the money.
Rob: Yeah, it’s— so we have stores all over the country, so we’re pretty familiar with telco expenses. So it wasn’t getting dedicated bandwidth into our cloud providers wasn’t that big of a deal. But yeah, it is a non-trivial expense.
Bridget: [00:39:39] And so we’ve mostly— Matt, for you, we’ve mostly talked about PayPal, but you’ve been at Allstate for a while now. Since you’re not so much the third party now, are some of your concerns when you’re scaling a little bit different? Maybe there’s less of a, hey, they’re going to take us off their site, and more, I don’t know, something else. Can you kind of address what the scaling concerns are for an organization like an insurance agency?
Matt: Yeah, sure. Our biggest worry would be that there would be some kind of event or disaster where people were relying on us for claims. We wouldn’t have systems available when people really need them. And that’s certainly much less predictable than the holiday season. But, you know, obviously everybody’s kind of on watch hurricane season comes around and things like that. I haven’t been through one of those yet, so I can’t really speak to the experience side. It would be really great if that just didn’t happen, not only for Allstate business, but we don’t want people going through those things. So yeah, but certainly the worry is there and the concern is there, and, you know, trying to manage that somewhat similarly to a holiday but in a less controlled way.
Bridget: [00:40:56] So do you find that looking at things like adding in public cloud is the solution to being able to scale, like, on demand more?
Matt: Yeah, I think we’re not there yet, but certainly there’s a desire for elastic capacity to some degree, and being able to know that we can address needs as they arise. So definitely we’re looking at it, and we’re hoping to get there, you know, over the next year or so, be able to do something in that space, but we’re not quite there yet. But, you know, the platform as a service side, like I said, we’re running platform as a service, so we can scale and move workloads within our own infrastructure. With a lot more ease than we’ve had in the past. And so at least we’ve made part of the journey. We’re getting used to the idea of how those things work, how to operate them, how to understand them, and know that, you know, we don’t have to rely on a human being to react to those things. It also forces us to be a lot more metrics-driven, which I think is really positive, a lot more metrics and data-driven.
Bridget: [00:42:04] Yeah, and so if you’re using something like platform as a service, I mean, obviously that’s a tooling change because you don’t have to, say, have the operations team provision you a VM or whatever. But I guess there’s also cultural changes there because the operations team does not get to say no to the VM or whatever it is that the person wants to put out there in whatever environment it is that they’re authorized to use. Matt, do you want to talk a little bit about the cultural work that you need to do in an organization to make that feasible?
Matt: We have a lot, I would say just a lot, a lot of work to do. And I would say traditionally we definitely have a security view of least access, and so talking through those conversations of, you know, what does least access mean, what does access, you know, to do your job function mean when the job function changes to more of a DevOps type model. I think that’s something that’s ongoing and we’re continuing down that journey. I certainly wouldn’t say we’ve arrived by any stretch of the imagination, but we’re certainly making progress and the platform is enabling and driving those conversations.
Bridget: [00:43:17] I know that Rob has had his team speaking at various tech conferences about like both the tools and the culture around the work they’re doing. Rob, do you want to jump in on this one?
Rob: Yeah, so we’re heavily invested in moving to continuous delivery and automation and infrastructure as code and becoming more metrics-driven, but from a lean perspective and a continuous improvement perspective. So, what is our next target condition and what are we doing to get there? And what are we learning along the way? Kind of the plan, do, check, act cycle. So that’s new for us, though, and that’s something else not evenly distributed throughout our organization. But where we’ve really applied that, like our customer mobile teams are kind of the unicorns within Nordstrom, and they’ve been really successful at it, and they don’t worry about They worry about performance, they don’t worry about running out of capacity. So they ship whenever they need to ship, and that would include holiday. So trying to model after them as much as possible.
Bridget: [00:44:36] So when organizations are trying to make that more evenly distributed, what would you recommend? If you’ve got one team that maybe has the will, the organizational will, and the tools that they need in order to move as quickly as the organization requires, and you want to distribute that more evenly, Like, what’s the next step? What would you recommend?
Rob: Yeah, it’s getting more— it’s a couple things. So some of it is getting the more senior leadership bought in, which was not how it started. It started kind of on the ground here and with some leaders that got it and some teams that got it. But to really accelerate that, senior leadership and being very open about it. So sharing weekly demos about what’s going on, and we’ve kind of created a dedicated team of kind of practitioners to help educate other teams on continuous improvement and some of these lean methodologies. So they will go in and kind of assess, is this team really ready to take this on? And if they are, they will go and do a workshop with them and help get them on their way, and then measure them after that workshop to make sure that they’re kind of sticking with it.
Bridget: [00:45:54] You’ve mentioned measurement and metrics a couple of times, and Matt mentioned that, too. Matt, do you want to address what some of the metrics, the most important metrics that you look at inside Allstate right now are in order to assess how you’re doing with your transformation?
Matt: Sure. So, we are looking at metrics, and this is still a journey for us, but we’re looking at metrics like when people use continuous integration servers, there are a lot of people who use it like a workflow engine. They don’t really actually understand what it’s meant to be used for. And so, you know, the quality of your tests, the code coverage, and like all of the things that we actually care about when getting you onto continuous integration server, you know, they kind of tend to fall by the wayside. So we’re trying to bring that— bring a transparent view view to those things of, here’s this service that’s great and you can use it, and by the way, this is how well you’re using it or not so well you’re using it. And we have, what I would say is, we don’t really label them coaches, but we have folks that are tied in on both the— we have an agile effort and extreme programming effort, and that’s kind of all managed under one umbrella, and so we’re providing a unified metric view of how people are using those systems, as well as things like cycle time. How long is it taking you to go from feature ideation out to being able to get in front of the customer, and how much is that improving over the way we’ve done things in the past with the waterfall model? And so being able to tell that story and share those wins is really important for building momentum. And it makes sure that we don’t kind of use our corporate culture that we’re trying to change and like take this process that was kind of pristine and well-managed and, you know, wrap a bunch of other process around it that makes it maybe not more— look more traditional than we would like.
Rob: [00:47:56] Yeah, the cycle time is the big one for us. Our VP of selling, Courtney Kistler, talked about this a little bit at DevOps Enterprise. She kind of just put a stake in the sand for our selling teams of, you know, we’re going to have a goal of reducing cycle time by 20%, and then we kind of tie all our other metrics to, are we, are we getting closer to that or further away?
Bridget: And that’s a really good point because the cycle time, if you’re— if it’s too long, then you could be reacting to some, you know, business conditions that have changed and passed you by. And then your competitors could be doing something that you wish that you would have done, but instead you were taking too long with whatever it is you were doing. Yep. All right, so the two of you have been hearing from each other’s, you know, perspectives and whatnot here. So just as we wrap up, like, I think it would be great to hear from each of you, like, what you think that you’re pretty excited about that you’re doing for your scaling, and what you’re interested in trying to grab from what you’re hearing from the other organizations on this Hangout.
Rob: [00:49:12] Okay, I’ll do that. So, I am most excited about our kind of move into public cloud and how our teams are taking more of their own ownership of their systems as a result and taking on responsibility for the system. So it’s not an ops problem anymore, it’s all our problems. They’re the ones who can fix it the fastest, especially if it’s a code issue, so give them all the power. I’m really excited about accelerating that. Talking to Matt, one of the things I would like to take away is more focus on creating a platform for engineering teams that they can use that that kind of abstracts some of that responsibility away. It’s handled by the platform. We don’t have that today.
Matt: Yeah, I think what Rob just said, I want what he has. Basically, what he just said, as far as public cloud developer empowerment, huge amounts of transparency, to us, that’s kind of what transformation begins to look like. What transformation begins to look like at Allstate is really become more of a technology-focused focused company, and we’re viewed that way not only internally, but externally. And so we have a lot of work that we’ve been doing in the community. We’ve been contributing back to some of the Cloud Foundry CLI stuff, and we’ve made some contributions for some of the BOSH releases related to ELK, and we’re fairly involved in some of the meetups, and we helped found the meetup in Arizona. Which is awesome, but we just want to see it continue to gain momentum and drive it out. We’re really excited about building services around these things that are somewhat non-functional requirements in nature. How do we provide the right level of abstractions so that developers can load test their own systems, and they can do it very easily and in a scalable way and on demand? How do we provide abstractions so that they can security and penetration test their own systems, and then the security folks have the right visibility into exactly what’s going on, and we’re getting them plugged in much earlier in the application lifecycle rather than kind of tagging them on the backend and then being delayed by X number of weeks before we can go into production? So that’s what we’re excited about.
Bridget: [00:51:42] That’s great. I really, I I feel like this is so cool to have retail and insurance and everyone realizing that we’re kind of all in the software business. I mean, I work at a vendor now in the software business, but it’s like even if you don’t work at a software vendor, you’re still in the software business apparently.
Rob: Yep, we’re all in this together.
Bridget: Yeah, so in that spirit, we should move into our community and event stuff, which is that there’s a bunch of upcoming conferences and events that some of us are going to have a presence By the time people are listening to this, QCon San Francisco may already be over, but I will be out in San Francisco next week for QCon between November 16th and 20th, and I’ll also be making an appearance over at Structure. And at both of those, I’ll be talking about platform, unsurprisingly. And also, I’m really excited at QCon, I’m going to co-run a Cloud Foundry and Spring Boot, Spring Cloud workshop with my coworker Josh Long, who I have not actually met yet because, you know, distributed teams. But so I’m going to meet him and run a workshop with him. It should be super fun. And there’s also a number of DevOps Days still happening this year. Detroit, Ohio, Warsaw are still coming up in 2015 yet. And there’s some OpenCFPs as well. The Velocity ones aren’t open yet. They’ll probably open right around holiday holiday when everybody is enjoying their Nordstrom and PayPal and insurance websites all working and up. But keep in mind that the OSCON CFP for next— I want to say it’s May in Austin— actually closes November 24th. So if you can sketch out what you think you might want to talk about in Austin next spring, you are almost out of time to submit a talk to OSCON, which you definitely will want to do if you’re into that open source thing. So yeah, of the stuff that I’m going to be at other than QCon coming up, I think I’m mostly done with travel for the year, but there’s— it looks like there’s a little bit of possibly Orlando, maybe a quick trip to New York and DC in the middle of December. So further updates on that as the situation warrants. And I have no idea where Matt and Trevor are going to be other than not here today. So And then there’s also some meetup involvement that, Matt, you said that folks are doing. And there’s a— I think Nordstrom has some involvement in KubeCon too. So let’s hear about that from you folks.
Matt: [00:54:20] Sure, absolutely. So this Thursday, which is— what is— what day would that be? The 12th? The 12th, we’re doing— we’re speaking at the Denver Cloud Foundry meetup talking about, I believe, the differences between Concourse and Jenkins. I mean, kind of doing a compare and contrast there. So that should be fun. And then in Phoenix, we’re running a BOSH workshop for that, which is open to the community and free on December 4th— how to use BOSH, run infrastructure as code using BOSH. And that’s what we have planned, at least at this point, for the rest of the year.
Rob: And then at Nordstrom, we have a number of speakers at QCon San Francisco, and then a number of other Nordstrom people there that are helping be track hosts. So definitely seek them out. They would love to chat. And then—
Bridget: Oh yeah, tell us who’s speaking. You have a few Nordstrom people.
Rob: Yeah, so Sarah Lake Hagen, John Madison, and John Schinner. Sarah’s with our People Lab, so anything culture or the science around culture, She is amazing to talk to. John Madison is a developer, and John Schinner is going to be talking about mindfulness, and so that should be interesting. Then, DevOps Days Ohio, Courtney Kissler will be speaking, and she is awesome to talk to also. Definitely seek her out.
Bridget: [00:55:51] Oh, yeah, I should mention that even though he was not able to make it to this episode of the podcast, Matt Stratton is actually speaking at DevOps Days Detroit. So, if you happen to hear this in time and/or are going to DevOps Days Detroit anyway, he’ll be speaking Wednesday, November 11th. And then your mention of meetups, Matt, reminded me, I am actually speaking at a joint meeting of the Cloud Foundry and Docker meetups in Minneapolis on December 1st with my coworker Casey West, who’s going to be flying in for that. And we’re going to be talking about the new native support for running Docker images on Cloud Foundry. Laundry, as well as orchestrating workloads with Lattice. So that should be pretty fun. That should be a good time. It’s so funny, I had written down Minneapolis as one of the places I would be, and what I didn’t write down was speaking for a meetup, not actually doing laundry between trips. So yeah, let’s hear some checkouts. Rob, do you have some cool stuff for our listeners to check Sure.
Rob: [00:56:58] Something that I am fascinated by and do a lot of reading around is kind of organizations and also kind of the biases and way we kind of rationalize away things. And there’s a book, You Are Not So Smart. It’s kind of a very quick read, but it gives you an intro to a ton of them. What I really belatedly realized is there’s a whole website and podcast around it also, so youarenotsosmart.com. Nice. Yeah, and then, so check that out. And then the other one was Ariel, one of our ops managers, sent this to me and said I must watch it. It’s a TED Talk, Why It’s Time to Forget the Pecking Order at Work. And quick spoiler on that, I guess, is it’s about chickens and farms, and a farmer was trying to make his chickens more productive and took the super chickens that could lay a ton of eggs and put them all together thinking they would, you know, mate and do their things and get a ton of egg production. What happened was they mostly killed each other off and the kind of mediocre chickens all drastically increased their production. So they did not have the influence of super chickens in the middle. So check out that All right, cool.
Bridget: [00:58:26] Matt, what do you’ve got for us?
Matt: So as you heard me talk a little bit about capacity planning, so I threw Neil Gunther’s great book Guerrilla Capacity Planning out on there. I think that’s kind of a must-read for anybody who’s playing in the capacity and performance space. Kind of goes into how capacity planning works on a technical level. And then whoownsmyavailability.com, which is always a fun site to send around when you have availability problems and, you know, people are finger pointing, especially like I always like to whip that one out when we’re talking about like what’s our SLA or like what our vendor’s SLA is and like whether they actually care that we’re impacting customers or not. So, you know, given the holiday spirit of the episode, I thought, you know, that would be a fun one to share.
Bridget: Nice. Okay, the couple of things that I have to check out, I guess you could consider them to be something that you could shop for for during this exciting shopping holiday upcoming period. I’ve been doing a bunch of travel for Pivotal, and I’ve found that after reading about 800 billion reviews, I bought the Bose QuietComfort 20 acoustic noise-canceling headphones. These are the in-ear ones, which I’ve always found not super comfortable, but these particular ones come with a lot of different sizes, like 3 different sizes, and I was able to get one of them to be comfortable. They really, really work for canceling out like the engine noise on a plane and/or the screaming baby behind you. Like you actually can get things done, mostly in my case, listen to podcasts while working on planes without a lot of terrible distractions and that dreadful hum. And then the other thing that I found super useful is the, I don’t know how to pronounce it, Lezone, Lezone. We’ll have a link in the show notes to all these things, but I bought this Extra Pro 2600 milliamp, whatever it is, the external battery charger that can actually charge laptops. Because probably by now all of us carry something that charges our phone, but when you’re on that flight coast to coast and you look and you’re like, great, this flight has no power and my laptop is not going to make it all the way to SFO, it’s really nice to actually be able to work because you have a battery you can plug your laptop into. So those are the stuff I’d recommend. So, I kind of feel like, and these do not have Amazon affiliate links, and I kind of feel like they should because I’m just going to keep buying these things for people and telling them to buy them. But in any case, let’s wrap up. We have a newsletter, arresteddevops.com/bananastand. It’s the best way to know about upcoming podcast episodes and cool news with DevOps. Stratton wants that to be said. I feel like cool news with DevOps is something you’re going to also find on Twitter. Twitter, but you can look in the newsletter for it. We also have an iPhone app if you dig that kind of thing. You can download it for free at arresteddevops.com/iphone. Thanks to our sponsor Datadog. Be sure to visit them at arresteddevops.com/datadog. Thanks to Rob and Matt for joining us. This was really fun, guys. I’m really glad you were able to make it.
Rob: [01:01:41] Thank you, it was awesome.
Matt: Yeah, I had a blast. Thanks for having me.
Bridget: Nice. And loyal listeners, if you enjoy Arrested DevOps, we’d appreciate it if you’d visit arresteddevops.com and leave us a review in the iTunes Store. We’d love to know what you thought of this episode. Please leave us comments at arresteddevops.com/seasonal-scaling. Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter. We’re always happy to get your input, ideas, or feedback at shows@arresteddevops.com. So please let us know any ideas you have for future episodes. I’m Bridget at BridgetCrumpton.com. No Matt and Trevor today, and we’re Arrested DevOps. And remember, there’s always DevOps in the banana stand.
