Scaling the Application Mountains

Igor: [00:00:02] It’s time for Arrested DevOps, a high-level biweekly panel discussion of DevOps concepts, giving you a tantalizing taste of the basic technologies and ideas of DevOps to entice you to try more. Here are your hosts, Matt Stratton and Trevor Hess.

Matty: Welcome to Arrested DevOps, episode 10, Scaling the Application Mountains. I’m your co-host, Matt Stratton, @MattStratton on Twitter.

Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.

Matty: This episode of Arrested DevOps is brought to you by 10th Magnitude, helping businesses realize true agility through DevOps and cloud-enabled innovation.

Trevor: Tonight on ADO, we’ll be talking about scaling applications, but since it’s been a while, you want to tell us about ChefConf, Matt?

Matty: Absolutely. So we had hoped to do an episode on ChefConf, which is the annual conference for Chef, the configuration management software, but We didn’t. So I just want to go over a couple little high-level things that I thought were pretty awesome. If you listen to our buddies over at The Ship Show, they did a real good recap episode, and I’ll link to that in the show notes. And it’s much better than any recap that I could do. I was going to join them, but I had a beer instead, which is a shame because they had bourbon and bacon while they were recording theirs. Which brings me to— those are pretty much the 2 best things about ChefCon for the bourbon and the bacon. In reality, though, there was the one thing I wanted to call attention to that I thought was pretty remarkable is one of the keynote speakers was Mark Russinovich from Microsoft. And so we’re sitting in a room of, you know, 400+ open source Linux, God, I can’t stand Microsoft people. I mean, everyone kind of around me when they announced the work, literally I’m hearing things like, oh my God, Microsoft presentation, really? Here comes the sales pitch. Mark got up there and talked about Azure and talked about how for Titanfall, you know, they’re running pretty much every single player who’s playing Titanfall at any given time is running on a VM in Azure. Showed how we’re able to provision Linux VMs using Chef directly through Azure, and everyone was really pretty surprised and impressed. That was pretty cool, and I felt really good about the community for for being open to that. A lot of other things that were announced. One thing that was demoed was Chef Metal, which is— if you look at Chef as a mechanism for defining a node, Chef Metal is a mechanism of defining your infrastructure through recipes. Really jazzed to start playing with that a bit more. And we also got a lot of insight into what the next generation of Chef running on Windows will look like. So I’m going to put some links to a couple talks. Actually, I’m going to put some links to a lot of talks in the show notes. There were some real amazing ones. A couple that I just want to make sure that I recommend: Pete Cheslock did a great talk about 2 different workflows using Chef for 2 different organizations he’d worked with, and also Fletcher Nichols’ Test Kitchen Roundup, and Jamie Windsor talked about the next generation of workshelf. So we’ll put links to those. But I can’t recommend that conference enough to anyone who’s interested in the DevOps space and the config management space. The Community Summit is in Seattle. That’s coming up in October. But mark your calendars for sometime around now next year for ChefCon 2015, and I would say try to make it. That’s enough about Chef right now. Let’s talk about scaling applications. Let’s introduce our awesome panel. First, we have Steve Corona. Steve? You want to tell us about yourself?

Steve: [00:03:44] Hey, yeah, I’d love to. I’m Steve Corona. A little bit about me, I dropped out of college when I was 21 and worked on a company called Twitpic. Twitpic I co-founded, and it was a photo sharing application for Twitter. We blew up, and I really had no idea what I was doing and kind of learned scaling by cutting my knuckles, just rolling over in the middle of the night to restart Apache and figuring out how it all kind of works together and how DevOps works. And so I wrote a book on the experiences, Scaling PHP book. Now I work at a startup here in San Francisco called Life360, head of API at Life360. But I’m just really excited, actually. I just moved to San Francisco this past week, so it’s all new to me, and I’m excited to be here.

Matty: Great. And Igor?

Igor: Sure. Thanks, Pat. My name is Igor Papirov. I’m based in Chicago area. I run a small software firm called Paraleap Technologies. We make a software service that’s called AzureWatch that helps folks monitor and manage and scale their Azure applications. In my previous life, I was a chief architect of a company called Restaurant.com, where I cut my teeth on a large-scale computing and helped Restaurant.com handle large peaks of users at various times and migrated to Azure pretty successfully. And I’ve been, you know, background software developer, been doing development since 1992, ’93, sometime then, and we’re excited to be here.

Matty: [00:05:28] Awesome.

Trevor: Hey guys, thanks for joining us. So we’re going to get started with the panel. So first, what is scaling? Let’s start really basic and get everybody kind of on the same page.

Matty: That’s that thing that happens when you don’t have a good rinse cycle in your dishwasher, right? And you get all the crap on your glasses.

Trevor: I thought we were going hiking.

Igor: Well, descaling is when you’re getting your— the crap— that crap out of your coffee machine.

Matty: That’s a different podcast where we have the arrested, uh, householdtips.com. So, but application scaling, because we talk about it a lot, the term gets thrown around, but in, you know, quick summary, what do you guys— how would you define that, you know, to someone who doesn’t know what they’re talking about?

Igor: Well, I think before one talks about scaling, you want to talk about what is scalability. And scalability is being able to handle large amounts of load or extra amounts of load by being able to provision extra resources. And it’s different than performance, which a lot of folks, you know, often compare. Performance can be, you know, fast or slow, but scalability is maintaining the same performance over different peaks of usage.

Steve: [00:06:56] Yeah, I actually— I really like what you just touched on right there, which was that scaling is not performance. Like, they’re not the same exact thing. It’s maintaining that existing application, maintaining through all the different use cases and people using it, and also main— it’s like almost this idea of maintaining the application as the application changes with the same user base. Like, it’s more infrastructure than it is code when you talk about scaling. There’s overlap there, but there’s a lot of infrastructure too.

Igor: Absolutely. It’s really about being able to handle versus being fast.

Matty: So it’s, in a way, it’s kind of the ability to react to a changing condition?

Igor: Sure, or be proactive and prepare yourself for the changing condition.

Matty: So I guess maybe not being reactive, but you can dynamically be elastic because the conditions are dynamic.

Igor: [00:07:57] Yeah.

Trevor: So what kind of conditions do you prepare for when you’re getting ready to scale something? Like, what are the sort of situations that you’re getting ready for?

Steve: I think that the classic conditions that you prepare for are things that you necessarily don’t expect or don’t see when you’re running something that doesn’t have a large user base. So you’re preparing for— a lot of times it’s failure, but sometimes it’s not— things like timeouts, things like how does this particular code or infrastructure server handle when I have a million people using it at the same time. There are a lot of really like fine-grained problems that you need to think about when you scale up that you don’t necessarily have to think about when you’re a smaller size. But I think really it all kind of comes down to timeouts and blocking. How do different parts of my application or my stack handle in different timeout or blocking scenarios?

Trevor: [00:09:01] How do you handle the Reddit hug of death?

Steve: How do you handle what?

Trevor: The Reddit hug of death.

Steve: The Reddit hug of death.

Matty: This is where you see the age difference between me and Trevor, because I would call it the Slashdot effect, which is, I don’t think, an effect anymore. I think being on the front page of Slashdot barely ticks you over anymore.

Igor: That’s true.

Matty: So that was— so I guess if I’m understanding correctly, or kind of breaking it out, so there’s The ability to scale from— the ability to respond to something predictable. So if you have some type of a seasonal business, and in one of my past lives I worked for Apartments.com, and so very seasonal business, right? Way more people looking for rental properties in May than there are in December. But that’s predictable, something you can predict. Or for example, one of our sister companies is Cars.com, and when they would do a Super Bowl ad, like the day after their Super Bowl ad was a quarter of the traffic for a year. And that’s a made-up number, by the way, so I can say it. But it was something ridiculous like that. But they knew it was coming, so you can prepare for it. So that’s one type of knowing that you’ve got that elasticity. But how does one be prepared for the Reddit hug of death? And Steve, I would imagine in your case, for just unpredictable success, right?

Igor: [00:10:26] Sure.

Matty: Were kind of built on a certain mechanism, and all of a sudden you became the default photo sharing tool for all of Twitter. You weren’t designed for that. Actually, how did you react to that?

Steve: How did I react to that? First, by jumping up and down, and second, by pulling my hair out for years, right? Because sort of what I alluded to earlier is when I started, I really had no idea what I was doing. Scaling expert by trade. I went to school for programming, and I had been programming for a long time, but really all this low-volume stuff and had never done anything at high scale. Learning was such an incremental process of how do I build something that doesn’t constantly crash over and over again, and learning what pieces of software to use, and learning best practices for doing timeouts, and learning that even some tools that we use and still use and take for granted for PHP, for example, has really poor support for doing fine-grained timeouts that you need at scale. Just so— how did I learn? It was mainly by practice and mainly by screwing up the site and crashing it over and over and over again for a long time and learning that, like, oh, wow, you need to do this certain thing in MySQL or else when you get to a lot of connections, it’s going to crash or not work well. There’s a lot of— it’s weird because there’s a lot of, like, tips and tricks or best practices out there that aren’t published, that only, like, the top people in the DevOps field sort of know, and they know inside their head, but they’re just not out there. You can’t find them on blogs. You can’t find them on Stack Overflow, whatever. But to your earlier point regarding, like, predictability, I would argue and inject into what you said that predictability plus configuration management, right? Like, you need to have Just because I know that tomorrow I’m going to have a lot of traffic, I also need to be able to set up a bunch of servers without having to actually go and set them up manually and that sort of thing. I think you learn— a lot of what you learn, especially DevOps and especially scaling, is very non-traditional learning. It’s learning by doing, I think, a lot of the times.

Igor: [00:12:35] That is very true. I’m curious, Steve, how big was your— Were your systems that you were running— can you give us a sense of scale?

Steve: Yeah, yeah, so I’d love to. So our team at the time— I’ll give you the team just so you can have background— was about 7 people. So we were always very, very tiny. I think at the highest amount we had about 90-something bare metal servers. So these are servers from— we use SoftLayer. They’re all like top-of-the-line SSD, hundreds of gigabytes of memory, 4 processors with 12 cores, something like that. It was somewhere close to 100, like 90-something, and then we started using more cloud stuff. Now, where we saved a lot in terms of server horsepower was that images, sharing images, you’d imagine you store a lot of stuff. We were able to leverage Amazon S3 for that and store our petabytes of data in the cloud, which was A, very expensive, but B, it saved us from having however many hundreds of extra servers you would need to store petabytes of data reliably, redundantly, and geo-redundant. You would need hundreds, if not thousands, of servers to effectively do that. Maybe less now because disks get bigger and that sort of thing. Using those cloud resources, we saved a lot of servers, but even still we had about 100 bare metal.

Igor: [00:14:05] Yeah, that’s pretty impressive. And actually, I think it’s interesting that, you know, when you say 90 servers to folks who are, you know, who are not doing large-scale computing, that sounds like a lot of servers, and it’s really a lot of servers. And one of the core principles of scaling, right, and being able to achieve scale is actually being able to take 90 servers and actually have every one of them do work, right? And a lot of folks, you know, originally when they start designing and when they start developing, you know, it’s hard to make that mental leap from running one server or running perhaps a couple of servers to actually making 90 servers or 200 servers be useful in the scenario. And I think that’s probably the, you know, the first, the very first step that one needs to think about. When they’re planning for large scale is what if you had— can you use 200 servers? Can you use 500 servers? Can you, you know, how many users can you get onto a server and be productive, if you will? Because a lot of times if you have bottlenecks in certain, you know, certain areas of your system, it doesn’t matter how many servers you have. You know, those bottlenecks may, you know, may give you issues. And often, you know, I’m speaking from the background of as a Microsoft developer. Back in the 2000s and late ’90s, you know, we’ve all been taught to do, you know, N-tier development and achieve scale by having SQL Server in the backend, you know, middle tier with with the business layer and the web layer in front. And it’s, there are challenges with that regular approach, right? Because there are many components within that approach that will not scale to the mega level, to the millions of people per hour level. So it’s, one has to think about it a lot when they’re attempting to take, yeah, 90 servers and, and actually having them all be useful.

Steve: [00:16:16] Yeah, yeah. That’s actually a really good point now that I think about it. Where I am now at Life360, we have something like 100 AWS instances. We have a lot of scaling experts that work at this startup. Even with a lot of scaling experts that know what they’re doing that are using best practices, you’ll still go and see, oh, some of these servers aren’t doing as much work as they could be. Or the work is not, like, being fairly distributed, and that’s a really hard problem to solve even if you know what you’re doing. Like, that’s a very difficult thing to do right is to give all of those 100 servers an even amount of work.

Matty: And I think that—

Trevor: something that makes me think of is a term I hear often used around scaling is scaling— the difference between scaling vertically and scaling horizontally. Could you guys go into a little bit about what that means?

Matty: Sure.

Steve: Yeah, I’d love to. I think that the easiest answer, and it touches on what Igor just mentioned, is this idea of scaling horizontally. You take a problem and you split it across lots of different hardware or servers. Scaling vertically, you have that problem or that program and you just add more resources. More direct resources to that server. You add more memory, you add more CPU, you add more I/O, and you try to just get it done really fast on one server. Everyone preaches scale horizontally, get more servers, but I think the problem that we just talked on is that it’s actually easier said than done. Just because you have more servers, you don’t necessarily automatically scale horizontally. You need to plan for that. It’s hard to do.

Igor: [00:18:03] I couldn’t agree more. I find typically that folks think about scaling vertically when they’re talking about relational databases and when they’re talking about, let’s say, SQL servers or other database servers like Oracle, where you can’t really scale horizontally. You have to add more power to the relational server to the relational brain. And it’s much easier obviously to scale web servers across. But, and this is really one of the interesting challenges of mega scale, is how do you get every layer of your system, you know, every component of your system to be able to take multiple servers and lots of servers and get them to be useful.

Matty: And I think it’s interesting because the old grizzled sysadmin in me looks at this, and, and we’ve been— again, like Igor, you said you kind of came up at a certain time and you were taught a certain pattern. And sysadmins were taught to be, systems are 70% utilized, time to start getting a little worried. Oh, 80%, how come you didn’t buy me shit yet? 90%, you’re screwed. Whereas in kind of the more web-scale world, you’re like, dude, 100%, no problem, use every last little bit of compute you got because the app can handle it. Because that was if it’s built, you don’t have to have that headroom, and I think the cloud is where you have that ability too, because we kind of were built on this. We have to start freaking out at 70% because it’s going to take us 8 weeks to get more. You know, like, we know that it’s going to take a long time to provision new hardware, so we have to get ahead of the curve, and in a cloud scaling world, that’s not the case, but we’re still so used to it that I have to imagine you can run into that challenge where, like Igor said, you want to be able to get every last bit of compute out of what your systems are doing and make sure it’s evenly distributed, but you still have this old way of thinking that that’s really scary when your systems are that committed.

Igor: [00:20:09] Yeah, and this is totally a legacy of the enterprise computing where it took you 8 weeks or and lots of signatures and a bunch of approvals to get the server in. And you had to prepare ahead of time for any kind of possibility. Nowadays, especially when folks are running either in managed environments or in cloud environments, it’s bad if you are at CPU 20% and your servers are— and you have lots of servers. If you have 2 servers, it’s okay. But if you have 20 servers and they’re all doing a little CPU, then you’re throwing money away.

Matty: Well, and that goes back to the vertical versus horizontal, right? Because when you’re talking in a bare metal scenario, your only burst ability is up, right? So you have to have that headroom so that when you have some atypical traffic pattern, you have somewhere to go. Whereas if you can scale out, and you really can only scale out cloud-wise, because unless you’ve got the fastest, you know, rackers and stackers in the West, you know what I mean? You’re not gonna suddenly throw 100 more blades into your chassis. All of a sudden. So you’re— again, I think Igor’s right. A lot of this is legacy thinking when the only place to go was up, but we can go out. And then I think about that though, and that leads me to another question, is where’s the fine balance, right, of not getting into analysis paralysis when you’re starting something and saying, okay, I don’t want to paint myself in a box, I want to make sure that I can scale this thing. I want to make sure that I’m not going to get caught with my pants down when all of a sudden I’m on the COVID of Wired or who the hell knows whatever the Reddit successor similar thing is. Anyway, but also be able to turn around quickly. I mean, there’s an old joke which is Rails app is up in 6 hours, it’s down in 6 months. Which is unfair to Rails, but I think it’s because It’s so easy to prototype something without thinking about scale. So where does that come into the lifecycle? How can we do rapid prototyping and rapid development and not wait till we’re totally screwed, right? And Twitter is kind of an example of this too, I think, because they started out as Rails, right? And then they outgrew it to their detriment and then had to go and refactor everything. So what’s the— what would you guys think is the magic balance? ’Cause I know if you know the magic balance, you’re holding out on us.

Steve: [00:22:43] We have the secret magic balance numbers. We’re just not— don’t want to share them.

Matty: It’s at 42.5 servers is when you should start caring.

Trevor: He’s got 100 Stack Overflow posts in his Google Docs.

Steve: So I think my perspective on it is that as much as I preach scaling, and as much as, like, personally it’s really interesting to me to scale things, I find that the easiest way to build something is to not worry about scaling. That classic, don’t worry about it until you get the traffic that you need to worry about it. Because like we talked about vertical versus horizontal, you can still scale vertically pretty easily to a point. If you’re on AWS, you’re one restart away from 500 gigs of memory and 8 SSD drives. Using a bare metal provider, it’s still pretty fast to get a RAM upgrade. You can scale a lot on one box with just a shit ton of hardware to the point where that’s enough to probably cover you from getting put on Reddit pretty fast. That being said, though, that’s not the end-all be-all. If you’re trending in that direction, you need to start splitting things up into different pieces as a starting point. But as someone that loves scaling, it’s really easy, like you said, to get into that research paralysis where you’re like, oh man, and then I’m going to do this and do this and do SOA applications with different backends. And oh, this database is better than this other database, so I need a whole data— it’s really easy to get into that trap. And so my solution anyways, maybe it’s a poor solution, but it’s to avoid that from the beginning and just say, you know, build it as naively as possible and then figure it out.

Igor: [00:24:30] Yeah, I will definitely concur. I mean, to get to a large scale, it’s not that simple. It takes a while. You’re going to have so many chances to rewrite your system and to throw people at it and throw resources at it to make it more scalable. If you’re just starting out and you’re building a startup, take it easy. Get the functionality out. You know, get folks to use it, get to hear, to listen to the feedback, because your system will change anyway multiple times before, before it actually becomes popular and viable, and you’ll have opportunities to, to adjust to scale. But if you can, obviously there are certain technologies and certain patterns that are better than others that, you know, if it’s, if it’s all the same in the beginning, right, then it may make more sense to try to know, not to be shooting into certain patterns that are kind of dangerous, right? But otherwise, yeah, it’s— big scale for a starting app is the last thing that they need to worry about, in my opinion, as well.

Steve: [00:25:32] I think that you said— so you said you will rewrite your app, and that’s very true. Like, you will rewrite your app, especially if you go from, like, low growth to high growth. Probably going to happen, and if it’s not the whole app, it’s— you’re gonna be ripping pieces out of your app and moving that somewhere else. Maybe, like, the lowest overhead way to start with something that’s scalable, I think the easiest way where it’s not too much overhead is when you’re building your app, the first revision is to have multiple separate small apps, sort of that Unix philosophy of, like, this little tiny app does this thing, this other little tiny app does this thing, like a services architecture, maybe that gives you some scaling headroom without having to, like, go crazy and detract you from focusing on the app too much. But, like, but to go back to what I started this with, which was you’re gonna rewrite your app, so it doesn’t matter because that’s inevitable. Like, it’s going to happen.

Igor: [00:26:35] Yeah, if you’ve never done a large-scale app before and you’re building a brand new system and in the beginning it’s low traffic and if you ever get to the high traffic, it’s almost guaranteed that you’re going to need to rewrite it. Yeah. So might as well just put your energy into features and get the traffic so that you can actually have the money and the whatever to redesign certain components.

Matty: So that you have the problem. So what are some tools? So I know, you know, Steve, you said one of the things is there’s a lot of knowledge out there, or whatever knowledge is kind of arcane and hidden away in the secret little cabals of high performance or whatever. So that being said, what are your tools or your thoughts of being able to ferret this out or being able to And also, I guess I would ask the question, do you think that’s something that’s changed? Is that different these days than it was when you were dealing with it at Twitpic?

Steve: [00:27:37] Yeah, so what ways do you ferret the information out of your system? You know, I still use strace, right? I just wrote a newsletter on this actually last week, because strace for debugging things like in production at scale is possibly the most valuable tool. What statsd monitoring, and you could have the most crazy modern advanced PagerDuty setup or whatever, you will never get the visibility that strace will give you when production is down right now.

Matty: So like, yeah, I was gonna say, I have, I have a running gag with monitoring where I’m like, I don’t care how much of an elaborate monitoring system, every good sysadmin still has a copy of What’s Up Gold sitting around somewhere, you know.

Steve: To that point, you need to— just because you have awesome monitoring, a lot of people say, oh, if you have awesome monitoring, that’s it. You log in, production’s down, it’ll tell you exactly what’s wrong. That’s not true. You need to learn all these tools that there’s really not that much information written on. strace, it’s an older tool, so it has a lot of stuff written on it, but comparatively speaking to Comparing it to something more modern, there’s not that much information on it because it’s not, like, a very sexy tool, but, man, it will save your ass when you have downtime. So I think that going back and, like, learning— really learning exactly how your app runs from, like, end to end, like, how the stack all comes together, how your app talks to Nginx, how— if you use Nginx or Apache, how Apache talks to the browser, how each of those settings actually impact Apache. Maybe even read— go crazy and read the source for the open source programs that you use because it all comes down to those traditional fundamentals. If you don’t know those, if you depend on some external monitoring to tell you what you need to know, you’re going to have a hard time. I think maybe that’s why there’s not a lot written about it. It’s a traditional problem but shoehorned into this modern way we develop applications that’s, like— there’s a gap there. Maybe it’s a generational gap, maybe it’s an information gap, but the two don’t necessarily connect. Like, the way we do modern development still runs on these older, more traditional systems, and that’s a good thing, but it’s a piece of knowledge you still need to learn even if you’re, like, doing Ruby on Rails and Node.js.

Matty: [00:30:07] You know what? I think that kind of summarizes in a lot of ways, or it speaks to one of the facets of the DevOps philosophy, which is that it’s not just this application that lives in a vacuum that’s just sitting here. It’s being able to understand the underpinnings of the infrastructure that it’s running on and how that code is working and how it’s interacting. Then likewise, if you’re the one who’s troubleshooting, because who knows who that is? Who is getting called by pager duty? It has to be able to have that, because if it’s just that you’ve got Nagios that’s telling you this, it’s not smart enough to tell you what might be— we’re still all troubleshooting.

Trevor: Right.

Matty: You know, and I think that’s the biggest thing that I’m seeing that people need to understand in this DevOps world is that there is no demarc. You know, again, I like to quote John Vincent, you know, it’s never saying that’s not my job. You have to understand a little bit of everything that’s important about your product. And then know when you don’t know what you know, right? Or know when you don’t— when you’ve hit the maximum of what you know and say, okay, I really got to escalate this to the guy who actually wrote the shit because I can’t figure it out. Yeah. But you don’t sit there and go, well, Apache’s up, so I guess I’ll call the developer because who the hell knows? That’s really common. So Igor, any thoughts on like tools or accessibility? Again, like you’re kind of have worked in a Microsoft world, like how would you chase those things down?

Igor: [00:31:35] Yeah, so I think one thing that Stephen touched upon that I really liked is that, I mean, there is a difference between monitoring and diagnostics, right? And monitoring is not going to tell you why it’s broken. Monitoring will tell you when things are broken, but not why they are. And being able to diagnose things and drill into an issue is a completely different problem. And being able to be alerted when something’s wrong, right? So the, you know, I’m not as familiar in the LAMP stack world as to what tools there are, et cetera. I think both Windows world and the LAMP stack world are pretty well stacked with a number of systems and tools and services that are, you know, that help with monitoring, help with diagnostics. And we can name a long list of systems that we like. And I’m biased, so I’ll probably start with my own. But yeah, the important thing is that I think when it comes to monitoring and when you’re dealing with a large-scale issues, monitoring needs to be done not just from one system. Monitoring needs to be not only reliant upon one system. You want to monitor from as many systems as you can possibly can without generating too much noise, right? First of all, monitoring systems are also prone to their own problems, so one may go down at the particular time when you really need it the most. Secondly, certain systems may give you a different visibility into a problem. For example, you know, a lot of folks who use our product also use a product called New Relic. And, you know, from Azure Watch perspective, it’s native to Azure platform. It really knows a lot about the Azure infrastructure. But New Relic, for example, knows a lot about the application code, if you will. It’s able to actually drill in and show you the stack traces when things are happening. And, you know, typically a question is, how do you differ from New Relic? Or, you know, should I use one or the other? And the answer is yes, you should use both because it’s really important to know before your customers do when you have a problem. And whatever tools are costing you pennies per hour to, you know, to deploy is definitely worth it when you’re getting millions per hour in revenue, right? So it’s just common sense. So I preach multiple monitoring systems for As long as they’re giving you valuable information. And not to confuse monitoring with diagnostics, because diagnostics is really being able to debug an issue as to why something is down. And there are better ways to— you cannot just find out everything from an alert. You have to, you know, alert is generated and somebody has to go dig in. And digging in is a different issue than being woken up at 3 in the morning, right? Typically, people get woken up. Typically, for some reason, those kinds of problems don’t happen unless you’re running an international business that everybody’s on 24/7, which is mostly these days our businesses are that way.

Matty: [00:34:53] Or you’ve got all your batch processing going at 2 in the morning for all your transactions the night before.

Igor: Then you don’t really have to wake up because nobody’s visiting the site.

Matty: Nobody’s visiting the site, but everything’s being billed.

Igor: Because why are you running? And speaking of which, then this is one of those things that you need to think about when you’re doing large-scale systems, right? There are no batch processing happening. You cannot have the batch processing because there’s always somebody and there’s always a lot of people who are doing stuff on your site or on your system. And you have to find a way. You have to find ways not to do batch processing and you have to find ways not to, not to impact your system ever. Unless you have emergencies or whatever, right? Or unless somebody approves it. We’re all, you know, it’s rarely an issue with big enterprise and it’s usually an issue with e-commerce or with popular social sites and nobody will approve, if you will, long downtimes or you know, can I take the system down on Sunday at 3 AM? It just doesn’t happen anymore.

Matty: [00:36:03] Yeah, it’s really a world of rolling maintenance windows, so to speak, because like you said, there’s always something in play. Even, you know, even if you’re not e-com, you know, if you’re talking web-scale IT, you don’t really have predictability in the world that we live in. People want to be able to do the work whenever they want to do it, even if you’re not a multinational company, even if you’re you know, confined to one country, that doesn’t mean that your employees, who are your customers, are stopping work at 5 o’clock. I mean, it’d be great if they could, but no one does that anymore. That’s maybe a bad thing. Maybe we should take outages so that our customers can have lives. That’s why systems go down. We’re altruistic.

Steve: I don’t know, but I don’t think you guys have ever used the IRS website, because if you try to register for an EIN number, it’ll tell you to come back at like 9 AM the next day. The system is closed after 5 PM, and that’s not a joke. It legitimately closes at 5 PM. The system won’t let you register anymore.

Igor: [00:37:10] That’s pretty cool. It’s as if somebody’s there from 9 to 5 who’s actually accepting—

Matty: That’s what’s actually happening. You’re typing the form in, and it’s one of those old CGI forms that just emails it to somebody, and they do something. And then they fire it back to you. So that’s why that happens the way that it does.

Igor: It’s actually funny that you mentioned this. If you want to hear a little story, is that when I was working at Restaurant.com, I heard about the other company that does restaurant reservations. And I don’t know whether it’s currently the way, but back in the day, when you would sign up, when you would ask for a reservation at the restaurant, they had an IVR system that would dial the restaurant and it would wait for a person to actually accept the reservation and press a button. And then it would confirm with the customer that the reservation has been accepted.

Matty: And then you’ve got this sort of like JavaScript spinning wheel that’s like processing, processing.

Steve: Wow, that’s, that’s a whole new form of blocking I/O that I’ve never heard of before.

Igor: [00:38:11] Hey, and it’s— if you think about it, it’s pretty scalable, right? Because you have The wait time is really sharded across multiple restaurants, and there are hundreds of thousands of them. So it’s not bad. It’s not like you have a long queue of requests pending, but each one is human interactive. It’s kind of cool.

Trevor: I just want to give a shout out to Fred, who agrees that the iOS website is an issue.

Matty: This is the first time we’ve had someone pop into Q&A and 8 episodes. I’m so excited.

Trevor: And I was— I decided not to tweet about the Q&A being active because nobody ever uses it.

Matty: So thanks, Brad. Well, it’s better than the first time where we didn’t realize how to turn it on and like 4 questions over there and we didn’t see them. So again, trust us, people, we’re professionals. By the way, we did not point out, and it’s late to do this, but I’m going to do it anyway, and I’ve had class of whiskey, so whatever. This is the 10th episode of Arrested DevOps. We are in the double digits, so hey, go us! It should have been 2 weeks ago, so there’s that. And so any— we’re coming close to the tail end here, but before we wrap it up, any kind of parting thoughts on things that you would want our listeners to keep in mind when they’re thinking about building applications that are web-scale IT, whether they’re for e-commerce or also just thinking that way within the enterprise?

Steve: [00:39:43] Yes, I have a good tip, or maybe a resource more than a tip. I don’t know if you guys have heard of this. You probably have. I think a lot of DevOps people have, but there’s a website made by a guy that works at Heroku. It’s called twelfactor.net, and it’s this guideline of how to build apps that are reasonable, how to build apps that, like, scale easier than if you didn’t follow these instructions. Basic things like have your logs set up this way, have your server run this way, have your server, like, not daemonize itself, and, like, 12 tips that are all very, very simple, but if you follow them when building your app, you’ll save yourself, like, so much headache and heartache and probably hair a year from now when you do have to scale it. So it’s— I believe it’s twofactor.net. Yeah, it’s twofactor.net, made by a Heroku guy. It’s a great resource. You should definitely check it out if you’re going to build an app that’s going to have any scale, or if you just want to keep your sanity and have an app that’s not going to scale at all.

Matty: [00:40:49] We’ll put a link in the show notes. Igor, parting thoughts on scaling?

Igor: So I really wish I knew about that website. Maybe I could have saved some of my hair. So I probably will speak to one of the tips on the website, perhaps, although I’ve never visited the website. But one of the things I really preach when one is thinking about mega scale is, and as I already mentioned, you know, being able to take any layer and scale it out. And I think one of the largest One of the hardest layers to scale out is the storage layer. And especially for folks who come from the Microsoft world where SQL Server rules for storage or Oracle rules for storage. And when it comes down to large e-commerce, large social, large sites that are just handling a lot of traffic, relational databases are kind of a little legacy world, if you will, or unless you’re able to shard them out really easily. I think it’s the object databases where the logic of relations is now moved over to your application servers is really the key to think about and to plan for. Basically, Relational databases, relational servers are awesome because they’re giving you so much power, but there’s a single brain that’s managing your storage, and that brain typically cannot scale out. It can only scale up, right? You can only add stuff to, you know, you cannot have 5 servers do a join query for you. But if you are able to piece things together within your app layer and relieve the smartness from your storage and bring it to your app layer, and every app request is now smart and is able to, you know, bring in the data from various places together. And it’s, yeah, it’s a little more complex, but you’re able to throw 1,000 application servers at the problem, and you can’t really throw more than one SQL Server at the problem. You know, there is definitely a benefit to think about object storage and non-relational storage, NoSQL storage. So that’s my tip.

Matty: [00:43:07] Great. Speaking of tips, we’re going to move into the checkouts. This is the section of the show where the panels and the hosts— the panels? The panelists and the hosts give a recommendation for something to check out. It can be a technology, a book, a beer, a bourbon, show, anything like that. So we will start with you, Steve.

Steve: Yeah, sure. Yeah, I do. I have— this is a quick thing I’ve been just playing with this week and checking out a little bit. It’s a DevOpsy Plus app development tool, I guess, but basically the idea is called Consul, c-o-n-s-u-l.io. It’s like DNS meets configuration management, so you have some configuration management for configuring different endpoints, and it takes that and puts it into DNS so your application can basically be smart and know about your other applications, other app servers, without having to manually configure, like, a bunch of IP addresses. You just use DNS in your app. The what app servers are where kind of magically happens, and then it all works together, and it’s awesome. It solves a problem that I’ve been fighting with and getting upset about for, like, the past 6 months. So check out console.io.

Matty: [00:44:19] That’s a good recommendation. I knew it sounded super familiar, and it was a pick— it was Jamie Windsor’s pick on the most recent Food Fight show, too, so we totally ripped those guys off. But I’m excited to check it out. I was just listening to it last night. I’m like, that sounds really cool.

Steve: It is. It is truly as cool as it seems.

Matty: So Igor, check out.

Igor: Sure. So I recently discovered a tool that I— for a problem that I posted about on Stack Overflow. One of the things that I always want to do when I develop systems is being able to not just monitor them, without, you know, by some third-party tool, but actually build them so that they’re monitorable much easier, right? Publishing stats about how system runs so that those stats can be monitored is much better than just monitoring in blind. And it’s— when you’re developing, it’s really bad if you’re polluting your code with your logic that keeps throwing, you know, performance metrics out to somebody. It’s really not as elegant. And I’ve discovered a tool called PostSharp that basically will make your, at least .NET code, really, really easily monitorable. And, you know, it’ll trap errors, it’ll basically capture all various stack traces and publish performance counters. Obviously you have to code all that stuff yourself, but it is easily injectable in your code without actually polluting your main logic. It’s really cool. It’s a commercial product, and— but it’s— but I kind of fell in love with it almost right away. So check it out.

Trevor: [00:46:03] Great, which usually makes it be my turn. We make Matt go last so he feels bad and has time to come up with things.

Matty: I’m prepared this time, thank you very much.

Trevor: So I got a couple things this week. I’ll start out with my non-programming thing. A couple weekends ago was the Chicago Comic and Entertainment Expo, where I got— people who are watching the video will be able to see this, but everybody else is going to hear about it. I got Demon in a Bottle, Iron Man #128, signed by the artist Bob Layton, who is a really cool guy. He said he enjoys defacing people’s property with his signature. Which is really cool. As for tech stuff, there’s a really great blog post which describes a lot of the issues I’ve come across doing software development called Programming Sucks. It’s written by Peter Welch, and it’s just— it’s hysterical. We’ll post a link to it in the show notes, but it’s at stilldrinking.org, which is what most programmers do. As you’ve seen, everybody we’ve had on the show almost has brought some sort of alcoholic beverage along with them.

Matty: [00:47:16] That’s how we roll.

Trevor: And finally—

Matty: go right ahead.

Trevor: One I happened to stumble across today that addresses a problem I came across a while ago, and I wish I had known more about when I did it, was there was an article from Sparks Engineering about the problems with System.IO.Portable. Ports dot serial port and serial ports in .NET, and where some instabilities come into play using it. That’s too too complicated a URL to say, so I’m just going to put it in the show notes.

Matty: Sounds sounds good. Great, thank you. Okay, I’ve got two. So my first one is this this app that I got called the Hub. It’s an OS X app. It’s used. It’s a very simple. Kind of single-purpose app. It’s used for tracking GitHub repos, and I like it because I hate email notifications from GitHub because it’s kind of like all or nothing. So this just sits up in the status bar and lets me know when there’s new commits or comments or issues on repos that I care about, and it gives me little toast notifications and then gets itself out of the way. So it is called The Hub. It’s $1.99 right now in the App Store, but they’re threatening to raise the price, but we’ll see. And then my other tip or checkout is— it is about ChefConf, but it was a talk that our good friend Sasha Bates gave called Whip It Good. And it’s basically a talk she gave on how not to be an asshole on the internet. And everybody should watch this talk. So I will put up a link to it. And it was entertaining, especially if you like lolcats. But it actually really gave me a lot to think about. At first, I thought that nobody who needed to see this would actually go to the session, but I think even those of us that don’t think we’re assholes on the internet can walk away from this talk going, yeah, you know what? I think I’ve done some of those things. Highly recommended. Before I wrap it up, I want to let you guys know we are beginning the planning for DevOps Days Chicago. If you are in the Chicagoland area and you would like to be involved in planning of DevOps Days Chicago, We will be having a kickoff meeting. Go to arresteddevops.com/devopsdayskickoff, and there’s a survey you can fill out. If you are listening to this sometime after the middle of May, it’s probably too late, but if you’re listening to it now, come join us if you want to help us organize this conference, which should be great. So, super special thanks to Steve and Igor for joining us. This was awesome and nerdy. At the same time. Check us out on the web at arresteddevops.com, or you can follow us @ArrestedDevOps on Twitter.

Trevor: [00:50:06] I’m Matt, @MattStratton, and I’m Trevor, @TrevorGS.

Matty: We are Arrested DevOps, and remember, there’s always DevOps in the banana stand.