Runtime Analysis with Brian Kelly

Brian: [00:00:00] We won’t say the Log4j word. It’s like saying, you know, the word that Shakespearean actors, you know, should not say.

Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton. We have a great show to you. We have a great show for you today, or some other comment. But first, a word from our sponsors. Thanks to our sponsor Gliffy, the leading diagramming solution for teams using Atlassian products like Jira and Confluence. Drag and drop shapes to quickly build a diagram capturing anything from code structure to a simple concept. You can start your free evaluation by visiting gliffy.com/arresteddevops and signing up via the Atlassian Marketplace. That’s gliffy.com/arresteddevops. Get started today. So Uffizzi is a platform for platform teams. You can stand up your developer platform in minutes, not months. What I like about Uffizzi is that it gives platform teams control and dev teams autonomy. It’s Kubernetes native and extensible. So you can customize it with tooling that meets your team’s evolving requirements. And these clusters, they spin up fast, like super fast. Out of the box, Uffizzi combines a great dev experience, secure multi-tenancy, and cost efficiency. But try it out for yourself at uffizzi.com. Download their CLI and you can spin up your first sandbox cluster in under a minute on their free starter tier. That’s uffizzi.com, u-f-f-i-z-z-i dot com. Let’s face it, no one likes writing or maintaining documentation. But when you start a technical project or pick up a new task, missing information can cost you valuable time. GitBook is a technical knowledge platform that fills that information gap, making it easy for your team to capture, maintain, and find information from a single source of truth. For example, With Git Sync, you can set up a 2-way sync between your repository and GitBook, so you can turn Markdown files into awesome user-friendly docs. And if you make a change in your codebase, the edits sync between the 2 automatically. Or what about when you need to find something in that knowledge base?

[00:02:30] Forget about searching. Just ask GitBook AI. You’ll get a neat summarized answer that is sourced directly from your docs. These are a few examples of what GitBook can do, so why not give it a try? Head to arresteddevops.com/gitbook to find out more. We are going to dig into the topic of runtime analysis today, and I have just as many questions as you do as to what those 2 words mean when they’re put together. But luckily, we’ve got a great guest who’s going to help us understand it better. So welcome to the pod, as the kids say, Brian Kelly.

Brian: Thank you, Matty. Great to be here.

Matty: I realize I’ve done a little like making fun of calling it a pod several times, and then now’s the part when we’re supposed to riff about like Zoomer slang. And things like that. So if you’re doing the Arrested DevOps Diet Coke drinking game, you should take a sip of your Diet Coke right now. Brian, you want to introduce yourself a little bit, just a little background on yourself to our listeners?

Brian: Yeah, sure thing. So given my last name, Kelly, you can probably guess, maybe there’s a hint of my Irish accent left, but I’m from there, but I’ve lived in the Boston area for 20, many, many years, over 20. I won’t say exactly how many. I’ve been in software since, you know, the early days. I started sort of distributed systems and then moved into SaaS. You know, generally have loved my time with developer-focused tools. So, I was at a cybersecurity company that focused on that. And right now I am at AppMap and we focus on runtime analysis, which coincidentally is a topic we’re going to talk about. So, I guess that’s lucky.

Matty: [00:04:06] What are the odds?

Brian: I know.

Matty: So I kind of looking at the things we’re going to chat about with runtime analysis, obviously want to dig into what that means. But my understanding, imperfect as it is, is we’ve talked for a long time about static code analysis. This is very tried and true. You know, I don’t know if I would say old and busted, but definitely, you know, mature and common. Right. And then we think about observability, which is the new hotness.

Brian: Right.

Matty: And we can have many conversations about what that actually means. But so where does runtime analysis kind of fit into this? I don’t know if it’s really fitting into the spectrum, but kind of into the conversation. And what even do we mean when we talk about— I mean, I can guess, but I’m probably wrong.

Brian: You know what, Matty, why don’t we start with your guess? I would love to hear your guess.

Matty: I was going to say, my guess is it’s analyzing during runtime.

Brian: It sure is.

Matty: There you go. That’s the show.

Brian: Thank you. You’re absolutely right. That is exactly correct. And that’s what differentiates it from static analysis. The difference is where in your value stream, where in your workflow does it do the runtime analysis? Because the industry right now is very used to runtime analysis happening in production. That’s kind of what, you know, has led to the massive proliferation of APMs and observability tools. And runtime analysis is a new direction to say, you know what, there are things we can figure out, there are analyses we can do that happen earlier, and in less risky places, where the developer is still directly engaged at the source code level in their laptop, or at most in CI, before it gets deployed. For the simple reason that If things get deployed that are faulty, yes, you might have your canary rollouts, you might have your mitigation systems in place and your auto undeploys, whatever. But we all want to just prevent those things happening in the first place. So if you think of that kind of value stream all the way from someone starting on a branch to writing some code all the way through to a full 100% deployment in production, where are people getting the best data for whether or not that change is performant, secure, stable, reliable, all those things? Right now, they’re getting really good security information in static analysis. You know, we’ve kind of seen a really good commoditization of static analysis for certain classes of problems. Like, if you have a vulnerable dependency, you’re going to find it pretty early. If you’re just linking in some Log4j or whatever library, we won’t say the Log4j word. It’s like saying the word that Shakespearean actors should not say. But if you have a certain class of problems that can be detected very reliably by static analysis, you are delinquent in your job if you’re not using those things. But there are classes of problems that developers assume are only catchable by sending the code into production, putting real heavy traffic onto it with real-life customer usage and PII and everything else, and then watching it with their observability tool. And runtime analysis says, you know what, we can do that earlier. We can actually shift I hate to say shift left, so I’m going to say shift closer. We can shift closer to the developer and do similar things. Obviously, it’s not— it’s never going to be the same as production, but you can do— there is a set of things that you can look for when the application runs. Now, where is it running? It can be running in, you know, on the developer’s laptop. In their local environment, or it can be running in a sort of a shared sort of common build system like your CI, GitHub Actions, Circle, wherever, you know, whatever system you’re using. But wherever you run your tests or even have manual QA and people clicking around to verify it, that is generating runtime data that is essentially ignored for a lot of teams. And they’re just missing this opportunity to like catch things That are absolutely catchable. And instead they send it to production. And what happens then? The developer’s like moved on to another PR. They’ve moved on to another thing. If you’re really lucky, you’ll find out in minutes if your deployment has screwed things up. If you’re not, and you find out like 2 days later when some DevOps person is like, oh, I’m going to click deploy on that batch of stuff now. So you’ve moved on to something and you found out in a really latent fashion about some problem. You’ve context switched. You are— your brain is elsewhere and you’re being told, hey, that thing you changed a few days ago, that’s now problematic. That is like super inefficient and has been proven in studies. There was a couple done recently, you know, to sort of prove out that if an issue goes into production, it is super less likely to actually get fixed. What’s more likely to happen is it goes on the Jira backlog. And then you know what happens, of course. This is why we have crazy upside-down hosting costs now, is they compensate with, well, throw another couple of CPUs at it, throw another stick, a few sticks of RAM in the cloud sense. And that is what gets done as a mitigating factor by some poor SRE who has to deal with it and doesn’t know why the problem is happening. So runtime analysis is saying, you know, this is kind of not a very optimal flow. Why don’t we get whatever classes of problems that we can find that are only detectable at runtime, so not, you know, not catchable by static analysis, but do that as early as possible and as close as possible to when the developer is literally living in that code, in that branch, in their editor, or in, you know, watching a PR, or in fact, if they’re a reviewer, looking through a PR, trying to cognitively map in their head the lines of code that they see in the diff, trying to figure out, is that going to collapse the company? Is it going to cause a massive, you know, PII leakage with some vulnerability, or is it going to be fine? That’s what runtime analysis aims to do.

Matty: [00:10:53] It’s interesting because one of the things that came to mind, and I run into this quite a bit in certain conversations, I think our engineer brain doesn’t like— we tend to, just as humans in general, maybe nuance is hard for us. But, you know, kind of you talked about saying, okay, so this, we’re not going to be able to completely replicate contact with the real world, you know. And I think sometimes we go down this path of, well, if I can’t get all of it, then it’s not worth doing, right? And it’s like, well, it’s a little bit of whichever metaphors we want to abuse about layering Swiss cheese or doing whatever, it’s like, okay, but cool. If we minimize, yes, there’s still going to be— we are not going to use these practices to ensure that we never have a security incident in production again, right? Or never have a performance issue in production again or whatever. But we’re saying, let’s get the 80%. Let’s get ones that maybe it’s a little bit easier to be able to mitigate the things in production because now there’s not hundreds of them. That all go into the Jira backlog, they’re, they’re fewer. And then maybe they actually get those— maybe that— so I mean, there’s some potential for actually making even stuff that doesn’t get caught before production to actually get fixed. Because when there’s less of it, it’s not just spiking on top of this, this never-ending, you know, well, here’s the next one, let’s just put it on top. You know, it’s sort of like Homer Simpson, like balancing the, the garbage to see, you know, they’re all, they’re all saying, like, who can Who could be the, you know, the one sort of doesn’t overflow so they don’t have to take out the trash? Like, that’s got to go in a deck somewhere. That slide.

Brian: [00:12:28] I love that reference. That’s a great one. And you know what you’re reminding me of is like the fact that when things like Dependabot or any of the really decent, you know, static analysis tools really upped their game and sort of solved a lot of things for developers, it didn’t suddenly end pen testing. It didn’t suddenly end dynamic application security testing or interactive security tests. There’s a place for all of these things. And developers need a signal. They need signals from each of these domains. The problem is right now there is an overreliance on observability tools to get data that honestly, A, we know academically and technically we can get it earlier. But it’s just easier to just crank up your spend on the APM and say, well, let me just do 1% of a canary rollout kind of thing here. And yeah, that’s basically the same, right? And no, it’s not. You know, you’re literally exposing customers to potentially flawed software. That’s not a good practice. And you can go faster and avoid things going wrong much more often. If you just tackle it a slightly different way.

Matty: [00:13:46] What are some of the places? Because I know one of the benefits or one of the things that I think you all have seen thinking about runtime analysis, thinking about this, is that there are types of defects or flaws that historically we might say you need a human to figure that out. That’s only going to be if a person looks at it or a person experiences it. The robots can’t help us. It seems like the robots can, but what are some of those things that maybe has like conventional wisdom is like it couldn’t be automated, but maybe you’ve seen that to be not true?

Brian: Yeah, so there’s a few that come to mind. So the most sort of obvious would be things like N+1 queries, which are coming about because you’re using some kind of object-relational mapping, you know, some kind of database magic layer that’s creating queries for you. Static analyzers are— they’re just not going to see that. They’re going to know that you’re using an ORM. They’re going to know that you’re using Hibernate or Rails or ActiveRecord, whatever. But they’re not going to see the queries that are actually generated and issued at runtime, or how many identical queries happen as you paginate through something in a sort of naive fashion because ORMs are They’re great, but they’re not, you know, they’re not DBAs. They’re not magically creating the best query interactions for you. So that one is kind of like the, the sort of the easiest example that everyone would give in the runtime analysis world to say, you know, you got to see how that resolves at runtime. And then you got the obvious things that follow on from that dependency injection or dynamic library loading. Your static analyzer. Will know that you’ve gone to a— that you can read a config file and that you can load a library, but does it see what gets loaded at runtime? No. And that once you open that door to anything can be loaded in, you absolutely need to see what gets loaded in. There’s no question about it. It’s going to get resolved in production no matter what. Those libraries are going to get loaded. Those queries are going to get issued. But the infinite possibilities is just too great for any static analyzer to solve. Runtime analysis says, all right, well, let’s watch where the infinite gets constrained down to the finite, to a reasonable set of things. And we can say, okay, now we see what libraries really get loaded. Yeah, you might have all these things in your— you might have all these JARs or DLLs or gems or whatever listed, but you might not even be using 7 or 8 of them. There’s a big difference between Dependabot screaming like, oh, you’ve included this in your gemfile, and actually using something. And not only that, but are you using it in a vulnerable way? So these are the kinds of things that can only be determined at runtime when you see code being actually executed.

Matty: [00:16:51] Yeah, that’s interesting because I remember when we had an episode of the show a few months ago with Michael Izbicki from Sysdig, and we were talking about what they saw in the Cloud Native Security Usage Report. And one of the things that really jumped out to me was how they saw so many JavaScript libraries that were loaded but never used. Right. Right. And this kind of a thing. And I can see how that makes it really challenging to sort of sit there and say like, is this actually doing something? Is this actually, like you said, am I just getting screamed at? For something that isn’t even there. Um, and, and so, you know, we talk about that this can help like identify performance issues. Um, but I mean, there’s a reason like static code analysis is fast as hell, right? You know what I mean? So how— when you think about this, we’re saying we have all this goodness of doing all this, this analysis during the runtime. How much does this really— I mean, like, kind of walk me through where this goes into my cycle as the developer, as the coder, like what’s my day if I’m doing all this? And now am I, are we that, you know, XKCD comic and we’re like, oh, sorry, what are you doing? You know? Oh yeah, runtime analysis, you know?

Brian: [00:18:06] Yep. So if there’s one thing that, you know, our class of vendors have learned is that, you know, you have to be automated, you have to be fast, you have to be reliable, you have to slot into the development workflow. With, you know, with ease. Because if you say, well, hang on, here’s a new step, or you got to wait, you know, an extra hour, like, that’s just not going to be adopted. That’s the end. You know, do not pass go, do not collect $200. So, what run— you know, high-quality runtime analysis tools will essentially adapt to the developer workflow as it already exists. So, for example, you’re working in your favorite editor, be it VS Code, IntelliJ, PyCharm, whatever it is. And right there natively in your workflow, you’re using your sort of the run button or your favorite keystroke. It just automatically instruments your application for runtime analysis. And that means there’s some level of low-level sort of byte watching, bytecode watching in the case of Java, or whatever interpreter or whatever language you’re talking about, the environment that your code is running in, there’s something watching, which your brain is probably thinking, Brian, that sounds just like an APM. That sounds like an observability tool. And that’s kind of right. It’s different. It’s not like aggregating data. It can do much more specific data. So, you know, it’s somewhere between what you get from an observability tool and all the way down to, you know, profiler level. So the runtime analyzer is collecting information as your tests run or as you run your application and click around it. And it’s basically saying, hmm, I’m building this corpus of knowledge. I’m building a dataset of knowledge of how this application truly runs. And I can see which functions get called. Which queries get issued, which libraries get called, and which ones don’t. And I can build up this picture, which can then be visualized, it can then be analyzed, it can, you know, you can do all kinds of things, but that data becomes the foundation for the analysis. Now, the analysis can be— there’s 2 sort of buckets of analysis. One is automated analysis. So, just like an observability tool might say, you’ve just, you know, exceeded 50% CPU usage on this cluster over here, and that’s, you know, you’ve got an automated alarm there. There’s automated analysis in the runtime world, and you’ll see things like, hey, we detected an N+1 query. Now it happens, you know, it’s like it happened and it was like 6 occurrences or 7 occurrences, and it’s like 0.00001% of your overall, you know, CPU time, you know, or whatever. Proportionally, it might not be terribly worrying. But the developer knows, hmm, that was just that little unit test that I ran or that little kind of smoke test. And I know if I deploy that, the dataset that query is going to run on, it won’t be doing 6 queries, it’ll be doing 6 million. So I can instantly extrapolate that. So you combine that kind of automated analysis where the analyzer is saying, I saw a pattern, It’s concerning. It’s also definitive, right? It’s not a— because it was a runtime analysis, it’s like, this really happened. This isn’t some like predictive fuzzy thing where it’s like, you might have a problem here. Like, no, you have this thing. And now it’s up to you as a coder or potentially the person who’s going to look at your pull request as a reviewer to say, I think this is a problem, you know, that will need to be fixed because I don’t want this going into prod. And that’s everything that runtime analysis is about, is giving developers that little extra information that they just don’t get at that point in time.

Matty: [00:22:05] What are some of the things that maybe might cause someone to not take on this way of work? You know what I mean? Like, what are some of the barriers, real or perceived, that you see around bringing this into a lifecycle?

Brian: Sure. I mean, there’s resistance to change in pretty much every workflow. So, The things that might give someone pause would be, well, the first thing they’re going to think of is like, is this just going to be noise? Am I, you know, because they all adopted, you know, variety of analysis tools, like static analysis tools. And most of them saw them in the early days, when they were getting things really wrong. And there was a lot of false positives. And there was a lot of just like, you know, eye rolling to, you know, people’s necks were getting hurt by just like, you know, just gasping. So, the first thing they’re going to be concerned about is noise. And that’s usually not a problem with runtime analysis, because by definition, it’s watching things run, and it’s not having to extrapolate it or predict. It’s like, it’s observation and saying, you know, this thing really did happen. I guess the other thing that might concern people is, you know, you have to actually run things to do a runtime analysis. And there is a bit of an expectation of magic, where static analyzers right now, they just suck in your repo. God knows some of them just send those repos, send all your code right up to some server somewhere, and it does some magic, looks at it, probably superglues on some GPT onto it, and then out pops a little static analysis report. But the speed of those static analysis reports has been increasing over the years, and they’re pretty quick now. And they also just don’t require much intervention. Turns out runtime analysis actually is really fast because you’re bound by the length of time it takes to run your automated tests, which you’re doing anyway because you’re a developer. Hopefully you are writing some tests. Hopefully at some point in your workflow you have tests. But the thing I guess that would give someone pause— oh, I thought of another reason and it just flashed out of my brain, but Yep, it’s gone. But you know what, the— I think the reason they gave it probably good enough anyway.

Matty: [00:24:20] It’s going to come back to you in about 10 minutes. So podcast listener, just fast forward 10 minutes from now and Brian’s gonna say something real smart, I promise you. So those are a couple of things. What, what, what are we seeing, you know, in, in maybe some ways that the industry is sort of reasoning about security and vulnerabilities when we’re kind of moving this closer, as you said, and where that goes? What are, what are some of the industry shifts and maybe conventional wisdom, if you will?

Brian: That’s a great question. So I’m going to refer you to the classic OWASP Top 10 for the answer to this. And I love whenever this comes out, you know, every couple of years we see it. And if you look at the evolution of the OWASP Top 10 and go back in time, if you sort of contrast, like, I don’t know, The first time I looked at it, I think was probably 2010. And right at the top was SQL injection, if I remember correctly. And why was that a problem back then? Why was it the number one problem? It was because there were really no tools there. I mean, people were totally used to just crafting strings of text for their SQL statements and sucking user input from a form on the web. Concatenating it. I mean, just, you know, it’s kind of embarrassing to think of it now, but that was the state of the industry. And what happened? A, we got good at detecting that problem. B, we got better tools to mitigate it and just avoid it in the first place. So I’ll never forget my first, you know, big pen test result. It was from a big bank, and the report was 33,000 pages long. Now, it was an automated tool, right? Which is like the very definition of the worst kind of pentest report. It was just like a lot, just immensely repeated stuff. But we fixed almost like, I think, 27,000 pages of that report by shifting to prepared statements, a thing that just guarantees that the user input can’t be used to influence the query. So, in a very short period of time, OWASP Top 10 issues became, you know, the ones at the top 10 back in 2010 got fixes that were very commodity, you know, solutions in the static analyzers and in the developer frameworks, you know, their libraries, their database, you know, ORM packages, whatever it might be. And what replaced them were the kind of harder problems. So, if you look now, The things that are at the top of the OWASP Top 10 are not, by and large, they’re not detectable by static analysis. And that should be worrying every developer because where are you going to catch those? You’re going to catch it in prod? Are you going to catch it, you know, hope that your senior developer with all those seasoned years of like, you know, you know, the sort of grizzled person who’s just like war weary is going to catch it, you know, by looking at your diff. In your pull request, but you’ve got things like, you know, your security, you’ve got broken security authorization and authentication ordering. You’ve got, you know, things related to cryptography. You’ve got things related to secrets handling. And you might think, you know, oh, well, of course I would never log a secret. Well, you might not, but the framework that, that data is passing through, those, that secret’s going to go somewhere. Something’s logging some, you know, you hit them all the time. Like, this thing logged it. I wasn’t expecting it to log it. Oh my goodness, they turned up the log levels. They shouldn’t have done that. Well, you know, that’s what a runtime analyzer can find for you is find those code paths and tell you maybe you shouldn’t have them here.

Matty: [00:28:07] What are some of the ways to even think about getting started? Right. You know, again, because, you know, sometimes we sit and we say we’re changing the way that our entire development team is working or whatever, but that’s a, that’s a big swing. But if I’m listening to the show now, I’m like, okay, like, what are the— not even necessarily the baby steps, because we also as engineers tend to like try to make everything as hard as possible, right? We’re like, let’s find like this gnarly, hypercritical application, which I will say sometimes that’s a good way to do things. I remember anecdotally when Adam Jacob and the folks at Chef were putting together the product Habitat, And Adam’s approach was like, for their first proof of concept of installation, he said, I’m going to think of the gnarliest enterprise software I can find. And if I can make Habitat install it, then it can do anything. I don’t know that I recommend that approach for things. So let’s say you’re not Adam Jacob. Let’s say you’re not trying to go the hard way. Maybe we want to look for some, some initial success patterns. Like how do we, how do we kind of get ourselves going? What’s something someone can walk away today and maybe go and try?

Brian: [00:29:12] That’s great. Yeah, so there’s 2 avenues they can take, and it splits apart. And by the way, I remembered, of course, the thing I was trying to remember 10 and a half minutes ago. And the thing was—

Matty: I think it was almost about 10 minutes ago, too. I think I’m gonna check the timecode when this comes out.

Brian: And you know what it was? It was, do you have automated tests? Because if you don’t have automated tests, runtime analysis is going to be a bit of a challenge. And I thought of it in this answer because if someone’s If someone wants to dip their toe into runtime analysis and they don’t have tests, it’s a little more challenging, right? But I’d also say if you don’t have tests, runtime analysis probably, you know, it’s not going to be the only problem you have. I hate to be mean, like, because there’s lots of companies out there that have, you know, we’ll call them heritage code bases that, you know, they just, they’re really hard to, you know, build tests around and they sort of supplant that with humans and Upworkers or whatever, and offshore teams that are just banging on the thing 24/7.

Matty: So the first thing you can start by doing is not have old code.

Brian: [00:30:17] Well, here’s the thing. The reason I mention it is because it kind of dictates which path you take. If you want to put runtime analysis in CI, you got to have tests. They’re very definite. If you want things to run and you want it to be sort of a reasonably fast thing that gives you quick feedback, then, you know, having a test suite that runs your stuff in CI and verifies it while it’s packaged and before it’s deployed, you’re just going to need that. That’s— there’s no way around that. But if you don’t have tests, you can still take the other path, which sort of puts you in and gets you a ton of value, which is instrument locally, you know, get runtime analysis running wherever you can run your application, be it your laptop or some UAT environment or wherever, and instrument it there, generate your runtime analysis dataset, and, you know, visualize it, you know, look at the reports and flaws that are found, and start that way. It’s kind of like, you know, you could think of it as personal observability, right? That’s a term we’ve used at AppMap, which is all the benefits of like seeing how something runs in a very safe place, And, you know, well before it becomes a problem. So if you think of it that way, you know, it makes perfect sense because no one’s going to say to you, you need tests to run in production. Of course, it just, it just runs. Your tests are your customers like hitting the thing.

Matty: [00:31:45] Uh, monitoring is simply testing with a time dimension.

Brian: I love that. So exactly, you know, if you have tests, you can bring it local into your editor, put it in CI, you’re off to the races. If you don’t have tests, Just ignore the CI side for now. You should still go write tests, of course, but, you know, go at least get some runtime analysis data wherever you can run it and exercise your API or your UI.

Matty: From a legal perspective, we’re not allowed to have an episode of a tech podcast where we don’t talk about AI, I guess. So, but seriously, like we sort of talked a little bit about even leading into prompt engineering and where these things go into play, but where are some of the really effective places that you’d see as this continues to grow and continues to go? Where is the role that AI, in whatever framing of that term, do you see as kind of playing a role in this?

Brian: When I think about AI, I think about what is the dataset they’re working on, the prompt, their own trained model, etc. You’ve no doubt seen the plethora of AI code reviewer apps that are out there. And most of them are not very novel or interesting. They’re just like, you know, sticking some kind of LLM onto a pull request and looking at a source diff. And, you know, that’s fine. I expect everybody to be, you know, doing something in that regard. The source data that those things are using are the same. They’re all just looking at the same diff, a static view. Of some source code changes and also the whole repository source listings. What runtime analysis does is it gives you data that does not exist there. It’s new data. It’s a new dataset, which we have experimented with at AppMap, and we’ve used to see, you know, how different is the kind of experience and the kind of quality of analysis when you take the LLM trained model, right? You take your repository of code, the diffs, you know, everything in your pull request, and you layer in the runtime analysis data that you got, that just like amps up and levels up entirely the kind of value you get. It strips away all the noise because you’re not dealing with false positives, you’re dealing with real issues. But we were able to craft some like really, really amazing— well, I guess we crafted prompts and we got some amazing responses. Where we’re like, hey, we have an N+1 query here. Can you tell us how to mitigate that? And it did. But you only knew about the N+1 query because the runtime analysis platform that we have informed us of that fact. So, it just constrains the problem space dramatically and gets you to the stuff that matters about, which is what every code reviewer wants to do is, Is this thing fit for production? Is this going to break or not? And the runtime analysis plus AI side of things has been just— it’s been kind of magical to watch. So it’s been a lot of fun.

Matty: [00:34:53] So this is my favorite kind of question. I think we can have a little bit of fun here. How are people going to screw this up? Like, like, what’s the anti-pattern for runtime analysis?

Brian: Oh, that’s a great question. Well, if we look at how people screwed things up for other domains and other tools, right? If we look at the DevOps hangover, as we’re calling it now, which is massive overspend on observability tools, you could, I mean, we, I mean, commercially, we’d be happy if everybody overspend on runtime analysis and reduce their spend on, you know, their APMs and observability platforms. But I guess there’s a point that, you know, it would be if you were over-reliant on it and not shipping code, then you’ve kind of, you know, it’s become counterproductive. I’m struggling to think of a really good, like, oh yeah, aha moment. Yeah, Brian, that’s exactly what would happen. And that’s really bad, like, because it’s a nascent domain, you know, it’s an up-and-coming kind of area of software analysis and software development as you know, as it pertains to like the general day-to-day workflow. So I will— I’ll probably be like sleeping in bed tonight going, goddamn, I just thought of the best example for the anti-pattern of runtime analysis.

Matty: [00:36:15] I don’t know, what I, what I hear here is that it’s impossible to screw up.

Brian: You can’t go wrong.

Matty: It’s, it’s, you know, there’s no mistakes to be made. Well, you know, it sounds perfect.

Brian: It could— maybe you could overstep its bounds and try, you know, to overpredict things in production and try and become like production light, or excuse me, observability light, and become like— if it becomes like a mosquito in your eardrum, like this, and we saw that with static analyzers when they first came out. They were just like, oh, great. Yeah, I’ve got 79 vulnerabilities. Great, thanks. And what happened? That amazing moment when Dependabot started creating PRs to say, look, I’ve already got this thing upgraded for you, just approve the PR. That was like a way to mitigate that overstepping of like just being too noisy, too interfering. And suddenly it was like, okay, your tool is part of my team now. People already feel that production tools like their APMs, their observability tools, they are part of the team. Like, they feel like really in there, even though developers probably have to like use some tool to get access to it, you know, keep themselves compliant. And, you know, there’s all these restrictions around them. They still feel like there is a— they have knowledge of how things run in production. There’s, you know, there’s all kinds of scope for runtime analysis to like give them that same level of data. But of course, anything can be overwrought or overused.

Matty: [00:37:49] This has definitely opened my eyes to a few things that I was unaware of, like all of them. We’ve certainly learned a lot about how we can— I don’t know if we can be more secure, but we can at least know when we’re not being secure. Speaking of security, my dogs are very concerned about the security at my house right now, so I apologize if you’re having a hard time understanding me, but Moxie and Cody have a lot to say about runtime analysis. If you would like to see what we have to say about runtime analysis in our show notes. If you go to arresteddevops.com/runtime-analysis, that’s where you’ll find them. And at arresteddevops.com/itunes, you can leave us a review in the iTunes store, which helps other people find the podcast apparently. If this is your first time listening to the show, you can find us on Spotify, iHeartRadio, Audible, and all sorts of great places to find other great podcasts. Brian, thank you for joining me on Arrested DevOps today. This has been really fun and interesting for me. Hey, my pleasure. Glad to have being here.

Brian: It was a lot of fun.

Matty: And as always, this is Arrested DevOps, and remember, there is always DevOps in the banana stand.

Brian: [00:38:53] Did I get it right?