CI/CD as Control System with Naga Sujitha Vummaneni and Sundeep Bobba

Matty: [00:00:00] It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I am Matty Stratton. have a great show for you today, or as my notes say, or some other comment. But first, let’s hear from our sponsors

Speaker: Today’s episode is sponsored by Attribute. I met the team, saw a demo, and honestly, this approach is quite clever. They call it FinOps without tagging. It’s the first FinOps runtime technology that analyzes cloud cost based on infrastructure traffic instead of relying on billing reports or tagging. For teams who need visibility into per team, per product, or per customer cost, Attribute enables this visibility with one line of code.

Think of a list of your teams or products or customers broken down to RDS, BigQuery, [00:01:00] Kubernetes, OpenAI, data transfer, and over 35 multi-cloud services based on actual usage, fully automatic, no spreadsheets, no tagging. They’ve been recognized in six Gartner hype cycles and are working with companies like Akamai and monday.com.

Arrested DevOps listeners, reach out by the end of 2025 and get their highest tier at the price of the base one for the first year. Check them out at arresteddevops.com/attribute

Matty: great. Uh,

Joined today by two great guests. We’re gonna be talking about CI/CD as a control system, and I think this is super relevant still, which is interesting when we think about the inflection point of when DevOps really started. It was a bunch of things that all happened at the same time, and one of those key things is the book Continuous Delivery by Jez Humble and Dave Farley. And here, going on whatever, 2026 minus 2009. What is that, 15 years? Are we 15 years? How many years? Are we almost 20 years [00:02:00] of DevOps? However many years and years later, this is still super relevant. But before we jump into our topic, I think it’d be great for let our two guests introduce themselves, and then we’re gonna get right into it.

Sujitha, you wanna go first?

Naga Sujitha Vummaneni: Yeah. Thank you, Matty. M- yes, I’m Naga Sujitha Vamaneni. I work for Ripple, and I work f- in the field of DevSec op– MLOps. I I do work on cloud infrastructure architectures and model developments and autonomous automated workflows

Matty: Great, and Sandeep tell the audience a little about yourself

Sundeep Bobba: Hey, Matty, thanks for having me, and thanks for doing this. So excited to have here in the show. I spent like last 16 years working across cloud DevOps, platform engineering, and large-scale software delivery. My name is Sandeep Bobba. I currently work for Southwest Airlines as a technical leader in cloud and DevOps.

Most of my work has been around building and modernizing CI/CD and DevSecOps platforms for complex enterprise environments. It was [00:03:00] understanding, like the book is all about understanding how entire delivery system behaves, how quickly we get feedback, and how we keep it stable while change is happening continuously.

So the book came directly from the problems we have been seeing and solving in real-world software delivery environments

Matty: Fantastic. Yeah, and I think we wanna dig into all of that. One thing that we can lay a little groundwork just for, as, as is always fun in our industry, there’s, we talk about CD, and it could actually mean two different things. Sometimes it means continuous delivery, sometimes it means continuous deployment.

I think for purposes of our conversation, it doesn’t really matter so much, right? The biggest difference is that last gate a manual gate or is it an automated gate? But thinking about that, and in fact maybe one thing, I have some questions, but maybe a good place to start is you said there were kinda three main areas that you identified and maybe we could kinda start by talking about them at a high level and then dig into each of those

Naga Sujitha Vummaneni: I can tell you the main idea behind the book. And it is [00:04:00] actually might be relatable to you. In the Arrested DevOps episode itself with Hannah Foxwell and Robert Warner. Once Hannah made a point that stuck with me quite deliberately that she remembered enterprise clients insisting continuous delivery would never work at their company, and now it’s just how everyone ships. The same thing happened with cloud and agile too, right? So that’s the pattern we kept hitting while writing this book. So the technical capability shows up years before anyone’s willing to use it. But I think it’s now the time to act on it because agents are moving at machine speed. We will need to be working on altering the current CI/CD workflow and systems.

Matty: Yeah, that, that always reminds me that, that story about say it couldn’t possibly work for me and I’ve talked about my sort of DevOps origin story of listening to a podcast that Jez Humble was a guest on, and he was talking about delivery, and I was working [00:05:00] at a dotcom s- like internet company at the time going, “There’s no way this could work.

There’s no way this wouldn’t work for us,” and blah, blah, blah, and then he starts talking about HP LaserJet firmware shipping with continuous delivery, and I was like, “Oh, okay.” But yet again, here we are almost 20 years later, and the ideas are maybe there, but, the best laid plans. One of the things that, that comes up is when we think about… Like you said, we have to move at agent speed as well, right? We’re running into things moving a lot faster, and these, these agents are probably having more requests, more things that have to be reviewed, and I wonder if we run into a little bit… This is a little bit of anecdata, but I wonder if people are better about doing reviews of code when it’s an agent that did it for them, or if they become a little less likely to do that.

I don’t really have any data to back that up, but I’m curious what what we might have seen.

Naga Sujitha Vummaneni: Yeah. Most pipelines are open [00:06:00] loop. They measure everything and on nothing. So the– that’s the automation with a human as a controller, right? So the control theory has a name for every part of what we are already building. The pipelines are activators, observability is the feedback signal, policy is the constraint, and deployment strategy is how you regulate the risk, right? So once you see it that way, lot of problems that look like tooling problems turn out to be like a system behavior problems. So here, a majority of the people, like even in my current company, like they’re working on the autonomous review agents so that it reduce the work for humans to review lakhs of ri– lines of codes. But we are doing is like we will take up the agent-driven review workflow as a summary and add the [00:07:00] human layer on top of it to final review so that it doesn’t compromise the authenticity or the ownership of like what is moving to the production gate and what we are passing through.

But I will allow Sandeep also to talk on this, about his experience.

Sundeep Bobba: I think if agents are creating more pull requests, more infrastructure changes, then the organization is making far more decisions per unit of time. At that point I think relying on humans to deeply review every single change doesn’t scale well. And you can also get rid of automation complacency where people assume that agent probably got it right.

So I think the answer is not remove hum-human review, but make review more risk-aware. Like low risk reversible changes can be validated heavily, like using automated tests like OPA policies and runtime feedback

Matty: Yeah, every time we have a conversation about AI and DevOps, and it’s always about like automated [00:08:00] remediation and all of this stuff, and it was like you don’t necessarily…

First of all, we’ve been trying to solve that automated remediation problem for decades before there was DevOps or AI or anything. But I always think about how it’s a useful thing to let the robot do the busy work for you so you have the information. So for example, you might say, “Hey, PagerDuty goes off, and before I get on the call, automation has gone and collected logs for me and done some initial analysis, so now I can use my human brain.”

And is that kind of the same idea here where we’re saying let the automation do this, but it’s not making the decision? And I think that’s it goes back to that classic like thing from the IBM manual in 1960 whatever, which says, “A computer should never be trusted to make a decision.” Like I don’t know how true that is literally, but what are maybe like conceptually we’re saying, “Okay, we can let this automation provide the human with information,” but how… are some things we need to do to walk that line about what is the thing to let [00:09:00] the automation do versus the point when it needs to become interaction?

Naga Sujitha Vummaneni: I think yeah, I think firstly we will need to insist some approval gates, I believe. Those approval gates are not just for the technical reasons, but they’re but they will be helping someone’s like what they can do and what decision they can make, right? So automation removes most of the manual work, but the approval is still left to the human. Not just for the accountability structure underneath, but it’s it’s because we will need to redesign the workflow, what it has been for decades, and what it has been running through just one stage of security scanning and then making a decision. But who knows the attacker can step in in what stage of the pipeline, right?

Sundeep Bobba: The next level is what I [00:10:00] would call bonded automation. I think the system can take action, but only inside a very clearly defined operating envelope. For example you wanna restart an unhealthy service, scale capacity roll back a cannery or quarantine a known bad artifact.

Like mainly the artifacts that are understood, measurable, and reversible. Like then there are high impact decisions, right? It comes down to changing security policy, could be modifying production data, altering infrastructure across multiple services or, taking an action on something which has large customer blast radius.

I think that’s where the threshold for human involvement should go up significantly. So I think the question I would ask is not simply can AI do this? I would ask four things. How confident are we in the signal? What is the blast radius? Is the action that is reversible? And who owns the risk if decision is wrong?

I think this [00:11:00] directly relates to our control system theory. So to your point, Matty, it’s this book is going ages back and then exploring the control system theory concept and then how we can model it and use it in the modern era.

Matty: I wrote a blog post a, a couple months ago that was a little bit of a tongue-in-cheek way. It started where I said how one of my coworkers who didn’t know what the CD command was can ship to prod. And it’s, and I had her full permission to say that. It was a joke, and it wasn’t, being super snarky. But part of it was interesting because it’s hey, this is the same kind of stuff we’ve been talking about forever. We have the same pro- it’s interesting, the same thing with documentation.

It’s like, turns out the best way to write docs for agents is write really good docs, right? That you should’ve been doing for people. But it becomes a little bit of a problem because you need extra guardrails, right? And it, and I thought it was an interesting analog because it goes back to the trust but verify way we used to think about things, where we would say, “Okay, we’re gonna run all these checks in the pipeline. We’re gonna assume good intent. We’re gonna assume [00:12:00] that the developer ran all this stuff it- themselves before they committed and pushed it up. But we’re gonna run it anyway.” And necessary because, as we know, you can give agents all the instructions you want in the world, and they will still happily go and just decide they don’t need to do them. I’d be interested to know how many people feel like they’ve solved their agentic security problem by writing something in Claude MD that tells Claude to always run GitLeaks before it does anything. Guess what? It can… How many times have we done this and you say, “Why didn’t you run this?”

And it goes, “Oh, you’re right, I should have.” But when we build these guardrails, like, how do we ensure… And this is not a new problem, but it seems like it’s an exacerbated problem in the AI world. How do we make sure that they don’t get bypassed, that they don’t just get skipped, right?

And we just go, “Oh, it’s important.” Just as much as, as Codex could tell you, “Oh, I should have done that,” many developers do we know, how many times ourselves have we gone, “Oh, you’re right, I really should have run that review”? What are some of the things to think about, either technically or, to me, these feel like they’re more organizational problems too, [00:13:00] to ensure that these checks, these…

Like you said, you have these different kind of angles of understanding. How do we make sure that they actually happen?

Naga Sujitha Vummaneni: The trust here it’s not just in the automations, but it is also based on the preconditions that we set up for the deployment all over the pro… Means, I call it as a progressive deployment, right? Making sure that the security is ensured at the every stage of the pipeline. We ha- the set of pre- preconditions that we could set up is as Sandeep told before that a signal that you do bet on if the rollback trigger fires on a noisy metric what does the automated rollback is like, right?

Nobody trust it. It’s worse than none. So you get you get you get flapping around it, right? So the signal quality is a gate. It’s not just like a an inbuilt process, but you [00:14:00] consider that as a gate. And you set a point, like a set point, so where the the health i- health of the process matters here.

When you make a judgment like irrespective of the time that you wake– you are being waked up by the pager duty call, like you should be able to make the right judgment. You can’t rely on the autonomous agents to make the right judgment because most of the autonomous agents, as you told, like they might miss the context itself, so like they don’t know what they’re doing too, right? There I think it comes to set up a bounded blast radius. So here the progressy del- progressive delivery matters and also like nobody trusts the auto rollback process across the whole fleet in one day, right? They shouldn’t do. And and also the, the reversibility matters. So the…

in the book that we have co-authored, the CI/CD control sys- as a control system, [00:15:00] so the main thing that we have mentioned is the feedback and the reversibility, right? So when something happens in the b- in between the pro- pipeline, although the securities checks scan, but if still the pipeline misbehaves or like it is giving something that it shouldn’t, then it should definitely recall every stage and be au-automatic rollback. And we also need to do the post-mortems, right? So the logs are crucial in this decision-making. And also make sure that the system is not too aggressively correcting or too aggressively roll– making the rollbacks. It has the rate limit checks and also some cool downs and automation correction correction safe.

So we need to ensure that all these preconditions are set up so that we don’t blindly trust the automation workflows, but we also make sure that it is going in a controlled way.

Sundeep Bobba: Yeah. I also think about it [00:16:00] in layers. It all comes down to prompts tell the agent what good behavior looks like. But platform controls enforce the non-negotiables. Like runtime feedback tells us whether these controls are actually producing the outcome we expected. And Matty, I agree with your organizational point.

You also have to decide who owns the control. I think AI is more organizational than technical to me. Like who owns the control, who can change it, what evidence proves it ran, and then what happens when it fails, and who is allowed to ac-accept an excep-exception. All of these need to be documented, and all the stakeholders need to be on same page.

Otherwise you end up with a lot of security tooling, but very little control. Like you have AI, but your hands are tied, you can’t do anything. So yeah, teaching the agent with instructions and then risk boundaries are very important

Matty: I think a lot of this reminds me of some of the early [00:17:00] days in DevSecOps when we first talking about shifting left, and then I got angry about it because everybody was like, “Let’s just shift the work to the left,” and we’re like, “No, you gotta bring the people with too.” But we used to sit there and say “Hey, if you care enough about something being a blocker or whatever, then you should care enough to write a test,” right? This was the, the advantage of the whole theory of tools like InSpec and stuff, right? Which could be, hey, we write our security controls as a test that we can always test. So the same thing when you’re creating these, these bits. It’s not necessarily the person writing the code or writing the code with the agent, but whoever ultimately is responsible, right? Is saying, “I’m gonna put an and properly tested control in place, and if you pass this, then we know everything is good,” right?

I We could do whole episodes around infrastructure as code, around that’s the idea. I remember when I was at Pulumi, had a customer that I can’t mention, but they had a scenario where if you were in this organization and you wanted a Kubernetes cluster, you could [00:18:00] either request one and it would take four to six weeks because it would have to get checked by security and everything like that, or you could use the Pulumi setup for it that was already tested and for sure was clean, and then you could have your cluster in 20 minutes.

I think it’s a lot of the same things here, which is they’re not just Like kind of that agents MD is like the documented… The sort of like back in the day when we like we wrote a run book and you’re supposed to read our security policy developer before you do anything. That’s an agent MD, is read the security policy, right?

But putting hooks or putting things into your CID system that are automated when it comes to that, that’s like the automated check that enforces it and is told in that way. What do people get wrong with that though? ’Cause some of this, I remember years ago, I, one my partner at the time was like, I was explaining DevOps to, to them, and they were like, “ Why wouldn’t you just work this way?

This just seems like common sense.” And it’s like, you’d be surprised. Where do people mess this up?

Sundeep Bobba: People could mess this up in, in number of ways. [00:19:00] Like they could understand the platform controls wrongly. I think when you’re using a-agentic AI using the OPA policies is the key. Those are all documented. They are part of the code and they build clear guardrails around, around the development processes

Naga Sujitha Vummaneni: Yeah, and the basically the control theory itself, it assumes a stable plant. So means a system whose behavior is roughly known and doesn’t change while you are controlling it, right? You’re m-mostly like with the AI agents, like our platforms are not like that. So we are keep on mon modifying the plant continuously. And so like everyone else is forgetting about the policies and procedures and governance and controls while they are focusing on the implementation at rapid speed, right? This is where the main problem is. So like you are delivering first and then fixing later. So [00:20:00] every one of us, I think will need to try to include the security from the design DNA itself, and the model itself has to be aware of like what it is doing in a secure way. As you have mentioned, the spec kits make lot of difference. So the spec kit have a control policy written before ahead when you ask for a project to be written with a wipe coding in AI. And also it implements the test cases first. So first it has to pass every security control before it even delivers the project. So that’s that’s where I think we can control it from the design DNA itself.

Matty: when we’re thinking about this organizationally, and to me these always come back to organizational and people problems, right? I always say tech is, the tech is easy. The people what make it hard. A lot of times we look at, say someone’s com- coming along and they say, “Okay, you know what? I read your book. I get [00:21:00] it. This is amazing.” And they do a project to start implementing this, and then everything is great for a little while. But, like, how do we think organizationally about how to… ’Cause we talked about how theoretically you’re constantly adapting this, you’re bringing it in from all the times, but what are some of the things to think about as how your team works and how your organization works, and where that responsibility goes to ensuring that this is continuous improvement, right?

Versus just a one-time, “Hey, I set up my pipeline.”

Sundeep Bobba: Yeah, I think this is where the model either becomes part of the organization or it dies as one-time transformation project. I think the, the book’s point is that, like moving to a control-oriented delivery model is not just a tooling change. It changes how teams are organized. It builds out a new mental model, how decisions are made, how platforms are designed, monitored, and governed.

I think if one team builds a great feedback loop then walks away the system will drift as the organization changes. So [00:22:00] organizationally, I would focus on few things. Like first, make platform engineering a place where common controls and paved paths live, rather than asking every application team to reinvent them.

I see this every day. Every team has their own set of processes, a set of tools and it leads to a lot of drift. Second, treat observability as a shared capability because if teams don’t have timely actionable feedback, they cannot adapt intelligently. And third make policies and decision rules versioned or owned so then the control logic can automatically evolve with it.

So I think the sustainable organizational model is not implement control once. I think it’s build an organization that continuously learns, updates its constraints, and evolves the delivery system as conditions change. Otherwise the, the organizations are gonna fall back very rapidly like where there are new models coming every single day.

The only way that we can [00:23:00] adapt is to is to build a sustainable organizational model.

Matty: So that’s interesting, yeah. When we think about that, so thinking having something like this, for lack of a better word, centrally managed, which brings its own problem, but if there’s some res- if you have a platform engineering function, this is the kind of thing that’s really good for that, ’cause platform engineering is thinking about holistic bits and bobs to that. It’s interesting too, ’cause we’ve seen different ways that things get adopted. Like on one hand, like doing a small pilot, I always talk about how what we can learn from DevOps that we learned from Agile transformation about you stack the deck for success, you make it look really great, and then everybody wants to have it.

But coming in by fiat, but when you’re trying to, your small feature team or your team that’s working on one small thing, they don’t necessarily have the context what matters overall organizationally, and that, that can become challenging. The, I was looking a little bit I’m, and I’m certainly gonna have to read your book because I’ve been reading a [00:24:00] little bit about it and I’m like, ooh, now I know what I’m doing this weekend. And we will put a link to where you can get the book in the show notes everybody, so everybody should definitely check this book out. A 30-minute podcast is not gonna get you all the way there. ’Cause we have people who listen to this show come from kinda all walks of of organizations, right?

So a lot of this we’ve been talking about in the context of a larger organization where you might have a platform engineering team. You might have this larger thing. What are some of the things that maybe a smaller team that’s a little scrappier, that’s a little bit in that, that, that can still get from this that’s still helpful?

’Cause I imagine it’s just as useful no matter what ’cause you’re still corralling your robots and trying to move faster and faster. But what are maybe I guess a couple of the places to sorta start with because this can seem very insurmountable. Where would you say someone should just start if they just wanna kinda understand this idea and get something going?

Naga Sujitha Vummaneni: Yeah so the small teams often close loops faster because the control boundary and the, the team boundary are, like very well communicated. So I [00:25:00] think when you implement these this control boundary as each gate, then the question that never should be like will this be automated?

But, like where should this control boundary sit, and who owns… what’s on either side of it? So these questions has to be, like discussed within the team very transparently. And most of the approval gates are like compensating controls for missing feedback. So the team can’t observe whether the deploy worked so a human looks at it or the fix when the observability and the gates justification evaporates what really happens when the gate stays because removing it requires someone on activity, someone who take the accountability of that risk. Now these compliance gates are the honest cases, like either it would be like a big team or a small team, right? And sometimes a [00:26:00] regulator genuinely requires a human signature. In that case we will be placing the reviewers, like for a tightly coupled pipeline it requires one or more reviewers and for where the f-functionality is very crucial, right? So now here we need to make the credible arguments as well. Like, when the decision la-latency itself is important to operate on those gates, then what, what would be, like, the timeline of acting on that system? In that case, I think I genuinely feel small teams do better because they stabilize the loop very well, and by the time the human decisions are made the system state shouldn’t be changed, so those should be controlled very well in these operating gates.

So the delayed feedback also need to be considered with the own… and ma-made sure that it is not [00:27:00] over-corrected.

Sundeep Bobba: You don’t need a formal platform engineering organization, I think, to start thinking this way. So what we have explored in this book is a, a new mental model that how we could use CI/CD. We have been always thinking, “Oh, CI/CD, I can use it for building an artifact, then deploying an artifact, then running tests.”

So this, this model just explores a new mental model. I think I recommend for someone with a small team, start with one service or one delivery path and make the loop visible. And what– and know what signals tell us the system is healthy, what decisions are we making from those signals, and what actions can we safely automate.

This book is very deliberate that observability is foundational. Without observability an actionable feedback cannot be taken. So you can add more sophistication later. So I think the important thing is not size of the platform team, it’s whether you have a closed loop of feedback, decisions, constraints, and [00:28:00] action rather than a pile of automation

Matty: That’s fantastic. This has been, this is really… I’m very excited to read the book. I’ll just put it that way. So this is, by the way, and I’m not getting paid to say that, everybody go check it out. I’m gonna have lots of questions I’m sure when it’s done, and I think it actually would be really awesome, we might have you guys on in after a while and say “Hey, this has been out there.

You’ve been talking to people. What’s, how’s it going?” ’Cause there’s a lot to, as we know, this is continuously evolving. Maybe before we just wrap up, maybe each of you give like a quick little like what’s the most key thing you want someone to get from your book?

Sundeep Bobba: the key thing that we wanna get out of this book is is to think beyond how a CI/CD can be used with the agent era is a different mental model for software delivery. I don’t want to finish someone finishing the book thinking, “I need a new CD-C- CI/CD tool.”

This book is about look at system they already have differently. Are we just automating steps? Are we actually controlling system behavior? Do we have a good feedback? [00:29:00] Are our dest- destination boundaries clear? So I think that becomes even more important with AI, right? If readers come away thinking automation is not maturity controlled, observable, adaptable delivery is maturity, then I think the book has done what we want it to do.

Naga Sujitha Vummaneni: Yeah. And also I would like to add a point in the security perspective or as well why we are insist on the control systems and feedback looping is the failure mainly comes f-where from although the automation exists, it will be quietly bypassed. So when the when there is an attacker’s perspective a misleading pipeline, then we should ensure that it doesn’t bypass quietly. So basically, the bypass is is a signal, not a violation, right? So when engineers route around a control, the control was misdesigned. It might be in a wrong place or too slow [00:30:00] or solving a problem they don’t even have. some common shapes to discuss here is the emergency change process that’s used for forty percent of changes and the security scan everyone has a documented ex-ex exception for, and, the staging environment nobody deploys to because it never… it’s never in a usual state. So and finally, the approve button that one clicks two hundred times a day without even reading, right? Why this is actually dangerous in today’s world the… it’s because the or-the org’s model of the system is now wrong, so like what we have been operating from years. leadership believes that there is a control, but the control chart says g-green. Nobody knows what is happening in the behind scenes. So in control terms you have lost observability of your own control layer. Now you are regulating a [00:31:00] system you can no longer see. We need to ensure and enable the guardrails more visibly on the pipeline at every single stage so that we don’t miss these silent fa- failures. And also making sure a, a tightening control to make sure that we productively graduate to production not the bugs, but the right code.

Matty: if you head over to arresteddevops.com/cicontrol, you’ll get this episode’s show notes, including all the necessary supporting links for our conversation. If you visit arresteddevops.com/itunes and leave us a review in the Apple Podcast store, that helps other people find the show. And we’re also available on Spotify, iHeartRadio, Audible, all the places that fine and less fine podcasts can be found. Thank you so much Sujitha and Sandeep, for Sand- for joining me today as always, this has been Arrested DevOps. And remember, there is always DevOps

Sundeep Bobba: In the banana stand

Naga Sujitha Vummaneni: In the banana stand