Johan: [00:00:00] I have read the Google SRE book so you don’t have to.
Matty: It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton. We’ve got a great topic and a great guest for today’s episode, but before we get into all that greatness, how about some greatness from our sponsors? This episode is sponsored by CircleCI. Designed for modern software teams, CircleCI’s continuous integration and delivery platform helps developers push code with confidence. Trusted by thousands of companies, from 4-person startups to Fortune 500 businesses, CircleCI helps teams take their software from idea to delivery quickly, safely, and at scale. Visit arresteddevops.com/circleci to learn why high-performing DevOps teams use CircleCI to automate and accelerate their CI/CD pipelines. This episode is brought to you by cloud-native consultancy Container Solutions. We bring culture, strategy, and technology together to help you get your cloud-native transformation right. To find out more, visit arresteddevops.com/containersolutions. This episode is brought to you by MacStadium, leading provider of cloud solutions built on Apple Mac hardware. As more teams are working from home, having your Mac build infrastructure in the cloud can make it easy for your app devs to work more efficiently. No need to have someone in the office keeping an eye on the Macs. Let MacStadium do it for you. And if you need a fast, scalable, modern way to run Mac virtual machines, MacStadium’s virtualization platform, Orca, is purpose-built for running iOS and macOS CI. Orca takes a standard macOS VM, puts it inside a Docker container, and then uses Kubernetes to orchestrate everything, all on Mac hardware. Orca is easy to integrate into your current workflow with plugins for all the popular CI tools like Jenkins, GitHub, GitLab, and Buildkite. If you’re building apps for the Apple ecosystem, learn more about MacStadium at macstadium.com/arresteddevops. From that link, you can also get access to a free 2-hour sandbox to give Orca a try. I’m really excited. I’m always really excited about our guests. I should specify that, but I’m excited in a different kind of special way maybe, and maybe you can figure it out. I don’t know about who’s joining me today. I, I, you know, I also, if you listen to the show, you know that I will commonly say things like, I can’t believe it’s taken this long to have this person on the show, but, but, but this time I mean it. So joining me today is, uh, my friend, uh, Johan Abildsko. So Johan, tell, tell people a little bit about, you know, what’s up with you. I know you, but they might not, so let’s fix that.
Johan: [00:03:07] Thank you for that fantastic setup. There’s no pressure at all based on that introduction, and I’m just so happy to be here. I, I really enjoy listening to the show and, uh, the profound conversations that we share in public through the medium that you guys have, you people have created. I’m a DevOps consultant and spend my time with teams, luckily not just fixing pipelines and talking about cloud, but very much so also talking about the co-creational skills around teams. How do we talk to each other? How do we interact? And through that, I’ve just been able to refer many episodes, many people. We have so many stories, so many narratives in our— in the awesome DevOps community. So I just feel privileged that I’m here to have an interesting conversation with you, Matt.
Matty: Let’s have an interesting conversation. I want to kind of jump right into something. So when we were talking over the last couple of weeks about what we might want to talk about and everything, you said something that really struck me, and I’ve been kind of thinking about it a little bit, and I want to jump into it. You said, if you invest enough into foundational practices, you can ignore them. And I want to kind of think about, you know, before I kind of say what I think about that, like, what, what were you— what was your intent around that? Like, what does that actually mean?
Johan: [00:04:34] So I think there are many layers in this reasonably simple statement. I think one of the first pains that you notice when you start visiting multiple software teams is that a lot of the conversation or a lot of the effort is spent around things that should be boring or not important or not present at the rate that they are. So ideally, of course, we know we want to spend all of our time creating value for our customers. And I’m quite sure that our daily ceremonies, our hordes of meetings, and intricate discussions around who failed what build and why is our Git branching strategy so complex are not value-adding for our customers. But because we fail to see that and we fail to invest enough time and energy and effort into those core foundational practices, then they keep being roadblocks for actually thinking at the level of abstraction that we want. Things like, are you skilled at operating your IDE? It’s just such a basic thing that we tend to forget that it even exists, yet I guess that At least I know I waste a lot of time when I’m in my IDE because it never feels important enough for me to really invest in it. So I just keep paying a little bit, a little bit, a little bit, and never invest in becoming awesome at something, so it disappears. So for me, where I had this insight, or like insight spilled over many, many talks and discussions, But a StarCraft streamer or video game streamer, Day9, who is one of the seemingly most wholesome and awesome personalities in the world, recommended The Art of Learning by— and now I will butcher a name— Just Waitzkin. I kind of like that. A chess champion turned MMA fighter or something like that. Who talks about his process and how he thinks about pushing yourself across, and how he pushes himself across plateaus in skill cap. And he talks about practicing the fundamentals so much that they become muscle memory. And we have a very apt understanding for that in terms of physical things, but we really don’t have any intuition about the truth of this when we become virtual or when we talk about programming, when we talk about all these things. And I really see that as a— we are missing out because we don’t have the vocabulary in our universe that doesn’t really exist about muscle memory, about focusing on creating these automated responses such that our minds become free to do this awesome creative work that developing software is and should be.
Matty: [00:07:54] I think it’s interesting because like developing that muscle memory has to come from— because you have to practice with intent too. Because I think that might be kind of the thing. And I’m thinking back a little bit to that example of just like learning to use your IDE and maybe using the example like being really good at Vim, right? Which arguably is a skill or not, you know, a valuable skill or not. We’re not going to get into that. But whatever, you know, being the most effective as you can with your tool and You can practice in a bubble. So you can use like a Vim playground, for example, or there’s a really fun website that’s like Vim Adventures that like lets you practice, but you’re not doing it in the model that you’re gonna actually do it. Like you have to, you have to use those tricks and techniques and skills in your tool doing the work that you actually do. Either you don’t practice it at all, which is one particular problem, or you say, okay, I’m going to actually sit down and I’m going to do a bunch of— because I’ve done this with VS Code all the time. I sit there and I’m like, I’ll read blogs about fun tricks and I’ll install plugins that look cool and I’ll figure it out. But that was in one little bubble of when I was in my slice of I’m going to learn stuff about VS Code today. Then I will then later that day or the next day, I will go and I’ll start writing some code and I’m going to keep doing everything the way I already do I did it because I’m in that place. So there’s 2 places that’s challenging. So one is just making ourselves do it, but also, and I think in the case of something like making your code editor technique better, at least that’s within your purview. But I think when we think about foundational things with how our teams work, a lot of times we’re dealing, you know, it’s hard to invest the like, okay, we need to, you have to go slow so you can go fast, right? We’re actually going to slow down a little bit because we’re learning a thing about how we’re functioning together. But then you’re like, but we don’t actually have time for that this sprint because we got to get this thing. And you’ll never have time. And one thing I want to know what you think about this, when I think about when— again, back to the muscle memory, I think about bowling. For a while, years ago, I really was into bowling. We had a bowling league at work and I loved to do it. My friends and I would go all the time and got a ball. I was really invested and I wanted to get good at bowling. And what I learned to do was, because there were always a ton of things I felt I needed to improve with my form, and what I learned to do was every time I would go up and address the pins or, you know, like go up to throw, I said, you know what, I’m only going to think about one thing. And I’ve taken that same, um, approach to how I’ve tried to— as I’ve tried to improve like my public speaking back when we used to get to do that, right? Is I, you know, I invested time in a speaking coach, you know, and like doing training. And so there was— I had a litany of things that I wanted to get better at with my stage presence. But for any talk I gave, I said, you know what, I’m gonna think about one thing, one improvement, and everything else will just take care of itself. Like, I don’t care, you know, if it’s like this talk, I want to think about making my gestures big. So that’s the thing. I’m not going to worry about, oh, I need to be better about moving with intent, or I need to worry about like how I phrase this thing. I’m like, I’m just going to do the talk the way that it feels natural for everything except my goal is to have big gestures. And so I think that can still apply. So if we go back to the IDE example, it could be like, you know what, this week what I’m going to do is I am going to use this one plugin I found to, you know, help me find, you know, a peak problem or whatever. Like, I’m gonna think about that every time I’m doing something. I’m like, am I using this one technique? And you do that for like a week or so, and then that gets it into this pseudo muscle memory of like, okay, now it’s part of my practice, right? And so you could do the same thing with how your team works. You could say this sprint we’re gonna— or this unit of time as a team, we’re not gonna try to like fix everything about how we work together, but we’re gonna have intentionality about this one practice so we kind of just get used to it, you know. And I think that’s why it gets hard with these foundational things is because they’re so broad and there’s so much to them, you, you feel like you have to do all of it, and then which means you do none of it. I don’t know. It might not— the metaphor might not work, but what do you think?
Johan: [00:12:36] I think at least you’re sparking many thoughts in me. So in that regard, at least it is working. And I think one thing that I would like to— I have 5 points, so we’ll probably only cover 2 of them, I guess. The first point is that you say intentionality, and I love that word. Not because it’s a particularly fun word, but I think most software teams are hugely lacking in intentionality, in discipline, in being deliberate, in being explicit. When we say that we want some scientific thinking or hypotheses, our base way of working really goes back to muscle memory, really goes back to the reptile brain. We really dive into the, like, oh, this complex problem, I know just where to tinker. And then we tinker, tinker, tinker, and we know I should probably have written the test first or created a drawing, not like a very excellent drawing, but where does this overall fit into our architecture or something about our deliberateness. And I think actually that is one of the key benefits from doing test-driven development. I think that’s actually forcing one to do things with intention. And because we’re difficult in our intention, everything becomes very fluffy and vague and hard to get a grip of what is actually going on. And we experience loss of control, I think, because we’re not being structured in the way we’re working. We’re just doing things and see what fits, which I think is sometimes okay.
Matty: [00:14:27] Like, that’s— I think that’s the one thing I want to take a, take a separate— is that not all work and all things you do are equal. So that working with intentionality is important, but there’s also exploratory work. So, because I think— and this is what the reason I bring this up is, I want people to realize that There’s— none of this is binary. None of this is a zero-sum. Like, if you do— you know, I think because that’s, you know, proponents— or not proponents, the opposite of proponents— detractors of like TDD would be like, but I don’t always know that. What if I’m just doing a spike to kind of see what works? Great, that’s a different kind of work. But you don’t— because of that edge case doesn’t mean you never do the other thing, right? So like, when you can put yourself into a like, okay, this is a process, you know, a part like that, then you have that. Then if you’re like, you know what, I’m just trying to figure out how this API works, and I’m just— because sometimes the work we do is throw it against the wall and see what sticks, and that’s an exploratory, experimental thing. But then you have to know when to stop doing that. Like, is that a thing you learn? You did that to learn a thing, and then now you’re like, now I want to make a thing. You know, so excellent point.
Johan: [00:15:41] I think for me, I too often forget to be, at least if not explicit towards my team or everybody else, I’m not even explicit with myself or honest with myself in what mode I am in. And I think that is not an advantage for me, at least.
Matty: I think you don’t have to be— to extend this a little further, you don’t have to have intentionality about knowing when you don’t have to have intentionality because you’re going to do it anyway. It’s sort of like my take on like why we— why people think we over-rotate on culture versus tooling in this kind of work. And it’s not because it’s actually more important. It’s— they’re equally important. But you don’t have to encourage people to play with tech. They will do it, right? You know, like I’m like, that will just happen. You know, I mean, like I don’t have to be like, make sure you’re playing with engineering stuff that’s really fun. They’re like, yeah, I got it, right? You have to, you know, it’s like the eat your vegetables thing, right? So it’s like, I need to remind you of this, but then it can feel like by reminding you, I’m saying it’s the only thing, and it’s not. It’s just the thing I have to tell you. I don’t have to, again, like, I don’t have to like remind my kids to play. You know, I understand maybe sometimes you do. All metaphors are suspect, but I’m like, you know, I want them to play with their friends, but generally speaking, that’s probably gonna happen. I don’t have to structure that into their day. They’ll fill in that blank. And then I think that’s the thing, right? It’s like, we’re saying have this intentionality, and sometimes you— there are cases where you don’t, but because you’re going to almost default to not having it, it’ll probably take care of itself.
Johan: [00:17:17] I, I think that’s a very, very apt point. And, and again, for me, the first thing that we need to be is able to see what’s going on, right? We need to be able to sense, and sometimes trying to give names to things Names can be important. Like, I am doing a spike. So then I know that this is something I will likely throw away. This is something where whatever goes, basically. And the only thing that I’m accountable for is see what happens, what sticks, right?
Matty: Yeah, I’m accountable to say, like, I learned a thing, and maybe the thing I learned is I didn’t learn anything.
Johan: Yeah, exactly. And there, so to move to my next point is you have this excellent public speaking thing where you say, well, then I will mostly practice in production, one might say, on stage, and then have tiny focus areas where I’ll say, okay, so this time I will be focusing on not saying um too much or not talking too quickly or whatever is the foundational public speaking practice that you’re practicing that day. But there’s also like, I really like that for you in speaking English, but I would bet that if you’re trying to do public speaking in French, you might— and please surprise me— but you might benefit from like taking a step back and then practicing like some very core basic practices. And I think sometimes we have a tendency to forget or have a difficult time. I have a difficult time, right? What are like our language? What is our vocabulary? What are the basic things that I would expect someone to be at least, if not proficient, then capable of as a professional software engineer, right? And what is the discussion around that? Do I need to To what level do I need to understand version control and Git? To what degree do I need to be aware of some architectural patterns? And we don’t need to be experts, everybody in everything, but there’s kind of like 2 different parts. First of all, we need to be able to see. So when you have caught a new pattern in your IDE, The reason that you practice that in a void, in a sandbox away from your production code is so you can become aware such that when the opportunity presents itself while you’re working, you can jump on that and then apply your new trick. And then you can apply your new trick in context and apply your new trick in context until that becomes muscle memory. That’s very Toyota Kata thinking where like saying, teaching you in a classroom, that will bring you only awareness. Until you have practiced in your context, then you haven’t learned anything. And that’s like the one point. The other point is that sometimes our foundational things, such as do we have discipline around keeping our builds fast and stable and those kind of things, or are we working in a programming language that I’m so unfamiliar with that I have a hard time grokking what’s going on on the screen, then I need to practice outside of my context in order to get the component building parts in line such that they stop mattering. So there’s like the awareness and then there’s like the proficiency. How can I make the smaller component parts go away? I don’t care, I can read a text, or text, I can read some code and I don’t care about variable declarations or something like that, or at least I don’t ponder what it means when it says float64. I know that, and I know that you don’t have to process that.
Matty: [00:21:19] You don’t have to sit there and kind of translate it, right? Like, it’s again back to the idea of— with going back to, uh, spoken languages— it’s when you think in the language versus having to translate. Like, when you’re learning the language at first, you’re like, okay, you said this word, I’m translating that into a context that I have. And then I hit the point when I don’t have to translate that anymore. And I think one of the things I really like where you’re, you’re going with about the foundational stuff, because I, I’m now realizing some of what I was talking about was talking about refinement, right? You know, again, like that idea of, okay, if you’re, if you’re trying to improve, but when you’re going back to that foundational thing, you, you know, it’s not going to be just one thing, but it’s— I like the idea of you have to take that and you have to put it into a context that resonates to you. And I also think You know, there’s a lot in anything about the kata is also about progressive disclosure, right? Like, you don’t try to learn everything all at once either, you know, right? Like, it’s like, okay, can you get this foundational first principle, you know? And yes, you’re gonna have all this other noise about like— and this, I, you know, I continually go on this with teaching where it’s like, because people, when there’s a brand new thing, you immediately want to get to how do I use this thing to solve this very specific problem I have now? And you’re just sort of like, dude, you’re not there yet. Because, you know, you want to be able to do that, you know, and, and this was something I always ran into when I was a chef. And because again, if you’re bringing me in, you’re looking at Chef, you have a problem to solve. So you’re like, how do I use Chef to do this thing? And I’m like, okay, but before we can— before I can show you how to do that, you have to understand what it’s about. And actually what might even happen is by the time you get to that, you might even see the thing that you want to do is not even the thing you want to do. Because you didn’t know you could do something different. But even if that’s not the case, so again, like, if you think about stuff with Git, right, it’s like you’re, you know, if someone’s jumping right into it, you’re like, okay, so I have this merge problem and I have to go back to a change that somebody did, whatever, blah blah blah. And you’re like, but you don’t even know how the staging area works, you know what I mean? Like, you have to—
Johan: [00:23:21] you don’t have the vocabulary to discuss this problem. Yeah. So, uh, there is, uh, I don’t remember the name of the didactic science person, but he talks about the zone of proximal development, which is basically saying that there is 3 zones. One is what I’m able to do on my own, what I am able to do with assistance, and what is beyond my capability, basically. And if we forget that there’s those levels, it’s easy as for us as teachers, whether that is talking about the problems that we want to solve, or we talk about all these awesome things that we can, but we as experts, we have probably forgotten what was difficult. And it’s so important to understand the learner’s journey in terms of what is they— we need to focus on what is the capability currently and improve that and thus expand the realm of infinite possibilities for whomever, right? And where that becomes very difficult, one of my pet peeves as a public speaker or teacher, or at least as someone who gives critique to those, so I probably, I do this myself, we put information, too much information on a slide or put too much content there. And either we know— so a good example for how you can do this, or not how you can do this, how we fail at this is, let’s say I want to talk about Kubernetes. So I want to talk about control planes and etcd, or however you pronounce that, and API servers. And we have like these 5 or 7 components in the control plane or whatever. So I put them on a slide, and then I talk about them. But I, as a newbie or rookie or someone who isn’t in on this, even though I might be smart enough or whatever, because I don’t have profound knowledge on this topic, I don’t know what I can ignore. And that’s why we have all these tremendously large discussions. Should you be in one repository? Should you be in many repositories? Should you use GitFlow or trunk-based development? Or GitHub Flow or whatever, merge or rebase. We can spend so much time discussing merge or rebase. But really, it’s not that important. But because we haven’t invested in that foundational knowledge, we are not at the level where we are able to see that this is something that we can almost ignore, or at least handle as we go. It will sort itself out reasonably. But because we don’t have that knowledge, we don’t know what we can ignore, or we don’t know what is extremely consequential and what is not very consequential. So in a teaching, do this exercise and just run the script, it will run Maven test. It doesn’t matter what it does. And then people are like, I need to know what’s going on in there. What is happening there? Why is it there if I can just ignore it? You’re making me uncomfortable. So it actually takes quite a bit of understanding to know what to ignore. So we need to either decompose the things or hide the details, even though we might, and this is scary, not be technically correct anymore, but only have the correct intuition.
Matty: [00:27:07] And I think you can do that. I was thinking back to that, like, test that you can ignore thing. And I think, again, this is where the hard work comes in a little bit, but you can— like you said, it’s hard because it’s not technically correct. But maybe I could be like, okay, so I’m showing you how this pipeline works. So we’re going to have a Maven test, but instead of either having something that does absolutely nothing or does something but I say just don’t pay attention to it, there’s probably somewhere in between, which is a way to illustrate just that little tiny bit that you need to know, that it tested a thing, right? And people, I think, can interpret that that’s not all of it, but it’s enough to get the nugget of the idea, right? You know, so like, it could be— the test could be, you know, is 1+ 1 equal to— and you’re like, no one would ever think and say like, yes, that would be a legit test you’d write, but it would be enough to understand it checked a thing. Especially for someone who doesn’t think about tests in a pipeline. You’re like, okay, it tested a thing connected to something, right? Or whatever. But I agree with you where sometimes it can be maybe I’m not even gonna tell you that that’s a thing that exists and it will hurt me in my soul because I know you should see a test there. But because I know I’m gonna talk to you about this again in 10 minutes, I can bring it back. And that’s that progressive disclosure thing, is you don’t have to show the whole map of the territory at the very beginning because people might not— it might be too much to be able to, you know, and they’ll get in there just— and I think you kind of nailed it, they’re distractions, right? Because you’re trying to say, okay, so again, go back to it, okay, you have the control plane, you have all these things, all these things. Someone will narrow in on a thing that jumped at them and they’re like, but wait, how does that CD part in this? I have all these questions about etcd. And you’re like, dude, I don’t even want to talk about that right now. You just need to know it’s there. But then you’re like, do you even need to know that for the thing I’m telling you right now? And this also goes back to like part of my problem with just agenda slides in general, not necessarily in a talk, but just like an overall, because a lot of times, or just it’ll happen with, I’ll be talking to customers and they’ll— I understand why people want an agenda because they want to know what they’re going to get. But part of it is like, I don’t want to tell you because you’re going to start to qualify and you’re going to be like, well, wait a minute, why can’t we talk about this thing first? You’re like, because I don’t even want you to know we’re going to talk about that thing yet because you’re not ready to know we’re going to talk about it till we spend 6 minutes talking about something else first, you know? So it seems progressive disclosure seems disingenuous to a lot of people, I think is why. Like they, they feel like it’s like you said, I think you kind of hit it, especially for engineers. It’s that, that specter of being technically incorrect or incorrect technically, you know, which is like— but you’re like, but that’s not how it works. You’re like, but it’s okay. And I think it’s okay because you— if you are taking responsibility as the educator to correct that quickly, right? You’re like, okay, to get you to get this thing, I’m going to show you this part of it, and there is more. And I’m not being dishonest by not telling you all the parts, by not giving the fidelity of the conversation. Now, if I never did that and I’m like, you need to— and I, I continue to act like that’s just what it is, that could be risky, but also maybe not. The reason I think people also feel like— because I guess this is where, whether it’s imposter syndrome or whatever, or just feeling like being challenged— is they’re like, well, someone will know, and then I won’t have credibility anymore to them because they’ll be like Well, actually, Johan, you know, there’s this part of the controller in there. And, you know, so sometimes it can be— so again, this is— now I’m just going into very solutioneering of how to solve this from an educating perspective, is I think sometimes you can tell people that you’re doing progressive disclosure where you can say, I’m showing you a very simplified view here and we’re going to go into more detail. And sometimes you don’t even say that. And to be honest, this is totally an it depends because it depends upon the control you have over the journey. That the person you’re teaching has. And, you know, like, again, if I’m gonna— if I’m not gonna be able to get to those other things in the 20 minutes I have with you, I’m probably gonna say, look, I’m, I’m showing you just a— I’m gonna actually own up to the fact that there’s a whole bunch of stuff outside of the scope of what I’m talking about, and we’re just not talking about that today. If I know I’m gonna get to that, I’m gonna, I’m gonna keep expanding that box. I’m actually not gonna own up to that at the beginning because I want to do that. So, um, yeah, I don’t really know what that has, you know, how relevant that is, but that’s the thing to think about as people are thinking about how they’re teaching. There’s no one right way to do this, right? Because it depends on context of so many different things.
Johan: [00:31:56] And there’s also the thing is sometimes we feel like we’re technically incorrect if we’re not exhaustive. And in almost all of the cases when we are being exhaustive, we’re doing someone a disservice, right? Because we haven’t filtered, we haven’t taken responsibility of being that filter that grasps the essence and hands like the nuggets. I have read the Google SRE book so you don’t have to. That’s kind of taking responsibility for saying, there are these topics, if you want to know more about these, go read them there. But now you have an idea and you should be doing very much better because I have enlightened you this way. But if I start by just being sort of, okay, so now we need to talk about service level objectives. Have I mentioned error budgets? They’re awesome. And then you just You add and you add and you add and you add. And just like this conversation, people will not know where we started. And like, that is excellent if you are more or less at the same level of comprehension, because then it’s a journey you go on together. But if you are in an asymmetric relationship, in terms of that understanding of that topic, then you will most likely be doing someone a disservice because the mind map just becomes longer and longer and longer and longer and longer and longer. And then we perhaps forgot to take responsibility for actually learning or understanding the topics that we intended to. And then we again can talk about intentionality. Because of course, we shouldn’t be so rigid in our communication, in our teaching, that we can’t adapt on the way. But as everyone who has been in a meeting ever has known, like the declaration of intent, that’s the agenda, will never be adhered to. And very rarely will that be with any sort of intentionality, right?
Matty: [00:34:12] Well, it’s a little bit like the model of open spaces, right? Which is whatever conversation happened is the right conversation. And I think it’s interesting you brought it from podcast. This is incredibly common on this show. You know, there’s— it doesn’t happen all the time, but there have been many instances where I’ll start an episode, I’ll have a guest on, and I’m like, okay, I want to talk about this thing, and we end up somewhere totally different. And it’s amazing. We had an episode with Tim Banks earlier this year, and that’s exactly what it was. It was— our intent was Tim and I were going to talk about like ops life, you know, like, hey, you know, kind of thing. And very quickly it turned into a whole conversation about gatekeeping and elevating under-indexed folks, and a much, much, much more powerful conversation than what we thought we were going to do. And so I think that’s the, the other, the other, uh, thing I think is interesting. Maybe, maybe this comes into play. And I know we, again, we kind of went off on a little bit of a different thing when we’re thinking about how we’re teaching people. Um, there’s a lesson we learned with this podcast, you know, through its history And it’s something that I always tell people because I get this a lot. You know, people are like, I want to start a show, you know, like, what advice? How do you do this? And I said, one of the things we learned really early on is you cannot control your audience. And in terms of you can’t control who they are. And when we started this show, when Trevor and I started, we had a very specific idea about like who we wanted to do the content for. And very early on, we had a whole bunch of folks who were you know, avid listeners and would, you know, give us feedback, be like, oh my God, I listened to the show, it was great. And I was like, why are you listening to our show? You should be on the show. You should do the show, right? You’re— we don’t— we’re not doing this show for you. You’re, you’re not our target audience. You know, not like upset, but just puzzled. And then it was like, you know what, we thought we were doing— and, and yes, the people that we were intending to target are still a part of our audience, but we never would have known. And so these, these things and like the journey that you go through and how you are communicating to people and what they’re getting from you You can have an idea, but you can’t— you need to be able to like see where they’ll go. And even the things that you intend are not necessarily what people will get out of it. Um, there’s a filmmaking, uh, metaphor or an analog to this as well. I remember many years ago when the movie Memento came out, my friends and I were obsessed with this movie. I probably saw it in the theater 10 times, you know, because we would talk about it and then we would just go watch it. So we could talk about it some more. And I remember standing in the parking structure outside of this movie theater with my friend after we saw it for how many times, and she said, you know, I really wish I could ask the director this particular question about the movie. Was Teddy ever really a cop? And I said, you know what, he might not know. And actually, to be honest, his answer might not be correct. And then we got into a big fight about, well, no, that’s right. I’m like, and, and I’ve made movies and there’s There are things I have learned about the film that I made from people who watched it where I was like, I, I didn’t mean that, I didn’t intend that, I didn’t think about that, but oh my God, you’re right, that absolutely is about that, and we didn’t even know that when we were creating it. And so the— and the thing can happen, I’ve seen this with talks I’ve given where people will come to me afterwards, they’re like, oh my God, in that talk I got this out of it. I was like Shit, I— it’s not what I meant. And not like that they misunderstood, right? It’s just that there’s a layer to it that you don’t know when you create it, you know? There’s a— and I want to be very clear when I kept saying like, that’s not what I meant, like, because there’s definitely— that’s not supposed to be a like, well, Johan, you said this thing and it bothered me, and you’re like, well, no, I didn’t mean it that way. It’s just like, it’s more of, I thought I was talking about this thing, but to you, you got something else more out of it. Like, you saw, you know, and that’s the beauty of this, the art of it, right? Which is that it, uh, and then, and I honestly think if you’re a good speaker, this is why you talk to people after your talks and you hear what they have to say. And, and it plays into the work that you do because, I mean, that’ll happen. I gave, you know, that, that zebra talk that I gave in 2019, I gave that thing like a dozen times. And I still, and, and, you know, my friend, you know, Aaron Aldrich will tell you he’s seen it more than anybody. And he always liked to see because he’s like, it’s continually evolving. Because I would give the talk and then I would talk to somebody afterwards, they’re like, this is what, this is what resonated with me. Or, wow, you know, I didn’t realize that, you know, you could connect this dot to this dot. I’m like, shit, I didn’t either, but I’m gonna do that now, you know. So I think that’s the thing to think about. Like, the more— and the reason I bring this up is it goes back to your, I think, the point you were making about the more structure we require for a conversation, we’re either fooling ourselves to think we can maintain that structure, or if we work so hard to try to, um, enforce that structure, you’re gonna lose, uh, value out of the conversation. And there are absolutely times, like, right, like a stand-up meeting is a time to have a structure to it and be like, we have— or if you’re— it’s a status update, you know, not every conversation is, is open. Some of them have structure, right? Like, I’m going to the Secretary of State to renew my driver’s license. I’m like, that’s not the time to talk to the person behind the counter about WandaVision, right? No, I’m there to do a thing, right? But like, if we’re saying we’re going to have a conversation, you know, I’m going to meet with a customer to talk about their transformation, to do whatever, yes, I should have a loose agenda of the things they think they want to talk about, but we need to all be comfortable to see Where does this go? Because we can’t predict the future. We don’t know what connections will happen. And if we are forcing ourselves to follow that, we’re going to miss a lot of really great opportunities of the, the, the sum is great— the whole is greater than the sum of its parts. I never remember if I get the order right in that expression, but right, you know, it’s, it’s not— yeah, yeah, you get it, you get it.
Johan: [00:40:15] But I think you have an excellent point, but there’s also again like being able to be very fluid, or whatever we call it, agile, let’s call it agile, right? The ability to be very agile requires some very strong foundations. The basics need to be there, right? So if we are to earn the right or the privilege or the capability of being very fluid and adaptive during our meetings, we also need to make sure that the basics are covered, that we’re not missing out on the important things that we set out to talk about. And I think there are, again, because perhaps of lack of either accountability or lack of awareness, we end up having the same meetings again and again. At least I’ve been in organizations where that has happened, where it’s like, I feel like we had this meeting yesterday, but I don’t think we got anywhere. So here we are again, and then it devolves again. So I think we can also be a bit more disciplined in order to allow to create the basics, the solid foundations on top of which we can go crazy. It’s like keeping our basic hygiene. We want to keep our unit tests excellent such that we can just bash around all over the place and feeling safe and still get there. But if we don’t have that basic hygiene in place, then it just becomes a pain that we keep going out through all different branches and diverge all the time. So I have an idea, and feel free to use it. I hope someone will, and tell me if it’s a success. I think it would be awesome to take like a stack of index cards or to-do papers, stick-its, Post-its, and then we create a stack of the agenda. So the last point of the agenda is on the bottom of the stack. And then every time someone notices that we change subject, we add that on top of the stack. And every time we complete a topic, we remove a topic from the stack. And I would guess that as any backlog, that stack would over time increase and not ever hit bottom. So it’s not— my point is not that we shouldn’t adapt during meetings, or we shouldn’t, we can’t predict the future and all those kinds of things. But we need to have some sort, especially if they’re internal, I think we have more, we should have more deliberate meetings when it’s like in-house. I would guess that otherwise it just happens organically. There should be a reason for the meeting. And if we don’t satisfy that root reason, to say a swear word, you said you were explicitly tagged. So if we don’t cover the root cause of the meeting, then we’re going to be sad because then again, that’s just work in progress. That’s again, one of the things that we should just have gotten out of the way so we can do all the more cool stuff. And I don’t think there is a right way or like it’s a binary solution again. But keeping aware of like, did we satisfy the purpose of this meeting? If we didn’t, was that because the purpose of this meeting was completely silly? Or is it because we got to talk about something that might have been more important? But how do we then avoid having the things that we meet about become urgent. Because if we’re meeting about them, they are important. But if we keep getting distracted by something else, then we’re just adding to our WIP. And suddenly the important things become urgent. And when the important things get urgent, that’s where it becomes much harder to be deliberate. And where it becomes much harder to have the discipline, because now there’s pressure also. And if we then don’t have had all that, if we have practiced our foundational skills, our basics, the discipline of doing software, if we have enough muscle memory, then even under pressure, we will do the right thing because it is the least— path of least resistance. And the path of least resistance needs to be the right way of doing things because over time, it will be much easier and it will be much cheaper to do things the right way. We know this, we feel this in our hearts. We know that if we do unit tests or test-driven development, we will be faster. But we will just not be faster right now. So therefore, we need to have the muscle memory such that when the release is close, when it’s just now, when the boss is yelling at us, we’ll just say, calm down, I know that you feel I’m wasting time on writing these tests, But believe me, if we measure our wall clock, I will be done sooner if I write these tests. And I just think that sometimes we have such a disconnect on how we believe we use our time versus how we actually spend our time.
Matty: [00:45:52] Oh, well, we’ve been— yes, that gets all into a whole other topic that the inimitable John Allspaw introduced me to, which is the you know, disconnect between work as imagined versus work as done. And I think that’s a thing that can happen. Most times when we talk about work as imagined versus work as done, we’re talking about the difference between what the blunt end imagines how teams work, and then the sharp end will tell you something different. But I think we do it to ourselves too, you know, um, both consciously and subconsciously. You know, you might ask me how I do things, and because I don’t want to uh, own up to not doing everything the best way possible. Be like, oh no, that’s totally what I do, and I absolutely do all that stuff. And I may know that that’s not true, and I may— or I may be fooling myself into thinking what I do. So having that awareness, and then also I think to be able to, to identify that, and but be able to— because the only way you can make it better, the only way that you can, uh, narrow the gap between work as imagined and work as done is to be able to honestly identify it, right? And to be able to honestly identify it, you need to be able to have a high level of trust both within the organization and even with yourself to be able to say, yes, I know we all think that this is the true north of how we should be doing things, and the reality is I don’t do that. And to be able to be like, but that’s fine, you know, that’s not— no judgment on me, right? Like, that’s just the reality, and then we’ll work towards it. You know, it’s sort of like You know, I can log all my food in MyFitnessPal and it can look great, but if what I’m writing in there is not what I’m actually eating, then who cares? It’s not helpful. Like, yes, it’ll make my stats look good, but I’m not gonna like get better at my nutrition by recording it wrong. But I feel like we do that a lot, right? We like do a certain kind of work, but then we go into our system of how we’re supposedly doing things and we just make shit up to be like, oh yep, totally wrote tests. You know, it’s, you know, so I think There’s a self-awareness that you can only have if you have a level of psychological safety, um, to be able to be like, no, you know what, we aren’t doing this right. And that’s not a bad thing. That’s not a failure. That’s just an area for like, okay, cool, well, let’s, let’s do something about it, you know, or not. Or just say like, or let’s just reevaluate how we understand how we work. And be like, that actually is how we work and we can’t really move it, so let’s move something else. You know, um, we’re kind of coming up to time, but there’s one thing I wanted to come back to that you also had, had said to me earlier, and I just wanted to kind of get some detail about because I think it’s interesting, and, uh, also to get us into a little bit more, uh, tactical, technical detail of things. So you said to me that you said people are spending too much effort caring about the wrong thing about So, what does that mean? What are those wrong things that we’re focusing on? Specifically, good stories would be awesome.
Johan: [00:49:00] I have an antipathy for the Git flow. I think it has done excellent things for the software development community. And most of the software community has outgrown it. And Envy also recognized that and has done some updated things in terms of his ancient knowledge, tome of knowledge that he bestowed upon us. The problem is that most of the time I have seen things like the Git flow being introduced or being used at organizations, they achieve almost the opposite outcomes than what they’re trying to achieve. And I think this goes very much back to one of the original DevOps paradoxes, I’d say. We claim that speed and quality are opposed, like we can either have one, but as it shows, they enable each other rather than the other way around. And when we do the Git flow, it’s because we want to know what is going where, what is the quality of different things, like have tight control of what’s going on, minimize risk, keeping our main branch clean, those kind of things. But what I see is develop or feature branches becoming huge And then we end up having a horrible merge in the end because it is— people do not invest enough in learning Git, so the Git flow feels very complex to them. So they become afraid of working basically through their workflow. And when we’re afraid, we postpone. And then the deadline hits and everything burns down, and that’s a reinforcing loop. So that’s a very concrete thing where people say, we want to know what’s going into our production environment, our main branch, but we just end up integrating tons of things at once, so we don’t know what’s going where. And we end up achieving the opposite purpose. The same thing about why do people go into a monolithic repository? Or multiple repositories. I have been an aggressive proponent for polyglotic repositories. And I have grown as a person. I think more now that it’s a spectrum. But it’s very difficult to have sensible discussions around that, I’ve experienced. And we end up spending a ton of time discussing, should we be in one repository or in 2 repositories or 4? Or usually the discussion is, should we be in one or should we be in 100? And I’m quite certain that neither is the right solution for most. But then the issue quickly becomes more about how do we feel about this rather than taking a step back and say, What are the developer workflows we want to enable? And what repository structure supports that? But because that’s a difficult conversation, we end up discussing— it becomes the mono versus many repository that becomes the important part for people because that’s something they can grasp. That’s something they can feel and understand. Kind of like the saying, containers will not fix your broken culture. Even though you might be able to work in mono repositories or in many repositories, you will make it work either way. Dependencies will still be hell. But let’s start talking about how do you want to manage your dependencies. And let’s see what underlying repository structure then best maps into that? And how can we consider software architecture where it is possible, not necessarily easy, but just possible to experiment with these kind of things? Because when you’re perhaps a traditional organization and you are in a monolithic repository, your architecture has suffered because of it, because you have not had the intentionality, because you have had the option, you haven’t had many repositories to to keep you accountable in terms of being explicit about your dependencies. So it has likely suffered. But the way that you have suffered is that your architecture becomes more and more connected, which makes it more and more difficult to experiment with possible remedies. So again, it’s kind of reinforcing. And that’s just like, we’re building everything because we need traceability, and we have everything traced because we’ve baselined them in a commit. It’s like, yes, but you don’t really know what’s going in there because everyone is just dumping their things in the same repository. Um, so I think my real point is, is we should look at Git the same way we look at Kubernetes or whatever. We shouldn’t care about Git, we shouldn’t care about Kubernetes, we should care about the platform that we can build on top. So I’ve probably told you something along the lines like, I wrote a book on Git, but Git is so boring that people shouldn’t care. Because it’s what it enables that is important. It’s the developer workflow. So it’s much more interesting for me to talk about when I talk to development teams. I want to talk about how does their workflow look like. I run Git in the command line because I have problems. But if you’re unable to solve all your version control tasks in the combination with your IDE, whether that is some IntelliJ or VS Code or whatever— VS Code for really many people these days— and in combination with whatever backend— GitHub, GitLab, Bitbucket, you name it, Azure DevOps, I don’t care— like, if those 2 tools are not enough to handle your version control things. Your version control things are way too complex. But when we don’t have that understanding, when we don’t have that reasonably profound knowledge on the foundational practices of version control, then we don’t know what we can ignore. We are back at that. So we spend a lot of time discussing all these corner cases rather than just saying, well, how does a developer go the fastest, the safest, from having something in their workspace that they would like to have in all the other workspaces. But the language we use around this, then people start talking about, well, then they commit and they should push to this branch and it should be named like this, and then we do a merge request over here. A developer doesn’t want to do a push. A developer wants to move some code somewhere.
Matty: [00:56:22] I read some blog post yesterday that was so pedantic about formatting commit messages and everything. Like, who reads freaking commits to that level that it matters, right? Like, I think the right way to format a commit message is a way that someone can read it and have some vague understanding of what you’re talking about. And the— that— but the thing is, what I think is the problem, and I don’t— this is a whole other thing, but I think people get wound up into if we find the one magic formula that will make sense to everybody. And you want to know something? There is no formula that will make sense to everybody. So you’re going to— you’re never going to have something like, if we only put the nouns and the verbs in the right order and had exactly 2 nouns for every verb plus an adjective in every commit, then suddenly collaboration is solved in your life. So I think that’s actually— when I think about it, that’s actually a really good— I mean, that’s a really good example, Maddie. You’re super smart. But I think that’s an illustration of caring about the wrong things. Right? We get wound up on the implementation of an outcome, which is write commit messages that vaguely make some sense, that someone can understand. And I think rather than getting— I think again to that over-rotating, we over-rotate to this, I have to write some tool that will lint my commit message to make sure I use the right ratio of nouns to verbs and a colon and not a dash. But then the other side of that to everybody is, fix typo. And you’re like, but there’s, to your point of the one repo or 1,000 repos, there’s something in between, right? There’s something in between the pedantic commit hook and the XY55, you know, like there’s that XKCD comic of like my Git commits over time, right? Which start out very clear and they end up with just like mashing the keyboard. So yeah.
Johan: [00:58:13] There’s sense in that. And I will not open this completely different can of worms. But you can have that discussion with someone else, I’m sure. But setting up these elaborate, very gated workflows, you can only move this Jira ticket from status X to Y if it has a value between 13 and 45 in column B, whatever, right? Where we know that’s not how we actually work. And where we have such a hard time, we discuss all the time our ideal workflows, like the very tough projects and we have all the checkmarks and we forget the pragmatic, like the work. Let’s make first the process documented as it is today. And then if that’s a bad place to be, then we have to change that organically together because otherwise the only thing that we will create is friction. And friction is both the killer of productivity and motivation and trust and everything.
Matty: I think that’s a great way to summarize that. I know you said Git is boring, but we will put a link to Johan’s book in the show notes. So if you want to read his book about boring Git, but you should, because while it may be boring, it’s also necessary. And where can you find those show notes? If you go to arresteddevops.com/foundationalpractices, that’ll give you the show notes with some links for things we talked about. You know, we’ll link to the article about the zone of Proximal Development and the book The Art of Learning, plus Johan’s book. And if we think of anything else, um, we’ll see. So go to the show notes and find out, find out if we thought of something else. Maybe we’ll put a little Easter egg in there. Uh, if you go to arresteddevops.com/itunes, you can leave us a review in the iTunes store. I know tech— sorry, it’s technically called Apple Podcasts, but I am not going to rewrite that redirect because, you know, reasons, which are mostly I am lazy and pedantic. But yeah, and you can find Arrest DevOps on Spotify and iHeartRadio and all sorts of places where fine and less fine podcasts can be found. Johan, this was a really great conversation. I got a bunch of things for me to start thinking about that came out of this. So thanks for joining me today, for sure.
Johan: [01:00:30] Thank you so much. It has been a pleasure and an honor.
Matty: This is, uh, Arrested DevOps, and remember, there is always DevOps in the banana stand.
