Naga Sujitha Vummaneni and Sundeep Bobba co-authored CI/CD as a Control System, which takes Jez Humble and Dave Farley's Continuous Delivery, now pushing 20 years old, and reframes it through control theory. As Sujitha puts it: "The pipelines are activators, observability is the feedback signal, policy is the constraint, and deployment strategy is how you regulate the risk." Once you see a CI/CD pipeline that way, a lot of what look like tooling problems turn out to be system-behavior problems, and most pipelines today are open loop: "They measure everything and on nothing." Sujitha traces the pattern back to a moment from the Arrested DevOps episode with Hannah Foxwell and Robert Warner: enterprise clients insisting continuous delivery would never work at their company, until it was just how everyone shipped. The premise of the book is that this framing was always available, but AI agents moving at machine speed finally make it urgent.

Bonded Automation and the Four Questions

Sundeep's answer to "should the agent handle this?" isn't a yes/no: it's a bounded operating envelope. Low-risk, reversible actions like restarting an unhealthy service or quarantining a known-bad artifact are fair game for automation. Changing security policy, touching production data, or anything with real customer blast radius pushes the threshold for human involvement way up. His framework is four questions: "How confident are we in the signal? What is the blast radius? Is the action reversible? And who owns the risk if the decision is wrong?" Sujitha adds the control-theory language underneath it: signal quality is itself a gate, because an automated rollback that fires on a noisy metric is worse than no automation at all, "you get flapping." Bounded blast radius, rate limits, and cooldowns exist so the system doesn't correct itself into a new outage.

Writing "Always Run GitLeaks" in Your CLAUDE.md Doesn't Make It True

Matty pushes on the gap between guardrails you write down and guardrails that actually run. Telling an agent to always scan for secrets before committing is no different than the developer who says "you're right, I should have run that" after skipping a check themselves, "it's the same trust-but-verify thing we've been doing forever, except now it's exacerbated." The book's answer is to stop treating this as purely a tooling problem: platform controls enforce the non-negotiables, prompts tell the agent what good behavior looks like, and runtime feedback tells you whether the controls are actually producing the outcome you expected. Sundeep frames it as fundamentally organizational: "Who owns the control, who can change it, what evidence proves it ran, and what happens when it fails, and who is allowed to accept an exception."

The Approve Button You Click 200 Times a Day

Sujitha's closing point reframes what a bypassed control actually means. "The bypass is a signal, not a violation. When engineers route around a control, the control was misdesigned. It might be in the wrong place, or too slow, or solving a problem they don't even have." She names the familiar shapes: the emergency-change process used for 40 percent of changes, the security scan everyone has a documented exception for, the staging environment nobody deploys to because it's never in a usable state, the approve button clicked 200 times a day without being read. Each one means leadership believes there's a control while the dashboard shows green and nobody actually knows what's happening behind it, control theory's version of losing observability of your own control layer.

You Don't Need a Platform Team to Start

For smaller, scrappier teams, both guests argue the model still applies, maybe more cleanly. Sujitha notes that small teams often close feedback loops faster because the control boundary and the team boundary line up. Sundeep's advice: "Start with one service or one delivery path and make the loop visible. Know what signals tell us the system is healthy, what decisions we're making from those signals, and what actions we can safely automate." Observability is foundational to all of it, since without actionable feedback there's nothing to close the loop with. The through-line for organizations of any size: "It's whether you have a closed loop of feedback, decisions, constraints, and action, rather than a pile of automation."

Long before "DevOps" had a name, manufacturing plants had already split into two tribes: the automation and manufacturing engineers who built up their own scrappy plant-floor IT, and the corporate IT department issuing laptops and running the domain a few buildings over. They grew up independently, with different tools and different goals, until, inevitably, they collided. IT wanted plant data flowing to corporate; OT wanted to reach out for data too. Add different incentive structures on top, and, as Matty points out, it's the same "wall of confusion" that gave rise to DevOps in the first place, just wearing safety glasses instead of a hoodie.

"I Broke the Rules Just to Get It Going"

Doug's clearest example: a robot on the plant floor goes down because of a network issue, and the routers are owned by IT: a help desk routed through Bogotá with zero context on an explosives manufacturing line. The ticket crawls through triage while production sits at zero and the CEO starts calling. Doug's fix was to log into the router console himself and get it running again. "Everyone's like, 'Ooh, yeah, you're a hero.' Like, no, I'm not a hero. I broke the rules just to get it going." It's shadow IT, industrial-plant edition: the AWS-credit-card moment of manufacturing.

Converging, Whether Anyone Planned It or Not

What actually helped, in Doug's experience, wasn't process; it was knowing people. "It's always gonna be personal relationships," he says, and his pitch to leadership has been simple: IT needs someone who's had OT experience, and OT needs a rep who understands IT's constraints, because the requirements on both sides have already converged. Neither side can keep pretending plant data stays on the plant floor; predictive maintenance and machine learning mean data has to leave the building, sent out to third parties for analysis, whether or not the old "peace treaty" between IT and OT accounted for it. That same shift is dragging OT into IT's security concerns for the first time, too. Doug is blunt that OT folks "just don't have a good concept of the security requirements," a gap he's only closed by talking directly to security engineers in IT.

Manufacturing Already Solved This: In Manufacturing

Matty pulls in The Phoenix Project and its inspiration, The Goal, both built on the theory of constraints and value-stream mapping from actual manufacturing. The irony: software loves borrowing metaphors from manufacturing (Andon cords, the Toyota Production System) while the people doing real manufacturing haven't necessarily absorbed the lessons about visibility and bottlenecks that DevOps eventually learned from them. Doug agrees the CAMS/CALMS framework (culture, automation, measurement, learning or lean, sharing) maps cleanly onto OT/IT, especially the parts industrial teams already live: lean practices are baked into manufacturing, but the value an OT change delivers is far easier to point to (100 widgets became 110) than the value of infrastructure work that only shows up when it's absent.

Incentives Explain Almost Everything

The conversation lands on a favorite Matty theme: people work to the incentives you give them, not the ones you intended. A mandate for developers to write ten tests a sprint produces assert(true). A cash bonus for testers finding bugs and developers fixing them produces a black market in planted bugs. And "litigating severity" on an incident call is really about whoever's compensation is tied to P1 counts. Matty points to the STELLA Report as the same pattern in postmortems generally: after a well-handled incident, it's easy to reason "nothing bad happened, so you shouldn't have shut it down," the same Y2K cognitive dissonance in miniature. Doug's closing advice for anyone straddling the OT/IT divide is almost embarrassingly simple: find out the name of your IT person. Treat them like a teammate with real expertise, not a request queue ("not AWS," as Matty puts it) and the rest gets a lot easier.

Hannah Foxwell, who has spent over a decade in DevOps and platform engineering, draws a striking parallel to earlier transformations: "It used to be that testers didn't trust developers and ops didn't trust testers and there were all these silos. Now we're putting AI agents in the mix. Can we trust them? Should we trust them?"

This isn't just déjà vu—it's a fundamental challenge that resurfaces with every major shift in how we build software. As Robert Werner points out, management had to give up control and push trust to the edges of organizations during the agile transformation. With cloud adoption came self-service and automation. Now, with AI, we're dealing with non-deterministic black boxes that we need to trust to be "right often enough."

The Fluency Gap

One of the biggest challenges isn't the technology itself—it's the lack of shared understanding. Hannah launched "AI for the Rest of Us," a community now with over 1,000 members, after realizing that AI fluency is essential for making good decisions about where and how to use these tools.

"I went to a talk at a conference thinking I'd learn about AI in one talk and become an expert by tomorrow," Hannah recalls. "It just didn't happen like that. There's a whole new domain with new vocabulary, new concepts, new techniques."

The community focuses on making AI accessible without dumbing it down—providing talks and content that explain complex concepts in simple language so more people can participate in the conversation about AI's role in software development.

The Speed-Responsibility Paradox

The technology is evolving so rapidly that best practices barely have time to solidify before they're obsolete. Robert describes how hiring strategies at startups are changing every few weeks as new capabilities emerge. "Things that weren't feasible last week are suddenly possible," he notes.

But this speed creates a dangerous tension. Organizations are pushing hard for AI adoption while the guardrails, workflows, and cultural practices needed to use it safely are still being figured out. As Matty observes, this leads to perverse incentives—developers required to "use AI" who find ways to tick the box without actually deriving value, just like teams that once added meaningless tests to meet sprint requirements.

Who Owns the Code?

A critical question emerges: if AI generates the code, who owns it? Who's responsible when something goes wrong?

Hannah frames it in familiar DevOps terms: "Does anybody really want to own a service if they didn't write it and they don't understand how it works? It's the ops challenge again—AI throwing code over the wall to us."

Robert's answer is pragmatic and honest: humans will need to take responsibility for validating AI-generated code, even if it's tedious work most developers won't enjoy. His company, Leap, is building tools specifically to make that verification process as convenient and enjoyable as possible, because he believes there's simply no other way to do it safely.

The Documentation Double-Bind

There's an ironic twist in how AI agents work best: they need excellent documentation. Organizations improving their documentation to support AI-powered development are inadvertently following DevOps best practices that benefit human developers too.

But as Matty discovered building his own project, AI-generated documentation can be dangerously unreliable. The tools will confidently document features that don't exist, pulling from incomplete PRDs or speculative notes in the codebase. Great documentation trains better agents, but agents shouldn't write that documentation—creating a challenge that requires human judgment and oversight.

Lessons from Past Transformations

The parallels to earlier shifts are instructive. Hannah remembers enterprise clients who insisted continuous delivery would "never work here." Now it's common practice. The same resistance appeared with cloud adoption and agile methodologies.

What worked then still matters now:

• Guardrails enable freedom: Constraints and safety nets let people explore confidently
• Make the right way the easy way: Transformations succeed when good practices are more convenient than bad ones
• Community and shared learning: Success stories and failures shared openly help everyone navigate change faster
• Start with good practices: Teams with solid engineering fundamentals—blue-green deployments, A/B testing, safe-to-fail production environments—are better positioned to benefit from AI-assisted development

Practical Advice for Explorers

For developers and teams trying to navigate this transformation, Hannah and Robert offer grounded guidance:

Keep your eyes open: Watch for patterns of success and failure. Who's making this work, and what do they have in common?

Build community: Find or create spaces where people can share honestly about what's working and what isn't, without the pressure to pretend everything's perfect.

Be selective about information sources: With so much noise and hype, focus on quality outlets. Ignore things for a few weeks, and if they keep coming up, that's when to invest your time.

Practice regularly: The technology evolves so fast that hands-on experience goes stale quickly. Even if it's not your main job, refresh your skills every few months.

Be specific and constrained: AI coding assistants work best with clear, narrow requests. Frustration comes from asking too much or being too vague.

The Future We're Building

We're in the Nokia phone stage of AI-assisted development, as Robert puts it—the technology will look completely different in just a few years. But unlike waiting passively for that future to arrive, developers and teams are actively creating it through the choices they make today about how to integrate these tools.

The question isn't whether AI will transform software development—it already is. The question is whether we'll learn from past transformations to build better practices, stronger safety nets, and more trustworthy systems. Or whether we'll repeat old mistakes at unprecedented speed.

As Hannah emphasizes, having more people with AI fluency means better conversations and better decisions at a pivotal moment in history. The rollercoaster is moving whether we're ready or not. The best approach is to keep your eyes open, stay connected to community, and remain thoughtfully critical about what works and what doesn't.

Learn more about Hannah's work at AI for the Rest of Us. Use code ADO20 for 20% off tickets to their London conference on October 15-16, 2025.

Worse Than Chain Emails

Kat's opening point: "Security is like a never not hot topic," and the stakes have risen as more of life and government runs on computers. The worms and Trojans of the 1990s look charming next to Meltdown, Spectre and Heartbleed, and a chain email that crashed your computer has been replaced by something that can take down much of the planet, as with the CrowdStrike outage, which Kat notes was a bug and not a vulnerability. Matty goes back to the Morris worm of the 1980s, when about 100 computers were on the internet and sysops hopped on conference calls because they were the only people affected. Today the whole thing is held together with "baling wire and good intentions."

Three Dependencies Deep

Matty says that complexity makes it worse, since a vulnerable package may sit 17 layers below the thing you use, and brings up left-pad and the earlier Who Owns Your Availability episode. Kat uses the Equifax breach as the example: a wildly outdated version of Apache Struts, which anyone could imagine they'd never let happen, until it's three dependencies deep and nobody knows it's there. Kat has credit monitoring for life because of it. The breach is also how the industry got software bills of materials and a pile of security tooling.

Matty notes that an SBOM only helps after the fact, while something like Dependabot tells you about known vulnerabilities in what you already have, which is a bit more proactive. Both agree teams pull in far more dependencies than they need.

Most CVEs Stay Put

Kat says companies spend an outrageous amount of time and money mitigating CVEs, sometimes with whole teams, and still only handle the very high and maybe high ones. If the fix costs more than the potential exploit is worth, it stays: "Most CVEs just get left chilling there." So anyone who thinks their software is secure is probably wrong, and nothing is immune. Kat cites a Kubernetes Ingress NGINX vulnerability scored 9.7 or 9.8, which affected 38 percent of cloud environments, and says it was not fun from a maintainer's perspective.

Matty asks about the dependency of a dependency whose maintainer left five years ago. Kat's answer is to fork it and fix it: "It's open source, baby. PRs welcome," and if the fork is public you risk becoming the new person in the XKCD comic. Kat still thinks open source is the best and most secure way to build software, since flaws are found and fixed faster, though they're also exploited faster, and a community of dedicated nerds can outrun proprietary teams.

What Belongs in the Kit

Kat's list for anyone who should pause the episode and install something starts with observability and monitoring, such as OpenTelemetry or Prometheus, because "you cannot just be like raw-dogging production": you need to notice weird traffic from a weird location hitting a weird endpoint. Next is alerting smart enough that engineers aren't firehosed with nonsense, then dependency management, meaning you know your dependencies and theirs. Kat adds infrastructure as code, instead of clicking through the AWS console, since most DevOps tooling has a security angle, and an SBOM, which Kat admits everyone is tired of hearing about after roughly three years of KubeCon talks but which still matters.

Kat's employer makes very small container images, most with zero CVEs, rebuilt automatically when an update lands, which gives an application a clean starting point, though it can't fix questionable code a team adds. Along the way Kat and Matty decide that raw-dogging alone wouldn't earn the explicit tag, and then Matty swears and it does.

Touch Grass

Kat says security and computers are a burnout factory and begs listeners to have a hobby that doesn't involve computers: "literally go outside and touch grass," especially anyone on the "mitigating CVEs treadmill," a much less fun cousin of the loot treadmill in games like Diablo. Matty recalls a friend who took up piano and kept it off social media for the same reason, and admits that all of Matty's own hobbies used to be reframings of work. Kat floats a miniature rage room with racks at conferences, like the therapy dogs and baby goats that have shown up at events, and both complain about status LEDs that can't be turned off on printers and routers. Kat's router lives in the bedroom, under a t-shirt.

Nuance in a Minefield

Matty frames the question by saying people are bad at nuance and asking someone's opinion on AI is like asking what they think about computers, since some people picture generative art and others picture assistive tools or agents. Kat lists the ethical threads that deserve attention: intellectual property and whether creators can keep a roof over their heads, accessibility and whether LLMs open the digital world to people who need it, and the ecological and academic impacts. Kat compares it to security and documentation, which end up being everyone's responsibility, and expects everyone to need enough AI knowledge to tell where the tools stop and where humans have to start. Even abstaining means gaining that awareness. Matty adds that the less educated we are the more likely we are to be steamrolled, and that for some uses "the juice is not worth the squeeze," like burning resources on a cute cartoon of a friend.

Kat notes that any line drawn today can move tomorrow as the tools change, and that LLMs have changed how long a tech career seems feasible given a tendency toward carpal tunnel. Kat also points out that significant models already run on a MacBook, which Kat compares to the room-sized computer that became a pocket one.

What Worked and What Didn't

Matty uses ChatGPT for alt text on social media, which makes doing it well faster. With Cursor, Matty's good experiences come when Matty already knows what to build. One was a private website for friends to watch old videos, with S3 and signed URLs, in a well-trodden React setup where Matty had the architecture in mind and Cursor implemented it. The bad one was asking Cursor to write a Steampipe plugin for Bluesky: "oh boy was that terrible." Matty concludes vibe coding has not made Matty worried that engineers will go away.

Kat says context is everything: about 30 to 50 percent of a context window goes to building context, including recent library versions, docs for new functions, and the project's structure and hygiene. Kat recently spent about 90 minutes on context and planning, with a task file where a markdown checkbox marks not started, in progress and done. When Kat told the agent to knock out the tasks, the work Kat expected to take four or five hours finished in a few prompts, leaving time to review the code line by line and check whether the first version was the user experience the team was after.

One More Thing

Both describe the rabbit hole: "just this function" becomes "just this entire feature." Matty ended up awake until 5:30 in the morning on a refactor of a podcast Hugo theme. Matty notes an upside, which is that an agent waits patiently with its context, so returning after two weeks costs nothing. Kat says burnout from layoffs and volatility has been real, and that Kat uses an LLM as an executive decision-making regulator, asking whether a tangent moves toward the milestone, which has helped with planning, estimating and sticking to a plan without difficult conversations with a manager.

Treat It Like a Junior Colleague

Matty compares working with agents to a senior engineer working with a junior, where mistakes are expected and the process of plan, develop, test, iterate, document and commit exists to catch them. Kat adds that LLMs were trained on GitHub and respond well to work framed as GitHub issues: have the agent write an issue, edit it to the real requirements, then tell it to pull the issue and work it. That demands good code hygiene, with modular code, documented interfaces and an easy data model so a context window stays coherent for a one to two hour pairing session. Kat also uses the GitHub MCP server to have the agent comment on issues when the plan pivots and to pick up from the issue history on Monday mornings, and thinks a service account would help distinguish what a person wrote from what the agent wrote. Matty keeps issues in solo repos and has one with over a hundred comments that are all Matty talking to Matty.

Guardrails and Private Setups

Kat is uncomfortable with data centers powered by gas turbines and plans to run a local setup, and cites a DevOpsDays Chicago talk by Paul Czarkowski, which wasn't recorded, about running open models locally on something like a Mac Mini. Kat doesn't want secrets in a context window, so secrets go in a file in the home directory that git commands can't reach, and the agent runs in a dev container, never directly on the host. Matty notes that an env file in the agent's workspace is exactly what it can read. "It's important to try and make it safe to make mistakes," Kat says.

Matty's last caution is that an agent will troubleshoot things that aren't broken: it once tried to uninstall a plugin because it ran the wrong CLI command, and it will "fix" a function that was already fixed, since the context window is small. Kat says healthy skepticism is warranted because it messes up a lot and isn't replacing people.

Be Polite to the Robots

Matty says you should always be polite to agents, partly as a joke about who they'll remember and partly because being polite to them makes us more inclined to be polite to everyone. Kat's version: neural networks are loosely aligned with how our brains work, and reinforcing demeaning behavior in a chat reinforces it in us, making it harder to tell the difference between treating computers that way and treating people that way. "We actually have to respect our own presence enough to appreciate that what we put out in the world will also change ourselves."

The Echo Chamber

Andrew says the first hurdle in working default open is the echo chamber feeling: you post things you'd normally keep in closed channels, and either the same few people respond or nobody does, and you can't tell whether the message is landing or being read passively. DevRel also ends up as a counterbalance between what the community thinks of the project and what is happening inside the company. Building trust takes a long time, and the rule is to listen and incorporate the feedback you ask for, because otherwise you lose that trust.

The Gravity of the Company

Matty adds that the company behind a project always has more gravity, since the people paid to work on it attend every roadmap meeting, while volunteers and even people at another company's open source program office can only look in now and then. Andrew agrees that the loudest and most contributory voices are usually paid staff. Andrew's answer is to give other contributors gradients of engagement and a way to validate their experience and reward them, whether with swag, a platform, shared stage time, or simply being heard. People on the edges who never see contributors like themselves celebrated don't find a way in, so part of the job is teaching people to contribute where they are.

Matty compares Chef and Pulumi. At Chef, Matty says, "I work at Chef because I believe in Chef, not the other way around": employees were members of the community first, and community summits had staff and outsiders on equal footing. At Pulumi, when a community event was planned, it was hard to get employees to see that they were part of the community too and to join on a workday. Matty is careful to say it wasn't malicious, and that leadership has to reinforce it, since engineers with deliverables won't otherwise take the day. "No one plans for it to go bad," Andrew says, and adds that this is a universal experience in open source. Andrew suggests sharing the tools too, such as cloud workspaces that let outsiders build the software the way staff do, since internal build systems are where open projects quietly stop being open.

What Can Stay Closed

Matty raises the open roadmap problem: customers can't be named, and a feature may matter because one large customer wants it. Matty notes almost no company would give the community an equal say over the project it stewards. Andrew says "being default open doesn't mean that everything has to be open. It just means that you have to default to being open," and that it's fine to ask at the start whether something needs to be open. The key is to communicate clearly about what isn't, to keep trust. If community-built features make a product sticky in the enterprise, Andrew adds, some of what comes back should go to the community, for meetups, lightning talks and swag, "by sharing the spoils."

Opening Up Advocacy

Matty asks how to open source advocacy itself, since people whose title is developer advocate can't do all of it. Matty describes a friend who asked how engineering blogs work, and the pattern of nothing for three months and then 12 posts in a day, since engineers write when they can. Matty's fix is to have people whose job is content work with subject matter experts. Andrew loves an engineering blog run by engineers, since the community wants to hear from the React developer about a new design library, not executive marketing, and says it needs pairing with a marketing team to tie back to strategy.

Andrew's core idea: "The biggest thing that you can do as a developer advocate is to create more developer advocates," meaning people without the title who have opportunities to engage. Open source communities are seasonal, since people contribute when they have free time, and "you can't control the waves, but you can control what the waves do when they wash up on the shore." Advocates should share strategy and plans, and write retrospectives about conferences, demos and talks, so contributors get excited and come along. Andrew has had contributors ask for feedback on their conference talks, which means trust has been earned and that advocacy created another advocate.

Andrew's example is GitLab's DevRel team, who plan events in the open on GitLab with epics and milestones and post retros with photos, talks and contributors. A contributor in Brussels can see the team is coming in three months and sign up on the issue. Planning for sponsorships, collateral and demos tends to happen invisibly on Slack, Discord or Asana, and fixing that invisibility matters.

Writing Programs and Contact Pages

Matty says a contributor writing program is "so much work" that contacts at other companies who ran one mostly wished they hadn't, since it multiplies content production problems. It doesn't have to be a formal program: signal boost a post about your product, or invite the author to collaborate. Matty also pleads with bloggers to put a way to reach them on their blogs, because Pulumi wanted to syndicate community posts with canonical tags and couldn't find the authors, and some authors turn out to have moved on from the tool years ago, though the post is still good. Matty's own most visited page is an old post about SharePoint 2010 search in a one-way trust, written to document a fix for a team.

Andrew ran a community writing program at Mattermost, which is rewarding for contributors, especially those in emerging tech hubs, and draining for staff, and can drift into a side project that doesn't connect back to company goals. Community members often assume the company won't notice their work, Andrew says, when in fact staff are right on top of anything that comes in and want to lift it up, and helping them get past that feeling builds confidence.

Links

• Matty's blog post about SharePoint (including broken images!)
• Community Pulse podcast

Every Job Becomes Management or Marketing

Chelsea started out planning to stay on the senior engineering track forever and write code for a whole career, and has since concluded that the skills early-career developers dismiss as soft become nearly the entire job: coordinating within and across teams, making sure everyone knows a product exists and how to use it. The system turns out to include people, regulations and corporate bureaucracy as well as the repositories. It's a lesson people seem to have to reach themselves, like advice to a friend in a bad relationship, and Jessica's version is that you don't break up with your technical skills, you open the relationship. Chelsea adds that you don't get to build things unless people want them, at least not if you want to avoid shelfware.

Knowledge Work Is Not an Assembly Line

Chelsea says measuring engineers on productivity metrics inherited from an assembly line, where nails accumulate linearly through the day, is a disservice. Software is knowledge work: "We don't create value by doing the same thing over and over." It creates value by gathering context into an understanding of how to solve a problem now while keeping as many likely directions open, which Chelsea credits to Kent Beck's term optionality. From outside, that looks like nothing, nothing, nothing and then a big release, after months of understanding the problem, talking to users and socializing changes, since changes that torch people's context take power away from them. Writing the code comes last and is the easiest step. "We are treating lines of code like nails." Jessica adds that "our job is not what we do. It's what we know," and Chelsea says much of the day job is finding knowledge for someone and routing it to them, and that early in a career, how much code you can write depends on how good the decision makers about the repository are.

Buying MLOps Tools

Mozilla's values include data privacy and ethical use of machine learning, which are hard to prioritize when teams spend their energy on getting any model into production through individual heroic efforts. In the fall a team began evaluating products to give data scientists and machine learning engineers a turnkey path to production. Chelsea notes the product landscape is young, with upstarts of around 15 employees, so the larger asks sometimes can't be met and documentation doesn't always match behavior.

Chelsea's advice on evaluating them is to separate optimizing metrics, where more is always better, from satisficing metrics with a good-enough threshold. Engineers tend to treat everything as an optimizing metric and end up in decision deadlock, for instance over scale, when a small startup isn't going to see a billion users at once. Jessica: "Problems you want to have." The optimizing metric that often isn't on the grid is the availability and flexibility of support. Chelsea fought to include it, and found that "the products we chose that have really, really responsive support teams are the products that we have managed to get into production at this point." A support engineer on Slack, for example at Weights and Biases, will take a custom chart problem and reproduce it in their own project. Free and open source tools the team liked ideologically didn't make the cut, since there's nobody whose job is making sure it works for the people using it, which Jessica sums up: "Wow, it's almost like the code isn't the whole thing." Jessica adds that at Honeycomb, good support gets renewals.

How MLOps Differs from DevOps

Chelsea came to MLOps from software engineering through data science, not through DevOps, and describes a data engineer as someone thrown into the model, the data science code or the app code as needed. Chelsea's view is that it isn't necessarily different in kind but that machine learning models add special considerations. Jessica compares deterministic program execution to the internal combustion engine, now one category among vehicle types. If the same input produces different output, that would worry a DevOps person and not necessarily an MLOps person, though there's a different decision tree, since some causes are fine and some are awful, and diagnosing a malfunctioning model is harder because the internals are automated.

Chelsea tells of someone distraught that ChatGPT claimed to have run code and reported wrong output, and of spending too long explaining the mechanism when the question was really "how could ChatGPT do this to me?" The person wanted to imagine a world where the tool had an interpreter, and Chelsea's answer was that one can imagine that world, but it's not the one we're in. A software engineering background alone isn't enough to debug these systems, and tools built to operationalize deterministic code lack needed features.

One such feature is checking that production data still matches the test data the model was evaluated on. Chelsea cites Andrew Ng's Machine Learning Yearning and a contrived example of a cat-photo model trained on high-quality images that then meets blurry phone photos. At Mozilla, Chelsea works on a system that sanitizes search data, discarding anything that might contain personal information. It drops anything with numerals unless it's on a human-made allow list, and uses spaCy's named entity recognizer to filter names. The team has to be sure the people using the feature are in the population the recognizer was trained and tested on, so it monitors an aggregate distribution of languages in search data, without storing the searches, to catch a shift before anything is stored long term.

Chelsea says MLOps and DevOps share a focus on catchability. There are three risk amplifiers: catastrophicness, likelihood, and insidiousness, "how likely is it that this thing goes uncaught if it happens." Security, DevOps and MLOps focus on insidiousness more than other roles do.

When to Use Machine Learning

Chelsea's view is that the most effective systems are a series of steps with humans in the loop, not one generalized system doing everything. The example is who-to-follow recommendations on social media. A popularity-based model produces "the Beyoncé problem," which spirals up the already popular and doesn't connect niche audiences. A person can often tell when someone has gamed the engagement system, and human judgment is hard to automate and, Chelsea adds, hard to beat with a model, even though it's spotty at best. Better would be to break it into three simpler steps: find what topics people discuss, figure out who is knowledgeable on them, probably with a human in the loop, and recommend those people to people who want to learn. Chelsea notes that Follow Fridays and hashtags were user inventions, and that classical models on tabular data are often enough. Jessica's summary: for any problem where you'd reach for machine learning or generative AI, break it down into parts where a model helps, parts where a deterministic rule works, and low-volume, high-value parts where a real person should be asked.

Chelsea writes at chelseatroy.com, linked below.

Read more of Chelsea Troy's writing here!

Sponsors and the First Sponsor Deal

Matty tells the story of the show's first sponsor. PagerDuty reached out, and when Matty and Trevor didn't know what a sponsorship costs, they said $50 an episode, and Matty's one smart move was limiting it to a six-episode deal. Later Matty heard PagerDuty's ad on another podcast and learned at a conference bar that the other host had been paid $1,000 an episode for a similar audience. The renewal went to a number between the two. Years later, after Matty worked at PagerDuty, a sponsorship of the show drew lawyers who needed to be sure it wasn't double dipping. Matty adds that some sponsors come for a couple of episodes and some keep returning, and that the long back catalog helps.

How It Started

Arrested DevOps began as a blog Matty never wrote, named by a friend who is a writer. Matty had learned DevOps from podcasts like the Ship Show, DevOps Cafe and Food Fight, and wanted a show for people who didn't know who John Allspaw was, the ones whose boss "read about DevOps in the in-flight magazine." Matty met Trevor at an Azure meetup in Chicago and invited Trevor to co-host, since Matty knew operations and wanted someone with a developer background. The two then got on a Hangout with Nathan Harvey and a fellow podcaster for advice, where the banana stand tagline was coined. Matty also has a lost episode 0, a five-minute recording testing Hangouts. Jessica says the "banana pants" variant came from a first episode where a script-reading panic and a lot of Dora the Explorer produced "in the banana pants."

The first episode with a cold open is 44. Joe picked them, and Joe's rule was: "If there's interesting profanity, that's the cold open." Matty also runs through the guest spiel: make peace with the explicit tag, stop and restart if you slip, and clap loudly if you blow an NDA, which has happened exactly once in ten years.

How the Hosts Joined

Bridget appeared first as a guest on an episode Matty remembers as being about conferences, whose art was a photo of Bridget's badges, published September 23, 2014, and Matty's checking turns up that the first episode Bridget hosted was November 18, 2014, DevOps in the Enterprise. They disagree about whether the invitation came at the DevOpsDays Chicago afterparty. Jessica had been on the show, liked it, and asked to join, after the two met at the first Redeploy conference, where Matty was a fan of Jessica's other podcast, Greater Than Code. Jeff Smith joined as an occasional host, and Joe, who started as the editor, also became a host, with a different editor before Joe. Since Joe went back to full-time work, Matty does the editing, which explains the quality, and the tail of each edit is worse because Matty got tired as an editor.

Live Episodes and GOTO Chicago

Bridget recalls curating a GOTO Chicago track as five live podcast panels in a row, with Matty, to an audience, in a breakout room with an air wall, and a guest loud enough that a proctor asked them to stop yelling. The first of them was Old Geeks Yell at Cloud with Andrew Clay Shafer and Bryan Cantrill. Bridget's takeaway: "If you're going to take on way too much, just have great guests. That'll cover a multitude of sins." Joe's tip for live episodes is to line up the AV crew ahead and pass a single mic, and says live episodes are easier to edit because fewer people talk over each other. The first live one may be eating sushi with Andrew Clay Shafer at DevOpsDays Minneapolis 2015, and Matty adds that the title was aspirational: the sushi took years to happen. Matty also notes that the DevOpsDays site doesn't handle a speaker called ADO well.

Memorable Episodes and Fans

Bridget highlights Who Owns Your Availability from 2016, recorded the day of the left-pad incident, as still current for software supply chain discussions. Matty calls out 2023's three platform engineering episodes (Daniel Bryant, Pete Cheslock and Matt Kuritz) and says "I still think that Pete is right, that platform engineering really is DevOps with better marketing," with the earlier platforms episode with Kelsey Hightower and Andrew Clay Shafer always linked. Jessica picks the one-on-one conversation with an enterprise practitioner, since those stories show real companies struggling and let listeners know they're not alone.

Fan stories: a stranger at KubeCon Chicago told Bridget that a computer science course uses the show. Matty recalls an early fan letter that went unanswered on LinkedIn for years. Matty credits the show for much of a career, since "I owe a fair amount, if not all of my career, to this podcast," and notes how easy it is to be a guest on someone else's show. Matty also mentions using the same pun twice, in database episode titles in 2019 and 2023.

Stats

The most frequent guest is Andrew Clay Shafer with eight episodes, about 4 percent, then Nicole Forsgren and Sasha Rosenbaum with seven each. Matty says downloads stood at 1,911,233 plus about 35,000 on Spotify, since Spotify hosts its own copy and doesn't show in the regular stats, so two million is near. Episode 1 is most downloaded, then episode 92, CI/CD Oh My with Jez Humble. The most viewed YouTube video is Old Geeks Yell at Cloud at about 16,000 views, then Bryan Cantrill's fireside chat at about 8,000, and Matty jokes that the show should have Bryan on more often.

Matty also fixes episode data live and discovers guest entries were lost in migrations: the site started on Jekyll for about seven episodes, moved to WordPress with a custom plugin, then to Hugo. After Bridget found a Google alert for another podcast using the theme, Matty turned it into Castanet, a podcast theme that is now poorly maintained. Jessica's verdict: "our memory is terrible and our data is also crap."

Where Everyone Is Now

Trevor is job hunting, looking at solution architecture or DevRel, and painting miniatures. Jessica is in a second year as engineering manager of developer relations at Honeycomb, is at a conference run by the Linux Foundation that added AI content to a Cassandra event, and would like the show to cover DevOps for LLM integrations. Bridget works in product at Microsoft, which means listening to other people's stories more than telling them. Joe travels for work to hotel ballrooms in Nashville and Cleveland. Matty leads developer relations and growth at Aiven, a Finnish data platform company, did not reach United's top tier this year, pays around $100 a day for dog care when traveling, and runs the team with a social contract about staying off Slack on holiday. Matty adds that because the company's field isn't Matty's expertise, leading the team works better than trying to do the work.

Bridget's closing idea, for anyone tired of telling the same stories: step back, help a colleague prep a talk, and "we can always make room to inspire someone else or to teach someone else or to go to a conference that we're not even speaking at."

• Every ADO Cold Open Ever
• "Episode 0" of ADO
• "Old Geeks Yell At Cloud" video

Nobody Wants You to Change Everything

Matty's warning for new senior hires: no matter how often people say they want a change agent, "here's the secret: they do not." Ben calls it the hero complex of hiring, the lowercase-m messiah, and describes doing something new in interviews: sharing a real, still-raw failure instead of a humblebrag, on the theory that if Fuel Labs still wanted Ben afterward, the fit would be real. Matty recalls the first manager job at Apartments.com, a role that was half manager and half sysadmin, where Matty arrived fired up about process and after about four weeks realized it was time to stop and listen. Matty's rule: "you have 2 ears and 1 mouth," and "every process, every way of doing things in a company is organizational scar tissue." The likelier explanation for something being done a certain way is that there's a reason, not that nobody has heard of Salesforce.

Ben contrasts the invading colonizer who uproots things with the person who sits with people, listens to their stories and helps shape direction from within, and admits that's hard when you were hired to do things.

The First Weeks

Matty describes starting a role while knowing a new boss would arrive in three months. A coach suggested using the time for discovery so that the new boss could be handed what Matty had learned and a plan, without having made changes yet. Matty's onboarding technique: in every get-to-know-you meeting, mostly about the person and not the job, schedule a follow-up four to six weeks out, when there's enough context for a substantive conversation. Ben adds asking each person who else to meet, which widens the circle quickly, and meeting ecosystem partners and developers who build on the company's infrastructure. Matty recommends The First 90 Days, with the caveat that it reads like Harvard Business Review articles stapled together, and that it helps with figuring out what kind of organization you're in.

Matty also uses a model from an executive coach, Bill Joy, who taught engagement and skill as two axes: a new starter is high engagement and low skill, even a principal engineer or a new CEO. "You could be the new CEO. You are low skill." Activity for its own sake looks busy but doesn't win collaboration, and Ben says DevRel suffers from filling calendars without knowing what moved the needle.

Showing Value

Matty's standing advice is to learn how the company makes money. In DevRel interviews Matty's philosophy is that "if you don't have a way of demonstrating value, a way of demonstrating value will be assigned to you and you won't like it." Take care of the things tied to business outcomes and nobody will question the rest. Matty jokes about wanting a Nicole Forsgren for DevRel. Ben says the same applies to SDK engineers, technical writers and DevOps: do less, more completely and at a higher standard, instead of covering everything.

Ben's Plan

Ben inherited a team of three, dispersed across the Americas, each there about a year, and wants a team "that knows more than you." Ben's first career was as a director at nonprofits, and Ben's earlier stint as a young manager was marked by opinions, talking more than listening and buying into the change agent paradigm, which cost Ben. This time Ben has been listening and taking notes, then drafted a quarterly plan, shared it with Ben's own boss, then the team with an invitation to tear it apart, and scheduled an offsite to build Q1 objectives and key results together. Ben's method is to be a facilitator and consensus builder, and says "ownership is how you create a sense of culture of staying." A key goal is career paths for the team, since nobody had held the role for a while, recognizing what people did in the past year and getting them to where they should be.

Team Lead Is Not a Manager

Matty points to Lindsay Holmwood's post "It's Not a Promotion, It's a Career Change," and tells how a new CTO suggested moving Matty from director to infrastructure architect because technology, not managing people, seemed to be what got Matty out of bed. Matty's conditions were no pay cut and a letter making clear it wasn't a demotion, which Matty says illustrates the problem. Both believe individual contributors should be able to earn more than their managers, and Ben cites Aaron Bassett's talk that promotion shouldn't require becoming a manager. Matty says team lead and people manager are very different jobs: leading an open source project is not performance management, career coaching or time-off requests, and as Ben puts it, "you're never putting an open source contributor on a PIP."

Matty credits Apartments.com for heavy manager training, since many managers were first-timers from internal promotion, and Bill Joy's framing that the hardest transition is the first one, from individual contributor to manager, because you must stop doing what you're used to. Matty also says that leading a DevRel team in data, and not in a space like Kubernetes or infrastructure as code where Matty knows the material, has been easier to do humbly, since "I couldn't get a job on my own team," while still knowing how to do DevRel. Ben calls that humility a good topic on its own.

• The First 90 Days
• "It's not a promotion - it's a career change" (Lindsay Holmwood)
• "Not All Leaders Are Managers" - (Aaron Bassett)

How DevOps Became a Department

Jeremy's first exposure was a DevOps Days in Atlanta where John Willis spoke on burnout, which showed that DevOps wasn't just about playing with Jenkins. Before that, Jeremy says, nobody thought you needed a department to deploy Jenkins servers: two people on a tools team built the machinery the rest of engineering used to ship. The real contribution of the movement was refocusing on the idea that developers are humans. Then, as with big data and digital transformation, big companies took the ideas, put them in a department with pay scales, and turned infrastructure teams into DevOps engineers doing the same things. Jeremy sees platform engineering as "the next logical evolution" of what DevOps should have been.

Matty adds that early DevOps deliberately avoided being prescriptive, with no manifesto, just the CAMS acronym of Culture, Automation, Measurement and Sharing from John Willis and Damon Edwards at the first US DevOpsDays in Mountain View, to which Jez Humble later added Lean, for CALMS. People still have to do work, and automation is where vendors make money, so DevOps drifted toward meaning Jenkins and Puppet. Jeremy compares it to Agile: a manifesto that gave rise to SAFe, which Jeremy calls garbage, and also to good things like Kanban and Lean. Matty adds that "SAFe is how to have Agile but let project managers still have a job."

Measurement and Incentives

Jeremy says the business community got its claws into both movements with Scrum, velocity metrics and somebody-has-to-get-fired accountability, which produces walled gardens, bureaucracy and a pathological culture instead of a generative one. Matty says measurement doesn't equal Taylorism, contrasts Nicole Forsgren's work on developer productivity at GitHub with McKinsey's, and names Freakonomics as the most important DevOps book: "Go learn about incentives and then you will understand DevOps."

Jeremy says what engineers do is hard to measure, so organizations reach for features shipped or hours against velocity points, and "A measurement ceases to be a good measurement when it becomes a target." Jeremy points to the developer productivity work of Abhinav and DX, which focuses on happiness, and cites Antifragile for thinking about teams that survive someone leaving or breaking an arm. Jeremy says Google, Amazon and Facebook get this right, with engineers empowered to deploy and build their own platforms, while Jeremy's retail clients are stuck in a 1990s CEO mindset, with Nordstrom as an example of a real turnaround. Matty adds that Courtney, who led that transformation at Nordstrom, has talked about internal platforms for the enterprise, linked below.

The Frozen Middle

Matty describes hearing at Chef that about 80 percent of CIOs said DevOps transformation was critical but only about 15 percent of enterprises had a plan, and says the C-suite can think big while middle managers whose job is executing one way of measuring will fight a radical change. At PagerDuty, banks had whole teams of incident managers, and they weren't bad people: "I would probably have done the same damn thing."

What Success Looks Like

Jeremy says most of the Fortune 500 now know DevOps is a thing, and successful teams have trust to make decisions and a paved golden road to production, with a platform or DevOps team, whatever it's called, building the frameworks while software engineering teams cross-skilled in things like Terraform get their apps to production without interference. The cloud means developers no longer wait two weeks for a Tomcat server. To avoid repeating the mistake, Jeremy says to avoid commoditization: "everything that's wrong with software engineering today is commoditization and value engineering." Jeremy builds teams with overlapping skills, "I don't want specialists in 2023," and says platform teams are software engineers, "They're not infrastructure people anymore." Matty wraps up with the 2014 episode How to Eff Up DevOps, linked below.

• John Willis’s talk at DevOpsDays Atlanta 2016 on Burnout
• https://platformengineering.org/talks-library/internal-platform-enterprise-courtney-kissler
• ADO - How to Eff Up Devops with Pete Cheslock, Nathen Harvey, and Randi Harper

Where Runtime Analysis Fits

Matty's guess at the definition is "analyzing during runtime," and Brian confirms it. The difference from the APM and observability tools the industry is used to is where in the workflow it happens: on the developer's laptop or in CI, before deployment. Static analysis has become a commodity for classes of problems like vulnerable dependencies, and Brian says a team that isn't using it is delinquent. But there are problems developers assume they can only catch by sending code to production and watching it with an observability tool. Brian says "I hate to say shift left, so I'm going to say shift closer."

The cost of finding out late, Brian says, is context switching: a developer who has moved on to another pull request hears days later that a change is a problem, and the issue tends to land on a Jira backlog, where an SRE compensates by adding CPUs and RAM, which Brian ties to crazy upside-down hosting costs. Tests and manual QA already generate runtime data that many teams ignore. Matty adds that the point is getting 80 percent: fewer defects reach production, so the remaining ones might actually get fixed, and Matty compares the backlog to Homer Simpson balancing the garbage. Brian says Dependabot and static analysis didn't end pen testing or dynamic testing, and each of them is a signal, but teams over-rely on observability tools and crank up APM spend.

What It Finds

Brian's best-known example is the N+1 query from an ORM such as Hibernate or ActiveRecord, where a static analyzer sees that an ORM is in use but not the queries generated at runtime or how many identical ones are issued while paginating. Others are dependency injection and dynamic library loading: a static tool sees a config file and can't see what gets loaded. Runtime analysis watches where "the infinite gets constrained down to the finite," and can show that seven or eight listed libraries are never used and whether the ones in use are used in a vulnerable way. Matty connects this to the Sysdig report finding of loaded but unused JavaScript packages in the Cloud Native Security episode.

What It Looks Like in a Workflow

Brian says tools in this class have to be automated, fast and fit the existing workflow: in VS Code, IntelliJ or PyCharm, pressing the run button instruments the application, watching bytecode in Java's case. That sounds like an APM, but it collects more specific data, between observability and a profiler, building a dataset of which functions, queries and libraries run. Analysis comes in automated form, like flagging an N+1 query, and human form. Brian's example: a unit test triggers six occurrences of an N+1 query that's a tiny fraction of CPU time, but the developer knows production data would make that six million. It's also definitive: "this really happened," not a fuzzy prediction.

Brian expects the concerns to be noise, since static analyzers started out with many false positives, and expectation of magic, but runtime analysis only observes, and is bound by how long the tests take, which developers run anyway.

The OWASP Top 10

Brian points to the OWASP Top 10 as an illustration: around 2010 SQL injection was at the top, and then tools and frameworks commoditized the fixes. Brian recalls a 33,000-page automated pen test report for a big bank where prepared statements fixed about 27,000 pages. What replaced it are harder problems like broken authorization and authentication, cryptography and secrets handling, which static analysis mostly can't detect. A secret might pass through a framework that logs it, and a runtime analyzer can find the code paths.

Getting Started and AI

Matty mentions Adam Jacob's approach for Habitat, installing the gnarliest enterprise software to prove it, and says that's not the way to start. Brian's answer splits on whether you have automated tests. With tests, put runtime analysis in the editor and in CI. Without tests, instrument locally, on a laptop or a UAT environment, and look at the reports, or "personal observability," which is safe because nobody needs tests to run in production. Matty: "monitoring is simply testing with a time dimension." Brian says you should still write tests.

On AI, Brian says most AI code reviewer apps just look at the same static diff, while runtime analysis data is a new dataset. In experiments, adding runtime data to an LLM prompt stripped out noise and produced answers like how to mitigate a known N+1 query. For anti-patterns, Brian can't think of one on the spot, and says the risk is overreach: a tool that becomes "a mosquito in your eardrum," as static analyzers did with their 79 vulnerabilities until Dependabot started opening pull requests. Brian ties the category to the DevOps hangover of overspending on observability.

• OWASP Top 10
• Stripe: The developer coefficient (quantifies the cost of bad code to companies to be $59B annually)
• Facebook: FAUSTA: Scaling Dynamic Analysis with Traffic Generation (how runtime analysis was used at WhatsApp to catch design flaws before they reached production)
• Dragan Stepanović - Async code reviews are choking your company’s throughput (from LAS 2022, a talk which highlights the systemic problems with developers trying to do manual code reviews of large PRs)
• AppMap, the runtime analysis company which Brian works for
• Cloud Native Security with Michael Isbitski ADO Episode

Are We Better Off?

Michael's question is whether operational availability, debugging and troubleshooting are better than in 2004 or 2005, given that "what we keep doing is inventing new problems and then inventing new solutions." At CircleCI the availability struggles came from "doing really complicated shit," not bad engineers. Both recall 2003: Michael was writing software to replace spreadsheets and manual administration with SSH and for loops, managing thousands of servers, while Matty was moving from Exchange 5.5 to Exchange 2000 after an acquisition, and rebuilding dev servers at Allstate from an answer file on a floppy and a ProLiant CD while sitting in a cold data center with a book. Matty says you could hold the whole stack in your head, and asks whether microservices and distribution have left anyone better off. Michael says unequivocally yes in many scenarios and no in many others, and the first thing to understand is whether you actually have the scaling problems or just think the tools are cool. "Keeping one thing online is easier than keeping 25 things online."

Build for the Problem You Have

Matty recalls a Rails app being "up in 3 days, down in 3 months," and Cars.com handling Super Bowl traffic by renting servers for 48 hours instead of re-architecting, since the spike happened at the same time every year. Michael says to find users and product market fit before investing in a service mesh and service discovery: "don't underestimate the power of rsync and cron," and a colo, Linode or DigitalOcean node can go a long way. The test is whether you're delivering the value, and time spent working out whether a Kubernetes minor version will change an ingress controller API is something no customer pays for. Matty adds that "you" is doing a lot of work, from a 12-person startup to JPMorgan Chase, and tells the story of a retailer's ops team insisting on site stability until management said the job was selling things.

How Containers Led to Kubernetes

Michael walks back the chain: a developer environment needed to be consistent, so "we're going to package up your laptop, we're going to pass it around until it gets to production," which is what a container is. That led to a scheduler, service discovery, a service mesh, and businesses scanning containers for vulnerabilities, all layered on top instead of asking why the decision was made. If everyone developed in the same reproducible environment, Michael says, containers might not be needed, and then neither would the rest. Michael misses typing service start and strace, where a problem takes ten seconds to find, instead of launching a debug pod. Kubernetes "was originally designed to solve Google-scale problems. Unless you are Google, you do not have Google-scale problems," and Michael's image is that not everyone needs Everest climbing gear to cross the street. Matty: "What's the best container scheduler? The one you don't need."

Organizations and Data

Matty describes booth conversations at an AWS Summit where Kubernetes people said data was another team's job. Matty is clear that's an organizational pattern and not a failing of the engineers, and says a platform engineer should care because data is part of a platform. Matty used to accept that DevOps means never saying "that's not my job," and no longer does: "not my circus, not my monkeys" is fine, and if your job is limiting, "maybe you need a bigger circus."

Michael adds that platform teams exist partly because the complexity was put inside the company, and asks whether they could have a smaller mandate. DevOps ideas of shared empathy and pain were good, Michael says, but operations expertise atrophied and developers reinvented tools, so a problem solved in 1995 gets rebuilt in 2018. "There's always a 27-year-old willing to redo everything you've already learned." Michael counts 150 AWS services, half competing with each other.

What to Do About It

Michael thinks simpler tools had a chance and missed: Docker Swarm beside Kubernetes, and a Rust tool like Docker Compose that runs plain processes without containers. Michael's wish is for a generation of tools that abstract the good patterns without all the complexity, solving the 80 percent case. Practical examples from Michael's company: a website behind a CDN and cache that could run on a Raspberry Pi with a cell modem and cost $5 a month instead of $600, and a monolith, with Knuth's line about premature optimization and "I hope that's a problem we have," since scaling problems mean users. One team with one microservice is a success; most places end up with more services than developers, and then Backstage to keep track. Michael adds that "only in software is legacy a bad word": a legacy system made the money, so don't be mad at it.

Inside a large organization, Michael suggests shortening a workflow from 12 steps to 10, automating the repetitive debug steps, showing a decision maker two workflows and asking what you lose, and learning that "you're still a technical decision influencer." Matty recalls learning from a colleague's resume at Chase that treasury services processed $1.5 trillion in wires a day, information that sat on the business unit's intranet home page. Michael describes a Caterpillar division where every transaction ran through six servers, about $6 million a day, which made an $80,000 software upgrade an easy ask. Michael suggests reading an S-1's risk section to learn what matters to a business, and both lament that value stream mapping is discussed less than it used to be, with Matty blaming Steve Pereira no longer going to DevOpsDays.

Michael's closing ask: understand the outcomes at the other end, even approximately, so that "my complexity is built because it achieves this goal," and send in stories of solving a problem with a simple solution like installing an RPM and hitting start. Sometimes, Michael notes, you can't do that 300,000 times, and then you actually do have the problems the tools were designed for.

Nobody Has to Explain DevOps Anymore

Matty frames the problem: code is close to immutable and easy to roll back, while data is living, and orchestration tools work well until someone asks about the data and it becomes somebody else's problem. Grant says that ten years ago every data person needed an explanation of what DevOps even was, and now "you don't have to talk about or explain what DevOps is anymore," which saves half an hour of every talk. Grant uses Donovan Brown's ordering of people, process and products.

Matty asks whether the organizational divide has improved, based on conversations at an AWS Summit booth where infrastructure people treated databases and Kafka as another team's job. Grant says it's getting way better, with a split between companies that build DevOps in from the start and incorporate data management, and companies that add DevOps later. Grant also notices that database administrator is a term going away: people who do backups, availability and query tuning now call themselves data engineers. Matty compares it to sysadmins becoming DevOps engineers with a pay bump, and both say to change the title if it helps, since the work matters more.

Still Struggling

Grant admits being "a bit of a negative Nelly" here: "We are still struggling." The obstacle is persistence, since you can't toss the database and start over, and deployments have to happen without taking the server down for three hours. Grant thinks data management people haven't explained well enough to developers what they need, and developers often look at the database and see something scruffy. The joke that data work is like sweeping up behind a parade with elephants in it is, Grant says, how it sometimes feels, though the job is fun. Teams that have been bitten by data problems either embrace automation or avoid the topic altogether.

Databases Are Code

For the platform engineer who runs Kubernetes and leaves data to a data team, Grant's message is "Databases are code," and Matty adds "just big code." It's big, so it can't move fast and self-provisioned development databases need third-party tools or empty databases, but it can be treated like the rest of the code, with the special part being persistence. Grant says the biggest misconception is that the database can't be automated. It takes a bit more discipline than automating development because the data must be kept, but it's fully automatable, and wildly successful organizations automate their data management and deployments.

Communities and History

Grant finds the PostgreSQL community as welcoming as the SQL Server community is most of the time, and notices a split in Postgres between committers oriented around academics, theory and science, and everyone else who uses it and wants help. Grant says the same split is familiar from DevOps, where "Peggy does DevOps" doesn't make a DevOps company. Matty draws on the history-of-databases talk with Kat Cosgrove, from 1970s Ingres through Postgres, and says knowing why things are the way they are helps. Matty recounts that the relational model's creator disliked rows, columns and tables and wanted something more mathematical, like tuples.

Grant says the old foundations persist because they work, "rebar inside of concrete," which the Romans used. Grant is building a LoRa and IoT project on Azure and still uses a relational store, since the data volume is small. Specialized databases such as Cosmos DB are like a specialty tool for taking antennas off a radio, and you still need a hammer and nails.

What's Exciting

Grant is excited by open source being everywhere: AWS, Azure and GCP all include or support it, and the divide between the commercial and open source camps is shrinking, though paid software will remain, as with Query Store in Postgres on Azure. Grant also expects people to try to put AI into production tooling.

Grant's advice for growing a career is to "assume automation from the start," then decide whether you enjoy the why of things, such as query tuning and design, or the how, such as automation in Azure and AWS, Kubernetes and containers. The local job market matters too: in Tulsa, Oklahoma, SQL Server dominates, while on the coasts Postgres is growing. Grant has submitted to PGConf Europe in Prague. Grant's last word: "Treat your database like code."

• Arrested DevOps - The Database: The Elephant in the Room
• Arrested DevOps - Data! Data! Data! With Francesco Tisiot
• Arrested DevOps - The New DevOps With Adam Jacob History of databases talk from Matty and Kat Cosgrove

Product Teams and Platform Teams

Matty starts from the Charity Majors post on the Honeycomb blog, which has a table contrasting platform engineers and ops engineers, including a row where SSH is a no for platform engineers. Matt Kuritz says the idea came partly from a diagram in Lean Enterprise about self-service operations building a PaaS, which Matt never wanted to build, but its goal stuck: in an ideal world there are no handoffs between developers and operations, and product teams own their software end to end. At The Farmer's Dog there are two kinds of team, product engineering and platform engineering, and "the only difference is who the customer is." Everyone cares about reliability and delivery. Matt Kuritz adds a disclaimer that if SRE or DevOps works for a company, there's no reason to drop it, and that some companies have to go deep on infrastructure.

Matty adds that nobody can talk about platform engineering without James Governor's line about "endlessly remaking remakes of Heroku," and says the point of Heroku was getting to value fast, with abstractions where they need to be, and not feeling like Dropbox. Matty repeats that "the best tool is the one you don't need. The second best one is a SaaS." Matt Kuritz says to "do what works at the right scale and then iterate and evolve from there," naming Vercel for front-end-heavy companies and Stripe as a business function nobody wants to run. The Farmer's Dog isn't building a PaaS but assembling a platform of preferred tools with glue where it pays off, which Matt sees as a pattern others could use.

How the Team Measures Itself

Matt Kuritz says the team has to be aligned with product teams on the goal, and their north star is ownership. The first milestone was continuous deployment, since every commit then has one clear owner who is responsible through the pipeline to customers in production. The team reached 100 percent continuous deployment of its applications after fixing a distributed monolith, and before any golden paths. Matt is frustrated by takes that say DevOps is dead and platform engineering is the future, and recommends Charity's DevOpsDays New York talk. The work, Matt says, wasn't different from what a DevOps team might do, but the language and structure changed: platform and product, not dev and ops. Ownership is still a multi-year goal, with each application having one clear owner, and "platform doesn't own anything."

For the product management side, Matt prefers to work hands-on with internal customers: do a real use case manually, maybe one or two more times, and only then invest in a code generator or golden path. "We don't upfront really decide anything. It has to be done at least once and put into production." The mistake teams make, Matt says, is making decisions in their own echo chamber.

Who Decides What

Matty's rule of thumb for what to standardize is whether something is an interface point between groups: a shared source control tool matters, the JavaScript form validation library doesn't. Matt Kuritz says platform steps in as the decider for cross-team choices, such as asynchronous messaging, where RabbitMQ, SNS, SQS, Kinesis and Kafka could all show up in one app. The team collects opinions from all teams and asks whether a tool that becomes the standard would be worth maintaining, and challenges assumptions. Matt says Kafka is powerful and flexible, with many ways to get hurt, and the team evaluated Confluent but couldn't find use cases yet. Matt describes path dependence with driving on the right side of the road, and says to be honest about migration costs. Matty adds that people who've never heard of a tool will say it's fine for their team, so the platform team has to bring the context.

Bounded Contexts, Not Silos

Matt Kuritz says "functional silos are unhelpful," but not having boundaries means everyone has to know everything, so the answer is bounded contexts in the domain-driven design spirit: vertically integrated teams mapped to business domains, like fulfillment and signup, that rarely need to coordinate. Jess Kerr's blog post on better coordination or better software supports the idea: build technical components that let people work asynchronously instead of perfecting the handoffs.

On the technical side, the team chose a monorepo, Node and one language, and not introducing another until necessary. The monorepo exists to practice trunk-based development and continuous deployment, and works only if there's one version, which is head. Upgrading Fastify across dozens of apps is one atomic commit that triggers the tests of dependents. The team has no Backstage or internal developer portal, since a code owners file in a repo organized by business domain is the directory, and Git is what developers already know. "What's our platform? It's a monorepo and a code generator. That's our platform."

Start and Stop

Matt Kuritz's thing to start is setting a long-term vision you can never fully achieve, an idea from Toyota Kata: for The Farmer's Dog it's that every dog lives its longest life. Then break it into 1 to 3 year challenges and near-term target conditions, like moving a 10-person team to three deploys a day. The thing to stop is copycatting, or cargo culting, which Matt ties to Feynman's Cargo Cult Science and to Lean being copied from Toyota by its artifacts like Kanban cards. In platform engineering it's launching Kubernetes and Backstage because other companies did. "Think about your problems, solve those problems from first principles, don't copy."

• Arrested DevOps - DevOps With Better Marketing with Pete Cheslock
• Arrested DevOps - Platform Engineering with Daniel Bryant
• Arrested DevOps - Platforms with Kelsey Hightower and Andrew Clay Shafer
• Lean Enterprise
• The Future of Ops Is Platform Engineering
• Charity’s talk from devopsdays NYC
• Jess Kerr’s blog that Matt mentioned
• Cargo Cult Science

What Happened

Ohad's account, as a participant: on August 10th HashiCorp changed the licenses of several projects, including Terraform, and a group of vendors and individuals wrote a manifesto asking that Terraform stay open source forever. When HashiCorp kept its decision, which Ohad calls "totally legit," the group started working on a fork. Cory adds that some CNCF projects began pulling away from HashiCorp tools, and that Massdriver wants to bet on a tool that the open source community keeps investing in, the "new lingua franca of infrastructure as code."

Matty asks why a license change matters to an ordinary engineer. Matty's concern is vagueness: depending on how a lawyer reads the license, distributing a project that uses Terraform could be a violation, and you can't always know who is competitive with HashiCorp. Ohad says the license was followed by a binding blog post of clarifications on August 21st and changes to the Terraform Registry terms on August 24th, which shows the community didn't understand the implications. Every fundraising round, Ohad says, means sending the license to investors, and a BSL license with a binding blog post and more updates is harder to explain than a clear alternative. Ohad contrasts it with SSPL, which exists to stop others from reselling a vendor's product as a service, and calls BSL flexible and, in Ohad's opinion, still vague.

Matty adds that enterprises already fight compliance just to use open source at all, and a complicated license makes that harder. Cory says Massdriver was affected and then wasn't, since it competes with Waypoint, which was carved out of the FAQ a week later, and that dynamism is what worries people. Cory gives examples of transitive dependencies: a CNCF project that uses Consul, projects pulling out Vagrant, and Jaeger considering removing a Go plugin, which is still MPL. "It is very much email us to figure out if you owe us money," Cory says, compared with how other projects handled BUSL changes.

OpenTofu

Asked what to call it, Ohad says OpenTF, and Cory breaks the news that will go public the day after recording: the Linux Foundation recommended a rebrand because TF could be confused with Terraform, so the project becomes OpenTofu and the binary is tofu. Ohad and Matty make the fork and tofu jokes. The project now belongs to the Linux Foundation, the natural step toward CNCF, and Ohad says it is meant to be a drop-in replacement: no code changes, just a different binary and registry. Gruntwork, the creators of Terragrunt and Terratest, and Harness are among the supporters, and Gruntwork has released a sneak peek of state encryption, which Matty recalls as a Pulumi differentiator since a Terraform state file can hold secrets in plain text.

Matty asks what makes the fork more than a burst of pledges, recalling a maintainer maxim: "as a maintainer, no is temporary. Yes is forever." Ohad points out that Terraform core was open source but stopped accepting community pull requests about two years ago, while OpenTofu has a five-member steering committee under Linux Foundation and CNCF guidelines, and has already deferred a community change to a later version as a new capability. Ohad says RFCs and voting rules are a work in progress. Cory hopes that teams replacing HashiCorp tooling will contribute and speed up a project that "really has been bogged down for the past couple of years."

The Registry

For modules and providers, Cory says the first registry will proxy requests to GitHub and resolve module names to repositories, and Ohad adds that provider naming conventions map to GitHub URLs. So publishers won't have to submit anything. Cory sees an opportunity to support OCI-compliant registries later.

How to Help

Ohad suggests starring the repo, joining the community Slack, filing and voting on issues, and sending pull requests, and sees an opening for large vendors and cloud providers to shape the project under CNCF. Cory suggests putting OpenTofu into CI pipelines, for example with Terratest, once the alpha registry is out, to surface edge cases in loading providers and modules. Cory's closing point is that the group is "a consortium of competitors" that get along well because they love the language. Matty promises to file a first pull request to nominate Cory's Instagram-famous Australian Labradoodle, Ziggy O'Doodle, as mascot.

• https://www.instagram.com/ziggy.odoodle/?hl=en
• https://twitter.com/opentofuorg
• https://github.com/opentofu
• https://linkedin.com/company/opentofuorg
• https://github.com/opentofu/opentofu

DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: NYC area, Chicago, Silicon Valley, Singapore, and London.

What DevOps Got Right

Adam starts with what went well by describing the year 2000: operations was separate from IT, siloed, and ran on long planning cycles, with developers requesting gear that took six to eight months to arrive and capacity planning a big deal. The organizations worked, Adam says, a lot like how people describe platform engineering today: operations stitches it together and builds systems so developers don't think about infrastructure, "and never the twain shall meet." The top achievement of the DevOps movement was recognizing a single continuum of work, and the pre-DevOps world was "objectively worse" in day-to-day work, tooling and what the systems could do.

Matty adds a story from leaving Chef: a customer engineer at a large Chicago financial organization apologized for not getting much done, and Matty, who'd watched them in the thick of it for years, said they had come a long way. Adam agrees and says the flip side is real, with teams wanting 100 deploys a day with no friction and still mostly making pipelines and hoping it worked out. Following the DevOps Handbook roughly as written, Adam says, leads to mediocrity, deploying once every six months, which is better than before but rarely great.

The Platform Engineering Smell

Adam says the platform engineering rebrand comes from the feeling that DevOps failed because outcomes didn't arrive, and the proposed fix "does smell a lot like what life was like in 2001": "let ops be ops, let engineers be engineers" with some software in between. Matty puts it as accepting that the silos can't be removed, "So let's just make sure the silos are better." Adam's answer is that the people who succeeded collaborated better, and that the tooling was never designed for collaboration.

Adam's history of automation runs from 1990s compute clusters and college labs, through ratios of 10 to 1, then 100 to 1, and up to 10,000 to 1 as EC2 and Facebook apps arrived, and each stage had to make up an answer for the world of the time. DevOps then "ossified the shape of the world into that shape." The Flickr talk shows it: a portal for deploying, feature flags, dark launches, configuration management, capacity planning and metrics, with deploys started from Subversion tags, so they "literally built a platform roughly the way that we describe it." Every piece of the stack has been rebuilt ten times in ten years without an appreciable change in outcomes, and Adam insists it isn't any one tool's fault.

Tools and Culture

Matty credits Adam with "tools influence culture, culture influences tools," which ends up in Matty's decks. Adam says regretting "the idea that it's about culture and not about tools was wrong from the jump," because culture is what you do, and a version of culture in DevOps was like being a lapsed Catholic: believing the right things hard enough. An organization will not become more collaborative without tooling that forces it, since individual people won't do it alone. "Tooling is culture because it's literally what we do all day."

Matty ties it to the book Switch, where a manufacturing machine that kept injuring hands was redesigned so that turning it on required both hands: make the right way the easy way. Matty's Asana fight over a defined process is the same thing.

Factories, Soccer and Collaboration

Adam says DevOps inherited factory metaphors from lean, but software is closer to a professional sports team or an orchestra: highly motivated specialists unified on one objective and making tiny decisions together in real time. Adam's picture is a soccer team retrofitted with pipelines, review by other people and no coaching during a Scrum window. The result was process and tooling that sucked out the creativity and collaboration, where people end up "working near each other at best." The most important insight of DevOps was that what separates great teams from okay ones is "the rate of collaboration," and nobody designed tooling for it.

Matty asks how much of this is outside the control of the people who can change it, comparing it to Agile needing changes in finance, sales and marketing. Adam answers that change is hard at scale but not insurmountable, and that the LivingSocial story of sales having already sold a removed experiment is a collaboration failure. At System Initiative, which Adam describes as "everything I believe at 12," one Miro board runs from the pitch deck down to an individual story, and Adam reads the whole strategy to everyone every Monday so engineers know why and for whom they're working.

Scale and the Enterprise

Matty asks about JPMorgan Chase scale rather than a 15-person company. Adam says large organizations can be convinced with data that DevOps is better, but they buy tools instead of building them, unlike Google and Facebook, whose tools are bespoke. The vendor's temptation is to adjust the tool to the customer's culture for a bigger check, which is "a tomorrow problem for tomorrow people," and Adam recalls a Cloud Foundry deployment that collapsed under its own integrations. The way out, Adam says, is to build a tool that teaches the right way, because enterprises reliably buy tools that promise better outcomes: "let the tool change them, not the other way around."

Matty pushes back that the cynic sees churn, and asks whether the tool will get used by the people in the middle. Adam says it will if it's good, pointing to Salesforce as sticky because "it's actually kind of good," and admits not being sure this is the only answer. What Adam is certain of is that "the status quo is not a good enough answer," since a new Terraform-like or Ansible-like tool will have roughly the value of the old ones. The thing not yet tried is changing the shape of the system: why a pipeline in the middle, why only one layer of the application, and "what even is an application?"

Simulations Instead of Code

Matty asks how System Initiative reasons about this. Adam starts from automation, which has followed one line of thought since Mark Burgess's computer immunization paper in the 1990s, and argues that code is a poor medium for collaboration and that feedback loops of 20 minutes or more, with an opaque Terraform plan, are too slow. Looking at adjacent fields, Adam lands on digital twins: Formula 1 teams simulate every variable of the car so only promising changes are tried at the track. System Initiative is an attempt to build a high-fidelity simulation of infrastructure, removing glue code and building the workflow into the system, and Adam says more people need to pull on other threads too.

Adam's closing hope is that people become willing to change the shape of the system and "throw some of the babies out with the bathwater." Matty adds that the past was the best available with the information at the time, and that maybe it was what got us able to learn these things.

Links to Resources Mentioned

• 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
• *The DevOps Handbook
• The System Initiative
• Switch: How to Change Things When Change Is Hard

DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: NYC area, Chicago, Silicon Valley, Singapore, and London.

You Already Have One

Cassandra's practical case is that a brand helps a career: speaking invitations come from what people know of Cassandra's work, and for jobs "the resume is kind of a formality at this point." Even an individual contributor needs some online presence so people can see what they're about. Matty adds that you have a brand whether you want one or not, since it's what people think of when they think of you, such as the SRE everyone asks about replicas, or the colleague who is difficult to work with. Matty brings up the Mad Men line, "if you don't like what people are saying about you, change the conversation," and says that we have some responsibility to take ownership of it.

From Accidental to Purposeful

Cassandra's own brand started by accident. As a recruiter told not to be the one who doesn't know Java from JavaScript, Cassandra went to Agile and cloud meetups, then followed everyone tweeting the #CodeMash hashtag, listened for a while, and started to participate, posting job search tips and skills advice. Developers turned out to welcome a recruiter as long as Cassandra was authentic and not working an angle, and invitations to panels, talks and conference organizing followed. Cassandra realized that being deliberate about what was posted could work in Cassandra's favor, hence the talk on a purposeful personal brand.

Matty's version: at Chef, Matty was known for infrastructure as code, and at PagerDuty, where incident response was the focus, nobody knew Matty for that despite years of doing it as a job. So for a year Matty took any stage to talk about incident response, blameless postmortems and learning from incidents. Matty insists that this wasn't inventing a persona, it was deciding what to be known for. When Matty's boss later asked for the same reputation in the ITIL and IT service management world, Matty's answer was that "you can't just decide to have authenticity." It takes time, and you have to be able to back it up. Cassandra says every role change means updating, not rebranding: the human skills of building relationships with a community's key leaders are the same whether the community is open source contributors or Microsoft MVPs.

Your Brand Inside a Big Company

Matty notes that almost nobody stays at one big employer for 20 years, so external brand matters even in roles that aren't public, and asks how it works inside an organization like Target or Visa. Cassandra says to choose opportunities that match who you are: not competitive, cutthroat cultures if you aren't competitive, and plenty of room for creativity if you need it. Matty adds that the tension is credit: talking about what the team did as well as what you did, since a story that is all about I doesn't show a team member, and ops roles are like a corporate lawyer, where nobody knows what you do until you don't do it. "If you don't, who's going to do it for you?" Nine times out of ten, Matty says, you think you're bragging and you aren't.

Three Questions and Framing Accomplishments

Cassandra suggests writing down answers to three questions, which are in the slides linked below. Who are you professionally: your top three technical specialties, three professional specialties and three contributions to the team and profession. Who are you personally: three interests and hobbies, three beliefs and values, and three parts of your social, family or community life you want to connect over. And how do you connect with people: shared interests, advice you're seeking and what you want to know about others. That gives you a bank of material. "Modesty isn't a career-enhancing trait, but that doesn't mean be an asshole." The trick is to frame accomplishments by who they helped. Cassandra's example is a KubeCampus Kubernetes Learning Day that sold out and gave 200 people an introduction to Kubernetes, which reads as "it helped all of these people" and not as bragging. Give credit to the team, and practice, even in a mirror.

Show, Don't Tell

Matty says the quickest way to get mocked on tech Twitter is to put thought leader in your bio, and recalls putting up a page of DevOps thought leaders a decade ago until J. Paul Reed asked to be taken off it. If you solve a problem, the story should be about the interesting way it was solved, and the expertise comes through. Matty also recommends keeping a brag book, an Evernote notebook of kind messages that started as a cure for imposter syndrome at Chef and turned out to help at performance review time and with seeing what's worth telling. Matty's DevRel team writes a monthly wrap-up for the company, framed around why it's useful to sales and others, and not as a pat on the back: "how do you brag by giving someone something useful?"

Making Time and Choosing Platforms

Cassandra keeps an hour blocked on the calendar every morning for social media, takes a day or two off the internet after a conference to avoid burnout, and pays attention to whether the people and things followed make things feel psychologically unsafe, since that makes content hard to produce. Platforms are shifting as Twitter usage declines: short-form video is growing, and Cassandra sees a generational split where Gen Z and younger millennials want to connect on Instagram while everyone else uses LinkedIn. The brand can stay the same while the tools change, so it helps to be on several.

Matty suggests blogging may have a renaissance, especially documenting what you're doing or learning in the open. Matty's most visited page is a post on configuring SharePoint in a one-way trust, written to document a fix for a team, which became the answer on an MSDN forum. Matty tells of Annie, who learned an infrastructure testing tool in public on a blog until the product team said "Annie's blog is our documentation now." Beginner content is the hardest to create and the most needed, Matty says, since hundreds of thousands of people need to install Kubernetes and only a dozen need to optimize sharding at a million users per second. Teaching also helps you learn.

Vulnerability, Asking and Paying It Forward

Cassandra is seven months into a role at KubeCampus, is teaching the networking fundamentals lab, and plans a blog series on the learning journey. "It starts with admitting you don't know." Nobody has ever responded by doubting Cassandra knows what a cluster is, they've offered to explain it. Cassandra's tip for new brand builders is to ask a question or for resources about something you want to learn, since developers love to share knowledge.

Matty adds that asking for help to learn is different from asking someone to do a task, and appreciation matters without genuflecting: "I'm not going to carry the bucket for you," but I'll give you a map, and sometimes you just need the bucket carried and can ask for that favor. Cassandra says helping people isn't transactional but pay-it-forward: Cassandra can't hire a former boss and mentor who is looking for a director of DevRel role, so the return on that help is paying it forward, and Cassandra passes the same knowledge on to others. Matty says to ask few favors and accept no, and that you don't need to go to the most famous expert every time, since thousands of people know Kubernetes. Cassandra's answer is meetups and conferences, where the hallway track has led to meeting an author whose Kubernetes book now helps, and a KubeCampus Slack for when learners get stuck.

Links to Resources Mentioned

• Keep a Brag Book
• KubeCampus: Free Kubernetes Training
• Purposeful Personal Branding - Nov 2018 Slides 13-15 have the 3 questions
  
  *DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: [NYC area](https://reg.rainfocus.com/flow/cloudbees/devopsnyc/webinar3/page/landing), [Chicago](https://reg.rainfocus.com/flow/cloudbees/devopschicago/webinar3/page/landing), [Silicon Valley](https://reg.rainfocus.com/flow/cloudbees/devopssiliconv/webinar3/page/landing), [Singapore](https://reg.rainfocus.com/flow/cloudbees/devopssingapore/webinar3/page/landing), and [London](https://reg.rainfocus.com/flow/cloudbees/devopslondon/webinar3/page/landing).*

Thinking Like a Product Owner

Sarah says product people tend to stop at the business case and the features and forget reliability, stability and the rest of the user experience. Ownership has also gotten more siloed as systems have grown, so nobody sees the big picture. Matty recalls a product owner for the SRE team at PagerDuty and failing to find any posts or talks from that person. Sarah contrasts the two roles: supporting a SQL Server cluster meant caring about the health of the cluster and little else, while a product manager has to care about all of it, whether or not every piece is understood, because it all affects the end user.

Matty's line from the talk is that you won't put an NPS score on your Jenkins pipeline, "but kind of are you," since it's about users and feedback loops, and that's DevOps. Sarah suggests a health rating for every piece of the system, covering whether it does what's needed to support the people paying the bills. Sarah has mostly worked in B2B software, where users often have no choice, and says internal stakeholders should be treated the same way: don't build things so hard to understand or so locked down that colleagues can't make sense of them.

A PM on the DevOps Team

Sarah was the product manager for a DevOps team at a company whose product was a messaging gateway for an IoT platform, a role the company hadn't had before. The team, spread across Boston and Budapest, managed development environments, AWS infrastructure, federation access control, security and disaster recovery, and was about six people before Sarah made it seven. After watching for a month, Sarah saw repetitive work that could be productized and automated, and spent much of the time running interference between the team and people used to going straight to a favorite engineer for access. The team's customers were the software developers, so the job was finding their pain points and building a strategic roadmap, where ops work is usually "pipeline-style ticketing" of whatever is oldest or loudest. The team liked having a direction.

Roadmaps and Promises

Matty's talk has a slide that says "this is why roadmaps are bad, and if you have one, you should feel bad," since a roadmap can be read as a promise, and asks how to keep transparency without that. Sarah calls it the bane of every product manager's existence and handles it sneakily: only the roadmap shared with engineers is called a roadmap, and that's where dates live. Everything else goes out as vague documents with names like "H1 priorities." Sarah tells sales to say "new alerting integrations" and not whether it's PagerDuty or a webhook, and adds detail as a release date firms up.

Matty recalls Marty Cagan spending two days at Apartments.com, and the LivingSocial story of running an experiment, removing it, and having sales say it was already sold. Matty's point is that you can't be agile in only one part of the company, and ties it to Andrew Clay Shafer's observation that everyone wants more reliability, stability and velocity without changing anything.

Platforms as Products

Matty says platform engineering is "fundamentally providing Heroku inside your company," and that an internal platform team might think it needn't worry about customers because the CIO mandated it. But people who don't get what they need will go around it, which is "shadow IT all over again," and platforms tend to be thought of as compute and orchestration while event streaming, data pipelines and messaging get left out. Sarah says a SOC audit reveals the rogue platforms, and that you have to think about use cases: at a previous company a backend service didn't handle bulk requests, a front-end engineer looped over a couple thousand rows, and it fell over as soon as two people used it at once.

MVP Means Prototype

Matty says Marty Cagan reads MVP as minimum viable prototype, meant to help you learn, even though most people hear minimum viable product and treat it as version 1. Sarah says experiments are supposed to be fast and needn't scale, but then they ship and never get a version 2: "MVPs are not MVPs anymore." Matty compares it to critical production systems under someone's desk, including a machine in a Bank One data center that nobody could identify. Discovery, Matty adds, means asking questions and finding proxies, since people know what they want but don't always communicate it in a form you can build. Sarah says to repeat requirements back and to have conversations, because a PRD or ticket can't hold it all.

Product Managers as Incident Commanders

Matty says the two places product people shine are community conference tables, where they spend the day talking to users, and incident command. At PagerDuty a product owner asked to join the incident commander rotation, which had been engineering management. Told that a product owner wasn't an engineer, the answer was that "that's a feature, not a bug." The product owner became the first non-engineering incident commander, and by the time Matty left there were no engineers in the rotation. The reasoning: "never half-ass 2 jobs, whole-ass one job," so an incident commander who isn't a software engineer never gets pulled into fixing. The skills overlap too: prioritizing, communicating and delegating. Matty says it also changes how product folks think about reliability, because the concerns arrive firsthand and not filtered through on-call.

Matty tells of a sysadmin on the team who couldn't get the product owner to care about a service throwing around 25,000 false errors a minute, because the sysadmin led with the symptom, and the argument that landed was that nobody would notice a real failure. Putting the error rate on the office dashboard got it fixed in about two days. Sarah's version is selling disaster recovery to a general manager by describing the worst outage scenario in dollars, and Matty adds that "money is a lingua franca," not that it's all anyone understands.

Sidecar Skills and Learning

Matty calls business cases, writing and speaking "sidecar skills" that set engineers apart, recalling developers who dropped an idea when the CTO asked for a business case, and saying the exercise works like a rubber duck. Sarah says none of the interpersonal, interviewing, listening or public speaking skills came naturally and all were learned. Sarah also warns against underestimating colleagues' ability to learn technical details, from sales to product. Matty extends it to incident commanders, who need to understand a system well enough to know whom to call, without knowing how to rebuild the carburetor.

Sarah's start and stop: start thinking about who your stakeholders are, and stop building things without validating them.

• "Everything is a Product" - Matty’s talk
• “More Buzzwords Won’t Help” - Andrew Clay Shafer
• Telemetry Hub channel on YouTube
  
  *DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: [NYC area](https://reg.rainfocus.com/flow/cloudbees/devopsnyc/webinar3/page/landing), [Chicago](https://reg.rainfocus.com/flow/cloudbees/devopschicago/webinar3/page/landing), [Silicon Valley](https://reg.rainfocus.com/flow/cloudbees/devopssiliconv/webinar3/page/landing), [Singapore](https://reg.rainfocus.com/flow/cloudbees/devopssingapore/webinar3/page/landing), and [London](https://reg.rainfocus.com/flow/cloudbees/devopslondon/webinar3/page/landing).*

What the Sysdig Report Shows

Michael says the report is based on anonymized customer data, not a survey, so it covers a slice of the industry: organizations that have acknowledged a security problem and use tools like Sysdig. The number of vulnerabilities is alarming, which says something about the state of open source and the hygiene of components, and scanning usually reveals a worse picture than anyone expected because of nested and transitive dependencies. One result Michael double-checked was the share of non-human identities, which dropped from 88 percent to 58 percent of identities in customers' cloud environments, a shift Michael attributes partly to organizations staffing up after the pandemic.

On SBOMs, Matty notes that everyone at KubeCon in LA at the end of 2021 wanted to talk about them and the report suggests the industry is mostly still talking. Michael says there are two problems: the SBOM formats aren't settled, and an SBOM has to be dynamic, since a system drifts from its design over time, and has to account for partners and suppliers as well as your own code.

Build Dependencies, Runtime and Noise

Matty points to the report's finding that fewer than 1 percent of JavaScript packages are in use at runtime, and guesses it's because build tooling is written in JavaScript. The DevOpsDays site is a static site generator with a long package.json, none of which ships in the built artifact, yet Dependabot calls it "insecure as hell." Matty then catches a mistake in that reasoning live: the site does load Bootstrap on the front end, so there is front-end JavaScript after all.

Michael adds that a website tends to accumulate JavaScript libraries, then marketing adds tracking and payment processing adds more. A scanner will list every dependency and known vulnerability, but it can't say whether the code is reachable at runtime, which is where Sysdig's runtime insights and "in-use exposure" come in. Without that, organizations are "flying blind" and taking a best guess at what is exploitable. Michael recalls engineering teams suppressing findings in open source libraries they don't own, which gets described as false positives, and Matty calls it normalization of deviance. In cloud native, with microservices, containers and ephemeral resources, the dashboard can be "a sea of red."

What Shift Left Means

Matty says "nuance is hard" and the shorter the phrase the more nuance it needs, and recounts writing a talk about shifting left securely out of frustration with a customer's sysadmins, on the idea that a title doesn't imply infallibility. Michael describes the traditional waterfall model with security as questionnaires and compliance, and shift left as pushing security into early design and automating tests in the IDE, at commit, in CI/CD and at runtime. Each stage produces scan results, which creates a correlation problem, and many of the problems of waterfall come back, just earlier.

Matty's definition: "shifting left to me is not shifting the work to the people on the left. It is actually moving that domain expertise earlier in the conversation." That's why NoOps never happened, since ops turned out to be a domain of expertise, and expecting software engineers to absorb InfoSec is unfair and a bit insulting to security people. Michael says many of the calls at Gartner were about pushing scanning onto other teams because the security team couldn't scale, and notes that dynamic scanning tools are "glorified fuzzers" that need good test automation to reach the functions, and that release decisions and pass or fail builds are still unsolved. Matty adds that monitoring is "just testing with the time dimension," so a check in pre-production and in production should be in parity, and that the old hardening sprint produced a note from the security team saying it was okay, which bad guys on the internet don't care about.

Matty says the tooling has improved: security scanners once cost around $15,000 a seat, and code instrumentation was limited by cost, so tracing ran on one of 30 servers. Both agree that doing this right means changing how product release and sales think, and not just engineering.

Security as a Feature, Privacy and Zero Trust

Matty asks whether organizations treat security as a product feature, in the sense of a secure product and not AuthN. Michael says marketing still wants security features, and that more training material exists through groups like OWASP. Privacy has brought more focus, which Michael traces to GDPR, and so has the US National Cybersecurity Strategy and SEC disclosure mandates.

Michael builds zero trust from least privilege: zero trust is "least privilege on steroids," assuming the environment is compromised and authorizing continuously. It includes zero trust network access, which Michael says is what people usually think of, along with BeyondCorp, a cloudified VPN, and BeyondProd. Like shift left, people cling to one actionable piece. The report says "90% of granted permissions are not used," which Matty ties to onboarding: a new hire gets cloned from a colleague's access, the same way Chef-era teams copied an existing VM to get a new Apache server, until nobody knows what's on it. At Matty's current company, new hires get almost nothing and get annoyed, which works because the culture accepts 30 to 60 days of ramp up.

Confessions

Matty's Netflix password is a variation on the domain admin password from Apartments.com almost ten years earlier, and it was never changed when someone left, because in four or five years only one person who knew it left. "You can have bad password policies if people don't quit." Michael admits to simple passwords that a spouse can remember, since a 3-year-old leaves little time for a password manager. Matty closes with Windows NT 4.0, where passwords expired after 30 days with warnings at 15 and no minimum age, so the team built a Visual Basic app that changed the password 15 times and then back again. Michael will be at a Gartner Security and Risk Management Summit and on LinkedIn, and Matty ends with Gartner's "rogue sessions," where an analyst argues against Gartner's own position.

• Sysdig 2023 Cloud-Native Security and Usage Report
• Pushing Left With Tanya Janca (ADO epsiode)
• Shifting Left Securely (Matt's talk)
  
  *DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: [NYC area](https://reg.rainfocus.com/flow/cloudbees/devopsnyc/webinar3/page/landing), [Chicago](https://reg.rainfocus.com/flow/cloudbees/devopschicago/webinar3/page/landing), [Silicon Valley](https://reg.rainfocus.com/flow/cloudbees/devopssiliconv/webinar3/page/landing), [Singapore](https://reg.rainfocus.com/flow/cloudbees/devopssingapore/webinar3/page/landing), and [London](https://reg.rainfocus.com/flow/cloudbees/devopslondon/webinar3/page/landing).*

DevOps With Better Marketing

Pete's hottest take is "DevOps with better marketing." The immediate reaction was the unoriginal one: isn't that what you've been doing all along, building a platform for your team? Pete cites James Governor's tweet that "we've spent a decade rebuilding Heroku poorly." Matty traces the lineage back through PaaS, with Azure's original web and worker services model, Engine Yard and Heroku, all of which boil down to getting software running somewhere faster without worrying about the bits that run it. Matty also admits to owning and maintaining Pete Chess Bot, which lived only in Heroku, with the code never put on GitHub, until an unpaid Heroku bill got it shut down.

Matty recalls an agile transformation in 2010 at Apartments.com, where agile coaches said infrastructure was a service that gets consumed and so didn't need to be in the conversation, which is "why the DevOps movement had to happen." Matty pushed for sysadmins embedded in squads, but with too few of them, each ended up on three squads with no time for real work after the ceremonies. When something can't be dedicated to one feature team, Matty says, an abstraction layer is the answer, and that's the evolution into platform engineering in theory.

A Numbers Problem, and Nobody Owns the Product

Pete has built this kind of thing three times in a decade: a knife command to spin up servers with Chef on EC2, provisioning bare metal with Chef at a DNS company, and a framework at ThreatStack where developers filled in the application's shape and committed code. The problem is a numbers problem, with about 100 developers and 3 systems people, and self-service is how to stop being the bottleneck. The missing piece at most companies, Pete says, is that nobody acts as product manager for the ops team, whose customers are the developers.

Matty says that if you offer a platform, it's a product and you have to treat it as one, referencing a talk called Everything's a Product. A platform team may think adoption is guaranteed because the CIO chose the tool, but that attitude is what produced shadow IT. Matty also says a lot of platform engineering conversation is "incredibly Kubernetes-focused" and treats the runtime as the platform, while data and observability get left out, and then the platform team ends up integrating with 15 different ways to do Kafka. Pete says picking Kubernetes because everyone does is the same as "no one gets fired buying IBM."

Pete ties it to what Pete calls "the DevOps hangover": 10 to 15 years of growth where nobody minded how much Datadog or AWS got consumed, and now someone asks about the Datadog bill and the cost of running Kubernetes, and half the team has been laid off. "I guess we should have just had product managers the whole time." Matty's cynical read is that a product owner on a platform team would be one of the first people laid off. Pete notes that product management is the least defined role Pete has seen, from mini CEO at some companies to owner of one feature at others.

Rebranding and Silos

Matty's worry is "the wrong way but faster," the Simpsons line about the max power way, where platform engineering is the SRE team, which was the rebranded ops team, which was the sysadmin team. A coworker once said of 12 years at the same desk and five different companies that the job never changed. Matty adds that "silos are okay" as long as domain experts work together earlier, and describes arguing about shift left and security on Twitter. Pete jokes that if platform engineering pays 24 percent more than SRE, Pete will be a platform engineer. Matty says that's good for the individual, but asks whether a team still doing tickets and requests on Kubernetes instead of vSphere is really treating it like a platform.

Pete says the step still missing is requirements gathering, which is talking to the other people in the organization about what outcome they want before writing the first YAML: "It's always a people problem." Matty adds that Backstage is a tool for building the portal, not a platform, so you can't "rub some Backstage on it." Platforms are a socio-technical system, and buying one takes buy-in from the people who do the work and the executives who fund it, with the frozen middle in between. Matty says selling OpenShift at Red Hat was hard for that reason, at half a million to $2 million.

Pete's story is a DNS company where "platform V2" became a trigger word. Pete wrote a ten-page product requirements document, renamed the project Honey Badger, and talked to every team to distill the minimum: provision a server anywhere in the world with a default operating system and a Chef role. It worked only with cross-team buy-in and budget. Matty also plugs a DevOpsDays Chicago talk on organizational politics.

How Do You Say

Pete's side project started with a long-standing idea to record friends pronouncing tech words in a game show format, and a Slack teasing about how Pete says a load balancer's name, for which Pete made up an origin story. Pete now works at AppMap, a 10-person company that let Pete record the series, with weekly posts across YouTube Shorts, TikTok, Twitter and LinkedIn. The first iteration is supercuts of about 20 words, about 30 seconds each, from 31 interviewees. Pete's favorites were SQL, epoch, fsck, which has about 20 alternative pronunciations on Wikipedia, and JWT, where the docs say it's pronounced "jot" and nobody Pete recorded said it, until some had implemented it.

Matty says that's bad marketing, since nobody would search for jot. Pete says some pronunciations exist to help a listener type the word, like /etc and /lib, and that "all pronunciations are valid," since the goal was never to make fun of people. Pete wants more non-native English speakers for a season two, and a form for words and volunteers is linked below.

Matty's advice on names is to ask everyone how they want their name pronounced, even names you think you know, which is what DevOps Party Games did, and offers "which one do you prefer" as a better framing than asking which is right. The episode closes on words that still trip people up: Pete's words like through and throw while reading aloud to a child, and Matty's arugula, which Matty replaces with rocket. Pete's daughter, after a show says a word from a book they read together, looks over and says, "wow, you weren't even close on that one."

• Pete's Video Project
• Pete's TikTok
• Want to join a future version of Pete's videos?
• Platform Engineering With Daniel Bryant (ADO Episode)

Who Gets Cut

Matty describes hanging out a shingle as open to work about a year earlier and watching layoff waves roll through from the end of 2022, while the same companies kept hiring, which could be a left-hand, right-hand problem or "eventual consistency." Sidney says early layoffs hit functional roles like engineering, product and data science, and since March Sidney has seen many sales, tech talent acquisition and HR people on the market, which Sidney reads as a sign of overhiring and a course correction. Even experienced engineering managers are struggling, because the open roles are so specific that the ask is for a "purple unicorn with a T-Rex on its back shooting off machine guns."

Sidney was laid off in September of the previous year from a Series B fintech when interest rates went up, was picked up at a Series A startup after leaning on Twitter, and was hit by a second reduction after that. The common denominator across all of these is that "when someone loses their job, they lose their, they feel like their value is gone."

Not Taking It Personally

Matty calls this "Rule 7: don't take it personally," a phrase from a former therapist, and says it's nearly impossible because so much of identity gets tied to the place you work. Matty recalls bleeding "Chef orange" at Chef, feeling betrayed when the company did things Matty disliked, and needing two jobs to learn the distance. Now Matty loves the company, but "the company is not me," and colleagues are coworkers: "we're not a family."

On layoff decisions, Matty says the best performers sometimes get let go, since the best performer may be the most expensive, and describes an organization where people couldn't volunteer to be laid off in place of a colleague because the decision was about a function and not about performance. Matty adds that managers who had no part in the decision are the ones who have to make the calls, and that CEOs crying on YouTube about it are tone deaf.

Sidney's advice for the mental side is to "give yourself space to process" before firing off resumes that aren't ready, to build a support system, and to find a cheerleader, because people in fight or flight end up on a hamster wheel. Sidney adds that a therapist can help. Sidney also notes that even with 24 years and a history of startups that were bought for billions, leaving still felt awful because the identity was wrapped up in "I built that."

The Cost of Settling

Matty describes the early 2000s, when a company acquired by GE laid the team off, the job search dragged on, and Matty took a cut of around 30 percent that took years to recover, since the next employer asks what you make now. Matty now takes recruiter calls for the information: asking for ranges, seeing what people are wanting to pay for a role, and watching people who need the work accept it. Matty suspects some companies see a chance to reset pay and get people cheaper. Sidney passes on advice from a woman CMO that it's okay to take a step back in uncertain times and cover what you need to, and that you can always keep looking while working.

Pay Transparency

Sidney says employers in California, Colorado, Connecticut, Maryland, Nevada, New York, Rhode Island and Washington are not allowed to ask what you made before, and that roles posted as remote that could be hired in those states tend to disclose a range. Matty says some ranges are games, like a posting spanning over $200,000 where nobody's paying the top number. Sidney says recruiters ask about threshold or expectation, which has nothing to do with last year's W-2.

Matty walks through Talk Pay: Lauren Voswinkel started the #TalkPay hashtag in 2015, and Paul proposed an in-person version as an open space at DevOpsDays Austin that year. Matty has helped run them at many DevOpsDays and did two at Austin in 2023, anonymous but with people free to volunteer their number. Matty says people fear that their number is too low or too high, and that in practice nobody thinks the lowest number is the person's fault: "They think your employer is bad." The links below cover Paul's write-up and the Corey Quinn and Sonia Gupta talk on salary negotiation.

Asking for the Range

Sidney's tactic is to ask what the level is for the role and what salary corresponds to that level, instead of naming a number, because "the first person to name a number loses." If the answer is higher than expected, keep a poker face and celebrate privately. Hiring managers should not hand out different salaries to people doing the same job at the same level, which Sidney calls out where it happens.

Sidney tells of a woman engineer whose offer came in $40,000 below what a same-level person in Portland, Oregon was getting. Sidney pushed to match the salary, and the engineer cried and said thank you. "It's not rocket science. It's influence."

Matty adds that most of the audience can't yell at talent acquisition, so the lever is transparency from peers. Matty describes telling a friend exactly what the offer was, and a case where a friend applied to the team without asking. Matty's advice to people who aren't cis white men is to find someone who looks like Matty to coach them on having audacity: before pushing back on a wrong title in an offer letter, Matty asked a friend to "remind me to be a white guy in tech." The recruiter's reply was that it was a mistake and the DocuSign needed regenerating. And if an offer is withdrawn because you asked for more, Matty says, "good."

Start and Stop

Sidney's last advice is to stop telling yourself you aren't worthy of the next opportunity, to process what happened through a support system, to ask for the level and the band, and to be as confident as possible. "All they can say is no. You got a 50/50 shot." Matty says a follow-up episode is likely.

• Sonia Gupta and Corey Quinn - Embarrassingly Large Numbers: Salary Negotiation for Human
• Talking pay in the public square

Data Is a Journey

Matty asks how the field has changed from the days when the DBA was the person in the closet. Francesco picked data because the work sits between computers and people: analysts, scientists and engineers translate business expectations into models and documents. "Data is data, but how you think about it, how you make sense of the data, changes every time you interact with different people."

Francesco frames data as a journey, using a company that sells shoes online. A purchase lands as a record in a transactional database. A startup then runs analytics queries against that same database to compare today's sales with yesterday's, which works at small scale. As the company grows, those queries make the transactional database suffer, so the data gets moved into an analytical database, and one piece of technology becomes a chain of them. Francesco notes the chain follows the growth of the company, including change data capture, event-driven architecture and analytical databases.

Batch, Streaming and Change Data Capture

Francesco explains the old batch approach: wait for the night when the website is offline, extract everything from the transactional database, and load it into the analytical database to feed the data marts and warehouses. That still suits reporting on last month's data. It doesn't suit cases that need an immediate reaction, such as reordering inventory when ten pairs of shoes sell. There the work moves from batch to real-time or near real-time, per event. With Kafka, a streaming technology, a change in the database can be propagated to downstream systems, for example an application that compares current stock against a data scientist's prediction.

Change data capture tracks changes in the database by reading its logs, without continuously querying it. Francesco says it lets a company evolve from batch to event-driven without touching the transactional database or the application in front of it, so the business keeps running while the backend changes.

Losing the Map

Matty asks how large organizations keep their data interactions aligned. Francesco describes enterprises where you don't know about a data mart sitting there, or who purchased a tool, because "people come and go and company remains and data remains and, you know, bills remain." With non-technical users building analytics through point and click, there is no longer a single view of reality. Francesco admits losing a ten-year battle against people exporting transactional data into Excel, and says what matters is a global view of which data assets and pipelines exist and how they connect. "If you don't have the map, you are lost."

Without the map, a company can pay a consulting firm six months later to solve an inventory problem it already solved, or let two teams solve the same problem and end up with different answers because of a small detail in a KPI definition. Francesco adds the financial and security side: GDPR aside, loose control over where data lands invites someone to dump a database export into a bucket by mistake.

Matty asks whether anyone does this well. Francesco has seen two approaches work. One is documentation, which is risky because it's an afterthought, though Francesco wonders if ChatGPT could help with parsing it. The other is a single tool for all transformations, such as Informatica, which can give column-level data lineage, but data now spans many technologies and teams. Matty says relying on documentation and training is the worst way and prefers guardrails that make the right way the easy way, and notes that Kubernetes is great until you want state.

Francesco sees a way forward through metadata. Within one database like Postgres, catalog views list tables, users and permissions. Between systems, the gap can be closed by parsing the configuration that connects them, for example the JSON payload of Kafka Connect, which says where data comes from and where it goes. Automated tooling can describe how a pipeline was built but never why, so the reasoning behind a KPI, such as using the last six months of sales instead of three, still needs documentation.

Four Directions for a Robust Data Platform

Francesco's framework has four directions, and the episode's existing links include the blog post on it.

• Scalability. It covers technical scale, such as going from one node to a cluster of 50, and whether the technology can segment all the business cases. It also covers whether the humans can scale, since a technology with no talent pool becomes a problem in five years, and financial scale, since a perfect solution that costs twice what each user brings in is not affordable. Francesco adds that data platforms are sticky and few backend migrations succeeded.
• Observability. Metrics, alerting and notifications matter because the data team is offering a service within the company. A bird's-eye view also helps when GDPR queries arrive, since you know who to ask. Francesco asks why versioning shouldn't apply to data too, with a way to replay yesterday's data after a mistake.
• Fast. Speed covers time to develop, time to deliver and time to recover from errors. A dashboard built in a day is not worth it if you always wait 23 hours for the right data.
• Trustworthy. With one analyst, that person's Excel file is the source of truth, but with 55 analysts there is no number anymore. The company needs a single golden truth, delivered on time, because a KPI that is right at 9 AM one day and 1 PM the next erodes trust. Data also needs to be secure, so nobody who shouldn't see or alter it can.

Francesco says some of these will bite immediately and some in the long term, and looking at all four lets you compare technologies and make a 360 degree decision.

Replaying Data With Kafka

Matty finds versioning and replay hard to imagine, since even restoring batch data to a point in time was nearly impossible. Francesco describes Kafka as a log: events are written one after the other, and reading doesn't delete them, so another application can read the same messages. Add schemas, and Kafka refuses messages in the wrong format. Schema evolution lets you change the schema while existing consumers can still parse messages. In the example, a boolean for putting initials on the shoes is added, billing ignores it and the printing team uses it. If the flag later needs to be a color, the log can be kept for days or years and a new set of consumers can be fed the data generated since yesterday, five hours ago or an hour ago.

Misconceptions

Asked about what developers and infrastructure folks get wrong about data, Francesco says the biggest misconception is that data systems are old and boring. The data world is hot, and ChatGPT and AI are data. Innovation with data depends on quality data, which is the biggest problem to solve everywhere, and "Data pipelines are cool."

Francesco is on Twitter as @ftiziot, which is consistent almost everywhere apart from LinkedIn, and the DMs are mostly open. Francesco also says the main mission in life is telling people how to properly eat Italian dishes, which for example excludes carbonara with cream, pineapple on pizza and cappuccino with lunch or dinner.

• Kafka Connect tool
• Metadata parser
• Francesco’s SOFT blog post

What Platform Engineering Is

Daniel's formal definition is the discipline of building toolchains, workflows and platforms to support the team going from idea to observable business value in production. Continuous delivery doesn't end when the app reaches production, since feedback from a business or operational point of view has to come back. Matty says that means Kubernetes alone isn't a platform, and cites James Governor's line that everyone is trying to build their own Heroku.

Is DevOps Dead?

Matty asks about "DevOps is dead" marketing at KubeCon. Daniel, with 20 years in Java where "Java is dead" recurs, says a shock headline draws attention, and that engineers want to bury the previous generation to look like they're innovating, while history rhymes. Matty says the question is what you mean by DevOps: if a DevOps team is an automation team that builds infrastructure, platform engineering replaces that, but the principles have been the same throughout. Matty adds that DevOps takes research seriously, and the research behind some of that marketing was talking to three companies.

Platforms Versus Portals

Daniel says some people say IDP for internal developer platform and others for internal developer portal, and the difference matters: Backstage is a great jumping-off point, but Daniel sees it as the UI, CLI, SDK and API on top of a platform. So Daniel asks whether people mean a platform soup to nuts or just a service catalog. Matty likes the idea of a service catalog that catches the 80 percent case and lets you do other things at a cost, and raises Charity Majors's maxim that the best tool is the one you don't need and the second best is a SaaS, asking whether there's any SaaS for this, and whether Heroku was the closest.

Daniel says Heroku and Cloud Foundry were built for web monoliths, but now the 80 percent includes machine learning apps, front ends and systems of record. Spotify talks about golden paths, such as one for machine learning apps, one for microservices and one for front ends, so there's no longer one true way and a one-size-fits-all platform hits at most half the apps. Matty says that makes it a big ask to build an internal SaaS, treated as a product.

Thinnest Viable Platform

Daniel borrows a phrase from Team Topologies, the thinnest viable platform. Large organizations like Intuit, which has a global team supporting its platform, can justify it, but for a startup of three people without product-market fit, "please don't build a platform," and something like Cloud Run or Knative with GitHub Actions is enough. Matty says the tech is the easy part, the hard part is how people communicate, and cites a line that source control is a communication tool for developers. Matty also cites Adam Jacob's point that tools influence culture and culture influences tools, a variant of Conway's Law. Matty says large platforms like OpenShift or Tanzu are hard to start small with and get decided at the CIO level, and Daniel says the cognitive load is high, while bottom-up requirements plus mid-management buy-in and then exec support works better than the old golf-course selling.

Matty says to begin with the end in mind, without solving everything, but avoiding decisions that make it impossible later. Daniel says this is the role of a platform product owner, who starts small and thinks big.

Who Builds It and Where It Sits on the Curve

Daniel says platform teams are often drawn from infrastructure or developer experience and enablement teams, and often a pain triggers them, such as an exec discovering 10 versions of a platform and nobody who knows how to maintain version 9. Matty says this is the early adopter part of crossing the chasm, and that typical enterprises haven't arrived, recalling arguing with PagerDuty's founder that enterprises still have production support teams and calling it survivor bias. Matty worries the late majority skips the pre-work and just renames teams: tech ops became cloud ops, then DevOps, then SRE, then platform, with the same remit, and says "get your bag" to anyone getting a better title.

Backstage

Daniel describes Backstage as a jumping-off point for an internal developer portal, with a service catalog, automation hooks, search, who's on call and ownership, and TechDocs for living documentation. Spotify sells extensions, and Roadie offers Backstage as a service since it's hard to install, and many people use it as a facade or for inspiration, for templates that spin up a new service with observability and security baked in and links to dashboards.

Do and Don't

Daniel's three keys are treating the platform as a product, since you can't have good developer experience without good user experience, focusing on workflows and tool interoperability, and making it composable. The thing not to do is buy a platform. Matty adds "you can't buy DevOps, but I can sell it to you." Daniel says late adopters worry about being left behind and throw money at it, while it's better to read the many blog posts from companies that have built platforms and spend time understanding the problem space first.

• ADO Episode - Platforms with Kelsey Hightower and Andrew Clay Shafer
• Daniel’s Kubecon talk
• Daniel’s blog/tweet thread the kicked off his interest in Platform Eng
• Spotify golden paths
• Crossing the Chasm - Technology Adoption Lifecycle
• Backstage project
• Backstage as a service

What the VOID Is

Courtney started collecting incident reports while doing product research on Kubernetes and Kafka, looking for non-marketing information on how they fail in the wild. Having gathered about 1,000, people said thank you and "we have more of that," and the database and its metadata emerged in late 2020, with a first report in 2021 and a second more recently. The database takes a broad view of incidents: postmortems, status page updates, tweets and media articles, anything where someone talked about a website or service falling over. It also collects metadata such as how long the organization said the incident lasted, severity and methodology. Courtney wants data, not abstraction, to address myths about software incidents.

The VOID focuses on availability incidents since security breach databases already exist. Courtney says the DevOps mentality of sharing failure is a long way off in security. The analogy Courtney uses is the US airline industry in the 1990s, where the push to share incidents came from pilots, not regulation, and they set competitive concerns aside. Courtney and Casey both worry about regulation of software, with Courtney citing the call to regulate software after the Southwest Airlines meltdown, which was an organizational and cultural problem, and Casey calling it a personal nightmare. Casey says the VOID can help redefine the value of availability work, be it incident analysis, response or chaos engineering, and that availability and security are two sides of the same coin for system safety.

Why MTTR Is Junk

Courtney explains that MTTR comes from physical manufacturing, where widgets wear in predictable ways with a normal distribution of repair times. Software incidents don't look like that: the duration data are heavily skewed, with a big bump under an hour or two and a long tail, so "you can't take averages of that. It's just garbage." An engineer from Google wrote an O'Reilly report that ran Monte Carlo simulations on incident durations, shortening some and taking averages, and the results were a mess. The VOID team did the same with its own data and got the same results. Courtney says people react either with "oh shit" or "oh shit, but I'm going to fight you on it." Casey adds layers: the statistics are wrong, the data coming in is garbage because methods for determining how long an incident lasts are fraught, and you'd need more data points than Google has.

Matty says the metric is used to report on the performance of people within a quarter and is about the closest thing to putting a Nagios counter on your people, and recalls incident command workshops where people said they spent their time on mean time to innocence. Courtney asks what decision you would make based on it, since either way you'd have to go look at what's happening in the system and the people operating it. Courtney says the industry's data-driven obsession overlooks data about human behavior, which is still data.

Bureaucracy and Taylorism

Casey says organizational researchers call software the bureaucratic profession, bought wholesale from manufacturing and scientific management. Every role, such as people manager, project manager and architect, takes responsibility for expertise away from engineers, which makes sense on an assembly line and is the wrong model for knowledge work. The business must change if it wants availability, resilience and security. Courtney's line: "Taylorism is a corporate disease that we haven't developed a vaccine for yet." Matty recalls the Upton Sinclair quote about salary and understanding, and the frozen middle, where people's jobs exist because of metrics like MTTR.

Courtney says reliability and security are now central to business concerns, as Southwest showed. Casey describes how every Netflix engineer knew how to look up stream starts per second, one metric correlated with value, and Netflix found that going from four nines to five was pointless because users' Wi-Fi and ISPs wash it out, so they invested in regional failover. Courtney says to find the core metric according to your business, and Matty notes public sector people know their mission better than private sector people know how their company makes money.

What to Use Instead

Courtney says what replaces MTTR is the recognition that no single metric captures reliability, and that software systems are sociotechnical, so you need people skilled in social science methods: interviewing, collecting stories, building narratives and finding patterns. Organizations are hiring incident analysts, and Courtney believes the ones who invest will gain a competitive advantage, though the data for that is further off. Courtney's favorite example is an organization in the CIO's office at IBM, described in a DevOps Enterprise Summit talk, where a skunkworks team did quality incident analysis, a bad incident gave the opening to try something different, and now monthly learning-from-incident reviews draw hundreds of people. Courtney also notes that the Microsoft Azure team changed its public reports from RCAs to post-incident reviews and the quality and depth improved dramatically.

Matty tells how a PagerDuty SRE quietly changed the word "root cause" in the postmortem template without asking permission, nobody objected, and suggests the homework for PagerDuty users: change the template to contributing factors. Casey "fully supports small acts of vandalism." Courtney's last finding is that the VOID data shows zero relationship between an incident's duration and its severity, and Matty and Courtney note severity is negotiable and changes during an incident. Casey says the whole model of severity is wrong at scale, since some group of users is always unable to reach your system.

• VOID
• The Verica Open Incident Database (VOID) makes public software-related incident reports available to everyone, increasing understanding of software-based failures in order to make the internet a more resilient and safe place. After scrutinizing nearly 10,000 incidents, one thing is crystal clear: Resilience saves time. Taking the time to understand how to better respond when something green turns red—learning from the people, the processes, and the systems—will make your next incident smoother.
• 2022 VOID Report
• “Taylorism is a corporate disease that we haven’t developed a vaccine for yet” - Courtney

Optimizing for Speed Only

Roni says development processes try to optimize for speed of deployment, cadence and time to release. If you only optimize for speed, "you're just creating a system where you're hurling features over the fence faster," 24 times a day instead of once a month, without improving the learning or the feedback. Jessica points out DevOps is supposed to keep caring after production, and Roni says the tools for that look for problems, which is reactive and generic. As a product manager Roni had tools like Google Analytics showing the impact of a decision, such as whether a navigation change increased adds to cart, and didn't feel like they were running blind, while developers had CI, CD and testing tools that say nothing about impact or performance in production. Jessica calls the missing piece observability, and adds it must be more than monitoring.

The Inverse Pipeline

Roni describes continuous feedback as the inverse of continuous deployment: the DevOps loop takes code from source to production, while continuous feedback starts with information from production, goes through stages to work out what's relevant, and ends back in the developer's tools, including the IDE. Roni stresses it's not actually linear, since feedback also exists before you start coding. Code ownership has grown from "done when I sent it to QA" to owning tests and deployment, which Jessica compares to moving from owning a car to parenting, where you want to know how your kids did in kindergarten.

Roni's examples of what code could tell you: how heavily the code is used and whether it's a bottleneck in a high-concurrency environment, which tells you what to optimize, whether it runs in production at all, which Roni has seen surprise teams who invested three years in a feature whose code path was never reached, and how it scales with concurrency, database size or payload. It could also report runtime errors, such as whether a "should never happen" branch does. Roni adds that developers misuse logging for this and forget to check.

Biases and Why It Doesn't Happen

Roni says continuous feedback makes the organization a learning process instead of a shipping process, and stretches the definition of done. Without it you accumulate technical debt and end up "running around, putting out fires." Roni cites biases: estimation anchoring and optimism bias, and confirmation bias in tests, which codify expectations and miss things nobody thought of. Observability injects relatively objective data, and "if you know about a bias, it seldom helps you actually overcome it." Jessica adds that combinations of features, data and ordering go well beyond edge cases.

Roni says very few engineering organizations actually practice continuous feedback, for three reasons: engineers are busy and can't keep looking for trouble in logs and dashboards, not all have the expertise, and they get data, not insights, and context switching is costly. Roni says an insight would be that this is a bottleneck and why, with a way to double-click for more.

Digma and OpenTelemetry

Roni started Digma, which is open source and entering beta, to tackle those three issues by codifying the tribal knowledge about how to measure latency and read time series, bringing it into the IDE so there's no context switch, and making it proactive, so the code sends "life signs" after it ships. Roni also wants to celebrate wins and not make observability all about blame. Readers can sign up at digma.ai, and mentioning the podcast gets a bump up the beta list.

Roni says OpenTelemetry was pivotal because everyone agrees on it, with vendors aligning around it and a spec that lets new open source tools make the data more useful. The libraries auto-instrument code, so getting from no telemetry to useful data is quick. Digma works as a pipeline and not an APM, ingesting OpenTelemetry data, scanning the code to correlate data to locations, and, if you add the commit ID via an environment variable in CI, relating insights to the code change that precipitated them, which Roni describes as a matryoshka design. Roni would like shorter loops, so adding a trace gives feedback in testing, CI and staging quickly. Jessica relays a friend's wish for something that says there's an N+1 query right here, and Roni says that's exactly Digma's point.

Roni says a developer wants control, not a 2 AM call three days after a push, and had written a blog post called Breaking the Fourth Wall, about code that talks back. The biggest risk is being spammy or sending people on a wild goose chase, so the feedback has to be accurate and pertinent. Jessica says to test in production as well as before it.

Roni is @DoppleWare on Twitter and writes on Medium, and the favorite board game Roni mentions is New Angeles, though Roni rarely plays the same game twice, since repeat plays become rule hacking.

Jess and Roni talk about what continous feedback: where it came from, what it looks like in the context of a dev proces, and the benefits it can bring to engineers and developers. They also discuss Roni's observability project, Digma.ai... and his other passion, complicated board games.

Clarify Assumptions

Dagna says everyone comes to a conversation with assumptions from family and culture, and clarifying them is a key communication skill. There are two kinds: the ones you hold yourself, such as how many people and how much time a feature has, which you can state, and the ones you make about what others assume, where it's better to ask and overcommunicate, because "there are no stupid questions." Dagna recommends starting with an "I sentence" to create a safe space: "I didn't get it. Why don't you give it another go?" so the other person doesn't get defensive. Matty says that when you keep saying people just don't get it, that's on you, and Dagna says you can't control other people's actions but have an impact on how they behave, and if you think you work with idiots, people sense it.

Empathy as Understanding Priorities

Matty asks how to operationalize empathy, recalling the 2014 devopsdays talks that were all about it and a talk title, "You've Convinced Me We Have to Collaborate Now. How the Hell Do I Deal With People?" Dagna says empathy is misunderstood as having to feel what others feel. What works in the workplace is understanding other people's priorities: an engineer wants to fix bugs, implement features, learn and have fun, while a project manager cares about business value delivered on time with no disruptions and isn't moved by refactoring without a tangible change. Dagna gives clients a worksheet to map stakeholders' priorities, and says pitching an idea by showing how it fits their priorities is "how brilliant engineering ideas get actually implemented." Matty says it's like a talk called The 5 Love Languages of DevOps: a change that makes sense to me might land differently for you, like 30 milliseconds faster matters because of an SLA.

Assertiveness

Dagna's own stuck point was assertive communication. Dagna is Polish and direct, and recalls a company-wide meeting of around 300 people after layoffs where Dagna tried to raise concerns with leadership with good intentions, and the boss asked why Dagna had called the entire leadership team idiots, which Dagna says was not what was said but how it came across. The lesson is that intent and how you come across can differ completely. Assertive communication means talking about your needs and wants respectfully so others feel heard. Dagna says probably every senior engineer sounds like an asshole now and then by being too direct, and that humans aren't rational beings who only look at facts.

Step Outside the Forest

Matty says systems are complex and have ramifications beyond your microservice. Dagna says engineers are so deep in the woods that they see one tree, while a manager sees the health of the project, a director sees groups of trees and the C-level sees the whole forest, and stepping outside the forest is critical for career advancement: "what got you to that senior position is not going to get you past that position." Once your technical foundation is solid, soft skills matter more. Matty points to earlier episodes on principal engineers and says even engineers who became engineers to avoid people deal with people.

Limiting Beliefs and Marketing Yourself

Dagna says the way you think impacts how you act, and compares the subconscious to background programs running on an operating system: people around you growing up "install beliefs" with a sudo command. A common engineer belief is that their work speaks for itself, and it doesn't, since nobody sees it unless they pull the repo. Dagna says marketing your work is a communication skill, and changing a belief is possible because people keep changing. Matty adds that ops roles have the same visibility problem as a corporate lawyer, and recommends knowing how your company makes money. Dagna tells of two features: a refactor that cut a mobile build from about 4 minutes to 20 seconds, interesting but affecting two engineers, and a boring build for a huge client that got praise from the boss's boss, the sales rep and the client. A conversation with the manager showed where the impact was, so ask how features affect the business. Dagna adds that if you can't ask your manager that, run. Matty recalls not knowing for years what the bank's treasury services line of business did until reading its intranet page.

Values and Interviews

Dagna's first coaching step is figuring out a client's values and the last is finding companies that share them, so you aren't reacting to job postings. Matty notes that interviews are full duplex: the candidate is also evaluating the company, and interviewers are usually untrained. Dagna says candidates who ask questions back are people Dagna wants to hire, and that confidence in your skills lets you negotiate, like the six-month sabbatical Dagna negotiated four months into a startup job, which was spent traveling across Southeast Asia.

Dagna's one recommendation is to ask more questions and put assumptions on the table. Dagna's coaching site is themindfuldev.com, with a case study video, and the two agree to do a follow-up on confidence.

• linkedin.com/in/dagnabieda
• www.themindfuldev.com
• How Dagna works with her clients
• https://www.arresteddevops.com/principal-engineer/

The Great Resignation

Matty notes that last time they hoped the pandemic was nearly over, and that devopsdays Chicago skipped 2021, the first year without one since it began. Matty hoped organizations wouldn't go back to the old way, and sees some insisting on it, with people not putting up with it. Tim says they'd both predicted that the second people felt comfortable leaving jobs where they were mistreated they would, and they have: people quit and figure it out later, or go to employers who treated people well, and the leverage is with employees now.

Tim says most companies "survived" and didn't thrive, because culture didn't change for remote work. Managers demanded Slack presence and cameras on, and couldn't manage people instead of overseeing them. Burnout and bad management are the leading drivers of quitting, and "retention problems" are a lagging indicator. If you can't staff positions, Tim says, "that is a you problem that you created." Matty adds that culture is a reflection of the behaviors of the people there, and that many leaders have not really tried remote work, since people working from dining tables during a pandemic isn't remote work. Tim says bad management "knows no industry" and may not be malicious, since a great in-person manager can become a poor one when circumstances change. Tim also says companies claiming you can't do culture from a bedroom can change the culture, because technology drives cultural change.

Who's Pushing People Back

Tim says the pandemic stripped away distractions, leaving "just you and your shitty job and your shitty manager," and that remote hiring let people apply anywhere. Matty asks about traditional enterprises like banks, insurance and manufacturing. Tim calls the Bay Area's mystique dubious and says the pandemic showed talent is everywhere, so influence can decentralize, though money remains concentrated there. Tim says older, risk-averse industries and managers whose careers were built on overseeing people directly are the ones pushing people back, along with real estate commitments, and that pushing people back is how you tell a tech company from a company that uses tech. Matty says most tech workers are at traditional companies, and that people on this podcast forget how small a sliver of the industry works in newer ways. Tim distinguishes tech workers from the tech industry, such as Home Depot's retail versus e-commerce sides, and says the disparity drives people away. Matty calls this bimodal IT, which Matty doesn't think is a good way to run an organization.

Why People Leave

Matty describes public-sector leaders who accept that someone might stay only a couple of years, and says a great manager says "you've outgrown this team." Tim, who worked in government contracting, says people in some public-sector jobs are incentivized to stagnate technologically, like protecting a platform you've certified in for decades. Tim says people used to leave for raises, and now they leave to not hate their job, and the only reasons to stay are the money, the job and the manager. Matty shares stories of old systems nobody knows the purpose of and is afraid to turn off, and Tim says people are still patching COBOL. Matty says whatever you're running, you're doing good work.

Making Conferences Inclusive

Tim is glad KubeCon North America 2022 is in Detroit and asks conference organizers to stop picking expensive places like the Bay Area, New York, Napa and Lake Tahoe. Tim wonders why events don't use spaces at historically Black colleges, such as those near the devopsdays Austin venue or in Atlanta or DC, and says virtual conferences let Tim talk to more people than ever, and "I don't want to give that up." Matty agrees it should be both, not just hybrid, since All Day DevOps has always been virtual, and some people prefer virtual for accessibility reasons. Matty says diversity of organizers matters because they may not know about those facilities, and venues are often repeated because "we've done it there before." It's too late for devopsdays Chicago 2022 because a contract is signed, but Matty commits to keep working on it.

Tim says ticket prices could be lower and sponsors could pay for plane tickets, hotel rooms or childcare facilities, and organizers should talk to neurodivergent people, parents and people who can't get days off. Matty says many devopsdays could be run fully sponsor-funded, and that free tickets should not require people to justify why they need one: a checkbox saying "my employer doesn't pay for this" is enough. Tim says having done it, being asked to justify is demeaning. Tim says the industry is more inclusive than it's ever been, which has allowed harder conversations about technology and how workers are treated, but "if you look out in that audience and it doesn't look like something you would consider inclusive, then you have more work to do."

Tim is @ElCheffe on Twitter and at the Duckbill Group.

Breaking Down Gates with Tim Banks (previous episode with Tim)

Hidden Work and Paper Security

Sue says that if software is eating the world, hackers are having a feast, and there aren't enough security professionals. DevOps teams already do a lot of security work that's hidden. Sue thinks security should be everyone's job but people don't know how, aren't incentivized with shared metrics, and often "black out" at the topic because the stakes are high. Matty cites a tweet, which turned out to be from someone the transcript renders as Cat Sweet, asking why security incidents don't get the same blameless "time I took down production" stories, with the reply that they cost money, to which Matty answers that tech incidents do too.

Dominic says both aim to make infrastructure run as intended: a service that's secure but down isn't useful, and one that's running while giving out credit card numbers like Halloween candy isn't either. Not every security finding will be fixed or needs to be. Sue calls the security team's mandate "paper security," as opposed to real security, which is what DevOps cares about and is hard to determine. Matty says organizations often claim a regulation requires something when it's how they implemented the control, and uses a Simpsons image of layers of physical security ending at a broken screen door.

A Story From Deutsche Telekom

Dominic recalls that about ten years earlier, at Deutsche Telekom, when cloud and DevOps were new, an auditor arrived with paper security manuals that didn't fit how they ran infrastructure. The team rejected the binders, distrusted security and ended up worse off. It resolved when a new pen tester and auditing team came in with fresh eyes and said some docs applied and others didn't, and together they wrote new requirements and put them into the automation pipeline. Dominic adds that ransomware attacks leverage the same automation that DevOps preaches.

Empathy and the CISO

Matty says policies are organizational scar tissue, and that almost nobody blocks you because they're a dick. Sue says security conferences lack empathy and team-building workshops, but that's shifting as security diversifies. Matty says classic security is adversarial, which shapes the whole outlook, like Matty's own old-school ops view of developers as the enemy. Sue says CISOs often feel peers dislike them, since CTOs can relate work to business value while security talks about risk, so the change needs to start at the top. Matty says ops is like being the corporate lawyer, known only for failures, and Sue wants to invite security to devopsdays.

Dominic says both sides have tried crossing over, but each lacks the other's context. Security asks why ops can't just auto-remediate, and ops worries about what that does to infrastructure. DevOps has built the muscle for safe, reproducible change with development teams, and it's time to extend it to security: don't hand over a list of 300 findings, give 10 critical ones to tackle together. Sue says that needs negotiation skills and a decision-making framework. Matty says shift left without nuance sounds like developers doing all the security, like NoOps, which failed because domain expertise matters, and Dominic adds it's subject matter expertise around one table.

Practical Advice

Dominic suggests picking one topic, like SSL/TLS, and discussing it with the security team, and not taking all 300 results. Sue says people in security have a hard time, like a SOC 2 checkbox about cameras at the entrance for a fully remote company, and DevOps people's answer is always "it depends." Dominic describes two trust-breaking stories: security freezing an environment for three weeks for an audit, and the DevOps team changing the environment the Monday after certification so nobody knew what happened. You can't change the other side but can change yourself.

Sue would love success stories to match DevOps metrics. Matty agrees and says that talk-worthy failures can be told without revealing attack details. Sue raises Equifax, and Dominic says there's technical security and legal security, and to find the security person who will have the technical conversation.

Supply Chain

Dominic describes the supply chain as everything that goes into building software: dependencies and infrastructure components. Shift left applied to supply chain with the same 300 findings means people react only to critical issues, and then findings get relabeled as critical. Tools are full of false positives, and "nobody's going to give me those 15 minutes of my life back." Dominic says progress comes in three steps: visibility, prioritization, then fixing, and thinks auto-remediation vendors are getting ahead of themselves. Matty notes that requiring CIO approval for every open source component isn't protection.

What to Do Tomorrow

Dominic's two suggestions: build a relationship with a security person, even by talking about anime or Star Wars, and get visibility into your infrastructure, since companies have a bigger visibility problem than they admit. Sue's are to make time to build relationships in a remote world, and to align on goals, since one DevOps team assumed security should own policies and tools while security didn't know.

What Changed in Year Two

Austin says the idea was the same: a virtual event in a deliberately constrained virtual space, with the constraints meant to inspire creativity. Austin announced it way ahead of time so fear of failure couldn't talk them out of it. This year added more production value, artwork and merch, and turned last year's charity shoutouts into a fundraiser for The Trevor Project, with a ticker during the event. Austin says it raised over $6,000, though the turnout wasn't the hit Austin expected. Angelina says the ticker made the event mean something more, and Austin says the commitments of no entrance fee, free on Twitch and accessible with transcription are founding concepts, and that getting all those people together means you can probably ask them to give.

Katy says as emcee there's no off mode, and introducing people talking about what they're passionate about brings joy. Katy ran a dedications line in a Discord channel, reading people's shout-outs aloud in a radio voice, since Katy once wanted to be a radio DJ. Austin also says introductions use fun facts in place of bios, which was accidental last year and is now deliberate, since a first-time speaker's short bio next to someone with a page of accomplishments invites implicit bias. Matty says devopsdays Chicago stopped having MCs do intros for a similar reason, since MCs know some speakers and not others.

The Talks

Ana, a first-time speaker, only started Animal Crossing around November or December of 2020, and had trouble with the abstract. Ana zoomed out from reliability to interpersonal things, like checking on neighbors and building trust and safety in teams, and tied it to reliability with the example of buying two axes because your tool will break. Rin's talk was on community and resilience, arguing resilience means having the tools to handle pain in a way that suits you and has to be worked at, like Roald, a villager who loves working out. Rin liked that the event was on Twitch in Animal Crossing and not just Zoom.

Angelina spoke about Apache Kafka, themed with Animal Crossing and a dinner party, partly to make it less intimidating for the computational biologists at a monthly meeting, and partly because burnout had made Angelina not want to give talks, so a video game theme was a way to do it differently. Based on feedback, Angelina will keep using it. Arri gave a talk on getting started with accessibility advocacy, since people often don't know how to begin, and liked that the conference mixes soft skills and technical talks. Matty and Arri compare single-track and multi-track conferences. Jack, a first-time public speaker who submitted a throwaway proposal in two minutes, spent two weeks on drafts and dry runs and an hour in GIMP making slide patterns, and is proudest of the line "Blathers was the developer because he's afraid of bugs." Laura's talk covered the dangers of third-party tooling, and Laura live-tweeted the whole event. Serena planned a technical talk about running Jaeger that turned into one about making friends with tracing.

Lessons From the Production

Austin says the production lesson is to tell speakers which side of the screen to stand on: the scenes assumed people would stand on the left, the first speaker stood on the right, and everyone mirrored, so Austin was furiously panning the camera. This year the slide inset was wider, about 800 pixels against 600 the year before, and speakers were asked to make text bigger. Matty suggests a set of best practices for speakers, including having someone else run the reactions. Laura says the Q&A after a prerecorded talk, with the speaker still in Zoom and Katy relaying questions from Discord, felt more like in person than typing into a chat. Laura and others note that participants were engaged, with watch parties on other islands and new channels, including one that spun off an ADHD ops discussion, which Serena says a platform like Discord handles better than others.

Resilience

Katy says last year had energy from the start of the pandemic, while this year the talks were more needed, messages of wholesome resilience. Austin says it's a story about resilience and shares something not widely known: LightStep had layoffs on April 30, 2020, the day of the first event, and Katy was affected and was offered an opt-out of hosting. Katy says, "don't you dare take this from me," and that Deserted Island got Katy through that, gave Katy a new community and friends, and "pretty sure it got me my next job." Austin says it was one of the hardest days professionally, and that this year Austin repeatedly doubted doing it again because the mood had changed, but decided to "take the limitations, take the things that suck and compress them into a tiny ball of suck, and then try to put that tiny ball over to the side and fill that space with good."

Serena and Arri's advice for other events is Discord's hallway-track feel and building a helpful, wholesome community. Katy asks people to watch the YouTube videos and follow the speakers on Twitter.

Melody: melody.dev

The Numbers

Matty opens with the YouTube year-in-review email: almost 6,000 views and 37,000 watch minutes, which is less than what one episode of the podcast gets listened to. That's why the show doesn't optimize for video. Matty adds that doing the show on Twitch would be less work than the podcast since there's no editing, and Bridget does not want a comment stream. Hangouts on Air history gets a mention: when Jeffrey Snover tweeted the join link instead of the watch link, a few extra people showed up on the call, and the show's highest live viewership was around 12.

The podcast had 11 episodes in 2021, which Matty notes is close to one a month and not unusual for the show, with four more ready to go. As of December 15th the most listened-to episode was All Things Docker. Bridget's theory is that recording it live on Zoom at US lunchtime, early in 2021, started a cascade of people linking to it. Second was Foundational Practices with Johan Abildskov, where Matty kept mishearing the pronunciation of the guest's name, which turned into a conversation about communication breakdown from context. Third was Doing Releases Right, the first episode of the year. Cumulative downloads are 1.6 million.

Spotify stats get their own stretch: follower count up 62 percent and listeners up 16 percent, top listening countries of Slovenia, Kenya, Serbia, Lesotho and Nigeria, and 137 people for whom the show is the most-listened podcast on Spotify. Over time the show has been listened to more than 14,000 times in Minneapolis-St. Paul, and San Francisco is the highest city. Bridget calls it "our podcast on podcasting."

Favorite Episodes

Jessica picks Words Are Hard with Emily Freeman, with one complaint: Matty and Emily kept referring to a class with a "secret formula for presenting to power" and never said what it was. Matty's pick is Drawing DevOps, with the sketch-recording side of Ashton's work and what drawing taught about DevOps over the years. Joe says it would have been a good episode for Joe to be on, since Joe came to DevOpsDays as an outsider who was voluntold into running audiovisual. Bridget's picks are the Open Service Mesh multicluster episode and Brigade with Kent Rancourt, for talking to maintainers about things that aren't in the repo. Joe's favorite was the Tech Twitter episode, purely as an editing challenge with around 15 guests.

Transcripts, Descript and the Back Catalog

Matty recommends Descript for editing, since the transcript is built into the editor and cuts are made against the text, though it can be overzealous about filler words and "verbal tics." Separate tracks per speaker (Zencastr or Squadcast) make the transcription much better, and the Deserted Island DevOps episode was hard because the tool can't tell ten voices apart. Jessica says text-based editing is the future for audio but not for video. Matty apologizes for missing accessibility and says transcribing the whole back catalog would cost over $45,000, so the show is doing it going forward.

High Points of 2021

• Bridget: IPv6 dual stack went GA in Kubernetes at the beginning of December, after behind-the-scenes work. "Something actually got done."
• Jessica: switched jobs to Honeycomb, and a partner moved from Tennessee to St. Louis, about three miles away.
• Trevor: finding a community and how much of it cares, plus riding a century on a bicycle. "I trained for it, and I didn't die."
• Matty: breakfast with Bridget three days in a row at KubeCon, "what it's like to spend time with friends again," and starting a new job at Pulumi a year after announcing Red Hat, plus a new dog, Moxie.
• Joe: a new cat, Ripley, who joined in June and became fast friends with Nimoy within hours. The rule of thumb for dogs, Joe says, is n plus 1.

The pets run long: "free as in puppy" and "free as in mattress" for open source, the dog and cat gods comic, and Joe and Bridget's house name, Gazebo 7, from a Simpsons episode, which is also the trivia team name. There's also a section on media: Trevor concedes that Babylon 5 is better than DS9, Matty and Bridget discuss the Cowboy Bebop reboot, Jessica watched Schmigadoon, and Joe reports on a water glass eggs video that ended with "We have the technology. They're called grocery stores or refrigerators."

Looking Ahead to 2022

Bridget is looking forward to being in charge of less: Andy Fehner takes over DevOpsDays Minneapolis, with Bridget in an advisory role, which leaves room for different decisions and other voices. Jessica is returning to a family beach trip at Tybee Island, Georgia. Trevor plans another century, a Seattle to Portland ride, and a triathlon in Chicago. Matty, who took over as global DevOpsDays chair from Bridget, hopes for DevOpsDays Chicago back in person in May, with the CFP open until the end of January, and for KubeCon in Valencia. Matty also ties the ADO streak of episodes to Seinfeld's method, and to normalization of deviance once a streak is broken. Joe is looking forward to a rescheduled Canadian fishing trip.

Favorite Episodes

• Drawing DevOps with Ashton Rodenhiser
• Multicluster Service Mesh With Phillip Gibson and Annie Wang
• Brigade With Kent Rancourt

Most Popular Episodes

• Most listened-to episode in 2021 was All Things Docker
• Second most listened-to episode in 2021 was Foundational Practices With Johan Abildskov
• Number three was Doing Releases Right With Scott Hain

Pet photos!

Why "Technical"

Emily's colleagues at ProAgile coach leadership, teams, processes and management. The technical coaching Emily does focuses on developers, to some extent testers, and on code and technical ways of working, which needs a different skill set and gives different results, and "to be really successful, you need both kinds of coaching." Matty says that's a gap in many Agile transformations, which stop at how to organize work, and the software engineering practices get left to the teams. Emily says DevOps is also very much about technical practices, and says the title technical agile coach, not DevOps consultant, is historical, since Agile came first.

Testing, Feedback and Approval Tests

For Emily, testing is about feedback loops: they give the feedback needed to write good code, know you're on track and know it's safe to deliver. Emily teaches unit test design and has worked as an architect on larger integration and system tests, and does a lot with approval testing. Emily explains that in a regular test you arrange, act and assert, while in an approval test you compare the system's output to a version you approved earlier, recorded from the system, using a straight diff. In essence it's "a fancy way of doing assertEquals," with a human decision on whether to approve. It works best with tools, and Emily is involved in two open source ones, Approvals and TextTest, both linked in the existing notes.

Matty raises shift left and the worry that getting developers to write tests means no testers. Emily says there's absolutely a place for people skilled at testing: they set the strategy for where automation matters, do exploratory testing and find areas lacking coverage, and can contribute to automation. Matty adds that Etsy's John Allspaw answered the question of why Etsy still had a web operations team with "I've got so much for them to do," and the same applies to testers. Matty also notes that DevOps is named after two roles but has always been about being cross-functional.

Small Steps and Pull Requests

Emily says most of the coaching is with developers: take smaller steps, get better feedback, commit more often, practice continuous integration and test-driven development, and learn to split tasks. Developers should push small commits several times an hour, each with passing tests, and the tests should run in a short time, which usually means unit tests. Many developers neglect test design because they think it's not production code. Emily says continuous delivery and pipelines depend on a steady stream of small, safe changes.

Matty asks about criticism of the pull request workflow. Emily isn't a great fan: review forces you to drop what you're doing, the discussion delays merging into master, and teammates can't build on the work until they see it. Emily wants to keep code review and design discussion but not tie it to a pull request or an integration gate, and wrote a blog post on a technique called pre-tested integration. Matty says pull requests suit open source projects where you can't pair with thousands of people, but inside an organization in the same time zone you can just pair, and that PRs encourage long-lived branches. Emily's test: if you diff the code on each team member's machine, the difference should be at most a few hours of work, so everyone designs from the same state of play, and "that's where you start to really see teamwork."

Pairing and Ensemble Working

Emily says pairing is a skill, and if nobody taught you, you may not be doing it well. Emily does more ensemble working, which is another word for mob programming, preferred because it sounds friendlier, and Matty adopts it. As coach, Emily can ask the ensemble to write a test now or back out a change to do it in smaller steps. About 10 sessions of 2 hours usually teaches a team to ensemble, and they keep using it for onboarding, starting a new task or a critical bug. Matty says live streaming coding while learning a new project has turned into pseudo pair programming with the audience, and Emily says remote work lowered the barrier to collaboration and remote ensembles work pretty well.

Learning, Getting Better and Learning Hours

Emily learns by practicing code katas, including doing the same kata in a new language, and designing exercises for refactoring, and likes learning in a group or with a teacher. On whether practice is getting better, Emily says it's big and diverse: at one organization with a 30-year-old C codebase, Emily is helping write better unit tests, while a JavaScript and React team was so agile there was little to teach. If things aren't getting better where you are, you could think about going somewhere else, and the Accelerate report shows the variety of organizations.

Emily's one thing is "keep learning." Emily runs one-hour learning hours with teams: a new technique in about five minutes, an exercise, and a reflection, with a growing collection of lesson plans, scheduled in everyone's calendar, and notes that sometimes people cancel for a crisis. Matty says intentionality around learning matters, and learning is easier to protect when the whole team commits, though it takes teaching other people your expectations. Emily says the Samman method is basically ensemble working plus learning hours.

• Emily's book
• ProAgile
• Emily's blog
• Emily's github

Approval testing tools mentioned

• https://approvaltests.com/
• https://github.com/texttest/texttest

Is DevSecOps Different From DevOps?

Steve's cynical definition is that DevSecOps is "security self-pending an invite to the DevOps party," a term security invented in reaction to the success of DevOps. The legitimate version is embedding security by default in DevOps "so we never have to use that word again." Matty says DevOps is unfortunately named, since it was so called because Agile System Administration was too long for a conference, and recalls Julian Dunn saying security is just another aspect of quality. Steve says people misunderstand DevOps as automation when it's cultural, and that security's self-induced purgatory comes from a world of throwing work over the wall and catching it at the end, so automation feels like the best way in.

Matty adds that in most DevOps transformations, security feels bolted on, and cites a talk called The 5 Love Languages of DevOps: what makes DevOps appealing to a feature builder differs from what lands with security. Audits are theater because people lie or misremember and computers don't, and automated change control won't let a deploy through when "the lights aren't green." Steve says security has to matter to everybody, and automation has to be "drip-fed, not sledgehammered."

NoSecOps and Shift Left

Matty asks how to get everyone to care about security without sliding into NoOps. Steve's definition of shift left is that it's not all the way left: a bit of security in the middle during CD for quick preflight checks, and sprinkles of security toward development and even education and threat modeling. Steve recalls static analysis tools that returned 10,000 findings that nobody acted on, versus everybody doing small things, like checking Dockerfile misconfigurations, so low-hanging fruit gets removed along the way. The aim is to find each person's easy version of security that takes no time and has a big impact later.

Matty adds that security tooling has been expensive and not democratized, so it lands on the right, using Qualys licensing as an example while inviting correction, and brings up "trust but verify": developers run checks at commit, and the pipeline verifies. Steve says the industry has made some things easier, with convergence on VS Code plugins that highlight issues in YAML, Terraform, Docker and Node.js, and GitOps, where a check can auto-generate a pull request or a logged suppression. The hard part is large rearchitectures, like monolith to microservices, which confuse many InfoSec people.

What Security People Need to Learn

Matty says the change is not learning a tool but understanding what Kubernetes does well enough to threat model it, and that nobody can hold the whole system in their head anymore. Steve says it depends on whether you're InfoSec, raised on networking and firewalls, or AppSec, obsessed with the OWASP Top 10, and notes security has its own silo between those two, which should take a page from DevOps. Steve's advice is to understand your attack surface and low-hanging fruit. Basic misconfigurations without CVEs are still bad, probably worse, and misconfiguration gets a bad rap. Steve recalls a podcast in which Ian Coldwater said early Kubernetes had real attackers because of insecure defaults, and now it's mostly misconfiguration.

Steve says security needs to hang out with the DevOps people, drop the culture of no, and include developers in tool-buying decisions so they aren't mandating. Matty says security reviews of new tools are often non-collaborative, handled through a ServiceNow ticket with no context, and Steve says if developers suggest a security tool, "just buy it," because developers like to pull tools and it opens a conversation.

Practical Tips and Learning

For starting now, Steve says use free tools: the CNCF security landscape, Trivy for container image scanning, which works in VS Code and on the command line, and Checkov from BridgeCrew for scanning infrastructure as code such as CloudFormation and Terraform for things like open S3 buckets. Steve notes that if developers use the open source tool and security wants visibility across 1,000 developers, then you buy the commercial version. Steve hosts a security-themed podcast (the transcript renders the name as CozyCast, and the existing links spell it Cosecast).

Matty shares a line from the Food Fight show's founder (name garbled in the transcript) that the dirty secret of tech podcasting is that it's how you get someone to spend an hour talking to you, and that you can't control your audience. For learning outside security, Steve recommends We Hack Purple, run by Tanya Janca, and certifications for a wide and shallow picture, and Steve did a software lifecycle practitioner certification years ago. For SREs and ops, Steve points to security chaos engineering and a red-blue exercise on a new Terraform deployment. Steve learns from short YouTube videos at 1.5x or 2x with a GitHub repo showing "what perfect looks like," going straight to the end and then back, then building and getting it wrong, and praises the CKA course for short sections followed immediately by hands-on work.

• Matty's dog on Twitter
• Shifting Left Securely (Matty's talk at devopsdays denver 2017)
• Steve's "Collaboration over competition" talk
• Trivy
• Checkov from bridgecrew
• Cosecast
• Pushing left with Tanya Janca (ADO episode)
• Cosecast with Tanya Janca
• We Hack Purple
• Security Chaos Engineering With Aaron Rinehart (ADO epsiode)

Where Brigade Came From

Kent came from a startup that Microsoft acquired around 2017. That company held an annual offsite with a Shark Tank-style exercise, a hackathon where you only needed an idea and a low-fidelity proof of concept. Helm won the first year and Brigade won the second, and both came from comparing Kubernetes to an operating system: what features of a traditional OS don't exist yet in Kubernetes? Helm closed the gap of no package manager, and Brigade closed the gap of no scripting environment. Kent isn't aware of any alternate name for Brigade, though thinks Armada would have been more fitting given Kubernetes's nautical names.

Event-Driven

Kent says the thing to emphasize is that Brigade is event-driven, unlike Kubernetes's declarative model of "do this and reconcile." Something happened, and now you handle that event. Kent compares it to AppleScript, where a gesture triggers a script. Kent calls it good for background work, "your minions," and says it's often mistaken for a CI/CD platform, which it isn't, though it does CI and CD well. The Brigade team uses it tied to GitHub, so opening a pull request triggers the build and tests and sends results back.

Why a v2

Kent says v1 was a minimal viable product and lightweight, with consequences: there was no API, and every event Brigade responded to was a Kubernetes secret, created either by a user at the command line or by a gateway, which bridges external systems like GitHub to Brigade, by talking directly to Kubernetes. A user therefore needed credentials to the cluster, and a cluster operator wouldn't hand those over unless the user was a competent Kubernetes user, a high barrier to entry. The aim of v2 was to abstract Kubernetes away from the end user, so Brigade went from "event-driven scripting platform for Kubernetes" to "event-driven scripting parentheses for Kubernetes." Kubernetes is now an implementation detail. There are no plans to support other orchestrators, but the architecture makes it possible, and Kent doesn't want to assume Kubernetes will be around forever.

Before starting v2 the team put a proposal of roughly 20 pages before the community explaining what was not optimal in v1 and couldn't be fixed without breaking changes. Kent says v2 is a complete rewrite, and the project is light on process compared with Kubernetes's KEPs or Helm's HIPs.

Community

Kent says a fairly vibrant community going into v2 seems to have stepped back and let the maintainers handle the shift, which "gives me the sads." The project is in beta, and Kent says it's safe now for people to come back. It's much easier to build integrations, with a rich API and language bindings for Go, JavaScript and TypeScript, and a Rust SDK in the works, and Kent has new swag set aside for anyone who helps kick the tires or contributes an integration. V2 work happens in the v2 branch on GitHub, and there's a Brigade channel on the Kubernetes Slack.

There's no exact release date, but Kent says v2 is definitely coming in Q4 of that year. Feature development is pretty much done, and the team is avoiding breaking changes, as Kubernetes does once something is beta. Brigade is a CNCF sandbox project that would like to reach incubation, contingent on v2 going GA and more community building. Kent says most maintainers work at Microsoft, a few others aren't currently active, and the project wants to diversify its maintainers, including by employer, and adds that contributions needn't be code.

Where to Look

Kent says most repositories under the Brigade core GitHub organization have a .brigade folder with the project definitions and scripts used to build those projects with Brigade 2. The main repo is the plain Brigade one, and there are gateway repositories for GitHub, Bitbucket, Docker Hub, Azure Container Registry and CloudEvents, with Slack and Teams in progress. Most are simple, and Kent invites people to clone one and adapt it to another system. Kent asks people to star and watch the repositories.

Bridget chats with Kent Rancourt about Brigade, a tool for running scriptable, automated tasks (in Kubernetes).

• Brigade website
• Brigade GitHub org
• Brigade blog
• Brigade v2 docs
• Brigade on Kubernetes slack

Brigade art by Ronan Flynn-Curran

What's Next for DevOps

Emily says the market is hungry for the next phase. Agile was about 20 years ago and DevOps 11 or 12. When Andrew Clay Shafer and Patrick Debois started it in 2009, teams were siloed with code thrown over a wall, and DevOps was a solution for bringing them together, which has mostly worked, though renaming the operations team a DevOps team "is not DevOps." Now the systems are distributed, the people are too, and the cloud adds managed services that remove pain but can cost visibility. Emily is thinking about how to capture telemetry and observability without getting into the weeds, and whether machine learning can help.

Matty says black boxes aren't inherently bad, and you need to know the promise is fulfilled and what to do if not. Matty compares it to Chef customers who asked how Chef does all 16 steps to build a server, when changing step 2 makes steps 3 through 10 unnecessary. Emily says systems are "a spaghetti of roots" like plant roots, so looking only at system A may miss the impact of B or C. Matty says you need to know things to the boundary at which you can influence change, like not knowing the power company rerouted your electricity.

Not All Silos Are Bad, and Left Means Awareness

Matty says DevOps over-rotated on culture because engineers will play with tools anyway, and that "not all silos are bad," as with literal grain silos, which exist for safety. The goal is cross-functionality, not NoOps. On shifting left, Matty says you push awareness and consideration left, not the work, and recalls the example of a colleague at a 1990s mail order company who kept being asked to teach everything about Photoshop in an afternoon. Emily agrees that being aware of someone's specialty, constraints and motivations is different from knowing how to do it, such as reading JavaScript without being a front-end engineer.

Emily adds that terms like "shift left" have gotten hand-wavy and people want to know how to apply them. Matty recalls 2014, when every devopsdays talk was about empathy because nobody knew what it was, and now people ask how to deal with people. Matty jokes that removing code from prod is "shipping left." Emily thinks DevOps is in late adoption, and Matty says the adoption phases overlap, noting that after a year working with state and local government many people still need the basics and others ask how to do it.

Cloud Engineering and Platforms

Matty has been thinking about cloud engineering as a discipline, which is what many people mean by SRE, although SRE is a specific way to implement it. Matty says a platform team is essentially an internal Heroku: a service with an interface, so feature teams don't need the nuts and bolts. Emily says it's a "service with an interface. That's it." Matty says a platform doesn't need bespoke Kubernetes, and can be cobbled together from other services. Matty also says the multi-cloud myth is the lowest common denominator unless you build a platform in front. Emily describes cloud engineering as being an infrastructure optimizer.

Visibility and Value

Emily asks how to stop valuing people by their place in the stack: front-end, back-end, platform, infrastructure. Some organizations favor devs, and Google in Emily's view over-indexes on SREs. Emily says much of spreading awareness is removing the fear of not being valuable, and that the empathy talked about for 12 years means "this shit is hard." Matty says it's a visibility problem: ops is like being a corporate lawyer, known only when something fails. Sales is the most visible part of an organization because everyone understands its value, and the further down the stack, the closer to utility, like a power company, the less visibility. Matty says this is why NoOps was a bad idea and likes the term operational excellence.

Words Are Hard

Matty recalls an early guest saying the value of DevOps is that it frees up the most important resource, the developers, which was the quiet part said out loud. Emily says massive companies saying they're obsessed with the developer is fine if inclusive, but about 90 percent of the operations community does not identify as a developer. Emily defaulted to engineer for the book and notes AWS uses builders, a term Emily hasn't heard outside that ecosystem. Matty: "Words are hard, which is that is what we're going to call this episode." Emily says you can appreciate someone's value without taking on their responsibility, like knowing how to change windshield wipers but not tires.

• The Best DevOps Blogs (the review giving ADO 5 out of 5)
• DevOps For Dummies

What Multicluster Means

Phil says the most accepted description of a multicluster service mesh is being able to manage multiple Kubernetes services connected in east-west communication across clusters, and notes it means different things to different people. Annie explains that before multicluster, pods had to exit a cluster to reach services in another cluster, which is north-south traffic, and with east-west traffic, pods can reach the other cluster's ingress without having to leave. Phil adds two pieces: a distributed control plane, so you can still configure the mesh if one system goes down, and an ingress for the service mesh, separate from the traditional ingress controller that exposes services to the world, where routes are populated and synced through the control plane so service A in cluster 1 knows the route to service B in cluster 2. On security, Phil says a cluster can itself be a security boundary with its own RBAC and policy controls.

Why Bother

Annie lists the benefits: a mesh limited to one Kubernetes cluster is limited to that cluster's size, so multiple clusters let applications scale horizontally. Disaster recovery and failover mean you can deploy the same service to several clusters and divert traffic away from an unhealthy one, and you can get zero downtime during Kubernetes upgrades by updating one cluster at a time. Bridget asks about API deprecations across clusters on different versions. Phil says that's the hard part, that Kubernetes became popular by offloading labor-intensive configuration, and that the team wants to keep the experience simplified and shield users from the heavy lifting, which is still being worked out. Annie says the philosophy is to align with OSM: enable complex use cases while keeping things simple. Bridget notes complexity is conserved, and Phil says to expect YAML, ConfigMaps and new CRDs.

An Intern on the Hardest Project

Annie says the internship was very overwhelming at first, with terms like Kubernetes, service mesh and even OSS new, since Kubernetes isn't taught in school. Annie is not a service mesh user, and talking to users about what they liked and disliked was valuable. Phil says the team gave the intern multicluster, the hardest thing they're dealing with. Annie says the most interesting part was learning how building features in open source differs from a regular product team, where you have a defined group of customers: you have to rally the community to figure out whether a problem is real, which is why Annie published a blog post to invite discussion.

Project Versus Product and the Spec

Phil says the benefit of open source is fast feedback, since a project will live or die on the vine. Phil looks for gaps and pain points that customers mention repeatedly, and says "you got to have thick skin in the open source game." On the Service Mesh Interface spec, which has no code, Phil says people ask where to download it, but it's an agreed-upon specification of how APIs interact, and getting consensus from a large community is slow, but listening to the community is good hygiene.

Who Needs It

Annie and Phil discovered that the need for multicluster is tied to maturity with Kubernetes. They assumed everyone wants it, from an enterprise mindset of a critical application, distributed service mesh, but early users had relatively simple applications. Phil says that if you've just refactored a VM application into a container and exposed it on port 80, multicluster is far away. Growth comes first by adding more apps to a cluster, and then the enterprise asks for regions and failover. Phil also explains mTLS with a shopping site: with TLS you trust the site, and with mutual TLS the site also knows who you are.

Learning and Contributing

Annie says Kubernetes is new enough that school still uses VMs, and Annie will bring Kubernetes back to classmates but not multicluster service mesh. Annie says university teaches the fundamentals to learn the rest. Phil's career advice is to follow your passion, noting Phil left Microsoft for Docker after a container tutorial made a light bulb come on, and came back. Phil says contributing doesn't have to mean writing lots of code, and filing an issue asking why something doesn't work a certain way counts. Bridget points out Annie's one-word documentation pull request, a clarification that was valuable, and Phil likes the embrace of content as contribution. Phil's and Annie's advice for learning is to spin up every service mesh that supports multicluster and read their docs. Phil is on Twitter as @PhillipGibson, with a lot of barbecue brisket.

• Previous ADO episode about service mesh with Michelle Noorali and Delyan Raychev
• Annie’s blog post about Multicluster Service Mesh
• Service Mesh Interface specification
• Open Service Mesh
• Service Mesh Comparison

art credit: "Spiral" by roland - CC0 1.0

What the Statement Means

Johan says that when you visit software teams, a lot of effort goes to things that should be boring: daily ceremonies, meetings, arguments over who failed a build and why the Git branching strategy is so complex. Because teams don't invest enough in core practices, those stay roadblocks to thinking at the level of abstraction they want. Johan's example is skill in an IDE, which never feels important enough to invest in, so Johan keeps paying a little and never becomes awesome at it. The idea came from the book The Art of Learning, recommended by the streamer Day9, about practicing fundamentals until they become muscle memory. Johan says we have an intuition for this in physical things but not in programming, and we lack the vocabulary for building automated responses that free the mind for creative work.

Practice With Intent

Matty says muscle memory comes from practicing with intent, and that you can practice in a bubble, like a Vim playground or Vim Adventures, but the skills must be used in real work. Matty has read about VS Code tricks and then gone back to old habits. Matty's approach, from bowling and public speaking, is to think about one thing at a time, such as making gestures big in a talk, and to do the same with a team, choosing one practice to be intentional about for a sprint, because foundational things are so broad that you feel you need to do all of it and so do none.

Johan says most software teams lack intentionality, discipline and explicitness, and that test-driven development forces doing things with intention. Matty says not all work is equal, since exploratory work like a spike is a different kind of work, and Johan says naming what mode you're in helps: with a spike, you're accountable to see what sticks.

Johan separates awareness from proficiency. Practicing a new IDE trick in a sandbox makes you aware, then applying it in context until it becomes muscle memory is "very Toyota Kata thinking." For other basics, such as fast and stable builds or a programming language so unfamiliar you can't read the screen, you need to practice outside your context so the smaller component parts stop mattering, like no longer pondering what float64 means. Matty compares it to thinking in a language instead of translating, and adds progressive disclosure: people with a new tool want to solve their specific problem right away, but first need the vocabulary, like the staging area in Git.

Teaching and Knowing What to Ignore

Johan brings up the zone of proximal development, meaning what you can do alone, with help, or not at all, and says experts forget what was difficult. Putting too much on a slide makes it hard for newcomers, who don't know what they can ignore, which is why teams argue endlessly about one repository versus many, GitFlow versus trunk-based development, or merge versus rebase, which Johan says isn't that important. To teach, you have to decompose things or hide details, even if not technically correct, so the learner gets the correct intuition. Matty suggests a simplified test in a pipeline demo, like checking 1 plus 1, to show that something tested a thing, and says it's fine if the educator corrects it soon and tells people it's a simplified view. Johan says being exhaustive does people a disservice, and taking responsibility to filter is like reading the SRE book so they don't have to.

Agendas, Open Spaces and Not Controlling the Audience

Matty says the best conversations often go somewhere other than planned, as in the Tim Banks episode, which started as ops life and became gatekeeping, and compares it to open spaces: whatever conversation happens is the right one. Matty says you can't control your audience: early listeners of the show weren't the target audience. Matty recalls that people tell Matty things they got from talks that Matty didn't intend, and gives the example of the film Memento and the question of whether a character was ever a cop. Matty also says some meetings need structure, like a stand-up, but others should stay open.

Johan says being agile requires strong foundations, and suggests a stack of index cards as an agenda where each topic change adds a card on top and each completed topic removes one. Johan says that if important things go unaddressed they become urgent, and under pressure you fall back on what's the path of least resistance, so the right way has to be the muscle-memory one. Johan adds that people often believe unit tests will slow them down when they'd finish sooner by writing them, and there's a disconnect between how we believe we use time and how we spend it.

Work as Imagined Versus Work as Done

Matty, crediting John Allspaw, says the gap isn't only between management and practitioners: we also do it to ourselves, and closing it takes honesty and psychological safety, like logging food in MyFitnessPal without recording what you actually ate.

Caring About the Wrong Things

Johan has an antipathy for GitFlow, which has done good for the community, but in organizations it often achieves the opposite, as feature branches get huge and end in horrible merges, and people afraid of their workflow postpone. Johan says mono versus many repositories is a discussion people can feel, but the better question is which developer workflows you want to enable. Johan says you shouldn't care about Git or Kubernetes but about the platform built on top, and that if an IDE and a backend like GitHub can't handle your version control tasks, they are too complex. Johan's point is that "a developer doesn't want to do a push. A developer wants to move some code somewhere."

Matty adds commit-message formatting as another over-rotation, between the pedantic commit hook and "fix typo." Johan warns against elaborate gated workflows that don't match how work is done, and suggests documenting the process as it is first, then changing it organically, because friction kills productivity, motivation and trust.

• The Art of Learning: An Inner Journey to Optimal Performance
• "Zone of proximal development"

Graphic Recording and Graphic Facilitation

Ashton calls the conference work graphic recording: listening as an outsider, processing what people say, and outputting it as drawings and text. Graphic facilitation uses the same skills in a facilitated session such as strategic planning or brainstorming, where the graphic captures the collective voice of the room and not a single speaker. Ashton describes being "a fly on the wall."

How It Started

Ashton studied early childhood education, worked at a nonprofit family center, moved into family support work, and fell in love with facilitation, which has the thread of teaching, and "you don't actually have to know the information." A friend sent a one-day graphic facilitation course in 2013, which Ashton says was the perfect marriage of creativity and group process. Ashton applied the skills right away with a young adult group exploring pluralism, spent 2014 and 2015 playing with it on the side, and got a scholarship to a 2015 conference of the International Forum for Visual Practitioners in Austin, Texas. Ashton left inspired that others had figured out how to do it full-time, and officially started Mind's Eye Creative in August 2016, after what Ashton calls a dark launch.

Ashton says the work sells by being experienced, and got conferences by cold email. The fifth cold email went to the devopsdays Toronto organizers, who said they didn't know what it was but it sounded kind of cool. It was the first tech conference and the first job Ashton had to fly to, and Ashton watched DevOpsDays talks on the plane, worried about not understanding them. At that first event Ashton was in the green room watching a feed, so attendees didn't see it unfold, but in later years Ashton was out in front of people. Toronto was followed by Dallas and Columbus in 2017, and Chicago in 2018.

Drawing Without Context

Ashton says not knowing the industry can be a strength, since it lets Ashton hear the larger picture without the baggage of the industry, and aims to draw what was said without interpreting too much. Having some context now helps with drawing things like Kubernetes and avoiding misspelled words, and Ashton challenges themself to draw time five or six different ways so as not to repeat.

Ashton likes talks with a human element and appreciates community and collaboration themes, and feels a responsibility when speakers share vulnerabilities to represent them in a way they'll feel okay about. Ashton recalls a DevSecCon talk by someone who helps people with autism find technology jobs.

Facilitation Versus Recording

Matty notes that a recording preserves a moment in time, while a facilitated meeting's output is fluid. Ashton says graphic facilitation can be used whenever a group, small or large, is having a conversation, and usually works with a separate facilitator, since it's hard to hold the visual role and make sure people feel safe and heard. Matty agrees with Ron Swanson that you shouldn't "half-ass 2 jobs, whole-ass one." Ashton says drawing in a session lets people see they're heard, telling of a group unhappy to be there that Ashton won over by hiding a little anchor in the drawing after overhearing a side joke. It's also a gentle way to tell people their point has been captured and the group can move on, and shows connections: in a session for a town's energy project, most of the conversation led back to one topic, which gave the town direction. Matty points out notes are linear and hide connections across pages, and Ashton says facilitation drawings are messier, but "we have to get through the draft."

Tricks for Mistakes

Matty says the game Drawful has no erase button, and Ashton has practice with that. On paper, Ashton has been at it long enough to judge lettering and drawing size against a talk's time, roughly so much space per five minutes, though speakers may spend 20 minutes on the first of five points, so Ashton sends out "invisible questions" about where the speaker will go next. Ashton channels Bob Ross and happy little accidents, and carries white mailing labels in different sizes to cover mistakes, since they photograph invisibly and can be written over. If Ashton can't spell something, Ashton leaves a blank and comes back, and recalls misspelling "health" three times during a mental health event. Digital has made erasing possible over the past year.

What Ashton Learned About DevOps

Ashton says what stands out is how tight-knit the community is, and that Ashton hangs out on Twitter because that's where the community is. As an outsider to technology, Ashton noticed how fast things change: the first Kubernetes talk Ashton drew was in 2018 and it was a new idea, and now "if you don't use it, you're a loser." Ashton says the last year was all observability, SLOs, SLIs and SRE, and jokes that buzzword bingo would be a win.

Drawing to Think

Ashton's advice to people who say they're bad at drawing is to try, reframing drawing as "a thinking and an understanding and an idea tool," not something for a gallery, so it's fine if it looks horrible. The second tip is to turn plain white paper on its side, because our brains think spatially and not in top-down lines, and with nowhere obvious to start you choose where to put your first mark and make better connections. Ashton has a free 30-page ebook on the visual communication method, linked in the existing notes, and is on Twitter as Mind's Eye CCF, for Creative Consulting and Facilitation.

(episode art by Ashton Rodenhiser)

• Ashton’s ignite at devopsdays Texas
• Mike Tozer's talk at DevSecCon
• Visual Communication Method - ebook by Ashton

Jess: "Cat is known internationally for her Penguin Power Stance and for standing on the edge of now!" Cat: "I like working with things on the edge of now. So that's either things that shouldn't exist anymore or shouldn't exist yet."

Jess and Cat talk about projects that fit Cat's definition of the edge of now.

Cat: "I do believe that DevOps is inherently feminist, because it puts the emphasis on that maintenance and reproductive work rather than producing working software."

Jess brings up Eric Evans' concept of software as gardening.

The panel discusses reproductive vs productive work. Cat talks about what she sees in healthy DevOps teams.

Jess and Cat talk about the challenges of working with legacy systems.

The panel discusses metacommunication and Gregory Bateson.

Cat: "In that situation, everyone is operating as specified but it's still not working! So that's when it becomes necessary to not have so much of that transactional interaction...It has to be something more generative."

Cat and Jess talk about the ethics of "care vs fair". Cat dives into how feminist theory has impacted her thinking on DevOps and systems.

Cat: "If we valued caring for the systems and caring for each other, rather than thinking fair or unfair...Let's check, are we caring for these systems, are we being mindful? Then rad, let's keep going."

Tech Asks a Lot

Katy observes that tech companies make things that make developers' lives easier and then ask the community to build things and show what they can do. It's voluntary, but Katy wonders about it: Katy doesn't follow companies on personal Twitter, so why would someone follow the company's account, and what are they getting? Being part of a community is often "more gives than gets." Katy adds that community is an important part of the business model for many tech companies, recalling one with a big open source arm whose core functionality wouldn't work without community contributions, so the company benefits economically.

Matty mentions Mary Thengvall's rings of community: people who are aware, users who aren't customers, customers, and at the center advocates, which isn't a job title but people who do it on their own and where scale happens. Matty says the community isn't only people who contribute code: people write tutorials, do Twitch streams, help others and tell others, which is hard to measure. Katy calls community a branch of the business, where if it isn't a success it stalls marketing and then sales, and wonders what makes the unpaid work worthwhile.

Is It Volunteer Work?

Matty asks whether other industries have this, using Peloton and CrossFit as examples where networks enthusiasts build add value, but notes you can't go work on Ford's assembly line for 20 minutes. Katy says developers have the same skills the tools require, so they can build the thing the company builds, just for something else. Matty notes that what companies ask of communities is often the same thing people did all day at work, and says when someone isn't paid for it, "that's volunteer work," so call it what it is. Katy says the reward then isn't necessarily for the volunteer, like planting trees along the highway as a kid, and asks whether the reward is knowing you did something nobody else did.

Matty says there isn't one reason, since people participate for different reasons: identity, recognition (Matty is recognition-driven), the satisfaction of solving a problem, or feeling smarter than everyone else. Katy says that makes personas hard to build, and ends up with buckets like swag-driven and recognition-driven. Katy also says a community member still expects to be treated as a customer by a vendor they pay, but wouldn't expect a candy company to care about their opinion on banana flavor. Matty adds that people's personas change by community and time of year, and that the value is in being inclusive of different reasons for participating.

Unpredictable Contributions and Ambassadors

Matty says depending on community contributions is hard because you can't capacity plan or burn down. Matty's open source podcast theme gets spiky attention, with months of nothing followed by seven features in two weeks, and contributors who are active for a couple of months and then leave. Katy says people have seasons of community, and used to write essays on Digimon forums. On ambassador programs, Katy was once a random Hotmail ambassador at college and received money and a sweatshirt, but says programs often come with criteria and pressure, like a second job. There's a tipping point where "I think I work here now, but for free."

Matty says the first few advocates at a company get a lot of pressure, since they're asked for everything, and organizations should encourage people to continue doing what they already did and not make it a KPI, since DevRel teams tracking advocates in a CRM may panic when someone stops being a super fan for a moment. Matty's maxim: "you can't manufacture a moment," so nurture where it's happening. Katy says Docker Captains worked because the skill became valuable on a resume organically, which wasn't repeatable elsewhere. Matty adds that you can make the soil fertile, but what grows might be an artichoke and not the organic avocado you wanted, and tells of a community manager held to Ubuntu's forum activity for a far smaller paid SaaS product.

Measuring the Value of a Community

Matty says the value of a community isn't inherently the number of people in it, so month-over-month growth may eventually stop, and that's probably fine. Matty asks how to observe the value, avoiding a KPI. Katy says after a year you instinctively know how it's doing, by seeing every interaction across Twitter, the forum and Slack. For a community Katy manages, success is people asking lots of questions and getting interaction: "I measure community in curiosity." Katy is on Twitter as TheCaterTot and wants to hear from listeners.

Listening to Customer Problems

Donnie joined Docker about five months earlier and has tried to focus the company on customer problems, since tech companies often build a solution and ship it, and then three-quarters of the time it flops. The pain points Donnie hears most are developer productivity and velocity, and finding things you can trust instead of a random repository with three stars or a Stack Overflow copy-paste. Bridget adds the question of whether to take a dependency on something that hasn't been updated since 2019.

The Public Roadmap

Justin says Docker tried to change how it labels readiness. Previously enterprise products were marked experimental and you had to "figure out the magic runes" to turn a feature on, which discouraged trying things. The team is culling experimental flags in favor of notices that something is new, and offers early access to developers who want it. The public roadmap has stages, which Justin walks through. Anyone can add an idea as an issue, and product managers look at new items daily, with a weekly meeting that looks at new items and thumbs-up votes. Items move through investigating, writing the code, almost there (asking early users to try it), developer preview or experimental with a public way to try it, and shipped, which means some form of general availability, with feedback still welcome.

Donnie adds that "investigating" also asks whether a feature is valuable enough that people would subscribe for it, since Docker has a long history of giving lots away and needs a sustainable business model. As an example, the M1 support item became the most upvoted item on the roadmap of all time within days of Apple's announcement. Donnie says Docker isn't a huge company and has to be careful about big engineering investments without a customer signal, so they hadn't started earlier, since it was unclear when it would matter and what else they could be doing.

CNCF and Open Source Decisions

Justin sits on the CNCF Technical Oversight Committee, and says cloud native is everything in modern software delivery. The community is large and open, people usually know what's coming, and there's less of a big-announcement culture and more collaboration. Justin says much of the CNCF is about the production end, but more developer-side projects are arriving, such as a Spotify project (whose name Justin forgot). Justin likes assessing projects, since it gives an excuse to ask users how a project is working for them.

Donnie says deciding what to open source comes down to what the company wants to accomplish, the maturity of that layer of the stack, and whether it's differentiating or becoming a utility or commodity. Open standards help align vendors around things that are becoming a commodity. Bridget notes that end users love any interoperability, anywhere they can just ship that container.

Remote-First

Justin says Docker chose to become fully distributed and dropped offices, going from a San Francisco company to one about half European and half US by the end of 2019. The lease on the Cambridge office expired about six months earlier, and it seemed weird to renew it. Docker used to send every new hire to San Francisco for a week, which had become a relic, and the pandemic made the change happen faster. The main limits on hiring are tax, legal operation in different places, and time zone overlap, which Donnie says means deciding whether to look in geographies that overlap or find people willing to work different shifts. Donnie says the one hour of overlap between Germany and San Francisco means you have to optimize for autonomous or asynchronous work, since "force-feeding" synchronous models doesn't work. Donnie adds that online communities and platforms suited to online events reach people where they are.

DockerCon

The DockerCon CFP was closing in a few days. Justin says last year's DockerCon was planned as online before the pandemic and drew about 80,000 people, and it showed how much more accessible conferences are without travel. DockerCon is about 80 percent people who identify strongly as developers, unlike KubeCon, where the strongest group works in infrastructure, and Justin wants stories from people who wouldn't normally be heard. The CFP asked about team collaboration, since helping teammates onboard and sharing images is part of the product, but Justin says to submit if it's interesting for developers.

Donnie says conferences let people swap stories, and that "nobody wakes up saying like, oh, I want to use this tool today." Donnie cites a stat, unverified, that 50 percent of developer time on cloud-native applications might go to configuring instead of writing new code, and Bridget wonders how much of that is usability and how much is security compliance. Donnie describes Docker's experimental Hub CLI tool, and says the interesting part is watching how people use it in pipelines, such as automating token rotation, making sure images have both M1 and x86 builds, or monitoring subscription seats, because people want a use case and not just a tool.

• Docker Public Roadmap
• DockerCon CFP open until March 15th - How to Write a Great Talk Proposal for DockerCon LIVE 2021
• Docker Career Openings
• Docker Hub Experimental CLI tool

Growing People Who Live in Production

Shelby says the idea grew out of a Twitter discussion the year before about what makes an SRE: is there a junior SRE, do you need an Ops background. Shelby wants more people living in production and sees observability as helping. Shelby is wary of being prescriptive, noting that many senior people cut their teeth in server rooms while newcomers have never seen a server rack, and Shelby is a cloud-native engineer who has never worked on-prem. Shelby's own lesson came from owning a tool without releasing a new version, since a year into a career Shelby didn't know how to release a Python package: pretty maintainable code is meaningless if it doesn't reach users. The question is how to teach deployment, maintenance and long-term software life cycle to bootcamp graduates and CS students.

There's No Linear Path

Matty says experience is the best teacher but "you can't do unless you have experience," and that it's hard for experienced people to shift their frame. Matty recalls a tech screen with softball opening questions that virtualization-era candidates, who never dealt with hardware, couldn't answer, and trying to learn an API at PagerDuty whose documentation was written by developers for developers, though many consumers were ops people. Matty says Gatsby is built for React developers, which is why it's frustrating for others.

Shelby says "there's no linear path to software practitioner knowledge," and with so much information, constantly changing and deprecated, you can't expect people to have the same knowledge. Even senior people benefit from building from first principles, and Shelby tries to state up front who the intended audience is and what readers should know. Matty learns best when solving a problem, like struggling to understand Habitat until finding a use case, and likes "journey tutorials" that start with the problem and not just how to call a function.

The "So What?" Question

Shelby says developer advocacy makes you want to have experienced something before talking about it, and describes a second internship outbrief that the CEO wanted to attend, which led to coaching from people who present to three-star generals on organizational impact and the question "So what?" Shelby needed the "so what" on Kubernetes and load balancers when reading the SRE book in 2017, and says product pages and open source repos often fail to say what problem they solve or why to choose them over the standard library.

Plan Your Message

Matty says communication is a skill, and that both took a structured course called Communicate to Influence. Having a plan for what you're communicating beats a brain dump, which is what podcasts are for. Shelby says that when feeling insecure about being technical enough, the instinct is to drop a stack of books on someone's desk, but "the firehose doesn't really help people learn." Shelby is a stronger writer than ad hoc speaker, and writes internal docs in half an hour in response to questions. Both admit they rarely use the method they learned, though it's effective, and Shelby's manager suggested using it for CFPs.

How to Learn

Shelby's first tip is to figure out your learning styles, which are a spectrum: visual, auditory and kinesthetic. Shelby is a multi-modal learner who needs to interact with a thing every way and takes three to four times as long to grok something, but then can teach it. The second is the book Mindset, about growth mindset. Shelby grew up very hard on themselves and thinks the industry has gotten better about the gatekeeping and incredulity toward people who haven't heard of something yet, and Matty links a talk by Sasha Rosenbaum on Mindset from a meetup the week before.

Matty ties it to psychological safety, which isn't just not being mean but includes how you answer a question in front of others. Matty recalls mocking a colleague in HipChat for a capitalized SSH flag, which sent the message that "If you make a mistake, Matt will make fun of you." You should think about the people listening, not the people you're talking to.

Shelby says early in a career, teaching PhD-level aerospace engineers a Python library showed that everybody has gaps and "there's no linear path to expert." A platform engineer at Honeycomb admitted not fully grokking something, and Shelby blogs about things like breaking DNS on a Hugo site to share those gaps, and learns by thinking out loud and tweeting. Shelby also recommends Lichtenbergianism, a book about procrastination as a creative strategy, and embracing abortive attempts, the half-finished repos and unpublished posts. When learning a codebase, Shelby refactors and renames to see ripple effects, breaks things, then clears the index and starts a new branch for the real work.

Risk and Privilege

Matty notes learning in public carries different risks for different people, but cites Annie Hedgpeth, whose blog about learning InSpec became the documentation linked from the product page. Matty, who learned by doing without a degree, credits Scott Hanselman with the observation that someone who doesn't look like Matty wouldn't have gotten the same chance to not know what they're doing, and recalls being asked about a VPN in 1998 and told to put one in the next day.

Shelby says luck mattered too: on the first day of a second internship, a product lead asked the linguist whether they wanted to write a translator, which turned out to be compiler theory. Shelby says being able to spin that into a positive story is not available to everyone, and that appearance and paper credentials lend credibility to some people. There's a disproportionate penalty for people who don't fit the mold when they raise organizational issues or push for psychological safety, and "Move fast and break things" doesn't work for everyone. Shelby thanks senior people and people outside minority groups for raising it, because it takes the burden off those more impacted and avoids being labeled as ruining everyone's fun.

Matty plugs the return of Deserted Island DevOps on April 30th, with scholarships for underrepresented speakers who lack a Nintendo Switch or Animal Crossing. Shelby will speak at Developer Week in February and join an InfoQ panel.

• Learning Stuff With Ali Spittel - ADO Episode
• Managing Your Mental Stack - ADO Episode
• Mindset: The New Psychology of Success
• Sasha Rosenbaum's talk about Mindset
• Lichtenbergianism: procrastination as a creative strategy

The Same Plots Again

Kat nominates gatekeeping, in which a startup CEO or VC posts a terrible opinion about women in tech or bootcamp students, everyone quote-tweets to dunk on it for 48 hours, and the same thing happens again about four months later. Matty adds that anecdotes get treated as data: someone says it didn't happen to them and treats that as a refutation. Aaron says arguing past each other is a big trope. Sasha notes that telling a story about one customer is how sales and DevRel work, and Matty says the problem is when a personal anecdote is treated as refuting someone else's. Matty's example is that Kubernetes is bad because it fell over in one organization.

The recurring topics include arguing about Friday deploys about once a month, whether developers should be on call, and front-end versus back-end, which Sasha says turns into gendered gatekeeping. Quintessence adds that Twitter works as a live status page of who's down. Aaron says that when the Friday deploy and on-call arguments die, the DevOps transformation will be done, and the periodic reminder not to deploy on Fridays inevitably tags Charity Majors and leads to the same argument with the same people.

Clever, Funny and Hot Takes

Matty says people who think they've come up with a clever joke should search for it, since the New Year's resolution joke has been made a thousand times. Sasha says not everyone checks Twitter 15,000 times a day, so every joke is new to someone. Matty, who studied improv, recalls the rule "don't try to be funny," and Kat says funny tweets are "intrusive thoughts." Matty says the carefully workshopped tweet never lands and the random thought gets the engagement. Sasha says Twitter downgrades a tweet that gets no likes in the first 30 seconds, and links to YouTube or Spotify, while native video does better.

On defining shitposting, Quintessence says hot takes, and Kat says it's a non-serious statement taken to an extreme, never intended as true or factual. Jeremy and Aaron say it should be nearly void of valuable content, and Jeremy adds there's a purposeful-troll version where someone drops a statement and walks away. Matty notes a hot take that everybody agrees with isn't one, and Kat calls the often-argued ones, like washing cast iron with soap, "room temp" takes, since people still argue.

When to Argue

Sasha says arguing with people made Sasha angry and obsessed for a week, while not engaging means forgetting within hours. Quintessence only engages if someone said something awful to someone. Kat says if something is actively harmful, having an audience of almost 13,000 comes with a responsibility to say it isn't okay, but Kat won't engage with a crappy take about 10x engineers. Jeremy checks a replier's profile quickly to see whether they're in good faith. Matty frames it as public versus private: you may be arguing for the benefit of the people watching, and if nobody is listening it just makes you louder and angrier, so take it to a group text. Aaron offers three levels: call out bad takes publicly, have good-faith arguments loudly so others learn, and walk away from bad faith.

Sasha says a blog post from another person with about 14,000 followers compared a follower count to a stadium of people listening, which stresses responsibility. Kat describes how a pithy tweet in a thread explaining a change to Kubernetes carried no weight at 4,000 followers, but 36 hours later at almost 12,000, people assumed Kat worked for Google or Docker, because Twitter has no context. Matty adds that as your audience grows, your tone and personal posts change, which isn't self-censorship. Sasha says putting "opinions are my own" in a bio doesn't help with screenshots, and Kat removed it since it's not a legal disclaimer. Matty says anyone who wants to go after someone won't read the bio, and shares a friend's email footer: "by opening or replying to this email, you agree that all of my opinions are correct." Sasha adds that Sasha's out-of-office replies are valid JSON.

Tips for Using Twitter

Kat's tip: if you're not a man and get a filtered DM from someone you don't follow that opens with just a greeting, don't respond, since often it's an inappropriate photo. Kat and Aaron add that nobody should open a DM with just a greeting, and to include the actual question. Sasha suggests that if you're not a man, consider not having your DMs open. Matty recommends private lists to curate about 100 accounts, since lists are always chronological and don't hit the algorithm. Jeremy suggests pinning lists in the mobile app. Quintessence says if your feed is monochrome, deliberately follow leaders who aren't white or aren't men, and Twitter's recommendations will follow. Sasha says to follow people beyond the big names, and that advice differs under or over 1,000 followers. Aaron says lists matter more when your follows act as endorsements. Matty and Sasha keep private lists of cute animal accounts for toxic days.

Jeremy's last tip is to use your walk-away power and shut the app off. Sasha and Kat say to have at least one friend who understands Twitter and can take a rant, since many real-life friends only see screenshots on Facebook. Quintessence says a major news cycle mixed with sales pitches and Friday deploy takes can be stressful. Sasha reminds everyone that Twitter is also a community for support, hugs and cat videos, and on most days it's good.

• The Six Main Arcs in Storytelling, as Identified by an A.I.
• The Seven Basic Plots by Christopher Booker

Delivery Versus Deployment

Scott uses CD for continuous delivery, meaning making an artifact from code, whether a single binary, a binary with installers and wrappers, or something behind a SaaS. Continuous deployment is what happens when you actually ship a service or upgrade it. Matty says continuous delivery means your software is releasable at any point and shipping is a business decision, and cites Ken Mugrage's point that a holiday code freeze only freezes deployment, not the work. Scott adds that the release cadence is a human decision, and the aim is the best artifact sitting ready at any time: "if it's not ready to go, it's not ready to go, but it should be able to go." Matty notes "ready" doesn't mean complete, it means ready to make the decision to release.

Scott says a release is also blog posts, announcements and other coordination that are mostly human-driven but can be automated.

Versioning

Once code is merged, the CI system creates a versioned, signed artifact, and the same code should yield the same binary, with caveats for Apple and Microsoft signing. Matty says it's a point in time, so you cut a new one, and versioning strategies stop you from ending up with "artifact.final.back.back.back." Scott says a rolling staged artifact removes the need for betas and release candidates, because customers could run it in their integration environments.

On versioning schemes, Scott likes SemVer but says it doesn't always make sense, since customers ask about gaps, and another common option is a date timestamp. Matty jokes about putting a year in the major version, as with Office 2003, and notes the branded version differs from the artifact version. Matty says the wrong way is to be inconsistent, and complexity costs the people who must understand it. Scott disagrees slightly that the whole organization needs one scheme, as long as each project is consistent, because "consistency breeds automation." Scott also mentions meta-versioning for bundles of multiple artifacts.

Workflows, Mandates and Tooling

Matty says that if you try to find one true workflow for every project, nobody will love it, since compromise is when nobody's happy. Scott says mandates don't work and "do whatever works for you" doesn't either. A cross-functional release engineering or quality team can recommend tooling and set the things teams must follow, with exceptions allowed if explained, like SOX. Scott's key point is that engineering tooling has to make engineers happy, be easy to use and add value, or people will use workarounds.

What Quality Means

Scott says measures of quality differ by company: MTTR, incidents, number of bugs, Sev 1s, or uptime for a SaaS. Matty says people work to the metric you give them, telling Jez Humble's story of adding one test per sprint and getting assert equals true, and warns against tying metrics to compensation, which Scott strongly agrees with. Matty notes that if you measure Sev 1s, teams focus on "mean time to innocence," and wants metrics that are interesting and actionable, making your ears perk up. Scott suggests ISO/IEC 25010 (SQuaRE) and the Quamoco framework as starting points, both linked in the existing notes.

Scott defines customer confidence as how much customers trust that your software does what they need, and how little they have to think about it. Matty recalls a tweet that customers want to be unaware your stuff exists and cites the Futurama line "sometimes when you do your job right, nobody even knows you did it at all." Scott says a telltale sign of low confidence is a customer saying "We're fucking tired of being your QA," and points to Nicole Forsgren's book Accelerate for why quality speeds you up.

The Nuts and Bolts

Scott says to make PRs run as many tests as is reasonable within a reasonable time, and to build an artifact from a PR so a developer can pull it down and debug locally. When merging, run exactly the same tests, because main branch creep is real. Build a workflow with tight feedback loops and quality gates in which each stage raises confidence: quick, high-value unit tests first, notifications, tests on the binaries, notarization, load tests, a long-running instance with customer-like data, and upgrade tests. Scott says to talk to customers, since a bad upgrade experience means people won't upgrade, and then you get a Sev 1 and a four-day upgrade. At the end, you have a releasable artifact in a staging area, and a button that makes it public.

Getting There From Here

Most organizations aren't greenfield. Scott says treat it as a migration in chunks with story mapping, noting that a year-long fix of a release process at one place was painful and that Scott isn't a fan of big bang. Matty agrees that any transformation should be iterative, since you'll miss some ifs anyway, and ivory-tower architects don't dictate everything. Scott recommends talking to engineers, without listening to everything they say, and to support staff, making a tactical plan for a small problem, writing RFCs, and tying it to company value and goals. Matty mentions a talk called Everything's a Product, applying product management to internal services such as release engineering, since your customers are the engineers, without the NPS score on a Jenkins pipeline.

Scott's parting advice is to be empathetic to customers, engineers and teammates, and quotes Adam Jacob's line "happy people make happy software, which makes for happy customers," paraphrasing slightly.

• Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models
• Operationalised Product Quality Models and Assessment: The Quamoco Approach

Favorite Episodes and the Numbers

Matty's two picks are Deserted Island DevOps, which Bridget had warned would have far too many people on one episode, and Breaking Down Gates with Tim Banks, which started as an ops conversation and "ended up talking about something else" that was better. Trevor and Jeff both pick the DevOpsDays Chicago 2020 episode, with Jeff calling the event "the best virtual event that I have attended." Jessica picks Don't Worry, Do Care with Aaron Blohowiak, about Netflix letting developers start as many services as they want: "don't worry about what it costs if this is worth it, but care how much it costs." Bridget picks Tea and Anarchy, for bringing together "overlapping, intersecting, yet disparate points of view."

Matty's stats caveat is that there are "3 kinds of lies: lies, damn lies, and podcast listening statistics," so no numbers get shared. The most listened-to episode of 2020 by a wide margin was Deserted Island DevOps. Second was We're Always Learning with Patrick Debois, which was also the first episode with new co-host Jeff. Third was the communities episode with Jono Bacon. Matty's method for real listens is to cut downloads in half, and to treat downloads within 24 hours of publishing as a proxy for subscribers, since podcast apps download new episodes whether or not anyone listens. Both numbers keep growing.

Pre-Recorded Versus Live

Bridget has come to like the pre-record format, which let Joe edit the KubeCon EU Helm talk as a pop-up video with commentary from the other project maintainers. For KubeCon North America, the talk was a podcast-style conversation with no slides, and about an hour and a half of footage was cut to a 35-minute slot. Jeff adds that pre-recording lowers a barrier for underrepresented and nervous speakers.

Matty describes having changed position on pre-recorded versus live after a long talk with Jessica, who argued that live lets speakers reference each other's talks, and then changing back. Matty ties it to work as imagined versus work as done: in a virtual event, people tend to "pop in, do their talk, peace out," and the one exception Matty saw was Deserted Island DevOps, where all the speakers sat in Zoom together all day. Jessica says it's the only virtual conference actually attended this year. The larger point from Matty is not to copy the physical event: for DevOpsDays Chicago the rule was "I don't wanna hear a damn word about technology," and to start from outcomes. Matty predicts a virtual buffet line will appear within two months, and Jeff calls the skeuomorphic approach a failure to take advantage of new avenues. Matty says to look at small events for innovation, since "the risk profile is less" and nobody has hundreds of thousands of sponsor dollars on the line, and Deserted Island DevOps was "Austin fucking around."

What Happened in 2020

• Trevor became a product manager, founded the Illinois Shuffleboard Association as its treasurer, learned green screening for the virtual DevOpsDays Chicago, and got a puppy on January 1st.
• Jessica gave a keynote with Avdi at Codebeam on March 7th and 8th, closing the conference, and calls it the close of a conference speaking career, for now. Jessica now teaches workshops, including Invitation to Systems Thinking with Kent Beck.
• At the end of 2019 Bridget announced a plan to travel less, and apologizes for "causing the apocalypse." Bridget also passed the global chair of DevOpsDays on to Matty after five years, saying it was time for the next generation of leaders.
• Jeff finished the book Operations Anti-Patterns, DevOps Solutions, and describes the kids seeing their names in the dedication. Jeff's framing for 2020 is that "we're all in the same storm. We're not all in the same boat."
• Matty's last in-person event was DevOpsDays New York, then a move from PagerDuty to Red Hat's transformation office, focused on state and local government, where the equivalent of "we're not Netflix" is "we're not the Department of Defense." Matty also started DevOps Party Games with Jeremy Meese, a monthly streamed game show built on custom Jackbox-style content, with a second league in a friendlier time zone planned for January.

Babylon 5

Matty tweeted that listeners could ask the hosts anything for the year-end show and got one question, from Josh Zimmerman to Joe: what's the best episode of Babylon 5? The question traces back to an Ignite talk Joe gave at DevOpsDays Madison in 2016, on the most influential TV show no one had ever heard of. Joe's pitch is that it was one of the first shows with an overarching plot, running one story over five seasons. The short version, Joe says, is that it's Deep Space Nine "but good," which Trevor and Jeff push back on.

Bridget tells of being asked in a job interview, "Star Wars or Star Trek? Show your work," and answering Babylon 5: Star Wars is fantasy, Star Trek is a utopian future, and Babylon 5 has a real future where dock workers strike and people are locked out of their offices for not paying rent. Joe's friend Christian Harrow replied on Twitter that the obvious answer is Severed Dreams, so Joe went with the best and worst episode of every season instead. The best picks:

• Born to the Purple (season 1, episode 3)
• The Long Twilight Struggle (season 2, episode 20)
• And the Rock Cried Out, No Hiding Place (season 3, episode 20)
• Moments of Transition (season 4, episode 14)
• Day of the Dead (season 5, episode 8)

The worst picks are Believers, Confessions and Lamentations, Walkabout, Racing Mars, and Phoenix Rising, the season 5 episode about the telegoths. Bridget explains that the show was rushed to wrap up its plot when cancellation looked likely, then rescued by TNT with a fifth season that had nothing left to do, and disagrees with Joe on Racing Mars. The telegoths also get flagged for lighting candles on a station where earlier episodes treated oxygen consumption as a serious concern.

The episode drifts from there into vehicle-themed 90s TV (Airwolf, Knight Rider, Street Hawk), the Star Trek books Trevor keeps buying, including one on the transition to a post-scarcity society, and the planned segment on looking forward to 2021, which Matty calls "an empty dock." The wrap-up ends with Trevor's projects (a Battlestar Galactica model kit), Among Us, Bridget's Hunt a Killer boxes, online trivia and three Dungeons and Dragons campaigns.

Favorite Episodes

Matty

• "Breaking Down Gates" with Tim Banks
• "Deserted Island DevOps" with a cast of thousands

Trevor

• "DevOpsDays Chicago 2020"

Jessica

• "Don't Worry, Do Care" With Aaron Blohowiak

Bridget

• "Tea and Anarchy" With Alice Goldfuss and Ian Coldwater

Jeff

• "DevOpsDays Chicago 2020"

What happened in 2020?

Trevor

• A year of PM land
• Streaming fun with Shuffleboard (and a little DevOps). Check out the Royal Palms Shuffleboard Shufflinsanity, or look at some shuff.io livestreams.
• Virtual conferencing!
• A doggo (Friday)

Jessica

• Jessitron, LLC
• Cut my hair
• Last Keynote Codebeam March 7&8 last talk with Avdi
• systemsthinking.dev

Bridget

• I planned to travel less. Sorry for causing (points to everything).
• Fewer events but playing with the pre-record format

Jeff

• Got a new gaming table
• Wrote a book! Operations Anti-Patterns, DevOps Solutions

Matty

• Last in-person conference was DevOpsDays NYC
• New job at Red Hat!
• DevOps Party Games
• Took over from Bridget as global chair for devopsdays
• Did "less" speaking this year ("only" 9-10 talks)

90s Sci-Fi Roundup

Joe's Babylon 5 talk at DevOpsDays Madison 2016

Joe's Top Babylon 5 Episodes

• Season 1, ep 3: Born to the Purple, Great Londo episode.
• Season 2, ep 20: The Long, Twilight Struggle. End of the Narn/Centauri War. One of the best FX shots of the whole show. One of the best Londo/G’Kar eps.
• Season 3, ep 20: And the Rock Cried Out, No Hiding Place. More good Londo/G’Kar.
• Season 4, ep 14: Moments of Transition. End of the Minbari Civil War. Good Delenn ep, better Neroon ep.
• Season 5, ep 8 Day of the Dead. Duh.

Joe's Worst Babylon 5 Episodes

• Season 1 ep 10: Believers. Basically Christian Scientists in space. I could have chosen TKO (Bloodsport in Spaaace!) or Infection ("He took a pretty bad hit!"). Believers is super-preachy and it has the pedigree of being written by David Gerrold (wrote the tribbles ep of original recipe Trek)
• Season 2 ep 18: Confessions and Lamentations. A deadly plague threatens the Markab. Another weak Dr. Franklin ep. Also, a deadly plague, read the room B5!
• Season 3 ep 18: Walkabout. Hey, look! It’s Wallace’s mom from Veronica Mars but oh, god, the songs!
• Season 4 ep 10 Racing Mars. Marcus AND Dr. Franklin together on Mars! Nuts and gum, together at last. Honorable mention: The Deconstruction of Falling Stars.
• Season 5 ep 11: Phoenix Rising. Fucking telegoths.

Chili as a Metaphor

Tim has strong views on chili, which for Texas has two ingredients, meat and heat, with no beans and no tomatoes, and uses it as a metaphor. Chili, like DevOps, means different things in different places: is DevOps a job title, a methodology, a culture or a group? The same title comes with different responsibilities at different companies. Matty notes the metaphor breaks down because chili has a definition and DevOps doesn't, and adds that at a bank everyone has the title vice president, which means you're not in leadership.

Standardizing Levels and Interviews

Tim would like tech to have something like the journeyman levels in the trades, so people know where they stand, and Matty adds that electricians have unions and certification boards. Tim says that if we call ourselves engineers, there should be standards, whether a union or a certification. Standardization would help with treatment, pay and inclusion, and lower barriers by making hiring less arbitrary, such as trivia interviews where someone is asked for an inverted binary tree they will never write in the role. Tim also worries about startups expecting junior people to work 60 to 80 hours for no equity.

Matty mentions Kat Cosgrove's All Day DevOps talk on gatekeeping, which showed a junior job posting that described an entire team. Matty says job descriptions get garbled between a hiring manager's nice-to-haves and an engineer who doesn't know how to write job requirements, and recalls keyword-stuffing a résumé for recruiter software, like listing every ProLiant model. Tim still gets Solaris contract offers because the résumé mentions it.

Two Kinds of Gatekeeping

Tim describes two gates: getting in, from résumé screen through interviews to the offer letter, and staying in, with promotions, raises, reviews and projects. Many companies measure inclusion with lagging indicators like hiring numbers, and Tim says to look at retention and promotion. Tim says it's not a pipeline problem: people leave, even if you get them in. A good gauge is an anonymous survey asking whether someone who was LGBTQ would feel comfortable coming out to co-workers and leadership, and the answer has to be an emphatic yes. Matty likens it to an NPS question because it has skin in the game.

Matty compares counting hires to monitoring and sentiment to observability. Tim agrees: observability is insight into what's going on inside, where hiring and retention numbers are "when you get paged at three o'clock in the morning that something is down because it's already broken." Matty adds that you can't put a Nagios agent on your people, and that becoming a people manager is a career change, not a promotion.

Being the First

Matty worries about the difficulty of being the first person from an underrepresented group on a team. Tim says you can't hire a junior person to be the only one and need someone with more experience who can navigate it, and that you have to fix the culture and listen without defending. Tim says "when you have a more inclusive culture, you will have more inclusive hiring practices. Full stop." Matty says a failed first attempt is not a reason to stop, because you have to keep trying.

Tim says the first hire is being asked to do their job plus fixing the culture and being a pioneer, which is three jobs, and so should be paid more, and the worst outcome is finding out you're a token. Tim recalls a company that announced a diversity push and then celebrated progress hiring more white women, and notes "the things that you measure are the behaviors you're going to incur." Matty describes a frozen middle: people who support diversity in principle and then stall when it gets hard.

People in a Pandemic

Tim says tech has often told people the job is typing on a keyboard, when people need to feel they matter as whole people. Tim says that in the pandemic tech is losing women, especially mothers dealing with kids at home, and it's "unforgivable." Matty adds that much of the loss is invisible. Tim says if a hiring manager loses a woman who has to take care of family, "you fucked up," not the person who left.

Matty and Tim say companies have the money. Companies are saving on rent, commuting benefits and parking, so they could offer stipends or a one-off payment, like $1,000 each to 100 people, which is a rounding error for most VCs. Matty argues investors don't have a fiduciary responsibility to make shareholders insanely rich, and suggests that if growth is 300 percent instead of 1,000 percent, that's fine, and to invest in people. Matty ties this to resilience: organizations rebound from unmodeled disruptions like a pandemic through depth of capacity, which comes from people, "because it's banked." Tim says you must put into the granary in good times and take care of people all the time, since they'll leave "the second they can." Matty says the bar is so low that being slightly less bad than everyone else will make a company look like the best in the world.

Tim ends by saying that without insight into what your people are going through, the only signal is a lagging indicator when something goes down. Matty adds that you'll see it a year or year and a half later, wondering where everyone went: "Here's your chance to get it right." Tim is @elchefe on Twitter.

• Texas Chili Parlor in Austin
• Chili John's - the chili place Matt's friend brought him to in California
• Gatekeeping and the DevOps Revolution: We Haven't Always Known Everything - Kat Cosgrove at All Day DevOps 2020
• Lending Privilege - Anjuan Simmons
• levels.fyi/

Shout-out to Matt's friend Marcelo for the link for Chili John's (and for taking Matt there so many years ago)

Deciding to Go Virtual

The organizers took three to four weeks to decide, and Matty says the focus was on outcomes and not on platforms, since "that's so very DevOps," being outcome-driven and not implementation-driven. Matty says engineers default to "how would we do that in Slack?" and argues the wrong question about virtual events is how to replicate the hallway track, since that gets you "augmented reality VR expo booth nonsense." The right question is what you get from the hallway track. Sasha pushed for a virtual event so the community would have somewhere to gather, and for it to be free, with talks watchable on YouTube. They looked at many virtual platforms and said no to all of them. Chris says the worry was making sure participants and speakers felt they got value.

The event was split between an AV team in a studio in Chicago, which recorded and streamed, and a virtual team handling participation, so Matty and Sasha, both in the studio, didn't see how the participant side went.

Audio and Video

Kevin says AV at devopsdays Chicago started out terrible and iterated upward year over year, including live captioning, but this year "we pretty much threw that entire thing out the window." Kevin's priority was quality, and Kevin insisted there be no live remote presenters over Zoom, since internet, audio and lighting are hard to control, and the first minutes of many virtual talks are AV problems. Presenters could record in a studio paid for by the event, with guidelines for two cameras, or self-record using a guide. Trevor wrote most of the self-recording guide, drawing on experience recording the podcast. AV Chicago, the local partner, did all the editing for consistency. After each talk, a live Q&A or fireside chat over Zoom, joined early so lighting and audio could be checked, ran on the main YouTube stream.

Kevin says the studio was in a Chicago music venue Kevin loves, and the AV Chicago team was wonderful. Matty adds that the fireside chats were in front of a fake fireplace that Sasha figured out how to turn on, and that speakers could be in the chat during their own talks, which Matty says gave more riffing opportunity than in most virtual events.

Why Discord

Matty says the plan started with Slack, then Matty wished Slack had video channels like Discord's, and realized Discord was the answer. The event was low-risk since it was free, so they could take a gamble. The main concern was abuse, and Discord offered more granular permissions and moderation than other tools. They used bots so people had to accept the code of conduct to get access. Matty recalls Dr. Richard Cook asking what they'd do if everything went wrong on Discord, and the answer was turning it off, plus recruiting volunteer moderators from across the devopsdays community worldwide. Matty also says they recorded how-to videos and over-rotated on onboarding. Matty is writing a blog post with details.

Kat has done roughly 25 to 30 virtual conferences this year and calls this one of the favorites, since Discord allows friendly, genuine communication with attendees and speakers, is easy to moderate, and "if DEF CON can pull it off with tens of thousands of attendees, than anybody should be able to." Kat says moderating was often facilitating, and once a good question started, a room self-managed for 15 or 20 minutes, feeling as close to an in-person conference with real breakout rooms as Kat has had.

Breakouts Behind the Scenes

Matty didn't call them open spaces, but they followed the spirit, with attendee-suggested topics in text or video rooms. Chris says the proposal channel went unnoticed at first, so moderators reseeded topics, people voted with emojis, and video rooms were capped at 25 users, so a moderator had to be in place first. In the first round, people hopped in and out and nobody turned on cameras, and it took five or six minutes to get people talking, but it got easier. Organizers and moderators had separate channels, Jerry acted as ringmaster for the virtual team, and a read-only FAQ was updated live. Chris didn't hear about any problems getting into Discord, joking "We didn't observe it, therefore it did not occur." Chris found the virtual side more draining than an in-person conference, with constant context switching.

The Speaker Side

Laura recorded the Ignite at home, following Trevor's guide, with a bedsheet taped to a coat rack to reflect light, and needed around 15 takes because of cars passing the window and a hand-off of an inflatable microphone. The Ignite speakers passed inflatable mics from one to the next, and Laura dropped the mic onto a pile of blankets that the dogs then lay on. Matty ordered the mics in sets of five with random colors, and had hoped someone would mess up the hand-off. Laura says watching your own talk while chatting is more nerve-wracking than live, but being in the discussion all day meant nobody missed the hallway track.

Jason recorded in a studio near train tracks in a sketchy area, ended up sitting, and was the last talk of the day, followed by a fireside chat with Matty and the DevOps Yak. Jason says the chat let Jason skip watching the video of the talk, and that Matty and Sasha were exhausted, which Jason says shows that virtual events are a lot of work. Sasha says Zoom-style events are more exhausting because you don't get five minutes chatting with a friend and a coffee. Jason says a polished recording helps with future CFPs, and jokes that the bar is now high for organizing.

The Yak, Deep Thoughts and Stickers

Josh Zimmerman traditionally gives a funny Ignite, and with extra stream space Josh recorded a series of DevOps Deep Thoughts that played through the day. Kevin says sponsors got a three-minute pitch, longer than usual, sandwiched between two Deep Thoughts so people would not skip them. The team also filmed the yak costume against a green screen and Trevor turned about 25 minutes of footage into yak bumpers, which Matty says led many viewers to think the yak was live. Sasha says they were a major improvement over running dry content.

Matty had avatar stickers of speakers and organizers made, designed by Kelly Mahoney, in place of the professional headshots of the previous year. Laura says it was the best speaker gift ever, because it was a drawing of Laura. Jason says it worked because it was something nobody thought to buy. Kat, who had never been to devopsdays Chicago, says the small things going wrong, like shuffling moderators and scrambling for topics, made it feel more like a normal conference than platforms that hide every hiccup, and felt "more normal for a day, which is extremely valuable right now."

Matty ends by reading part of an iTunes review from a nurse who finds the podcast gold, and says reviews like that are why the show has kept going for seven years.

• Matt's blog post about "howto"
• Rich Burroughs’s wrapup post
• https://matty.wtf/yak-wtf
• https://twitter.com/SoSplush
• Love, Yaktually video
• Recording of the event livestream

Containers Are as Secure as the Stack

Ian says container security has to be thought of holistically, because containers share resources with each other and the host, so "your containers are as secure as your stack is," including silicon, operating system, kernel and what runs in them, and defense in depth matters. Alice has never worked on a container team with a dedicated security person, and says security is usually an afterthought, a checkbox before shipping. People often choose containers for security, such as running customers' arbitrary code, while security people say an insecure box means insecure containers, possibly more so with more ports open. Alice warns about false prophets and marketing, and says if you pick containers for security, you need an expert like Ian and must implement what they say.

CVEs and Disclosure

Ian explains a CVE as a taxonomy: someone who finds a vulnerability submits it to the CVE Numbering Authority, it gets a number, and you can look the number up in a large database. The numbers aren't memorable, so people name their vulnerabilities. Ian recently got a first credited CVE with a group, a credential leak in containerd 1.2.x, and mentions an earlier Kubernetes CVE for which friends published a proof of concept that "honked Kubernetes to death."

Alice says when a CVE lands, you need an inventory of the versions you run, since "otherwise, you find out about them on Twitter." Then ask whether you run the affected version, how likely and severe it is on your fleet, and how much of your infrastructure is affected, and set a mitigation deadline. CVEs typically aren't announced until a patch is in the works, and an older version might make patching gnarly, which is one reason to keep doing rolling upgrades.

Ian says responsible disclosure is a matter of some debate. At best you write to the security contact, get a friendly reply, file the CVE through maintainers, who set severity, and agree on a timeline. Sometimes vendors threaten to sue the finder, which is bad behavior and how "you get 0-days dropped on Twitter on you." Ian notes that someone who reports a bug is showing good faith, since they could sell or post it. Alice adds that if you get an email saying someone found a vulnerability on your site, do not respond with threats, because that person is trying to help unless the email continues with demands for payment. Bridget's analogy: "hi neighbor, your window's unlocked."

Below the Software

Alice says the kernel is software too, and has a well-entrenched maintainership that gets patches out, while operations teams are used to patching it. Hardware has its own issues. Alice recalls an F5 announcement a couple of hours before a leap second that some load balancer versions had a vulnerability triggered by it, which meant interrupting an important meeting with executives. Alice also describes the "hotel maid" scenario of a device being placed in a laptop and security researchers scanning machines before and after travel.

Public Life and Personal Security

Bridget asks where they draw boundaries between public work and private life. Alice takes a physical safety mindset, doesn't tag locations, shares restaurant visits only hours after leaving, and has been recognized from Twitter on the street and in restaurants. Alice is a protected voter in Oregon and currently doesn't share an employer on Twitter, because the boundary has been good for mental health. Alice enforces parasocial boundaries, inviting approach at events but not at a restaurant or crossing the street. Bridget posts about work because it fits open source, and says familiarity doesn't mean friendship.

Ian says everyone has their own threat model and comfort level. Ian tells of a single week when a parent at a kid's karate class and the person at a bodega counter both announced they followed Ian on Twitter, and Ian decided to accept being visible. Ian's advice: "err on the side of not being creepy," and "if you know for a fact that you're being creepy, maybe stop." Alice adds that constantly being watched has paid off in professional settings as a kind of trial by fire.

Tea and Geese

Asked for the most delicious tea, Alice says it depends: coffee drinkers may like smoked teas like Lapsang Souchong or malty ones like Assam, people who like fruity flavors might try an oolong, and people who like savory might try a Japanese green like a Fukamushi sencha. Alice's recent favorites include a Weishan Bao Zhong and a Dan Cong oolong that smells like currants. Ian isn't a tea snob but enjoys friends' tea.

The goose thing came from the video game Untitled Goose Game, which Ian saw as an allegory for hacking because the goose chains together innocuous objects to exploit them. Independently, organizers of the Kubernetes Contributor Summit at KubeCon made a goose-themed CTF where GitOps makes a stuffed goose honk, and Ian's keynote was about it. Now "lots of Kubernetes people honk at each other."

Making the Year Better

Bridget is trying to elevate voices other than Bridget's own. Alice says impact depends on bandwidth, and changed the PDX DevOps meetup, a group of about 60, to go online and host topics beyond technology, including ethical organization at a company after the George Floyd protests, resources for protesting safely, stress release and emergency preparedness after wildfires. Ian says that as someone with access to tech community resources, the answer is redistributing money and resources to BIPOC youth doing organizing on the ground. Alice agrees.

Image credit: Tea and Anarchy, modified from Anarchist Revolt

Font: 1403 Vintage Mono Pro by Jeff Kellem

Dependencies Behind Dependencies

Alyssa says the number of packages keeps growing, with npm's count nearly doubling every year, and that most vulnerabilities come from indirect dependencies, not the ones you chose, especially in Java and JavaScript. An example from the report is an 80-line JavaScript app with 7 dependencies that expands to 59 more and turns into 750,000 lines. Matty asks whether to fix this or accept it, and Alyssa says we need awareness and tools because it isn't going away: security has preached "don't roll your own encryption," and reuse was the panacea when Alyssa was a developer. Alyssa raises the software bill of materials, noting an FDA advisory about a popular open source package that medical device makers couldn't answer for, because they didn't know what was in their software, much like Heartbleed.

Jessica says that if you write it yourself you'll still have vulnerabilities, only nobody sends you mail about them. Matty cautions that open source might have been reviewed, not that it was, and Jessica says nobody looks at what Jessica publishes on npm. Alyssa says package health has no generally accepted measure, but popularity, age, active maintenance and accepted PRs give an idea, and popular packages get more scrutiny now and later. Academic researchers, such as a security lab at UC Santa Barbara, report vulnerabilities to Snyk after scanning thousands of projects for patterns.

What Improved

Alyssa says the total number of new vulnerabilities grew more slowly than the year before, with fewer reported in 2019 than in 2018, which Alyssa calls a positive indication but "not ready to say, hey, we're getting better at security." On who's responsible for application security, about 85% said developers both years, but security rose from 23% to 50-55%, and operations went from barely registering to about the same, which Alyssa reads as awareness that DevSecOps needs all three. More organizations review their YAML and JSON and audit production clusters, though 31% said they didn't know or weren't doing anything, and 44% of respondents use Kubernetes.

The Scatter Plot Surprise

New this year was a scatter plot of vulnerabilities reported versus projects impacted. Cross-site scripting had many reports but few projects affected, while prototype pollution and deserialization had few reports but wide impact, including a Lodash vulnerability in 2019. Nothing landed in the upper right quadrant, which Alyssa reads as a sign that big, popular projects have eliminated the common flaws, while newer attack vectors have the big impact.

Official Images Aren't Safe by Default

Alyssa calls this the "stranger danger" story and says it was personal. A blog claimed official Docker Hub images have been scrutinized, but the report found the top 10 official images still had many vulnerabilities, with the Node image off the charts. Alyssa pulled the full Node image with 642 vulnerabilities, versus about 53 with the slim image, and compares it to the old server advice to minimize the operating system. Jessica says the full image is handy for development but production is a different animal, and Matty points out the temptation to go back to the image that works. Alyssa notes Docker has been adding scanning and higher-scrutiny image programs.

SnykCon and Threat Modeling

SnykCon is a virtual conference on October 21-22 with vendor-agnostic talks, keynotes such as Wendy Nather, and a fundraiser for the Bill and Melinda Gates Foundation. Alyssa will do a workshop on threat modeling in DevSecOps.

Alyssa says threat modeling answers "what could possibly go wrong?" Traditionally it's a heavy process of data flow diagrams taking days, fine for waterfall but not sprints. Alyssa's approach adds a continuous-improvement CI: threat model each user story, where someone from the business can guess what attackers would want to steal, expose or deny, and that informs coding, test cases, automated tools and production monitoring. You won't make the system "unhackable," but you get incrementally better. Matty adds that monitoring is testing with a time dimension, and Jessica says talking to domain experts about what it shouldn't do gives a better understanding of what it should. Alyssa points to a Puppet survey finding that collaborative work like threat modeling builds more confidence in security posture than siloed pen tests and scanners, and says it helps prioritization: which vulnerabilities protect the crown jewels, and which are exploitable at all.

Gates Break DevOps

Alyssa has attended no fewer than 30 talks on DevSecOps that put quality gates between stages. Security has to be integrated in each phase, since "If you push motion in the pipeline back to the left with the feedback from a gate, you just broke DevSecOps." Matty adds that people who can't get through a gate figure out how to go around it, and tells of expensive security tools with few licenses that act as gates themselves. Alyssa cites a study where 85% of organizations said they'd pushed known vulnerabilities to production and 54% of those said it was to meet a timeline. So you have to accept that software ships with vulnerabilities and shorten the feedback loop. Jessica adds that gating slows security releases, since "change is on our side."

Alyssa's favorite t-shirt says "Unhackable?" with "Here, hold my beer" beneath, linked from the show notes.

• Snyk's State of Open Source Security report
• SnykCon is coming on Oct 21-22! Register now!
• Guess what you can threat model in devsecops! More about threat modeling in Pushing Left With Tanya Janca
• Alyss's awesome t-shirts

What a Retrospective Is For

Matty defines the topic as what happens after service is restored, however you name it: postmortem, after-action review, retro. Amy frames an incident as an unplanned investment, with people time, software and cloud spend going in, so the question is whether the organization got the most out of it. Rein adds the view that an incident is an encoded message the system is trying to deliver, and while responding you decode just enough to restore service, so skipping the rest wastes the investment. Alex calls retrospectives the most sophisticated end of responding to a ticket, where you fix a problem in the best possible way and not just click close.

Matty says during an incident the goal isn't fixing the problem but restoring service, and that the two halves depend on each other: you can only skip the rabbit holes during response if the organization has a social contract to decode the message afterward. Amy describes different paces of engineering, with incident response at the fastest and a retrospective on a much longer timescale, like the architecture phase. Rein compares this to Kahneman's thinking fast and slow, where fast is about performance, not long-term learning.

Start Before the Meeting

Matty notes the irony of "thinking slow" in a one-hour meeting, which should be a jumping-off point. Alex says you can start the slow learning during an incident with the incident command system, used conceptually, by asking someone to start the incident state document or the retrospective while responders focus on mitigation. Amy adds that assigning the scribe role is also a way to keep a nosy manager busy. Matty cautions with Ron Swanson that you should not half-ass two jobs, and Alex agrees that it takes a practiced organization where everyone knows who is doing what. Amy says if you show up to the meeting and most of the analysis isn't done, the meeting is a waste.

Rein says the idea that learning happens in one hour is a little silly, since learning is happening in a dozen or more brains for days and weeks, and the meeting is for the things only possible with those brains in one room.

Action Items and Their Deadlines

Matty disagrees with a line from the PagerDuty postmortem guide that the most important outcome of the meeting is consensus on action items, though not if action items include questions to investigate and not just Jira tickets. Amy and Alex say that in the real world, follow-ups are the main point for most SREs, because they are the easiest to tie to business value and executives and directors want them. Rein's example is a junior SRE paged five times a week for the same thing, who won't accept "we're going to stop worrying about action items."

Matty warns against SLAs on action items, such as completing everything within two sprints, since people will only agree to items they know they can finish, and an engineer should be able to come back and say the plan changed after looking closer. Amy's workaround is to get follow-ups into a prioritization process and then let go, turning choices over to the engineering and product teams. Matty says the people who prioritize work should be in the retrospective. Alex says many organizations lack buy-in, and Matty replies that nobody has it everywhere and change has to happen in both directions. Amy and Matty also note that when people manage the numbers they will game them, the Pareto-inefficient Nash equilibrium problem: people work to the numbers you give them.

Narrative and Timelines

Alex's goal is always to tell a story, since "we're storytellers," and finds a timestamp table less useful than a narrative of what happened first and next. Amy disagrees on timelines: the timeline is the outline before writing the narrative and common ground with readers, especially for complicated incidents. Matty calls the timeline supporting information, and notes that not every line in the Slack channel is worth including. Amy supports cranking out shallow incident reports cheaply and ubiquitously. Rein says there is no single timeline, with 12 people in a channel there are 12, and asking people to compare theirs is where the richness comes from. Matty adds that stories are more memorable than log entries, and that people don't read retros from other teams, which are the ones they most should.

Matty says one of the biggest anti-patterns is only doing postmortems for Sev 1s, and Alex has been on teams where every page got a retrospective, even if it meant deleting the alert. Amy notes organizations that aren't ready to hear the reports, where small insurrections matter more, and Matty argues that writing short reports helps engineers learn to speak in business value.

Making It Easier to Try

Rein's rule is to ask what would have to happen to make a change easy, and suggests a Goldilocks zone between big, scary incidents and small, boring ones, and getting an organization used to trying things first, which can take six months. Alex has had success with facilitator rotations where people sit in on teams on the opposite side of the company, and Matty adds that a good facilitator isn't invested in the content, and that management facilitating is a problem. Matty says to stack the deck for change by starting with people who are interested, since they'll sand the rough edges.

What They Changed Their Minds About

Alex used to believe timelines were crucial, and now thinks the narrative is the important part, though they remain good starting points. Rein says timelines are what let you reinstantiate context in cognitive interviewing ("it was Friday, it's 9 PM"). Rein also no longer believes the hour in the meeting is the most important part, which Amy also said, and Amy no longer favors an independent meeting per incident, preferring the weekly incident review run at GitHub, where everyone came for a cadence of caring about incidents and people shared what happened in narrative form, with follow-up done out of band.

Better Questions

Alex suggests templates with prompts such as "where do we get lucky?", who happened to be online, and how to make sure we don't have to be lucky. Matty adds a prompt asking what questions aren't on the template. Rein suggests asking how priorities should change as a result of the incident, since action items tell people what to do without why they should care. To choose which incidents to study deeply, Rein suggests looking for cues like confusion, surprise or frustration, and in the mundane looking for the surprising. Matty's closing advice: find the mundane in the interesting and the interesting in the mundane, write good narratives, and "ask why 10 times, because if 5 are good, 10 must be twice as good."

Alex's book - Implementing Service Level Objectives: A Practical Guide to SLIs, SLOs, and Error Budgets

Aaron discusses the three big projects he's working on. The first is cost efficiency, or, as Netflix thinks of it, "cost compression."

Jessica: "Oh, so you have to achieve cost compression without telling engineers not to spend money."

Aaron: "If you don't believe you can predict very well, then what you should do instead is get really good at reacting."

The panel discusses the difference between autonomy and agency.

The panel talks about the utility of data dashboards.

Aaron: "The dashboards are more like cost debugging ultilities."

Access Isolation

Aaron talks about his second big project, "a unified strategy for access isolation."

Jessica: "You want the cells to have access to the other cells in the same muscle tissue, but if they need a nerve ending they have to say so!"

Regional Growth and Availability

Aaron explains the third big project he's working on, "Netflix's regional growth and high availability story."

Aaron talks about how Netflix produces original content all over the world, and the coordination required to serve all the computation needs of the various projects in different places.

Shared Reading List

Ecology, the Ascendent Perspective

Thinking in Systems

Aaron's blog post, The Sufficiently Smart Engineer

What a Service Mesh Is

Michelle gives the textbook definition: a dedicated layer of infrastructure that helps you manage, secure and observe service-to-service communication. The problem is that in highly dynamic environments, where pod IPs change as things come and go, networking needs to be more dynamic as well: traffic encryption, access control, which service can talk to which, traffic shifting from one version of an application to another, and observability metrics.

Delyan frames it from the point of view of a CTO who wants observability, security and traffic management but has busy engineers: run an install command and "you get all those extra features." In the past this came from libraries tightly coupled to a language, such as Twitter's Finagle, Netflix's Hystrix and Google's Stubby. A service mesh instead bundles a sidecar reverse proxy with your workload and pipes traffic through it.

Is It a Man in the Middle?

Bridget asks whether intercepting traffic is a man-in-the-middle attack as a service. Michelle says it's meant to prevent them. A common requirement, especially in enterprise and government settings, is mTLS, mutual TLS, where both client and server prove who they are, and it's nice not to handle that in code. Delyan lists three components: the reverse proxy sidecar, a certificate used to encrypt and decrypt traffic, and the control plane that tells the sidecar what to do. All three are open source, so you can review the code, and traffic leaving the sidecar is encrypted and flows only to services explicitly permitted.

Do You Need One?

Michelle says "You don't need a service mesh unless you need those things," and it suits environments with lots of microservices and specialized teams, such as Twitter or Lyft, which Kubernetes and containers made possible. Delyan adds that operators get lost in Kubernetes complexity, and a mesh controls east-west traffic between namespaces, helps with zero-trust networking among teams that don't trust each other, and gives auditability of which services exist and who talks to whom. Michelle explains that east-west is service-to-service traffic, and north-south is external traffic coming into the cluster. The data plane is the set of proxies carrying user traffic, and the control plane is the source of truth that configures proxies and manages certificates. Michelle notes the sidecar approach isn't the only one, since some meshes run a proxy per node. Delyan says the data plane must run nonstop with minimal latency because customer data flows through it, while the control plane has more flexibility for upgrades.

SMI and Open Service Mesh

Michelle says SMI, the Service Mesh Interface, is a set of APIs representing the most common functionality people want from a mesh, so that tools can build against a consistent API regardless of provider. Delyan compares it to a shared language invented by college friends from different Eastern European countries: you can try mesh A, then mesh B, without changing your policies, because SMI stays in the cluster and only the data plane swaps. SMI describes the topology, the control plane ingests it and tells the proxies what to do.

Open Service Mesh is a lightweight, Envoy-based, Kubernetes-native, SMI-compliant mesh. Michelle says SMI covers mTLS, traffic shifting, access control and metrics, and OSM chose Envoy for community momentum and WebAssembly extensions. Delyan says the goals are source that's simple to understand and contribute to, effortless to install, painless to troubleshoot and easy to configure with SMI.

The design philosophy is "no cliffs." Delyan uses the analogy of service meshes as motorcycles in a garage, each fine-tuned for different uses, so there's always room for one more. SMI doesn't cover everything in the proxy, such as circuit breaking and back pressure, so when you hit that cliff there's a dirt road instead: switch to XDS, Envoy's own configuration protocol, which is harder but lets you fine-tune the proxies.

Getting Involved

Michelle points to the GitHub repo's install guide and demo, which work on a local cluster such as Kind or minikube, and says feedback via GitHub issues or Slack is welcome. Bridget adds that a newcomer reporting that the walkthrough did not work issue is valuable because the authors can't see what they've assumed. Delyan hopes people enjoy reading the code and rename variables to make it their own. Michelle is @michellenoorali on Twitter, and Delyan is @DelyanRaychev.

• SMI
• Open Service Mesh

OSM logo art credit: @flynnduism

What Developer Experience Means

Stephanie sees developer experience as a subset of user experience design: the experience developers have when using your product, which for Stephanie is a web browser. Edge Developer Experiences brings together the DevTools team, the web apps and PWA team, and the Ecosystem team, which Stephanie is on, after the move to Chromium. Stephanie explains that the Ecosystem team works like developer relations, helping other teams scale up and get good documentation out, and works closely with the HTML platform team on standards and features such as CSS requests. Stephanie describes WebView2 as a way to embed HTML, CSS and JavaScript in native applications, and has seen it demoed in Excel.

Matty compares it to the days of throwing things in Notepad and putting a green border on every div, and says tools like Firebug were revolutionary.

Ask Developers What They Want

Stephanie says the biggest thing is that the Edge team now comes "from a place of humbleness" and asks developers what they want. Stephanie understands that building a browser used to be more closed off, with an assumption that browser makers are web developers and so know what developers want, which isn't true. Focusing on your developers is "the key to building a great developer experience," because "if you can build some cool feature, but if no one uses it," it doesn't matter. Matty adds the danger that when you think you're close to your users, you're really orthogonal to them, and skip the research.

Documentation, Support and Crisis Design

For products where developer support is a sidebar, Stephanie says two things to bake in are documentation and support, which "can make or break a developer's experience." Documentation tends to be left to the end, with an assumption about what users know. Stephanie calls out some popular static site generators where debugging leads to an endless loop of docs with nobody to contact. Stephanie recalls an Eric Meyer talk about designing for users in crisis, the idea being that an experience a user in crisis can navigate will work for everyone, which applies to developers whose site has broken. Stephanie's advice is to keep iterating on documentation and stay open to feedback.

Matty ties it to empathy: things make sense to you because it was your idea, and swagger output isn't documentation, since examples matter and "I am coming to solve a problem." Matty adds that if someone keeps asking how to do the same thing, "that's on you," and compares it to learning to drive a manual so you can drive an automatic.

The Web We Want

Stephanie spends about 60 percent of the time on The Web We Want, a cross-browser and standards initiative that started on the Edge team but isn't Edge-specific. It's a forum for developers to say what's missing from the web platform, asking what they'd change if they could wave a magic wand. So far it's had about 150 valid feature requests or gaps, and "developers are really hungry to give their feedback." HTML controls is one request that matched work already underway. Some submissions are things standards groups decided years ago weren't worth the investment, and now there's data from developers.

Matty links it to the saying that in open source "no is temporary, yes is forever," and that for no to be temporary you have to keep looking. Matty adds a change-management tip: reassure people that with the information they had, they made the right decision, and now things are different.

Design Skills and Tracking the Work

Stephanie says "at my core, I am a designer," solving problems and looking at the whole developer experience, such as what a developer sees when arriving at the website. Stephanie gave design feedback on the Grid tooling going into Chromium, since a designer debugs layout differently from someone who only develops.

Stephanie says the team tracks engineering work in Azure DevOps, and that Web We Want submissions and problems extracted from interviews are hard to track there, because it isn't an engineering task you can give a number of dev days, and there's a "bucket of wants." Stephanie doesn't have a solution, noting the tool was built for dev work. Matty says you'd have the same problem in Jira, and it's the classic DevRel problem of tying work to value. Stephanie adds that features in DevTools aren't viewed as done when they ship, since usage and feedback continue to drive iteration, and Matty says organizations' measures don't map to continuous improvement.

Empathy and History

Stephanie's steady message is empathy: talk to a subset of your users about their pain points and what they like, and "embracing your empathy and shed your assumptions." Stephanie likes telling stories about history, and in the HTML Controls talk dug into a 1994 or 1995 specification. Stephanie says an unresolved developer complaint can linger for years, and calls Internet Explorer a great example. Stephanie is speaking about HTML controls at FrontCon in Latvia the next month, and has a YouTube channel with the February version.

• The Web We Want
• Webhint tool
• Designing For Crisis - Eric Meyer talk
• FrontCon - upcoming speaking appearance for Stephanie
• Stephanie's current and past talks
• Stephanie's talk on web controls
• Go to Stephanie's YouTube channel for past talks!

How It Happened

Austin says it happened because Austin and Katy were bored. With COVID, they started Twitch streams, and Katy's idea was a Friday stream about something other than technology, which began with Animal Crossing. Austin joked on Twitter about building a trade show booth in Animal Crossing, someone suggested a conference, and with April 1st the next day Austin put up a web page, planning to call it an April Fool's joke if nobody cared. There were a hundred signups in a day for an event with nothing behind it, and Katy remembers a message saying "I made a joke, but now it's real." Austin calls that the hallmark of a good idea.

Austin says the event was built on the belief that tech events should be accessible in a broad sense: closed captioning, speakers who don't all look alike, and being findable in unexpected places, like Austin's own chance encounters. Austin says conferences like KubeCon or re:Invent are marketing and sales events where learning is a side effect, and Austin was inspired by devopsdays to make something that felt like that but online. Twitch made it easy for people to jump in.

The Keynote

Ian's idea had been a weekly Zoom call for Kubernetes folks who missed the contributor summit turning into an informal meetup in Animal Crossing, not a whole conference. Austin thought bigger, and Ian then got picked for the keynote. Ian had planned a script on the in-game tablet with reactions and costume changes, which fell apart immediately once Ian tried to talk on Zoom, drive slides and use in-game reactions at once. What survived was the tone: warm, positive and welcoming. Ian's keynote was about DevOps and security working together with empathy, and Ian says the reactions and claps from the audience felt more real and supportive than talking into the void on Zoom. Matty adds that the speakers sat in the same Zoom all day, muted, which replicated some connection, and says other organizers should not dismiss this event.

Dave describes it as "the best of the Internet," reminiscent of an old forum, and coordinated real-life and in-game outfits, only to learn the video wouldn't be shared. Dave and Matty both recall long pauses while hunting for the right reaction, and Matty got stuck on one reaction for three minutes without noticing.

Why the Community Worked

Kat says it felt like magic and doesn't know how to recreate it, since virtual expo halls with corporate-branded avatars aren't as intimate as visiting someone's island, where they spent hours and millions of bells on a little movie theater. Kat calls it "the most pure thing I have seen in tech in a long time." A participant who refers to "Katy and I" says much of it was "me getting out of the way": they gave people a space and watch parties came from attendees. Ian pushes back that Austin and Katy did a lot of active work to make the space inclusive, wholesome and accessible, and compares it to open source communities that act with care. Katy says setting that tone attracted the right people and led volunteers to offer to moderate.

Mia says being genuine worked, but it may not be replicable, and that the tone set from the first talk mattered since Twitch chat can get mean. Mia has played Animal Crossing since 2004, and says friends and family watched and finally understood the job. Austin says about 100,000 viewers in total, including many people who weren't Animal Crossing fans or in tech, and at one point the stream ranked third in concurrent Animal Crossing streams. Ian notes a lot of security folks attended who had never been exposed to DevOps culture.

Preparation and Speaker Camaraderie

Kat, intimidated by the speaker list, bought Animal Crossing and wrote the talk two weeks ahead, unusual for someone who usually wings it, and showed up wearing a million-bell crown, having got lucky on the turnip market. Jacquie decided to submit after a team discussion, then got pulled into a three-day hackathon building a videogame, and proposed a talk about building a videogame from inside another videogame. Jacquie says the overlap between DevOps and Animal Crossing is building a space that empowers and welcomes people. Adrienne asked what in Animal Crossing could make awesome parallels and set out to write a CFP too creative to turn down. Katy says there were no bad CFPs and they kept adding talks, and Austin says they did add two more.

Matty notes that Austin set up a Discord before the event, which gave speakers a way to trade slides and titles, something like a speaker dinner. Aaron started with about 60 minutes of content for a 25-minute slot, and calls single track conferences powerful because speakers who see the talks before theirs can tie themselves together and say "go listen to Katy talk about that." Matty says the event changed Matty's mind about prerecorded talks. A participant who works on a Twitch channel for a company's developer content argues that Twitch viewers expect a messier, live experience and that virtual conferences work better if you treat them like you're flying out.

Next Time

Austin wants another big one next year, with shorter Animal Crossing sessions over the summer and possibly other games, joking about Fortnite. Austin says 90 percent of what people saw was done the week before, and that Austin learned enough of a vector illustration tool to make rounded curves, which Matty calls the best DevOps metaphor: learning from doing. Keep an eye on desertedislanddevops.com. Matty plugs Irreverent DevOps Party Games, a live-streamed DevOps game show on Twitch on Tuesday, June 2nd at 8:00 PM Central, with Kat as an early contestant.

Conference content

• Videos of talks on YouTube
• Deserted Island DevOps Postmortem
• Shoutout to Tori Chu who both spoke and did the amazing in-game artwork and swag!

Recaps

• Deserted Island DevOps Wrapup (FireHydrant Blog)
• Deserted Island DevOps Recap (Blameless)

Press coverage

• RedMonk article
• VentureBeat article
• TechCrunch article
• Vice article
• TechRepublic article

How Security Chaos Engineering Started

At UnitedHealth Group, Aaron was chief security architect and helped lead the DevOps transformation. The company hired its first SRE, who described chaos engineering, proactively breaking parts of a system, and it blew Aaron's mind, because Aaron had never seen the system and its security as separate things. The team decided that control validation made sense: you build security measures into a system with a design in mind, and need a way of continuously verifying they work as intended. Aaron was also frustrated as chief security architect that a data architect and a solutions architect would bring different diagrams of the same system, and wanted a way that wasn't subjective "to ask the computer a question." Does the firewall fire when this condition occurs? Does configuration management catch these misconfigurations?

Aaron's short definition is a proactive methodology for understanding an inherent failure within a system before it manifests as pain, for customers or for engineers. Jessica says the key is forming a hypothesis about how the system works in some non-optimal condition and asking the real system.

What Chaos Monkey Was For

Aaron says Chaos Monkey began during Netflix's move from DVDs in the mail to streaming, in 2008, when AMIs were disappearing in AWS and causing outages. Netflix had no chief architect to mandate anything, so it designed services to be resilient, and Chaos Monkey would pseudorandomly take down one during business hours. Aaron says that puts a well-defined problem in front of an engineer, and "when you put a well-defined problem in front of an engineer, they solve it." Jessica says it turns "works on my machine" into a reproducible test.

Matty adds that chaos is about testing a hypothesis that everything will be fine, not one where everything goes to hell, and quotes Netflix's line about running it in the middle of a business day in a carefully monitored environment with engineers standing by. Everybody knows the experiment is happening, and when things look squirrelly it's done, so if the key business metric heads south, pull the plug.

What a Security Experiment Looks Like

Aaron says most security experiments focus on accidents and mistakes, the low-hanging fruit: a weak password, ports open that shouldn't be, too much access. With 680 accounts and 200 services with conflicting IAM policies it's easy to miss a misconfiguration, so they proactively introduce these mistakes to build confidence that the tools catch them. Jessica's summary: engineers aren't perfect, so stop asking them to be, notice when they're not, and let them learn.

Aaron's example is the open source tool ChaoSlingr, written at UnitedHealth Group, whose original name was a poop-themed joke that kept a side project fun. It had three functions, a generator, a slinger and a tracker, written in Python on AWS Lambda, with opt-in and opt-out tags. The main experiment opened an unauthorized port in AWS security groups, on the assumption that the firewall would immediately block it. They found the firewall detected it only about 60 percent of the time, due to configuration drift between their non-commercial and commercial AWS environments. The cloud-native configuration management tool, which they weren't paying extra for, caught it every time. Both tools sent log data to the security operations center, but the operators couldn't tell which AWS account and instance the alert came from, which could take hours to work out, so they added metadata to the alerts. Aaron says "Nobody's freaking out" and they learned all of it without customer pain.

Aaron's boss, the CIO, said the tool keeps the incident team sharp by testing the tools, people, skills and runbooks. Matty adds that business hours are the best time to have an outage, since everyone is available, and practice makes incident response normal. Jessica compares it to unit tests, which give you privacy on your own computer, while chaos tests give you privacy within the company.

Logging, Root Cause and Who Security Is

Aaron says there is no software security logging anywhere, and that log events must be written by a software engineer and need to make sense to a human. Aaron also says "Root cause is a fallacy," and Jessica adds there are many necessary conditions, any of which could be called the root cause. Aaron says security is always an engineering problem, and that the book's audience is about 70-30 security people, trying to bring them toward the software engineering and SRE communities.

Matty says security has come from two directions, business risk and controls in the 90s and engineering now, and that zero trust replaces the fence with locking your door. Aaron wants security in the value chain, and says DevOps helped. At UnitedHealth Group, Aaron taught over a thousand security people to write Python, not to make them engineers but to build empathy, and Aaron figures 15 to 20 percent wrote interesting scripts. Matty says ops and security are like a corporate lawyer, known only when something goes wrong, yet security and reliability are aspects of quality. Aaron also says security spending takes about 30 percent of project cost in unregulated environments and 40 in regulated ones, and mapping an experiment to the control it verifies gives "free compliance." Matty says audits are often theater and an automated trail is better than "a bunch of information that a human being typed in," and Jessica's version is "That is not an audit trail. That's a blame trail."

What Makes It Hard

Aaron says security chaos engineering is only about three and a half years old, there are few open source tools, and ChaoSlingr is somewhat deprecated since Aaron left UnitedHealth Group. Others are writing their own Python and bash scripts to inject failures, mostly for cloud and container security experiments. Aaron adds that one of the better tools is a Java one from a person in Berlin that hadn't been open sourced.

Aaron is @aaronrinehart on Twitter, and there's a chance to win a printed copy of the O'Reilly book through the show notes.

• Last time Aaron was on ADO
• ChaoSlinger
• Enter to win a free copy of the upcoming Security Chaos Engineering O'Reilly book

Where Helm Came From

Matt Butcher's one-phrase description is that Helm is the package manager for Kubernetes, and a chart is a package Helm can install into a cluster. The idea came out of a hackathon project at the company with Karen and a couple of other engineers, from wanting the package-manager experience for people starting out on Kubernetes. It's now "I don't know what, 1.8 million downloads a month or something like that."

Karen didn't expect it to get this big, since the company was still scrappy, and recalls helping debut Helm at the first KubeCon with a turnkey booth and socks, not long after the hackathon. Matt Butcher says the booth was about 6 feet by 8 feet and KubeCon had a couple hundred people.

The Charts Repository

Matt Farina joined about two and a half years earlier. Farina co-chairs SIG Apps, which oversaw Helm when it was a Kubernetes subproject, and started contributing to the charts repository, a community-curated collection of packages such as MySQL and MariaDB, accessible out of the box in Helm 2. Matt Butcher says they expected 12 to 30 charts at most, and then the pull requests piled up. Farina saw a painful manual process, learned from an amazing maintainer whose reviews everyone trusted, and automated it: linting charts and pull requests, then testing installs in live clusters. That became a standalone tool, Chart Testing, now available to people who self-host repositories.

Matt Butcher describes four phases: the Wild West, design patterns and a best-practices document written by Farina and maintainers from Bitnami, automation, and tooling general enough for anyone's chart repository. Farina adds that charts grew from simple replacements to logic and design patterns, like specifying a URL to expose an application and having everything around it created.

From Scrappy to CNCF

Karen organized the first Helm Summit shortly after the acquisition and before Helm joined the CNCF, and calls it scrappy and modest. Afterward, with CNCF support, logistics like the schedule and CFP were offloaded, so they could keep the conference technical, not salesy or flashy. Farina calls the first Helm Summit one of the favorite conferences Farina has been to, with different track styles, roundtable time and an intimate setting, in contrast to CNCF conferences with well over 10,000 people. Karen says they deliberately didn't scale the second one to 1,000.

The Issue Queue

Bridget asks how to connect with a community that wants many things. Matt Butcher says the early community was small and eager, so they could pair on Zoom or Slack with people having problems. Now it feels like "the Time to Make the Doughnuts commercial," and early issue interactions are almost robotic, asking people to fill out the template. Butcher says the biggest emotional challenge is treating each issue as filed by a person with needs, "often filing the issue out of frustration, because they don't file issues when they're happy with something."

Farina says the way to keep focus is to say what Helm does and doesn't do: if someone has another idea, suggest a Helm plugin, or wrapping Helm, and Helm will list related projects, since it's "not a junk drawer." Generic Kubernetes questions also land in the Helm queue. They assign someone each week to triage, and Farina's best answer is better documentation and pointing people to it. Bridget agrees and mentions having sent a couple of doc updates.

CNCF Benefits and Virtual Connection

Karen says CNCF brings webinars, conference support, and project pavilions, such as a Helm-dedicated booth at the last KubeCon in San Diego, a neutral space for a project built by many companies. Matt Butcher says the pavilion and Helm Summits are the opposite of the issue queue, with time set aside to talk to people as people. Farina says some people just came by to say thank you, which you don't see in issue queues, and that Karen organized maintainers, more than 20 from more than 10 companies, to staff the pavilion.

Bridget proposes a drop-in "Helm happy hour," distinct from the maintainer call, to replicate serendipity. Karen says it would remind people why they work on this. Farina says virtual conferences make speakers isolated, while a happy hour would be two-way, since "a community is a lot of people, not some people who know lecturing others."

Phippy and Friends

Bridget asks which Phippy and Friends character each identifies with. Karen, who Butcher says was the brainchild behind the visuals, picks Zee, who is full of questions. Matt Butcher picks the pods that carry things around until they die, and says Captain Kube came from the owl, a favorite animal of Butcher's daughter, and that Phippy was the answer to the daughter asking what Butcher does. Farina picks Phippy, for a secret PHP past and because giraffes are a favorite of one of Farina's daughters, and Butcher and Farina first worked together doing Drupal. Bridget picks Goldie, tied to the University of Minnesota's Goldie Gopher mascot and the Go community.

Where to Go

Matt Butcher points to helm.sh and its accessibility push for documentation, where you can learn it and re-express it in other languages. Farina points to hub.helm.sh for charts, and Karen to the Helm Twitter account and the helm.sh blog.

• Helm.sh
• Celebrating Helm's CNCF Graduation
• CNCF announces Helm graduation
• An Introduction to Helm - Matt Farina & Josh Dolitsky
• Phippy and Friends
• Keynote: Phippy Goes to the Zoo: A Kubernetes Story - Matt Butcher & Karen Chu - KubeCon North America 2018
• Seven Hard Truths About Open Source Community
• Helm chart testing
• Helm hub
• Helm twitter

Helm logo art credit: @flynnduism

Banner image: @bridgetkromhout

What a Cloud Economist Does

A guest explains that the title was made up because they're two words nobody can define, then found out other people use it, including someone with a PhD in cloud economics, which led to a choice between owning up and teaming up. What the Duckbill Group does, in a guest's words, is look at companies' AWS bills, "because those tend to be the big ones," and help them become smaller and less terrifying. A guest's example of a typical surprise is an EMR cluster that fails to start but doesn't turn off the old one, and with no idempotence check spawns a new one every run, so "you're not building the cloud for what you use, rather for what you forget to turn off."

Matty asks why a guest who had been a cloud consultant joined Duckbill. One guest says that while consulting for a couple of years the lesson was to know only a little more than your first customer, and that the two kept saying they should do something together. The move came when projects finished early and the company was looking for people, and they slid into the CEO's DMs, a message that sat unseen for a while because of a Tweetbot bug with group DMs. As the economy got questionable, reducing spend seemed more important, since it can be the difference between laying off engineers and turning off servers nobody remembered.

What the Customer Says Versus the Pain

A guest says that what customers say and what actually hurts aren't aligned. Someone in finance sees a bill that looks like a phone number, the concern passes through about five levels of corporate telephone, and the real pain is that it's too difficult to figure out what the cost drivers are and allocate them: "understanding, optimizing, and predicting it." Now, with a recession-style pandemic event, customers who say "we're here to save money" mean it.

Jessica notes that DevOps was supposed to give people feedback loops and cloud took that away for engineers who can't see the bill. A guest says data centers had the problem too, buried in multi-year cycles, and that you can still do financial hijinks in the cloud. Another guest says the number of pages in a large bill can be in the hundreds, and mentions a bug found in AWS data transfer pricing where it's cheaper to transfer data between us-east-1 and us-east-2 than between availability zones. Both guests tell stories about tiny charges, one of a 22-cent charge running for years and the other of spending about 5 hours to delete a 2-cent Glacier vault, to which Matty replies that they have that 22-cent charge too.

Tagging and Enforcement

Asked about misconceptions, one guest says to tag your cloud usage, thinking about how your company makes money, since "current you is going to have the CFO roll over to you one day and say, what is our cost of goods sold?" The harder part is assuming users won't follow the policy and enforcing it, and the guest's approach is to delete untagged resources, which the guest calls the scorched earth approach, with the gentler alternative being permissions and security rules, "but no one understands IAM." A guest describes customers fixated on the wrong thing, like a company building tooling to cut its dev environment spend when development was 3% of the bill, and says an unbiased third party helps by avoiding internal narratives about the bill. Another points to Terraform plugins that estimate cost, and worries about Kubernetes, where containers make it unclear what's underneath and how to tag it.

Matty adds that a dollar amount without context means nothing, such as a $500 a month button. A guest adds that attributing cost to teams or users also fails without context, as when accounting asks who Jenkins is, and a data science user costs a king's ransom because that's what they do.

First Steps and Over-Specing

For the one thing to do first, a guest says to turn on the AWS billing reports, which aren't on by default and deliver data to S3, and since some take weeks to produce useful insights, do it on day one. Someone should own the Amazon bill, since "someone should have a number on their head in some way." Matty notes the old pitch was to move from CapEx to OpEx, and a guest says most people are still over-specing, picking an instance, hearing "it's slow," doubling it with no metrics, and never going back. One guest tells of quietly moving developers' unused workloads to a smaller instance type and no one noticing.

Asked if Lambda will fix this, a guest says Lambda solves a different problem, and jokes that many AWS blog posts conclude with "fix it your damn self" with a Lambda function.

Worst Names in AWS

Matty asks for the worst-named AWS thing. One guest says Snowball, and another says the many Systems Manager services and invents Systems Manager Cost Manager in the moment, then finds AWS Cost Categories was announced that day. A guest thinks any service starting with the word Simple sends the message that it's easy. For the worst name in cloud, a guest says Azure DevOps, because a hiring manager junk-piled a resume for listing it as if it were a skill. Jessica says it should have been called Arrested DevOps. Matty adds that you can't buy DevOps, "but I sure as hell can sell it to you."

Pandemic Bills

The pandemic brought new work. Customers made multi-year commitments assuming spend would rise forever, and now ask how to deal with commitments they may not meet. Traffic is skyrocketing for some and falling for others, but bills don't fall as much, because people "misunderstood auto-scaling to mean it only ever scales up." A guest adds that billing systems run on at least an 8-hour consistency model, so you don't learn what an expensive change cost until later. The guests close by saying that companies still paying retail prices should negotiate, since no one really pays retail, and Jessica's line on elasticity is that "the definition of elastic is not that it stretches, it's that it snaps back after it stretches, sometimes with lawsuits."

What WebAssembly Is

Taylor says WebAssembly is a compiled language invented for the web, running in a complete sandbox, with modules importable into JavaScript in a browser. WASI, the WebAssembly System Interface from the Mozilla Foundation, lets WebAssembly modules run anywhere and not just in a browser. Brian says it's a binary format that executes anywhere there's an interpreter, so the same file works on a Linux AMD server and an ARM32 Raspberry Pi, the "write once, run anywhere promise Java brought us so long ago and we all laughed at."

On security, Brian says execution is entirely sandboxed and memory has to be explicitly exposed in or pulled out, so "unless there's a bug in the implementation of your WebAssembly host, it's completely secure." Taylor adds you can only do what you explicitly expose to the runtime.

What Krustlet Does

Brian says Krustlet is an implementation of the kubelet specification, written in Rust, that executes WebAssembly instead of Docker containers. It takes a pod spec, fetches the WebAssembly file and runs it, with no containers involved, so workloads are more secure, though the Kubernetes installation is only as secure as you made it. Taylor says you don't have to worry about AppArmor, SELinux or a Linux file system in a Wasm pod, which cuts the attack surface and lets platform builders worry less about Linux permissions.

Brian says WebAssembly on the server is a new concept for most people, and the cost angle is big, since the same files run on a 96-core ARM processor that is more energy efficient than an AMD processor. Taylor says everything runs as a thread in one process, so idle work parks its thread, unlike containerd's shim, a separate process per container, which helps on edge devices with a gig of RAM. Brian adds that WebAssembly is designed to be interpreted as a stream, so you don't load the whole file to start and the memory footprint is lower. Brian doesn't expect it to stop Internet of Things vendors from making insecure configurations.

Two Runtimes and an Evil Capability

Krustlet has two execution runtimes. WASI currently has no networking, so pods can't listen on a port. The other is waSCC, WebAssembly Secure Capabilities, created by folks at Capital One, which uses RPC between host and module and so can do network calls. Capabilities, such as logging, key-value storage and databases, are configured once on the host, and modules just request a key-value store. Brian says you can hot-swap a capability, such as Consul to Redis, without the actors knowing.

Brian describes building "95% of an evil thing" that afternoon, a capability called Shell that lets the sandboxed module make an RPC call to the host and run a shell command as the host process, such as ls or rm -rf. It shows capabilities are "just as smart as you make them." Taylor finds it terrifying but a demonstration of flexibility, and notes WASI is very new, with only two or three languages having strong support. Brian says this is a least common denominator interface, so you won't get every feature of each provider, similar to the Service Mesh Interface spec. WASI's specification work is done through the Bytecode Alliance.

Is It Ready?

Taylor says the repo has construction-sign warnings that it's not ready for production, since a provider lacks networking and init containers and volumes are missing, though basic pods run. They want people trying it for feedback. Outside Krustlet, Taylor says most major websites use WebAssembly in the browser, and gives Autodesk's web AutoCAD as an example, while server-side WASI has few production examples. Brian names edge providers Cloudflare and Fastly, which let you upload WebAssembly to run on the edge, and says Brian's own website runs on WebAssembly in Cloudflare.

Ways to Get Involved

Taylor suggests a demo with one Krustlet running the Wasm provider and another running the WASI provider, doing HTTP-triggered job processing, or serving a low-traffic API or web page from a Wasm module. Brian wants to turn a Go-based Raspberry Pi controller for a barbecue pit, which turns a fan on and off, into a WebAssembly module with a smaller memory footprint. Brian adds that the Krustlet process can run on anything from a Linux server to a micro:bit, extending a cluster across the globe, and Bridget reminds listeners the devices should be theirs. Krustlet is a virtual kubelet in the sense that it tells the control plane it's a kubelet, and differs in being written in Rust and not Go.

Why Rust

Taylor says the reasons are that Rust has some of the best WASI support and most of the related projects are in Rust, and that its compile-time safety guarantees around how long data lives prevented bugs. Brian says there's an opportunity cost, since Rust is harder to read and start with, and it took more than a month, closer to two, before Brian wrote useful code, leaning on Taylor. Once past the learning curve, you appreciate how much the compiler does. Taylor disagrees that Go is easier to follow as projects grow, and likes Rust's match blocks and error handling. Taylor, a core maintainer of Helm, says updating Kubernetes libraries in Go has been "an absolute nightmare," and calls Cargo "an absolute gem," with conditional compilation through features, making the dependency story "infinitely better than Go."

Brian streams live coding of Krustlet on Twitch, and Taylor jokes it's good for anyone with imposter syndrome. Taylor invites contributors with experience on EKS, GKE, DigitalOcean, IBM and other clouds, and stresses it's not meant to be a Microsoft-focused product.

• WebAssembly

• WASI

• Bytecode Alliance

• Krustlet

• Kubernetes Rust Kubelet on GitHub

• Introducing Krustlet, the WebAssembly Kubelet by Matt Fisher

• Kubernetes and waSCC by Brian Ketelsen

• waSCC capability that will make your ops folks cry - GitHub link to evil project (don't install this)

• Kubernetes: A Rusty Friendship - Taylor Thomas

• WebAssembly meets Kubernetes with Krustlet by Ralph Squillace

Guests Dan Bergh Johnsson, Daniel Deogun, and Daniel Sawano join host Jessica Kerr to discuss their book Secure by Design.

Daniel: “There’s a lot of good designs which come naturally to us as programmers but which has the interesting side effect that they also prevent security-related bugs.”

Domain Primitives

The panel discusses domain primitives as an example of coding practices that naturally provide security through good design.

Dan Bergh: “It’s a good starting point to understand that using domain-driven design not only makes your code more expressive, solves more domain problems. Even though these designs were not crafted to address security to start with, they’ve also had that as a side effect.”

Jessica: “I love that what you’re recommending in this part is to think harder about what you do want in the system, express that in the code, and suddenly a bunch of things that you don’t want in the system just aren’t.”

Testing

The panel talks about the ways in which testing contributes to secure design.

Daniel Sawano: “It tends to be so much easier and more robust if you start defining your own domain types.”

Immutability

The panel discusses the benefits of immutability.

Dan Berg: “It’s possible to...configure and mutate them until they are kind of safe-ish.” Jessica: “Kind of safe-ish?” Dan Berg: “Well, we are on a DevOps podcast.”

Logging

The panel talks about the security implications of logging practices.

Daniel Deogan: “One thing that’s very important is that if you log input directly into your logs, it becomes an attack surface for second-order injection attacks.”

Dan Bergh: “It’s a perfect launchpad for doing a really, really hard attack inside your system.”

Daniel Deogan: “The common mistake that many developers do is that they more or less dump inputs blindly.”

Jessica: “We have this illusion that logging is simple, but it isn’t.”

Cloud Thinking

The panel discusses the chapter on cloud thinking.

Dan Bergh: “In a way, we’re instructing the system to become more intelligent.”

Symmathesy!

The book is available online in its entirety.

Forced Digital Transformation

Jessica notes that working from home during a pandemic, as opposed to voluntarily working remotely, makes everyone more dependent on the technical parts of the system for social interaction. Andrew says many organizations are being forced to live a digital reality they didn't want, which forces the question, though it's unclear how it will play out. Digital transformation as a tagline is not new. Andrew found an IDC projection of over $7 trillion invested in digital transformation initiatives from 2020 to 2023, made before the pandemic, and that most people consider the vast majority of those initiatives failures, with failure rates between 66% and 90%. Andrew adds that with no clear success criteria and a culture that doesn't allow failure, things that are failures by any measure are often called successes.

Pareto Inefficient Nash Equilibrium

Andrew explains the framing Andrew has used in talks for about seven years, Pareto inefficient Nash equilibrium. It describes a situation where changing certain things would leave no one worse off and at least one party better off, but no player would change their strategy unilaterally. What makes DevOps hard is that no one group can do it unilaterally, since all the players need to change behavior at once. Jessica notes that teams can act collectively but we lack a unit of collective action larger than that.

Andrew says it's slightly worse than that, because with the wall of confusion one group is incentivized to change things and so introduce instability, and the other is incentivized to maintain stability, sometimes with compensation tied to it. Jessica adds that those concerns are inherently intertwined, and separating them into units of responsibility creates conflict.

Systems Thinking and Identity

Andrew says most people don't think in systems. Dominant management thinking focuses on metrics, KPIs and OKRs and pushes them in the desired direction, while systems thinking argues you get more return by removing the resistance to those things happening. Those barriers get institutionalized over time. Andrew also says people attach identity to their task, particularly in the Western world "where our last names are literally connected to our vocation," so when told to do something different, they hear "you are erasing my identity" and produce an immune response. Jessica's version: "I am a Java programmer."

Double-Loop Learning

Andrew explains single-loop learning as picking a metric, taking action and checking the metric, while double-loop asks "why are these our KPIs? Should these be our KPIs?" It gives an individual or an organization the ability to update the model when the data doesn't match. Jessica says the hypothesis is most useful when it supports or invalidates a model, and the model is what's valuable. Jessica offers the example of a code change that makes something faster but also overheats a laptop.

Matty asks what drives leaders to think this way. Andrew says dominant management theory is an artifact of the Industrial Revolution, and applying Taylorism-driven metrics to knowledge work doesn't give good results. Andrew points to small communities pushing things forward, including Agile, DevOps, Cynefin, Wardley Mapping and resilience engineering, and to observability and chaos engineering as subgenres. Jessica says double-loop learning is needed at every level, not just among leaders. Matty notes that new ideas get retrofitted to old metrics.

Andrew says organizational learning is only possible if the central nervous system, back to leaders, has a good sensor network connected to line workers. If leaders don't have that mindset, they can actively prevent the rest of the organization from having it. Jessica adds it's more than changing how you think: you have to want to continually change it.

Doing Versus Learning

Andrew describes Senge's two domains, action and doing, and enduring change and ideation, both pathological in excess. The quip is "I can't learn anything, I'm too busy getting things done," attributed to the least productive person in the world, and the other danger is being disconnected from the doing. Andrew wants a balance and cycle between them, and says learning has its own exhaustion, using chess, guitar and learning Arabic as examples. Jessica adds a motorcycle riding book's advice: devote 5% of your brain to observing while riding, and in a race put 100% on going fast.

Matty ties it to the moment: there are times to stretch your learning and times to just do the work, and right now people don't have the bandwidth to improve everything. Andrew says "Some people are born to transformation and some people have transformation thrust upon them," and Jessica adds that right now everyone is the latter. Matty's advice is to give yourself, colleagues, family and your boss some flexibility. Andrew's final hope is that humans have "the resilience and adaptive capacity to build something better when this is over." Jessica's advice: eat more chocolate. Matty adds: wash your hands.

• Pareto Inefficient Nash Equilibrium
• Systems Thinking
• Double-loop Learning

Why It's Hard to Get Started

Laura says the definition of DevOps is hard to nail down, and "you can't buy DevOps," since it's a cultural shift. Getting into that culture takes history and hands-on experience of things not working, but you need to be part of that world to get the experience, a chicken and egg problem. Matty says context matters: people who've "been in the shit" see why it's better, while newcomers may not see why to push through the hard parts, or may wonder why it's a whole movement at all. Laura is one of the people for whom it was common sense, having started on a system built by a team focused on DevOps, and forgets that others find it unfamiliar, like being told dev does this, hands it to ops, and marketing and sales are elsewhere.

Matty says people worked the old way because those structures solved the problems of their time, not because they were dumb. Matty recalls PagerDuty new-hire orientation, where newer colleagues couldn't see why managing notifications and on-call schedules had been a problem, while Matty remembered spreadsheets and email aliases. Matty also points out that some DevOps beginners are senior technologists who are new to this approach.

Permission to Ask

Laura says the biggest thing for any newcomer is understanding "you're allowed to ask questions." People who've been around in a waterfall culture may not feel it's a safe place to ask why they're being asked to do something, and Laura says a good devopsdays provides that space through open spaces and people who are open to "ask me anything." The hard part is getting people there to begin with.

Matty adds that veterans are jaded because advocates used to be the digital darlings like Netflix and Facebook, and people replied "that's great, but we're a bank." Matty says DevOps Enterprise Summit was among the most powerful things to happen to the movement, since large enterprises and government organizations started talking about their work, and people could say "now you're talking my language." Once you know something can be done, you can probably do it, and "if you don't know that it can be done, it can seem insurmountable."

Laura suggests offering help directly: Laura answers questions in Slack communities and gets DMs from people who are stuck, which models helpfulness. Matty adds that you have to meet people where they are, since not everybody goes to conferences, meetups or Twitter.

Onboarding Onto a Team

For a team welcoming someone new, Laura says to identify the people with the patience to answer the same question a few more times, since not everyone has it, match the person's style with documentation or a person, and set up sandboxes so they can learn by doing without taking down something important. Laura says "you're just going to figure it out as you go" is too intimidating.

Matty says some of this way of working inherently feels wrong to newcomers, like giving devs access to prod, so encourage them to say why they're uncomfortable, since they may raise things the team has stopped questioning. Matty recommends pairing and shadowing, and a living "rules of engagement" document reviewed a couple of times a year on how the team communicates and talks about problems, as a lot of it is undocumented knowledge. Laura likes having each new person update the docs, like being the scribe on an incident, and calls the norms guardrails that can be repaired if they head toward a cliff. Matty cautions that you shouldn't measure success by how many changes were made, but by validating and updating as needed, and says even the author of a runbook should follow it. Laura, who has been a technical writer, says to audit docs every time, because the arrow may now be on the left of the button.

An Unwelcoming Community

Laura says joining any group is intimidating, since you don't know who's safe to ask, and that there's so much going on it's hard to know where to go. Laura was lucky to fall in with a meetup and volunteer at devopsdays Austin, and didn't learn about the community Slack for years. Laura also says that for people with experience, "all your war stories" can be intimidating for people without them.

Matty suggests sharing experience as "thank goodness this isn't true anymore," not "you sweet summer child," and asking what still sucks. Laura's own war stories are about communication, and Laura says marketing, support and sales have the same stories, including the morning after when the emails come in. Matty adds that some exclusivity can be fine in a space such as an open space on AS/400 admin war stories, not as a requirement for the bigger conversation, and Laura calls this a community of practice, like birds of a feather.

Tutorials as Journeys

Matty's tactical point is that tutorials should tell a story of why: "Why does Kubernetes matter?" and "Why did I need an S3 bucket?" Make them journeys, since context is where it fits "into the big smushy world." Laura's closing idea is that DevOps is about zooming in and out, like a camera lens that catches the butterfly on the flower and the whole picture.

Laura will be at Spring Live on March 19th, runs A Minute on the Mic, and is @nimbanatus on Twitter. Matty will speak at FailoverConf on April 21st.

Check out A Minute on the Mic for bite-sized videos from experts on various topics!

Laura

• Spring Live - March 19

Matt

• Failover Conf - April 21

Not Patrick's Idea

Jeff asks what gave Patrick the nerve to change an industry. Patrick says "I didn't have the idea," and never takes credit for it. Patrick put a lot of effort into promoting the ideas, but they weren't Patrick's, and compares it to research, where you find little nuggets and put the pieces together. It stuck because the first devopsdays got people so excited that they took it further, to the US and Australia, and Patrick says the timing was about "amplifying the zeitgeist." Patrick admits to organizing the first events because it would be cool to travel to country X, finding random people on Twitter, and says the drive was learning, which "might sound selfish."

Matty mentions the 10th anniversary of devopsdays in Ghent and the Day Zero for organizers, which had more people than the first devopsdays. Patrick calls it surreal, and is struck by the cooperation among random individuals on their free time, helping people on a budget fly to Belgium, and by growth in places like Brazil and China. It "sometimes does feel a little bit like a religion," and Patrick rants on Twitter about the echo chamber in dark moods.

Tools, People and Safety

The big thing Patrick learned from the global community is that tools and human relationships go hand in hand, though we don't want to believe it. Patrick points to the irony of automation and promise theory, which Patrick learned about from Mark Burgess, plus safety and resilience. Jeff asks why safety conversations thrive on Twitter but not inside organizations, where people are still being asked to do blameless postmortems. Patrick says there's a lag, and that change takes time, including time to recognize you need to change: "it's really hard to sometimes see yourself honestly in the mirror." Matty says devopsdays has always had an underlying human factors approach, so resilience engineering fits in naturally.

Patrick recommends a forthcoming IT Revolution book by Jeffrey Fredericks and Douglas Squirrel about honest conversations, where you say you'll go along with tool A while thinking something else, and being transparent about what you mean is part of safety at work.

Measuring What You Can Do

Jeff asks how teams evaluate what they're capable of delivering. Patrick says the first step is being aware you're not good at something. Patrick first thought more talking meant healthier collaboration, then read that it's not the number of interactions that makes a better party. Patrick's paradox is that we now use services and don't talk to them, so is that the end of DevOps? Patrick's answer is that you don't need to communicate if the other side helps your goals, and "if you don't hear too much of the friction points, then things are good." Matty adds Dunning-Kruger and a point about not knowing how far the spectrum goes, and says sharing at devopsdays and DevOps Enterprise Summit resets where people think they are.

Saying "I Don't Know"

Jeff says it takes vulnerability to be the first to say you don't know what an acronym means. Matty tells of a game with a fellow systems engineer, where one texted the other a made-up phrase on a two-way pager and the other used it in a meeting as if it were a saying, and nobody ever questioned it. Patrick recalls a 2010 conference where Stephen Nelson-Smith, "this impeccable Englishman," stood at the microphone during a panel and said, "in all due respect, I call bullshit." Patrick says Patrick was shy, and learned to speak up because people kept asking for an opinion. Patrick's tip from someone else: you don't have to talk like you know everything, just say what you found and ask for another opinion.

Laggards, Context and Moving Bottlenecks

Jeff asks about people in an organization who aren't bought in. Patrick wouldn't go in saying it's what everybody is doing, but would try to understand the person's problem, because bad past experiences make trust hard, and says you first learn the history and context, like a firewall auditor who flags open ports that the business needed. Patrick prefers supporting people with little successes to being on the barricades, and says if you can't change beliefs "you have 2 options. Either you leave or they leave," while wanting diversity of opinions too. Change is baby steps: find champions, book successes, and question yourself either way. Jeff adds that organizations have preferences, and that the purpose of the business isn't the technology.

Patrick tells of spending about 4 years at a mobile and video streaming company, getting the tech straight in a year and improving sales, marketing and legal, and realizing "we're not making any money, so there is no business fit." The bottleneck "just moved, moved, moved." Jeff says the win you celebrate this week is next week's bottleneck.

Projects, Kubernetes and Resume-Driven Development

On project management, Patrick mentions the No Estimates book and says to treat plans as a projection, play the milestone game while asking for a stake in the ground on features, time or resources, and build confidence by delivering smaller chunks, talking to project managers about their real goals.

Jeff asks whether it's fine to skip a technology, like jumping from the data center straight to Kubernetes, compared with the leapfrog to mobile. Patrick notes Jeff is already biased, and says to ask what you're trying to achieve instead of which tool to use. At a recent meetup, asked who isn't using Kubernetes, Patrick was the only one to raise a hand, because Patrick felt no friction in what they were using. Patrick says being mature is picking the right tool at the right time, same with serverless, and squads at Spotify, which Patrick says Spotify doesn't use anymore, and to avoid "resume-driven development or operations." Jeff jokes the episode title is that Patrick Debois doesn't need to run Kubernetes.

How Patrick Learns

For a new domain, Patrick starts with blog posts to find the keywords, then buys several books for different perspectives, since they share about 70%, then follows blogs, looks at slides on SlideShare, and finally follows people on Twitter. Patrick is on Twitter and will speak at QCon London, DevOps Talks Melbourne, DevSecCon Sydney and the new O'Reilly conference in San Francisco. Matty will be at devopsdays New York March 3rd and 4th and runs a workshop on blameless postmortems at SREcon Americas West, and mentions devopsdays Chicago's CFP, in its seventh year.

Referenced in the show

• Patrick discussed his love for continuous learning and referenced some of his favorite books.
• The Ironies of Automation is a paper discussing how automation can expand problems rather than eliminate them.
• Promise Theory is an approach to coordinating actors and agents in a system about their intentions to each other in the form of promises. The model was originally proposed by Mark Burgess
• NoEstimates: How to Measure Project Progress Without Estimating is a 2016 book that explains the "No Estimates" Agile concept.
• A trip down memory lane with the very first DevOps Days Ghent back in 2009.

Where you can find us

Patrick

• KubeCon in London
• DevOps Talks Melbourne
• DevSecCon Sydney
• O'Reilly Infrastructure Conference

Matt

• DevOpsDays NYC March 3rd - 4th
• SRECon Americas West March 24th - 26th

Jeff

• Panda Express usually on Thursdays

Open CFPs

• The CFP for devopsdays chicago will be open at the time of this publishing, so go to devopsdays.org/chicago to check it out!

The panel discusses the origins of the book Team Topologies. The project started with a blog post.

Matthew: “Back in 2013, I actually wrote a blog post in my personal blog. I actually wrote it in a rage.”

In 2015, Manuel joined the team to help expand on the ideas from that blog post and create Devops Toplogies.

Manuel: “What the hell are you calling a DevOps team? DevOps is not about creating a new team called DevOps.”

DevOps Topologies

The panel discusses the impact of DevOps Topologies and some of the companies that have used it, including Netflix and Conde Nast.

Matthew and Manuel explain how the project has evolved over time as DevOps Topologies was being deployed in the real world.

Matthew: “It’s not just a set of patterns or templates. We wanted to provide an organizational capability for detecting when things have changed and have gone wrong.”

The panel discusses Conway’s Law and its implications for DevOps.

Manuel: “Teams are the means of delivering value.”

Flow!

The panel discusses the importance of flow in both living systems and organizations.

Manuel: “It’s a more experiment driven approach where we have this goal or this need we need to meet and then allowing the teams to find the right solution.”

Jessica: “In modern systems, the flow is of changes to the flow of the product. It’s a very different level of work.”

The panel discusses the need for different team configurations that are constantly evolving.

Matthew: “It seems quite important to understand different kinds of dynamics in the organization at different times.”

Three Team Interaction Modes

The panel discusses the three team interaction modes laid out in the book: collaboration, as-a-service, and facilitation.

Four Team Topologies

The panel discusses the four team topologies in the book: value stream aligned teams, enabling teams, platform teams, and complicated subsystem teams.

Jessica: “The limitation of a team is cognitive load. It’s not resources, it’s not pizza.”

Manuel: “Although pizza is very appealing.”

Manuel discusses Dunbar’s Number and how that concept can put useful constraints on teams.

Buy the book!

Episode images by Jessica Kerr. Show notes by Tyler Wilson.

The Kid Who Walked Two Hours

The book opens with an email Jono got about six months into working at Canonical, from a kid in rural Africa with no computer at home. The kid earned money from chores in the village, walked about two hours to a town, bought an hour of internet at a cafe to contribute to Ubuntu, and walked two hours back. Jono says it showed what's possible when someone feels part of a bigger mission, and notes that Ubuntu is an ancient African word meaning humanity towards others.

Consumers, Champions and Collaborators

Jono defines a community as a group of people interconnected around a mission or an ethos, and says social media can play a role but isn't one by itself. Jono breaks communities into three models. Consumers share a common interest, such as Star Trek fans or Kubernetes users, without much influence on the core thing. Champions go the extra mile with content, videos, blog posts and events, which add to a stockpile so the community gets more valuable. Collaborators come together to build something, either inner collaborators on exactly the same project, as in open source, or outer collaborators building things on top of your platform, like WordPress plugins or npm modules.

Across those models, personas have different needs. Jono's example is a company building a community for business decision makers who will never go to a forum and communicate by phone and email, alongside technical implementers who are used to forums, Slack and Stack Exchange. Defining the personas, including where they hang out, what motivates them and what they fear, makes the other decisions easier. Jono says you can't attract everyone at once, so you have to decide who is most critical.

Structured, Unstructured and Long-Term Memory

Jono splits communication into structured, such as a GitHub or GitLab issue or a Stack Overflow question, where there's no "how was your weekend," and unstructured, which is everything else. Unstructured platforms divide into short-term and long-term memory, a term Jono borrowed from Jeff Atwood. Slack and Mattermost are gratifying but transient, and "anyone who claims that they can do this is lying to you" about finding old discussions, so the same question gets answered repeatedly. Forums such as Discourse are easier to reuse, and Jono likes the Solved plugin, which lets you mark the 16th post as the answer and gets found on Google. Stack Exchange and Askbot, in contrast, "build value, but you don't build community." Matty says that the conversation around an answer has value too.

Finding and Enabling Champions

Jono agrees you can't create advocates, but you can facilitate them. The project has to be interesting, and people are motivated by dreaming big, so build a mission around what you're doing. Jono describes a member journey of casual members, then regulars, then core members, encouraged by incentives and mentoring. When someone's name keeps showing up, ask them to do things, without pressure. Jono's example is content: plan the tutorials, demos and events you'd like, then ask community members to write them with editing, guidance and promotion. "You just got to ask."

Matty wonders about legal and PR worries over outsiders contributing. Jono says it varies, startups are less anxious, and "lawyers are not incentivized to let things through. They're incentivized to reduce risk." Companies should tell the community, "you're not giving any control," but treating people more like equals and setting clear expectations about where the line is.

From Open Source Project to Project Inc.

Jono says a common mistake is tacitly agreeing to everything for fear of upsetting the community, when it's better to say where the line is. Jono puts it as "I'd rather piss you off now than piss you off in 6 months." Companies that do well build inclusive environments, with open issue tracking, open code and regular meetings, where engineers, product people and the CEO are members of the community too. Jono names HashiCorp, Docker, Microsoft, Red Hat and Mattermost, with Red Hat praised for training non-open-source staff in open source nuance. Matty adds a saying about the first rule of open source, that "no is temporary, yes is forever."

Measuring Community

Jono says to slice metrics by persona and avoid "data fetishism" dashboards with hundreds of graphs. For contributors, tangible metrics include pull requests submitted, time to first response, time to review and merge, and issue lifespans, and intangible ones, like whether people are happy and learning, are best tracked by community managers. Track the action and also its validation, such as a pull request being merged. Avoid counting repetition, like a forum ranking after 500 posts, which rewards repetitive posting. Pick 5 to 10 metrics and look at patterns on a regular cadence, asking whether a slowdown is summer or a new release.

Personas That Aren't You

For keeping up with personas unlike you, Jono's advice is to be organized, breaking a big list of tactics down by quarter, and to be intentional, talking regularly to the people involved. Jono would put a recurring open-ended meeting on the calendar with translators, for example, since "all of the answers to how to do this kind of stuff well live in your audience's brains." Surveys give some data, but a call gets more valuable answers, especially if you ask where you can make a bigger difference. Jono's last advice is to start simple and track a limited set of metrics for a limited set of personas.

• "Solved" plugin for Discourse - https://meta.discourse.org/t/discourse-solved-accepted-answer-plugin/30155
• Discount code for ADO listeners for $4 off Jono's book - https://www.amazon.com/gp/mpc/A2JVV0MPZUIQ7D

Matty

• State of Devops 2019 with Dr. Nicole Forsgren and Jez Humble
• Certification Fun with Jay Gordon

Bridget

• Too many to name! Three recent ones in a row about k8s - the Kubernetes Best Practices book, Ian Coldwater, and Kelsey Hightower today.

Trevor

• Catching Up With Steven Murawski
• Devopsdays Chicago 2019

Jessica

• The Meltwater Transformation

What happened in 2019?

Bridget

Switched from devrel to product. Helped with Helm 3 launch!!! Traveled less. Dragged Joe to a personal trainer because if he’s going maybe I will too.

Matty

Lots of speaking (spoke at 24 conferences in 2019). Moved back to Chicago. Went to lots of devopsdays. Got even more interested in resilience engineering and realized I don’t know shit.

Trevor

More travel! A lot of customer work, focus on delivering solutions, now moving officially into product. I also got really into shuffleboard and DND

Jessica

Consulting!

Why Teaching Makes You Better

Ali says that if you're going to advance in a development career you'll end up mentoring, leading a team or moving into management, so knowing how to onboard people and help them learn concepts, not just fix one bug, pays off. Teaching also makes you a better programmer, because people ask questions you'd never think of, and "if you can teach something well, you really know it." Ali took education classes in college, would have been an education minor if they hadn't left to be a software engineer, and got more from them than from computer science classes.

Matty says teaching Chef Fundamentals workshops at customers for two days over and over taught Matty things about the product they hadn't known. Teaching forces you to think about the why, and the process stuff sticks more with some background behind it. Ali's key teaching concept is linking: take what somebody already knows and link it to a new piece of knowledge, because teaching in isolation is "impossible to learn," which is why a first programming language is harder than a second.

Teaching at Scale

Matty asks how to do linking in a big room or a recorded course. Ali's answer is to subtly reteach things several times with different formats, such as diagrams, videos, a code-along and a traditional lecture, and to use different examples for different people. Ali notes that Star Wars tutorials appeal to some, but "my students freak out when I do Britney Spears fans apps." When teaching variables and functions Ali links them back to algebra class.

Matty adds that as learners we think we know more than we do, recalling the swing dancing saying that beginner dancers take intermediate classes, intermediate dancers take advanced classes and advanced dancers take beginner classes. Ali agrees from experience: after years as a software engineer who jumped straight into React, teaching at a bootcamp meant relearning foundational JavaScript from the ground up.

Write It Anyway

Ali says everyone has a unique perspective, and it can be terrifying to publish when someone has done it before. Ali wrote a React tutorial well after the initial burst of React tutorials, sure nobody would read it, and it became the most read post. Matty tells of a SharePoint fix posted on a blog mostly as self-documentation, which an MSDN forum answer then sent thousands of views a day. Matty also says some posts have an intended audience of "future Matt."

Ali has a talk called Yes, You Should Write That Blog Post, about writing for three people: past self, present self, since teaching cements learning, and future self. Ali returns every year to a post on filling out calls for papers. Matty's popular post was on getting a Powerline font working on a Mac in Visual Studio Code, and Ali's was about how they set up computers, "nobody wants to read that. Well, turns out they do."

Making Content Better

Ali's advice is to make it skimmable, since "people don't want to read an essay," with headers, images and lists, and to add the why, what problem this solves, in the introduction. Also appeal to multiple learning styles with code demos, graphics and other formats such as podcasting and video. Matty says lists get people in, and then they find your deeper posts. On talks, Matty calls the why versus how balance "eat your vegetables" and likes single-track conferences for that, and Ali says conference talks aren't the best format for education because people aren't hands-on, so the best thing is to get people excited enough to learn afterward. Matty says attendees asking for more technical content usually mean practical examples.

Getting Read

Ali says the two main channels are social media and search engine optimization. On social media, consistency and being a real person matter. Matty says you don't have to share everything or engage with everyone, "You don't owe anybody a conversation," but to be successful "have some conversations." Ali adds that LinkedIn posts live longer than tweets. For SEO, Ali says to add a couple of keywords but not to keyword stuff, and backlinks come from making good content and relationships, not spamming. Matty adds that when asking for a favor, "don't give 'em more work to do."

Ali recommends cross-posting to a platform with a built-in audience such as dev.to, using a canonical URL so Google knows it isn't duplicate content. Matty suggests the company engineering blog, which is usually "dying for content," and says that cross-posting with a canonical URL helps everyone and is good for recruiting. Matty also says repurposing talks as blog posts and the reverse works well. Ali says making it part of the workday is one strategy for finding time.

Dealing With Criticism

Ali says the number one question in blogging workshops is how to handle people online, and that when criticism gets personal it comes from "a place of hurt." Ali expected the worst from day one as a young woman on the internet, but it didn't happen for about a year, until a post hit the front page of Reddit, so it probably won't be as bad as you think. Ali's own strategy is to write out a response, screenshot it to a couple of close friends, and delete it. Matty says that if you're someone who doesn't have to deal with it, you can be part of a vent network: say that's really shitty, and that you're sorry, and be done. Ali adds not to minimize it, because "nobody should have to deal with that."

Where Ideas Come From

Ali suggests writing the post you were Googling for and couldn't find, writing your own story, or writing about what you want to learn, and Ali's first blog was learning a new technology every week, building an app and writing about it as a beginner. Matty tells of Annie Hedgepeth, who blogged while learning Inspec, and for a long time the project's documentation was that blog. Ali likes ConvertKit's slogan "teach everything that you know" and likes complete beginner guides. Ali's biggest tip is to write down ideas when they come, while walking a dog or Googling, and not when you're looking to write. Matty ends by asking for ideas for 2020 talks on Twitter at @MattStratton and pointing to devopsdays.org/speaking for open CFPs.

The 56K Modem Era of Kubernetes

Kelsey compares Kubernetes to the 56K modem: some people liked the dial-up sound because it meant they were getting on the internet, and people now look at their nodes and clusters and feel like they're doing computing. The internet got interesting when that went away, with DSL and wireless routers, and now "the internet is now just a thing." Kelsey says "Things tend to get better when they disappear," and that we're in the 56K modem era of Kubernetes, and hiding it will let more people use it without learning to manage it.

Bridget raises the fast-moving release cadence, and someone stuck on 1.12 or with no Kubernetes yet. Kelsey says that's a good place to be, because "if you don't have this problem, you don't really need this solution quite yet." Linux went through the same transition, from rolling your own distributions to Red Hat and Canonical, and Android users benefit from Linux without touching the kernel. Kelsey says it's early: Kubernetes is only 6 years old, VMs still work, and some people will skip containers and go straight to serverless, but Kubernetes-style APIs are resonating, and some people will use parts of it without ever being a cluster administrator. Kelsey describes GKE, AKS, Fargate and K3s as "checkpoints" in an ever-moving project, and says Kelsey is now a consumer of Kubernetes who goes to the checkpoints and uses it as is.

Kubernetes the Hard Way

Asked what people can still learn from Kubernetes the Hard Way, Kelsey recalls that early on there were no docs, and learning how to install it came before the first line of code or PR, which revealed what the scheduler did and what went where. Kelsey's argument is "you can't really fix a system that you don't know how it works." The guide goes step by step with no scripts, so people see what the kubelet does, how it connects to the API server, and where the certificate goes, and gain the foundation to troubleshoot and debug.

For production, Kelsey says there are many layers, and security is number one, since performance can be tuned later but "once that security hole is too big, it's a little too late to go rewind the clock on a breach." Most clusters come out of the box with flexibility and not security, so you can run random images and run things as root, and Kelsey compares it to SELinux, which every system admin turns off. Kelsey says 30% of Kubernetes the Hard Way comes from security feedback, which is why it generates certificates for each component and encrypts secrets in etcd, and why it leaves the dashboard out. Bridget says they show the dashboard in workshops with a giant disclaimer about cryptocurrency miners. Kelsey is encouraged that KubeCon talks and docs now say to do exactly 4 or 5 things, raising the security profile for many people at once. Bridget mentions Ian Coldwater's keynote and Gareth Rushgrove's work with Open Policy Agent.

Complexity Moves

Kelsey uses CDNs as the model. People once glorified FTP and then SFTP, and thought FTP would just become more robust, but CDNs took the problem of getting files close to people and made it disappear as the complexity grew and the number of people who understood it shrank. Compute is slower because so many people believe they understand the compute problem, so a new person invents a new platform every couple of years. Serverless says there are about 80% of compute use cases that are understood and never need to be built again, but mainframes and VMs don't go away either. Kelsey says "I really look at this as we're going to have multiple things in parallel," and that if starting from scratch, "I would probably try to go as high as I could and focus on building my app and the product before going to play infrastructure again."

On trust and lock-in, Kelsey says nobody digs up a wire to connect to the backbone of the internet, and compute needs providers that earn trust with open interfaces. On compliance, Kelsey says some banks are 100% online and see a single building with a door as too much risk, so it's "different degrees of understanding the risk."

Stop Saying Legacy

Kelsey asks engineers and executives to stop using the word legacy, since it has a derogatory context, and says "classic infrastructure" instead: "It's the stuff that actually worked cutting everyone's paycheck." Bridget calls it "the place where all the customers and money are." Kelsey asks what problems they have now, such as service discovery or scripts and large on-call rotations for failing over, which is the opportunity, since a part of Kubernetes solves that specifically. They may not need all of Kubernetes, and Kelsey asks them to have a good reason why: if 15 people maintain a scheduler that looks like Kubernetes, those people could work on another problem, or half of them could contribute to Kubernetes.

Kelsey likes to spend time on-site asking what's on the backlog, often observability and security, and says even if Kubernetes automated you out of a job, "which it won't," it would give you time back. For people deciding whether to adopt, Kelsey says the world will continue to move with or without you, so put the tool through its paces, and if the answer is no, write an internal doc on the reasons and revisit if they get solved. That's "just engineering."

Infrastructure as Data

Kelsey has worked at Puppet Labs, used Ansible and contributed to Ansible and Terraform, and describes the last 10 to 15 years as an attempt at infrastructure as code, with DSLs, for loops and then the problems of any codebase. Kubernetes tries something slightly different: "No more infrastructure as code. Now, we're doing infrastructure as data." The logic and state machine live in the controller, which some people call operators, and on the front end you restrict yourself to data, which is YAML, even though Kubernetes itself only supports JSON and protocol buffers. The drawback is a lot of YAML, but like assembly language, any language can compile down to it. Helm can run as a preprocessor, pipe to Kustomize to patch, and go through an admission controller, which is "the dream come true" of describing infrastructure with a type system and interchanging tools.

Bridget asks whether the flexibility makes the complexity unapproachable. Kelsey says Kubernetes is "formalizing the complexity," so it can be seen in one place, and people deal with it at different levels. Someone managing the environment can create a custom resource definition that says deploy my app across 50 countries, and control loops do the heavy lifting. Kelsey says that if you just want to deploy containers, "writing CRDs and operators is the equivalent of writing kernel modules," a job for the people who need to extend the system and not for most people, who need only declare a load balancer, a certificate, a DNS name and a container.

The End Game

Bridget says picking the technology first is resume-driven development. Kelsey says it's hard when you don't know what question to ask and every deploy goes wrong for 10 years in a row, and then you go to KubeCon and see someone deploying all over the world, and think you need some Kubernetes, which Kelsey admits to being partly responsible for. Bridget jokes, "Ask your doctor if Kubernetes is right for you."

Kelsey's story is teaching their daughter to make a GeoCities-style web page in a text editor, viewed in Chrome. When the daughter asked why a friend couldn't see it at 127.0.0.1, Kelsey used Firebase, and "she said Firebase deploy and it spit out a URL." Kelsey calls that the end game: people with an idea and finished code who want to see it come to life, and any platform that gets closer to that moment is exciting. Kubernetes will evolve that way from the ground up and serverless will work its way down to support other workloads. Bridget adds that we should remember we're building things that produce actual value, and Kelsey agrees, adding that the middle layers matter and aren't the end game. Kelsey is on Twitter, DMs open, and on YouTube, with meetups announced about two weeks ahead.

Bridget chats with Kelsey Hightower about Kubernetes and the future.

• Kubernetes the Hard Way

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2019 or ADO2020 for discounts on lots of devopsdays

Possible, But Not by Default

Bridget asks if Kubernetes security is possible. Ian says yes, but Kubernetes is not secure by default, so if you assume it's secure out of the box you may be unpleasantly surprised. Asked for the over-under on new CVEs between finishing the keynote and giving it, Ian says they don't personally know of anything anybody is sitting on, so the number is completely unknown. For learning, Ian points to the engineering blogs from Aqua Security, StackRox and Twistlock, the Kubernetes security announce list, the Kubernetes documentation, past KubeCon talks, and people on Twitter. Ian acknowledges it's a pile of stuff, because Kubernetes has a lot of moving parts, but "you can do it."

Before You Go to Production

Ian's top item is admission control, "the biggest thing that you can do to stop attackers from being able to compromise your cluster." It's a set of policies that dictate what privileges get run and who is allowed access, and pod security policies are part of it. The user experience could use work, with pod security policies particularly notorious, but it's worth learning. Ian's second item is being careful about what's exposed to the internet, and suggests looking at Shodan, which indexes every 24 hours, so the idea of being too small to be a target isn't true. If SSH ports are exposed, use SSH bastions.

What Developers Can Do

For developers who don't run the cluster, Ian says one way in is supply chain attacks. Software has a supply chain, and Ian uses npm as the example, where a Node module has dependencies that have dependencies. Libraries and container registries are all potential vectors, so know what you're running and keep it up to date. Ian adds that people are a vector too, including yourself: developers know they're smart and may be more vulnerable to phishing than marketing or sales, since they're confident and may "pay less attention to those trainings by the numbers," and they often have more access, so they make exciting targets.

On pinning versus staying current, Ian says containers done correctly make patching much easier, since you can spin down a container and spin up an updated one. In Kubernetes, have a plan for upgrading, and especially for upgrading in place, since newer security features may not come into an existing cluster to avoid breaking changes.

Defenders Think in Lists, Attackers Think in Graphs

Bridget calls back to the keynote line. Ian explains that people who aren't attackers think in lists: a list of compliance boxes, or sprint points, and "did we get them all?" They aren't necessarily looking at the layout of what's running and how things connect. Attackers want to get in, find out what's there, how it talks to other things and whether anything is vulnerable, to get "a lay of the land." If you don't know the connections between your resources but the attackers do, "that's a disadvantage for you." For building the graph, Ian recommends a VMware tool (the transcript garbles its name) that's useful both to administrators and to pen testers, since you can run it locally with the privileges you have. Ian has heard of cluster visualization resources in Visual Studio Code but hasn't tried them. Bridget notes the Kubernetes extension for VS Code is open source and plugs colleagues who work on it.

Not the Team of No

Bridget asks why Ian is "suspiciously positive for a security person." Ian says security people are notorious for being the team of no, and that it doesn't help relationships, since people who don't like you won't talk to or listen to you, and doesn't help security either, because developers and operators have their ears to the ground. Ian has worked in DevOps and knows what sprints are like, thinks most people mean well, and says "being a negative jerk doesn't really seem like it helps with that, so I just don't do it."

Getting Into Offensive Security

For people who want to become pen testers or red team members, Ian recommends capture the flag games, in which the flags are on servers you're sanctioned to compromise, and mentions overthewire.org as a beginner-friendly site. Ian has put on CTFs internally for developers and operators, and it's amazingly effective when people find out how fast it is, like "you can literally just hit that button?" Ian warns that some developers get bit by the bug and want to do it all the time.

Ian is @IanColdwater on Twitter, says Twitter is a good place to learn about security, and warns that their LinkedIn says it's only good for phishing. Ian's parting thought: "You don't have to be anybody other than who you are," diversity is strength, and it's important to step into other people's shoes, whether a security person stepping into a developer's or an operator stepping into an attacker's. "Lead with empathy, it's important."

Bridget chats with Ian Coldwater at the devops Minneapolis meetup about their KubeCon North America 2019 keynote.

Video from KubeCon: Hello From the Other Side: Dispatches From a Kubernetes Attacker

Art credit: Sarah Becan - original tweet, threadless store, website

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2019 or ADO2020 for discounts on lots of devopsdays

Short Essays, Not a Narrative

Brendan says the project has moved from something people heard about to something everyone wants to implement, but people struggle with specific tasks, and hands-on help doesn't scale. They've seen "lots of people sort of shoot themselves in their foot." Unlike general introductions to Kubernetes, the book is focused on specific topics, to dip into when working on machine learning or setting up a cluster for a bunch of developers, so it's "a series of short essays rather than a whole put-together book." The 258-page PDF Bridget has in front of them has no narrative flow, which is on purpose.

Lachlan says now is the right time because adoption has grown, the ecosystem has become more complex, and Kubernetes has a sprawling variety of APIs, so the book shows where to start on topics like policy, rolling upgrades, governance and security. Eddie says organizations are already down the path and don't want another step-by-step walkthrough, and the authors tried not to make it a snapshot of one version. Dave says users need to focus on the core concepts and often skip them to over-engineer. Lachlan likes the mix of philosophy, meaning why you'd want policy, and the tactical how.

Chapter Favorites

Lachlan's favorite was Chapter 11, on policy. Lachlan notes "everybody loves hearing Chapter 11 for anything," and says enterprises moving workloads to Kubernetes ask how to make sure workloads conform to policy, whether regulated or just wanting to understand configuration. Bridget notes the chapter covers the open source project Gatekeeper, and Lachlan says it's a Kubernetes-native implementation of OPA, the Open Policy Agent.

Dave's was resource management, which "doesn't sound really exciting at all" but is something users struggle with and affects scaling. The book covers best practices around requests and limits and how workloads behave when capacity runs out. Lachlan says most of the outages Lachlan was paid to handle in the early days came from resource management, as clusters got to 80, 90, 100%, and would have liked to have had the chapter in 2015, to avoid a cluster going into cascading failure at 3:00 AM.

Eddie's was Chapter 9, covering networking, network security and service meshes, which was the most challenging to fit into a concise format. Eddie calls networking the foundation, where little things trip people up, like the move from kube-dns or SkyDNS to CoreDNS, and describes a customer where divisions put Kubernetes in without telling anyone, and then security asked why their controls were gone. Eddie says people want a service mesh as an "easy button" for observability, security and policy, and find it's "Thousands of little buttons that you have to press in the right combination." The chapter describes what all service meshes should do, what to prioritize, and the SMI spec, a common API for those things.

Brendan's favorite, after the first chapter on laying out a service, is the one on developer workflows. Brendan worries that operators love the technology, or it's great for continuous delivery, "but we've made the developers' lives miserable." The chapter covers partitioning a cluster with namespaces, onboarding a new developer, RBAC so people don't step on each other, cluster-level logging and monitoring that's just there, and testing and debugging, since "if it's not easy, people will do less of it, and then you ship buggier software."

Will Organizations Do This?

Bridget asks whether people in the field actually follow these recommendations. Brendan says people want to, and "the recipes just aren't there, necessarily." Eddie describes a customer creating a new division to build patterns for developers and ops so onboarding is easy and everything is automated. Dave says there's a big cultural impact, since you leave control to Kubernetes that you used to hold tightly, much like adopting a DevOps culture. Lachlan says there's no single right answer, but a set of tools and techniques that have worked, which the book offers as blueprints, so you aren't "left scrounging."

Best Advice

Lachlan's advice: this is a journey and not a destination, there will never be a point where the system is perfect, and people paralyzed by indecision should use best practices to make a decision and keep adjusting. Dave's is to walk before you run, and get good at the core capabilities, network security, resource management and policy before layering on tools like a service mesh installed with one Helm command. Eddie's is to keep an open mind, since practices from on-premises servers and VMs may no longer fit, and the bias of "that's not how we did things" is the biggest challenge in the field.

Brendan's is to understand why you're making every decision, including Kubernetes itself, and not adopt it because "Everybody needs a Kubernetes strategy" appeared in a magazine. Brendan says Kubernetes makes deploying easy without helping you understand the system, and that things hard to manage over time are easy to start, so people assume the ease will continue. Brendan's summary: Kubernetes "is alive," a living thing and not a static one, that "could turn on you at any moment." Bridget calls it the Admiral Ackbar principle: "it's a trap if you think that it's going to be easy." The episode closes with a joke about putting a paper copy in a time machine, or a DeLorean, so Biff can't steal it.

Bridget chats with the authors of Kubernetes Best Practices: Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson. (Kindle version available now!)

Community & Events & Stuff

Dave:

• All Things Open - Talk on Non-Code Contributions To OSS
• KubeCon NA - Lightning Talk on Non-Code Contributions to Kubernetes

Brendan:

• K8s meetup in Heidelburg Germany
• Microsoft Ignite
• KubeCon North America

Eddie:

• Medium Blog
• Kubernetes on Azure Best Practices Series
• KubeCon NA and Contributors Summit in November
• Austin Kubernetes Meetup

Lachie:

• OSS Unboxing on Youtube
• At KubeCon and the Kubernetes contributor summit in San Diego in November

Bridget:

• Twin Cities Startup week, devopsdays Philly, devopsdays Ghent, Velocity Berlin, KubeCon
• Van Moof ebikes! Joe and I just got these.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2019 or ADO2020 for discounts on lots of devopsdays

Curating a Single Track

Peter says the program is single-track, in its fourth year, so it has to balance culture talks, technical talks and talks that feel timely. Attendees complain in both directions, that it's too cultural and that it's not cultural enough. Peter tries to make the opening talks lean cultural and relevant to the moment, and if possible frames other talks around that theme.

Indifference and Software Defining Culture

Jocelyn's keynote was about ethics in technology and the responsibility technologists have. Jocelyn says technologists can seem indifferent to the problems that larger tech companies they use or work for cause for people who aren't technologists. These conversations aren't new, and Jocelyn says social media, Twitter especially, has been a vehicle for people who might not have been heard inside companies.

Tim's earlier talk at Philly turned Conway's Law around: the systems we build and the choices around them also shape our organization's culture and the wider culture. That includes building for reliability so people trust each other, and which technical communities you join, such as a language community with toxic behavior. Tim continued the theme at devopsdays Minneapolis the year before, on whether we build technologies with a good purpose or a neutral one that can be used in bad ways.

Do You Need This Data?

Bridget says technologists tend to look at the happy path and ask how realistic it is that something will never be abused. Peter says engineers look at data as inputs to a service, where the inputs can be extremely sensitive and could harm the people the data is about if they got into the wrong hands. Jocelyn says we've reached the point where we take all the data because it comes with it, and should ask "do we really need this data in order to perform the function that we have?"

Tim gives an example from a job a couple of jobs back, a sensor that counted people and detected movement through a space. The designers saw how individual tracking could be abused and designed it so that identifying individuals was impossible, which became a competitive advantage because customers also didn't want their workers tracked. It also detected mannequins moved through a doorway as people, which Peter confirms meant people carrying mannequins in retail. Peter says the general consensus is the opposite, recalling Etsy's "if it moved, graft it" for metrics, and that for customer data it's "Storing bits is cheap." Bridget answers that it's cheap until the lawsuit, and cites a line Bridget can't attribute, that PII is like toxic waste, something you have to carefully corral.

Peter has worked at two organizations with HIPAA data and says the care was amazing, with legal review, access controls and training, but most other data gets a shrug. Tim adds that seemingly meaningless data points can be combined in harmful ways, and says data lakes are a blob of data with models applied that aren't magic, since "we're really encoding like, all the biases of the people who are doing that work into those models." Tim's example is parole models, and "oh, weird, it turns out that that algorithm is completely racist." Peter adds "And they blame the algorithm, not the people who wrote it." Jocelyn says the question is also why the data is collected, and that "just collecting data just to have it" isn't a good enough reason.

Neutral Tools and Responsibility

Tim says some technologies are neutral, like Kubernetes, and all can be turned to bad ends, but asks whether there are any good uses for facial recognition. Jocelyn says many engineers see it as a job and don't think about the impact, and hopes people take away that "it's your responsibility to think about that," and that not being on the executive board doesn't mean you can't have an opinion about what your work is used for. Bridget says you don't need a dramatic splash, and that if you think about operability and how systems break, that extends to data breaches and privacy, such as a provider who emailed that everything had been in an S3 bucket open to the world.

Money, Regulation and GDPR

Peter says it comes down to money: if a breach would be company-ending it gets bolted down, but if the cost of fixing is more than the cost of the damage, it doesn't. Tim says companies have been allowed to externalize these costs, like pollution, and that if technologists don't fix it themselves, lawmakers may eventually decide how technology works, which probably won't be a good technical solution. Jocelyn thinks it will take federal law with real penalties. Bridget notes GDPR has penalties with teeth, and that some US companies respond by dropping Europe. Tim says California enacted a similar law, and most US companies won't cut off California. Peter says a toothless law won't work, and fines need to scale per infraction, not be a one-time fee.

What Individuals Can Do

Tim says technologists have a lot of privilege and power, and "if I were to ask who here is hiring, everybody would raise their hand." That power includes choosing who to work with, which projects and who the customers are, and Tim says it's time to exercise it together. Jocelyn says to start at work by being the squeaky wheel, and outside work, to start on Twitter, where Jocelyn began before podcasts and talks, and says "Twitter activism does work." Peter echoes the "vote with your wallet" idea for where people work, acknowledges leaving a job on principle is hard for some, and says that if people banded together and left companies that don't meet their standards, it "would really, really send ripples through the industry."

Bridget chats with guests Peter Shannon, Jocelyn Harper, and Tim Gross in front of a live studio audience at devopsdays Philadelphia 2019.

• Jocelyn Harper's keynote at devopsdays Philly 2019: CI/CD (Constantly Ignoring, Constantly Deflecting): A Case Study of Pervasive Indifference in Technology
• Tim Gross's talk at devopsdays Minneapolis 2018: They Didn't Stop to Think if They Should: Machine Learning and the Internet of Unpatched Things

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2019 for 20% off lots of devopsdays
• Velocity Berlin Nov 4-7 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes

DevOps at Meltwater

The panel discusses how DevOps has transformed the working environment at Meltwater. When Joan started at Meltwater, engineering and operations were kept entirely separate.

Joan: “There was this huge wall because engineering knew how things were built and operations knew the production systems...but there wasn’t any cross-pollination there.”

The lack of communication between the teams caused friction and slowed down product delivery. The process could take as long as six months.

Joan: “We’re a Software as a Service company so we always want to build products that are sticky and keep our customers around.”

Joan discusses how Meltwater began to move to a more agile business model. Getting engineers and operations people in the same physical location to work this out was important. Creating truly cross-functional teams couldn’t happen over Skype alone. DevOpsiCon was born to facilitate this transition.

DevOpsiCon!

Buy-in wasn’t immediate throughout the firm. Teams were updating legacy systems throughout the company and some were more focused on work than enablement.

Joan: “The first one was a little bit sneaky. The next one was in Manchester and that one was a little less sneaky.”

The panel discusses the importance of investment in enablement. This allows the feature teams to focus on the features they were actually building instead of all the underlying infrastructure.

Joan: “Having them build everything they need is not realistic.”

The fact that Meltwater has offices all over the globe makes any wholesale shift of company culture difficult. The panel discusses the need for face-to-face interaction to facilitate change.

Gene: “There are these events like DevOpsiCon where we solve the problem by bringing a broad set of people together.” DevOpsiCon has grown beyond just developers. Product, UX, support, and even sales have joined the event to help build relationships and give insight.

Jessica: “That is very DevOps in the sense that DevOps says that you need to take these concerns that cannot be separated...and you can’t put those responsibilities on different teams and make them fight.”

Unconference

The panel discusses the value of DevOpsiCon as an unconference. Attendees collectively set the agenda on each day of the event.

Gene: “That format has been so much fun and created so much value.”

Gene: “The event becomes the conference you didn’t know you needed at the start. It adapts collectively to what the organization needs at that moment.”

Ongoing Transformation

Gene discusses how allowing teams the freedom to experiment can yield results that a top-down structure could not provide.

The panel talks about the structure of mission, framework and support teams.

Joan: “By keeping things loosely coupled, it gives teams a lot more freedom to make the technology choices that are best for them.”

Ownership Challenges

The panel discusses the ownership challenges of moving from the datacenter to the cloud specifically and large enterprise changes more generally. The investment in enablement is critical.

Joan: “There’s very much a not built here mentality that can take place.”

Gene: “Software can benefit from being owned by many people.”

The panel talks about overhead costs to both software and human changes.

Gene talks about the challenges of moving to a full mission responsibility model.

Gene: “You need to find efficiencies more effectively than you’ve ever had to find efficiencies before.”

So! Has It Worked?

Gene: “Oh wait, you’re looking to me for an answer?”

The panel discusses the results of the ongoing DevOps transformation at Meltwater. The six month process for production changes has transitioned to a constant stream. Beyond the productivity, it has also been a quality of life improvement for the employees.

Gene: “We’ve got a Slack channel for all the production changes and it is constant activity in it all week long of changes going out.”

Joan: “We’re now getting hundreds of features out every six months to our customers.”

The panel discusses the ways in which transformation is, or should be, a constant process. The change goes on.

Find out more about Meltwater, their DevOps transformation, and the process of an unconference on their blog.

Episode art by Evelyn Kerr.

The Gap Between Advice and Implementation

Sasha says devopsdays keeps having non-technical conversations, partly because the audience is so mixed that you don't know who is dev, ops or PM, and partly because as an open source event it doesn't want to endorse products or go deep on technology. People leave with amazing ideas and "have not seen anybody do any implementation of stuff." Ken says every conference has a DevOps track, but the talks say to do security testing, not how it was done on a specific project. Looking around, there were DevOps conferences and language conferences "but not CD conferences."

Ken picks on the talks Ken gives, which stay at a high level because the audience's tech stacks are unknown and 30 minutes isn't enough to go deep. The hope is that what was 5 minutes of a talk becomes a 30-minute talk on how compliance was done on one specific project. Sasha adds that the conference aims to be vendor-neutral, so people can showcase products and techniques and compare them. It's for hands-on practitioners, closer to "hone your craft" than an introduction, and Sasha says a first-timer is probably better off at devopsdays. One gap Sasha hopes to close is continuous delivery for databases: "you can't really continuously deliver your entire software application if you're not able to continuously deliver your database."

Discussions as First-Class Content

The format is a 30-minute talk followed by a 20-minute facilitated discussion, which will be recorded. Sasha says they love open spaces and wanted to make the conversation first-class content, so people who couldn't attend can listen. Ken says the conversation is about the topic, not the talk, and the aim is "to take the speakers off the pedestal," though the speaker can join or not. The facilitator will ask what challenges people have, what successes they can share, and the magic wand question: what would you want to see in the next two to five years, in product or culture, to make this easier. Sasha notes that Q&A isn't for arguing with the speaker, and this gives people with a different opinion a place to voice it, with rooms to continue offline.

The recordings are audio only, to respect privacy, with a microphone used as a totem, and participation is optional. Ken's hard metric for success is that the discussions get at least as much traffic as the talk videos. Sasha's softer one is seeing the light bulbs go on and people learning they're not alone in their experience. Ken adds that it's a "paid focus group after every talk" for vendors.

Program

The tagline is learning from today and shaping tomorrow. The opening keynote is Jez Humble and Dave Farley, co-authors of the book Continuous Delivery, which has its 10-year anniversary next year. Heidi Waterhouse from LaunchDarkly will speak on feature toggles. Ken teases machine learning in pipelines and hands-on security sessions, and a second-day panel of executive-level people on what CD will be in the future. The CFP closes the night of the recording.

Fast Versus Safe

Matty asks where the state of the art is. Ken says the industry is at a crossroads: there's a focus on how long a pipeline takes from commit to production, and a 3-minute pipeline can't have done compliance and security checks. Jez's definition of continuous delivery on continuousdelivery.com "specifically says safely," and Ken was glad DORA now says "on-demand" for the top tier. Matty's version is that it's not as fast as possible but as fast as you need it to be. Matty also tells of an e-commerce boss who said the target for a site performance dashboard was "not slower than last month," which is good while you're improving and less useful once you reach what's appropriate.

Ken argues that what you measure isn't the business value of deploying on demand but the value of the thing you deployed, framed as a hypothesis, and wonders whether machine learning in the pipeline could learn what a change did to the business and kill the canary automatically. Ken also says "I don't think hardly anything that we call AI is AI." Sasha says not everyone is a startup deploying straight to production: people have different compliance requirements and levels of legacy, and "I don't even like the word legacy because like, hey, this is software that's making money for you." Matty supplies "heritage systems."

Matty brings up incident response, where you'd like to restore service without bypassing your normal checks. Ken describes a pipeline on the most recent team that was fairly long with all the security checks, but had a short circuit for emergencies. Basic unit tests and fast tests ran and produced the installer, and someone with the right permissions could click a button to reach production, with the rest of the pipeline tailing it. It was still the same pipeline with things skipped.

Tickets, Sponsors and Who's Missing

Tickets are at deliveryconf.com, and the code ADO gets 10% off. Ken says the price, about $400 to $450, is higher than most community events because of three tracks and all the recordings, and people who can't attend for financial reasons should reach out, since some sponsors may help. The conference is a not-for-profit backed by devopsdays Seattle, and Ken says it lacks the budget for big promotions. Sasha says hearing that people who didn't know of Sasha's involvement call it a solid conference means "we're actually hitting the mark."

Gold sponsorship includes a sponsored talk, and the goal is deep technical talks and not product pitches. Sponsors submit decks two weeks ahead for review. Ken tells sponsors to bring engineers and product managers, not just sales staff, and says the aim is real solutions to real problems, without requiring live coding. Sasha closes by citing a study that only 3% of women identify as technical in the Dev/CICD space, says it's proving hard to find non-male speakers for the CFP, and invites diverse participants onto the stage.

Matty Stratton talks with guests Ken Mugrage and Sasha Rosenbaum about their new event DeliveryConf.

Transforming a Bank Founded in 1888

Devi's talk went straight to the practicalities of transformation at a giant organization. Devi says "we were not a Spotify": Nedbank is a traditional bank established in 1888, with traditional values and local and international regulation, which makes bringing in "a sense of looseness" very difficult. The approach was to embrace those traditional concepts, accept who they were, and start from there. A bank has to demonstrate value before anyone adopts anything, so they went in with evangelists who had credibility and used them as poster boys for the message. Devi says it's about generating financial value, doing things cheaper and better, and that as long as they generated value they could get more investment. That is how they're growing "slowly but surely, one pipeline at a time."

Adrian says it's easy to find good stories from unicorn companies, so the program needed balance, and if a big enterprise known for moving slowly can adopt DevOps, "it's possible anywhere." Devi's bank is a sponsor and has spoken at the event for the past few years, and Adrian notes many South African banks have adopted DevOps. Bridget says Minneapolis likewise has people from large local retailers, since it isn't only for the tech vendors.

What MMA Taught Me About Working in Tech

Daniel's talk title was What MMA Taught Me About Working in Tech. Computers and mixed martial arts have both been big parts of Daniel's life, and Daniel realized that a lot of how they act, make decisions and interact with people came from martial arts training. Cobus says balancing technical and cultural talks is always fun, because "tech is never the problem. It's always people," and the organizers were surprised how much culture showed up even in talks they expected to be technical. Bridget says that's also true of open source: adoption depends on the people deciding whether to trust a project, and a technically excellent tool with no documentation and six months of silent pull requests looks risky.

Adrian agrees that if a project's pull requests sit for months and the last commit was six months ago, "if there's no community around it, it's not going to happen." Daniel pushes back that vitality is worth checking but doesn't settle it, since some software Daniel uses hasn't been updated for 10 years because it still does what it was designed to do, unless there's a security problem. Daniel adds that tooling is getting better at flagging that, and suggests building checks into pipelines or occasionally looking at the CVE list.

How a Bank Picks Its Tools

Bridget asks how Nedbank decides whether an open source tool is okay. Devi says open source is allowed in principle, subject to tight security checks. The method starts with the community that will use the tool: unpack their problems, map the whole value stream, have the team prioritize the biggest issues, then look for tools that would solve them. A proof of concept runs for open source and paid tools alike, and the outcomes are evaluated against the original problems. Where timing matters they "literally stopwatch our developers" before and after a learning cycle, rate the tools on a Harvey Ball scale of 1 to 4, and sometimes test two tools in parallel with different teams.

An easy tool takes about two weeks, starting at the beginning of a team sprint after a few days of training, and a heavier one about six weeks. Sign-off takes longer than the proof of concept "because we're a bank." Devi says a full process map can be done by locking people in a room for four hours. Bridget likes that it happens inside the sprint, since for the period of the decision it has to be the work, and Devi says that makes it practical and not "a science experiment scenario." Daniel says the depth of the answer was impressive and unexpected.

Cape Town, Joburg and Who Runs the Conference

Bridget asks about the link with the Johannesburg community. Adrian's involvement has been mostly Cape Town, and jokes that Capetonians give Johannesburg a hard time because "They don't have the ocean." Adrian says a growing DevOps community exists in Johannesburg and the Cape Town team should help them start a devopsdays without running it. Adrian mentions the Python community, where PyConZA needs meetups in both cities to qualify under the Python Software Foundation.

Daniel says devopsdays events are meant to be local, and that in 2013 when the first Paris event ran, people traveled from all over Europe because it was the only game in town, but that was never the goal. Daniel says roughly 75 events were planned for 2019 with no central committee authorizing them, and that if a team in Marseille wanted to run one, Daniel would share the mistakes made and then back away. Bridget relays how Matty Stratton used Minneapolis, which ran a few weeks ahead of Chicago, as a model to copy what worked and skip what didn't. Adrian adds that cross-pollination across organizations means Johannesburg could eventually help Cape Town improve too.

Running a Conference Out of GitLab

Cobus asks Adrian to walk through the process used to run the conference. Adrian says nothing is public, and the idea was to use DevOps to run a DevOps conference, after taking a management job and starting to care about project management. There are private GitLab repositories, one for the meetup and one for the conference, each with a wiki as a knowledge base (venues, sponsors, potential speakers) and issues for actionable items. The meetup gets one issue per month from a template checklist, covering the venue, time slot, meetup.com page, sponsors and announcements on Twitter.

For the conference they use a milestone per year and an issue per sponsor, labeled as a lead until they sign, with templates for each sponsor level because, for instance, a platinum sponsor gets to choose the Wi-Fi password. The same idea covers speakers: whether their bio picture, abstract and Twitter account are in place. Adrian says it's still a work in progress, and the stated goal is to automate it so a script can run next year's event. Bridget plans to deputize Adrian to help rewrite parts of the organizer guide.

Takeaways

Daniel's is a talk by Rory, who works at Microsoft, on building things usable by people regardless of their abilities, an area Daniel says they don't know much about but have some personal experience with. Adrian heard the talk earlier in the year at DevConf and says it made the point that building accessible websites is easy and "it's no longer an excuse to have a website that's inaccessible." Devi's takeaway is from Daniel's talk: "the success of any organization is based on the well-being of your people," and work-life balance "isn't just a couple of words on a piece of paper."

Adrian has a list of 10 things devopsdays could do better, and most aren't about Cape Town, so Adrian may aim at improving the global DevOps community. Bridget, who says they're the current titular lead of the global devopsdays community, would like to subscribe to that newsletter. Cobus noticed new people asking questions in the open spaces and more experienced people helping, and says the community should invest in making it easier to ask, since many people take remote jobs and disappear from the community scene. Bridget closes by praising the way the organizers pre-selected open space ideas, gave participants a web interface to add to them and sorted them ahead of time, a format Bridget hadn't seen before that got discussions going right away.

Bridget chats with Devi Moodley, Daniel Maher, Adrian Moisey, and Cobus Bernard at devopsdays Cape Town 2019.

• Devi Moodley's talk: From Science Experiment to Enterprise Rollout
• Daniel Maher's talk: What MMA taught me about working in tech

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2019 for 20% off lots of devopsdays
• Velocity Berlin Nov 4-7 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes.

Historical Context

The panel discusses the history and purpose of the report. This is the sixth year it has been produced. How did the report start and what questions is it seeking to answer?

Nicole: “Once upon a time, kids, making software was sad. Making software used to light people on fire!”

Nicole: “So why do we even do this DevOps thing? It’s because we want to make that software process easier and better.”

Jez: “These approaches have worked even in highly regulated environments. They’ve worked everywhere.”

Nicole explains that capabilities and practices are more important than tools.

Nicole: “There’s no such thing as DevOps in a box.”

The group discusses the limitations of systems data vs survey data, and the importance of collecting both. Survey data is low resolution but high signal and vice versa.

Jez: “Over-precision is something that’s a real problem in our industry.”

Nicole: [“How to Measure Anything” - Douglas W. Hubbard] ( https://www.amazon.com/How-Measure-Anything-Intangibles-Business-ebook/dp/B00INUYS2U )

The panel discusses the importance of starting from hypotheses instead of looking for any spurious correlations that exist in highly related systems.

What’s New This Year?

The panel discusses some of this year’s findings. The “retail apocalypse” over the last decade has had some surprising effects. Regulation has less of an impact than many industries assume.

Jez: “We do see high performers in large companies who are highly regulated.”

Matt: “People will sit there and say well, I have over 5000 employees and I can’t DevOps so now I have an excuse, and that’s not the case.”

Enterprise!

Change management processes are one of the key areas of difference between large and small enterprises. Jez discusses how to make that process more lightweight even in a large organization.

Jez: “Risk is also about upside risk. If you can’t move fast at delivering software, that’s a risk to your business.”

The panel discusses the importance of keeping a holistic view even in a large enterprise with specialized roles.

Nicole: “You go from change management theater to strategic change management for a massive organization. It’s scary but it is also dope!”

The report has found year after year that a slower change management process can paradoxically result in more instability, not less. The panel discusses the reasons this experiment was not successful and how enterprises can implement more nimble processes going forward.

Productivity

The panel talks about what productivity actually means. The report uses an interesting definition: “Productivity is the ability to get complex, time-consuming tasks completed with minimal distractions and interruptions.”

Nicole: “You may be just closing the tickets that are meaningless but are easy to close.”

Research shows that real productivity reduces burnout and improves work-life balance. The group discusses how to separate that real productivity from gamification and busywork. Jez talks about scaling up and how centers of excellence may not be as useful as previously thought. Nicole points out that the report has given a lot of starting points for people to begin making changes in their organization.

Remember to read the report!

First Impressions and Fifth Ones

Veronica arrived wondering how public transit works outside New York, and found a team that helped with transportation, served "the best food you've ever eaten at a conference," and even asked speakers what song they wanted when walking on stage. Jeff says the event has grown and the audio and video have gotten smoother, and calls it "the bar at which I judge all of the other conferences." Jessie praises how well organized it is and the varied audience, including a data scientist and a technical writer, as a way to avoid groupthink. Veronica's favorite conversations were in open spaces about bringing the lived human experience into a talk submission alongside the technical content.

What Could Go Wrong

Matty sees a parallel between Jessie's talk and Jeff's, since it may have seemed a good idea to put a web server somewhere it didn't belong, and asks how these things can be misused. Jeff says we misapply technology ourselves in operations and engineering, choosing tools because they're interesting without thinking through downstream consequences. The example is the recent Zoom breach, where a local web server was meant to make reinstalling the software easier. Jeff recalls Google's image-classification mistake involving Black people and says "there wasn't a Black guy in that room," so the fix is broader perspectives and thinking one step beyond the task in front of you. Trevor proposes asking "what could go wrong" seriously instead of facetiously, and Jeff says an IAM policy statement ID Jeff saw in AWS was literally "what could go wrong," with allow star star.

Jessie says many employees are process-oriented, so if asking the question isn't in the process, you do A, B, C, D and finish the task. Veronica says it comes down to psychological safety, and the innovation possible where it's okay to ask why something is the way it is differs greatly between organizations. Trevor ties it to Jeff's morning idea of an ethical body for technology: people feel pressure to finish but rarely the moral pressure to challenge getting it done. Jeff adds that standards and culture move the bar, recalling a job 10 years earlier where a password in a config file wasn't crazy, until a later job where the reaction was "what's wrong with you?" Jessie says it's culture: if the culture is to make money even at the expense of exploiting customers and nobody faces ramifications, "that's the culture that you set."

Veronica remembers a government research center that wouldn't put anything in the cloud, where losing data was an expected part of the process: "You lose data, you grieve a little, and then you put your head down and create it again." Looking back, Veronica says, it sounds bizarre.

Building Psychological Safety as a Leader

Matty asks Jeff, as someone who leads teams, what to do. Jeff's first answer is vulnerability and being unafraid to say "I don't know." On Jeff's team that's an okay answer, just not the end of the conversation. Jeff also pushes education as part of the job, asking why people who are already on call should learn a new technology on their own time, and says spending an hour or two a week reading in the kitchen is fine. The third is creating space to dissent constructively, so people who disagree know "we're still a team."

Veronica says the difference between helpful and unhelpful workplaces was time to learn and how things went wrong. In a research job, a machine had a problem nobody noticed for a month, and the team traced it back in what Veronica calls a blameless way, and everyone left knowing how the machine worked inside and out. Veronica says both fields produced the same pattern: learn at work instead of being burned out in the evenings, and when something goes wrong, focus on the process more than the person who made the commit.

Influence Without the Title

For an individual contributor without that safety, Jessie says candor with leadership can go one of two ways, depending on the person. Jeff says to think of leadership as a role and not a position, and that whoever is the emotional leader of the team can provide some psychological safety cover, such as going to the manager to say something isn't working, or inviting a quiet teammate who is stewing over a technology to speak. Veronica adds "praise in public, reprimand in private," and to remember a manager may feel just as vulnerable, so pick a moment instead of ambushing someone in the hallway.

Matty brings up a talk by Anwen Simmons a couple of years earlier on lending privilege, and says one mark of a team with a high level of psychological safety is that everyone speaks about equally. Jeff says that as experience and confidence grew, calling out a bad idea got easier, and that if fired tomorrow Jeff is confident of getting another job, so using that to shield someone else can be huge. Jessie agrees, having started out unable to be aggressive and get away with it. Trevor adds that lending privilege doesn't have to happen in front of the person: talk about their idea in conversation and remind people whose idea the team is now following.

Key Messages and Plugs

Jeff's one message is that we should take responsibility for what we create and own the consequences, because the stakes get higher as technology advances. Veronica's is "Comments are a form of documentation, and documentation is good." Jessie's is that people working on different layers of software should talk to each other more and actively listen, to solve problems between the interfaces.

Jeff is writing a book tentatively called Real World DevOps, written from the perspective of an individual contributor in imperfect organizations, due out in 2020 sometime. Veronica announces, for the first time publicly, the Quick Developer Guides YouTube channel with two other first-time speakers, a series of 10-minute videos on topics from getting started with version control to budgeting for a career change into development. Jessie, who is not actively looking for a job, plugs two books, Soul of a New Machine and Ben Horowitz's The Hard Thing About Hard Things. Jeff says, "She's not looking. She's choosing."

Matty and Trevor chat with guests Jessie Frazelle, Veronica Hanus, and Jeff Smith in front of a live studio audience at devopsdays Chicago 2019.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2019 for 20% off lots of devopsdays
• Velocity Berlin Nov 4-7 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes; early price price ends 20 September.

Keynotes in a Sentence

Liz's talk was about focusing on people, processes and culture to run more reliable systems, since "you can't really just buy an alphabet soup of tools." Alice's talk was four and a half years of pain running containers in production, and its theme is that containers are just a tool. You shouldn't deploy them because someone talked you into it in the hallway track, so assess whether they fix your problem.

Liz adds that doing it correctly takes serious investment, so why not pay someone else to run it. Alice says the startup habit used to be building things to save money, and encourages people to look at what their business does to generate revenue and whether what they're building is part of that core value. Liz connects it to Heidi Waterhouse's Ignite talk the night before, about which thing is the goose that lays the eggs, and Matty says to find out how your company makes money: "If you don't, go find out. I'll wait."

You Can't Buy DevOps

RJ's takeaways from the open spaces were that legacy and complexity exist everywhere, and that everyone's DevOps is different, with no need for Jenkins or any particular tool to know you're doing it. Liz says automation and tools help with things you already know how to do, but what you don't know how to do or what your culture doesn't represent has to be worked out first. Alice says it needs driving home because people think buying Docker gets them DevOps, and "DevOps is a culture. It is not tools." Matty's version: "you can't buy DevOps, but I sure as hell can sell you some."

Liz says you can't hope to succeed until you know what you're optimizing for, which is why the talk stressed service level indicators and measuring the customer's experience, and starting somewhere instead of targeting all the nines. RJ says clients often arrive after an executive announces "we're DevOps now," and ASBE coaches them through it while training. Most of ASBE's DevOps audience skews toward developers, and RJ asks whether the infrastructure side is agile too, since if not, "you're not doing DevOps."

Humans in the Loop

Liz says automation exists to empower people to do more, and that you won't get better decisions by taking humans out of the loop. RJ says talks at the event kept returning to the human side of automation. Matty says automation should get the human everything they need for the judgment call about whether a spike meant anything, and jokes that Skynet will eventually run PagerDuty. Bridget says this is the golden era of AI washing, and Alice's version: "It's AI when you're fundraising. It's ML when you're hiring engineers. And it's an if-else statement when you're actually making it." Liz points to Jessica Kerr's talk debunking the idea that smarter automation requires stupider humans.

Bridget also recommends Nivea Henry's talk on how the beliefs of a system's creators, conscious or not, end up in its algorithms, with the Babbage line "if we put the wrong figures in, do the right answers come out?" Liz adds that the exposure of people's information matters too, citing a large retailer that mailed pregnancy congratulations to people's houses.

What Minneapolis Does Well

RJ would steal the closed captioning, and praises the thoughtfulness and inclusion behind it. Liz says that in 2019 they only speak at events that have live or after-the-fact captioning, and that in 2020 they'll require live captioning for the room: "you wouldn't have a conference without a code of conduct. Why are you having conferences that are not accessible?" Bridget says participants who never thought captions were for them later say they caught something they missed, and Alice, who has a history of hearing problems, says the captions help people whose need isn't visible.

Bridget describes the room for nursing mothers, which costs no extra budget: organizers who won't be in their hotel rooms between about 7 AM and 10 PM have the room cleaned and leave a key card at the info desk. A nursing mother used Bridget's room that day.

Alice says the AV, lighting and stage make Minneapolis stand out, compared with Portland's first year, when the organizers borrowed a screen from a local BSides group. Bridget credits Joe, an organizer who works for PSAV, and gives the recipe: time travel to the '90s and date the cute audiovisual tech. Liz says sponsors pay for that magic, and Minneapolis has worked hard to support them. Alice likes that the sponsor hall sits between the door and the food, so attendees have to go through "the sponsor gauntlet," which Alice loves as a sponsor. RJ chose this event by seeing how many sponsors it had. Bridget notes it depends on the venue, and that local companies may send 20 people instead of 4 to the expensive coastal conference. Matty adds that the Minneapolis meetup will show 250 RSVPs on meetup.com and 245 people will show up.

One Thing for Your Own Event

Matty says the devopsdays Chicago badges say participant, not attendee, and this year none will say speaker or sponsor. Organizers may get a different color for identifying staff, and their badges will say participant too. Matty reminds everyone that for most attendees this may be the one event they attend all year. Liz wants a more explicit New York call for papers process and a concrete guide to facilitating open spaces so newcomers raise a hand. RJ says the inclusion bar has been raised and is taking notes back to the Raleigh organizers.

Alice, as MC, normalizes inclusive behavior, and each year locates the gender-neutral bathroom at a shared conference center and announces it every day. Bridget says Minneapolis signs say "stalls and urinals" and "stalls only" instead of labeling by people, which gives information instead of assumptions, like documentation that doesn't make untoward assumptions about its end user. Bridget adds that Minneapolis participants often work in large, brownfield enterprises with a lot of legacy, and are pragmatic about building on what exists.

Closing Thoughts

Alice says Bridget got their room extended when a flight was delayed three hours, and the venue televisions showed flight status for anyone waiting, small touches that make a speaker feel welcome. RJ says people were open about their transformations, which is the on-the-ground view that training clients don't usually share. Liz says that talking at the speaker dinner with Serena, who spoke about logging pipelines for security, turned up that they were fighting the same Apache Kafka problems, even though Serena is outside the startup bubble. The show ends with Matty's sign-off: "there's always DevOps in the banana stand."

Upcoming events mentioned: devopsdays Chicago, August 27th and 28th; devopsdays Portland, September 10th through 12th, with a hackathon on the first day; devopsdays Raleigh, October 1st and 2nd; and devopsdays New York in early 2020.

Bridget and Matty chat with guests Liz Fong-Jones, Alice Goldfuss, and RJ Williams, in front of a live studio audience at devopsdays Minneapolis 2019.

Liz Fong-Jones is a new organizer of devopsdays New York City. Alice Goldfuss is the MC for devopsdays Portland. RJ Williams is an organizer of devopsdays Raleigh.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2019 for 20% off lots of devopsdays
• Velocity Berlin Nov 4-7 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes; early price price ends 20 September.

Being Nice to Systems

Martin's way of getting people to behave is to understand their motivations, give them interesting things to do and show them value, the best way being to let them meet real users, since so much software goes over a wall. Martin is hired for performance and finds it leads to delivery: you can't know you made something faster without tests and measurement, which needs CI and continuous delivery, so short cycles matter. Queuing theory and Little's Law are the fundamentals behind lean delivery, and "System being code or system being people, it's still the same thing. Same mathematics apply." Mechanical sympathy, Martin notes, is a term taken from racing driver Jackie Stewart, and is really empathy.

Slack, Utilization and the J Curve

Martin describes the scientific method as the learning cycle: have an idea, design an experiment, analyze the results, and go back if wrong. The faster the cycle, the quicker you learn. Buffers exist because components run at different paces, and a team with no slack has no buffer to react, so backlog grows. Martin's example is a service that takes 100 milliseconds per job with jobs arriving at one per second, at 10% utilization. At five per second it's 50%, and as utilization climbs, response time follows a J curve, so past about 70% queues form. At 9 jobs per second, 90% utilization, a job waits about a second, and cutting the job to 50 milliseconds makes the system 20 times more responsive, not twice. That holds with the same resources and arrival rate, because it reduces utilization.

Adding resources only works if the work can be distributed without contention. Martin cites the Universal Scalability Law, which includes the coherence cost, the time to reach agreement, and says Brooks's Law in The Mythical Man-Month is the same math: adding people to a project costs time bringing them up to speed. Martin quotes Rob Pike: parallelism is doing multiple things at the same time, and concurrency is dealing with multiple things at the same time, which requires coordination. A team needs steady state, since Little's Law assumes it, and reshuffling teams constantly changes all the parameters. Conway's Law means dysfunctional team communication produces dysfunctional software.

What a Protocol Is

For Martin, a protocol is "just the rules for engaging or interaction between components in a system," whether people or software: the etiquette and the precedence, meaning behavior in order. In real life we deal with out-of-order events by handling exceptions and not expecting perfection. Martin says English is directive with little redundancy, while languages elsewhere approach things from several angles, which is less efficient but safer. Claude Shannon's information theory says communication isn't complete until feedback confirms reception, yet software is often built as if delivery just happens. Martin calls two-phase commit a protocol sold as a silver bullet that is fundamentally broken.

Zombies, Versioning and Idempotency

Martin says "nodes just dying is not a problem. Zombies are the real problem," and "Partial failure is much, much worse," so shoot a node in an indeterminate state and move on. Versioning helps at all levels: messages, protocols and state, since a process waking from a long GC pause may act as if no time has passed, and old messages from an earlier session can turn up in a new one, which Martin ties to TLS 1.3 fast-restart replay attacks. At the application level, give every message a unique, monotonic sequence number or correlation ID so duplicates, replays and out-of-order messages can be ignored, as in a banking system with transaction IDs. Martin calls this hygiene, like "a surgeon will not consider performing an operation without washing their hands," crediting Florence Nightingale, a statistician who popularized the pie chart to show infection rates. Tests, CI and monotonic sequences save time in the end.

Development as a Protocol

Martin treats testing order as a protocol of precedence: write the test after fixing a bug and it may be bogus, while writing it first and watching it fail gives falsifiability, which is scientific maturity. Jessica adds "Never trust a test you haven't seen fail." Martin says software has only been around a few decades, without generations of trade knowledge, so we should admit mistakes and shorten feedback cycles, and that every protocol needs a way to change, which is why protocols need versioning. Legal systems are codified protocols that change as we learn.

Amplification, Buffering and Canaries

Martin says, citing Dijkstra, that software is so novel that metaphors break down, and that one small change, such as a single incorrect bit, can have a catastrophic effect, more than almost anything in human history. Jessica notes that software amplification can be nearly instantaneous, while human systems take time to propagate, and Martin says buffering contains change, comparing it to shockwaves through air versus liquid. Martin gives a historical example of a society that ran by committee in peace, which slows change on purpose, and appointed a war leader only in wartime. Jessica mentions a Cloudflare outage from a pathological regular expression pushed globally, and Martin says use isolation and suitable buffering, like a canary, and Jessica says the same works for trying a process change on one team.

Slowing Down

Martin's favorite thing learned that year was an article saying people are always trying to do the right thing, including when procrastinating, which usually means insufficient information or a worry ahead, so it's a canary: "don't treat anything as bad behavior. It's just interesting information." Martin's advice: "just slowing down and pausing is actually one of the best ways to speed up."

• Martin’s talk on protocols from J on the Beach
• Mechanical sympathy:
  • https://mechanical-sympathy.blogspot.com/2011/07/why-mechanical-sympathy.html
  • https://groups.google.com/forum/#!forum/mechanical-sympathy
  • https://dzone.com/articles/mechanical-sympathy
• Universal Scalability Law, Queuing theory, Little’s Law
  • http://www.perfdynamics.com/Manifesto/USLscalability.html
  • https://blog.acolyer.org/2015/04/29/applying-the-universal-scalability-law-to-organisations/
  • https://www.infoq.com/presentations/little-usl-scalability-performance/
  • http://perfdynamics.blogspot.com/2014/07/a-little-triplet.html
  • http://www.vissinc.com/2012/09/07/littles-law-isnt-it-a-linear-relationship/
  • https://medium.com/@__bbak/dont-be-fooled-by-littles-law-18e18dba3717
• Florence Nightingale and Pie Charts
• Dijkstra on the radical novelty of software
• Image credit: Ylva, Ebba, and Kashti Grimm

Community

• Velocity Berlin Nov 4-7 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes, and Best Price ends August 2.

• For any devopsdays, try the discount code ADO2019!

Threat Modeling

Tanya first saw threat modeling when the CISO brought Tanya, newly on the security team, to a meeting with the business about what kept them up at night, and found the business worried about completely different things than a developer who just wants the app to stay up. The idea is to work out the threats to your system, then fix or protect against them, or accept the small ones. Formal frameworks like STRIDE and PASTA exist, but an informal conversation works as a warm-up, such as asking how you would hack your own app. Tanya tells of a friend with an IoT app whose view of the threats changed when Tanya asked about the users, and says it's "basically evil brainstorming, and the more point of views you have, the better."

Where to Start

Tanya suggests a half-hour to hour meeting with someone from the business, someone from tech and a security person, asking about confidentiality, integrity and availability: how sensitive the data is and where it's stored, what happens if something changes it, and what could knock it down and what you can tolerate, from a pacemaker to a neighborhood flower shop. Don't start with attack trees and a long formal process, which can scare people away. Mistakes include thinking threat modeling is the only thing needed, using a heavy process, and starting late: doing it at the end beats not at all, but it's "so much cheaper to find a design flaw really early than at the end." The OWASP application threat modeling wiki page is a good place to start.

Pushing Left, Not Shifting

If you draw the system development lifecycle left to right, from requirements and design to coding, testing and release, left is earlier. Shifting left, Tanya says, implies everyone's on board, while at previous workplaces Tanya and a friend from the Canadian government had to fight to start security earlier, so it was pushing. The blog series runs 14 posts because the fifth one was going to be 20 pages: a security activity at each stage, such as security requirements like HTTPS-only and key strength, secure design principles and threat modeling, secure coding and code review, static analysis and dynamic scanning. Tanya writes it as what Tanya wished someone had said two years earlier. Matty says we learn by teaching, and adds the swing-dance saying that "advanced dancers take beginner classes," so experts should read beginner material and just absorb it.

Serverless Security

Matty asks whether serverless is Amazon's problem. "No, serverless is not Amazon's problem." It's still an app, and functions appear and disappear, so a function that runs five minutes a week can be missed in testing, while malicious actors don't punch a clock. OWASP has a Top 10 for serverless risks, nearly the same as for web apps, and injection is still possible if a function talks to the operating system or a database. Keep an inventory, since "if you don't know you have them, how can you secure them?" Tanya has responded to an incident for an app no one knew the company had. Logging matters even when functions are fast, since without it there's nothing to investigate. Tanya says to log usernames and failed-login bursts, not social insurance numbers or dates of birth. Matty adds you can't log retroactively, and recalls an application error that simply said something has happened, and Tanya recalls apps that sent a daily ping to an inbox that everyone ignored. Matty calls it normalization of deviance.

What Developers and Ops Should Know

Tanya wishes developers knew the CIA triad, which isn't taught in school, and that the security team wants to help: "keep annoying us till you get what you need because that's our job is to help you." Tanya tells of a design that called double Base64 encoding encryption, and said another team had already built the real thing, but nobody knew who to ask. For searching, Tanya says "Whatever is at the top is the worst in regards to security every time," and recommends searching for OWASP cheat sheets for what you're trying to do, which surface the right answer. Matty adds that being good at searching has always been the secret, recalling using AltaVista in 1998.

For ops, Tanya says ops people get beaten up for unpatched systems though they work in slow waterfall settings, and that smaller, more frequent changes make emergency patches quicker. Security teams should buy ops people licenses and training for scanners like Nessus, and add container and VM scanning to pipelines. Tanya says assume breach and zero trust, recalling a network that drew zones on paper and was one flat network, and says a database should talk only to the app and its administrators, with the perimeter gone.

Mentoring Monday

Tanya mentors a few people and couldn't take on more, so started a #MentoringMonday tweet that thousands answered, now a weekly hashtag where people post what they want help with and others respond. Tanya retweets it, and women can also get a retweet from the WoSec account. It isn't only for infosec: Python, blockchain, project management and startups are welcome, since "Everyone is welcome." Tanya says to consider mentoring after two years in an industry, even if it's just naming the first book. Tanya recommends The DevOps Handbook, The Phoenix Project and Accelerate, and Matty says telling people to read those is most of Matty's job.

• Pushing Left, Like a Boss: Part 1
• OWASP Serverless Top 10 Project

Ignite the Tour and Learning From Failure

Ignite the Tour is Microsoft's 17-city, six-month, two-day event with Microsoft 365 content and Azure learning paths for migrating apps, running services in the cloud and hybrid operations. Steven's team ran the modern operations track: infrastructure as code, instrumenting applications, troubleshooting in the cloud when you can't crawl under the floor, scaling and global resilience. A favorite session, with input from Jason Hand, was about responding to and learning from failure. A colleague, David Blank-Edelman, coined the phrase "you cannot fire your way to reliable": if people fear for their jobs they'll minimize their part and stop sharing what happened, so the question becomes what about the system allowed the failure and how to engineer it out. For bad actors, Steven says to minimize the opportunity with a just culture, and recognizes that HR or the law may be involved.

Steven says a core SRE question is the appropriate level of reliability. Some software, like airplane systems or pacemakers, needs better than five nines, while others don't, and incidents can show that the impact on users is lower than expected, so a service might need four nines. Trevor's example is a coffee shop site that only matters from 9 to 5. Fail gracefully in dependent apps, or invest more where a service is critical.

Infrastructure as Code and Open Source Integrations

At Build, Steven's session was on infrastructure as code in a pipeline, focused on testing, because Steven likes code that does what it says. Steven has spent a lot of time on integrations with Ansible, Terraform, Jenkins and Spinnaker. People at Build see Azure and DevOps signs together and assume Azure DevOps is the only way to deploy to Azure, which isn't so. "We don't want you to have to change your toolchain just to be successful in Azure," whether it's Habitat, Chef, Ansible, Jenkins or Octopus Deploy. For people with nothing yet, Steven recommends Azure DevOps, but otherwise keep what works. Steven says a service company has to keep earning business: "If we can't make it easy and effective for you to consume our services, we're going to have a bad day," which Steven also liked at Chef's transition to services, unlike the old enterprise model of a pile of money up front.

PowerShell 7 and Windows Terminal

Steven is excited about PowerShell 7, the next open source drop, moving from PowerShell Core to just PowerShell, built on .NET Core 3, with expected compatibility of 70 to 90% with Windows PowerShell and a path forward from PowerShell 5.1 to bring back into Windows. Steven notes AWS bakes PowerShell into its Linux images, and describes PowerShell Summit the previous week, with a new on-ramp track and scholarships. The biggest thing for Steven is Windows Terminal, which can host WSL, command.exe and multiple side-by-side PowerShell versions, making it possible to test across versions without a box per release. It was due in June, and Steven said "I want it now." They also talk about Cortana as a framework automakers build assistants on.

Interest in SRE

Steven says a trend from the tour is interest in site reliability engineering, because operations people find its definitions more prescriptive than DevOps, which can feel fuzzy, like having a CI/CD pipeline. People ask whether SRE has to look like the Google book, and Steven says Microsoft is figuring out its own practice, and that service level indicators, objectives and error budgets give useful language for negotiating reliability. Monitoring also shifts: black-box monitoring of CPU and memory infers application behavior in a known environment, but in the cloud you need application performance metrics and the business drivers behind them. Trevor says the same question about DevOps having a uniform shape has the same answer: a core set of structures, and you figure out which fits.

Trevor chats with Steven Murawski of Microsoft about Azure DevOps, Windows Terminal and all the cool things from Microsoft Build 2019.

Tripping Onto Databases

Silvia says nobody grows up wanting to manage databases, since people trip on them and never come out, and Matty adds that nobody wants to be a sysadmin either, though an intern once said so and was hired. Silvia started with the DBA title at SendGrid, which changed to DBE as the role involved more code, and now works to expand beyond MySQL.

What a Principal Engineer Is

In Silvia's org, the role "is not like a senior, senior engineer." It is more strategic and business-oriented, built on influence without a manager's title or performance review authority. Silvia says a principal must be a force multiplier: Silvia's own shift over about a year and a half was from writing code to teaching others how to write it without trouble down the line, and in a large org principal engineers write design documents and help the team build, with mentoring the biggest part of the job. They can have a home area of the stack and still need to be T-shaped.

The skill Silvia calls most controversial is learning to talk to people other than engineers: product managers, finance and security. Principal engineers answer security and compliance questions about encryption and backups. Silvia says it would be a red flag for a principal not to understand what problem is being solved for customers.

Talking to Product

The process at SendGrid starts with a product canvas that lays out the problem and customer type, followed by solution validation with engineers, where principal engineers come in and explain, in English, what a request will cost, like a multi-region consistent database requiring Spanner and a large budget. Silvia dislikes the tech community's dismissiveness toward "just the product person," who is the voice of the customer. Silvia admits it didn't come naturally, having once been a DBA who got cranky at customers who called an API too often, until realizing that the company let them. Rate limits are an example: if you allow a behavior, you need to support it. Jessica calls it setting expectations. Silvia says "That's the biggest part of a principal engineer's job, is to make sure that what we're promising is what we're building."

Blueprints

Once product settles on what to solve, the delivery team writes a blueprint in a Google Doc covering what they're building, which helps onboarding and lets changes be explicit, with product able to see and comment on technical limits such as a service's SLA. An architecture team made of senior principal engineers reviews blueprints for one-way doors, decisions that can't be undone. It's a gate to production but not to proof-of-concept work. Silvia says the process can look waterfall-ish but tries to keep it fast, because customers build businesses on the product, and it shouldn't reach production by accident. Jessica says rewrites are appealing because it's the only time requirements are nearly complete, and Silvia adds that rewrites need a higher bar than Go being cool. Silvia notes that SLA math starts early: a service promising four nines in one region gives fewer nines across regions, and the more components, the lower the overall SLA.

Titles and Challenges

Asked about misuse, Silvia says "All over Silicon Valley," a title lottery, and "Staff engineer at Google does not equal principal engineer or architect at a company that's 18 months old." Jessica says it's about salary bands. The biggest challenge is the calendar, and finding the middle ground, since senior people become aware of other limits such as customer revenue, deadlines and security risks. Silvia's team motto is "strong opinions, loosely held," and Silvia used to be a no person earlier at SendGrid and earned flak for it. Jessica says the job is not to say no but "how do we get to yes?" Silvia also names the calendar as the best part, since it lets Silvia swap the DBA hat for a product or security one. Principal engineers partner with engineering managers, who are less in tune with the technical implementation.

The Org's Archaeologist

SendGrid hires principal engineers from outside, and onboarding is a team exercise, like Support Bootcamp, a four-day course where the support team teaches how to use every part of the product. Silvia, with the longest history, is one of the org's archaeologists who can explain what a table does. Jessica says to find such people and make friends with them. Silvia is now learning data stores beyond MySQL, and is a pragmatist: none will work all the time, and the question is whether somebody else found the sharp edges. Matty jokes that Silvia disrupts electronics, and Silvia tells of a hotel booking that failed twice and then said it was already booked, "I'm a living Jepsen," and a network flap at a Chicago data center the moment Silvia landed in Denver.

Advice

Silvia's advice for aspiring technical leaders: strong opinions loosely held, be an enabler, not the person who always says no, expect to spend a good chunk of time mentoring, and learn why things work the way they do, with healthy skepticism of new tools. Silvia is "very much of the Dan McKinley school of like, use boring tools to build cool things." Silvia recommends a talk by Tanya Reilly on glue work, and says "the internet is duct taped together with Bash."

• On being a principal engineer - blog post by Silvia

• Silvia's talks

Image credit

Community

• Velocity San Jose June 10-13 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes.

• For any devopsdays, try the discount code ADO2019!

• CFP for devopsdays Chicago: open until May 3rd

Checkouts

• Silvia: the Beyoncé movie came out on Netflix: Homecoming

• Jessica: Do something outside! It’s spring!

• Matty: Stocksy - for affordable stock imagery that benefits the artists and Super Team Deluxe for great pins and stuff

Start Small, Find Allies

Arup's step zero is to start small, with one team, project or codebase instead of a 10,000-person department, then find champions, meaning people who reply to your emails about change with enthusiasm, or build them by giving context. Arup says the journey gets lonely, and allies make it easier. Matty compares it to making a movie, since there are days you hate it. Arup adds that leaders should expect some short-term negative business impact, such as more incidents, before long-term gains, and that conviction tends to spread.

Metrics and Context

Arup likes number of deploys as a metric, because it is a proxy for operational maturity, including incident response and end-to-end ownership, and suggests moving from yearly to quarterly, not straight to continuous deployment. After that comes SLAs, SLOs and SLIs, and the Google SRE book. Arup describes a previous company that tracked revenue no longer lost to downtime, reviewed weekly with the question of whether the team had done something to change it. Matty says "you can't know if you're moving the needle if you don't have a needle," and stresses there's no magic number of deploys, so Amazon's frequency isn't a target. Arup says there is a lot of DevOps FOMO, and "you're not PagerDuty. You're not any of the companies that you saw at a conference," and quoting another company's way alienates stakeholders.

Matty adds that stage talks are success stories, because speakers feel better telling them and PR departments block failure stories. Arup says a bank has constraints but can use its customers' priority of an accurate ledger as an advantage. Matty recalls listening to sales calls at Apartments.com after an office move, learning the value of a lead, and says to learn how your company makes money. Arup says to talk to finance and product managers. Arup's examples of a right metric: Amazon's order graph, Netflix's stream starts per minute, and three at PagerDuty, endpoint availability, time from event to incident to notification, and web experience, which took years to settle. Metrics are proxies, so halving incidents doesn't double the customer experience, "but we know we've made it better."

When Metrics Backfire

Matty says to question metrics, asking why, and to avoid targets like "not slower than last month." Arup says to set metrics early, then "question the crap out of it" a month or two in, and tells a story of measuring availability as the percentage of 200 responses, when an engineer served every 500 from the load balancer as an empty 200, and the team congratulated itself until the manager asked what happened. Matty cites Andrew Clay Shafer on people working to metrics to the organization's detriment, and Jez Humble's story of a goal of one test per sprint that produced assert-equals-true tests. Matty says "There's a difference between being committed and being compliant," and that people usually just lack the why. Arup compares intent of a metric to the intent of a law. Arup tracks median pull request duration but sets no goal on it, so it stays a temperature reading, though an engineer noticed it could be gamed. Matty says don't litigate severity in an incident call, and that when teams are measured on counts of Sev 1s, the metric becomes "mean time to innocence."

Anti-Patterns

Arup says a common mistake is expecting a transformation in months: "if it took you a decade to get into a problem, it's going to take at least a year to get out of it," and you're never done. Matty adds the urge to plan everything first, recalling a large insurance company that took six months to plan a first change with Chef. Arup says rigid plans and assuming no risk are dangerous, and starting small accepts a bit of risk that de-risks later.

Treat It as an Experiment

Matty draws the parallel to chaos engineering: a hypothesis, a limited blast radius, known measures and a learning experience, though the word experiment may unsettle stakeholders. Arup likes the word because it implies humility, and tells stakeholders Arup will be first to acknowledge a failure. Arup describes PagerDuty's introduction of Chef about seven years earlier as an experiment at the lower levels of the organization that over about a year and a half became the way. Another customer of around 5,000 engineers feared engineers would quit if everyone went on call, and so tracked the number who quit and slowed the rollout if it rose, and the number went down as they invested in explaining why. Matty warns about mistaking correlation for causation, such as an acquisition happening alongside.

Wrapping Up

Arup's tactic for any leader who can't say how the business makes money is to go talk to the CFO's finance team, who are transparent, and jokes "Can't boil everything down to a single shell script, unfortunately." Matty says to find a buddy if you're not good at selling change, and to make the tent big, including testers, product and FinOps, since DevOps is unfortunately named. Arup says DevOps means pulling in whatever stakeholders you need and owning the problem, not throwing it over the fence, including to product management.

• Image credit: photo by GotCredit

Community

• Velocity San Jose June 10-13 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes.

• For any devopsdays, try the discount code ADO2019!

Why Change Is Where Failures Spread

Mark says complex systems are always failing, and failures become big problems when they cascade. If failures are independent, the probability of a combined failure drops, and change breaks independence: version 1 and version 2 of a service are coupled through their data, and clients couple through interfaces. With two versions of every client and service but one version of the data, everything is coupled to everything. So "your reliability is particularly dependent on how good your deployment process is," which Jessica promises to quote.

Mark says a deployment process needs a full feedback loop, where production tells you whether it's working and the process adapts, citing an AWS talk on closing loops and opening minds. Jessica compares it to test-driven development, asking "how will I know it works?" in terms of users, not just not crashing. Concrete examples: running old and new services in parallel, using only the old results, and comparing performance. Jessica calls it a shadow deployment, and mentions progressive delivery, a term from RedMonk. Mark says rewrites create a cliff, temporal coupling that removes independence, and that a deployment can carry its own checks, such as verifying that the first 100 queries to a spatial service return sensible shapes.

Building, Operating, Changing

Mark's three topics are building, operating and changing systems. On building, accidental coupling happens at design time, and Mark cites Michael Nygard's entity service anti-pattern, which Mark says describes nearly every microservices tutorial, with a service per noun that makes everything chatty. A smell is a state field describing which part of the system is using the object. Using an order moving from cart to shipping to archive, Mark argues for services around lifecycle stages with handoffs, which Jessica ties to bounded contexts in Domain-Driven Design. Mark's chess example: in-flight games need a fast database for 5,000 games, while 5 million historical games suit a slower searchable one, instead of a Cassandra cluster with hundreds of nodes for terabytes of data when only a fast fraction needed speed. Toy examples in tutorials teach small systems, which isn't a problem unless people assume they scale.

On operating, Mark says health checks are commonly coupled so that one service's failure makes everything report unhealthy, and the cluster shuts production down. Jessica says as soon as you automate around diagnostics, "that's production code." Serving some requests beats serving none, which means aggressive timeouts, and Mark recalls a licensing system for a large antivirus product with 40 million clients where fast checks shared a server with 45-megabyte update downloads over dial-up.

Unreliable Things Can Be the Most Valuable

Mark prefers a positive spin on reliability: "unreliable things are often the most valuable things," especially early in a product. Mark works with data scientists, and instead of handing their work off to programmers who take six months, ships their code to production independently, so that if it crashes it can't break the rest of the system. A similar approach works for feature spikes shipped in an afternoon, with the caveat that customers may come to expect a feature. Jessica notes that programmers take six months because of guidelines built for systems that can't fail, and calls it disposable code until it succeeds and needs hardening. Mark says reliability techniques can enable experiments as well as defend.

Change as Science

For change, Mark says developers who know the operational environment should write a production check with each feature. Mark cites GitHub's Scientist library and Facebook's Hack language, where a programmer adds an indicative type and ships it, production monitors whether it's ever wrong, and after weeks of consistent data it raises a change request to enforce it. Jessica summarizes it as making a prediction, injecting an observation so you can be surprised, then escalating the consequences of surprise. Mark likens it to pair programming with the user, and Jessica mentions Dark, a language and environment for changing a running system. Jessica says this works for teams evaluated on how successful users are, not on cards completed. Jessica connects Safety II, which asks what causes success, to Mark's point that we should sell these techniques for what they enable, not only what they prevent.

Kinesis and Cities

Kinesis brings together urban datasets, from weather and mobility to land use and electricity consumption, for analytics and predictions, such as the impact of a building standard on cost and greenhouse gas emissions. Mark describes a Sydney tower, where solar panels created excess power, which feeds a recycled water plant, which needed a way to use the water, so plants were grown up the side, which shades the building and reduces air conditioning. Kinesis found up to a 7 degree Celsius difference between parts of Sydney on a hot day, tied to tree canopy cover, and helped a government change policy on where to plant trees, correlating hot zones with at-risk populations. Competing customers share data in building partnerships that cut carbon footprints more than other buildings, and Mark says the work is "definitely better than selling ads."

• Mark's talk from YOW! Australia: Principles of Reliable Systems
• Progressive Delivery by James Governor of RedMonk
• [Scientist](Scientist - https://github.com/github/scientist), a Ruby library for experiments
• Dark, a language/IDE for coding into a running system
• Entity Service Antipattern by Michael Nygard
• Facebook PHP Typechecking example from Julien Verlaguet
• the excellent closing loops / control plane talk by Colm MacCarthaigh
• Safety II
• That cool building in Sydney
• Credit for the drawing of Mark goes to lyn.ia

Community

• Velocity San Jose June 10-13 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes.

• For any devopsdays, try the discount code ADO2019!

Why Jay Took an Exam

Jay needs to speak to an Azure audience credibly, and wanted a new way to learn, so returned to the certifications that Jay used in youth, having come into technology without college or an internship. The exam covered the big picture, like governance, subscriptions and business units within a subscription, and a certification was a check on what Jay did and didn't know. Jay's view: "I like them as a way to do some self-check. I don't like them as a way to justify whether or not you really do know something." Jay adds that Microsoft lets employees take the exams for free, and suggests taking a vendor's test if you can afford it.

Matty's Certification History

Matty got into the Microsoft world doing Apple field service for prepress shops whose RIP stations ran NT 3.51, then took a job at a systems integrator that needed staff to be Microsoft Certified Professionals within 60 days. Matty failed the first exam by one question, developed a method of marking questions known to be right and counting up to a passing score, and later collected an MCP+I and an MCSE of six exams, retaking only Networking Essentials, which had a slightly higher passing score. A coworker once passed all of the MCSE exams in a single day, and Matty once bet a senior sysadmin over who could pass more exams at TechEd and won, despite the sysadmin being a better engineer. Matty last certified on Microsoft in 2003 and was among the first Chef Certified Developers, an exam that was mostly lab.

Memorization and Hiring

Jay says between about 1999 and 2006 IT recruiters believed certifications were the bellwether of a good technologist, which Jay thinks was mostly memorization, and for-profit schools promised day-one jobs. Matty says the exams taught "the way that that vendor wanted you to answer the question," a stamp of approval as their representative, and that without a college degree the certs gave hiring managers reassurance. Matty adds that later exams, like the CCIE lab and Chef's, became hands-on. Jay notes interviews moved from paper credentials to practice.

Interviews That Match Real Work

Matty argues tests that forbid Google are flawed, since work involves API docs, a search window and an editor. Matty prepared for a coding interview by writing a little Go against the company's API, and the interviewers were happy to let Matty pull up the existing code. Matty says "if you want to understand how someone is going to work, you need to replicate the conditions upon where they will work," and praises an old interview where three team members posed a real problem and asked the candidate to lead. Jay says they should expect people to build things, not repeat information, and Matty adds that it's about being able to "synthesize information and not just memorize information."

Learning Again

Jay says learning new things as a 40-year-old is worth continuing, through video tutorials, better documentation, and the cert as a self-check, and that tools change, noting Corey Quinn's comment that Kubernetes won't matter in five years, even if every company keeps a legacy thing. Matty says a certificate is "a nice carrot," and what matters is that you care about it. Matty recalls rereading old Outlook email from a network engineering job about frame relay, knowledge now irrelevant.

On-Call Stories Without the Glory

Jay says on the podcast Jay tells old-days stories without glorifying them. Jay's worst on-call moment, around 2005 or 2006, was a Saturday-evening storm when the automatic transfer switch at a New Jersey data center failed to move to the UPS and generator, taking down about 2,600 servers, followed by a night of fsck and repairs. Jay's reflection: "I didn't learn a damn thing," except that the company hadn't made the right decisions. Matty describes patching SQL Slammer by logging into thousands of servers in different domains, which wasn't a battle scar worth being proud of. Matty tells of a night early in a career when a mail server went down and the boss and owner, who had said to leave it alone, took the server completely apart on the floor overnight, so the informal postmortem lesson was "keep the boss away from things that are broken." Matty also recalls a change request rejected for lacking a backup plan for turning a server off, where the answer was to turn it back on.

Wrapping Up

Jay hopes the episode said something about learning: some certifications are worth it, Amazon's for example, for certain roles, as a way to test yourself, not as a mark of being good at a job. Matty has been relearning incident management through PagerDuty training and is taking an incident commander rotation, carrying a pager again, and jokes about an Arrested DevOps Listener Certification, with questions about Bridget's preferred airline and Trevor's favorite anime.

• Previous episode: Punk Rock DevOps

• Oncall Nightmares Podcast

• Image credit: aesthetictech.net

Community

• Velocity San Jose June 10-13 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes.

• For any devopsdays, try the discount code ADO2019!

Job Shadowing

Jessie ran out of New York museums and went to Washington, where a friend at the US Digital Service gave about half a day's tour of the Pentagon, including an office called Protocol that reminded Jessie of Parks and Recreation. The next day Jessie shadowed a friend who is a surgical resident and watched a liver procedure, finding it less dramatic than Grey's Anatomy. The takeaways: the military is an intense authoritarian system, which the US Digital Service is trying to shake up with support from the Secretary of State, and a childhood wish to be a doctor gave way to being far more interested in computers. Jessie liked how respectful the residents, nurses and attending doctors were, and how knowledge moves between them by doing your time and getting promoted.

Andrew points to the book Team of Teams, by General McChrystal of the Joint Task Force, which has DevOps themes: empower the edge to make decisions, and avoid silos that deprive people of context. Jessie says the USDS practice of bureaucracy hacking is like cold-emailing another team across an organizational boundary.

Organizations as Distributed Systems

Andrew says the CAP theorem papers say nothing about computers, only nodes passing messages, which applies to humans, except that humans will acknowledge writes that never happened. A designed organization must choose consistency or availability, and partitions get injected through acquisitions and personalities. Bridget wonders about speculative execution, with many similar initiatives running in a large organization. Jessie says weird organizational structures keep coming up, such as laptop firmware teams at big vendors that apparently don't talk to each other, and Andrew says "It's almost like Conway's Law is true."

Pressure in Medicine and Tech

Andrew, whose wife is a doctor, calls residency a medieval system and an extended hazing ritual not optimized for learning or patient care, with 80-hour weeks. Jessie agrees tech stakes are lower since no life is on the line. Andrew notes that experienced doctors feel less pressure because they've run the procedure a hundred times, while tech pressure comes from anomalies nobody has practiced, for which there aren't good algorithms. Bridget adds that anything easy has been automated, leaving only mysterious corner cases.

Ethics, Supply Chains and Carbon

Andrew raises the geopolitics of Huawei and chips, and Bridget the difficulty of sourcing parts locally. Andrew says the price of computers and clothing rests on a chain of human suffering that cost-externalizing hides, and doesn't have an answer. Bridget says every choice has an impact and suggests looking at the carbon footprint of computing, such as cloud providers moving toward carbon-neutral data centers and the CoEd Ethics conference in London, while Andrew says to skip Bitcoin. Bridget mentions Astrid Atkinson leaving Google for a clean energy startup.

Designing an Organization

Asked to imagine being the CEO, Jessie says Jessie hates titles and career ladders, like the military's authoritarian rule, and cites a talk by Bryan Cantrill about everyone having the same title. Purpose and mission motivate people more than climbing a ladder. Jessie describes the n+1 shithead problem: the person one level up is a shithead, so why climb. Andrew and Bridget note incentive structures like OKRs get gamed, and Andrew says the industry's state of the art is Taylorism, which doesn't unlock the creative potential software needs.

Andrew describes the cube-square law: an ant, with an exoskeleton and no lungs, can lift 50 times its weight, while an elephant has the highest bone-to-mass ratio, and an ant scaled to elephant size would suffocate and crush its own organs. So "the majority of the DevOps presentations" from cat-pictures companies "are the equivalent of ants explaining how they can lift 50 times their body weight." You can't copy what someone did without the context of why, and an organization should evolve for its habitat: "if you have an undifferentiated mass in a human body, then that's a tumor," but an amoeba-sized organization looks like one, and that's fine. Andrew's goal is that "leaders should make more leaders, not leaders should have followers," and there's no magic formula, since scale breaks culture in phase shifts. Andrew recalls Brian Foote's talk on the Ball of Mud, asking what to call people who build such architectures: "Millionaires."

Upcoming Talks

Andrew will speak at devopsdays Atlanta, starting with chess puzzles to argue that seeing the board, as Wardley maps emphasize, isn't enough without understanding its dynamics, with John Boyd in the mix. Jessie is speaking at QCon on Intel SGX, where new information from experts changed Jessie's mind, and at dotGo on eBPF in Linux and Go, which Jessie would like to see replace iptables, which is "just archaic," though eBPF is hard to debug. Jessie explains SGX, built first as DRM for Netflix, then used for running code in enclaves in a cloud you don't trust, which just shifts trust to the hardware provider, and Andrew says "All security starts with physical security."

• This is the unofficial pilot for their new podcast: weird trick mafia

• Jessie's job-shadowing blog posts: Government. Medicine. Capitalism? (Wednesday, February 27, 2019) and Trust and Integrity (Friday, March 1, 2019)

• Andrew mentions a book: Team of Teams

• Jessie wrote a blog post about Intel SGX.

• Image credit: oldpatterns

Upcoming talks

Jessie: QCon London, dotGo Paris

Andrew: devopsdays Atlanta

Community

• Velocity San Jose June 10-13 2019 - discount code "ADO2019" gives 20% off for Gold, Silver, and Bronze passes.

• For any devopsdays, try the discount code ADO2019!

What Is DevOps?

Baron says DevOps is something you live, hard to put in words, and that the core community, by not defining it, unintentionally gatekept while vendors like AWS, Microsoft and New Relic wrote decent definitions. Baron once wrote on O'Reilly that a manifesto might help. Matty says the official definition is CALMS, culture, automation, lean, measurement and sharing, which is a starting point for conversation and not a definition. Jessica offers one: DevOps "is not a thing, capital T. It's a situation," where operations and development responsibility sit with the same people, which gives them more options. Matty adds that a DevOps team or engineer can be "an organizational smell," since it's often an automation team, and Baron suggests calling it a team that supports DevOps outcomes.

The Second Age

Baron cites Charity Majors: the first age of DevOps was operations people writing infrastructure as code, and the second is developers owning things in production, accountable for performance, operability and observability. For databases, Baron sees DBAs automating away manual work, but less often developers owning database performance. In a survey Baron tweeted, developers owning database performance of their code ranked only about halfway, which surprised Baron, who would put it at the top. Customers that succeed with VividCortex are in the second age, with developers engaged daily and DBAs having moved from guarding a walled garden to running a self-service platform, while some companies declined to replace a departing DBA. VividCortex can't push a company there, but it helps when a company is going anyway.

Matty asks how to avoid assuming software engineers know everything, citing the islands and bridges metaphor from the Effective DevOps book in place of silos. Jessica says developers have respect for DBAs that they didn't always have for ops people, and Baron says the database is scary because we push statefulness down the stack so the upper layers can be stateless, and "It's terrifying down there," with logs, file systems and RAID controllers turning into distributed systems on one server. Baron says everyone needs to model data well, and 80% of indexing and schema design is reachable by developers, who then need specialists for hard cases and to "make friends with the optimizer." Jessica says making friends with the DBAs early in a career earned query permissions. Baron adds that SQL hides intent, so as a consultant, Baron would ask what a query is trying to do.

What Doesn't Work

The first thing is that "you can't bring in a vendor to solve a culture problem." Baron thinks culture is "emergent from the ways that things are done," from incentives and what is praised, so changing incentives or making things easier changes culture, and no vendor can be asked to create culture change. Matty agrees a good vendor helps you see what to change, though you can't just rub DevOps on it, and uses the Switch story of a machine redesigned so both hands had to be away from the blade, the right way being the easy way. Jessica says Agile's manifesto was co-opted by vendors selling culture change, which Jessica calls garbage. Baron notes cloud providers sell a new way of life, like Google's customer reliability engineering team, and that professional services or customer success matter in early markets, so Baron wants customers to learn from each other.

What Works

Baron groups what's correlated with success in four buckets, people, culture, structure and process, and tooling, which align with CAMS and CALMS: deploy and release tooling, monitoring and observability that make you a better programmer, and shared knowledge and process, such as deploy confidence dashboards linked from deploy tooling. Baron mentions the full-cycle developer idea from Netflix, and Matty prefers full-cycle to full-stack, being involved through the whole cycle without being responsible for all of it. Baron says it's more important to be present with what you built and shipped and your customers' experience through time, not through layers of the stack.

• Greg Burrell at QCon SF 2018: Full Cycle Developers at Netflix
• Baron's talk from QCon SF isn't public yet, sorry. It will be. Meanwhile, here are the slides
• Bonus: Baron tweeted to ask ppl about their favorite on-call resources, and Mike Julian compiled all the answers into https://monitoring.love/articles/how-to-improve-on-call/

What is DevOps?

According to...

• Microsoft
• Amazon
• New Relic
• Atlassian "DevOps is the next most famous portmanteau next to Brangelina"

Check these out

• Jess: Quantum Mechanics Without the Observer
• Baron: Podcast rec: http://www.sceneonradio.org/
• Matty: John Allspaw on Incidents as We Imagine Them Versus How They Actually Are

Favorite Episodes

Matty picked three and then noticed that all of them were recorded solo. Let's Be Careful Out There with J. Paul Reed and Mary Thengvall is about resilience of organizations, people and teams, recorded before their Redeploy conference. Punk Rock DevOps with Jay Gordon was mostly about punk and metal, and comes with a playlist for DevOpsing. Shouting at the DevOps, a crossover with Corey Quinn's podcast, is about getting started with conference talks, and Matty heard from listeners who went on to submit abstracts.

Bridget's pick is the GOTO Chicago episode, a panel of the speakers from the distributed systems track Bridget curated, which was "like fanfic": getting speakers to discuss each other's talks. Trevor, who did five episodes, liked the theater episode, the DevOpsDays Chicago episode, and the Ignite interviews with Microsoft people about the history of Git and TFS and why Azure DevOps exists in its form. Joe picks the theater episode and again favors live episodes for being easier to edit. This year's live episodes came from Amsterdam, Minneapolis, Kansas City, Chicago and Salt Lake City, with Jessica DeVita and Nicole as guest hosts.

Trevor pitches revisiting old episodes the way Alton Brown revisits Good Eats, with commentary on what was wrong, like Trevor having Googled the definition of DevOps for the first episode. Matty suggests a riff-track crossover with Software Defined Talk. Matty also mentions a two-hour crossover episode with hosts from the Ship Show, Food Fight and Software Defined Talk that proved "mostly unlistenable," and jokes about putting it behind Patreon.

Numbers

The only figure Matty wants to share is that the show passed one million listens, "a big number that has two commas in it." The most listened-to episode of 2018 was Punk Rock DevOps, and second was Theatre Geeks Unite and Tech Over the World, which Matty says keeps coming up in Twitter threads from people who came to tech from acting.

What Each of Them Did

Bridget joked that airline status is "the gamification of poor life choices," and after cutting back in 2017, said yes to too much and is Diamond on Delta again. The year included fewer talks but more Kubernetes workshops with Jerome Petazzoni and more involvement with the product side. Matty traveled about 135 days out of roughly 200, visited more countries for the first time than in all previous years combined, went to Australia and to Amsterdam twice, finished the first full year at PagerDuty, where the advocacy team grew from one to three with a developer advocate being hired, and moved from Chicago to San Francisco. Trevor spent the year as a Chef partner architect with Microsoft, launched the Chef Automate managed service for Azure preview on stage at Ignite, started learning guitar properly, and played the Chicago House of Blues with the Chef band. Joe followed Bridget around the world, with Norway the new country, noting that "the secret pipeline is me" for getting Bridget to say yes to a conference.

Predictions for Conferences

Joe asks what the talks will be after Docker and Kubernetes. Bridget expects service mesh, noting that Kubernetes, Prometheus and now Envoy are CNCF graduated projects, and hopes for unvarnished experience reports on gluing Envoy to Kubernetes. Matty says DevSecOps will be a consistent theme and that business response matters as well as technology. Trevor predicts "the minimum viable application modernization project," where an old application shoved into the cloud gets optimized without a rewrite. Bridget says it's a familiar pattern: "You put your data center in the cloud and you wonder why the cloud is expensive." Bridget expects customers to tell these stories on stage at vendors' first-party events first.

Podcasts and Places

Bridget listens to Pod Save America, Matty to Hidden Brain, Trevor watches Techmoan on YouTube, and Joe's only podcast is Extra Hot Great. Matty's favorite trip is a tie between Sydney and Helsinki, where the organizers took speakers to a cabin two hours away for a sauna and dinner, and speakers warned Matty that Finns don't ask questions. Bridget recommends taking the Eurostar between Paris and London, with terrible Wi-Fi all the way. Trevor went to Tokyo for fun and to Las Vegas for Inspire, which went as badly as expected. Joe's highlight is DevOpsDays Amsterdam and HashiDays, including a dinner that grew from 10 people to about 45 and a restaurant that seemed glad to see them leave.

Trends That Need to End

• Bridget: events that serve only beer and pizza, which leaves out vegetarians, vegans and anyone who avoids gluten or alcohol, and food with no labels. Also lineups of all men, including vendors who send the same person to give the pitch: "remember, not everyone is you."
• Matty: speakers who pay for their own tickets, which falls hardest on underrepresented speakers, and the resume slide: "we need to kill the resume slide," or at least move it a few slides in. Bridget disagrees, since a new speaker facing an unwelcoming room may benefit from bona fides on screen, and Matty adds that boring slides are the real problem and to "use it with intent." Matty also wants better speaker intros, walk-on music like at DevOpsDays Chicago, and more Diet Coke. Bridget cautions that paying international speakers can have tax implications.
• Trevor: Bluetooth beacons on badges at Microsoft's Ignite and Inspire that tracked people through the expo hall, with the opt-out buried in fine print. Trevor also wants more citing of sources in talks and posts, and mentions using gender-neutral names in product personas, which a colleague noticed and appreciated. Bridget tries to quote people other than the authors of the same three talks everyone cites.
• Joe: slide design that can't be read from the back of a 500-seat room, and complaining about the stage lights.

Looking Forward to 2019

Trevor wants to get deeper into product ownership, do more speaking, and play in a shuffleboard tournament in St. Petersburg, Florida, having placed 25th of 64 in Chicago. Bridget plans more product work, including helping PM the Helm 3 release, and significantly less travel, which Joe doubts. Matty plans more targeted travel and more writing, including on burnout and psychological safety. Joe wants to be prepared for a fat bike race in March and for guitar lessons, and sums up the job: "I look at PowerPoint professionally, and then I get on a plane and I go look at PowerPoint recreationally."

Favorite Episodes

Matty

• Let’s Be Careful out There with J. Paul Reed and Mary Thengvall
• Punk Rock DevOps with Jay Gordon
• Shouting at the DevOps with Corey Quinn

Bridget

• GOTO Chicago 2018

Trevor

• Ignite 2018 Catch Up With Jessica Deen
• Ignite 2018 Catch Up With Ed Thomson
• Ignite 2018 Catch Up With Steven Murawski
• Ignite 2018 Catch Up With Donovan Brown

Numbers

• We passed 1,000,000 listens this year
• Most listened to episode of 2018 - Punk Rock DevOps with Jay Gordon
• Second most listened to episode of 2018 - Theatre Geeks Unite and Tech Over the World

What happened in 2018?

Bridget

• Travel. And plenty of it.

Matty

• Moved to SF and a bunch of personal stuff
• Visited 8 countries for work
• Spoke at like 20 events or something
• 5 new tattoos

Trevor

• Lot’s of time with Microsoft
• Launched a product (preview)
• Learning guitar! Formally, finally!

Podcast Recommendations

Bridget

• Pod Save America

Matty

• Hidden Brain

Trevor

• Tech Moan

Joe

• extra hot great

How Jessica Got Into Software

Jessica wanted a steady paycheck and the ability to work in any city, and a college internship at FedEx in operations research, programming in ArcInfo to make maps, showed programming was doable, with no homework at 5:30. Grad school in physics offered a long road to job security, while a developer could earn more right away. The motives were selfish, Jessica says, but now software and automation are what Jessica thinks about in spare time.

Development Automation

At Atomist, which Jessica joined almost two years earlier, the work is development automation, in which Jessica studies the domain of software development itself. Jessica is reading the Domain-Driven Design book. Everyone needs their own delivery automation, but not everyone can have a team devoted to it like Netflix, Stripe or Facebook, and Atomist's service does triggering and event correlation for event-driven delivery choreography. Matty likes the word choreography, which Matty once suggested on the show in place of orchestration, and Jessica says "when the world is ready for an idea, it doesn't come to just one person."

Asking Better Questions

Asked how the practice of software could be better, Jessica says "we don't know what we're doing yet," and there are no universal laws or silver bullets: not everyone is Netflix, so not everyone should use Spinnaker, but everyone needs their own automation system. Teams learn by asking better questions. From re:Deploy, Jessica takes the safety questions: not only what made this fail, but "what made this succeed? What made it not worse? What keeps this system together at all?" Also who keeps it running and what matters to them. Matty adds the question of how your company makes money, and Jessica adds that staying profitable is a necessary condition, but "Profit as an aim is not interesting or useful." Matty says deploying 13,000 times a minute is no use if the business isn't changing at that rate, and Jessica adds that software shipped as a part may be updated every few years. Jessica cautions against asking how to go faster, since "that's a destructive question," and asking how to go smoother instead.

What the Industry Does Well

Jessica says the industry is thinking about systems and teams, and excitedly notes non-software businesses learning from Agile: letting a team become a learning system and influence the organization turns paperwork into knowledge work. Software lets us create, modify and study very complex systems in days and weeks, not lifetimes, which makes us better at systems thinking. Jessica adds that humans are wired to track relationships, and we're learning to do that with software.

Symmathesy

Matty asks Jessica to explain the word from the re:Deploy talk. Symmathesy, Jessica says, is more specific than system, which suggests a mechanical thing that could in principle be predicted: it is "a learning system made of learning parts," where the relationships between the parts keep changing because the parts learn from each other and the whole. An ecosystem is one, as is a team or an organization, and a learning team can't avoid influencing the organization. Jessica has a short blog post with the summary.

QCon and Conferences

Jessica was track chair for QCon's DevOps track, after an earlier experience hosting the frontend track, which felt awful because Jessica is a backend developer. Jessica says "Software delivery is not a nuisance. It's not something that's in the way of your job. It is your job," and "We don't write code. We operate useful software." Jessica compares QCon, structured and practical, with Code Mesh in London, which is informal and opinionated and curated by the program committee. Matty, at a first QCon, says some of the theme language like engineers over evangelists felt exclusionary, and Jessica agrees it's the "tech rules the world thing." Matty praises the track, including the opening keynote by Nicole Forsgren and Jez Humble, and Jessica notes QCon speakers are invited, so tweeting about wanting to speak got Baron Schwartz into the track with a talk on DevOps for the database.

Greater Than Code

Jessica says the podcast is "we talk to technologists about things more important than technology." It started when the panel of a Ruby podcast lost its editor, Mandy, and started a new show around Mandy, aiming for a more diverse set of panelists. Matty notes Mandy was the first editor of Arrested DevOps. Jessica says all episodes are transcribed, and being a panelist means showing up for an hour and a half to talk to someone interesting. Matty says Arrested DevOps is more like putting on a podcast in the barn with no schedule, and Jessica calls that continuous delivery: "you make a podcast when you have a podcast."

Learning and Sharing

Jessica's career took off seven years earlier with speaking. Jessica liked being on stage but didn't feel like an expert, then learned "you don't have to be an expert in anything. You just have to learn enough to talk about it for an hour." Jessica's advice is to pick several topics you'd like to learn, submit abstracts, and build only the ones accepted, giving you a deadline and incentive, or give them at user groups first. Blogging stays around and lasts longer. "Learning begets community, which begets learning and new ideas," and the sharing makes it geometric instead of logarithmic.

Greater Than Code podcast

Check Outs:

Jessica

• Atomist.com
• glitch

Matty

• Pagerduty has open-sourced our incident response training - https://response.pagerduty.com
• Buffalo - rapid web development framework for Go - https://gobuffalo.io

Get the Bad Talks Out

Corey's rule: "The secret to giving a good conference talk is always to give a bunch of crappy ones first," then iterating until you learn a joke will never work and cut it. Matty cites Robert Rodriguez in Rebel Without a Crew saying everyone has 10 bad films in them, and Matty hopes the ten bad talks are behind now, and notes they can be versions of the same material. Matty also takes a tip from Jez Humble to iterate on roughly one talk a year. Corey's joke is to give the ten bad ones in a row and get fired.

Stories, Especially Failures

Corey's pet peeve is talks where the speaker is the hero with the solution for everyone's problem, where a coworker in the audience doesn't recall the project. Corey wants more failure stories, which are "fantastic, and no one wants to give them," including how the decision was made, since "we're all in the process of making terrible decisions." If you lack a story, start there: "People want stories. They don't want you to read a man page to them." Matty adds that stories can be fiction as a narrative device, like a made-up Acme Corp with characters, and recalls Adam Jacob telling a talk through a personal story. Matty also describes a speaker at a large enterprise whose PR department asked to remove everything that said the company did it wrong, which was the point of the talk.

Nobody Wants You to Fail

Corey says "no one in the audience is hoping a presenter screws up," and everyone is panicking the night before and nervous on stage. Matty adds "Remember always that nobody knows what you meant to do but you," as in the story of Hitchcock never watching finished films because they never matched what was in Hitchcock's head. Corey notes that the talks they hold up as examples have their speakers cringing too, and that the talks people love most are often ones the speaker would rather not show.

Preparation and Practice

Matty admits to writing slides on the plane and a history of last-minute success, and now wants to reframe it: how much better could it be with preparation? Corey does it from poor time management, not pride. Giving a talk more than once means the first time isn't practice on an audience, though Corey warns not to show another conference's title slide. Matty's advice on practice: record yourself every time and watch it, and focus on one improvement per talk, like bowling one frame at a time, so "I am focusing on one improvement." Corey jokes that a list of 18 things to fix doesn't stick.

Finding Ideas and Submitting

Corey keeps a running note on a phone, jotting ideas from conversations, and gets themes from other people's talks and consulting. Matty does best when a shower or cutting grass turns off the thinking, and does expense reports at the desk for the same effect. Giving a talk is a good way to learn: Corey's Terrible Ideas in Git came from using a tool without understanding it well, and Matty's Keeping Calm with Azure DevOps came from wanting to learn it. Both recommend a fun title and a suitably vague abstract, and waiting until it's accepted to build it out. Corey says "Submit early, submit often," about six proposals per conference, expecting one accepted. Matty adds to submit appropriately, not a shotgun, since organizers get to know you, and use the notes field to explain. Anyone who says "don't you know who I am?" has lost the fight.

Ignites, Slides and Openings

Corey says new speakers choose a five-minute auto-advancing Ignite to get it over with, which is almost always a bad idea, since "it takes me somewhere between 3 to 5 times longer to build the Ignite talk than it does the 45-minute session," with no chance to back up. Matty's friend prefers Ignites because every beat can be rehearsed. Both say reading from notes is deadening, with caveats from Corey's first Ignite.

Matty says an audience can either read or listen, so a slide should be mostly empty while talking. A plain black slide makes a technical audience think "something broke," so Matty mentions Emily Freeman's approach of using a calm photo. Corey recalls Emily opening a keynote at devopsdays Indianapolis with "the dumpster is on fire" as a fire alarm went off. Both recommend a cold open with a story, because "you have about 45 seconds to grab people's interest before they get back into their phone," and the resume slide can wait a few slides, a snap from Decker Communications training. Corey says the urge for a resume slide comes from the voice saying you're a fraud, and Matty answers that "You don't need to prove that you have the validity to be there because the talk selection committee already did that by giving you the stage." Corey: "If you're holding the microphone, you deserve to be there."

Two Kinds of Learning in 48 Hours

Katie says the conference balanced deeply technical material with emotional conversations about bias and ethics: "I didn't know what a service mesh was until yesterday," after an open space where people explained it well, and then a session on bias where people "got really emotional."

The Bias Open Space

Katie describes the session starting with a video of a race where people step forward if they've never worried about where the next meal comes from, so that by the end some are near the finish and others are far behind through no choices of their own. The point was to look at who is around you, and to use an advantage to help people behind. Open spaces on bias are usually a self-selecting group in agreement, but this one had varied opinions. Katie describes one participant who graduated in a class where the only three people hired during the recession were women, which affected that person, and an immigrant woman of color who has had a hard time navigating life and work. Katie says the aim is to listen to voices not yet heard while bringing the perspectives together.

Jeff says the conversation followed a talk by a speaker named Sonia, who directly challenged white supremacy, and Jeff's table had an intense reaction. Jeff's observation: "people have a problem accepting a narrative that runs counter to their experience," and the challenge likely filled the open space with diverse opinions, more than the usual virtue signaling. Trevor says a similar circle at ChefConf in 2015 was mostly white men, so the change is notable.

Is Software a Profession?

Jeff says the ethics part raised whether software development should become a profession with an enforced, rigorous governing body, as systems move into safety-critical settings, such as Teslas that park themselves and may run outdated software. Katie recalls a DevOps Enterprise Summit panel with a pilot, a doctor and Gene Kim on whether engineers are responsible for how software runs in production, such as in a self-driving car, and says that without a decision "we're really just a bunch of cowboys pressing buttons and then saying, well, I didn't do it, not my fault."

Matty points out Sonia's point that a consortium publishing ethics guidance was nearly all men, with one person of color, so how can you enforce ethics from one set of experiences. Jeff says messaging matters, because someone at the table heard it as white men being incapable of being ethical, when the point was that no single group can represent the whole spectrum.

Who Uses Our Software

Trevor asks whether the discussion reached who uses the software. Aaron raises the reports about ICE using Microsoft's platform, and asks what an engineer does who learns their software is helping doctors and also enabling agents to do harm. Matty says that without governance that backs you up, quitting on ethical grounds is "a place of privilege" since you're alone when you do it, and compares it to lawyers who can be disbarred and so can refuse an unethical request from a firm. Jeff adds that much of the industry rests on questionable uses of user data, so a serious ethical conversation has to acknowledge the consequences for companies like Google and Facebook.

GDPR and Privacy

Aaron says GDPR acts as a governing body for the domain world, catching registries and registrars off guard, since transferring domains needs WHOIS information that now needs the owner's permission, and some countries and domains don't care. Jeff says GDPR is noble but technologically a nightmare, since companies use delay tactics on requests for data, and "we can't just design a rule for privacy, we have to design a system of privacy," or we end up with "a EULA 2.0." Matty says making the right thing hard means people half-ass it. Aaron says reactions to disasters tend to be poor, so voluntary action is better, citing Target's breach and Have I Been Pwned. Jeff notes people still shop at Target, and Matty calls the shrug at breaches "normalization of deviance," with Equifax as the example.

Remote Work

Aaron, who works for a fully remote company, heard in another open space about companies moving toward remote by allowing one day, then three, for people with families and long commutes. One company used software to keep everyone's webcam on, and made a reluctant person join, which Aaron says "would be terrifying." Aaron adds: "You don't want to see me at 3 in the morning."

What Changed in Year Three

Dan says the event started in a much smaller theater and has kept the theater theme, with better chairs than last year, new workshops, and an Ignite karaoke tradition where a visitor from another city does one about Kansas City, and a local does one about the visitor's home. The event has "doubled pretty much our attendees in the last 3 years," and there is a video wall and roaming cameras. Dan also notes this year was the first with volunteers, who helped with registration and other tasks.

A First DevOpsDays

Monica, from a non-tech background, was struck by the amount of talent and the differing views on how DevOps is used at work, plans to tinker with Kubernetes afterward, and admits to nerding out in front of speakers. Dan shares that feeling when learning Mike Julian had submitted. Monica says the DevOps community is "a really judgment-free zone," where people ask why you think so, instead of dismissing you.

Chaos Engineering

Ana's keynote was an introduction to chaos engineering in a fun manner. The reasons start with scale: systems are getting more complex, and downtime is expensive, with a report of "around $300,000 for every hour that a company is down." Chaos engineering gets teams thinking about resilience first-hand, can help onboard people to on-call, and frees engineers for development by reducing incidents. Jessica says talks on the practice only recently appeared at devopsdays events, and Ana says antifragility is becoming part of the conversation, and that while Ana has done this for three years, the practice is about ten years old, coined by Netflix with Chaos Monkey, with other companies doing it quietly.

Matty says it's been a year of resilience and notes the re:Deploy videos had just been released, with talks from Nora Jones of Netflix and John Allspaw of Adaptive Capacity Labs, and adds that "our systems are always in a state of some type of degradation."

User Groups and Volunteers

Ben says the virtualization community has realized it has to embrace DevOps culture, and tells engineers at VMUG to learn at least scripting and DevOps principles. Ben appreciates that this event is volunteer-led.

Open Spaces

Matty says the morning talks exist to give people something to discuss in the afternoon's open spaces. Matty shares that Andrew Clay Shafer said at devopsdays Chicago that in the first devopsdays, Andrew and Patrick mainly cared about open spaces and had talks only because companies would pay for them. Monica's favorite was an open space on workforce transformation, hearing common issues that aren't usually discussed. Jessica urges people to run open spaces at their company, like a lean coffee, since "It's not PowerPoint and presentation. It's conversation." About three quarters of the room were first-time attendees, which surprised Dan.

What Surprised Them

Matty says the variety of topics made Matty better rounded, including a keynote on flintknapping that left Matty thinking "I might want to make Paleolithic tools as a hobby." Ben sees a community change, with companies once hesitant now sending many people. Ana, based in San Francisco, says it's good to see Kansas City's conversations match the ones on the coasts, about monitoring, Kubernetes and chaos engineering. Dan was surprised by the number of first-timers and of companies not historically involved in DevOps.

Asked to describe devopsdays Kansas City in three words, Ben says "Culture, fun, and networking," Ana says "that was rad," Dan says "Growth, community, and challenge," and Monica says "Mind-blowingly awesome."

Matty (with special guest host Jessica DeVita) is joined by Ana Medina, Dan Barker, Ben Clayton, and Monica Hart at DevOpsDays Kansas City 2018.

The League

Jessica is part of the DevOps advocacy team led by Donovan Brown, which the team calls the League, and whose stage motto is "rub a little DevOps on it." The team splits between dev-focused and ops-focused members, and Jessica says they try to embody the culture they talk about. Three members were at Ignite, all five at Build earlier in the year, and they keep in touch to borrow each other's knowledge even when apart. That week Jessica was in Scott Hanselman's general session showing Azure DevOps, led a 75-minute container DevOps session, recorded for Channel 9, gave an MVP interview and led another session with JFrog.

Everything Goes Back to DevOps

Asked for a favorite topic, Jessica says every topic goes back to DevOps, even Kubernetes, Helm and Draft, which are built on infrastructure as code and automated release, like the Jim Carrey movie where everything adds up to 23. Jessica mentions Donovan opening sessions with a pit-crew video. A technical glitch that day meant switching to a failover demo: "I work in ops and I consider redundancy."

Announcements Jessica lists: serverless in Kubernetes clusters, tying Azure Container Instances into the Kubernetes service, changes to Cosmos DB billing, and the rebrand from VSTS to Azure DevOps. Another announcement, which Corey Sanders tweeted after leaving it out of a blog post, is a virtual machine image builder based on HashiCorp's Packer, supported for Ubuntu 16.04 and 18.04 and in private preview, with Windows containers on the roadmap. Jessica also learned this week that a KubeCon talk on Windows containers with Patrick Lang was accepted. All sessions were recorded and uploaded quickly, so Jessica watched the recording of a session the next day and will queue some for a flight. Jessica says staying current is hard even inside Microsoft: someone asked about in-cluster ACI and AKS three hours after it was announced, and the answer was that it was news to Jessica too.

Prioritizing What to Learn

Jessica says "I need a Kanban board for my brain," and uses Trello for personal projects, demos and new content. Jessica is known for a dotfiles project called Badass Terminal, named by the community and popular on Reddit, which works on macOS, Windows Subsystem for Linux and Ubuntu, and opens PRs for the community. Trevor admits not having learned Kubernetes yet, and Jessica offers sessions and demos.

Conference Feedback

Jessica says this may be the largest Ignite ever, and the conference center is about half a mile end to end. The recurring complaint in Jessica's session feedback was that rooms were cold, which the speaker didn't control. The moral: "always relay your feedback to the people who can make that change," using the survey for logistics and leaving session feedback for speakers and content.

Kubernetes for Realsies

Jessica's content is mostly at the 200 to 300 level, and Jessica wants a real-world demo of a stateful application with a database: WordPress with Helm charts and MySQL on a persistent volume claim, then high availability, failover, load balancing, canary and blue-green deployments, using NGINX ingress or Azure's HTTP routing and Istio for traffic splitting. Jessica wonders whether the database belongs in Kubernetes or an external service. Trevor notes the trouble of finding an open source legacy application with a complex enough architecture, and used an old movie database sample from ASP.NET MVC 1. A working title for the session: Kubernetes for realsies.

Know What's in Your Images

Jessica talks about artifacts: if you use an upstream image from Docker Hub you don't control what's in it, and if you write your own, you own the vulnerabilities, such as one Alpine Linux announced the day before. A base image with vulnerabilities baked in carries them through a multi-stage build. Jessica mentions a post by Jess Frazelle that found about 700 versions of Java in one image. Jessica's practice: "build small containers." An example is cloning a private repository in one stage without baking a personal access token or SSH key into a layer, with the artifact passed to a final image. Using Debian the image is about 86 megabytes, and with Alpine about 8.

Jessica tells Trevor that as a former consultant Jessica said "good, fast, and cheap. You can only pick 2." Pulling an image from Docker Hub is cheap and fast, but you don't know if it's good. The week's big takeaway: "there's 30,000 new people that I never knew existed that are obsessed about the same technology stuff I am."

Trevor is joined by Jessica Deen at Microsoft Ignite 2018, and have a quick catch up on the event and the state of the DevOps world.

Program Manager

Edward says the role means different things for different people: feature PMs shape the direction of a component like Azure Repos, mastering the backlog, while Edward is more customer-focused and doesn't own a feature. Edward began with Git and version control customers and noticed that people now struggle more with the rest of the pipeline, so Edward looks increasingly at CI and CD.

Decades of Version Control

Edward started in scientific computing, then went to SourceGear in central Illinois, where the first product was Source Offsite, a TCP/IP server layered over Visual SourceSafe to stop its habit of corrupting itself, then worked at Teamprise, building cross-platform clients for Team Foundation Server, until Microsoft bought the company. The requirement to explicitly check out files in early Team Foundation Version Control came from huge teams like Windows and Office: the Windows source tree is about 350 gigabytes, so scanning for changes would take forever. Later versions offered scanning, since most people don't have 350-gigabyte trees.

Moving Windows to Git was a challenge, since LFS pages in large files, not a giant tree. Edward's team built the Virtual File System for Git, a kernel driver built with the Windows team, because "nobody should trust me writing kernel code." A clone gets metadata only, takes about a minute or two on the Windows repository, and files are pulled from Azure Repos as they are opened. The first try at a 350-gig Git clone finished overnight, and git status took about eight minutes.

Edward says centralized version control still makes sense for game development, since TFVC lets you lock files and handles big files, that Perforce influenced TFVC, and that "Git is just dominant." Edward thinks SVN's time is past and appreciates Mercurial users.

Getting Git Into Microsoft's Tools

Edward's friend Martin, who came over from Teamprise, pushed to bake Git into Team Foundation Server and what was then Visual Studio Team Services. Edward first thought it absurd, then they worked out how to put GPL code into Visual Studio. The lawyer for the developer division, who had been a software engineer, answered the pitch that it would be great, since the lawyer was tired of using GitHub for after-hours projects. Edward wrote much of the code, while Martin did the planning and "sneaky execution," including getting it past Steve Ballmer. The work used libgit2, which built a relationship with GitHub, and Edward later left Microsoft for GitHub's Git infrastructure team. Edward missed customer interaction and returned as a program manager, now writing code at night and on airplanes.

The Azure DevOps Split and Open Source

Edward says it isn't a rebrand, since "rebranding suggests that all we did was change the name," and it was split into separate products like Azure Boards, Repos and Pipelines, so a Jira and GitHub shop can adopt just Pipelines. Services are free for up to five users, and open source projects get 10 parallel pipelines with unlimited minutes for free. Edward moved libgit2's CI, which used several providers of mixed speed, some paid out of pocket, to Azure Pipelines, with Mac, Linux and Windows agents, for lower cost and faster builds. Edward's answer to what open source projects want: "make it free." Because the agents are real virtual machines, QEMU can run PowerPC and ARM images, which Edward says libgit2 is about to add. Edward credits the DevDiv lawyer, Jason Barnwell, with changing Microsoft's open source culture, and Trevor says the explanation of the name makes sense, though some people object to it, and some CIOs might hear DevOps in a box.

Announcements

Edward missed most Ignite announcements, as a last-day speaker. Trevor names Azure Blueprints, a governance framework that deploys a subscription with resources and policy, and, for Chef, the announcement of a managed Chef Automate service on Azure, which Trevor says is a private preview and describes as the first product Trevor has released as product owner. Edward is looking forward to seeing open source projects adopt Azure Pipelines.

Trevor is joined by Edward Thomson at Microsoft Ignite 2018, and have a quick catch up on the event and the state of the DevOps world.

A New Ops Advocacy Team

Steven's team is operations-focused advocates, started after Microsoft's developer advocacy effort, a reboot of its technical outreach about a year and a half earlier, had few people with operations backgrounds. The team became official at the end of May and spent the summer hiring and onboarding. Steven lists Jason Hand, David Blank-Edelman and Jay Gordon, with Emily Freeman joining to focus on DevOps and incident response. Steven was a developer advocate on Donovan Brown's team before that.

Microsoft's term IT pro is a huge bucket for anyone who isn't a developer, and the cloud ops advocates focus on server admins and the people moving environments into the cloud, whether on Windows or Linux. Steven's team focuses on the ops crowd around DevOps, site reliability and cloud-native, pushing practices like source control and automated delivery from a single source of truth to reduce manual intervention. All the advocates report to the same general manager, and "Our job title does not reflect the tooling and capabilities that we talk about and expose." It reflects the audience, because "the lines are blurring" between ops and developer topics. Steven likes docs.microsoft.com, which put the documentation that used to be split between TechNet and MSDN in one place, and was part of the reason to join.

Being the Conduit

Steven says the team can be found at devopsdays, SREcon, LISA, PSConf Asia, WinOps in London and Chocolatey Fest, but best online. The foundational idea is that "we exist to help be a conduit between our communities and the product engineering teams," since product teams are incentivized when people use their services, which they won't if the services don't fit existing workflows, like Terraform, Splunk or Jenkins with Azure. Problems are chances to improve documentation, bring feature requests or support bug fixes. Trevor compares it to Chef, and Steven says it's the same work as on Chef's community team. Steven is a latecomer to IT, on a third career, and says that freely shared podcasts, blogs, code samples and IRC answers made it possible, so getting paid to give back is a blessing.

Continuous Monitoring and SLOs

Steven's session was on continuous monitoring. After CI/CD comes the feedback loops of the second and third ways from The Phoenix Project, through monitoring and instrumentation. Azure Monitor ties together App Insights, Network Watcher, container and VM monitoring, and thresholds can feed back into CI/CD pipelines as quality gates, stopping deployments if rules evaluate unfavorably and replacing manual inspection. Steven's teammate David gave a session on setting service level objectives and indicators in Azure, using the same metrics and Log Analytics tooling that Microsoft uses to run Azure DevOps.

Azure DevOps and Announcements

Steven doesn't love the name Azure DevOps, which is what VSTS became, but likes the componentization into Pipelines, Repos, Boards and Artifacts, so you use what complements what you have, for example tracking issues in GitHub and not using Boards. Steven tips that people not yet moved can turn on preview features in their user settings and move at their own pace.

Favorite announcements: Chef Workstation in Cloud Shell, which Steven is increasingly using as a default place to work and can be wired into Visual Studio Code through the Azure account plugin, the public preview of a managed Chef Automate service, and the rebrand of Azure Monitor, which is run out of the Azure SRE org. Steven admits the picks skew toward personal interests, and notes the announcements came as an ebook of 50-some pages of two-sentence entries.

Slow Down and Define Quality

Steven's closing thought is that discussions of moving faster jump to automation tools, when the first step is to define what the work is and what testing and validation look like, because "you can't inspect quality into a product," a line Steven says Deming was quoting from Harold Dodge. Focus up front on what quality and done look like, and be confident what's in your environment so you can move with confidence.

Trevor is joined by Steven Murawski at Microsoft Ignite 2018, and have a quick catch up on the event and the state of the DevOps world.

Aligned Incentives

Donovan is a 20-year developer and notes that Jason's hiring interview collided on one topic, since developers usually tell Donovan the ops team is in the way, while Jason says it's usually developers applying the brakes. The two were brought into different floors of the same kind of organization. Donovan says ops teams can be rewarded for keeping the lights on, while developers are rewarded for changing things, so the two teams' bonuses conflict. Donovan tells companies to make the bonus depend on both teams reaching the same point, since "you literally told them to work against each other, and they're doing exactly what you told them to do." Jason adds that in large companies value to the customer gets abstracted away, and the fix is aligned goals and transparency, with dashboards showing the business, since SRE "isn't about reliability of infrastructure, it's reliability of the business." Trevor's first question to people is what they're incentivized on.

Azure DevOps and the Keynote

Donovan's keynote was about the rebranding of VSTS, once nearly monolithic, to Azure DevOps, whose components can now be used individually. The stage demo wired a GitHub repository to pipelines through a GitHub Marketplace extension, showing only pipelines, with no boards or repos, since customers are on GitHub. Donovan stresses it's a platform that can be extended, with the agent and tasks open source on GitHub, and Donovan learned to write a task by cloning the repo. The message is "any language, any platform," which Donovan backs with a blog post on VB6 and a demo of a Node app built on Linux and deployed to Kubernetes.

New Microsoft and GitHub

Trevor says four or five years earlier, learning C# felt like a poor choice. Jason and Donovan say the open source shift and Microsoft Learn show competition dissolving. Trevor wonders aloud whether 90-year copyrights hold society back, which Donovan says is for another show. Donovan is most excited about the announced plan to acquire GitHub, and says "I want GitHub to stay GitHub," and not be absorbed into Microsoft. Donovan cites the mission to empower every person and organization to achieve more, which "you don't do that by holding patents over them," and says it's the only Microsoft Donovan has known, joining in December 2013, just before Satya Nadella took over.

Donovan loves VS Code and once let an audience pick the language (.NET, .NET Core, Node or Java), the platform and the Azure region for a live demo, with VS Code on a Mac, a PC and Linux. Trevor notes VS Code pieces appear in Cloud Shell, and Donovan says Azure DevOps's quick edit uses Monaco, the engine behind it.

Rub a Little DevOps on It

Donovan explains the catchphrase. On the VSTS team, Donovan and two other PMs were at a large customer's headquarters trying to make a big Gradle build work with Java, knocking down pain points one after another, and blurted "we're just going to rub a little DevOps on that." The room laughed, Donovan tweeted a picture of a tube labeled DevOps, and the first time saying it live was Build 2016. Some people hate it, and one who did came to a meetup and was disappointed when Donovan forgot to say it. A birthday cake carried the hashtag. Donovan means it literally: find what hurts most in your pipeline, fix it, and when the next thing becomes the bottleneck, go back to it, which is continuous improvement.

Jason plans to bring SRE and incident response to Microsoft's advocacy on the operations side. Donovan ends by pointing to the League's website and hashtag, which reach the whole DevOps advocacy team.

Trevor and Jason are joined by Donovan Brown at Microsoft Ignite 2018, and have a quick catch up on the event and the state of the DevOps world.

Redis and the Commons Clause

Matty raises Redis's announcement of a licensing change in the previous day or two. VM explains the company posted that certain unnamed components would be relicensed under the Apache license with the Commons Clause, which prevents making money from those components. Core Redis, VM says, according to the lead developer, is and will always be BSD-3, but the Commons Clause only implies open source, since open source means the freedom to do whatever you want, including make money. VM has blogged and written on Hacker News about it, and Matty will link the post.

Open Source Is Not a Business Model

VM says people think open source is a business model, which is foolish: "Open source is not now and never has been a business model." Open core can be a business model, but VM says few open core companies are a success, and those were exits by acquihire, since the technology is already free. VM points to John Mark Walker's article series on the subject. VCs will fund an open core company through a runway, but it then struggles to sell support or add-ons. VM does make a living from free and open source software, but thinks it needs case-by-case reevaluation.

Asked for success stories, VM says a company can't release everything and expect payment without a good reason. But free and open source software is in anywhere from the 80s to 98% of software developed in proprietary companies, which is an amazing way to bootstrap innovation, while contributing back falls short. VM cites Nadia Eghbal's Ford Foundation study on the infrastructure of software development, which set off alarm bells and the sustainability conversation, and points to Heartbleed and OpenSSL as a tiny underfunded team. Silicon Valley's answer is to throw money at it. VM believes maintainers deserve pay but says what they need most is help, including learning to let others help: "It's not actually a business issue." Matty adds burnout and thankless maintenance, citing Jess Frazelle's post The Art of Closing on learning to say no.

Doc Your Shit

VM's advice to maintainers is to document. VM gives a talk on drive-through contributors, people who give one contribution and never return, and argues they're a good metric: if someone can show up, get a patch merged and leave, the project is doing something right that makes it easier for others to stay. In VM's research, drive-through contributors found no install docs, user docs or developer environment setup. Writing is hard, but it's the best force multiplier, and "even shitty documentation is probably better than no documentation at all." Matty says a blank page is intimidating and a half-written doc can be edited. VM adds that writing docs requires knowing the project, so "just write docs" is not good advice for newcomers, but fixing docs is, and as a maintainer, starting people off with a doc or two helps.

VM's first job was at a library automation software company, where VM learned to write everything in an issue tracker, treating it "like a scientist's lab book." Matty adds that doing PRs for oneself models the workflow for other contributors.

Contributing From Inside a Company

Matty asks about lawyers getting in the way of customers contributing back. VM says the answer to almost everything is it depends, IP lawyers are rightly conservative, and VM's IP advice in the book boils down to "Don't fuck with IP law," a phrase the editors keep out of print. Some companies, VM says GitHub and GitLab among them, changed employment agreements to allow contributions to free and open source software on any device at any time, and some publish a checklist for when you don't need to ask permission. But a checklist varies with each company's risk profile, and VM says borrowing another company's is a load of hooey.

Not Just Code

VM says contributing isn't only code, and that free and open source software has become programmer-centric, which is why usability is poor and there will never be a year of Linux on the desktop. Designers, usability and accessibility experts, marketers and finance people are needed, along with organizations like Software Freedom Conservancy. "Software is about all of the people that have to come together to make it happen." Matty raises the ops side, such as running crates.io for Rust with volunteers on call, and VM says at HPE, the team dedicated to upstream open source included many who ran OpenStack's contribution infrastructure. The book covers every way to contribute, not just code.

Better Talk Proposals

Matty and VM were both at re:Deploy the week before, which VM calls content-packed, relevant and well connected. VM will have reviewed over 1,000 conference proposals this year, and offers two tests: find a unique angle on a hot topic by checking YouTube for the same talk, and "tell me audience takeaways." VM says to write, "by the end of this talk, the audience will know," and to give three things people can do afterward, which beats 95% of proposals. Employers send people to learn, VM says, and a conference is not the place for story time.

Three Things From the Book

Put on the spot, VM says readers will be able to find a project that fits them, not just any project, and to accept feedback in an empathetic way and give it, because communication is the key to free and open source software.

We Have Let People Down

VM closes with a rant: people are still asking how to contribute, twenty years into open source and nearly forty into free software, and "we have let people down" by not making it obvious. The GitHub Octoverse shows millions of new public repositories a year, several million new open source projects even after discounting non-OSI licenses and forks, and "this is not sustainable." VM says "we're just going to crash under the weight of our own success," so contributors should contribute and maintainers should make it easier.

Matty is joined by VM (aka Vicky) Brasseur, Vice President of the Open Source Initiative, for a chat about contributing to open source software.

• OSI - open source initiative
• Forge Your Future with Open Source - book
• Redis blog post from VM
• Apache license + commons clause
• Ford Foundation ‘Roads & Bridges’ study by Nadia Eghbal
• Article series by John Mark Walker about how to do business with open source
• Ashley Williams PagerDuty AMA
• Public speaking repository

What Nylas Does

Nylas is an API company. Christine's thesis is that email hasn't seen much product innovation since Gmail because it has become so complicated to develop against, and Nylas exists to make it easier to build on email, contacts and calendar. Email is the lingua franca of business, and its usage keeps growing, unlike SMS.

Writing Down the Culture

Christine says early values went unsaid, which works with a few people in a room but not as a company grows, and the company doubled in size in the past eight to ten months. A change in the founding team led to writing and publishing a company handbook, which Christine declines to dig into but calls valuable. Christine describes a leadership style rooted in trust and listening: "as a founder, you are the leader of the company, whether you say so or not," and listening and empathy build trust over time, which is the foundation of a great culture.

To build trust, Christine says lead by example, give people responsibility without micromanaging, and be consistent. Asking someone to write a blog post and then rewriting it undermines trust by not letting them share their voice, and switching direction constantly makes it hard for a team to commit, so be upfront when you change your mind.

On-Call for Everyone

Christine says "our infrastructure is our product," and the team is backend heavy, so every new engineer joins the on-call rotation within about three to six months, first on a front-line rotation, and later some move to escalation. When the company hired its first full-time operations person, that person was surprised at how easy it was to ask engineers to join. Christine thinks younger engineers expect to own what they build. A past period when the company had two products, an API backend and a desktop email client, left the on-call rotation with three people on a three-week rotation, which Christine calls soul-crushing, and Christine wrote a blog post about it. Matty notes PagerDuty's incident commander rotation is three days, and that in Australia on-call pay complicates putting everyone on call.

Nylas caches a copy of the mailbox and calendar data it serves, since reconstructing a thread from IMAP can take half a dozen calls, so it runs a fleet of horizontally sharded MySQL clusters with a team of DBAs. One of them, based in Russia, volunteered to cover the nights, which helps the rotation and means fewer pages.

Going Remote

Nylas began fully co-located in San Francisco, but after Series A hiring, office space and housing costs limited growth, and Christine says that affects diversity, since people with families are at a disadvantage. About 80% of the team is still in San Francisco and four engineers are full-time remote, with more offers to remote candidates. Steps to include them: moving the Friday all-hands to directly after lunch West Coast time once people were on the East Coast, limiting time zones to North America for now, and putting cameras and area microphones in conference rooms. The company writes things down in Slack and uses Dropbox Paper as a wiki. Matty, who was remote for eight years before moving to San Francisco, adds to start with noise-canceling headphones and to remember time zones.

Why Structures Exist

Christine says scaling from two people to many shows why company structures form: past about ten people communication breaks down, and more diverse teams need to be clearer about the words they use. That has given Christine lasting empathy for other companies' processes, which can look like a black box unless you watch them grow: "there's always a reason why things end up that way." Matty's summary: "Context is a thing."

Matty has a chat with Christine Spang of Nylas about company culture and on-call techniques and war stories.

• Student Information Processing Board - MIT
• Ksplice
• Paying back technical debt - How we scaled infrastructure 20x and kept developers sane

On-Call, Culture and Aha Moments

Wes is on a second devopsdays Salt Lake City, and says last year's takeaway was the realization "we are doing things well," while this year has been more specific. Matty likes the event's practice of asking attendees for an aha moment after a talk, and gives a model of one: "interrupts actually caused me to lose productivity for 45 minutes," as against liking the Star Wars slides. Nicole recalls Alice Goldfuss's on-call selfies as a way to make a lonely practice visible. Matty remembers on-call at a bank, when the requirement to reach a terminal within five minutes meant staying home all weekend and not walking the dog more than a couple of blocks away.

Wes says Pluralsight has autonomous product teams that own applications all the way into production and carry their own on-call, and are decoupled enough that cascading failures are rare, since each team owns a vertical slice from front end to database. Wes took away from Matty's talk the idea of having the business conversation about noisy, non-actionable alerts, turning off monitoring that isn't worth making actionable. Matty adds that people assume the answer will be no, and says to push back with information reasonable people understand, such as "we have to invest time in this flapping alert because right now it's having us fly blind."

A First Conference

Chris, on Qualtrics' data platform, working on automation and alerting, came because a friend at Elastic mentioned going and the manager said to expense it. It's Chris's first tech conference, after seven years at Qualtrics and a narrow echo chamber. An aha moment was the idea that "complexity leads to fragility": a system launched before it was ready took about eight months to stabilize, and the fix reduced complexity, moving from very fine-grained message handling to a per-database queue with fairness protocols. Nicole calls it a smart tradeoff, and says Qualtrics is known for data-driven releases and got its start among professors. Chris hopes to submit a talk this year.

Organizing

Jason says the event improved by documenting everything last year, holding a live retrospective with the staff, and having a stable board, with most members there three years running and one person as dedicated CFO. The conference is about 70% funded by sponsors, with only two new vendors, and "repeat offenders" the rest. Jason's wish for the future: "Not plan it during a major security conference." Matty notes there are over 50 devopsdays that year, so avoiding overlap with other events is not always possible, and mentions Chicago's past conflict with VMworld and this year's with GopherCon. Matty also asks attendees to visit the sponsors, even without a demo.

Sponsoring a devopsdays

Matthew describes Blue Matador as "a recommendation engine" for proactive monitoring, aimed at getting ahead of reactive alerting, and works near the venue. The story in the morning keynote minute: Matthew was in the hospital with a newborn son when a former employer called, in a very Star Wars-esque manner saying "help us, you're our only hope," and later got an alert at a family sledding outing. Matty jokes that a tool should recommend whom to page based on who is doing the least important thing.

Matthew has been to the first devopsdays Salt Lake City at a different venue, and it's the first time working a booth, as an introvert. Matty and Matthew say big conferences are "trick-or-treat," with attendees collecting swag, while devopsdays visitors want to talk, helped by a sponsor passport with a drawing for stamps. Matty recalls an old manager's spouse saying Chef was "a t-shirt design company who also sold software."

State of DevOps and Accelerate

Nicole asks listeners to take the State of DevOps survey, about 20 minutes and open until June 8th, covering monitoring and observability, cloud platform, database and reliability, this year in partnership with Google Cloud. Nicole's book Accelerate, on the science of lean software and DevOps, launched March 27th, and VictorOps sponsored a book signing with free copies.

Matty chats with Nicole Forsgren, Wes Novack, a shadowy figure known only as Chris from Qualtics, Jason Vance, and Matthew Barlocker in front of a live studio audience at devopsdays Salt Lake City 2018.

• Matty's talk at devopsdays Salt Lake City 2018
• Nicole's book: Accelerate

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2018 for 20% off lots of devopsdays, 5% off GopherCon.

Jam Leomi is a past organizer of devopsdays Silicon Valley, and they just moved to Minneapolis. Christian Herro is a founding organizer of devopsdays Madison, and Debbie Gillespie is a new organizer of devopsdays Minneapolis.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2018 for 20% off lots of devopsdays, 5% off GopherCon.
• MATTY for 20% off REdeploy

Human Factors and System Safety

Paul is finishing the Human Factors and System Safety program at Lund University, a two-year program founded by Sidney Dekker, author of the Field Guide to Understanding Human Error, who built it after getting type rated on the 737 and noticing odd questions about how the airline industry works. Paul's classmates include pilots, air traffic controllers, accident investigators and doctors, among them an orthopedic trauma surgeon and an investigator who looks into deaths in live-fire military exercises. John Allspaw went through the program earlier, making Paul the second IT person. Allspaw's thesis looked at decisions under high-tempo, high-stress incidents. Paul's is on what happens after: how organizations use the artifacts of postmortems and retrospectives.

Paul says classmates in other industries deal with high-tempo, high-stakes situations and keep raising technology's effect on their fields, from electronic health records to airline flight decks and the NTSB report on the Uber crash. Paul says tech people in the program bring context, such as how software gets developed and into an emergency room. Nora Jones of Netflix is in the year behind Paul, and Paul says it opens up how you think about the world when reading postmortems and NTSB reports.

What Safety Means in IT

Matty asks what safety means beyond redundancy. Paul says many people think they're not in a safety group since they don't write flight control or nuclear software, pointing to the old Java license that excluded certain industries, and Matty mentions Chef's similar EULA. Paul gives two meanings: the business's financial safety, which means understanding the value stream, and the fact that "we live in an increasingly interconnected world," where you don't know how technology gets used. Examples: Wi-Fi pet feeders that starved pets for about a day during a cloud outage, security hardware sold with a cable service that failed to unlocked when the network was cut, and Netflix reaching out to someone who had watched one show for about 80 hours to ask if they were okay. Matty raises Google Duplex, and Paul a report of an Alexa sending a family's conversations to an employee. Matty recalls a tweet that Black Mirror is not supposed to be a how-to, and Paul says letting humans train AI without rules means bad things happen, as with Microsoft's chatbot.

Paul says more backups and redundancy was the right answer up until Three Mile Island, and that Three Mile Island, Chernobyl and the Challenger explosion made people think about the problem differently.

What Postmortems Produce

Paul's research question asks how the artifacts of a post-incident review are used in a software development and operations company. There was an industry survey and a case study of a high-performing company. In the survey, postmortem was by far the most common term at about 60%, followed by retrospective at 17%, and root cause analysis came up in write-ins. The top two items collected, each at 85 to 90%, were a list of remediation items and an event timeline. One respondent wrote in blame and fault. Some organizations record luck, meaning where they got lucky. Operations engineers update documentation notably more than managers or developers, and larger organizations are less open with their retrospective reports inside the company, which Paul says means people without a conscious bias toward transparency may unconsciously default to less.

The Company That Doesn't Chase Action Items

The case study company is one everyone would know, called DevOps Co. in the thesis for research protocol reasons. It calls the process an after-incident review, and "they don't focus on remediation items," and sometimes decide not to fix something. Paul found three things. First, the aim is context sharing, not action items. Second, the reviews continuously map the complex sociotechnical system, both technical connections between systems and human ones, such as an overseas team under attack that didn't know who to talk to, and small fires before they blow up. Third, the reviews curate tribal knowledge and culture: anyone can deploy at any time, within guidelines mostly generated by outages. Matty's question for any protective process, such as a change board, is "how many times in the past year has your process saved you?" and nobody can say. Paul says you can get the same outcomes by trusting people's gut feel and not heavyweight checklists.

Resilient People

Mary has dealt with burnout personally and looks at how to prevent it, how to treat people respectfully, and how to ensure work is valuable in context. Paul tweeted that it's an organizational anti-pattern when some people's vacations matter more than others, and Mary says you can't pay people enough to be online all the time and miss family occasions. Mary says developer relations has no real downtime, since low-season time goes to conference prep and content, and burned-out people say they can't step away because nobody else can cover. Paul describes the sharp end of the system, where work gets done and trade-offs are made without all the information, and says we assume we know how work is done there. Mary says to keep track of your work and be your own PR system to show why your work is valuable and why you need time off.

re:Deploy

Paul and Mary describe re:Deploy, August 16 and 17 in San Francisco, where the RE stands for resilience engineering, to look at the intersection of technology, organizations and people. Mary says you can't separate resilient tech, people and teams: "You can't have resilient people without having resilient teams to support them." Paul adds that many have run disaster recovery sites that don't work when turned on, so redundancy isn't the whole answer. The conference is looking for sponsors.

Matty's checkouts include Chef Workstation and Automate 2.0 from ChefConf, and the Cardhop contact manager. Mary's pre-order book is The Business Value of Developer Relations, which Matty says Matty is in.

Guests J. Paul Reed and Mary Thengvall talk about resiliency, safety, and a great new conference - REdeploy!

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2018 for 20% off lots of devopsdays
• MATTY for 20% off REdeploy

Getting Started

Thijs gives workshops and talks inside the company but hasn't made the first CFP. Jessica says it took a long time for the first one, because Jessica is very critical of self and didn't know what people wanted to hear. Friends just say it will be fine. What helped was presenting a 15-minute talk about a project at small meetups, seeing how excited people were afterward, and expanding it to 30 minutes for the first accepted CFP. Bridget says people love hearing what didn't go well, and that a talk at a meetup with good video is golden for a submission.

Where Talk Ideas Come From

Coté says that starting from zero, without a software project to show, talks can still be built: watch customers' presentations, aggregate smart things they say, footnote them, like a liberal arts paper. Once in the loop, it's "sort of like sourdough," building on conversations and questions. Coté also thinks of "a talk I would like to have," and assigns a deadline by having a conference accept it. Matty goes title first, and ideas come from the shower, mowing the grass, or explaining the same thing repeatedly, which is good fodder for a blog post or a talk. Matty says personal stories are key, and people would rather hear how someone did Kubernetes in the real world than a hand-wavy theory. A talk proposal can also be a way to learn something and show the journey. Matty's trick is a self-help book title with the word DevOps in it. Bridget describes the talk, Docker in Production: Reality, Not Hype at OSCON 2015, which came from a year of running it and the bugs and janky workarounds, and "People eat that stuff up." Coté adds that "a lot of people don't know a lot of things," so you have to remember when you were ignorant of the topic.

How Organizers Choose

Kris says talk selection is hard: you want the best speakers, and new people whose quality is unknown, so Kris looks at whether they've spoken at meetups. When Kris helped run the DevOps track at DrupalCon, speakers needed to have spoken at a local meetup first, which raised the bar but also meant the first time wasn't at the big conference. The usual suspects submit a lot, and Bridget says that's because speaking is their job. Kris says a small topic you think is trivial, like what you did over three weeks and why, may be valuable to someone else. On tech talks, Kris says the audience is not the authors of the tool, and being an early adopter means you can say how you broke it, which Kris did in talks in 2004 and 2005 about features the Linux kernel team had just released. Jessica wants more tech talks, which are hardest to do, and Kris says "Pro tip, don't do demos."

Coté describes selecting for a Pivotal track: first eliminate talks that fit the wrong room, then ask whether Coté is interested in it, consider speaker experience and then balance the slots, such as two on a DevOps topic, three on build pipelines, some miscellaneous and "a floating one," without eight talks on empathy. Matty notes you never write a talk until it's accepted and Bridget adds not to submit three if you'll only give one.

Rejection

Matty says not being accepted doesn't inherently mean the proposal was bad, since seven CI talks may have come in, though it might mean more work is needed. Looking at what got accepted helps, and if organizers offer feedback, be kind and don't litigate: a speaker once wrote back to say why the reasons were wrong. Bridget says saying who you are or where you work leads organizers to be less interested, and Jessica says they've refused a company's later submissions for that attitude. Bridget suggests asking colleagues or friends to mark up the abstract in a doc, or asking on Twitter, and Matty mentions a website of volunteer speaker mentors.

One Sentence of Advice

Kris jokingly says "don't," then "don't submit too many because you might actually end up speaking too much." Jessica says it's the same as applying for jobs: "Let us say no. Don't say no for us." Bridget: "Don't self-select out." Matty advises trying material on the road at small meetups, perhaps not in your hometown. Coté gives three: make sure the talk lines up with the conference's topics, make the title say what it's about with a keyword, and state in the abstract what people will learn, which also makes the organizers' fit decision faster.

On this episode of Arrested DevOps, Matty and Bridget discuss conference speaking with participants at devopsdays Amsterdam 2018.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2018 for 20% off lots of devopsdays, 5% off GopherCon.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Discount codes

• ADO2018 for 20% off lots of devopsdays, 10% off ChefConf, 5% off GopherCon.
• Mongodb world: JAYGORDON for 25% off

Where We'll Be

Chloe

• Sentry Scouts every month

Nathen

• London Infrastructure As Code
• ChefConf
• DevOpsDays DC
• Velocity San Jose - Workshops on Habitat & InSpec

Nell

• ChefConf May 22-25
• DevOpsDays DC June 6-7

Trevor

• Microsoft //build May 7-9
• ChefConf May 22-25

Check Outs:

Chloe:

• Search Party
• Eric Butler's talk from Def Con
• Off Book Podcast

Nathen:

• Lost in Space
• The World Cafe
• Tip...nobody pays full price for a conference

Nell:

• Dear Evan Hansen Soundtrack
• San Junipero episode - Black Mirror

Trevor:

• Barry - HBO Series
• Hazards of Love
• JSON Viewer
• Postman

Keeping State Safe

Bridget asks for tips on storing state safely. Kyle says do as much immutable as possible, since it solves consensus trivially: the Lamport proof says it takes two rounds only when proposals conflict, and immutable data has no conflicts. Alena adds Kyle's point not to trust the documentation, but to learn the distributed algorithms behind a system and test it against your own use case, and if it works, maybe keep running it. Jordan says to think clearly about which services are stateless and which are stateful, to keep the stateful list very short, and to keep the design simple, since "we're not very good at reasoning about these things on our own."

Bridget asks how to tell content you want to keep from content like spam. Jeff says you can accept writes without distributing them: rather than fanning out a tweet, analyze it, and if it looks suspicious by your heuristics, don't deliver it. Jeff says that's common, because "the acceptance of writing doesn't mean that you have to accept all the reads that are gonna come out of it," and there are far more reads than writes.

Building on Kubernetes

Bridget asks Erik how to know what to build when building systems no one has thought of. Erik says many of the pieces would have been created anyway, like schedulers, and the important thing is to build on what exists, such as Kubernetes and etcd, because that's where mistakes are made, and leveraging existing guarantees prevents repeating them and adds less complexity.

Learning From Failure

Bridget asks Kyle whether organizations learn from other people's failures. Kyle says testing under failure conditions like partitions and clock skew has been adopted rapidly, often after a customer reports a production problem. There's also a settling-in period where a system must run at scale with weird inputs to solve corner cases, which a greenfield project doesn't have. Jeff says a talk from five years earlier had to be updated, with the future idea of Docker, Mesos and data center schedulers now being reusable pieces, and "Fortunately, we still screw up the same exact things."

Alena says there are engineering errors and algorithmic errors, and that specification languages like TLA+ describe the algorithm as mathematics to catch logic errors before engineering ones. Kyle sees a continuum of safety from proof to implementation, and gives the example of a missing fsync invalidating a correct proof. Erik notes that proofs rest on known failure modes.

Unknown Unknowns in Production

Jeff says feature flags still lack a robust implementation, because they integrate with your deploy processes and user models, though metrics systems like Prometheus and Stackdriver have become reasonable, and dark rollouts need both. Even Let's Encrypt had to build its own. Jordan says Joyent's team is wrestling with how Manta behaves at scale, including whether every service for an instance will fit in DNS packets, and having to think about infinite scale from the beginning. Jeff adds that there was a patch for an old Samsung mobile platform that didn't do HTTP as expected, kept for four years.

Erik is excited about chaos engineering, and notes how shared suites can teach people failure modes like exhausting file descriptors and local ports. Jordan adds running out of TCP connections, Kyle says "What up, time wait?", and Erik explains connections closed in batches can be handed back to the operating system faster than it makes them reusable. Kyle mentions a major cloud provider that occasionally swapped pages of a VM's RAM with other VMs', which Kyle says is fixed. Jeff says anyone running about 20 machines will one day stare at a TCP state machine diagram in lsof wondering why time wait is crying.

Conference Talks as Hooks

Bridget notes Alena's cautions about Spark on Kubernetes. Alena says the demos work but production may make you the first person to run it, and gives corner cases, such as a driver pod dying as a single point of failure, and a Cassandra-to-Spark connector that didn't support Spark 2.3. Erik recalls a DEF CON badge maker's comment that talks aren't exhaustive training: they give hooks to research further, so don't feel bad if you can't follow everything in 30 minutes.

Best Advice

Erik says "it doesn't matter how many distributed systems you build or how long you do it for, it's still hard." Erik recommends understanding what consistency guarantees your use case needs, and building in backpressure, circuit breakers and idempotency from the beginning. Jordan says "everything fails all the time," and suggests bounding request time, avoiding compound operations, and remembering that adding instances won't fix a memory leak. Alena recounts Leslie Lamport telling a TLA+ workshop, "It took me 20 years to get all of the pieces," and recommends building observability into as many pieces as possible to find new unknown unknowns. Kyle points to Jeff's post Distributed Systems for Youngbloods and Kyle's own class, and says "Assume your clocks are garbage. Assume your runtimes will pause." Jeff takes the social ground: distributed systems involve more capital, teams and organizations, and without a migration plan to get people onto a new system, it goes nowhere, since there is "a technical set of problems embedded inside of a social space." Bridget concludes that distributed systems, like Soylent Green, are made of people.

Bridget curated the distributed systems track at GOTO Chicago 2018. In the last timeslot of the day, she gathered all the speakers from the track to discuss their topics.

Show notes

• Jeff Hodges - Practicalities of Productionizing Distributed Systems, 2018
• Jordan Hendricks - Designing Features for Mature Systems: Lessons Learned from Manta
• Erik St. Martin - Building Distributed Systems with Kubernetes
• Alena Hall - Distributed Data Stores on Kubernetes
• Kyle Kingsbury - Jepsen 9: A Fsyncing Feeling; Kyle's distributed systems class

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays
• Velocity EU - CFP closes May 8

Discount codes

• ADO2018 for 20% off lots of devopsdays, 10% off ChefConf, 5% off GopherCon.

From On-Call to Advocacy

Jay relays a line from a coworker, Adrian Howard, and Mary Thengvall that "developer advocacy is kind of like the good fat on certain companies, like avocados." Matty notes that advocates, evangelists and DevRel folks have very different jobs, and that a survey of the field showed little consistency, with anonymous compensation numbers ranging from $5 to $1 million. Jay adds that advocacy rarely has definable metrics, unless you work at one of the biggest companies, which counts how many people each advocate talked to. Matty says the effect can be indirect, and at PagerDuty and MongoDB the audience includes developers, admins and architects. Matty calls the role a full-duplex connection between a user community and a product team, and describes hearing from meetups what people say in words that differ from the product team's.

Jay tells of an open space MongoDB held for users at its Chicago event the day before the conference, where MongoDB's VP of engineering sat in and by the end a Jira had been written and a pull request submitted and merged. For Jay, that is a sign of success: people who use the product told the company what didn't work, and a correction followed, though "just a minor, minor change."

Asked what was hard, Jay says learning marketing and writing, including grammar, and the public face of a company. Jay compares it to an auto mechanic for 20 years who decides to start a newsletter about being auto mechanics. Jay still looks at the phone waiting for something to do, though it's no longer a pager world.

The Phantom Pager

Matty calls it a phantom limb and says it was a revelation to leave the phone on another floor after moving off on-call. Matty describes Nathan Harvey leaving the laptop home on a family vacation and then the phone, only telling the family once on the plane, and a summer trip to northern Minnesota with the phone turned off in a lodge and Slack uninstalled. Matty's point is "there's no such thing as a developer advocacy emergency," and that unplugging is "a muscle that you have to exercise." Jay admits to pressuring to get everything done before vacation, which doesn't work, since the work will be there no matter what.

Matty says a vacation can be planned for, finishing things or leaving them in a starting state, and a coworker declares email and Slack bankruptcy: if it was important, people will reach out again. Jay says that's a tough move at some places. Matty's answer is "you have to train the system": tell people you'll check email once a day, or that mail during vacation is deleted. Matty adds that this is a privilege not everyone has, and Jay says changing how you communicate without telling your team is the antithesis of DevOps. Matty compares it to getting a TiVo and feeling compelled to watch everything.

The Memcached Attack on GitHub

Jay wanted to talk about the memcached DDoS on GitHub, a 1.7 terabyte attack from UDP reflection. What troubled Jay was the reach of GitHub, for businesses, teams, students and kids learning to code, and that so many systems weren't firewalled and providers weren't blocking ports by default. Memcached, Jay says, is like a dumb protocol you can exploit, and Matty adds that you have to go out of your way to open all the ports on a cloud instance. Matty says people think a dev box doesn't matter, but compromising small things affects neighbors, in the same way two-factor on Facebook matters because of OAuth. Jay says fast and loose startups leave the security team to clean up after a unicorn shitting rainbows, and "failure is a great teacher."

Hiding Mistakes

Matty repeats that people punished for mistakes won't make fewer of them, "they're just going to become really good at hiding them," as with a leaked key nobody reports. Jay once took down a major site for a company and didn't hide it, which Jay credits partly to seniority and privilege. Matty says leaders set the expectation, even by making fun of a junior person in Slack. Matty recalls joking with a friend in Chef's chat about a Knife bootstrap flag, until a colleague pointed out that newer people only saw the joke as mockery: "you are, as a more experienced member of your team, and you're setting an example."

Fast Food and DevOps

A listener asked about the role of fast food in DevOps. Jay says every topic at a conference gets compared to DevOps, then repeats the Chipotle line. Matty disagrees, since Matty dislikes cilantro, and picks In-N-Out as the most DevOps, because it's simple with lots of hacks, and you adjust for yourself without cargo culting. Jay orders double-double animal style with fries well done, and lives in Manhattan, where delis make fast food matter less.

Punk Rock Playlist

Matty named the episode after Jay's Fugazi posts, and both give picks. Jay suggests the Void side of the Faith/Void split, Fugazi's Cassavetes, and a MongoDB metal Slack channel playlist with Entombed, Carcass, Electric Wizard, Nails, Morbid Angel and Snapcase. Matty's coding playlist has Black Flag, Misfits, Minor Threat, Dead Kennedys, Slayer and Social Distortion.

• What is Developer Advocacy? - Ashley McNamara
• Community Pulse podcast
• Developer Avocados: The Good Kind Of Fat
• The GitHub Memcached DDoS: It shouldn’t have happened
• AW.. Sh*t: Amazon S3 borkage takes down GitHub, Yahoo Mail and more

What is devrel anyway?

DevOps Music Playlist

• Wolverine Blues - Entombed
• Heartwork - Carcass
• Electric Wizard - Funeralopolis
• You Will Never Be One Of Us - Nails
• Highway 101 - Social Distortion
• Raining Blood - Slayer
• Holiday in Cambodia - Dead Kennedys
• Piles of Little Arms - Morbid Angel
• Set It Off - Madball
• Caboose - Snapcase
• Rise Above - Black Flag
• Last Caress - Misfits
• Straight Edge - Minor Threat
• Nervous Breakdown - Black Flag
• Who are You?? - Void

view on Spotify

Community & Event Stuff

Where are we going to be?

Matt will be at the devops meetup in MSP on March 20 and then home for a bit. In April he'll be at DrupalCon in Nashville, Devopsdays Des Moines, and GOTO Chicago. See mattstratton.com/speaking for more.

Open CFPs Discounts

Lots of devopsdays: https://www.devopsdays.org/speaking/

Discount codes

• ADO2018 for 20% off lots of devopsdays, 10% off ChefConf, 5% off GopherCon
• MongoDB World JAYGORDON for 25% off

Check Outs

• Muzzle - simple mac app that turns off notifications when you are screen sharing
• Tailor - iOS app that automatically detects overlapping screenshots and merges them

How the Newsletter Started

DevOps Weekly is 7-plus years old, about 360 issues, with 25,000 to 26,000 subscribers, one of those "side projects that got out of hand." Gareth's colleague Dean Wilson went to the first devopsdays in Belgium and came back saying Gareth would have liked it, and Gareth went to the second, in Hamburg. Around the same time Gareth was doing Ruby and liked Peter Cooper's Ruby Weekly, a curated weekly email, and decided to collect the things already being read and share them. The first issue was issue 0, and early on it wasn't on Sundays or truly weekly. Sunday stuck, and the regular habit helped Gareth and the readers, who read it with coffee in New York or first thing Monday in Australia.

Gareth jokes you should never put the cadence in a name: "including the cadence of something in the name is not a great idea," since a fortnightly version would need a rename, though the name also creates social pressure to keep publishing. Growth has been organic, with no advertising, plus an occasional bump when someone mentions it at a conference.

Curation and Bias

Bridget asks how Gareth's varied jobs shape the opinions. Gareth has been the first employee of a founder-led company, worked at agencies, been an in-house developer at a radio station, worked for government, and at two growth-stage technology vendors. The downside is less depth in any one of those, but the perspective helps. Gareth doesn't claim to be unbiased and will link to own work, but includes things Gareth disagrees with, using "is it interesting?" as the barometer, and doesn't cover industry news, acquisitions or hires. Bridget notices the blurbs never say who wrote an article or where they work, and Gareth says that's deliberate: "the thing is the interesting part, not the personality." Containers, serverless and Kubernetes each show up in the newsletter's volume over time, which reflects Gareth's own interests.

Gareth says DevOps as a banner has been "purposefully broad from the start," and the conversations at devopsdays have moved on, which is good, and the newsletter changes as the conversation does.

Speaking and Visibility

Gareth did meetups before the newsletter, starting with a local mailing list in Newcastle after asking a Brighton organizer how they got theirs going, and found that organizing means you always need a speaker. Travel was a bonus Gareth couldn't otherwise afford. Gareth values that devopsdays recordings, slides and videos create a large corpus of shared content. At vendors, content from practitioners already trusted by a community is more powerful than pure marketing, and at the UK government, "Us being super visible was strategy, not tactics," for recruiting and openness. Gareth also uses talks as a design tool, since acceptance validates interest and the audience gives feedback, and Bridget notes people from Chef or Puppet who propose a configuration management topic and then let the room talk.

On capitalizing DevOps, Gareth says "Patrick didn't camelCase it, therefore no one else should do," though Gareth has lost that argument often and once restored lowercase in a UK government publication that someone later changed.

Debate Club and Containers

Bridget recalls the Velocity Debate Club in Amsterdam in fall 2015, arguing whether containers and microservices are good for business, and switching sides halfway through. Gareth says taking both sides builds empathy, and that for someone trying to win an argument on the internet it doesn't matter, while for someone trying to learn it does. On containers now, Gareth says it's a messy period where it isn't about containers: Docker is trying to build a business helping large organizations modernize technology, infrastructure and applications, with gnarly problems that aren't about containers. Gareth joined as a product manager to see how the pieces work better together.

Open Source as Strategy

Bridget asks how a vendor decides what is open source, such as Moby versus Docker. Gareth, new at Docker, says "open source is strategic, not tactical," and the question is what gives the most benefit back, sometimes direct revenue, sometimes a large user community. Gareth points to recent posts by Luke Kanies, who founded Puppet, about licensing decisions, and notes earlier open source companies like MySQL AB were acquired before the industry could learn from their data. Now Hortonworks, Cloudera and Mongo are public and will produce data about the model.

The Microservice Conversation Isn't About Technology

Gareth says the microservice conversation is about people, teams and independence. Technically it adds latency and operational complexity, but it reduces complexity for an independent team: "You've moved complexity around." Bridget calls that the conservation of complexity. A 20-person company changes team structure all the time, but an organization that's been around for decades, like the UK government, can't easily reorganize to use technology the way fluid companies do. So the vendor goal is getting benefits to organizations that can't adopt early. Gareth adds that "everyone is a software company" is easy to say, but Google has about 40 to 50% of headcount in R&D, and there aren't enough software engineers to staff the FTSE 100 that way. Large organizations are also spending far more getting onto virtualization than on serverless or unikernels, and both can happen at once.

Context, Not Chasing

Bridget asks where things are going in early 2018. Gareth: "I'll be boring and say context." If you chase Kubernetes, Docker or serverless, you'll always be chasing the next, and should ask what business problem and what context you have. Gareth would also like the industry to get better at technology that stops changing, since large organizations have far more technology running than developers working on it, and asks what it would take to build software that lasts ten years. Bridget jokes that becomes a talk called Built to Last.

Gareth is helping run an ethics track at QCon London with Anne Curry, arguing ethics for technologists is becoming a pressing issue since the industry is increasingly powerful and lacks ethical education, and that conferences are a way to reach people. Gareth's pet project is Kubeval, a local validator for Kubernetes configurations.

Bridget discusses Devops Weekly with Gareth Rushgrove.

Referenced in this episode:

• Devops Weekly
• Velocity Debate Club, Amsterdam 2015
• Luke Kanies - Should You Open Source Your Product? That’s the Wrong Question

Header image

Postbox

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays
• GopherCon: CFP closes March 15; Conference Aug 27-30.

Discount codes

• ADO2018 for 20% off lots of devopsdays, 10% off ChefConf, 5% off GopherCon.

Checkouts

Gareth

• kubeval

• QCon London 2018 ethics track

Bridget

• IOT DevKit
• Male Ally Summit New York March 29

Charity Majors (Honeycomb), Jill Jubinski (Fastly), and Eric Sigler (PagerDuty) sit down with Matt to have some fun with a few great DevOps "hot takes" and talk about what everyone is doing wrong.

Referenced in this episode:

• Veteran of the Process Wars by Michael Ducy
• Crucial Conversations
• Charity's tweetstorm about on-call
• On-Call Doesn't Have To Suck by Cindy Sridharan
• The Five Love Languages of DevOps
• DevOpsDays Chicago 2016 - DevOps'ing Recruitment by Jill Jubinski
• Containers Won't Fix Your Broken Culture by Bridget Kromhout
• Matt's LinkedIn post about DevOps Engineers

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays
• GopherCon: CFP closes March 15; Conference Aug 27-30.

Discount codes

• ADO2018 for 20% off lots of devopsdays, 10% off ChefConf, 5% off GopherCon.

The Agile 2008 Story

Bridget asks for Andrew's version of the Agile 2008 gathering that led to devopsdays. Andrew calls that conference formative for reasons beyond Patrick Debois: it is where Andrew met two people who influenced Andrew, one of whom introduced Lean, while Andrew was working on Puppet and talking about agile infrastructure. There was a board of index cards for topics, and Andrew posted one about Puppet and agile infrastructure, then showed up late, because a conversation about moving from Scrum sprints toward flow, Kanban and Lean was too absorbing. Patrick had already written a paper on bringing agile practices to infrastructure and sysadmins, with small chunks and standups, ideas Andrew had been articulating from a development background. Andrew says Puppet let you bring tools and practices honed on software development to infrastructure problems.

Arguing Over Words and Attached Identities

Andrew observes that people like to argue over the meanings of words, as in the new term GitOps, when "everything's been Git-centric" from a Puppet perspective for ten years. Andrew says there's no need to argue whether it's new. Andrew spent three years on a debate scholarship, where you have to disassociate your ego from arguments, since you're assigned positions arbitrarily. Andrew says people get defensive because "they're in love with their identity" and attach it to their tasks and even the definitions of words.

Matty recalls a recruiter's post asking why DevOps engineers are so hard to find, a joking tweet that upset someone, and a long LinkedIn post about it that reached about 80,000 people in four or five days. The argumentative response came from someone who built a consulting practice on DevOps engineers being a thing. Andrew says it's worse when it's livelihood, and Matty adds that ITIL takes years to implement, so telling the person who did it they're wrong is hard. Andrew adds, speaking as the attached ego, "it's also my artwork."

Why Transformations Succeed

Andrew says the conversation about organizational learning has been sad, because it's too meta for most people, who want paint-by-the-numbers tools, and because "most organizations don't actually want to change." They get sold transformation in waves, most rooted in different metrics on top of Taylorism. Andrew has seen two archetypes of success and many failures, quoting the line that all happy families are alike: "the unhappy ones, the unsuccessful ones are all different." Success needs an impetus, either a visionary with social capital at the highest level, or an existential crisis for the business.

Andrew adds institutional theory and isomorphism, meaning organizations ending up with the same shape. Early adopters chase competitive advantage, and later adopters are motivated by legitimacy, because a trade magazine or Gartner said a buzzword is what legitimate organizations do. Andrew used to use the cargo cult metaphor, and now leans toward "they actually don't believe in the religion," and since they're not doing it for advantage they don't get one. Matty says the outcome such organizations want is to say they're doing DevOps or ITIL, not better uptime, and asks everyone, from CIO to the junior sysadmin changing backup tapes, how their company makes money. Andrew was always baffled by colleagues who focused on technology details and disregarded why the system existed, and says in an organization you either build or sell.

Complexity Below the Value Line

Bridget asks what's new to pay attention to. Andrew says not everyone needs to twiddle cgroups, and that designing an organization is like designing a web service, with inputs, outputs and throughput, noting distributed systems papers don't presuppose the nodes are computers. Matty cites Cindy Sridharan's post that everyone is not ops, and says you can't know everything about Kubernetes, and someone should know cgroups.

Andrew describes the arc from Puppet, wrangling mismatched data center boxes, to the cloud-native world that eliminates complexity by collapsing variation: identical racks, standard operating systems, fewer runtimes. Andrew recalls a talk by Jeff Hodges at Twitter arguing "polyglot is bullshit," and says people excited that Cloud Foundry and Kubernetes can patch operating systems and collect logs forget that some teams did that with Puppet and Chef years ago, only each had to solve it alone. Bridget asks if the future will be more evenly distributed, and Andrew says no: "the consolidation below the value line is going to continue ever upward," and value is created above it.

Foundations and Irrational Choices

Andrew is not always a fan of foundations, which can act as kingmakers and invite projects to chase legitimacy, as with OpenStack and possibly CNCF. There is a Cambrian explosion and then a contraction, so Andrew's advice is to experiment, fix the obvious and not go all in until it settles. Bridget notes large enterprises have pockets of everything, and Andrew says sometimes one group picks one because the other didn't.

Matty says a sales leader asked why companies have salespeople, and the answer was that humans are irrational, and recalls a blog post from Michael Hedgepeth saying NCR chose Chef because of its pre-sales experience, not steak dinners. Matty, now at PagerDuty, has spent time at Chef. Andrew says it's usually legitimacy, not the best solution, and people buy a story they can see themselves in. Matty says people think they're snowflakes, and that Sasha Bates has said every snowflake has six sides.

Team of Teams

On a morning Twitter argument about tools, Andrew says the two sides talk past each other: tools aren't enough is not tools don't matter. Andrew recommends Team of Teams, about the Joint Task Force in Iraq, where each team had the qualities you want but there was a command of teams and no horizontal collaboration. Andrew says that parallels DevOps silos: you don't want to tear down the silos or functional specialties, you want to leverage each group's context, and "The mission is not to configure servers. The mission is not to develop software." Andrew says the book argues what must change is not how workers do work but "It's how we manage people," and that one of the dimensions of organizational learning is participation in dialogue regardless of rank.

Consolidation, Gold Rushes and Witch Hunts

Bridget asks about Red Hat buying CoreOS and CloudBees buying CodeShip. Andrew expects more consolidation: "You're not gonna have 2 dozen Kubernetes startups that, like, survive," and thinks CoreOS's team and etcd fit Red Hat. On the CNCF chart, Andrew says it will be obvious what to adopt when it's too late, with patterns emerging and dominant ones legitimized. Bridget notes KubeCon proposals for projects with two contributors. Andrew describes gold rushes and witch hunts: people rushing for gold that may or may not be there, and tribes uniting against something. Andrew sees people saying DevOps is over because of containers and serverless, but "if you actually think about it as a systems thinking optimization problem, then DevOps will never die." The two laugh that getting expense reports approved is simple but not easy: "It's simple. It's not easy."

What's Next

Andrew plans a book on a five-element model of DevOps, essentially CALMS, to make the meta ideas more actionable, and has joined a reverse book club that meets every two weeks.

Bridget and Matt discuss the past and future of tech with Andrew Clay Shafer.

Previous episodes with Andrew Clay Shafer

• Platforms with Kelsey Hightower
• Devopsdays MSP 2017 with Bryan Liles & Jess Frazelle
• Devopsdays MSP 2016 with Nicole Forsgren, Charity Majors, and James Watters
• Devopsdays MSP 2015
• GOTO Chicago 2017 with Bryan Cantrill

Referenced in this episode:

• Cindy’s post about Everyone is not ops
• The full-time job of keeping up with Kubernetes
• Team of Teams: New Rules of Engagement for a Complex World

Header image

Guernica by Picasso

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays
• GopherCon: CFP closes March 15; Conference Aug 27-30.

Discount codes

• ADO2018 for 20% off lots of devopsdays, 10% off ChefConf, 5% off GopherCon.

Annie's Path

Annie has an art degree with an emphasis on film, started as a casting director, then stayed home with three boys for about ten years, blogging and running a home decorating business. Coming back in the late 30s, Annie had nothing of interest on a resume, didn't want to start at the bottom, and had lost the film network. Michael kept suggesting technology and Annie thought that was crazy, since "I have a freaking art degree, man." Then InSpec came along, which is written for non-coders and is human-readable, and Michael asked for a three-week experiment. InSpec, Annie says, "lowered the barrier to entry into technology," because writing controls to check that things are as they should be teaches IT in an inverted way, and then Chef teaches remediation.

The hardest part was belief. Annie had to get over imposter syndrome and the feeling of being the dumbest person in the room, and learned that not everybody knows everything: "It was just my own psychology, I guess." Annie's InSpec tutorials got buzz on Twitter, and Matty asked Annie onto the podcast at ChefConf 2016 about six weeks into technology, which Annie finds too cringy to relisten to. Trevor worked at Tenth Magnitude then, and that is how Annie got the job.

After the Hire

Getting the job wasn't the endgame. At a consultancy driven by utilization, Annie knew a narrow slice, cookbooks and InSpec profiles, and "I was on the bench a lot," learning on the side until a project translating ARM templates to Terraform. A colleague, Scott Nowicki, spent so many hours training Annie that Scott's own utilization took a hit that quarter, which the company treated as an investment, and Annie was much more independent afterward. Michael had warned that it takes three to six months to be effective in a new role. Annie says new people have to be scrappy and get help where they can, like the Chef community Slack. Annie adds that the scope was the surprise, since at first it was all cookbooks and InSpec, and only over two years did the context of MS SQL, PowerShell, ARM templates and Jenkins fill in.

What the First Career Brings

Sonia asks how prior careers inform the new one. Annie says casting and parenting taught reading people and empathy, "the greatest asset that I have from both of those things," while home decorating taught planning projects. Michael adds that people with art degrees had to work extremely hard to succeed, unlike the business school, and that what drew Michael to suggest technology was Annie's work ethic: "There's a lot of entitlement in IT."

Megan's Path

Megan spent 15-plus years in supply chain operations, worked as a liaison between IT and operations and loved it, but "never really felt like I was smart enough to do that kind of job," believing a computer science degree was required. After a visit where Annie talked about Ruby and Chef, and three or four conversations with Annie and Michael, Megan started self-teaching Ruby online through Udemy and Udacity. Annie then asked Megan to join the organizing board of devopsdays DFW, as sponsor liaison, which led to job interest and a scholarship from the organizers to Tech Talent South's 12-week DevOps and continuous integration course. Megan says the DevOps crowd's culture of sharing is "so open and, I don't know, inspiring," unlike the business world. Michael says "I don't think that anybody could make a career transition without a strong mentor."

Degrees, Code Schools and What Surprised Them

Trevor asks about first degrees: graphic design, marketing, English and American literature, and management information systems, and Matty's unfinished degree was in theater directing. Matty notes that privilege helped get opportunities without a degree. Megan's course was small, and Megan worked through tutorials ahead of the class with the instructor, which Michael says is the kind of person to hire from a code school. Sonia's school was a seven-month program of 60 to 70 hours a week, in four six-week modules of Ruby, Sinatra, Rails and APIs, and Sonia says the biggest takeaway over self-teaching was working in a team. The surprise at work was that "everything is brownfield," navigating code from engineers who've left, and understanding the whole workflow from mobile app through AWS and Lambda, which school doesn't cover. Annie adds that newcomers can't tell whether confusing code is elegant or just bad.

Michael says what was hard for Annie was "being comfortable with being just okay at something," since the world of technology is too large, and Matty adds that's the T-shaped idea: nobody knows everything deeply.

DevOps Native and DevOps Jumpstart

Annie coined the term DevOps native: never a sysadmin, starting in 2016 and learning only automation, configuration management and working with other teams, taught by people including Michael, Christoph Hartmann, Nathan Harvey, Dominic Richter, Trevor and Scott. Michael says job postings ask for ten years of traditional IT experience, when many of those people won't code anything, and Matty jokes "Those 10 years of experience come with 8 years of doing it wrong."

Michael and Annie describe DevOps Jumpstart (devopsjumpstart.org), which grew from the devopsdays DFW experience. It aims to sponsor people from underrepresented groups in technology with mentoring and financial help, such as childcare or scholarships, and build a repeatable pattern. Annie says the goal is life-changing for single parents or people who grew up in poverty who have the drive.

Advice

Sonia's advice: "Don't be afraid. Actually, be very afraid, but still do it." Annie recommends the book Feel the Fear and Do It Anyway.

What is it like to change careers and get into tech later in life? Annie Hedgpeth and Megan Bohl tell their stories.

This episode also features special guest Sonia Gupta!

Referenced in this ep:

• Podcast with Annie when she was brand new
• Greater Than Code episode with Scott Hanselman
• https://www.techtalentsouth.com/
• http://devopsjumpstart.org
• Feel the Fear and Do It Anyway

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays
• GopherCon: CFP closes March 15; Conference Aug 27-30.

Discount codes

• ADO2018 for 20% off lots of devopsdays, 10% off ChefConf, 5% off GopherCon.

Where We'll Be

Matt

• Time Series meetup in Denver, Feb 28
• Minneapolis DevOps Meetup, March 20

How They Came to Go

Brian saw the Go announcement in 2009 and played with it, and six to eight months later had a problem that needed concurrency: a big Ruby on Rails monolith that wasn't meeting its SLAs while calling many data sources. Go "blew the doors off of what I expected out of concurrency," and Brian has been a fan since. Erik remembers a group at Disney in 2009 playing with it without seeing a selling point. Two years later, Erik was job hunting, interviewed at Brian's company, and was asked to maintain a service Brian had written, too busy as CIO. That meant learning Go in the days of makefiles and a language changing about weekly.

Brian explains that before 1.0, releases were numbered like R56, which was the first version they put in production. The team shipped go fix, which rewrote old code to the new syntax, and Erik recalls only one case needing manual fixing, when rune was introduced. Brian's view is "ship it with a tool like go fix." Erik says that care for the developer is part of the love. Since the 1.0 API freeze, "anything that compiles on Go 1.0 will compile on Go 1.10."

Why Concurrency Matters

Bridget asks for the ops-audience version. Brian says Ruby and Python historically execute on one core at a time, and Erik adds the global interpreter lock, so only one thread interprets code at a time, though blocking I/O can run in parallel. Go has goroutines, which Brian calls lightweight, low-memory threads, with many running on one OS thread. Erik adds that the concurrent code is easier to reason about, since threading was added to many languages after the fact while Go designed around concurrency from the start. Bridget calls that concurrency as a first-class citizen, and Brian asks if they can record that and put it on the GopherCon website.

Go in Action and the Next Book

Brian and Erik and a third author wrote Go in Action, published in November 2015. Since the API froze at 1.0, Brian says the book isn't out of date at all after three years. Brian has time booked that afternoon to finish an O'Reilly proposal with a co-author, which Brian won't describe, and would pick an otter for the animal. Erik says the original book began with wanting to tech review a stalled Manning Go book, and being asked if anyone they knew would write it: "screw it. Let's do it. How hard could it be?" Both say writing is like a baby: after swearing never again, you forget how painful it was.

Go Time FM

The pair co-host Go Time FM with Carlesia, which came out of a relationship with Changelog, who invited them on to talk about GopherCon and later wanted to produce other podcasts. Erik says both groups wanted a Go podcast, and they merged. They want three co-hosts each time, and when one is missing, a guest sits in, such as Ashley McNamara, Kelsey Hightower or Scott Mansfield, and when more than one is out, they skip the episode. Erik says the format is "like we're all sitting around just having a conversation at the dinner table," with only loose notes, and the show had just reached its 65th episode.

Starting GopherCon

Asked how GopherCon started, Brian says "It was a dare." Brian and Erik had been saying for two years that there should be a Go conference, and someone on Twitter said they should run it, so Brian registered a domain name. That was mid-2013, and they had hoped for 200 or 300 people and sold out at 750, needing to rearrange the venue. They had reserved a hack day for people waiting for flights and expected 100 people, but far more stayed, leaving them wondering how to feed everyone. Hotel attrition in the first years meant close calls, with Erik saying "Brian, we're gonna lose our houses, man," and the second year lost about $10,000, before hiring Convention Designs in Colorado. They remember committers to the Go project stuffing 750 swag bags on a production line in the Denver Marriott.

Attendance was about 1,200 in 2015, 1,400 in 2016 and just over 1,500 the previous year including staff, with about 1,800 expected. The dates were August 27 to 30 in Colorado: a workshop day, two days of talks and a community day. The CFP runs on PaperCall, which Brian says filled a hole in CFP management, and Erik recommends the community day, with a room of electronics programmable in Go and a Go contributor room last year where the Go team helped people get first patches in. Brian mentions a post saying conferences are dead, and agrees that loosely structured time to network matters as much as talks. They also lend the GopherCon name to events outside the US, provided there's a code of conduct and no selling speaking slots.

Microsoft and the Community

Bridget asks whether Microsoft running a conference for a Google-born language makes it a corporate effort. Erik says from the beginning it was community-first: "we won't sell a speaking slot," and the conference has lost sponsors over it. GopherCon is run by Gopher Academy, which employs Erik and Brian legally, and Microsoft's role is letting them do it on company time. Brian says the interview made clear Microsoft wanted Brian "not in spite of the fact that I ran GopherCon, but because of it," and says "there's no forced shilling." Erik's reaction to the arrangement: "Pinch me."

Virtual Kubelet

The new team's first project is the Virtual Kubelet, an interface anything can implement to look like a node on a Kubernetes cluster, be it a container, a virtual machine or a bash prompt. Brian says the Hyper.sh team built one to run virtual machines under Kubernetes, and an Amazon group is also working on the spec, with the goal of tools useful beyond Azure and of patching other projects to work better on Azure. Erik wrote a blog post explaining it, and asks people with a project to come say so.

Bridget and Erik also discuss Ashley McNamara's Gopherize Me, which began with Ashley making gophers for Brian and Erik and, Erik says, became a tool built with Matt Ryer in a day.

Bridget discusses all things Go with Brian Ketelsen and Erik St. Martin.

Referenced in this ep:

• GoTimeFM podcast
• GopherCon
• Go in Action
• Microsoft CDA team
• Gopherize Me by Ashley McNamara

Artwork credit: Ashley McNamara's Gophers art repo on github

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays
• GopherCon: CFP closes March 15; Conference Aug 27-30.

Discount codes

• ADO2018 for 20% off lots of devopsdays, 10% off ChefConf, 5% off GopherCon.

Checkouts

Brian

• kubed-sh - web-based Kubernetes distributed shell
• Isomorphic web applications using Go

Erik

• Blog post about virtual Kubelet
• Building abstraction atop k8s
• Machine learning setup on k8s
• Reverse engineering circuit boards (next book on my reading list)

Bridget

• Honeycomb engineering blog
• Bombsheller leggings (again! because people ask for this at every conference!)
• Leakproof Nalgene water bottle

Favorites and Numbers

Bridget's favorites are the live recordings at GOTO Chicago and the one-on-one fireside chats. Matty agrees, and recalls that the DevOps track day at GOTO Chicago meant recording six episodes in a row, which Bridget vows never to do again, along with the live call-in show with Nicole Forsgren and the fireside chat with J. Paul Reed in episode 97, an episode the two had been trying to record for years. Joe, who edits, likes live episodes because people in the same room get visual cues and there are fewer awkward pauses.

Matty's numbers: about 22,000 website visitors, up from 16,000, about half from search, and Twitter still at 6 percent. Listens were about 263,000, up roughly 30,000 a year for three years, with about 750,000 total episode downloads. Matty's usual caveat is that download counts don't say anyone listened, though Apple's podcast app will now report minutes listened. The most listened-to and most watched episode was Old Geeks Yell at Cloud with Andrew Clay Shafer and Bryan Cantrill, and Matty notes that Bryan's episode was also the most watched video last year. The hosts disabled comments on the site, since nobody used them and Twitter is the place to talk.

What Each Person Did in 2017

Bridget started a developer advocacy job at Microsoft in September, kept scaling DevOpsDays to 51 events on 6 continents, and gave far fewer talks, on more program committees. Matty became DevOps Evangelist at PagerDuty at the beginning of December, a former sponsor of the show, after realizing that much of what Matty did in spare time had become a job. Matty spoke at DevOpsDays Denver and ChefConf, visited the first DevOpsDays Hartford, and ran Chicago, calling the result acceptable. Trevor wrapped up a first year at Chef, spoke at DevOpsDays Dallas, and moved back to Chicago. Joe hit 20 years at the same employer, which has changed names and owners, and took trips to Iceland and on a cross-country drive in an electric car. Matty also recommends an escape room run by a friend.

Ask Us Anything

Matty got the idea from the Go Time podcast's AMA episode and was glad nobody asked anything too personal. Joe reads the questions.

What does a developer advocate do? Matty, new to it at PagerDuty, describes a bidirectional job: communicating to the community and customers, and bringing what's learned back to product teams. That's hard for PagerDuty because "it's really hard to do feature discovery in a product that you use during a firefight." Matty also wants to develop practices, like incident command, that are good for the community whether or not they use the product. Bridget sees advocacy as being the voice of the customer inside the company, which doesn't fit well under sales or marketing. Matty adds that Mary Thengvall is writing a book on developer advocacy.

How did you come to work for Microsoft? Bridget has never owned a Windows machine, and got a Mac with a Microsoft asset tag for a job in Linux and containers advocacy on Azure. Organizations that have been around a long time change, and joining at the right time to shape that is exciting. Bridget points to Azure Container Instances, Kubernetes work and the Virtual Kubelet, with colleagues hacking together in Austin before KubeCon.

Planes. Bridget recommends Michael Cote's podcasts, since they're soothing and fine to fall asleep to, and Matty picks books already read, with a sleep timer and a bookmark, and recommends the audiobook of The Goal done like a radio play, a pick Trevor shares. Trevor plays Nintendo Switch on planes. On drinks, dehydration is a rookie move, and Matty's pick is ginger ale, where you ask for the whole can. A side trip into small conferences, including a sci-fi convention in Fargo and one in Cardiff, runs long enough that Matty predicts a drop-off in the analytics.

What is Pete ChessBot's purpose? Matty registered it under the table at dinner with Fletcher Nichol and Pete Cheslock at the Agile Conference in Orlando, then forked a Markov bot. Its purpose is to cause mirth and to confuse people: someone once talked to the bot for an entire event while trying to reach Pete.

Can you buy better microphones? The hosts have good microphones; the guests are the problem, and so is Hangouts, which Bridget keeps as the lowest common denominator while Matty wants double-ended recording, where each person records locally and uploads as they go. Matty's example is the episode with Paul Reed, at high quality, and another with a guest in China on AirPods that came out better than this one will. The catch is that double-ending can't do video or livestreaming. Matty says every episode gets about 7,000 listens and maybe 200 views, though YouTube has close to 1,000 subscribers and Bridget says those viewers come up at conferences. Joe counts 18 episodes released, 10 of them recorded away from home, and notes that the most popular episode of last year was recorded on Bridget's iPhone. Matty's summary: "compromise is when you discuss things until you reach a point when nobody is happy."

Advice for new graduates. Trevor says go to meetups and "don't blindly accept the status quo," like a two-week change freeze nobody remembers the reason for. Matty pushes back: new people should learn context first, and "challenging to me is more about asking questions," since a fresh set of eyes "should be questioning, not correcting." On mentors, Matty says to find someone who is where you want to be in ten years, to ask them to talk about their history over coffee, and to "never do anything that looks like you're asking somebody to do more work." Trevor adds asking colleagues for feedback right after projects and meetings.

SRE versus DevOps. Bridget says SRE is a job title and DevOps is a cultural practice: one is a job you do, the other is something you do at that job. To get into either, Bridget says, you'll have to learn on your own, and Matty suggests volunteer work, adding that your first job will not be SRE at Google.

Hot tools with zero ops headcount. Bridget reads hot as trendy, says picking tech because it's trendy is "great for resume-driven development and terrible for your employer," and cites Charity Majors: the best tool is the one you don't need and the next best is SaaS. There's a SaaS for canary releases, for instance, through feature flags. Matty adds that if a tool needs someone who knows ops and there's no ops headcount, don't use it. Bridget's advice is to "use the minimum viable complexity to get you there," and notes that Facebook scaled on a PHP monolith.

Ops people afraid of automation. Matty says that "automation is not a headcount reduction practice," and suggests asking what the sysadmins would rather be doing and how big their backlog is. Fear of automation is also a chance to listen: automation run through tests is less risky than manual change, since "humans will do it differently every time because we're fallible meatbags." Trevor adds small projects to show leadership the value. The last resort is "change your job or change your job," which Trevor first heard from Steve Murawski and Matty traces to Nathan Harvey's talk.

The episode closes with upcoming CFPs, the discount code, and a long aside on sticker inventory.

Matt, Trevor, Bridget, and Joe discuss 2017.

What were some of your favorite episodes?

• Bridget: the GOTO Chicago eps were fun! And I love the 1-1 fireside chats.
• Trevor: I enjoyed the Twitter banter around getting the live call show with Dr. Nicole Forsgren going, sad I had to miss it! There’s also a ChefConf episode sitting on my Surface that I just found the charger for!
• Matt: The Live call in show with Dr. Nicole Forsgren. Also the recent fireside chat with J. Paul Reed. And the crazy-go-nuts overload show of Windows Stuff.
• Joe: Live eps were easier to edit.

Let’s talk numbers

Website

• 22K visitors to the website (up from 16K visitors in 2016)
• Little less than half of traffic comes from search
• Twitter still accounts for about 6% of all traffic

Episodes

• 263,558 listens in 2017 (232,200 listens in 2016; 204,001 listens in 2015)
• Most listened-to episode in 2017 (and also the most-watched YouTube Video!) was Old Geeks Yell At Cloud With Andrew Clay Shafer & Bryan Cantrill (https://www.arresteddevops.com/yelling-at-cloud/)

What happened with you in 2017?

Matt

• Got a new job (recently). Now DevOps Evangelist with PagerDuty
• Spoke at DevOpsDays Denver. Went to DevOpsDays Hartford and MSP and Madison.
• Pulled off an acceptable DevOpsDays Chicago

Bridget

• New job in September! Microsoft dev advocate
• Kept scaling devopsdays up (51 events on 6 continents)
• Way less conference speaking; was on more conference program committees

Trevor

• Wrapped up first year at Chef!
• Spoke at DevOpsDays Dallas
• Moved back to Chicago!

Joe

• Iceland!
• Tesla roadtrip!

Referenced in this ep:

• Old Geeks Yell at Cloud

• Fireside chat with Bryan Cantrill

• The Oatmeal on Tesla

• Clued In escape rooms

• GoTime podcast (AMA episode mostly about BBQ)

• ADO ep 1

• Mary Thengvall book

• Bridget’s "getting into devops" blog post

Past year-in-review eps:

• 2016

• 2015

• 2014

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays
• SREcon - March 27-29 2018 - CFP closes Wednesday, November 29, 2017, 11:59 pm PST
• ScaleConf Colombia - April 27 - 28th 2018 - CFP closes Jan 15
• Velocity San Jose - June 12-14 2018 - CFP closes Jan 17
• ChefConf 2018 - May 23-26, CFP open now, closes Jan 10th

Discount codes: ADO2018 for 20% off lots of devopsdays, 10% off ChefConf

Hero Culture and Mixed Incentives

Matty says the theme of Katie's talk was hero culture: rewarding extra hours gives people no reason to work more efficiently. Katie says organizations assume automation is attractive, and people outside the conference room don't care what you call it. Bill says people want to do what looks immediately right, like spending 16 hours beating on a problem, and that a weekend of miserable firefighting is a good thing done that shouldn't have been necessary. Sasha would go further and say it's a bad thing when recurrent, noting companies give double incentives, praising collaboration and efficiency and then a prize for the most hours worked. Sasha adds research that productivity stops after about 55 hours a week.

Matty recalls Katie's line that no one gets claps for working two hours on Friday. "You get claps for working 10 hours on Friday." Matty describes a sysadmin who worked 70 hours every week, and whose review said the person "takes our site's uptime personally," which Matty meant as praise, and now sees as putting responsibility on the human instead of the system. An old office poster asked "how does it feel to save the day every day? Not all superheroes wear capes." Sasha notes a burnout talk that same day, and that email, laptops and remote work make around-the-clock demands easier. Katie adds that a manager is likely to like whoever answers the phone on Saturday, even while everyone says to have a good weekend.

Leading by Example

Matty recalls an interview with a CEO who said the interview process includes a Saturday call to see whether candidates answer, which Twitter destroyed. Matty thinks that direct toxicity is less common than the unintentional kind, like a manager sending email at 10 at night, even if that manager would rather you watch a show. Sasha says messaging can be direct in the healthy direction, too: a Silicon Valley startup CEO who personally called anyone who answered email on vacation, because no single person can be available 100% of the time. Sasha says at Microsoft nobody minds disappearing for a few hours mid-day as long as results are delivered.

Matty says when Chef changed to unlimited PTO, the CEO, CTO and CFO took three-week vacations first. Bill, at a previous company with unlimited vacation, told the team about time at conferences and shrugged "unlimited PTO," and before a four-week honeymoon pasted junk into Outlook's change password form so there was no getting back in. Bill likes the regulated-industry practice of surprise vacations for accountants, and wants to do that to ops teams: "If we can't survive for 2 weeks without you, it should burn." Matty calls it "a chaos monkey for your team." Katie admits to the arrogance of thinking the team can't live without you, and says at an older-style enterprise, employees may have to stand up for themselves. Matty adds that an unenforced boundary gets crossed, and if it isn't respected, getting out may be the answer. Sasha cites Sheryl Sandberg's Lean In: a McKinsey mentor was surprised how many people quit from burnout with unused vacation.

Tactics

Katie began charging meetups to the company's training bucket after a speaker said it's training: "Charge this to your training bucket." Matty says flexible remote work usually means working more, since the commute demarcation is gone, and Sasha agrees. Matty blocks time for what the job owes back ("Chef owes me 2 hours from last night") and keeps 2 hours a day on the calendar for tasks, treating it like a meeting with someone else. Sasha signs up for a class that starts at 6, and Bill tracks time as if billing customers, and checks vacation quarterly: "If it's August and I've taken a week off, I need to take a break." Bill notes HR likes unlimited vacation because nothing is owed at departure and people take less, and Sasha agrees that it's often "a trick which actually works against you." Matty shares stories from Chef, where a colleague's manager insisted on choosing a vacation budget and dates, and from Travis CI, where vacation is tracked.

Coming Back

Sasha says working on vacation can be self-protection, to avoid coming back to 1,000 emails. Matty's answer is email bankruptcy: set an out-of-office that names who will help, then delete the backlog, and even declare Slack bankruptcy, because "If it's important, it will happen again." Matty says you have to train coworkers first and calls it maybe controversial. Katie takes a more conservative route, scheduling a full day of meetings with the IM off on the first day back so nobody asks anything while going through email.

• Devaluing Hard Work - Katie Prizy, DevOpsDays Chicago 2017
• Security, Don't Fear The DevOps - Bill Weiss, DevOpsDays Chicago 2017

Referenced in this ep:

• ChaoSlingr
• Waffle.io
• The Hand-Waver's Guide To Contributing To Open Source
• Rugged Software
• Stranger Things

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays
• SREcon - March 27-29 2018
• ScaleConf Colombia - April 27 - 28th 2018
• Velocity San Jose - June 12-14 2018
• ChefConf 2018 - May 23-26, CFP

Discount codes: ADO2018 for 20% off lots of devopsdays, 10% off ChefConf

Matt refers to an episode of “Coupling” on the BBC - the episode was Series 1, Episode 2, “Size Matters” - here’s the quote:

Patrick: Oh, don’t be so piecey.
Howard: Typical lefty puritan..
Sally: Typical what? Come the revolution-.
Patrick: What revolution!? You guys are in power! We’re the revolution now!.
Sally: No. No, it can’t be right...
Patrick: You’re the evil empire..
Sally: No!.
Howard: Yes! Like Star Wars, and Patrick and me are the Rebel Alliance..
Sally: No! You’re not the goodies, we’re the goodies! We’re lefties!! We’re always goodies!.
Patrick: puts glass on his mouth and makes a Darth Vader voice No, Sally. You’re the establishment.

Open CFPs

• https://www.devopsdays.org/speaking/
• https://conferences.oreilly.com/velocity
• https://chefconf.chef.io/cfp/

Image credit
By inkknife_2000 [CC BY-SA 2.0 (https://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons

Curating LISA

Alice was talks co-chair for LISA, the Large Installation System Administration Conference, which USENIX runs and which has been around for over 30 years, back when a large installation meant 10 workstations. USENIX handles logistics and brings in people from the community each year to select talks, workshops, tutorials and keynotes. Alice likes that it is non-vendor-specific and for practitioners, and describes three days of training and workshops followed by three days of a more traditional conference. The 2017 event was in San Francisco, the next in Tennessee and the following one in Portland.

In choosing talks, Alice wants the how: "there's a difference between how to use Docker and how we used Docker to change up our infrastructure." Alice also says "I also really love a good outage story," and wanted Niantic to talk about the Pokémon GO launch. Bridget asks how to get stories with failure in them past a company's lawyers. Alice says it's hard, since companies fear customers questioning an SLA, but an engineer hearing honest failure stories is more likely to buy, and talks from Google have gotten through when the incidents were old or vague enough. LISA, Alice says, sits at the end of the year when people are tired of flashy conferences and more willing to talk.

Reaching Out for Talks

Alice ran devopsdays Portland's vendors and MC duties, and did not choose talks that year, but encouraged people to submit who otherwise wouldn't have. A more diverse selection committee used its networks, telling people it would be a safe space. Bridget says that when you post a CFP and do nothing else, you get the people whose job it is to submit, so a broader range means reaching out. Alice's first conference talk, at LISA in 2015, happened because someone who was a chair that year reached out, and "someone reached out to me and said, I think you have something to add here." Alice adds that looking at past talks by people with PhDs would have scared Alice off.

Alice says it matters who is reaching out and why. An invitation to a Python conference said it was looking for more women, for a framework Alice hadn't worked with in years, and Alice's reaction was "you just want a woman, that's all this is." Bridget says no one wants to be a decorative prop. Reaching out works when it names something specific, which Bridget calls "conference invitation Mad Libs," and when it says why the conference is good for the speaker, since Alice has to justify conference travel to an employer.

Learning to the Bottom of the Stack

At GitHub, Alice started on the Edge team, which handled the network tier and a custom load balancer, and now works on the Kubernetes platform that all of GitHub.com runs on, with plans for kernel-level work. Alice grew up in a very small upstate New York town with a Windows 95 computer and no internet until 16, so learning happened alone, with Compton's Encyclopedia for AP Calc. Alice has a film degree, started in tech support, tried Ruby and disliked it, then liked Python, and took an early Coursera programming class that built games in eight weeks. Alice says "I've had a career of doing one job during the day and learning to do another job at night," and prefers "self-guided" to self-taught, since other people wrote the materials. The method is the Socratic why, which keeps leading lower in the stack.

Alice's laptop was resting on Understanding the Linux Kernel, the Practical Linux Security Cookbook and Docker Up and Running, and since the kernel book's examples are in C, Alice learned C for fun. Alice brought the kernel book to jury duty. For concepts like Kubernetes, YouTube videos with diagrams help, and Stack Overflow answers supply jargon to search the man pages. Alice carries a chip on the shoulder about lacking a CS degree, but says "I've made it this far without having to know really any computer science," and that knowing arrays and hashes gets you far. Bridget notes that even with a CS degree you still Google constantly.

Interrupt Designs

Alice made a t-shirt with a Panic! at the Colonel design after wanting shirts that weren't unisex tech swag, and when it sold, began donating the money. The first went to Women in Linux. Then came the "ladies is gender-neutral" shirt, made to answer "guys is gender neutral," offered only in fitted sizes and sold with every line conference organizers use on unisex shirts. It raised over $5,000 for Outreachy. A Manic Pixie Dream Girl shirt with PXE as the pixie benefits Free Geek, a Portland organization that refurbishes donated equipment. Alice moved from Teespring to Threadless for more cuts and colors, and includes sizes up to 4X. Fitted shirts, Alice says, should be table stakes for vendors, though they don't fit all women. Alice adds that scarves made things much easier for devopsdays Portland's swag, since no one needs a size.

Humor, Loudness and the Internet Persona

Bridget asks what Alice wants people to take from posting things that poke the bear. Alice says humor works: "comedy allows you to point out things that otherwise would be considered offensive or people would instantly shut down to," like the fool in a Shakespearean play, and people remember it better than a blog post with citations. Jokes about sexism may also embolden people to talk about it at work. Alice says the loudness comes from being vocal about things that get in the way, like sexism and racism: "I talk about them a lot because I want them to go away." Alice also talks about technical things and baking.

Chairs, Meetups and Practice Talks

Alice had been at GitHub over seven months before ordering an office chair, a Steelcase Gesture with narrow armrests that suit a smaller frame, describing the research habit as being "like the missing stair enabler of office equipment." Alice runs the PDX DevOps Meetup on the last Wednesday of the month, and encourages speakers to come, since the bar is lower than at a conference, with no recording, and a repeated talk is welcome. Bridget's tip is to give a conference talk at a local meetup the month before, pointing to a local speaker who changed about half the slides after the first run. Alice gives at least two practice talks, and plans to use the PDX meetup for the next new one. Alice likes devopsdays for the local angle, since it's harder to claim something can't be done locally when an organization three blocks away has done it.

Planning 2018 and Containers in Production

Alice plans the year during the holidays, because of a need to manage emotional highs and introvertness, and wants to keep 2018 domestic with not too many new talks. A planned January keynote was postponed indefinitely. Alice would like to give a technical keynote on three or four years of running containers in production: at New Relic, Docker went into prod at version 0.6, and Alice has run Dockerized databases and now runs Kubernetes at GitHub. Alice calls it "the mechanic's view of containers," leaning on the podium with a rusty wrench saying it's going to break, and jokes "Your problem is all of the whales that you put all your apps in." Bridget counts 29 public talks in 2016 and calls that too many.

Career Advice

Asked how people grow up to be Alice, the answer mixes self-guided learning with a network outside your employer. Alice says soft skills are a real deficit in tech, and that giving talks and keeping a social media presence is "kind of like a bridge between all of your jobs" and a safety net if a job ends, since you don't want your only network inside one company. Befriending women in tech has made life easier, Alice says, and the network doesn't have to be women. "Don't let your job be your whole life," because failing at work shouldn't take everything else down with it. Bridget adds that networking means connecting over common interests, and that following someone on Twitter doesn't make them your BFF. Alice adds that you can't control a promotion, but you can control learning a new language.

For listeners trying to DevOps at their own companies, Alice says things are easier with buy-in from the top, since a grassroots effort without a VP's backing is hard, and convincing a manager's manager helps. If you keep hitting obstacles and aren't enjoying work, Alice says it may be time to look elsewhere, since "your job shouldn't be your whole life."

Bridget sat down for a fireside chat with Alice Goldfuss (GitHub). No actual fires were harmed in the making of this episode.

• LISA17
• devopsdays Portland
• Charity T-Shirts - Interrupt Designs - awesome nerdy shirts with proceeds going to charity

Checkouts

Alice

• Chair
• Baking Scale - round metal, good for scones
• Increment Magazine
• SysAdvent
• Airplane food critique website
• Sailor Mercury zines

Bridget

• Go Pirates - Veronica Mars podcast
• Bombsheller leggings - again! because they’re just that great
• Jewelbots - because I got Joe’s 10-year-old niece in the gift exchange
• Nerdy Baby - great STEM gifts

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Use code "ADO2017" for a discount on many devopsdays.

Open CFPs

• lots of DevOpsDays
• SREcon - March 27-29 2018 - CFP closes Wednesday, November 29, 2017, 11:59 pm PST
• ScaleConf Colombia - April 27 - 28th 2018 - CFP closes Jan 15
• Velocity San Jose - June 12-14 2018 - CFP closes Jan 17

Bridget and Matt sat down with a speaker (Emily Freeman) and a couple of organizers (Joshua Zimmerman & Christian Herro) to talk about the wider themes of organizational change when you may not call all the shots in your org.

• Emily Freeman's talk: Scaling Sparta: Military Lessons For Growing A DevOps Team

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Use code "ADO2017" for a discount on many devopsdays.

Open CFPs

• lots of DevOpsDays

• The Manager's Path: A Guide for Tech Leaders Navigating Growth and Change by Camille Fournier

Check outs

James

• Launching new classroom collaboration project Empatico
• Product-launch survival food Indomie Mi Goreng instant noodles
• Bundle of Terraform and Packer books

Inés

• Fastly Teams growing the EdgeApps services - ImageOpto, Video, Load Balancing
• Recently got into Almond lattes (like a true Californian)

Bridget

• Bombsheller leggings - I just bought circuitboard ones, and I see also that they have Settlers of Catan ones. WANT.
• Cloud Developer Advocates at Microsoft - I just joined this awesome team!

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Use code "ADO2017" for 20% off at Velocity New York Oct 1-4.

Use code "ADO2017" for a discount on many devopsdays.

Open CFPs

• lots of DevOpsDays

Recorded May 8, 2017.

The unofficial title of this episode: "Ya sure, Nicole?"

• DORA - DevOps Research and Assessment
• This ep on Food Fight Show
• Kelsey Hightower's devopsdays Austin 2017 keynote
• Release! The Game

Check outs

Nicole

• Velocity San Jose 2017
• DOES London
• Metrics Workshops
• 2017 State of DevOps Report

Nell

• devopsdays Seattle 2017
• The Expanse

Matt

• Robot Unicorn Attack 3
• Robocall protection with Nomorobo
• School of Rock

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Use code "ADO2017" for 20% off at Velocity New York Oct 1-4.

Use code "ADO2017" for a discount on many devopsdays.

Open CFPs

• lots of DevOpsDays

What DORA Measures

DORA's assessment surveys an organization's people against 20 capabilities the research has linked to IT performance, from culture to practices like automated deployment and managing work in process. It compares the organization to the industry, and with a group of 50 or more people it can say which investments give the most return. Bridget raises the customer who wants a RACI chart for who should do each thing. Jez's answer is that an organization should have a few strategic priorities and each team should work out how to collaborate, agreeing on a measurable goal for the next month. There's no way of knowing in advance who will have to change, "but, you know, my money is on everyone." Jez says DORA is "all about data," including measurable culture, and that picking a fight with Nicole over stats is a losing one.

What the Continuous Things Are

Jez defines continuous integration as building and testing every change, working off a shared trunk and not long-lived feature branches, with a quick, comprehensive automated test suite, so that the default state of the system is a working one. Jez tests people with three questions: does everyone check into a shared trunk at least once a day, do tests run on every check-in, and when the build breaks, is it typically fixed within 10 minutes? Most people can't answer yes to all three. Running Jenkins against feature branches and ignoring it when it turns red isn't continuous integration: "It's a practice and mindset, not a tool." Bridget notes the CI/CD theater, and Jez says DORA uses yes/no and agree/disagree questions, and that "asking the questions is, in many ways, the hard part," and "People love redefining terms so they can say they're already doing it."

Continuous deployment means a passing build goes to production automatically, with high performers deploying hundreds or thousands of times a day. Continuous delivery is behaving as if you'd do that without deploying every build, which suits firmware or mobile apps, and still improves quality, cost, time to market and feedback loops. Bridget plays devil's advocate with Schrödinger's deployment, where a build could go out but a schema migration wasn't accounted for. Jez says to "do continuous deployment wherever you can," and that Jez wouldn't build non-web software unless forced to, noting people at Facebook who deploy mobile apps every couple of weeks aren't happy about it. Jez points to a recent Charity Majors blog post about testing in production, and agrees that production needs good monitoring, logging and tracing, and cheap, low-risk deploys through blue-green deployments, canaries, feature flags and circuit breakers. Jez says to accept that "failure is inevitable," and recalls that John Allspaw's focus on mean time to restore over mean time between failures was an aha moment. Bridget adds Andrew Clay Shafer's state of continuous partial failure.

Organizations That Keep Improving

Asked about the common thread among organizations on the path, Jez's answer is that "The best organizations are always trying to get better," and are never satisfied. Even Amazon and Facebook invested huge effort, and Jez notes Amazon was a monolith that spent four years re-architecting after making it a priority at all levels, consistently and with money. Jez adds that Amazon has miserable, dysfunctional pockets too, and Bridget asks whether that means software is made of humans.

The Agile 2017 Keynote

Bridget asks for the Cliff Notes of the end of Jez's Agile 2017 keynote. Jez explains that an ex-Google employee, James Damore, had written a 10-page document about Google and the causes of low representation of women and people of color in the industry, and that Jez was angry we're still having the discussion in 2017, like being angry about talking about continuous integration after 15 years. Jez holds up Douglas McGregor's The Human Side of Enterprise, from 1960, on motivating teams. Jez says there are biological differences, but real outcomes come from a complex interaction of genetics, epigenetics and environment, and "you can't go from those biological differences straight to differences in ability." Jez recalls telling the audience that math and science can't explain the lack of diversity, but sharing a work environment with people like Damore can, and some people walked out. Jez argues that unequal access to highly paid, high-status tech jobs is inequitable, produces worse products and affects the economy, and that people who care about the minutiae of tech but not how teams are composed will do worse, since these are systemic problems and the system is what dominates.

Bureaucracy, InfoSec and Lunch

Bridget asks about siloes. Jez says it's highly variable and depends on who leads each team. A high point of Jez's career was the federal government, where Jez met mission-driven, brilliant, hardworking people, including in InfoSec, and learned from Mark Schwartz's talk that bureaucracy can be helpful because it encodes what people think is the right way to do things. Jez's "number one DevOps hack" is to find the person everyone slags off, often InfoSec in the dev world, take them to lunch, and spend an hour actively listening to what's in their way. Bridget asks for a British-to-American translation of "slag off," and Jez says it means trash-talking someone.

Bridget asks what individual contributors can do. Jez says it's making friends and understanding different perspectives, as in learning why a rule exists from the person who had the problem. Jez argues that "Most of the problems we face are due to a lack of empathy," pushing back on the idea that the obstacle is a failure to systemize. DORA measures collaboration between Dev and Ops, asking whether the outcome is win-win, and measures culture with the Westrum model, a construct from sociologist Ron Westrum, which John Allspaw also pointed Jez to, studying safety outcomes in healthcare and aviation along six axes. Jez says information flow is critical for resilient organizations, and "You can build the best systems in the world and use the best tools in the world, but if your cultural organizational structure is wrong, it just won't work."

Cargo Culting and the Andon Cord

Jez says large agile frameworks get adopted without changing procurement, contracting, leadership or organizational structure, and then "you're just shuffling around the chairs on the Titanic." The point of practices is to change the organization's behavior, and Jez objects to cargo culting. Jez says knowing where you are and where you're going in measurable terms comes first, and when people copied Toyota, "You're copying the practices, you're not copying the mindset and the culture of continuous improvement." Bridget gives the example of an andon cord in a place where pulling it gets you fired. Jez points to the NUMMI story on This American Life, where other GM factories copied the cord, but managers were rewarded by how many cars came off the line whether or not they worked, so pulling it got you fired.

Where Change Needs Buy-In

Jez cites John Kotter's Leading Change, whose first step is that most employees, about 75% of management and virtually all top executives need to believe that considerable change is essential. That shows up as ruthless prioritization, and it's "much easier to change an organization where there's a burning platform than it is to change an organization where everyone thinks that everything's just fine." Some leaders won't state priorities in measurable terms because it takes away the ability to change their minds. Jez says individuals can still effect change, though the impact will be limited and making it stick is the hard part, and organizations have backslid after people left. The original Continuous Delivery book came from a team of eight and colleagues working on miserable projects with Bash and CVS, transforming how an application was deployed, which shows "you can achieve great things with small teams." Jez adds "There is no linear path from A to B."

Jez closes with the Zen temple story from the opening, told by the teacher at an introduction to meditation: sometimes you come to the temple feeling down, sometimes feeling great, "But don't worry, because that feeling will pass too." Jez thinks that sums up DevOps and the human condition. Jez also encourages listeners to speak at conferences, since you're probably doing things other people would be excited to hear about.

Bridget chats with Jez Humble (DORA) about continuous... everything!

• Continuous Delivery - ADO Ep 15 - Jez's previous ADO episode
• DORA - DevOps Research and Assessment
• Westrum model of organizational culture
• The course website for Info 290M Lean/Agile Product Management, a three unit graduate class taught by Jez Humble at the UC Berkeley School of Information
• Jobs at 18F - director job opening soon!

Check outs

Jez

• Manjula's Kitchen
• Plutopia - Nuclear families, Atomic Cities and the Great Soviet and American Plutonium Disasters

Bridget

• Favorite food blog: Smitten Kitchen
• New-to-me food blog: Mountain Mama Cooks
• Great Electric American Roadtrip

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Use code "ADO2017" for 20% off at Velocity New York Oct 1-4.

Bridget will be at Uptime in Pittsburgh this week.

Open CFPs

• lots of DevOpsDays

Use code "ADO2017" for a discount on many devopsdays.

Incentives With Nothing but a Smile

Bridget asks how to incentivize people in tech. Bryan says "the only incentive that I have is a smile," and describes spheres of influence: tensions occur where bubbles touch, so Bryan tries to make a bubble more permeable and bigger. Jessie says incentives are hard in open source because you can't pay people, so you use their interests, and don't push them into work they don't want to do, since it's free labor. Matty notes that Jessie's slides paired each deep technical point with a person problem, and that making security hard is itself an incentive, since people turn it off. Bryan adds that incentives are harder at big companies because "you can really blend in" and do the bare minimum for years.

Let Users Fail Well

Bryan took a note from Jessie's talk: "We should always allow our users to fail well." People will do the worst thing possible, so the default should leave no room for it: "You can't add privileges to Docker. You get what you get and you can only take away." Bridget says defaults apply to organizations too, and asks what a team does if no one takes any action. Bryan says people do the easiest thing, like turning off SELinux or booting cloud instances without thinking about security groups, and wants all that to be on by default, like a banana you have to peel.

Matty tells of a sysadmin who wanted a release manager reporting to the CTO with hiring and firing power over developers so they'd follow a new release process. Matty's answer was to "make the right way the easy way," as in the book Switch, whose factory machine was redesigned to need both hands on switches away from the blade. The happy path should be a glide path, where "the default should be the happy path," and doing it wrong means fighting the current.

Saying No as a Maintainer

Bridget points to a GitHub thread in Jessie's slides where users wanted a use case that would harm the general one. Jessie says the maintainer takes the heat for saying no, and the aim is to leave the mass of users unaffected, while "someone's always going to be pissed off at the end of the day." Bryan recalls one key moment in that thread, when someone came 100 posts down and admitted reading none of it. Jessie remembers the reply, "maybe if you find the time to read the rest of the issue, you will know what is happening right now," and says friends in Slack absorb the heat not sent back on GitHub. Matty calls it a troll flame radiator.

Bridget asks Bryan how to handle people fighting for their local optimizations. Bryan says a job isn't a place where you get your way all the time, and "at the end of the day, it's org first," but the approach is to set expectations early and offer tradeoffs, inside what Bryan calls a "circle of respect." Jessie says in open source there is mutual respect, and multiple maintainers will take the wheel when someone gets frustrated.

Vendors, Customers and the Scale of Jerkiness

Matty, who works for a vendor, explains that Matty and Bridget are advocates inside their companies, and a vendor doesn't have infinite resources. When a large customer demands a feature, a company may redo everything, run a 90-hour week, and compromise 10 to 20 other customers, without anyone asking five whys. Bryan's rule is "It's don't be a jerk," and every decision asks whether it's being a jerk, since there's a scale of jerkiness. Bryan wouldn't demand a feature now, but would point out that a vendor hinted at three months, and that was four months ago. Matty says roadmaps are not promises of dates.

Bridget wonders whether unpaid users are more demanding. Jessie says "Everybody wants their thing, their Turing-complete thing that will make their life so easy," and that some users eventually say thanks after a maintainer adds the feature out of spite. Jessie notices when an issue comes from someone at a company using the tool, and those people are mostly willing to contribute the fix. Bridget notes Allstate contributed authentication and authorization work to open source Cloud Foundry, which Pivotal sells in a commercial distribution.

Intersourcing and Pressure From Above

Bryan describes intersourcing, open source to a company and no one else, and says it exists for good reasons in a specialized vertical like a bank. Inside Capital One, Bryan uses GitHub and gets pull requests all the time, and some are closed because they ignore the roadmap. Bryan says when people push hard internally it's because someone else is prodding them with a fork, and "we're all archaeologists." Matty calls it closed-door open source, and says open source has already solved collaborative coding, so vendors should stop trying to invent it. Matty adds that asking why uncovers the driver three levels up, and Bryan would call the person and say both of you are trying to get things done.

Matty asks about incentives for smaller maintainers. Jessie says "giving people responsibility actually goes a long way," meaning commit rights or code review, and it becomes gamification of reaching maintainer, though some people just want a thank you.

Open Source in the Enterprise

Andrew says open source isn't always shiny, happy people. You can create a lot of value and capture little, and people assume they're entitled to enterprise-level support for weekend efforts. Bridget asks Bryan the right way for an enterprise to consume and contribute. Bryan says it's complicated: licenses and patents, for example, and without patent indemnity, using someone's open source could lead to being sued for IP, so "there's levels to this." Jessie says Jessie stays far from Google's secret sauce to avoid messing up.

Andrew says some people want to give money to projects and there's no good way to, and that adoption sometimes stalls for lack of governance or indemnification. Others go YOLO, and Andrew says "I would be shocked if most enterprises realize what code's actually in production." Matty knows a CTO who requires code review of every open source project, and someone submits 10,000 lines of code that goes straight to prod. Bryan has been at Capital One since the previous October, after doing open source full time, including contributing to Terraform and writing Go for DigitalOcean. Bryan says the bank tries to know everything that goes into production, given the stakes. Bridget adds that during a startup acquisition Bridget had to find every license and rip out one library whose terms would have given the acquirer's IP away.

Closing: Tech and People Together

Asked how to get people to do the tech work and the people work, Bryan says to be the example, with empathy. Jessie says being kind makes people kind back. Matty, who coaches customers and isn't a line manager, says to be genuine, because fake coaches get smelled out. Andrew reinforces that, and says the two can't be solved separately: "there's not really a tech and a people thing. That it's one system."

Bridget and Matt chat about enterprise transformation and open source with guests Bryan Liles, Jessie Frazelle, and Andrew Clay Shafer, in front of a live studio audience at devopsdays Minneapolis 2017.

• devopsdays Minneapolis 2017

• Sys Admins, DevOps, SRE. Oh My! - Bryan Liles' opening keynote

• Security In A Containerized World - Jessie Frazelle's closing keynote

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Use code "ADO2017" for a discount on many devopsdays.

Why Split It Up

Daphne's reason for microservices at the ABC was scale. One part of the system, transcoding, has to scale far more than everything else, and the microservices part "tended by accident really," once that piece was separated. Transcoding means converting a video or audio file from one format to another, and the ABC needed to do it over its whole catalog, with requirements that commercial transcoders and Elastic Transcoder didn't meet. The transcoding service takes a JSON packet saying which file to transcode and where to save the output. It can run on a larger AWS instance type suited to the job, and Daphne says the cost of that instance goes 100% to transcoding. Matty calls that an interesting angle on showback and chargeback.

Bridget points out that spreading work across services adds complexity, and Daphne answers "I think you just move the complexity. Because there's always complexity everywhere." Bridget says that is Tim Gross's "conservation of complexity." Daphne says independently deployed services tend to be stable, and adding a captioning extraction service would sit apart from the others, leaving the main complexity in coordinating which service is called when.

What Is a Microservice?

Matty asks how this differs from service-oriented architecture, which Matty worked with at a dot-com years earlier, with a mail service and a lead conversion service behind versioned APIs. Kenny says recent research into continuous delivery changed the definition Kenny started with. A shared delivery pipeline means you're "technically taking public transportation to production," batching up changes from, say, 500 engineers on one monolith. Splitting the pipelines lets people commit and deploy independently. Kenny admits "We're really bad at naming things," then lists small teams organized around business capabilities, independent deployability and a decomposition strategy, and calls it "a better SOA." A rule of thumb: if a new engineer takes longer than a day to ramp up on a service, it should probably be two services. Bridget, speaking as someone on the receiving end of the pager, adds that if a health check can only say some of it is working, too much is crammed into the service.

Kenny's talk covered event-driven microservices, using events to maintain the integrity of foreign key relationships when a large shared database is torn apart. Somebody had found the Cloud Foundry management endpoint on a Spring Boot app in one of Kenny's demos, and crashed the application of ten microservices from an iPhone. Matty guesses that someone thought they were at DEF CON.

Don't Just Rub Microservices on It

Kenny passes along Matt Stein's line that microservices aren't the solution: you already have a problem, and want to go faster or scale. Matty says every VP wants a fill-in-the-blank this quarter, and Bridget recalls an open space at an Agile conference where someone said their VP wanted microservices that quarter. Kenny's advice is to get data first, for instance "How much unchanged code are you deploying per deployment?" Kenny notes that even a monolith on continuous delivery could be deploying every 11.6 seconds. Matty adds that the question is whether you need to go faster at all. Daphne says unchanged code is risk, since every deployment might go wrong, and Matty says small changes minimize the blast radius, which is a reason even if speed isn't.

Kenny says shared resources are a good reason to consider microservices or serverless, and expects a healthy microservice architecture to reduce unchanged code per deploy. Kenny would like to test that by mining GitHub, though no enterprise is going to put its code there. Kenny and Josh Long spent two years writing a book, Cloud Native Java, on cloud-native applications with Spring Boot, Spring Cloud and Cloud Foundry, which had gone to press, with a copy expected in early June.

Daphne's own route to microservices wasn't a formal journey. A team averaging about three people, each knowing a different part, built small pieces, and the scale question drove it. The project "kind of secretly started a bit off-piste" as a proof of concept that would save money.

Conway's Law and Weaponized Microservices

Bridget says teams sometimes want independently deployable services to avoid interacting with other teams, and asks about weaponizing microservices. Kenny: "Oh, they're already weapons." Kenny says teams need empathy for the services they consume and produce, and describes consumer-driven contract testing, where you publish a contract and other services test against a mock, so you can't reach production without passing consumer tests. That pushes you to go to other teams instead of them coming to you, "a good way to prevent evil." Matty adds that Conway's Law also works in reverse: at places with little trust, a change from hunter green to forest green on the front end meant testing the whole website down to the data warehouse, because nobody trusted the contracts. "No matter what, people are terrible," Matty says, and Kenny says to make the less terrible thing easy.

Serverless, Replatforming and Product Lifecycles

Bridget asks how serverless fits. Daphne says you still need to deploy and debug the serverless pieces, and the smaller the piece the easier it is: "it's a tool that you should wield in particular circumstances, and otherwise you're just gonna be shooting yourself in the foot." Kenny is researching how Lambda binds you to event sources that lock you into AWS, and suggests a Spring Boot microservice as an event source with Lambda functions as event handlers, though Kenny doesn't know of anyone doing it in production.

Matty says enterprises move slowly, and most enterprise customers on AWS or Azure use only compute, not things like RDS, because they want to hold on to configuration, and a DBA wants to look at a slow query log. Matty's private serverless GIF is an empty data center. The appeal of tools like Habitat is getting some of the value without rewriting an app into a 12-factor app, and Matty can't make a legacy .NET app on Windows 2008 serverless. Kenny defines replatforming as modifying applications to suit a platform that runs them differently, such as a cloud-native one, and says it makes sense where competition is rough. The IRS, Kenny guesses, wants to move faster but faces higher risk and little competition.

Matty brings in Marty Cagan's kill, maintain and innovate for products. A product on the kill list shouldn't be rewritten, one in maintenance doesn't need faster delivery, and only the innovate ones do. Matty says that's where bimodal IT goes wrong, though the Gartner idea properly read is that "different teams move at different speeds," like "one transmission for all of our teams, they're just in different gears." Bridget calls bimodal IT horseshit, and Matty's point is that declaring "microservice all the shit" skips thinking about each product's lifecycle.

The Ridiculously Stupid Thing

Asked for the most ridiculously stupid thing to do, Kenny says creating microservices for things the business doesn't drive, or special snowflake services like image filtering for one consumer. Daphne's answer is languages: the ABC system used different ones for different services, which is fine as long as you don't end up with 20, so "Don't write them all in 10 different things." Matty adds "Don't write one in Ada." Kenny adds shared libraries, since upgrading one across 500 microservices is trouble. Bridget warns about the hidden distributed monolith, where everything still talks to the same database, and Kenny's test is that if one change means deploying all ten microservices, "you've got a distributed monolith."

Bridget and Matt chat with Daphne Chong (Amazon) and Kenny Bastani (Pivotal).

• Daphne's GOTO Chicago talk: Video Transcoding at the ABC with Microservices

• Kenny's GOTO Chicago talk: In the Eventual Consistency of Succeeding at Microservices

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

From Lightweight VM to Just Processes

Matty asks what the paradigm shift is, since customers often just want to spin up a container instead of a VM. Jérôme says the lightweight VM idea helps people grasp what a container is, but it should be dropped quickly: "it's just processes," or for the technically inclined, cgroups and namespaces. Containers will be many things to many people, the way virtualization went from stacking machines on a host, to cloud APIs, to disposable environments for CI. Matty asks whether calling a container a lightweight VM is itself a wrong statement as a global claim, and Bridget says it depends on the use case. Jérôme sticks to the point, calling the metaphor "just the tip of the iceberg," and noting there are use cases where it doesn't make sense.

Mark adds the developer view, which is packaging. A VM is "just like driving a nail with a sledgehammer," while a declaratively configured image is repeatable, lightweight and easy to deploy. Bridget adds that a golden image stamps out your application and also a Heartbleed baked in, and Mark says rebuilding and retesting an entire golden image each time a backing service moves is heavy compared with a new Docker build.

Runtimes and Choosing One

Bridget asks for the most incendiary line in the talk, and Mark says nothing was flame-worthy, except that some people will not run on a Docker runtime while others will only run on one. Every new Docker release brings cries of anguish when something breaks, and some people are frustrated by the move-fast approach. Bridget asks Mark to explain what a runtime is. Mark sketches runc for initiating containers from images and containerd as the Docker runtime, with other vendors using other execution engines: Joyent Triton, which Mark describes as running Docker containers on Solaris zones, Cloud Foundry, which builds a container per the Docker image format and still uses runc, and CoreOS Rocket.

Jérôme says Docker's strategy is to give people options, such as swapping runc for Rocket, or SwarmKit for Kubernetes, and compares the choice to picking a hypervisor. For a first cloud VM, you shouldn't spend an hour deciding between EC2 and DigitalOcean, and after hacking at the "cloud jungle" with a machete you'll know enough to choose. Matty agrees that you can't judge engines by letters you don't understand yet, and that "you'll know you need a scheduler when you need a scheduler."

Diving In

Bridget asks Jennifer about the actual adoption at Edward Jones. Jennifer says some teams are using containers, and some even run in production, which Jennifer only learned a few weeks earlier. Jennifer describes the financial advisors as the main moneymaker, and the concerns as security and reliability across six time zones. Containers connect to reliability because a sick container is replaced by a new one. Jennifer describes approaching the topic like standing on a beach and looking at the ocean, trying to find where to enter the water, then downloading VirtualBox, Docker, Kubernetes and PCF Dev, following Docker's docs, and running commands like ps to see the output. Jennifer says "You're not gonna horribly break something." Mark adds that Mac users should use the stable builds of Docker.

Matty, who uses Docker to test cookbooks because a container is faster than a VM, warns that at some point you have to test on something that looks like production, unless production is a container. Mark agrees that less deviation between dev, test and prod is better. On databases, Jérôme asks "should you be running your database in the first place?" If you have one Postgres server and a manual failover in the middle of the night, it probably shouldn't be in a container. If you spin up thousands of databases, say one per CI test, containers pay off. The one deviation Jérôme encourages between prod and dev is running the database in a container in dev when production uses a third-party service.

Ops Visibility and Buildpacks

Matty says ops people see a container and ask what the hell it is. Bridget asks whether it holds an unpatched operating system. Jérôme calls the fear unfounded technically, but says techniques that work with VMs don't map to containers, and uses SSH as the example: you can attach a shell to a running container, so there's no need to cram an SSH server and keys into it. Communities, vendors and bloggers exist to share those tricks.

Mark raises the difference between a Docker image and a Cloud Foundry buildpack, where the container is built around your application. When something like Heartbleed hits, the ops team can patch the underlying platform and reach into those lower layers, so developers don't need to rebuild, and Mark says "That's good and bad, right?" Matty says Habitat is a similar idea of abstracting a layer away.

Domain Experts and Guardrails

Bridget asks what to tell a developer who wants to create their thing and not care about buildpacks. Matty says to get off "this full-stack nonsense," since domain experts need to be domain experts, and that whatever is built, be it a Docker image, a Converge node or a binary, is an artifact that should be tested for compliance on its way through deployability. Jennifer says there are two sides: wanting to customize everything, and not having all the time in the world. Matty adds the example of a developer who reads on Stack Overflow that disabling SELinux helps a Node app and changes the cookbook, and says guardrails and fast feedback catch that before security wags fingers. Jennifer says you have to know what's valuable to your company. For a financial firm, that means regulation and privacy, plus user experience teams that once rejected a product for not being user-friendly to financial advisors. Mark says it's a matter of prioritization, developing your expertise and relying on others, and that containers "don't fix your broken culture."

Aha Moments

Matty asks for the surprising moment. Jérôme says there was no single one, but a series of crazy experiments, such as running a container on Linux that held a VM showing a screen of Moby running containers inside containers. Matty compares it to Sean O'Meara's configuration management parlor trick of CFEngine installing Puppet that configures Chef. Mark's moment was a gradual realization, from asking why we need this when we have VMs to spinning up Redis or Mongo in containers and seeing the potential. Jennifer says reading Docker docs wasn't enough until building and deploying a simple application, with the realization that you deploy one application in one container, not onto a massive piece of machinery.

Bridget asks Jérôme for a two-sentence explanation of Moby, and the answer is one: "Docker Inc. is a company that makes Docker, a product that uses Moby, an open source project." Jérôme credits Laura Frank with the explanation.

Bridget and Matt chat with Jérôme Petazzoni (Docker), Mark Heckler (Pivotal), and Jennifer Heckler (Edward Jones).

• Mark & Jennifer's GOTO Chicago talk: Clouds & Containers: Hit the High Points and Give it to Me Straight, What's the Difference & Why Should I Care?

• Jérôme's GOTO Chicago workshop: Container deployment, scaling, and orchestration with Docker Swarm

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

Open CFPs

• lots of DevOpsDays

Bringing Audit Along

Bridget asks how an insurance company's customers and regulators shape this. Matt says the compliance and security teams were the tough part of the continuous integration journey, because "explain your job in an algorithm" puts people on the defensive, as if they're being replaced by a robot or a shell script. Anthony adds that insurance means state-by-state regulation plus PCI and SOX. The team reached out to its internal audit organization early and asked them to look at the work, and the greatest outcome was that the lead auditor eventually worked for Matt as a product manager. Anthony relays that auditors saw every production deploy trace back to a GitHub commit, and the reaction was, "I've never walked out of an audit with a smile on my face."

Matty says every company has compliance with a lowercase c, meaning the standards important to the organization, and everybody thinks they're special. The "dirty little secret" is that one of the biggest ways Chef gets into companies is through audit and compliance, "because people lie. Computers don't lie." Nicole says compliance teams get scared when you go fast without them and hand over a spreadsheet or PDF, but once they're part of the process, they see the value and can collect data programmatically. Matty adds that nobody is automated out of a job, since the risk officer's big brain still decides what's important and the task is describing it consistently. Matt says engaging audit early made communication bidirectional, educated them on what new tools could do, and let the team understand their incentives, which turn out not to be checking the box. That was a chance to "build a bridge rather than kind of pile another brick on the wall," and Bridget notes that bricks into a bridge instead of a wall sounds suspiciously like DevOps.

Audit Theater and the Compliance Sine Wave

Matty draws a sine wave of compliance: a company does its regular business and drifts down, then scrambles before the quarterly audit, the auditors show up and leave, and it drifts down again. Bridget asks whether it's really more of a cliff. Matty says that when compliance is part of the process, you're continuously compliant, and a compliance officer knows an auditor could walk in at any time. Anthony describes the typical response as adding process on top of process: one audit finds a missing document, so a process is added to check for the document, and the next audit finds that the checking process failed. "That's what the system solves for you."

Shifting Left, Hardening Sprints, and Democratized Compliance

Matty describes a project that ends with a hardening sprint for security testing, which fails because nobody has looked all along, leaving a choice between delaying or getting an exception. Matty's point is that "the bad guys on the internet don't care that you have a note from your mom that says it's okay you didn't patch Heartbleed." Nobody would accept saving QA for the last sprint, and the closer to a defect's introduction you find it, the cheaper it is to fix. Bridget adds what happens if you find out six weeks later, after eight dependencies rely on the hole. Matty says you have to "democratize your compliance."

Bridget recalls a line from Nicole's talk that Bridget tweeted: a raise of hands for whose job security and compliance are, with the point that all hands should be up. Nicole says it's not a joke. Anyone who touches a system is responsible for compliance, and before you even get to testing, the systems should be hardened, or you reach production with a gold-standard image and find the hardened images break the app. Nicole says you can't make everything compliant right away, so start with the lowest barrier to entry.

Make the Right Thing the Easy Thing

Anthony tells of a security team's preferred scanning tool, which Anthony won't name beyond a company that starts with an I and ends with an M, that the devs tried to get into their pipelines and could not. Once security acknowledged it wouldn't work for agile and went with another tool, about 60 dev teams switched in around two weeks, and every commit now goes through the scan. Bridget cites Andrew Clay Shafer's "make the right thing the easy thing," and Matty recalls the example from the book Switch of a machine redesigned so both hands had to be away from the blade.

Bridget asks what leadership does when it needs to impose a choice. Matt says "if you have a problem that every developer needs to solve, that should become a platform concern," and imagines a world where you don't opt out of certain parts of the CI pipeline. Artisanship is fine inside constraints, and Matt explains the thinking in systems terms: "committees are not scalable," since teams that haven't moved this way can't get a meeting for months. Matty adds that the list of things needing human intervention is shorter than people think. Writing the standard needs a human brain, while checking a system against it does not, and Matty quotes the Continuous Delivery book saying that asking a highly skilled person to do a boring, monotonous task "introduces more errors than inebriation or sleep deprivation." Nicole adds that the standards still need the right humans, those who interact with the systems, to supply context to committees, since you'll never slap a CIS benchmark document on and be 100% done.

One Pipeline Shape

Matty says Allstate's decision that there's one way of doing CI and CD means not 60 different teams, and a feature team's core competency isn't building a pipeline. Bridget calls that resume-driven development, and Matty offers "excitement-driven development." Matt says consistency matters for compliance because it makes deployments auditable, predictable and boring. Matty explains the shape stays the same even when Maven differs from another build, so you can look in one Jenkins log no matter the project. Bridget asks Anthony how to motivate people away from what's on the front page of Hacker News, and Anthony says it's a never-ending fight between good and evil, kept grounded by user focus. A team shipped from start to production without talking to the platform team once, which makes it worth it, and constraining the outcome leaves flexibility over tooling.

Culture, Tools, and Vendors

Nicole asks how hard it was to change the culture. Matt credits open-minded partners in security and compliance, and says the process is often based on tools already bought, so it becomes a financial conversation, with a sales rep having promised the tool would solve world hunger. Matty adds that organizations tend to throw good money after bad because a decision was made. Bridget notes vendors outnumber customers on the panel. Anthony says that when choosing between Cloud Foundry and OpenShift, the team minimized vendor interaction to judge how well they could run the software on their own, though some vendors won't give access to a download site until a big check is signed. Anthony doesn't advocate cutting vendors off, and credits a great partnership with one.

Matty says a good vendor partner wants to understand what you're trying to do, not force it a particular way, and Nicole, who is on the presales side, says it means asking the hard questions, such as why do it like that, and serving as a go-between for teams. Matty tells of Bridget referring a contact, who was having compliance problems, to Nicole about Inspect, which surprised the contact because Bridget works at Pivotal and Inspect is seen as a competitor's product. Bridget says the best thing for that customer is often Cloud Foundry, and Cloud Foundry is not Inspect. Matt adds a test for vendors: ask whether they say "yes, totally, I've done this before," or go find the one person in a 10,000-person company who may have kludged it once.

Closing Advice

Anthony says to apply the DevOps habit of small, frequent changes to organizations and people: "Small things, small incremental things." Matt says partnership, trust and credibility are the foundation, and that you may have to invest upfront in establishing them. Nicole says everyone is working toward the same goal, even with different paths to getting compliant or secure. Matty says if competitors can hug at a DevOps conference and "give each other hug ops," teams inside one company can too.

Bridget and Matt chat with Nicole Johnson (Chef), Matt Curry (Allstate), and Anthony Lee (Allstate).

• Nicole's GOTO Chicago talk: Automating Security & Compliance (for Fun & Profit)

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

• Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

• lots of DevOpsDays

Organizing a First devopsdays

Bridget notes the overlap between ops people and conference organizers, and asks Amy how the organizing team recruited a newcomer. Amy met Steve Pereira, one of the organizers, at a DevOps Toronto meetup about a year earlier, and asked a million questions about the role before saying yes. The appeal was a grassroots event in many cities, and Amy does community building in the day job. Planning one is "a full-time job in and of itself." Amy's pro tip is to talk a lot on Slack, and Amy admits the team may be annoyed by the number of direct messages.

Steve was too sick to attend, which became a test of the team. Amy says the organizers kept information in the group email and group chat, even for things needed from one person, and held regular meetings and kept shared files. When Steve said Steve couldn't make it, "there was almost just like a playlist that you could go through and check off." Amy adds that the alumni organizers were welcoming, which made the onboarding feel comfortable.

Foreground, Background, and Deep Work

Bridget asks Roderick and Arthur how they balance interrupt-driven work against heads-down work. Roderick says to be unafraid to say you need time to focus, pointing to research that "every time you're interrupted, it takes 25 minutes to get back to what you were trying to do earlier." The model from the talk splits a team in two: a foreground group handles the firefighting while a background group works on preventing fires.

Bridget wonders what happens when everyone who understands Project X is in the background. Arthur says that if only one person can answer a question, that's the problem to begin with, and the team is still working on spreading the knowledge. Arthur hopes to reach a "pair programming or pair opsing model." Bridget pushes back that deep contemplation and conversation are hard to combine, noting that Pivotal is a big fan of pairing. Arthur agrees that foreground pairs can learn from each other, then separate when someone needs contemplation, though the team isn't there yet.

On onboarding, Roderick says the team timeboxed the model for a month, and it worked well enough to keep iterating, and that being senior doesn't remove the chance to learn from other groups, which is one reason Capital One sponsors the event. Arthur credits Piyush Chugh, a former colleague now at Capital One Canada, with creating the foreground and background idea, and says Arthur took the credit at first.

Incident Commanders and Backups

Bridget asks Aaron what happens if the incident commander is a single point of failure. Aaron says the odds are low while an incident is active, but the best practice is to have someone who can step in, and adds that "best practices is a great word we all like to throw around and then pretend they're not real when we actually put them in place." Aaron says onboarding people to the incident process is about leadership setting the tone: being deliberate, and not letting someone work past capacity when they say they're fine, instead saying "I really need you to go home and get rest." The talk came from an incident that went well, and Aaron wanted to capture why, since the company culture side of incident response seemed more important than individual tooling.

Bridget says ops culture spends a lot of time on what broke and less on what went well, and asks Amy how events handle that. Amy says that on the day of an event, feedback is nearly instantaneous: are people smiling, talking, on time, did everyone eat lunch on time?

Keeping Questions in Public

Arthur says the challenge is redirecting conversations that would happen in private chat into a group chat where others can see them, since another engineer who has seen an answer before can give it when the one expert isn't there. Aaron says the team Slack has shifted from private to public messages, in part by reinforcing "default to the public channel," and by copying and pasting private messages to the public channel. Bridget adds that safety matters: if asking a question gets people laughed at, they'll ask in private. Arthur says senior people can use that privilege to show that even someone senior doesn't know everything.

Bridget chats with devopsdays Toronto local organizer Amy Mansell & speakers Roderick Randolph, Arthur Maltson, and Aaron Aldrich.

devopsdays Toronto 2017

Roderick & Arthur

• Deep Work: A New Working Model For Ops Teams In A DevOps Environment

Aaron

• Managing Fires: The Role Of Leadership In Crisis

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

• Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

• lots of DevOpsDays

Context Is Not State

Jeff's big lesson was about context. Alerting on everything can be information overload, so you have to frame the information to tell a story, because you don't want to assemble breadcrumbs mid-outage. The other trap is silencing all the annoying alerts at once, since the landscape of the incident can change, and "you end up fighting the wrong fire." Bridget notes that Bryan Cantrill's keynote made a similar point about alerts that depend on the very systems that are down. Jeff's example is a billing error that got an Azure account shut down, where the only alert was that the website couldn't be reached.

Mark says the context problem gets serious in long outages. The Heroku outage ran 67 hours for the last piece to come back, and most companies never think about what happens past 12 hours, when everybody goes into superhero mode. At Heroku they staffed shifts so that two people always worked the incident, offset by 50%, so that with an eight-hour shift, someone new arrived four hours in and could build up context while the previous person was still there. Jeff says a single person carrying an incident start to finish is probably storing all the context, which is dangerous for the organization and the individual.

Bridget asks why an oral handoff matters when documentation exists, and Mark separates "context versus state." State is what's up or down on the monitoring page, but the story that led there comes only from the person who lived it, who was too busy to write anything but notes, "marks on trees as they went through the forest solving the problem." Jeff adds that notes record facts, while context is also the theory being chased, and a test script left running overnight means nothing to whoever reads it. Mark adds "you don't write down the negative results," so the next person may walk the same trail.

Managing Up During an Incident

Matty asks about individual contributors on a bridge with someone seven layers up asking what's going on. Jeff describes Grubhub's common bridge with business and technical people, where business people need to message customers about orders and keep asking how long the fix will take, and "you have to give a number." Matty's answer is the Scotty principle, and Jeff's number is 10, with no units.

Jeff's fix is to separate the troubleshooter from the communicator. A troubleshooter gives periodic updates to one person, who becomes the parrot of that message, instead of repeating it each time someone joins the call. The other tool is a running Google Doc, which Jeff thinks was borrowed from the Google SRE handbook. People thought it was crazy at first, then the chat room link to the doc answered the status questions. Mark says the same idea shows up as an incident command role at Heroku and GitHub: the commander captures context and remembers that Jeff is looking at the thing. And "plot twist, that person shouldn't be the one who's communicating externally to your customers either," because choosing how to word a message about bringing a database back up, without saying "recovering" and making people think of backups, takes its own effort. Jeff adds that writing a 15-minute update takes 10 minutes, and that "jumping in to help isn't always helpful" if you haven't checked in with the incident commander, since you could undercut a debugging theory.

Matty asks what a sysadmin can do in an organization that has no incident commander, and suggests offering a struggling peer to act as a firewall for communication, then taking the result to management. Bridget suggests a shared doc built from the chat logs, which Bridget says aren't publishable because of the rabbit holes, and building it during the incident might prompt someone to ask whether they missed something with the database. Jeff points out that the first person to grab an incident is already the communicator, scribe and incident commander, so offering to help means asking "what do you need?"

Mark's advice for the person being harassed for status is to say "I don't know. I'll tell you something new in 20 minutes," and to give that update even if nothing has changed. Mark compares it to calling the cable company, where the cadence matters more than the answer. Matty calls acknowledgment huge for reassurance, and Mark adds "Not knowing is worse than getting bad news."

When an Outage Becomes a Disaster

Jeff asks how you mark the point where you concede you have a disaster rather than an outage, since a 20-minute update promises progress and a disaster declaration lets customers stop checking back every 15 minutes. Mark says "I don't know, but I know it when I see it," and describes stretching the cadence: if nothing new is expected in 20 minutes, say you'll update in an hour, or two, and promise to tell them sooner if you can.

Mark describes the Heroku case. In 2011 EBS went dark in US East, with a cascading control plane failure across multiple availability zones, and for the first part you couldn't launch apps or restart idled dynos, which took about eight hours to resolve. The long tail was the 200,000 Postgres databases, where some EBS volumes took up to 67 hours to come back or for AWS to give up. Jeff says that with consumer apps you can time a visitor's lifecycle and declare a disaster once the pizza order isn't coming. Bridget says sometimes you know immediately, and describes the 3:00 AM call from an on-call developer saying the HBase cluster was gone and Amazon said it was terminated. Bridget remembers knowing right away that the startup might be done, and it took days to resolve with broken backups, though the company did not go out of business and was later acquired. Jeff's takeaway is "the cloud is just someone else's computer," which doesn't absolve you of recovering from an Amazon failure. Bridget points listeners to the Who Owns Your Availability episode and tells them that if they haven't checked their backups, "you have Schrödinger's backups."

A Plan Before You Need One

Mark says that joining a company, one of the first things to look at is how it communicates during problems, because it's all about managing expectations. The plan covers frequency of updates, tone for the audience, like Heroku's technical readers versus Basecamp's small businesses, and canned responses. It should be prescriptive enough that nobody makes value judgments mid-incident, usually less than a page, and "you should be completely transparent," since misleading people and later finding you undersold the problem is the worst outcome.

Jeff says the way to test an incident process is to simulate incidents. Jeff has been thinking about a talk on pulling the environment's incident creation into the system, like Chaos Monkey for incident management. They ran Failure Fridays and War Room Wednesdays, and Jeff's example prompt is "I just socked Cassandra in the face." Jeff adds that it's best if someone outside the exercise says what's dying, and that with a War Room Wednesday coming you can pre-generate your templates. Bridget warns listeners to clear it with people before taking production down for fun. Matty says to start on paper: "Make your database cluster a sticky note on the board" and pull it off, as good a test as breaking the database. Jeff describes a tool called NetImpair that adds jitter, injects latency or cuts off a node's communications, which simulates failure from that node's perspective. Matty tells a story of turning off the wrong server at Allstate years earlier, with no way to decode the naming scheme, and waiting in the cafeteria to see if anyone came running.

Please Don't Be Me

Bridget says the reaction inside a company can't be "am I fired?" Jeff describes the week's AWS cost-savings cleanup, in which everyone agreed to delete 67 terabytes of logs on EBS volumes and shut down instances, which broke things. The engineer who ran it was paranoid about being walked out, and Jeff says you can't assume people know they're in the clear, so you have to reinforce it, before repeating the cold-open joke, "Don't worry, it's fine, you're fired." Matty adds that it has to actually be true first.

Matty says even when blamelessness is true, nobody believes it until they break something and don't get fired. Matty also says punishing mistakes doesn't reduce them: "It makes them become subject matter experts in hiding mistakes." Matty recalls John Cowie of Etsy on an earlier show asking "how amazing is it when the only thing that happens when you make a mistake is you learn something new?" and cites Etsy's three-armed sweater. Mark says you'll always have the first reaction of "oh my God, what did I just do," but the healthier one is "Please let that be me," because then you know the problem. Jeff says senior people can give air cover by publicly owning mistakes, and Mark notes that a GitHub co-founder wrote a public post about deleting the production database, thinking it was a test database.

Bridget says that even at a place that understood all this, the morning after the HBase loss came with the sinking feeling of being probably fired, and Mark says the healthy fear is whether you let customers down, not whether you will lose your job. Matty adds that someone who doesn't care about mistakes isn't someone to run the systems.

Jeff says a related burden is being on the hook for not knowing something about a system with 5 million lines of code, and Matty's example is the CTO who asks why you weren't monitoring for that. Jeff says "You're always fighting yesterday's war." Mark points to research by Richard Cook and David Woods: "The way that complex systems fail is fundamentally unknowable," so "Give yourself a pass to be human." Bridget says Tim Gross once sent an email quoting Etsy's principles of blamelessness after an outage when Tim was a director of operations at Drama Fever.

Dialects and Postmortems

Jeff describes a people-heavy practice: one scribe passes updates to key contacts across the organization, who translate them into their audience's dialect, so developers get a developer version and business gets a business version, and marketing gets what it will publish.

Asked for a final word, Mark gives a formula for public postmortems: apologize and mean it, demonstrate a thorough understanding of what happened, and describe what you're going to do, saying "we think it'll reduce the likelihood of this kind of thing happening again" instead of promising it never will. Mark says that can rebuild confidence to a higher level than before the outage. Jeff says to bring the people you communicate with into the process and ask what they needed that they didn't get, and that there are three axes for action items: reduce severity, reduce likelihood, and get better at detection.

Bridget and Matt chat with Jeff Smith (Centro) and Mark Imbriaco (Pivotal).

• Who Owns Your Availability?
• Wide Columns, Shaggy Yaks: HBase on EMR

Jeff

• Troubleshooting Tiered Tragedy: A Peek Into Failure
• DevOps in the Windy City

Mark

• Disasters!

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

• Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

• lots of DevOpsDays

Measuring the Squishy Stuff

Matty says it's easy to think you can't put science behind culture: "You can't put a Nagios monitor on a human." Nicole says people often respond by proposing to pull measures about people out of HR systems, which won't capture what teams mean by culture, which is high trust, information flow and collaboration across silos. The alternative is a proxy, and turnover is Nicole's example of a weak one, since someone may leave for a better culture, a worse one, a partner's job, or $1 million. Chat logs work only if Slack is the only way people talk, and Nicole's example of what no system will catch is a coworker dropping off a Diet Coke at a desk.

Nicole's answer is psychometric methods, meaning survey questions, done in a research-based way. The Westrum model is the example, named for the researcher Ron Westrum, whose work shows that in high-risk, high-performance teams, a culture that values information flow, high trust, risk sharing and boundary spanning predicts performance. Nicole says the Westrum typology was rewritten, with Nicole's help, into survey questions, six in the short version and seven in the extended one, and that they are open sourced for teams to use every quarter. In the DORA findings over the previous four years it was one of the highest predictors of delivering software with both speed and stability, and it also predicted profitability, productivity and market share. The 2017 State of DevOps Report was due June 15th.

Small Teams, Moving Needles

Tim has worked mostly in small and mid-sized organizations and worries that a survey would have no meaningful sample size. Nicole says "you can do it with small teams. It still works." Matty, who did this with Nicole at Chef and with customers, says to scope down to the size of a feature team, and adds that Matty doesn't care what number a team lands on, since there's no magic score. What matters is that the parts important to you move, which means asking regularly, not once for a pass or fail.

Four Principles of Software-Defined Culture

Bridget asks Tim to walk through the four areas from the talk, and Tim says the four are reliability, operability, observability and responsibility, and Tim takes them out of order. Tim jokes that they're a map, not an array, and Matty adds that it will be unsorted every time.

Reliability

Tim says unreliable software has knock-on effects on the organization. People up all night because of bad on-call burn out and fight with each other, and chasing the shiny normalizes risky decision-making. Nicole says teams need to understand that taking risks is a safe bet and that risks will be shared. Matty cites Charity Majors, who would say that if you haven't broken production, you're not trying.

Nicole says the S3 outage was a favorite case, because the published postmortem described a fat-finger incident and "nowhere does it say human error," which speaks well of both the culture and the systems. Bridget takes from that that building for reliability implicitly means not blaming people when reliability falls short. Matty agrees that "you can't work around human error," since people are going to make mistakes and that doesn't scale. Tim adds that many system measurements are themselves proxies for culture. Uptime, for example, is often a proxy for what your people are doing, and Bridget says it may just reflect whether you patch. Nicole says "anything that's a metric becomes a proxy," representing something in someone else's head, and Nicole recalls that as a hardware performance engineer, response time was "my jam".

Operability

Tim says operability is about delivering quickly and about keeping an application's behavior understandable and self-contained with the team that owns it. Tim objects to the trend of pushing intelligence out of the application into a third party or a platform, because it creates a cultural imperative that it's fine not to understand these things, and it widens the gap between the platform team and the development team. Matty says black boxes let people treat a problem as someone else's, and supply an excuse: if I don't understand how that works, how could I have done it better? Bridget sums it up as "microservices are a game of point the finger and plausible deniability," and Matty says "All of IT is a game of point the finger." Nicole adds "Wait, you mean containers won't fix my culture? What?"

Nicole says metrics shape culture and can help teams communicate across boundaries, but they turn problematic when a team throws a container or an app over the wall with a metric that only makes sense to that team. Bridget asks what makes or breaks operability, and Tim says the measurements have to mean something to the consumers of the system, not serve as cover. A vague service uptime number isn't what a consuming team needs. They need to know whether they're being throttled, or whether clients should refresh service discovery. Nicole says to tie metrics to a line-of-business goal, and Bridget cites James Turnbull's argument in The Art of Monitoring that you are not the consumer of your metrics, which Bridget says ops-focused people forget when they build dashboards around what wakes them up.

Outcomes, Not NGINX

Matty says that outside the echo chamber the simple point still needs making: "Outcomes are like the only thing that matters," specifically the business outcome, and for a nonprofit, Tim adds, that is a mission. Matty recalls a sysadmin freaking out that SQL Server was using all the memory on the server, when that is what the memory is for. Matty also says to write a Chef test for whether a web server does its job and not whether it installed NGINX, and Bridget says the test should look at ports 80 and 443.

Matty retells a story Sasha Bates told on the Ship Show about working for a large retailer right before Christmas, when a product team wanted to push a release and Sasha objected on stability grounds. The boss's answer was "your job is not to keep the website up. Your job is to deliver the features that the company needs." Bridget's reply: "I think the company might need a feature of being up."

Nicole tells of chairing the LISA conference in 2014, when Courtney Kistler, who had been leading the Nordstrom transformation, stood in for a closing keynote speaker who had a medical emergency. The ballroom of old-school sysadmins was restless about a talk on business transformation, so Nicole and Tom Limoncelli told them to listen, asking "Who the F do you think you're keeping email servers up for?" After about ten minutes of business translation, Nicole says, the room was into it, and "Courtney won them over hard."

Responsibility and People

Tim says the fourth principle is about externalities. When talking to people about containers, Tim says they don't care about containers: they have a mission, and people whose lives should be fulfilling. Tim adds that people are not just there to fulfill the mission, and on the CEO's duty to shareholders Tim says "fiduciary duty, which is bullshit, by the way. That's not actually a law."

Nicole supplies data: over four years, the DORA research found that employees of high-performing teams are 2.2 times more likely to recommend their organization as a great place to work, and research from Harvard found that employees who recommend their workplace predict higher revenue growth. So "even if you want to be a selfish asshole," making the workplace better increases hiring, retention and revenue. Bridget asks the room who is hiring, and every hand is up. Nicole says hiring costs more than retaining, and "Don't be a jerk." Tim says Tim's gut says the same thing, and it's good to see data.

Observability and Debuggability

Tim says the observability section moves from traditional monitoring toward tools for exploring systems iteratively and collaboratively, not a lone sysadmin watching a dashboard. The stronger point was debuggability, which Bryan Cantrill discussed in the keynote. When a stack has a black box, whether the operating system, a platform or a web server's event loop, people end up saying "and then magic happened." That leaves software less reliable and is dissatisfying for technical people, Tim says, because "This is all software. It's not magic."

Nicole adds that it's not enough to have data: you have to act on it, and the highest paid person in the organization "sucks at this. Use your data." For teams without instrumentation, Nicole says to start by asking people. Can you roll anything out without asking other teams? Are you testing, have you shifted left on security? Bridget adds that if you can't deploy a microservice without deploying three others, you may have built a distributed monolith. Matty says to do it iteratively: one question is better than none, and it's information you didn't have yesterday.

Nicole says some things only surveys can give you. Systems can tell you what's in version control but not what isn't: "Only your people can tell you what is not in version control," and what is bypassing your systems. Bridget asks Tim about the automation Tim built to capture the state of an AWS account before a migration, and Tim says you can't capture everything, so you start from the top level. That meant documenting the network first, "because nothing else runs without the network."

Nicole cautions that instrumenting the easy thing for a quick win creates a pile of metrics, and once something is measured people start paying attention to it. Matty adds that it creates a culture that cares about CPU. Tim asks whether asking people questions has the same effect, and Nicole says it does, because collecting any metric sends the signal that it matters. The problem comes when it turns into a demand to answer 10 on a 1 to 10 scale or else. Matty says the survey for the car Matty had just bought worked that way, and did the same for a Microsoft TAM years earlier, where a scale of 1 to 10 was really pass or fail.

Closing Advice

Nicole says to start measuring, do it honestly, and keep measuring periodically: "Even a bad baseline is super powerful." Tim agrees and adds that you will chase what you measure, so measure the right things. Nicole is glad they agreed, and Tim says there was no screaming on this stage at all.

Bridget and Matt chat with Nicole Forsgren (DORA) and Tim Gross (Joyent).

Nicole

• Be Awesome With DevOps (Through Data!)
• State of DevOps Report
• Westrum model of organizational culture

Tim

• Software-Defined Culture
• ContainerPilot

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

• Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

• lots of DevOpsDays

The Nineties Sucked, and Then Everything Closed Up

Bridget asks how we got into the pickle we're in. Bryan and Andrew dispute the pickle, but Bryan is clear that "the '90s really sucked": a very proprietary, closed era in which people thought systems were done. Andrew points to the dark ages of the relational database and the Java middleware stack that "totally paused everything for a decade." Bryan adds that Java was not open source, and that Windows was deeply proprietary, "then there's asshole proprietary." Bridget sees real willingness to open source coming out of Microsoft now. Andrew says Microsoft has been forced to, since it lost its monopoly.

A New Proprietary Era

Bridget says the proprietary era is over, and both guests disagree. Andrew says that once you're in the cloud, it doesn't matter how much open source built the stuff at Google or Amazon, because you can't change that code. Bryan says "We are in a new proprietary era," and that it rhymes with the '90s, when everything was going to Microsoft and doing anything else was stupid. Now everything goes to AWS. Andrew says "Nothing's more proprietary than Lambda."

Bryan argues that the great myth Amazon created is that cloud is a terrible business nobody should be in, when cloud computing has very good margins and Amazon's overall margin is very low. In Bryan's telling, AWS is underwriting a war on big box retail, and a lingerie retailer that was a Joyent customer answered Amazon's $9 bra with "no, forget it". Andrew says Amazon runs analytics on what sellers and partners do in its marketplaces and turns around with its own products on both the retail and cloud sides. Andrew calls Jeff Bezos "the Genghis Khan of the internet." Andrew remembers Rackspace saying it was playing a different game than Amazon, after throwing a Hail Mary with OpenStack, and Bryan says "Genghis Khan does not follow you on Twitter. He does not care."

The Serpentor detour begins with Bryan raising Larry Ellison's philanthropy and a longevity institute, and arrives at cord blood. It ends with Bridget remarking that vampires and zombies came up earlier in the day than expected.

Lambda, State, and the Borg Paragraph

Bridget asks what the two see among enterprise customers, given all the hype that Lambda functions will save us. Andrew says nothing ever goes away, and describes sedimentary layers of mainframes and Java middleware. Some places are like opening a time capsule, where "you can kind of tell what year they stopped learning." Bryan agrees that mainframes still run but are not a growth area, since nobody holds a conference of several hundred people on z/OS.

Andrew says people over-rotate on running a function. What Lambda represents is a fabric of event sources, and S3 is one, so "you can't build useful things with functions, stateless functions, until you have these things." Bryan says Lambda is "a needle exchange for AWS services," and Bridget adds DynamoDB as the other example of services built to keep you in. Bryan, from "stateless land," is asked whether there's state in the world, and Andrew says there is, along with all the hard problems. Bridget says state is all the customer data and money people do business because of.

Looking ahead, Andrew says new applications should be much more aware of their own state. Andrew recommends the Borg paper, which has pages on schedulers and then a single paragraph that Andrew considers more impactful than the best scheduling algorithm: every application running in Borg has an HTTP endpoint that broadcasts metrics about its health. Andrew says that if you did just that in your applications, "you'd get 85%, 90% of the benefit of the way Google runs their applications." Bryan calls Prometheus the Google idea taken to the open source world. Andrew adds that frameworks should make doing the right thing the easy thing for the app developer, and that people are already struggling to monitor Lambda infrastructure.

Bryan wonders how much Lambda will be used in anger rather than for prototyping, noting that the people who love utility billing are utilities, since you can't predict costs in a utility model. Bridget describes a Minneapolis meetup talk from SPS Commerce engineers, who used Lambda a year earlier and then built their own Lambda-alike in-house because Lambda billing didn't fit. Andrew says that when people cite cost savings from Lambda, it's because they were running mostly idle compute instances.

Open Source Doesn't Die

Bryan says "the open source business model is the second worst business model on the planet," and the worst is being a proprietary infrastructure software company. The Docker CEO had just been replaced that morning, and Bryan recalls the painful period at Joyent when parts of the stack weren't open source. On OpenStack, Bryan says the problem it was solving was a middle management problem in soon-to-be-dead infrastructure companies, and Andrew disagrees that this was the initial goal. Andrew says it became a weird political marketing exercise with little engineering in the core, and that it pulled attention from other promising projects. Andrew's history begins with Eucalyptus, which Andrew says had the birthright to be the open source cloud and mismanaged its community, so that "OpenStack would never have existed if Eucalyptus didn't mismanage its community in the beginning." Bryan says both made the same mistake by trying to be an open source AWS. Andrew's view is that Amazon's advantage wasn't necessarily software but the socio-technical system that had run a massive distributed system for a decade, and that many of the organizations trying to build clouds couldn't manage a multi-node Rails app.

Bryan argues open source outlasts any company: "because open source software can't die." The system Bryan works on descends from Unix, with parts 40 and 50 years old. Postgres was dead on the operating table for a long time, Bryan says, and was revived when things changed. Bryan insists that Illumos has been independent of its Solaris roots for seven years, and that Solaris "is just a very brief proprietary era in a much longer system." Andrew says the Linux and Solaris kernels differ in the quality of engineering, especially around observability. Bryan likes small communities that share values, and says large ones are a mixed blessing. Node.js is the example: Joyent was the company behind it, and Bryan says Joyent and the V8 team shared values around observability, debuggability and rigor that the broader community did not. The community wanted promises, which Bryan calls a bad model "in the operability of software years down the line." Bridget sums it up: day one is very short and day two is forever.

Doing It Right Now

Bryan says doing things properly initially makes you faster than the limit, and that executive leadership has to understand it. Bridget pushes back that doing it right the first time is seductive and impossible, since nobody has perfect future knowledge. Andrew says "there's nothing more expensive than building the wrong thing," and that sometimes testing a hypothesis with a hack that lives forever is cheaper. Bryan says the complaint isn't about prototypes but about the corners you know you're cutting, like being in someone's code that has never been executed and thinking it would have taken an extra 20 minutes. Andrew says every developer on every keyboard is choosing "between doing things right and doing things right now," every second of every day. Bryan passes on advice from a senior engineer that every line of code is a business decision, so an organization has to value not cutting the corner and "be a craftsperson." Andrew adds that values are about who gets rewarded, and that on a resume with stints of 18 months, 18 months, 18 months, each move often came with a big raise. Matty says the same pressure shows up as analysis paralysis among Matty's customers.

Integrity

The reward point sets Bryan off. Bryan says the leadership principles of Amazon and similar organizations leave out integrity, noting that Amazon has 14 leadership principles and integrity is not among them. Bryan contrasts that with corporate values of a generation ago, when integrity topped the list, and says "we have stopped aspiring." Bryan admires a final email to Sun employees saying that in 30 years there had been no need to hide the newspaper from the children, and sets that against Uber's behavior. Andrew asks whether Uber has been rewarded or punished, and Bryan answers that Uber's valuation is pretend. Andrew pushes back that many developers have mortgages and families and are managed like factory workers from the Industrial Revolution. Bryan says they are not, that "our children don't die," and that anyone using their brain in a cube farm is among the haves. Andrew answers that the choices people make to get through the day under imposed management structures aren't something Andrew will begrudge them.

Closing Advice

Bridget asks for a closing statement in 60 seconds. Andrew says "This might not be the podcast you wanted, but it's the podcast you needed," and that integrity starts with individuals: to build structures that have it, you have to think more globally about politics and the implications of decisions, and elevate yourself inside your organization or by creating new ones. Bryan says we live in a great time, making "castles from thought," and "We are literate in a society that is broadly not literate," so it's incumbent on us to increase literacy. Bryan adds "Anyone who bets against humanity is simply ignorant of history," and tells listeners to find their own motivation and take the luxury of being true to themselves. Andrew's last word is that the panel might not have yelled at the cloud, but "we definitely yelled."

Bridget and Matt chat with Andrew Clay Shafer (Pivotal) and Bryan Cantrill (Joyent).

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

• Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

• lots of DevOpsDays

Start With Why

Matty says that a couple of years ago the statement was that DevOps won't work at big corp, and now the question is how to do it at big corp, which is a fundamental shift. Bryan says how is easy, since you can read a book, and what people miss when they look at Google or Dropbox SREs is why: why they needed that solution and what issues you have. Without understanding why, you never know what success looks like. Even at Capital One, Bryan says, the DevOps teams bring in CI/CD, which is a tiny smattering of what DevOps is.

Matty describes a CIO buying the big-picture message and never getting it to the boots on the ground, who conclude a DevOps initiative means installing Puppet. Matty recalls Jez Humble saying Jez will have a job for life because nobody gets this, and says leaders are now asking people to come to a town hall to explain the principles, not the product. Bryan says to talk about CI, CD, logging, monitoring and declarative infrastructure as principles first, then bring in tools, and Matty asks how you pick a CI tool if you don't know why you're doing CI. Bryan says a big company will just tell you it uses Jenkins, with 20 or 30 people on it, and the conversation loses whether the tool is helping ship code or just being used. Bryan says developers are often given a problem with constraints and told someone else handles ops.

Strategy, Practitioners, and Constraints

Bryan says at a larger org leaders such as directors, VPs and CXOs should talk strategy and where they need to be, and let practitioners figure out how to get there, with a feedback loop. Because tools like Terraform and GitHub Enterprise cost a lot, higher-level people get involved, and should realize they are there for financial support and direction. Matty adds that outcomes are all that matter, and Bryan adds that they must be repeatable, since if you can do it on Tuesday you'd better be able to on Wednesday.

Bryan's view is that "Constraints are a great thing." People who say their startup's problems would go away with more money are naive, since projects with lots of money aren't guaranteed to succeed, and Google succeeded partly because it was trying to do it with fewer people. At Capital One, moving into AWS, they hit account limits, like how many VPCs can be peered, how many images you can copy across regions at a time, and how many security groups you can have. Bryan's example of copying an AMI between accounts, when you have hundreds of accounts and can copy five at a time, means deploying images alone can take more hours than a week has, which forces creative solutions that wouldn't have appeared with unlimited freedom.

Cloud Custodian and Why Enterprises Matter

Bryan says open source is hard at a bank, with lots of lawyers and regulation, but they created Cloud Custodian, a governance tool they released to the community. It enforces 100% encryption at rest: Bryan booted an instance with something unencrypted and got an email 30 seconds later, and again after trying a second time. It also lets you limit things like which instance types can be booted. Matty says that feedback loop through open source and vendors matters, since compliance ideas on a whiteboard differ from a customer's real compliance issues, and someone will be the first at a given industry's problem for Pivotal or Chef.

Bryan says enterprises are also where the money is for vendors, since companies like HashiCorp can't live on VC money forever and have to work with bigger companies. Matty had written off enterprises as where innovation goes to die, and being on the inside as a vendor changed that: it's harder, but the payoffs are better. Bryan says "we're a big, slow-moving ship, but guess what? When we turn around and we actually point in a direction, you get all the force of that ship." The problem is people: ten people have about 100 conversations, and 10,000 people have far more, and Matty says that means 10,000 conversations where good ideas come from. Bryan's favorite thing about working at a big company is that "we have internal tech conferences that are bigger than some conferences that I've been to," where people can talk about specifics without guarding. Matty says a healthcare customer's conference is bigger than every DevOpsDays combined, and that Matty can't go to the talks.

Enterprise DevOps and the Security Door

Matty says "let's all temporarily mourn the term enterprise DevOps and be glad it died," since it was really DevOps without the culture. What Matty sees now is that the entry point into an organization has shifted from ops or dev to security and compliance, since InfoSec people love it. Bryan says governance and compliance are hard at scale, and Cloud Custodian showed there is a marketplace for it, though compliance isn't Capital One's business. Bryan also notes that being in tech at a financial company means not needing to know how the company makes money, and Matty says at Chase the line of business only became clear on updating a resume: $1.3 trillion a day in wires went through the systems Matty managed.

The Changing Role of Ops

Bridget asks about Bryan's Twitter thread with Alice Goldfuss and Kelsey Hightower on ops. Bryan says there are two Bryans, the worker and the person, and the worker can do things wrong without being a bad person. Likewise, "Bad ops teams are teams that don't want to get better," as distinct from overloaded ones, and they need to go extinct. The advice is that you're responsible for your output, and if you can't effect change, "you shouldn't be in a place where you can't make positive change." Bryan says velocity is increasing, and if ops sits on the sidelines saying it's too hard, someone younger will take the job. Bryan is over 40 and expects to keep doing this because times change, from Sun pizza boxes to other people's hardware in data centers. Bryan says to change the solution: if the dev team throws things over the wall, use the SRE book's production readiness tenets and tell them you have production requirements as well as business requirements. Bryan says it's all about being the adult and owning the idea of change for the positive.

Closing Advice

For anyone at a large enterprise, Bryan says to distill a problem to the basics: you're not curing cancer, so solve one little change, then combine them. Startups and enterprises are similar, with a few more rules because more money is on the line. Bryan adds that a big enterprise gives women and people of color more opportunity to move up, and Bryan sees more Black VPs and women at the SVP level than ever before. Bryan leaves listeners with a question: what did you do yesterday, and how can you make today better, and to do something for someone you hadn't yesterday.

Bridget and Matt chat about devops in a large enterprise with Bryan Liles (Capital One).

Check Outs

Matt:

• GFM supports folded details (like, disclosure triangles)
• Tables Generator
• Hugo plugin for Atom written by me
• hub is a cool tool

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

• GOTO Chicago - Matt and Bridget hosting an entire day of Arrested DevOps Live! May 1-2 - $75 off with discount code "arresteddevops"
• Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

• lots of DevOpsDays

Image credit

The Crossroads

Nicole says the July conference found Nicole at a crossroads, having left academia just before tenure and spent about a year and a half at Chef, while DORA had been a side project that was getting big. Other companies were approaching Nicole, who didn't know whether to join a bigger one or run the startup. Bridget told Nicole to talk to Charity, whose advice was to decide what Nicole wanted, and if taking a job, to be transparent about the other project and promise a company only a year. Nicole says being about a year out hadn't been articulated until then. Charity says it was clear Nicole was in love with the thing. Nicole took over DORA, committing to Jez and Gene, who had been trying to convince Nicole for months.

Charity says that at the time of the last conversation, about six months had been spent heads down after a rough co-founder breakup, without knowing whether what they had was worth doing. Around Minneapolis, they decided it was real, and time to push the baby out of the nest. "The first 2 months that you were showing your product to people, you should be humiliated. You should be embarrassed." Now they have their first paying customers and are working through roughly 800 signups from Charity's Twitter feed. Charity says what's hard is to shut up and listen, and that hearing Bridget pitch Honeycomb to others was painful and one of the most valuable moments of recent months. Nicole says the same about the channel partner program.

Why a Startup

Nicole says the startup did the hunting. People kept asking to compare themselves with the industry using the State of DevOps data, and some things can only be measured by people, not systems, so Nicole and colleagues built it. Nicole swore never to do a startup and is completely risk-averse. Charity had always been an implementer, not an ideas person, and had assumed a startup would come at some point since it seemed like a missed opportunity in Silicon Valley. The core truth, Charity confesses, is disappointment with the roles offered coming out of Facebook and a successful startup: employers wanted a proof period as an individual contributor or a manager of three people. After a career of saying titles don't matter, Charity found they do, and "it was rage." Bridget is also angry all the time, and it fuels the work. Charity also notes pedigree counts for a lot in Silicon Valley, and that Charity was never going to be more fundable than right then.

Charity says "Nobody's a startup person," roughly 100% of people, but once in a while something smacks you, and it isn't always the idea. Sometimes it's the team, and the idea comes later, and everyone retcons it as if they always knew.

Teams and Balance

Nicole says the partnership with Jez gelled, and "individuals don't make software, teams do," so sometimes when a person leaves, you take the team. Charity says when your strengths and weaknesses mesh with someone's, it's magic. Charity's co-founder Christine says little but is right every time, so Charity copies Christine on everything. Nicole describes the balance of the three founders between focus and creative openness. Charity adds a rule: "Christine and I never freak out at the same time." They joke about whose turn it is to freak out and who's going to be calm in the next conversation. For the first six months it was mostly Charity, who hadn't expected to be CEO, and then Christine couldn't carry them any longer.

Hiring for a Startup

Charity says the risk profile is different: at a small startup each hire is one of two or three people and materially shapes the product, which Charity says is true of everyone they've hired, up to seven or eight. You're building a family, technical skills can be taught, but caring about what you care about is hard to tease out and matters more than anything. Charity has never simply interviewed and hired, and who Charity wants to sit next to eight to ten hours a day matters. A big company needs a filter, Charity says, but Charity criticizes Facebook for saying they couldn't lower the bar while having five women in production engineering out of 350, and admitting they found no correlation between interview scores and success. Bridget says a diverse interview panel prevents unconscious bias. Charity says "Great teams can fight well," like couples, and Nicole adds it's how well you resolve conflict. Charity favors transparency below ten employees, including cap tables, and shielding people only from chaos that would distract them.

Funding and Staying Lean

Charity says they took $2 million because it was offered on very good terms, are extremely fiscally conservative, and are racing to profitability. When people celebrate raising $100 million Charity sees failure to execute with what they had, and says giving up equity and control is often the unmentioned part. They are raising enough to get comfortably to break-even, so they can walk away from terms. Nicole says DORA is revenue-funded with no investors, which is the best way if you can do it but limits growth, so Nicole uses contractors for accounting, design and copy editing and is feeling scaling challenges. Bridget mentions Honeycomb becoming an ISV partner with Pivotal, and partnerships as a way not to hire.

Charity says startups need generalists who are fine changing their minds, and that you don't hire into a role until you've done it enough yourself to know what makes someone successful and have six months of work that justifies it. Charity adds that success shouldn't be measured by headcount but by what you deliver divided by the number of people, and is irritated by VCs and acquirers asking how many bodies Honeycomb has, while turning away world-class engineers.

Advice for Joining

Charity says the younger the startup, the more ownership comes with it, and you should look for founders aligned with where you want to be. Charity dislikes founders with a "them and us." People should ask far more questions, since many don't know to ask how many shares are outstanding, and look for ways to enrich the relationship before signing, perhaps with a trial period, or by taking friends who know out for drinks. Nicole's advice is to know who you are. Nicole is a big-company person who likes titles and transparency, and jokes about the nickname Crusher of Dreams, earned by spotting things that won't work and needing to be high enough to say so, with a filter that drops in front of the mouth. Charity adds that in your first 10 years you should push yourself, trying big and small companies, and "Running towards being uncomfortable is, I think, generally good life advice." Nicole says give it six months, and Charity says learning to tell good uncomfortable from bad uncomfortable is part of growing up, and that after staying a year and a day at two startups, Charity would now cut it short in a week.

Bridget chats about startups with Charity Majors (Honeycomb) and Nicole Forsgren (DORA).

Check Outs

Charity:

• Honeycomb has a public announcement/launch on April 11
• @goturbine for splitting traffic
• also check out buoyant.io

Nicole:

• DORA has an ROI white paper coming out soon
• State of DevOps Report releases June 7
• Book with Jez and Gene; early release is summer/fall

Bridget:

• Systems We Love - coming to Minneapolis March 16th!

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

• GOTO Chicago - Matt and Bridget hosting an entire day of Arrested DevOps Live! May 1-2 - $75 off with discount code "arresteddevops"
• Velocity San Jose - discount code "ADO2017" gives 20% off for Gold, Silver, and Bronze passes.

Open CFPs

• lots of DevOpsDays

What's Working

Chris says DSC works, with the caveat that it's challenging. Brandon says it's automation pipelines and ChatOps to expose tasks to groups who don't have the skills or access. Glenn says PowerShell has been an absolute saver, and Reuben adds PowerShell and Chocolatey for distributing everything from desktops to test environments. Matty notes that a robust shell and package management are what the Microsoft world lacked, and that the old automation in VBScript was really macros. Brandon says they don't like to talk about the VBScript days, and Chris says it took about 10 years before they had a way to package and distribute modules.

Open Source Comes to the Windows World

Matty asks about sharing code in a historically closed-source platform. Liam says many organizations needed Microsoft to make the first move, and that "it required Microsoft to make that first move," because after Ballmer stepped away in 2014 the company got permission to change, and that gave everyone else permission. Chris says Microsoft had a history of building its own version and killing an open source project, and now seems to contribute instead. Glenn says that having worked for banks, military and government, contributing outside the firewall is still a big challenge. Brandon says using open source is a lower barrier than contributing back. Matty says it's often lawyers worried about setting precedent for other IP, and about telling apart plumbing from what's different. Chris says permissive licenses like MIT have helped, since companies now separate plumbing code from business code.

The .NET Developer Stack

Matty asks whether anyone deploys with Microsoft's stack. Reuben's team uses TeamCity, not VSTS, and objects to building differently on a server than locally, and to the one-pane-of-glass developer experience and right-click publish: "their definition of done, when they're finished is just a package," which is fine for a monolith but makes ops people mad in distributed computing. Matty recalls the stack being built around a developer with a codebase spraying it somewhere, config transforms and a per-seat license of $8,000 to $10,000 for every sysop who needed to deploy, so everything ended up manual.

Liam says culturally "you win most of these places over just by making it easier." Once you beat the Microsoft tool once or twice, people begin to question the rest. Liam sees TeamCity and Jenkins increasingly, and the source control part of VSTS dying quickly as Microsoft itself moves to Git, with the Windows core apparently 40 billion lines of code. Matty ties it to the old "Microsoft is the answer, what's the question" attitude, and to Microsoft's own not using SCOM to monitor microsoft.com. Matty adds that the company has improved by decoupling, and that its leadership and engineers get it, while middle management and sales might not.

Bringing Puppet and Chef to Windows

Glenn says some people used Windows in the dark days and never want to touch it again, and some new ones love Visual Studio, and Glenn has even met a Linux admin who loves PowerShell. The bigger shift is teaching people to model architecture, since Windows admins are used to next, next, next, finished. Matty compares it to Microsoft's own Office for Mac lesson: a tool should fit where its user's comfort zone is. Glenn says Chef and Puppet need to do a better job onboarding Windows people.

Matty recounts Jeffrey Snover's line that Linux is a document-based operating system and Windows is API-based, which is why IIS, being document-based, is the easy example. Bootstrapping is easy on Linux with SSH, while enterprises turn off WinRM, and securing it means running a command and setting registry keys, not dropping a file. Brandon says onboarding matters because if Windows admins hit roadblock after roadblock, they'll give up. Glenn praises the Docker for Windows installer as the best of both worlds. On DSC, Chris says it's "delightful for about one server, and then after that it starts to go downhill," and Brandon says you need a management platform, which is where Chef and Puppet come in. Matty quotes Snover that DSC is a printer driver and Chef or Puppet is Microsoft Word.

Maturity, Culture, and Table Stakes

Liam says initial usability of the tools is pretty good now. The roadblock is the second and third step of maturity, managing a whole fleet, and that story isn't told well. It depends on the company: if config management is in an internal IT organization that doesn't think of IT as its business, the jump feels epic. Matty says it's Conway's Law, and that in a low-trust culture people argue with math, as Adam Jacob says. Matty also notes that what vendors consider table stakes, like testing infrastructure code, isn't. Brandon says quite a few admins still use ".bak as their source control system," and one colleague with 30 years in IT said they were learning their job all over again.

Chris thinks Linux configuration management came from developers and Windows from ops. Matty has seen many Windows admins shred PowerShell and many Linux admins who want a click-button tool, and "being bad at command line knows no operating system." Liam calls this a golden age of managing Windows, with config management, Terraform and the PowerShell gallery, yet just as people reskill, the industry pulls the rug out with containers and .NET Core on Kubernetes. Liam sees mostly the other end, legacy apps never written for this, and cites that apparently 30% of Windows workloads on AWS are 32-bit.

Advice and Daily Drivers

Matty's advice is not to boil the ocean or fall into analysis paralysis: don't design high availability before you have one node under management, write some code and improve later, and avoid yaks. The principles, like continuous delivery practice and thinking of infrastructure as code, carry over when tools change, and they are harder than learning any syntax.

Asked for their daily drivers, Brandon uses an old Mac Pro running Windows 10 through Boot Camp, and PowerShell is the indispensable tool, Chris a corporate Lenovo with Visual Studio Code, Glenn a MacBook Pro with Windows 10 and no macOS, with Hyper-V and VS Code, Liam Ubuntu with Atom and a Windows VM, and Reuben a Surface Pro 3. Matty notes it is ironic that VS Code's Git integration is better than Atom's. The checkouts include Liam's pick of Terraform and a forthcoming book on it, Brandon's Operation Validation Framework and Pester, Reuben's Serilog and Seq, Glenn's Neo4j and Flow Perth's hack days for nonprofits, and Chris's GitKraken.

DevOps Cafe w/ Jeffery Snover - Linux is docs based, Windows is API based

Check Outs

Liam

• James Turnbull Terraform Book
• Black Mirror - My latest show to binge watch

Brandon

• Infrastructure testing with Pester

Reuben

• Paramore, aka Brighter - My goto reference implementation for understanding how to write robust “microservices” that gives you options
• https://serilog.net/ & https://getseq.net/ Logging is the “new” debugging...
• Phil Haack - Be the scientist

Glenn

• neo4j Graph database- Works on Windows too!
  Free OReilly ebook on graphdatabases
• Flow Perth create unique events in the technology community bringing skilled volunteers and not-for-profits together

Chris

• I’m a big fan of GitKraken. I can actually do some useful things with Git without spending a couple hours reading man pages.
• termeter - A Go app for “rendering” ascii graphs in the console.

Trevor

• Blue Raspberry Portable mic I’m looking at getting, tired of lugging the Yeti around (much like the 17” laptop I abandoned for my Surface)
• Factorio - super fun collaboration game
• pvpgn open source classic Battle.net + Westwood server, got it running- trying to stand it up inside habitat now

Matt

• Maybe weird to talk about Apple-only app on the Microsoft show, but I’m now enamored with Bear Writer - just a nice markdown notes thing.
• I’m kind of obsessed with Golang now - the go Fundamentals Pluralsight course by Nigel Poulton was pretty dang rad

Bridget says the university is worth a visit: it has a giant Android lab and its own Linux distribution, and it sits about a 30-minute drive out of Havana on an old Soviet military base, along a road that was once used as an airstrip. Joe warns the audio sounds like a big echoey room because it was one.

The Closing Session

The recording begins with Hugo, an organizer, closing the fourth day, an intense few days in which Hugo saw people learning a lot, and says of the Ignite talks that "everybody was on fire." Hugo thanks the invited speakers, including Patrick, whom Hugo calls one of the founding fathers of DevOps, which is a rare opportunity for Cuba, and who answers "Two words: thank you." Hugo also thanks Bernard, Bridget, Mike for making everything understandable, Joe for the podcast, the local team, and a sponsor that covered all the expenses. Bridget adds that none of it would have happened without Rudy, who has spent years building the Belgian and Cuban cooperation, and leads a standing ovation.

Attendees then line up, in Spanish, with Joe translating in the episode. One says the event was great, interesting, instructional and intense, and that some of what was discussed will make life better and some will make it worse because they'll want to rush to implement it, but that the best part is the friendship. Another says the best part is the sharing of information, knowledge and experience among professionals. Maria Lina thanks the invited guests for dedicating their time and knowledge. A representative of UCI says the event helped identify bottlenecks and important things to do. Bernard says it is a pleasure to have been invited to be in Cuba, with "good weather, good food, good company, good conference."

Patrick and the Future

Bridget asks Patrick Debois, who started DevOpsDays in Belgium in 2009, what was different about this one. Patrick says in other countries people are saturated and wonder if they can learn one more thing, but here people started sharing from the first exercise, with the same enthusiasm as in 2009, when everyone left to take it home to their friends. Patrick says the retirement from DevOpsDays became official two years earlier, and that Bridget is now in a good lead, helping people organize elsewhere.

Bridget asks Enrique what comes next, for the conference and for DevOps in Cuba. Hugo answers, and Joe translates, that that the survey results are coming, everyone wants to do it every year and to keep developing the cultural movement of DevOps, and the idea is to share, and to work with all companies to advance.

Ignite in Spanish

Joe, a fan of the Ignite format, says the Cuban participants had never heard of it before they were supposed to give them, and took to it like fish to water. Ignite is a five-minute talk with 20 slides that advance automatically every 15 seconds, which is difficult for first-timers. Bridget says English-speaking Ignite speakers manage about three sentences per slide, while the Spanish speakers were probably putting about six or seven. Joe says they were going a mile a minute, and even catching every fourth or fifth word, they were really good. Joe singles out Henry, the "on fire" speaker, whose Ignite was one of the lead-offs, and Maria Elena, whose evening talk on telenovelas was very funny.

Bridget says of the year's conferences on five continents that this one stands out as significant. The show ends with the banana stand line in Spanish, and Bridget's note that in Cuba the bananas are really plantains.

Bridget and Joe discuss their experiences at devopsdays Cuba and share audio from the closing session.

Dramatis Personae

• Rudy Gevaert @rgevaert
• Patrick Debois @patrickdebois
• Bernard Grymonpon @wonko_be
• Mike Rosado @MikeRosTX
• Enrique Carbonell Muela @kikicarbonell
• Genry Leyva González @genrylg
• Marialina Ballesteros Hernández @MarialinaBall

Further Viewing

A few videos of devopsdays Cuba have made it to youtube. They can be found [here.] (https://www.youtube.com/watch?v=bTMwHcjpMf4&list=PLobspijdw3822IEFotAHz_vWrc0nTqwxn)

Special thanks to Mike Rosado for his help in preparing the translations for this episode.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Open CFPs

• lots of DevOpsDays
• ChefConf 2017 - closing on January 18, 2017
• Monitorama - May 22-24 - until Feb 1

The Cold Open Supercut

The episode starts with a supercut of the year's cold opens, which Matty describes as the lazy route: in the first year-end show Matty tried to make a highlight reel and had to listen to every episode, but now "you only have to listen to the first 5 seconds of our show." Matty also says the cold opens only started this year, and that Joe, the new audio editor, picks them and keeps surprising Matty.

Why the Audio Isn't Better

Listeners keep asking for better audio quality, and the hosts agree Hangouts is a poor choice but the "lowest common denominator": nobody has ever said they couldn't use it. They've tried other tools at least half a dozen times, including TriCast and Zencastr, and Trevor says that "every time we actually go to do it, something goes catastrophically wrong," usually with three hosts and a panel of guests. Matty says the show is unusual, since most podcasts have one host or a host pair in a room, and a panel show with few repeat panelists is "0.00001% of podcasts out there," so the software isn't built for it. Bridget's takeaway for anyone thinking of starting a podcast: "software is terrible." Matty also notes that two-person episodes are the easiest to edit, and that pre-recorded sponsor pre-rolls replaced the hosts reading them live.

This was also the year of cross-podcast episodes released on other shows' feeds, with the Goat Farm and Software Defined Talk, and a DevOpsDays Dallas super-episode with Software Defined Talk and Food Fight. Matty floated a roundtable of podcast hosts talking about how they make their shows, and every host who replied said nobody would care.

Favorite Episodes

Bridget's favorite is the first episode of 2016, on platforms, which set a bar the year managed to live up to. Trevor picks the DevOpsDays Dallas episode and one recorded in Singapore, where people "are getting all the concepts that we're talking about now, but they didn't have to go through all the pain." Bridget compares it to skipping landlines and going straight to cellular.

Matty's picks:

• Personal brand: the episode that reached friends and family outside the usual audience.
• Who Owns Your Availability: recorded within hours of the left-pad incident after Bridget said it had to happen that day, with a guest who was about to board a plane to Japan and was told to do it now.
• CareerOps: an episode that came entirely from a listener's idea about disaster recovery for your career, which Matty uses to repeat that guests should arrive with a topic. "we have a long list of people and a short list of topics."
• Operationalizing open source: an experiment where Michael Hedgepeth hosted and the hosts were panelists, after a proposal that ran ten paragraphs.

Matty adds that the show gets "one episode a year to be self-indulgent," and this is it.

The Numbers

On the numbers, Matty notes that the website had about 16,000 unique visitors, almost half of traffic comes from search, and Twitter accounts for 6 percent. The number one search term is Arrested DevOps. 1 percent of traffic came from being listed in the Hugo gallery, and another podcast turned out to be using the show's Hugo theme, which Bridget discovered through a Google alert on a name. Matty then wrote a Hugo theme for podcasters. Listens were 232,200 against 204,001 in 2015 with about the same number of episodes, so Matty reads audio popularity as roughly flat. The most listened-to episode was Application Configuration with Adam Jacob and Tim Gross. The most watched video was Bridget's fireside chat with Bryan Cantrill, at almost 2,000 views, then the containers and security episode with Jess and Ben Hughes at about 1,200, then the Jeffrey Snover episode recorded in 2014. Matty credits big personalities and topics people are hungry for, and that if every episode were a chat with a personality the show wouldn't have lasted, and without any it would have burned out.

Matty also tells the story of a blog post that put a personal site on the front page of Reddit, and after Bridget asks, agrees to take down a post a commenter had asked to have removed.

What Each of Them Did in 2016

Matty traveled little, lost status on everything, spoke twice (Pink16 and a CloudBees Jenkins conference in Chicago), attended one DevOpsDays, Chicago, which Matty ran, and got married. Trevor spent months leading a data center transformation in Asia-Pacific, saw Hong Kong, Singapore and Tokyo, and says staying on Singapore hours "broke me in some way," which is when Trevor understood burnout. Trevor spoke at the PowerShell Summit in Singapore and at ChefConf, moved to Los Angeles and joined Chef in the same role Matty holds. Trevor also recounts applying for an evangelist job at the lowest point of the burnout, and Matty adds that it was mature of a hiring manager not to hire for the wrong role at the wrong time. Bridget's advice is not to compromise on what you'll be happy doing and where you want to live.

Bridget gave about 25 talks, visited DevOpsDays in London, Toronto, New York, Detroit, Havana, Philadelphia, Madison and Sydney while running Minneapolis, and with Joe covered five continents. The plan had been to travel less, which became "the gamification of poor life choices" once the airline status arrived. Matty says the remote-work life has made travel something to dread, and that roles built on being in the room with customers make it hard to avoid. Trevor calls elite status "the worst wonderful thing in the world."

DevOps in 2016 and DevOpsDays

Matty's read of the year: "this was a year when we stopped talking about doing shit and we just started doing shit." After the empathy talks of 2014 and the thinking of 2015, people in the open spaces at DevOpsDays Chicago were talking as peers who were all doing it, instead of sitting at the feet of thought leaders.

Bridget says DevOpsDays grew from about 22 cities in 2015 to 42 cities on 6 continents in 2016, with first-time events in places like Istanbul, Porto Alegre, Raleigh, Kansas City, Philadelphia and Cape Town, and Moscow, Beijing and Zurich planned for 2017. Matty joined the core team as web team lead, doing the work behind the scenes of the site so that updating it is easier, while deliberately keeping the Git requirement: "we want to make it delightful but not easy." Bridget has notes on the mockup colors, asking only for better contrast.

Matt, Trevor, and Bridget chat (at length) about podcasts, podcast recording, and podcast recording software. Oh, and the highlights of 2016 if they get around to it. (Don't miss the supercut of all 2016's cold opens, which was edited by Joe, even though Matt takes credit for it!)

What were some of your favorite episodes?

• Bridget: So many great ones! Kicked off the year with Andrew Clay Shafer and Kelsey Hightower, and we kept going at that pace!
• Trevor: Soon-to-be-finished Singapore PowerShell Summit Episode
• Matt: Personal Brand episode, Who Owns Your Availability (Bridget says “omg, left pad, we need to have this episode NOW”), CareerOps

Let’s talk numbers

Website

• 16K visitors to the website
• 46% of traffic comes from search
• 16% comes from referrals, mostly Twitter (Twitter is 6% of all traffic)
• 1% of our traffic comes from the hugo site itself (our site generator)

Episodes

• 232,200 listens in 2016 (204,001 listens in 2015)
• Most listened-to episode in 2016 was Application Configuration with Adam Jacob and Tim Gross
• Most watched YouTube video in 2016 was Bridget’s Fireside Chat with Bryan Cantrill

Website updates

• Didn’t change much, but we list episode numbers now (maybe we’ll add dates sometime)

Other improvements

• Matt did a session with Daniel J. Lewis of the Audacity to Podcast (we were a featured podcast eval in Podcasters Society) and we got a whole bunch of ideas on how to improve the show - some of which we have already started to implement). You can check out our outstanding issues on GitHub
• Welcome to Joe as our main audio editor!

What happened with you in 2016?

Matt

• Didn’t travel much - only spoke once twice - once at Pink16, and once at a Cloudbees conference in Chicago
• Only went to one devopsdays - Chicago. I’m failing!
• Hey, I got married

Bridget

• 28 talks. Give or take. A few of those were at devopsdays - I made it to London, Toronto, New York, Detroit, Havana, Philadelphia, Madison, Sydney - and of course ran Minneapolis.
• Counting North America Joe and I hit 5 continents this year. And I was going to travel less. Instead, all the airline status. Rethinking next year. Maybe get into webinars?
• Devopsdays: growing! New core team! New cities!

Trevor

• I spoke 3 times this year, but got to attend so many more conferences than before
• I was literally on the other side of the planet to my usual place of existence for the first time.
• I moved to LA
• I work for Chef now!

Check Outs

Bridget

• Systems We Love - videos at https://twitter.com/SystemsWeLove/status/809117528374972416

Trevor:

• [Westworld](http://www.startrek.com/article/klingon-mug-and-collector-lapel-pins-ready-to-beam-up Klingon Blood Wine Mug)

Matt:

• There’s this thing called Minecraft I “discovered” via my kids.
• Also reminder about code.org and everyone is talking about Hour of Code
• It’s been super retro time for Matt as he re-read a bunch of the Dragonlance books, and discovered a bunch he hadn’t read, including The Soulforge.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Open CFPs

• lots of DevOpsDays
• Velocity San Jose until Jan 10th
• ChefConf 2017 - closing on January 18, 2017
• Monitorama - May 22-24 - until Feb 1

photo credit 1, photo credit 2

Where Sydney's DevOps Community Came From

Lindsay and Mick started the first ever DevOps meetup, in February 2010, and Matthew says Sydney's DevOpsDays in 2010 was the second one outside Ghent, making it the longest running. Mick says the reason was that at the end of 2009 Mick worked at a startup that missed its next round of funding, and everyone was fired one morning. With no name in the industry and no community events behind them, Mick did the hard slog of job hunting and decided no one should have to. So the meetup would be inclusive, not just for practitioners but recruiters and anyone else, and helping people find jobs is the only reward Mick needs. Lindsay says they're actively inclusive of recruiters, on the condition they don't cold-call the members, and sees them as allies.

Katie says the event has skipped beer at the first-day activities, with bowling instead, for the past two years, which helped attendance the next morning. Lindsay gave an opening talk at the 2013 Sydney event, and says this year's attendance was spectacular by comparison.

A Conference Where Everyone Participates

Matthew likes that half of DevOpsDays is open spaces, shaped by attendees and not organizers. Lindsay says there are no silent witnesses, and people who participate most enjoy it most. Matt Ray, as a vendor and former organizer, says a table isn't the point, since you have to show up and talk. Lindsay says at corporate events the contrast is stark, and that at DevOpsDays a vendor person is just an actual person.

The challenge is selling open spaces in Australia to people who haven't been. Mick passed along feedback from someone asking "why am I giving you money to come to a conference that has no speakers." Bridget says about 85% of any room has never been to one, so the stock template now says attendee-suggested breakout sessions instead of open. Katie calls it "a supercharged hallway track and some talks," and says the conference emphasizes that talks are recorded so you don't have to attend them all, and people travel to meet people. Mick heard a non-technical attendee say they felt very involved.

Topics: Lambda, Business, People

Lindsay's morning talk was about building teams as complex distributed systems made of humans. Lindsay came for the technology in Ghent in 2009, when people wanted agile systems administration, and found that "none of the technology problems are the hard ones in our industry. It's actually people." Matt Ray says the talks break into technology like containers, and how to do DevOps with people in the enterprise, like getting your boss to listen. Lindsay says functions as a service is a fundamental game changer, and expects the talks in two or three years to be about different technology. Matt Ray says last year and the year before were about containers, and "The people problems never go away." Matthew notes recurring topics of DevOps and the rest of the business, including an Ignite about finance and DevOps, and security. Katie says language communities are picking up on culture, hiring, diversity and inclusion, and that a linguist gave the closing keynote, and adds that organizer burnout is real, which is part of why Katie volunteers.

Australia and the US

Matt Ray doesn't sense the same fear of competition in Australia as in the States, where Amazon and Walmart are torching retail. Lindsay pitches changing technical practices to government by saying you want to go faster to deliver value and meet user needs, and that "going fast and being safe, they are not mutually exclusive." The Puppet survey stats show fast teams recover from failure faster, and Matt Ray calls it compliance at velocity. Lindsay says the government's Digital Transformation Agency supports hundreds of apps with a team of two. Matthew says Australians are open at meetups, but US and European companies moving in are bringing an NDA culture, which Matthew is mindful of as an organizer.

Where DevOps Is Going

Mick says more effort is going into what developers deliver and less into the operating system: "The OS layer is really more a utility these days," like a tap you turn on without wondering where the water comes from. Katie, who is newer to ops, says there is still gap filling and legacy work, since not everyone is running containerless Lambda functions. Lindsay says operations will consolidate, with fewer of those jobs, so learn at least one other programming language, since you can help developers who now can get things running easily. Matt Ray adds that there are more places things can go wrong, so operability wisdom still matters, and that "there's always gonna be servers somewhere." Matthew says the future is about people, and adds that everyone talks about business value but should remember the customer who pays you. Bridget ties it together with the electricity generated on a river in Appleton, Wisconsin, and says you decide what to abstract away, and which to pay someone else to do.

Bridget and special guest host Matt Ray of Software Defined Talk chat with Matthew Jones, Lindsay Holmwood, Mick Pollard, and Katie McLaughlin at devopsdays Sydney 2016.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Open CFPs

• DevOpsDays Baltimore - closing on Dec 12, 2016
• ChefConf 2017 - closing on January 18, 2017
• Velocity San Jose until Jan 10th
• Monitorama - May 22-24 - CFP opening soon

The CAP Theorem, Reconsidered

Julia has been trying to understand distributed systems in a way that maps to a real system. Stripe cares about both consistency and availability, though some systems can be a little less consistent than others. They say it took a long time to get the CAP theorem even with a math degree. Strong consistency means linearizable, meaning all the actions in the system can be put on a line, which is a very strong property, and availability means people can use the database. Because networks partition, you can't have both at once. Bridget cites Katie McCaffrey's point that you don't get to choose to skip partitions, and Julia adds garbage collection pauses as another way a system goes quiet. Bridget mentions the Juliet pause Bridget saw running HBase, where a system decides the other side is dead and kills itself.

At Strange Loop, Julia told Martin Kleppmann they were confused about the CAP theorem, and Kleppmann said not to pay attention to it, since there are more interesting things to say. Julia read Kleppmann's paper, A Critique of the CAP Theorem, the night before. It proposes thinking about what happens to reads and writes when the network is slow: a linearizable system has slow reads and writes, a weaker causal consistency has fast reads and writes, and some intermediate models have slow writes and fast reads. That is a richer model than CAP's two scenarios but not a complicated one, and it describes a replicated database that writes to a primary and reads from a secondary, which CAP doesn't help you reason about.

Discomfort as a Guide

Julia mostly doesn't read papers. They go to Papers We Love in Montreal, listen, and ask dumb questions, and they read a paper when someone hands over a printed copy. What motivates them is a feeling of not understanding: "I got all the words, but I still don't really understand what's happening." They spend time being uncomfortable about how well they understand something and asking why, and with CAP the answer may be that the theorem isn't the answer to their questions, not that they're missing something. Bridget says people in tech are drawn to things that sound like answers.

Service Discovery at Stripe

Julia is on the team that owns service discovery, didn't set it up, and wrote the blog post after giving an internal talk to solidify their own understanding. If you lose instances, you don't need service discovery, since load balancers do health checks. It matters for registering new nodes. Before, Stripe used Puppet to write a configuration file on an HAProxy load balancer listing the nodes, which was slow and toilsome. Consul runs an agent on each host that reports what it is running to the Consul servers, which hold a database you can query.

The interesting part is consistency. Consul is strongly consistent, so it sometimes says it is having a leader election and gives no servers, and "Consul was like too consistent." For service discovery, "a lot of the time you don't really care if your results are exactly right. You just want to be mostly right." So Stripe uses Consul Template to generate an HAProxy configuration file every minute, which stays in place if Consul goes away, so the worst case is a slightly old list of servers. HAProxy does a graceful reload, forking so the old process handles old connections. Health checking happens in the load balancer, separately from Consul.

Blogging, Asking Questions, and Working Remotely

Julia started blogging at the Recurse Center three years earlier, writing a post every day about what they learned, joking it was a media strategy to get a job, and it worked. Their rule was that it doesn't have to be perfect. Bridget appreciates that they say what they don't know, and Julia says it matters at work too: when they joined the team they didn't know how the service discovery cluster worked, and now they do and can work on it responsibly.

Most of Julia's team, and most of the teams they've worked on for almost three years, is remote. They are aggressive about asking questions. When they joined a data infrastructure team, on the plane back from San Francisco they interrogated people about every noun they didn't understand, from HBase to Spark to YARN. They use Slack, sometimes schedule a Hangout, and visit San Francisco, and once gave a colleague who set up the cluster a list of questions. They think it helps the person who built it to hand a system off, since it's bad to be in charge of something forever, and Bridget says being territorial won't get you promoted.

Zines and Comics

Julia describes a zine as a tiny magazine about something you love. In 2014 they were giving a PyCon talk about Linux debugging tools, and people asked what to read afterward, and the links didn't get read. Inspired by a movie about riot grrrl and fanzines, they wrote a zine about strace to hand out at the talk. Bridget saw copies at DevOpsDays New York. They made a second zine in September about Linux debugging tools.

For comics, someone suggested a cute drawing at the top of the service discovery post, and Julia drew it on a six-hour flight. The person said the post was much easier to understand from the drawing, which works as a summary tool. They also drew the eight steps for setting up a new web service at Stripe. On Twitter, images communicate more, and a comic on /proc took off. The inodes comic came from running out of them once. Every inode lives in a flat numbered array on disk, and Bridget notes it's possible to run out. Julia says everyone has a day when they learn that, and it's better if it's when they read an adorable comic than while troubleshooting.

Containers and Hype

Julia's interest in containers is about making infrastructure easier for Stripe developers, and they find the Kubernetes hype frustrating. The appeal is a uniform infrastructure where every box is configured the same and all the special snowflake configuration lives in the container, so that setting up a new service is less work. Containers have been very successful on the desktop as a developer tool, but people conflate that with production, where it is less clear. Utilization makes sense as a business reason, though 99% is too high, and going from 20% to 80% would be reasonable. Bridget asks what business problem you're solving.

How to Stay Excited About Learning

Julia says they became more excited about learning at the Recurse Center, with 12 weeks to learn whatever they wanted, and got stuck learning things all the time. Bridget notes their employer lets them stretch, unlike places that just want you to ship. Julia says asking questions is a skill, and that "I don't understand" is not a very good question. They try to understand it a bit alone, then describe how they think it works to someone and ask them to check their understanding, and they wrote a blog post on asking questions. They also have an unshakable confidence that they can figure things out, and think "I haven't taken the time to learn it" and not "this is too hard." Learning one thing at a time adds up, and "no one has ever anointed me the expert of anything." Their comfort limit is the Linux kernel code.

Bridget takes the opportunity to talk about CFPs, and Julia says the first conference talk they gave was at PyCon Canada, after the person who runs Montreal Python told them to submit, and they said someone had already talked on the topic. The organizer said it didn't matter, and both talks turned out to be great.

Distributed systems, service discovery, load balancing: Service Discovery at Stripe

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off.

• DevOpsDays Berlin Nov 16, 2016 - Nov 17, 2016
• DevOpsDays Brazil Nov 18, 2016
• DevOpsDays Warsaw Nov 22, 2016 - Nov 23, 2016
• DevOpsDays Paris Nov 28, 2016
• DevOpsDays Sydney Dec 1, 2016 - Dec 2, 2016

Open CFPs

• DevOpsDays Baltimore - closing on Dec 12, 2016
• ChefConf 2017 - closing on January 18, 2017
• Velocity San Jose until Jan 10th
• Monitorama - May 22-24 - CFP opening soon

Check Outs

Julia

• A critique of the CAP theorem: https://arxiv.org/pdf/1509.05393v2.pdf

Bridget

• Catchafire - matching volunteers with skills to orgs that can use their skills
• Minnesota Literacy Council - look for your local literacy org to teach English, math, civics, and more to immigrants & refugees

From the F-117 to Ten Business Units

Pauly's background is varied. Pauly worked on the F-117 stealth fighter in the Air Force, then moved into telecom and satellites, did development, got roped into being a network engineer because the company didn't have one, and then moved back to development and operations. At Silverpop in Atlanta, Pauly started digging into DevOps, after some cultural and tooling transformations, and found Chef, after starting with a few other tools, some homegrown. Pauly then bumped into a Chef employee at a bar in Seattle who said they were looking for an ops leader, and moved the family to Seattle, where Pauly spent about three and a half years as VP of Operations at Chef, talking to customers at different maturity levels. Hearst Business Media has 10 business units going through transformations to agile, lean and DevOps, and they hired Pauly as VP of DevOps. Pauly tried to talk them out of the title. Pauly's job for the last two years has been building a DevOps community across ten islands that interact mainly at the senior leadership level.

Unique Snowflakes

Matty says that's common, and relates a story from a Chef customer's private community where someone asked why a person from another business unit was telling them something. Pauly says the first hurdle was building trust with each unit, since a VP title can be menacing, and the second was the belief that DevOps works everywhere else but not here because "we're a unique snowflake." Pauly's response was "you're incredibly unique, just like everybody else": where humans are involved there is a limited set of emotions, and those shape culture. Drawing a ten-circle Venn diagram showed more overlap than they expected, in language and tools if not process. The magic, Pauly says, was seeing people help each other who hadn't known each other a week earlier, including during outages, for no reward beyond paying it forward.

Matty says it's more facilitating a community than building one, since a community forms out of humans and you can stymie it but not force it. Matty shares the opposite: at a previous employer, a bank, Matty and later colleagues at Apartments.com found they had worked on the same product one floor apart, dev and ops, without ever meeting. Matty also quotes Sascha Bates that every snowflake has six sides.

Don't Make a Framework Into Dogma

Pauly agrees with Matty's "don't take a framework and make it dogma," and admits to originally going in looking for economies of scale, which was the wrong approach, so the choice of tools came off the table, since business units have the context. One level up, Pauly looked at common SDLC approaches, without change for change's sake: work that is cyclical, with a fixed start and end each year, may not need agile at all. Big bang plans like everyone being 100% agile and in the cloud by August were dropped, since it's like throwing someone in an icy pond instead of teaching them to swim. Instead they changed processes iteratively, with DevOps principles underneath, like turning up the water on the frog. Facilitation is guiding, not dictating, and some people won't come along.

Make the Easy Path the Right One

Matty tells of a sysadmin asking how to make developers follow a process, and Matty's answer was to make the right way the easiest way, which the book Switch validated. Change annoys people because learning a process isn't their job. Pauly says ops people are so mired in daily brushfires they don't do fire prevention, so you show the investment makes their life easier, and they join organically. Matty says a cultural change has users, and the UX of that change matters, so put yourself in the ops person's shoes, and that "Everything's a product."

Pauly's principle is to work with the downstream work center in mind, since every customer in the value stream matters, and if work has to be sent back that is waste, measured as percent complete and accurate. Matty says the easiest waste to hammer home is people's time, and that, as Damon said, we should just call it common sense.

Value Stream Mapping

Pauly says the value stream mapping workshops turned out to be the most powerful tool, because everyone is in the room at once. Pauly adds that most of the time what someone cares about isn't the root cause, so you dig as with five whys, since "fixing the output doesn't fix the cause": Pauly compares it to giving a cough drop to someone diagnosed with lung cancer. Pauly's example is a color the customer wanted blue that was never specified, fixed each time, while the feedback loop from product never improves. They use Little's Law for efficiency ratings, and numbers on lead time, process time and percent complete and accurate, which confirm what people already feel. Matty says decisions from the gut are reactive, like the screaming customer versus the quiet one buying less, and that data doesn't have to be numbers, and Pauly says use your gut, but don't fly at night with no lights, or stare only at instruments.

Pauly says every workshop has one person with folded arms who rolls their eyes, and "that person should have representation." Matty recalls a proof of concept where a skeptic turned out to be the biggest champion, telling stakeholders to go buy it. Pauly says being a facilitator means letting that person be heard, and that they may either join or opt out, and leaving is a fine outcome. Pauly also cautions against forcing the top engineer, the Brent, through a transformation, which hurts the organization and the person. Matty recalls Adam Jacob saying most people want to do good work and haven't had the opportunity for a long time, that some people just don't want to be on the bleeding edge, and that change agents have to think like salespeople and understand drivers, as in Adam Jacob's talk on the five love languages of DevOps and the Bill Joy episode.

Conferences and Speaking

A listener, Michael Lombardi, asks whether companies should pay employees to attend and speak at conferences. Pauly says employees should get paid T&E and not take PTO, with common sense, and got the business unit leaders to go to the DevOps Enterprise Summit, which was an aha moment for them. Pauly has stopped going to purely technical conferences where Pauly doesn't gain anything. Pauly worries paying people to speak could push an agenda, and describes a DevOps conference in New York that was a day of sales pitches. Matty says DevOpsDays are strict about vendor pitches, and tells of a manager who offered a cash bounty per talk, which only rewarded people who were already speaking. Pauly says pushing someone terrified of speaking hurts quality, and a blog post is another medium, and Matty adds a podcast. Matty warns that an engineer who speaks constantly becomes a better speaker and a less awesome engineer, unlike a full-time evangelist. Both suggest bringing back an internal blog post or brown bag, and rotating who goes so no one becomes a conference pariah.

Staying an Engineer

Matty loves seeing a big-muckety-muck VP in a customer's Slack talking about attribute precedence. Pauly loves being an engineer, and says that if the role doesn't allow building, Pauly will create an open source project. Pauly has to speak engineers' language, and when visiting a business unit and offered an office, Pauly sits in the queue with everyone else, because engineers can put on a show for a day but then start complaining across the walls, and that unvarnished view is the real world. Pauly works with CTOs, CEOs of business units and new engineers, and says the exposure is incredible.

Matty admits to over-engineering a Go application for the DevOpsDays website partly to keep chops up. Pauly says it's less about street cred, since developers will sniff out bluffing quickly, and more like speaking Spanish: saying hola gets you a torrent of Spanish, so you have to understand how it works. Pauly tells of building an elaborate Lego train and Arduino plan to wean a son off a sound machine when Pauly's wife suggested turning it down a little each night, and the domain name was already bought. Pauly points to Adam Jacob, who never stopped loving technology, as a guide.

Pauly's checkouts are the DevOps Enterprise Summit talk, which will be delivered as a story about middle managers in a transformation, and DevOpsDays Kansas City, and Pauly is starting a blog called DevOps Therapist next year.

• Switch: How to Change Things When Change Is Hard
• Little’s Law
• Dan Pink: The puzzle of motivation (aka "the Drive TED talk")
• The Five Love Languages of DevOps
• “As a great conductor, I don’t know how to play every instrument. But I know how they should sound” - Pauly
• Pauly is launching a new blog in 2017 at http://www.devopstherapist.com/

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off.

• DevOpsDays Cape Town Nov 7, 2016 - Nov 8, 2016
• DevOpsDays Nashville Nov 10, 2016 - Nov 11, 2016
• DevOpsDays Berlin Nov 16, 2016 - Nov 17, 2016
• DevOpsDays Brazil Nov 18, 2016
• DevOpsDays Warsaw Nov 22, 2016 - Nov 23, 2016
• DevOpsDays Paris Nov 28, 2016
• DevOpsDays Sydney Dec 1, 2016 - Dec 2, 2016

Open CFPs

• DevOpsDays Baltimore - closing on Dec 12, 2016
• ChefConf 2017 - closing on January 18, 2017

Check Outs

Pauly

• Pauly is speaking 9:30a.m. Wednesday the 9th at DevOps Enterprise Summit 2016 or check out the live stream!
• Pauly's talk at DevOpsDays KC

Matt

• MarkdownToPDF
• Errors Azure Throws
• Chicago Cubs win the World Series!

A Small World

All four have crossed paths before. Matty traces how Classified Ventures, the company behind cars.com and apartments.com, is connected to Grubhub through people who left HomeFinder. Jeff recalls meeting Grubhub's CEO at a Twitter meetup around 2007 and turning down the startup for a safe job at Accenture. Jeff adds that Chicago feels like a really tight community where you keep running into the same people, so it doesn't pay to burn a bridge on the way out.

Sameer moved from Boston and New York, including a stint at Citi, and found a work culture that felt like a family. On starting at kCura, Sameer looked through the Outlook directory and recognized Matty and Trevor from the podcast. Jerry grew up in Miami, worked for FedEx swapping out computers for Y2K in tropical places, and spent four or five years in the Bay Area, through the crash. When Jerry's wife got an opportunity in Chicago around 2003, all that could be found online was banks, until Orbitz stood out. Jerry notes the meetup gets hosts, sponsors and speakers whenever it wants. Jeff says that in upstate New York there was information hoarding, and in Chicago people were humble, cognizant of imposter syndrome and willing to mentor.

Midwest Pragmatism

Matty says the Midwest adopts things after they hit the West Coast and then the East Coast banks, and that Chicago has very traditional IT and a lot of startups with little in between. Jeff says it comes from Midwest practicality: the region isn't interested in the brand-new shiny, it's about "solving actual problems. And generating revenue." Businesses here have straightforward models, like delivering food and taking a slice, and Jeff tells the team to ask, "How does this help us deliver food better?" Jerry counters that innovation still comes out of Chicago: Graphite was developed at Orbitz, and Gogo had just presented Foremast at the meetup, a tool to automate Spinnaker pipelines. Matty calls it pragmatic innovation, like Etsy's, solving a real itch. Sameer adds that this leads to an iterative approach, where you can share work in progress and get "this is great progress" and not "you've got a long way to go." Jerry says the solutions from Netflix and Google look cool but aren't the scale of problem Jerry deals with.

Jeff reminds the audience that big, slow institutions are still bringing in revenue, and that people from them at DevOpsDays feel left behind. Jeff's message is that you may not be able to move as fast as others, but that doesn't mean you have to sit still.

How the Conversation Has Changed

Matty says that two years ago at DevOpsDays Chicago the story from traditional companies was "this will never work here," and now it's "this is really hard and we're not moving as fast as I wish we were," which is a sign you can do something about it. Jerry says the meetup, held downtown at 5:30, still misses suburban companies, though they post videos.

Jeff says DevOps doesn't have to be a top-down transformation: pair with the developer you work with most for half an hour a day, or have lunch with the team for the service you support. It spreads, and exposure between teams reveals "learned helplessness" on both sides, like developers who don't have access to log aggregation or ops people who restart the same service every night. Jeff tells of visiting the NOC, hearing a PagerDuty alert for a Splunk log velocity threshold, and turning it off on the spot, and getting donuts. Sameer says it's also middle-out, with a central team establishing practices and engineers deciding on their own to speed up deployments, and describes an ideal of engineers deploying to production with supporting people embedded on the team.

Matty says wherever you work, part of your organization is already doing this. Matty adds that a mandate from a Fortune 10 CIO won't magically create DevOps, and that managing sysadmins means being the person who says "I know you can think of 100 reasons why this won't work. For a minute, let's just pretend it will." Sameer spent five years implementing config management at Citi, sometimes tracking changes quietly when told to just get it done. Matty tells the story of learning Chef out of spite when told it would take a year and tens of thousands of dollars in training, and says that's not recommended.

Chicago Companies Doing Interesting Things

Jerry names Gogo, Hyatt, which is moving website work to Docker, Braintree, DRW, which is doing a lot of bare metal DevOps, and ThoughtWorks. Jeff likes the way Hyatt is building a startup-like DevOps culture inside a big company, and Jerry says US Foods, around for 100 years, began with a single new application. Matty warns you have to avoid "bimodal" as an end state, and points to Target's dojo, which starts small but is for everyone. Jeff treats bimodal as a transitional state, and says that a pilot makes other managers say they want that. Matty says pilot it, stack the deck, and beware that people want the magic button, which was the point missed in the 10 deploys a day talk. Matty adds McDonald's, Walgreens, Hyatt and the Mercantile Exchange, which need velocity with safety. Sameer adds that Morningstar rotates developers through ops teams, and Jerry adds 1871, the incubator. Jeff says the startup challenges Jeff took part in felt very Midwest: less about technology, more about the minimum viable thing.

Matty ends by noting that Chicago's DevOps meetup has the best URL, meetup.com/devops, which to Matty means Chicago invented DevOps.

• ADO episode at devopsdays Chicago 2014 with J. Paul Reed
• Jerry started his career swapping out computers for Y2K in tropical locations. Nice job to have!
• Most sportsball talk on any episode of ADO to date!
• “You may not be able to move as fast as others, but that doesn’t mean you have to sit still” - Jeff
• Conversations Matt has had with local folks in the past two years have changed from “this will never work at my company” to “ this is really hard, and we aren’t moving as fast as I would like us to be”
• 1871 - Chicago incubator
• Chicago DevOps Meetup
• 2016 Chicago Tech Madness Bracket

Companies doing cool devops in chicago

• Kcura
• Grubhub
• Gogo
• Hyatt
• Braintree
• DRW
• ThoughtWorks
• US Foods
• McDonalds
• Walgreens
• CME Group

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off.

• DevOpsDays Ohio Oct 31, 2016 - Nov 1, 2016
• DevOpsDays Madison Nov 2, 2016 - Nov 3, 2016
• DevOpsDays Bangalore Nov 4, 2016 - Nov 5, 2016
• DevOpsDays Cape Town Nov 7, 2016 - Nov 8, 2016
• DevOpsDays Nashville Nov 10, 2016 - Nov 11, 2016

Open CFPs

• A lot of devopsdays CFPs closing soon - see devopsdays.org/speaking

Check Outs

Jeff

• Neptune.io
• Netflix’s Winston
• Chicago DevOps Book Club

Jerry

• DevOpsDays Chicago videos
• Spantree Vimeo - a Meetup recording company that also does tech consulting

Sameer

• Edmund Lau’s Effective Engineer a great, fast read that’s made big differences in day to day life (reminds me of Time Management for System Adminstrators)
• Can’t wait for Halloween and the finale of the web series Edgar Allan Poe's Murder Mystery Dinner Party
• OMG I’m loving http://armviz.io

Matt

• Storm Trooper Whiskey Decanter
• Hamilton in Chicago!
• Westworld on HBO

Back in the day, James also wrote a book called Pro Nagios 2.0.

Three stages of monitoring maturity:

• Manual, user-initiated, or no monitoring (aka Bridget’s example of “we know things are broken because the customer calls us to complain”)
• Reactive
• Proactive

"You will eventually get to CPU, memory, and disk, but a lot later after you start with the things you should really care about" - James

"Too much monitoring is binary - this thing either works or it doesn’t" - James

Also mentioned:

• Large-scale cluster management at Google with Borg
• RFC 1149 IP Datagrams on Avian Carriers - IETF

Community & Event Stuff

Where we’ll be for the upcoming fortnight

• Matt is getting married at the Jim Beam distillery on Saturday
• Bridget will miss the bourbon wedding as she’s heading to Joe’s family reunion followed by GOTO Copenhagen.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off. Also now works on O'Reilly Security conference.

For more DevOps awesomeness, check out the Chef Community Summit, October 26th and 27th in Seattle, WA. This Open Space event provides a great opportunity to connect with the DevOps Community and Chef Engineers over two days of engaging sessions and hallway discussions. Bring your ideas, passion and excitement for Chef and DevOps to this highly interactive event. Go to summit.chef.io to register for this awesome event and use the code ARRESTEDDEVOPS to get 10% off your ticket!

Open CFPs

• A lot of devopsdays CFPs closing soon - see devopsdays.org/speaking
• OSCON's CFP closes Oct 25

Check Outs

James

• Skyliner.io
• Jason Dixon
• Vote, but not for Trump

Bridget

• Cloud Foundry Summit is going on in Frankfurt right now
• honeycomb - explorable operations metrics from Charity Majors
• MicroBadger - for docker image inspection Liz Rice & Anne Curry at Microscaling Systems

Trevor

• Windows Server 2016 is GA!

Matt

• InSpec has shipped 1.0! You can check it out at http://inspec.io/ InSpec is compliance as code – a human-readable language for automating the continuous testing and compliance auditing of your entire infrastructure. You can also use it to verify if your servers and applications are configured correctly.
• Self-promotion: working on a shareable theme using hugo for podcasts. Check it out at github.com/mattstratton/castanet

Building a First DevOpsDays

Annie says they had a little over 300 people, with everything going off without a hitch, and they wanted to be Texan but different from Austin, which has a huge DevOpsDays. Annie adds that many attendees were new to DevOps, and there were people around to mentor them. Michael says the reward is realizing the months of work helped people change their lives, and when Nathen asked on day one who was at their first conference, people stood up. The show-of-hands exercise, repeated for the recording, gives about 20 of 100 who had been to a DevOpsDays before.

Michael tells how the organizing came about: a local group called DevOps Live decided on a DevOpsDays, and Michael had read an old email from Doug Ireton about a conference open space where a bullet point said DevOpsDays needs more women organizers. Annie was looking at a career transition, and Michael suggested organizing, since Annie "can organize a conference." Annie became the sponsor liaison and raised money from about 25 sponsors. Michael watched the kids on Wednesday nights during Annie's sponsor meetings, and three weeks in got a text asking if Michael wanted to handle speakers and emcee.

One Track, Many Backgrounds

Nathen was adamant about a single track. When you have choices, you go where you're comfortable, and Nathen heard people say they never would have gone to a networking automation talk, but they got a lot out of it. Michael, who had only been to Austin's multi-track event, mixed technical and culture talks on purpose and picked speakers from networking, security and development backgrounds. It's about empathy and common ground for faster delivery, and Nathen adds: "Imagine a DevOps conference where there was a dev track and an ops track." Dillon, who normally focuses on Microsoft subjects, found the exposure to different speakers surprising and delightful.

A Local Community Event

Nathen says the first DevOpsDays in a city is special because the community has two days to gel, and 90 to 95% of the audience lived within 40 miles. Coté says a lot of business and IT happens in North Texas and few people know it, and likes talking to "normal people doing normal things," since heroes are heroic and that gets boring. Trevor says the weekend brought a first: Dillon recognized Trevor from the podcast, which recalled the experience of meeting people Trevor admired, and it was the first conference where Trevor felt past imposter syndrome and unafraid to share thoughts.

Audience members share their reactions. Reuben Garrett, a Unix sysadmin at IBM SoftLayer, attending a first professional conference, wanted more technical talks but was glad for JJ's Ignite on introverts at conferences and a talk about empowering women in tech. Jean Bennett had been to many conferences but never a DevOpsDays, and the single track and open space worked well even though Jean isn't very technical. Jean is "a talkative introvert." Clay Schroeder of CBRE brought nine people, which would have been hard in Austin with travel and hotels. Michael says at NCR only a small group gets sent to conferences on planes, and this one had no hotel and an inexpensive ticket. Annie says leftover sponsor money can go back into DevOps Live to keep the community going.

Local Celebrities and First Talks

Michael says the mix of speakers, including Franklin Mosley, whose security talk was a first at any conference, made the event more engaging than an all-heroes lineup, since the authenticity engaged the audience. Trevor says DevOpsDays is a great place to give a first talk, and that Trevor's was at ChefConf after about three weeks of Chef work, which was much more intimidating than a smaller venue would have been. Nathen says "you should build local celebrities within that meetup" instead of flying in people with name recognition.

Confusion, Learning, and Empathy

Coté values meeting people who think differently about obvious things, and says it's fun to sort out the confusion on both sides. Coté recalls introducing Scrum at BMC around 2003 or 2004, when as a snarky young developer Coté figured they were replacing nothing, since nothing existed. Nathen says some attendees ended the first day asking what exactly DevOps is, and the answer on stage was that it's a journey, not something you finish after 18 months. Dillon found the most clarity in open spaces, since people shared similar problems and Dillon heard three or four different valid answers.

Michael and Annie stayed up until midnight after the kids went to bed working on InSpec profiles and pull requests, and Michael realized that "when you know something and you become the expert, you start to forget what it's like to learn it, and you lose empathy for people." Watching Annie struggle with something obvious to Michael made Michael a better change agent. Trevor says teaching others Chef while still green did the same for Trevor. Michael adds "you kind of don't learn empathy until you come across somebody who's disagreeing with you."

Getting Your Boss to Come

Michael asks how to get bosses to attend, and Coté says to figure out their motivations and goals, and either free up their barriers or be persuasive that unless management is involved it won't work at scale. Coté suggests giving them permission to attend, getting them on a panel like a hot dog on a string, sending them recorded talks, and writing up your conference notes as things to do here, not just what happened, and being ready for homework. Nathen suggests inviting managers to speak at a meetup to say what keeps them up at night, and points to books like The Phoenix Project as an audiobook and Mark Schwartz's book on business value, at 98 pages, and the DevOps Enterprise Summit. Annie suggests finding three or four of the boss's pain points and solving one in a DevOps way with a tiny project.

An audience member asks when you reach success with DevOps. Nathen gives the definition that Adam and Nathen use: "DevOps is a cultural and professional movement focused on how we build and operate high-velocity organizations built from the experiences of its practitioners." Nathen isn't sure all of NCR is bought in, but Michael is doing DevOps, and they treat it as planting the flag. Michael cites Toyota Kata: on Monday, set a target two weeks out in the direction of the goal state, like having lunch with someone to ask their pain points and mapping them to what automation could solve, and then create win after win. Nathen ends by saying that when you reach a milestone, stop to market and share it across the organization, and that you'll feel like you're telling the same story again and again, which is fine, since you're telling it to fresh ears.

The Floppy and the Skeuomorph

Bryan has just come from HashiConf, where the guest gave the closing keynote and, for the first time, used a physical prop: a 3.5-inch floppy disk to make a point about hardware virtualization. Bryan explains that the disk is a skeuomorph, like the save icon or the fake wood grain on a station wagon, and says "I do think that VMs are skeuomorph." Your VM has a floppy controller, and these legacy devices have no place in a modern container architecture. Worse, Bryan says, provisioning containers inside virtual machines on hardware wastes resources, "and this can't last forever."

Containers on the Metal

Most containers today run inside VMs, a layer people don't see, and Bridget raises the objection that unprivileged containers may not be production-ready without another security layer around them. Bryan is up front that this is talking one's book: SmartOS and Triton have run containers securely on the metal for over a decade, with zones designed to be completely isolated. Bryan contrasts that design center with Linux, where "containers" are really namespaces and cgroups that cut across the system. The guest draws a parallel to ZFS and DTrace, designed to be production-ready on day zero, and says the difference between a facility at birth and later is the scope of features, not readiness.

Bridget concedes that Linux won, and Bryan refines it: what won is the Linux binary interface. Because that is settled, Joyent implemented a Linux system call table for SmartOS, so a Linux stack can run in a Triton zone and looks like a VM but is a container on the metal. Bryan notes that FreeBSD and Windows have done something similar, including Bash on Windows, a phrase Bryan never expected to hear in a sentence. The guest believes open source has won an unconditional victory, and everything in the container ecosystem, from Kubernetes to Docker Swarm to BOSH, is open. That is why Joyent implemented the Docker Remote API, to have Docker without the Docker engine, and why Bryan wants an API separate from its implementation, so people can compete on the engine.

Forks, Agendas, and Open Thinking

On the rumored Docker fork, Bryan says people shouldn't talk about forking, they should do it or not, and suspects some who talk are goading others. The guest would rather see de novo implementations of a stable Docker Remote API than a fork of the engine. Bryan advises people not to get swept up by vendors with an agenda: the people ginning up a fork have a solution in search of a problem, not a problem to solve. Bridget raises Red Hat's downstream patches, and Bryan says "I am a much stronger believer in incompetence than malice," since there are 20,000 issues on those repositories. Bridget adds that taking a patch is like adopting a free puppy.

Bryan says Joyent open sourced its stack almost two years earlier and found it wasn't enough, because the design discussions were still in hallways and chat rooms: "we actually need to be open sourcing our thinking, not just our code." Their answer was requests for discussion, or RFDs, written in an RFC style, which anyone can search to see the Triton roadmap. Customers read them and react to specific numbers. Bryan thinks design discussion in GitHub issues is an anti-pattern, since an issue gets closed and the valuable discussion is buried.

Foundations

Bridget says the Cloud Foundry Foundation is valuable because employees of competitors pair on the open source project. Bryan, who is involved with the Cloud Native Computing Foundation, says it is still finding its footing and it can't be the Kubernetes Foundation: "If we are the Kubernetes Foundation, then we have failed." Bryan wants the CNCF to have as public a mission as it can, contrasting a nonprofit with a public mission and an industry consortium, and says its constituents should be the people running and contributing to the infrastructure, not the vendors that paid for a seat.

The GitHub Resume and Paying for Open Source

Bridget respects companies that pay employees to write open source, including Joyent. Bryan objects to the GitHub resume because contributions to a fork don't show up on your activity. SmartOS and Illumos Joyent are GitHub forks of Illumos, so Bryan's own history looks empty. Bridget says paying people means they can write open source all day, then go home and not write more when exhausted.

Bryan says we're deep into what Bryan called supply-side open source a decade earlier, where infrastructure comes from companies dedicated to it, and asks how you monetize it. The guest thinks the era of proprietary software will be completely over, using ZFS as an example of a substrate where duplicated effort is over. In Bryan's view, people won't pay for the software, but for running a cloud, a service or metal, and for support: they will pay "for the ability to pick up the phone in the middle of the night and call for an upgrade that has gone sideways." Bridget says Cloud Foundry customers pay for integrations they could build but don't have time to. Bryan adds that if you intend to be the one who picks up the phone, it pushes you to make the software as reliable as possible, and to invest in debuggability and observability.

Production Empathy

Bridget mentions a blog post by a Joyent engineer on a storage disaster, and Bryan says the details matter. Bryan doesn't think every software engineer should carry a pager, since it's like trying to train a dolphin with a shock collar, but does think "everyone needs to develop production empathy." Software engineers underestimate the stress of a working system that stops working while everyone asks why you can't turn back time. Bryan says that during an outage the personality types separate, and operators keep a cool head, while the dev mindset loses its mind. That makes Bryan a developer at heart, one who immediately goes to the worst case. Bridget suggests DevOpsDays organizers are often ops people because running live events is operations, with no do-over, as the event technology work of Bridget's spouse shows, and Bryan says if things go right nobody notices, so lighting and sound crews rarely hear thanks.

Bridget connects this to Simon Wardley's pioneers, settlers and town planners, placing the host in settlers mode caring about day two. Bryan says people have a preferred mode but can learn the other's empathy. Bryan tells of being told someone complained that Bryan talks about production too much, a complaint that, in Bryan's view, denigrates the people responsible for keeping systems up, since everything ultimately has to boil down to production. Bridget: "Otherwise, we're just making toys."

Computational Literacy and Who Gets Access

Bryan takes the 40-year view and says we're doing well, since so much software simply works, and that "computational thinking has to become literacy," a way to think analytically, not necessarily to code. The guest says there's a bifurcated economy, and the dividing line is how much you're participating in this revolution. Bridget pushes back that there are systemic barriers, citing computer science professors hassled by campus security because they're Black. Bryan agrees and adds that Bryan's inner-city magnet high school needed an IBM investment of millions for a computer lab, while online courses and the cloud now cost far less, though you still need quality instructors.

Bridget says DevOpsDays Minneapolis had 700 people, and with sponsor support they donated about 10% of the surplus to local initiatives, including a job training center run by the American Indian Council whose graduates move from fast-food jobs to data center jobs. They talk about how "the world does not need another dating app" and how Bryan's marriage began on Match.com. Bryan says climate change is going to be kind of exciting, as a grand unifying engineering challenge, and describes deep optimism given how close the Cold War came and how the best minds were briefly in government.

History, WarGames, and Container Summit

Bryan is an unapologetic Xer, and tells how a young engineer at Joyent who hadn't seen WarGames got demerits from the company's online demerit system until the engineer watched it, and afterward said it was a really good movie. The guest adds that Back to the Future is in the canon for millennials and WarGames isn't. Bryan also finds people are interested in history, which we don't teach, and Bridget says it involves rivalries like the split between Cray and Control Data in Minneapolis. Bryan recommends a book on the CDC 6600, with its parallel execution units and rotating Gatling gun, which is in the show notes.

Container Summit began as a marketing event disguised as a conference, Bryan says, and they took it on the road to have conversations with local technologists in front of local technologists. In smaller cities, people turn out for events, unlike in the Bay Area, and Bridget says about 135 people came to the Minneapolis one. The last topic is Joyent's acquisition by Samsung and Bryan's trips to Suwon, where, Bryan says, a baseball game in Korea is amazing and defies description.

Bridget sits down for a classic fireside chat with Bryan Cantrill (Joyent), ranging from containers to social justice to lawn care.

• Container Summit
• Design of a Computer: The Control Data 6600 by J. E. Thornton

photo credit

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off. Also now works on O'Reilly Security and Velocity conferences.

• Devopsdays.org has dates announced for Sydney - Dec 1-2

Open CFPs

• A lot of devopsdays CFPs closing soon - see devopsdays.org/speaking
• OSCON's CFP closes Oct 25

Themes of the Conference

Nell saw a strong emphasis on humanity and technology and how they aren't so different. Mike saw open space after open space come down to testing: "testing all the way down." Jill says a constant theme is that "technology is hard, people are harder." Matty recalls Adam Jacob's keynote on humane systems, context and thriving together, and Jill's morning talk on DevOpsing recruitment, which included a game of guessing whether a recruiter or an engineer said each line, where the answer was always both. Matty also liked a line from Nell's talk on refactoring that what software does matters more than what it was intended to do, and Nell adds that "the only source of truth is when you execute the code itself."

Testing and the Long Road to CD

Matty says test-driven infrastructure has only really been possible for about three years, and can't remember how cookbooks got written before Test Kitchen. Nell says Nathan Harvey, Nell's boss, suggested that to teach TDD to sysadmins, you point out they already test: when they configure by hand, the first thing they do is log in and check it worked. Testing is the same thing, faster and more reliable than a human. Nell sat in an open space on applying the testing pyramid to infrastructure code, where people from Chef, Puppet, Ansible, sysadmin and app dev backgrounds showed the principles are the same whatever the tool.

Matty says people at infra-code vendors are in a bubble where table stakes aren't table stakes for everyone. Mike describes conversations where people know the outline of a transformation plan, but every bullet is a journey that can take nine months or a year, and people who read about 50 deploys a day don't have 50 tests yet: "just because you know the roadmap doesn't mean you can actually drive." Nell recommends what Martin Fowler calls the strangler method, adding tests around one small part of a legacy mess until the new well-tested application strangles the old one.

Matty tells of a bridge in Louisville that was built on top of an old bridge, which let the old one fall away once the new one was strong enough, as a metaphor for keeping the old thing until the new is ready. The host connects it to shims and temporary bridges, which product owners hate, and to Jeff Smith's Ignite about Dungeons & Dragons DevOps: you fight level one monsters first, and learn to write a Git commit message before the 50-deploys-a-day dragon. Customers want to manage their Hitachi storage before they've written a recipe to install a package. Mike finds writing the tests "way, way harder than writing the software," and has mad respect for test automation engineers. Matty says writing the tests makes the code easier, since you've done half of it.

Matty ties this to time to first delight, Matty's own term, which Andrew Clay Shafer calls mean time to dopamine: people have to experience results, since you can't talk them into it, and if you're selling against something that gave them a dopamine hit, as Adam Jacob would say, they will argue with math.

Empathy for Recruiters

Mike says it was the first recruiter talk Mike had seen at a DevOpsDays, and it was great. Jill's goal for 2016 was to speak at an engineering conference, and this was the third talk toward it, after Monitorama, where the topic was Taylor Swift and open source, and Boston. Jill wants to spread empathy for recruiters, and says the good ones are not rare. Mike says Puppet grew from 35 people when Mike arrived to 470 and couldn't have without recruiting and hiring pipelines. Nell praises Chef's in-house recruiter, who also trains interviewers, since some questions engineers are used to asking are actually illegal. Jill says to rely on recruiters for that, and for diversity, since they understand the market, while engineers assess the technical side.

Matty says engineers can be arrogant about everyone else's job, assuming sales or marketing or recruiting is easy. Mike calls it Dunning-Kruger, "The more you know, the more you know you don't know." Why do recruiters get a bad rap? Jill says there are internal recruiters who are ingrained in the culture, and external ones who work on commission and fill seats without knowing the companies. Jill's advice to companies big enough is to "get an internal recruiter and I promise you it will change your life." Jill has gotten emails pitching a DevOps engineer role, and Nell one pitching an administrative assistant job. Matty says the reason we hear about it is that we live in a bubble, and IT people get complained about on Facebook. Making fun of recruiters is lazy, and Mike loves fake internet points.

Jill says recruiters often lack effort as well, and should know their audience, since spamming LinkedIn is wrong for engineers but works for marketers. Matty sometimes replies to recruiters offering to tell them what is wrong with their job description, and about half the time gets a response saying let's talk. Jill runs messages by engineer friends for honest feedback.

The Event Itself

Nell, who is hard of hearing, enjoyed the party at the bowling alley with card games, since it wasn't a loud bar. Jill liked the close venue compared with Boston, where a larger stage felt disconnected from the audience. Mike was thrilled with the deep dish lunch on day two, and Matty recalls that in the first year a speaker got mobbed with questions and missed the pizza.

Three first-timers in the audience share feedback. Mark liked the networking and hearing how companies integrated DevOps, and wished open spaces were organized more tightly. Yasser, a developer, preferred the non-technical talks and suggested speaker ratings. Mike says DevOpsDays went through an arc where culture talks pushed out tools talks and now a few technical talks are coming back, and non-technical talks travel better since they're relevant even if you're not on that technology. Matty recalls two people at the first Chicago, one from a smaller company saying it seemed like it was for big companies and one at a big insurance firm saying it was for small ones, after the same event. Dongmin Liu, also a first-timer, says the non-technical part is the harder one, because it's not about technology but how you package it to convince other teams.

Nordstrom's Four-Year Journey

Doug was the unofficial go-to person for open source questions at Nordstrom, a setup that needed no formal committee or title, just people who care, as Doug tells it. Doug's main takeaway is that "culture change takes time": it took about four years to go from open source contributions being allowed to open source being the default for fast-moving teams. Doug walks through the timeline. In 2008 the only open source they used was Red Hat Enterprise Linux and nobody went to conferences. They bought Chef in spring 2012, and in fall 2012 a VP said contributing is part of using open source, and employees signed an intellectual property agreement. In 2015 developers wanted to contribute to React.js, which required a contributor license agreement, and getting Facebook's approved took about four months, a lot of back and forth from legal and two VPs' approval. By 2016 the Google corporate CLA, for Kubernetes contributions, was signed in a week.

Doug says the close relationship with legal never developed, and wishes it had. Leadership approval was in place, but the VP first approached sent the request to another VP who then left, and Doug suspects a non-technology company's legal team may not focus on differences between licenses.

From Config Management to Everything Else

Michael asks how starting with Chef led to open source in core development. Doug says Nordstrom is a big company, Doug's team never used Kubernetes or React, but leadership came to see that open source lets you move faster instead of a six-month vendor selection, and app teams co-evolved with the Chef users. Bridget says it isn't only for frontend teams, and that if a vendor tells you their platform is the only thing, you should ask whether it's built out of open source. Michael says for a Microsoft organization like NCR, the world of many tools instead of one umbrella is a big shift, and Chef helped them take baby steps toward being community members.

Matty says the same holds for IBM shops, with the old joke that nobody got fired for buying IBM. Matty describes going to a vendor with a problem and being told they don't quite have it, then buying the closest thing anyway and taking on tech debt, as with a BizTalk system that could have been a small open source tool. CIOs love "single throat to choke," but buying everything from one vendor doesn't mean it works together.

Standardization Versus Freedom

Michael says NCR wants to standardize for efficiency, which runs against the open source idea of letting teams pick the best tools. Bridget says a well-composed platform like Cloud Foundry isn't opposed to devs using small tools, as long as choices are open and compatible. Doug uses Netflix, where most teams use Java because there is a paved road, but you can choose something else and support it yourself. People resist standards from a central architecture team that reads white papers and talks to vendors, and accept them from architects embedded on dev teams who write code and support production, and Nordstrom's API teams have moved to Go and Node.

Matty says the best way to get people to do something is "to make the thing you want them to do to be the easiest way to do the thing they want to do," not "thou must." Matty advises a customer to think about the result they don't want, such as an artifact that enables remote logging, instead of policing how people write infrastructure code, and notes that in a huge enterprise, standards have to get vague as you move up. Michael's takeaway is that "standardization is great when it enables freedom," and that Chef works that way for NCR, since it made adding HashiCorp Consul easy. Bridget adds that organizations adopt Netflix OSS like Eureka and Hystrix because good, battle-tested tooling gives an incentive to choose the tried-and-true path.

Open Source and Hiring

Doug's talk argued that an open source stance affects hiring. Several engineers said they looked at Nordstrom's GitHub profile before interviewing, and Doug says "your GitHub organization is your organization's resume in the open source world." They were able to hire someone from Chef primarily because they contributed, and it helps retention too. Doug ran a Twitter poll asking whether people would work for a company that didn't allow open source contributions: 68% disagreed, 25% were neutral and 7% agreed. Bridget says companies building on open source can't expect it all to be written in people's spare time, and should pay employees to contribute. Doug adds that private forks become untenable quickly.

Internal Community, Vendors, and Pull Requests

Michael asks when paying for help with open source makes sense. Michael describes the open source workflow of running it, Googling the error and running it again, and Matty says the important part is then sharing what you learned so the next person doesn't repeat it. Companies with healthy internal communities look just like public ones, behind the firewall, and Matty can tell what wiki software customers use from referral links to ADO episodes. Michael says the relationship with Chef morphed from support to consulting, which helped with thinking strategically about who talks to whom, and gives an example of a colleague in Prague who disagreed with how Michael did Chef, and Michael told the colleague to go with it, which was freeing.

Michael asks whether relying on a vendor for support of an open source tool is unhealthy. Matty says the vast majority of Chef users don't pay, and that if all a vendor sells is support, it isn't a sustainable business, because you'll eventually get better at the product than they are. Matty adds that with closed source, support might be all you get and you can't stop paying. Bridget says Cloud Foundry has contributors like IBM, HP, Swisscom and GE, and Pivotal contributes about 65% of the commits, and that a free 60-day trial of the commercial product lets people try it before paying. So "I can't install it from GitHub, so I can't cut a PO" is a straw man. Matty says if a customer has to spend a week standing up infrastructure to see what a tool does, the vendor has failed, and the goal is time to first delight. Bridget says Andrew Clay Shafer calls it "mean time to dopamine." Bridget also says look at the community around a project, like the US government's 18F building cloud.gov on open source Cloud Foundry, with Terraform scripts on GitHub. Doug pushes vendors to hand over a Terraform script or CloudFormation.

Doug says enterprises new to open source respond to bugs with "the vendor should fix that," but they have unique insight into the problem. Doug found a bug in Supermarket that could be fixed in five minutes, and it was merged within an hour, versus filing a support ticket that takes two and a half weeks. Michael says Chef support once demonstrated the same lesson by opening a GitHub issue, submitting the PR and reporting back with a link. Doug has also been on a closed-source project where the vendor said it might get to a fix in six months, after millions were paid. Bridget likes subscription models because vendors care about churn, and Matty says the dirty secret is that "Renewal time is the best time to ask for anything from your vendor."

Freemium and Build Versus Buy

Michael asks about the freemium model like Chef Automate. Matty says that in Chef 11 you had to reinstall everything to drop the enterprise version, whereas since Chef 12 and through Automate, the core is open source and everything around it is a wrapper, and "you want that premium thing to be sticky because it's valuable, not because it's in the way." Michael asks whether the answer is instead to build a Jenkins pipeline and Elasticsearch cluster in-house. Bridget's answer is that it depends, but the fallacy in organizations with smart people is thinking they must master everything down to chip fabrication: "you should only build the things that make a differentiating business value for you." At DramaFever, they didn't build their own CDN, and when the Akamai bill was high, they prepared to go multi-CDN and it turned out Akamai would negotiate. They did build a Docker, Packer and Chef-based platform, since it let them stand up a K-drama site or a horror site from the same image.

Matty says there's no reason to run your own email, and that "modern DevOps, we're all about coulda, not about shoulda." Pay someone to do the thing you'll never do again, like setting up a cluster once every five years, but not to write automation you'll change. Matty recalls dev leads at Apartments.com who wanted to write their own authentication for an internal customer service app when they were a Microsoft shop with Active Directory, and calls that hubris and resume-driven development. Doug adds that consulting is most useful when the consultants partner with you and you dedicate engineering hours, and that paying for Chef was worthwhile for the support and the add-ons customers expected, like a web UI. Matty ends with an example: at Chef they didn't write their own search provider, they use Solr and Elasticsearch.

Matt and Bridget chat with Michael Hedgpeth (NCR) and Doug Ireton (1Strategy) about organizations adopting open source.

Checkouts

Michael

• DevOps Days DFW on September 15 & 16 - I’ll be speaking
• Toyota Kata book
• PBS SpaceTime

Doug

• Habitat for application automation
• Adam Jacob’s DevOps KungFu talk

Bridget

• Coté on “the ROI on devops/agile/etc”
• BWCA is part of Superior National Forest. The National Park Service is 100 years old this year and National Forests are different than parks!
• The West Wing (streaming on Netflix), because it’s election time.

Matt

• Wakatime - fitbit for programmers
• Homeland!!!

Image credit: https://flic.kr/p/3V99c8

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off. Also now works on O'Reilly Security and Velocity conferences.

• Devopsdays.org has dates announced for Sydney - Dec 1-2

Open CFPs

• A lot of devopsdays CFPs closing soon - see devopsdays.org/speaking
• DevOpsDays Madison, Detroit, Cape Town, and Nashville open until August 31
• DevOpsDays Berlin open until September 1
• DevOpsDays Bangalore open until September 4
• DevOpsDays Ghent open until September 6

Where we’ll be for the upcoming fortnight

• Bridget - Canoeing with no phone or internet, and then Velocity NY/devopsdays NY.
• Matt - DevOpsDays Chicago Aug 30-31

Checkouts

Tim

github.com/joyent/containerpilot and example applications at github.com/autopilotpattern

Adam

habitat.sh

Bridget

The Art of Closing - open source maintainer advice from Jessie Frazelle

Badass by Kathy Sierra

Matt

DevOpsDictionary.com - I’m having a bit too much fun contributing to it.

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off.

Open CFPs

• Devopsdays.org!!!
• DevOpsDays Singapore open until August 15
• DevOpsDays Ohio open until August 26
• DevOpsDays Madison, Detroit, Cape Town, and Nashville open until August 31
• DevOpsDays Berlin open until September 1
• DevOpsDays Bangalore open until September 4
• DevOpsDays Ghent open until September 6

Where we’ll be for the upcoming fortnight

• Bridget - SpringOne Platform in Vegas Aug 1-4
• Matt - in Chicago. I am boring now. But you should all come see me at DevOpsDays Chicago Aug 30-31...a few tickets left, ADO2016. Plus now with extra Adam!

Everyone Is a Software Company

Bridget asks what kinds of companies are realizing software matters. James mentions Merrill, a sponsor, whose executives were betting on a new software product as a global SaaS brand. Charity says platforms mean you never know who will show up: Disney signed up and created an account without talking to them, next to 13-year-olds building software on the same platforms as corporate America. Charity calls it the democratization of tools that used to be a competitive edge for big vendors with giant R&D budgets. Andrew describes a reinforcing spiral that started with open source, where the primitives for building Facebook, Google and Amazon became available, and has gone up the stack to things like TensorFlow, so you can start thinking about your domain and have neural networks. "The democracy is going up the stack," Andrew says, and James adds "abstraction levels going up democratize technologies," like a toddler using a touchscreen.

Nicole says the companies doing it right reach customers earlier, get to market and feedback faster, and reduce complexity with an MVP, and the ones that spend a year on an RFP fall behind. Charity says that's why hashtag NoOps draws a wince: operations isn't vanishing, good operations is a competitive advantage, and what is shifting is the definition.

Operations Isn't Going Away, and Neither Is Its Value

Charity describes "the shadow self of DevOps": the industry has spent almost a decade telling ops people to write better code and tests, but hasn't told developers what they need to learn, the operational skills that let them build, ship and maintain products. Andrew dislikes labeling people as ops or dev, and prefers thinking in capabilities, since not everyone can do everything as organizations scale. Andrew says NoOps is a reaction to a world where sysadmins were responsible for the mail, and that "the systems are actually a reflection of the organization." Charity says anyone giving advice without context is selling nonsense. James asks enterprises what problem they're solving before talking about product differentiation, and asks how long deploys take. One bank told James eight weeks and 10% of their staff, a year into building their own platform. Nicole says that is solidly mid-to-low performance, and not even the worst of it.

Charity says an inherent trait of infrastructure done well is that you don't notice it, like roads, and Bridget recalls the Minneapolis freeway that fell into the river in 2007. Nicole adds "when the work is done correctly, the work disappears," which contributes to devaluing it, and points to systematic salary differences between development and ops, and Charity says Google couldn't retain SREs until it paid them more than software engineers. Andrew says leading-edge cloud-native companies don't have as much of a disparity, and that treating IT as a cost center forces bad behavior. James says offshoring tried to lower the unit cost of developers as paying less for ops did. James describes operators running thousands of containers and hundreds of apps and says raising abstraction lets an operator have leverage, so James tells the operators' bosses how valuable the operators are.

James's advice for IT professionals is that "application architecture and operations architecture are not dissimilar things," and to think about how a database is operated, not just its API. Andrew says lots of people who think they have operations problems have architecture problems, and adds, "Day 2 matters."

Meeting People Where They Are

After James leaves, Charity says context is everything, and Andrew says continuous delivery and microservices can sound like fairy tales to someone far from them. Nicole says you have to meet people where they are and help them envision what's possible, remembering the comfort of doing it manually back at IBM. Andrew uses a marathon analogy: if you're not ready and you try it, you'll hurt yourself. Charity's version is a patient in the emergency room with a broken leg and blood spurting from their head, who shouldn't be worried about skin cancer yet. First stabilize the patient, and the top skill in startups is "ruthless prioritization." Andrew says overfunded startups fail because cash protects them from their context.

Asked how large organizations get startup nimbleness, Nicole says it happens in small teams that try something and scale, not in an organization-wide nimble process with posters. Charity says Facebook was a revelation, with a horizon 18 months out, and that what works at scale is "Outcomes-oriented and trust," define an outcome, develop trust and then go hands-off, empowering people to make mistakes as long as they won't destroy the company. Andrew says smart people solve problems, and also, "Smart people also cause problems." Charity says every workplace of Charity's had an outage caused by nearly dropping all the data.

Careers, Teams, and Performance Reviews

An audience member asks about HR and career development in DevOps teams. Charity says what an org values differs, like Heroku's five nines counting toward promotions, and signaling what you value is how people are incentivized. Charity likes asking reviewers who they would most like to be paired with on call, or would call at 3 a.m., and who they least want to be paired with, since it differs from who the best engineer is. Andrew says work can be done so that you're better at it afterwards, and if not, reevaluate the work, and adds that not confronting incompetence demoralizes the rest. Nicole says university doesn't prepare you for distributed software or continuous delivery, for developers or ops.

Nicole's rant is that individual performance reviews are nonsense, because "Teams deliver software, individuals don't," and invisible work like the person who glues the team together may show the fewest commits. Andrew says "As soon as you make a measure a target, you made the measure useless," and takes a contrarian position that humans are non-fungible. Nicole says Google's study of 36,000 engineers found team dynamics, with psychological safety first, matter most, and Charity says Google still hires "as though they're hiring for Lego bricks." Andrew describes the 80 percenter persona who can prototype anything overnight and will never build anything that belongs in production. Bridget recalls Alice Goldfuss's talk on rock stars, builders and janitors. Charity says startups have the advantage of hiring for what the team needs, and that constrained resources drive creativity, and you can carve out a small budget for a team inside a big company, like a startup.

What the Data Says About Enterprises

Nicole responds to the assumption that the data doesn't apply to enterprises: "there are no statistical differences among the different enterprise sizes," or by industry, including highly regulated ones. Nicole classifies teams as high, medium and low performers by throughput, meaning deploy frequency and lead time, and stability, meaning mean time to restore and change fail rate. High performers score significantly higher on the Westrum culture measure, with high trust, good information flow and messengers not shot. Practices like version control of infrastructure, application and configuration differ, and the firms' characteristics don't. Nicole also sees a 50% difference in stock price performance between high and low performers over three years. Charity summarizes: you have no excuse. Bridget notes a recent conversation with a large company excited to be getting Git this year.

An audience member asks whether high performers do stability or throughput first. Nicole doesn't have data on sequence and doesn't see trade-offs, and says gains now come in speed because "slow is the new down" and stability is near its ceiling. Andrew adds "Each nine costs 10 times more than the last one." Another audience member asks Andrew about synthesizing ideas, and Andrew says the best thing a technical person can do for their career is learn to speak and to write, helped by Andrew's time on a debate scholarship.

Takeaways

Charity has said no to 26 conferences over the next six months, and wants more acknowledgement at DevOpsDays that this is for software engineers too. Nicole shares Andrew's ideas from the conference: "all code is technical debt," so people should be rewarded for removing it, and "software is never the endgame," so do things for the business, the customer, not because of DevOps. Andrew adds that tests are code and thinks DevOps, microservices and continuous delivery are a single phenomenon, a cloud-native paradigm that can't be treated as separate initiatives in separate silos, and cites Deming: "change is not mandatory. Survival is not mandatory either." Bridget closes by praising Jeff Smith's talk from Grubhub about what happened when they did DevOps, with the spoiler that not everything is unicorns and rainbows.

How do large enterprises transform the way they do IT? What does it mean for every company to become a software company? Our panel of experts at devopsdays Minneapolis 2016 has worked in some of the largest orgs out there and has seen a lot of transformation first-hand.

• devopsdays Minneapolis

• Alice Goldfuss - Rockstars, Builders, Janitors: You're doing it wrong

• 2016 State of DevOps Report

Big Companies Shipping Ideas

Matty notes the keynote theme of shipping delight and velocity. Jon says what has been most interesting over the years is that this rapid delivery has spread to much bigger companies that traditionally didn't work that way: the National Football League and GE spoke that day, and GE said it generates something like 40% of the world's power and needs to iterate on features. Trevor adds Alaska Airlines to the list and says the point is that they're succeeding, since so many have said they can't because they're big.

Fletcher brings an example to conversations with friends whose companies say they can't move faster: Standard Bank went from days or weeks to hours or minutes, and Disney said last year it had been at it five years, so a competitor starting now is five years behind a company that's already faster than it was two years ago. Fletcher argues the audit-every-three-years compliance theater "just does not ring true to me anymore." Matty says Alaska presented itself as an 84-year-old multi-billion-dollar startup that doesn't throw money at problems. Jon says it is easy to call startups disruptive when the cost of being wrong is slight, and it's cool to see an airline attack the idea that travel just sucks, including a demo of e-ink baggage tags that update from the mobile app.

Matty adds a customer who said Chef had started to break down their silos, because everyone in the room had never worked together until they started doing it. Matty also recalls enterprise customers saying Matty was the fifth person that day to tell them to be like Facebook, and the Etsy episode line that they're "not a unicorn, they're a sparkly horse." Trevor has just been in Singapore, running a Chef training at a PowerShell meetup where DBAs, bankers and machine automation people all had the same story.

Accessible, Not Dumbed Down

Annie wants to show people that InSpec and Chef are accessible, and says, at three months in, that "there should be no fear." Matty says the idea behind Chef Compliance and InSpec was to communicate with code, so compliance and audit teams hand you compliant InSpec code in place of Excel sheets and PDFs, and has been asked whether they'd want to. Annie's point, which Matty loves, is that it is accessible, though not easy. Annie credits Test Kitchen, and a long conversation with Fletcher the day before, for lowering the barrier to entry, and says open source is making things more accessible to non-technical people, and "I don't think it's dumbing it down at all."

Matty says that moves the value to the content, not the arcane, and admits some veterans took protectiveness or job security from being the only one who knows. Matty's response is to let the robots do the boring stuff and use your big brain on other things.

The Community Summit

Annie loved the Community Summit, calling it a hallway track times five. Annie proposed a security compliance topic and only about eight people put a dot on the card, which was informative, given Barry's keynote claim that compliance is the bottleneck. Fletcher's take is to give it a year or two: at the very first Opscode summit, where Fletcher got a free ticket for having contributed to Chef, conversations about testing infrastructure got mixed reactions, and it later became a real thing. Matty adds having asked Fletcher about Test Kitchen for Windows two years ago, then 45 people showed up to an open space six months later, and six months after that it shipped.

How ChefConf Has Changed

Fletcher says the DNA is the same, professionally done and with talks you want to see, but at the first one companies were half saying this is what we're thinking of doing, and by years two and three they were sharing lessons learned, from automating infrastructure to testing it to continuous delivery. This year's theme is going from someone's brain to production, which pushes people to talk about the business, and CEOs now come too, so the audience is the whole organization. Fletcher's mind-blown year was when Facebook and Disney talked.

Trevor recalls telling Jon last year, a month into using Chef, that it felt like lying to clients, and Jon replied "slow down, you're an expert because you can find the answers." Jon adds that Chef is a for-profit company whose future is in enterprises, but the open source core hasn't been lost: Facebook invests heavily in the community, Target contributes cookbooks, and the governing board and most project lieutenants aren't Chef employees.

Automate and Habitat

Jon says earlier Chef web UIs were built by and for terminal people, and someone told Jon during the announcement "thank God Chef has a sane UI at last." It gives a view of converging nodes, changes deployed and compliance, and Jon hesitates to call it a single pane of glass but says it's a tightly scoped one. Matty likes that the decisions were about what belongs in a UI, meaning visualization. Automate's three pillars are Chef, Habitat and InSpec, which Matty notes are the three guests, more or less, and Fletcher corrects Matty that they're products, not projects.

Fletcher wants well-behaved services from Habitat. Startups with infinite time might build everything like Netflix components, with health checks, streamed metrics and self-clustering, but most business software, even a mainframe, doesn't, and if every service, even Postgres, behaved the same way, you could compose systems on top. People try to fit it into something they know, like Docker, and Fletcher says the confusion comes from the pieces. The guest says the seed of the idea "has legs and I can feel it in my gut." Matty says people wrapped shell scripts in execute blocks when they first met Chef, and Habitat needs the same step back: articulate what you want and let the product handle the sausage. Fletcher says there's a "Chefness," a consistent company DNA across three products that come at you at right angles.

DevOps 2.0

Trevor has been hearing people ask what comes next for DevOps 2.0, and hopes that saying it on the show will bring enough ridicule that it never sees daylight. Trevor called the person after the keynote and said the thing they meant already has a better name. Jon says these labels are usually a logical evolution of what people have been doing, moving into new areas, and "we just happen to like sticking names on it that look good in headlines." Matty says jargon is "a linguistic shortcut for communication," which fails when it means different things to different people.

Favorite Talks and What's Next

Annie loved the InSpec talk by Christoph Hartmann, standing room only, where the speaker tested something for compliance and remediated it with a cookbook in 20 minutes. Fletcher's moments were the GE presenter and a colleague's Habitat talk that revealed an automatic sharding approach that was new to Fletcher. Jon's favorite was Tim Smith on composable cookbooks with custom resources, using the Tomcat community cookbook as an example of one that grew to include Gentoo and systemd support until it did none well, an honest retrospective from a Chef employee. Jon calls Chef "a toolbox that gives you the tools to solve your own problems," and says that is why composable resources beat a drop-in cookbook. Trevor liked the CTO and psychologist keynote, which helped with a transformation project Trevor is leading in Singapore, and the Target talk on Chef and SharePoint, which they are working on open sourcing.

Looking ahead, Jon is more interested in organizational transformation as Jon's role moves toward management, and this is the first ChefConf where the guest isn't speaking. Fletcher is looking forward to Adam Jacob's second-day keynote, which has been a personal turning point at earlier ones, and to talking to people about whether Habitat means you need less orchestration. Annie predicts next year will be a lot more about security, since it's still siloed and a huge bottleneck.

Awesome videos from ChefConf:

• Chef Presents: HugOps
• Barry Crist - ChefConf 2016 Keynote
• Chef Automate Demo - ChefConf 2016 Keynote
• Veresh Sita, Alaska Airlines - ChefConf 2016 Keynote

Other delightful stuff:

• Habitat
• Chef Automate

Severance agreement timelines: Employee Checklist: What to Do When Your Employer Offers You a Severance Agreement ... Esp: “If your employer has not given you a reasonable amount of time, or rushes your decision, this is a red flag...If you are being rushed, ask for more time [in writing]”

How to bounce back: Let folks know - Mike Fiedler and Annie Hedgpeth had good examples on Twitter recently.

Use all your networks

• https://www.themuse.com/advice/help-me-find-a-job-emails-to-send-to-your-network

Check Outs

Jill

• Newly obsessed with this site (and not just because I’m being interviewed for it in a month or so) https://usesthis.com. Its super interesting to read about how people do their day to day activities from all different industry/skill backgrounds

Peter

• Infinite Jukebox: http://labs.echonest.com/Uploader/index.html
• My Job Went to India (was renamed in the 2nd edition to “The Passionate Programmer”) (Chad Fowler, 2009)
• Fox in Socks, Dr. Seuss:

Bridget

• Jill’s epic Ignite at Monitorama
• Curating the content you produce. Yes, I think everyone should set up their website like I do. (see also http://www.devopsdays.org/events/2015-detroit/proposals/how-to-train-yourself/)

Matt

• Arrested Westeros

Trevor

• DOOM E1M1 Cover - DOOM is fun

Further Listening

Want to dig deeper into some of the things we discussed today? Check out these past episodes:

• Building Your Personal Brand
• Career DevOps

From Checks to Event Streams

Jason describes the old Nagios model as monitoring how something is doing right now, which loses the historical context of how the service has behaved. Over six or seven years, open source projects for different functional areas produced the event stream model, with collected metrics, window threshold queries and alerts, and eventually paid services for specific areas of the architecture. Aneel says the old way only asked about one thing, whether it was up, and nothing about the cluster, the service or the user's experience, and between checks you have no idea how it is doing. That isn't enough if you measure yourself on real performance for real people.

Thresholds, Machine Learning, and Business Context

On thresholds, Jason says monitoring works like security in depth, with layers, since no single answer says how a service is doing. Jason is not a fan of machine learning as the solution, though it is a great addition, and thresholds are fine where you know what to expect. What matters is business context: am I doing the work I'm supposed to be doing, not just returning 200s? Jason recalls a customer at OmniTI who said something to the effect that they didn't care if their servers were on fire as long as they were making money. Aneel says people think moving beyond static checks is only for ephemeral infrastructure, but what you always wanted was to know whether you were within your expected performance envelope, and now you can measure it. Matty adds a story about a sysadmin alarmed that a database was at 90% CPU when it was SQL Server doing its job.

The Single Pane of Glass

Matty asks "how big is your pane of glass?" Jason says a single pane is specific to your role, and that tools should be dynamic, which is why Jason built dashboards that let you look for hotspots on the fly, and says horizon charts are handy for spotting problems at the surface. Aneel, who spent a long time at IBM watching single panes come and go, says each role wants one, but the systems shouldn't be walled gardens, since the CEO's may be Tableau and marketing's Mixpanel. Data has to flow in and out so everyone can compose their own view. Jason says "if you're siloing your data within your organization, I think you have bigger cultural issues." Aneel describes a company where one metric, the number of documents users open in a time period, is tracked by everyone from the CEO to network engineering, though not every business reduces to one.

Bridget cites James Turnbull's point that the operations team isn't the only customer of monitoring, and concludes that it should be self-service and composable. Jason loves self-service monitoring, and describes a Heroku tool that queried Graphite and returned an HTTP status code, so anyone could put a check into Pingdom in minutes.

Monitoring Is Testing With a Time Dimension

Matty quotes John Sheehan, from an earlier episode, that "monitoring is just testing with a time dimension," and asks customers whether something important enough to monitor in production is being tested. Jason jokes that testing is monitoring without context. Aneel says the engineers at SignalFx bake metrics into code from day one and carry pagers, and that if you want to know whether a change had its intended effect, the same metrics have to follow it from code through test, QA, canary and production.

Jason asks about usage-based pricing, which both companies use. Aneel says it is the scale of monitoring you want to do, not the number of things, and that it gives you freedom to change rates and stop measuring things, like buying bandwidth, but it comes with the mindset of iterating on which handful of metrics matter.

What's Worth Monitoring

Jason says the most common and hardest question is what to monitor, and the honest answer is that you're the only one who knows. Aneel adds that the approach has to be non-static. Aneel uses Kafka, which has no metric for average message size, though a change in message size affects throughput, unless the producers and consumers are designed for size to change. Bridget confesses to a cron job checking whether an Elasticsearch cluster had gone yellow, and Aneel says you should get one alert when a cluster changes status, not 40 alerts from static checks.

Ephemeral Systems and Serverless

Jason cites an Adrian Cockcroft talk about systems moving from months to weeks to days to minutes or seconds with containers. You can't cron that, and hostnames don't matter since the system is gone before you finish a sentence, so you have to think in work units: how much work has to happen, and is it happening? Aneel says with Lambda-style systems you still measure the timing and performance of the functions and the experience downstream, and with service level objectives, changes in the underlying model change only how you measure. Aneel also says ephemeral infrastructure isn't what drives the need for metrics. A customer of Aneel's runs 100 physical machines with 20 to 30 Docker containers each across five locations, measures performance so precisely they don't need auto scaling, and still wants metrics.

Where to Start

Jason's advice is to start with open source until you hit the point where your time is better spent elsewhere, then outsource to someone like Librato or SignalFx, since knowing the technology makes you an informed shopper. Aneel recommends collectd as a collection agent, for its plugin ecosystem and, compared with every other agent tried in 20 years, its stability, though the warning is to watch its plugins. Jason trusts the team because many are OpenBSD developers, and says the biggest problem with agents is leaking memory and consuming CPU. Aneel adds that Graphite is a good starting point, and Jason mentions that the Graphite book is nearly done. For Windows, Aneel and Jason mention a collectd port by a team at Bloomberg, a PerfCounter reporter that SignalFx engineers wrote, and Metrics.NET.

Aneel returns to machine learning. In Aneel's experience in operations, nothing has done better than about a 50% false positive rate. The right use is as an additional tool, like a system that classifies incoming metrics as in-line or out-line and sends the outliers to another process, and "the single worst thing you could do" is turn on an algorithm and let it generate alerts.

Monitoring Versus Alerting

Bridget asks about the difference between monitoring and alerting. Aneel says "there's no monitoring without alerting": in operations, you monitor to figure out what is worth alerting on, and you should page only when something takes you outside the performance envelope of your service. Everything else should be an event you can go back and interrogate. Jason says alerting is a subset of monitoring, and the same data feeds capacity planning and analytics. Aneel adds availability to the mix: "if something is available and non-performant, it might as well not be available," so capacity, availability and performance move together.

Closing Advice

Jason's advice is to "try not to chase the shiny stuff," to start with basics like instrumenting your code and to ask why configuration management providers don't do more to emit instrumentation to storage engines. Jason also says that if a tool is painful to use, try another, since there's so much choice now. Aneel says no one will figure out what metrics matter to you: "If you don't take responsibility for figuring out what metrics matter to you, no amount of money and technology is going to do that for you." Matty's version is that the hardest part of monitoring is figuring out what you care about.

• Jason's Graphite book: Monitoring with Graphite: Tracking Dynamic Host and Application Metrics at Scale
• James Turnbull's Art of Monitoring

Checkouts

• Jason - Charity Majors' recent blog posts about serverless conf, etc

• Matt - Fish-like autosuggestions for zsh

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off.

• DevOpsDays Silicon Valley June 24, 2016 - June 25
• DevOpsDays Minneapolis is July 20-21

Open CFPs

• DOD Dallas & Raleigh June 19, Philly June 30, New York July 15, Singapore Aug 15, Detroit Aug 31
• New cities: Porto Alegre in Brazil, Baltimore

We have t-shirts now! And mugs! They are available at store.arresteddevops.com! Only unisex for now, but more styles coming! Buy one today. Or not. We’re not the boss of you.

Curating the Event

Dave says this was the third Toronto event, in a couple of venues, with a diverse audience and a growing enterprise focus. Steve says it was the second year at the same venue. Asked what they were curating, Dave says they take the code of conduct seriously and had an incident to deal with promptly, and that they look for speaker diversity. Steve is candid: they did an outstanding job on topic diversity and a poor job on representation of groups and demographics, with no excuse, and next year they need to start earlier and go out into the community. Attendees were happy with the range of culture and technical talks, and Steve says they tried to serve the enterprise audience without creating an echo chamber.

Sarah has been to a couple of DevOpsDays in Australia and notes that Toronto has more of a focus on financial tech and banks. Steve adds that Toronto has many attendees from fintech and legacy banks, but the banks don't show up as sponsors, and Steve would like to see them invest at the organization level. Joe says London was very fintech-focused, with Barclays as a sponsor.

ChatOps

Sean's talk was about the team's chatbot, Waterboy, one of two talks Sean submitted, and Sean didn't expect this one to be accepted. Sean's team is very distributed and conversations were happening in private, so moving tools into the chat room made troubleshooting collaborative and left a history. Their tools let you look at a URL and see timing, or take apart a microservice to see how the CDN affects it.

Steve is a fan of ChatOps as an easy path into DevOps patterns: everything happens in the open, is recorded, and is a single interface to many backend operations. Sarah says PagerDuty's operations team moved from deploys to provisioning, decommissioning and Chef converges via ChatOps, so the team no longer provides "keyboard as a service." A plugin that returns context for an IP or hostname means "we stop getting questions about it," a very easy metric. Sarah adds that you should log chat to centralized logging so it's searchable. Dave, a product manager, likes chat in the incident response lifecycle, such as support paging the incident commander from the bot. Steve values not having to interrupt people. Joe says the organizers run their events in Slack, and an audiovisual vendor form posted once gets found by search.

First-Timers, Open Spaces, and Failure Stories

Sean was a first-timer and had been told not to overthink open spaces. Sean enjoyed conversations about metrics, chat and incidents, and liked that the talks were "by people just like us." Dave says "you don't have to be a professional speaker in order to get up and talk at a DevOps Days." Bridget praises the talks about what didn't work, and Steve says that was curated on purpose. If you don't know what to expect, you may think the speakers are beyond your organization, so mixing success stories from companies like Shopify with teams that failed inspires people afraid to start. Dave adds "We all fail. We suck in places," which breaks down boundaries.

Sarah says open space selection was done differently than in Australia, where people pitch for about 20 seconds and others tally marks on cards. Toronto used a Trello board and a show of hands. Bridget likes pitches, since Sean pitching a talk about ChatOps beats the bare word on a board, and Sarah says a pitch framed as a problem gets a bigger response. The Toronto organizers also tried bringing the mic to people so introverts could pitch sitting down.

Getting Under-Represented Voices Into the CFP

Bridget says that as an organizer, the approach is to chase the groups you want represented before the CFP opens. In Minneapolis this year Bridget hand-fed the link to a few people, and their talks were upvoted by the rest of the team with names and companies stripped. Bridget thinks a large bank should get an invitation to tell its journey. Steve says the work is continuous, since 2017 starts as soon as 2016 ends. Sarah, who'd usually talk on ChatOps, will get voluntold to give the ChatOps talk next year, and Dave notes they won't invite Sean back: "Sean's done. He had his one chance."

Legacy Isn't Boring

Dave says the different cultural pockets inside large companies are undervalued, and the only thing that works in an enterprise is land and expand, a change agent in one area. Bridget says bimodal IT is bad because it tells coworkers that some get awesome mode and others get sad mode. Sean says "I wish people would stop treating the old stuff as boring. It's the stuff bringing in the money," and that experimentation needs the users and traffic that are on the old stuff. Bridget agrees: "if the old stuff didn't matter, you would just turn it off."

Other Events and Formats

Sean's earlier version of this talk was to federal workers in DC, where the idea of a chatbot doing work seemed foreign and the jokes didn't land. In Toronto people were far more open about their problems, which Bridget says is a hallmark of DevOpsDays, including DevOpsDays DC at the US Patent Office. Joe says single-track events have lulls in vendor booth traffic because everyone is in one room, and Steve says multi-track brings FOMO traffic. Sarah says it's recorded, so you can catch it later.

Looking ahead, Steve wants to get back to speaking, Sean has a ChefConf talk in July, Sarah looks forward to Monitorama, and Joe has t-shirts, AV and menus for Minneapolis. The thing Joe plans to steal from Toronto is a structured tech check for all the day's speakers half an hour before the event, since AV people like early testing more than "I'm speaking in 15 seconds and I haven't tested this." Steve says to take their laptops away afterward so they don't change settings.

devopsdays Toronto

Community & Event Stuff

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off.

• DevOpsDays Silicon Valley June 24, 2016 - June 25
• DevOpsDays Minneapolis is July 20-21

Open CFPs

• DOD Chicago, Boston open until June 1, Dallas & Raleigh June 19, Philly June 30, New York July 15
• Singapore Aug 15, Detroit Aug 31
• New cities: Porto Alegre in Brazil, Baltimore
• We have t-shirts now! And mugs! They are available at store.arresteddevops.com! Only unisex for now, but more styles coming! Buy one today. Or not. We’re not the boss of you.

What Security Is

Bridget asks Ben what security even is. Ben's answer: you have the stuff, and you don't want other people to get it. Computers aren't as deterministic as we'd like, since CPUs and microcode have bugs and Rowhammer showed that physics can corrupt memory, so insecurity goes all the way down. The role of security people is managing that risk, and Ben adds that "security often loses sight of the fact that they are a business function." For most companies the goal is a profit, not being the most secure company in the world.

What Containers Buy You

Jessie's view is measured: "you're better off with containers than without them," as long as you run them correctly. If someone gets into an app inside a container, the world they see is different from what they'd see on the host. It can't save the world. Ben puts containers in the broader category of sandboxing, like Chrome sandboxing Flash, and says the main thing you can do is reduce attack surface, especially by dropping as many permissions as you can. Bridget and Ben both remember jails and chroots.

Jessie explains unprivileged containers, from Jessie's blog post and CraftConf talk. Docker on your host runs as root, and adding a user to the Docker group also gives root, which some people don't realize. With unprivileged containers, the user starting the container is a normal local user with no added capabilities. Jessie also runs Chrome in a container with cgroup limits on RAM and CPU, but had to remove the limits because of a Chrome memory leak, so if Ben and Bridget see Jessie drop out of the Hangout, the out-of-memory killer is probably to blame.

The Mundane Beats the Dramatic

Asked where the sweet spot is between securing everything and nothing, Ben says humans are really bad at risk analysis. Nobody says "safe ride to the airport," they say "safe flight," though Ben has been more terrified by taxis than by pilots. Ben loves a kernel-hardening project that makes whole swathes of kernel exploits stop working, but would rather have people use longer passwords and a password manager. Most compromises are found credentials or very old software, not an amazing new kernel zero-day. The dramatic wins headlines, logos and RSA booth sales, as with the $100,000 appliance that claims to stop all zero-days. Ben uses ImageMagick as an example of an exploit whose barrier is nonexistent: "If you can write a sentence, you can probably exploit it." A logo and a domain got more people to patch than a blog post did.

They wander into online voting, where Ben says the paper system is reasonably trusted and the government contracts go to companies like Diebold. On chip-and-signature cards, Ben has heard the banks didn't want to change two things at once. Jessie says the line is Linux in cars, though open source, Jessie admits, is better than every company writing its own firmware.

Impostor Syndrome and Who Gets Hired

Ben's recent blog post, written on a flight back from Berlin, is about impostor syndrome in security, a field with a lot of posturing and an attack-defense mindset. "We should try and be nicer to each other." The field has a huge skill shortage, but it's an unwelcoming place if you have to be popping shells on day one. It also tends to want only breakers, and a team of breakers doesn't build secure software, it finds bugs in all the software you have. Ben's example is Wireshark, which parses loads of wire formats in a big C program: "Wireshark is just a CVE-generating machine." Jessie has a container for it, though it would need a custom seccomp profile, and Ben recommends capturing with tcpdump and loading into Wireshark in a VM or container.

Jessie chose container security because the problem of real multi-tenancy appeals, and says that nobody has 10 years of Docker experience, so you hire people who are good at what they do and can jump in. What Etsy actually finds more useful, Ben says, is people who can talk to developers and explain an attack, which won't get a conference talk but helps the business more. Jessie sums it up as "enabling people to do the right thing versus telling them that they were wrong in the first place," and Bridget adds that at Etsy corporate security enables people to do their job, which differs from how security usually interacts with the business.

DevOpsSec and curl Bash

Ben spoke in Berlin on the topic, whose name Ben blames on Gareth Rushgrove, to an audience mostly of executives. It included Pete Cheslock's image of the DevOps unicorn emitting rainbows while security shovels them out. Etsy doesn't use the term DevOps much, but the point is to get security involved early, embed security people on other teams and stop shouting at people for writing code with bugs. Ben also mentions an article on detecting curl piped to bash through server-side timing, since the shell buffers differently than a plain curl, which lets a server send different output.

Tooling, Config Formats, and Containers Done Right

Jessie says the talk was for anyone running containers, and surprisingly popular with the academic physics community, whose servers don't allow running as root. Real sandboxing needs custom seccomp and AppArmor profiles, which will land on the security team. Better tooling would help, since no one likes writing SELinux policy. Jessie's proof of concept for AppArmor uses TOML, which Jessie thinks should be JSON. Ben objects that "JSON isn't a config format," since you can't put comments in it, and that YAML's significant whitespace is "not acceptable in a config format."

Predictions and Wishes

Jessie predicts containers will keep getting more secure. Ben's bold prediction: "I predict in the next 12 to 18 months, there will be another OpenSSL vulnerability," and more ImageMagick bugs, since image and config parsing is a minefield. Ben thinks "the container security story is great in the kernel but is terrible on the actual things in the container," with hundreds of things running out-of-date code where there used to be one monolith. Bridget says that if you can't build images repeatably and roll them at a moment's notice, "you probably have no business using containers in production." Ben says the speed everyone was sold on disappears once you build a repeatable, testable build system, and Bridget answers that you can still push fast through CI with tagged images.

For wishes, Ben wants security to stop blaming everyone. You tell people not to click links in emails while employing a recruiting team to click on PDFs from the internet, so "the tools have failed. So make better tools. Stop blaming users." Jessie wants a desktop OS made of containers, like Subgraph, and people to stop making gigantic images. Ben adds: "People should stop using curl in Dockerfiles," and stop using HTTP there too. Bridget adds pinning versions, and Ben's advice is to find an ops person and work it out.

• Jess on unprivileged containers
• Ben on infosec, hubris, and impostor syndrome

Community Stuff

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off.

• DevOpsDays Salt Lake City June 14 - June 15
• DevOpsDays Silicon Valley June 24 - June 25
• DevOpsDays Minneapolis is July 20-21
• DevOpsDays Chicago is August 30-31

Open CFPs

• DOD Amsterdam open until May 30
• DOD Chicago open until May 30

ADO Merchandise

We have t-shirts now! And mugs! They are available at store.arresteddevops.com! Only unisex for now, but more styles coming! Buy one today. Or not. We’re not the boss of you.

Check outs

Ben:

• The 2016 DZIR report
• Which is similar to the DBIR from Verizon
• Phrack 69 is out! great stuff from Joern on RoR, horrors of Adobe, how OSX gets rootkitted.
• Detecting curl bash

Jessie:

• Linux Container Security Paper from NCC Group
• Subgraph OS

Bridget:

• Having a lot of fun with terraform lately - be sure to read the great posts Charity Majors wrote about it

How It Started

Ryn started speaking in 2013, when Jason Dixon, who organizes Monitorama, asked for a talk at Monitorama EU in Berlin. Ryn's first reaction was that there was nothing to say, and Jason's answer was "I follow you on Twitter because you have things to say." Ryn had just changed jobs and had opinions about monitoring changes they hadn't been empowered to make at the previous one, so the talk was about that, and it was well received. Ryn's reasons for speaking are to share stories as a way of helping others learn, and it helps a career to be an established speaker. Conferences also led to Etsy, and to meeting Bridget at DevOpsDays New York, where they were both giving lightning talks.

How Organizers Choose

Ryn is involved with DevOpsDays New York, which anonymizes submissions to reduce unconscious bias, then rates them and discusses as a group, leaning toward newer speakers. "I don't want the DevOps community to become an echo chamber where we just hear from only the same people over and over again." Bridget says that is why Bridget spoke at fewer DevOpsDays in 2015 than in 2014. Ryn says some conferences are invite-only, which can produce a diverse lineup or a group of friends who look just like the organizer. Bridget adds that organizers have to encourage participation from communities who haven't attended. Ryn wants DevOpsDays New York to include more of the organization beyond dev and ops, and Bridget mentions a marketing speaker from Atlassian at Minneapolis.

Preparing

Ryn is slightly less nervous each time, and thinks a little nervousness shows you care. Preparation starts about three weeks out with a written outline, sometimes as long as a blog post. The Codemania talk drew on a Code as Craft post. Slides come a couple of weeks before, starting with five or six header slides for the problem, the solution and what was learned. Ryn's message to organizers: "I'm not going to give you my slides 2 weeks in advance. I'm a professional." They're still iterating the night before. Ryn rehearses three to five times, since more makes the talk boring, and an Ignite, which has 20 slides that auto-advance, gets much more rehearsal. Full-day trainings are the opposite, because so much depends on the students. Co-speaking works best when it feels like a conversation, and Ryn will present on Nagios at Velocity with a coworker.

Bridget shares that the shortest gap between giving the same talk twice was before and after lunch at a local conference. Bridget used the same deck and it turned into two different talks.

Getting Started as a Speaker

Ryn's first step is to work out what you want to say, since enthusiasm makes for a better talk than something to check off a list. On slides: "If you have enough words on your slides that it takes people more than like 2 seconds to read them, you have too many words on your slides." Bridget adds that corporate decks are different, and Ryn says a deck should still make sense after the fact, unlike some of their own decks that are "just cat pictures with no context."

Next is finding a conference that fits. Look at past lineups. A tutorial submitted to a conference with no tutorial track won't be accepted. Ryn recommends the Callback Women Twitter account, which tweets CFPs from a wide range of tech conferences, and the Technically Speaking newsletter, which also notes whether travel or lodging is covered. Bridget recommends trying a talk at a local meetup first, especially a lightning talk.

Writing the Proposal

As a reviewer, Ryn wants to know who the audience is and what they'll get, since the abstract often becomes the schedule listing. At a multi-track conference you have to say why someone should choose your talk, and at a single-track conference you should still think about who that conference's audience is and check past agendas for how-to versus advanced. Ryn loves 101 talks. "Talking isn't about you as a speaker. It's about the audience." Use the form fields, since they're there for a reason. Bridget, who reviews for Velocity, says vagueness hurts, and "don't leave this a mystery." Ryn suggests two to three paragraphs and a few bullets, since one isn't enough and more than three burdens reviewers, and to put detail in the field for organizers and leave a little mystery for the schedule.

For a speaking portfolio, Ryn keeps videos when a conference makes them available, and has a page on their website listing past events with links to slides on SpeakerDeck and embedded video. Bridget adds a high-resolution headshot, and a short bio saying who you are and what your background is.

What Conferences Owe Speakers

Ryn has strong feelings on this. Some conferences don't cover admission for speakers, which Bridget calls complete nonsense, and Ryn points out a talk can take 30 to 40 hours of prep. Conferences that say exposure is payment don't understand that exposure won't cover the rent. Ryn gives leeway to new or small conferences, but a five-year-old conference with too many sponsors to fit on a page that doesn't cover travel and lodging is "taking advantage of them." Bridget prefers to take speakers at their word if they say their company won't pay. Both like a low-key speaker event beforehand, and organizers sending information about where to stay and what to do, since radio silence leaves speakers overwhelmed. Ryn says "there's nothing more stressful to me as a speaker than having no idea what I'm walking into," and praises Bridget's blog post about what organizers should tell speakers.

Day of the Talk and After

Ryn prefers to speak early in the conference so they can relax, though Bridget notes speaking later lets you refer to earlier talks. Bridget recommends going to the other talks, so you know what has been said before you add your voice, and can reinforce or push back on other speakers. Bridget's summary is to "go to the other talks, kids."

Afterward, people come up to you, and Ryn's advice to those people is not to be condescending, since the speaker has considered the basics. For Q&A, Ryn sometimes times a talk so there's no time for questions and invites people to find them afterwards, since fewer people make statements when they don't have a captive room. Bridget will sometimes ask a "strategic softball" that lets the speaker elaborate, and as an organizer likes it when the crowd moves outside the room. Bridget also suggests a buddy who holds your things on stage, and says a partner who is an AV professional plugs in and packs up Bridget's laptop. Ryn likes a friend in the front row.

They both recommend putting your Twitter handle on every slide so people can live tweet you, and Bridget embeds the tweets on a page for each talk and has shared them up the management chain at work. Ryn admits to being too polite and Midwestern to retweet strategically, and Bridget says conference organizers love it when speakers publicize their talks. Slides go on SpeakerDeck, and usually as PDF without speaker notes.

Advice to an Earlier Self

Asked what they'd tell themselves in 2013, Ryn says "it's hard to A/B test your own life," but would spread the talks out, since five in two months is exhausting, and would be less nervous about giving technical talks. They gave mostly cultural talks partly from imposter syndrome. Bridget says you don't have to be pigeonholed, and that Bridget's 2015 was all Docker until a job change sparked a wish to talk about how organizations learn. Ryn says branching out gets you to other conferences, like Codemania, a development conference that was out of their comfort zone.

They finish with ways in that aren't conferences: internal lunch and learns, Etsy's weekly Ops School series, local meetups. Ryn's last word: "I used to be super intimidated by all the cool Etsy people who were up on stage talking at Velocity, and now I'm one of them. You can do it too." Ryn has committed publicly to only four conference talks this year, including a keynote at Continuous Lifecycle London in May.

• Effective DevOps by Ryn Daniels & Jennifer Davis

• Lara Hogan - Day-of-talk countdown

• Bridget - tl;dr: Your Talk is Accepted

• Ryn Daniels - On a Conference Speaking Routine

• Nekro Atsume shirts

• Draconian - Sovran (Oct 2015)

• Walls of Jericho - No One Can Save You From Yourself (March 2016)

• Lovemeow.com

Community Stuff

Upcoming conferences

For any devopsdays, try the code ADO2016! It should get you 20% off.

• DevOpsDays Rockies April 21st-22nd
• DevOpsDays Atlanta April 26-27
• DevOpsDays Seattle May 12-13
• DevOpsDays Salt Lake City June 14 - June 15
• DevOpsDays Silicon Valley June 24 - June 25
• DevOpsDays Minneapolis is July 20-21
• DevOpsDays Chicago is August 30-31

Open CFPs

• DOD Washington DC open until April 15
• DOD Salt Lake City open until April 19
• Puppetconf open until May 2
• Texas Linux Fest open until May 5
• DOD Amsterdam open until May 30
• DOD Chicago open until May 30

ADO Merchandise

We have t-shirts now! And mugs! They are available at store.arresteddevops.com! Only unisex for now, but more styles coming! Buy one today. Or not. We’re not the boss of you.

You Own It, But Not Just You

Bridget points to whoownsmyavailability.com, which keeps telling you that you do. Charity loves it as a gut check, but says it is also not only you. It's your team, your processes and the teams around you. Operations people tend to have a "hero-martyr complex," and Charity wants to say both that "You can't pass the buck" on a vendor or platform and that it isn't up to you to be a hero. Pete says blaming is comical, since the one guarantee is that "if it's online and on the internet, it's going to go down."

Matty notes podcasters have their own version, own your own RSS feed, since everyone loved FeedBurner, and it's a reminder that the service you depend on is somebody else's business. They pour one out for Parse, though Charity says it didn't fail for technical reasons and won't say more. Bridget was impressed that the Parse community pulled together on open source options. Charity says the same holds for Parse, Heroku and AWS: if an availability zone goes down and you chose to be single-homed, that may have been the right choice, but you can't say it's their fault.

What Happened With npm

Trevor reads from the npm blog: a package many projects depend on, directly or indirectly, was unpublished by its author in a dispute over a package name. Bridget's summary is that the namespace is global, and lots of sites were importing the package live, so when it disappeared they couldn't build. Go, PyPI and Chef Supermarket users have all had a version of this problem, which brings up Pete's rage-tweeted advice to vendor your dependencies.

Vendoring Your Dependencies

Pete says Threat Stack has a lot of Node.js, and it wasn't affected because they use Artifactory. Vendoring means taking a copy of a package, bringing it internal and serving it yourself. Every dependency adds risk. Pete vendors all Chef dependencies and doesn't talk to Supermarket, and does the same for full Debian packages such as Cassandra's third-party packages. Charity says Cassandra stopped publishing an older package they relied on, and they couldn't bring up new instances without upgrading the cluster. Pete says it was the exact scenario they hit: only the newest version was kept.

Charity adds a caveat about company lifecycle. At a three-month startup vendoring every package would be a waste of time, since availability requirements and time are both in different places. But there's a stage where these failures affect your ability to push code and roll back, and at that point you should have a local cache of every apt package, gem and npm package. Bridget recalls a broken Docker registry release pushed without changing the version number, which they worked around by copying the official registry container into their own account. Trevor says Jenkins only hosts the latest Debian package, and a release that broke Trevor's Groovy scripts forced learning to build a package feed.

Matty points out that the enterprises everyone teases for banning internet access from build systems didn't get burned, though not for availability reasons. Matty adds that the depth of dependencies is scary, since people use systems that depend on left-pad without knowing they touch Node. Charity's example is a first boot script that turns out to call half a dozen apt repos and gem sources, when the goal was just to bring up a host. When trying to fix a scaling problem, the last thing Charity wants is to "detangle somebody else's outage."

Bake It, Don't Bootstrap It

Pete is one of a few people who sign the packages Threat Stack ships to customers, and says there is "a high pucker factor" when thousands of systems can grab what you push. Charity's advice on resilience is to avoid reinstalling every package every time a node boots. Build a base image with Packer, bake in everything except the package that changes, such as Cassandra, and cache that in an apt repo you control, so if it fails you can blame yourself and fix it quickly. As Charity puts it, "you should have as few things that can fail while you're doing critical things as possible." Bridget's team at Drama Fever built AMIs with Packer from a Chef definition using Jenkins jobs, and a failed image build is annoying but doesn't affect production.

Pete says Threat Stack started with a ten-minute Chef run on every node, and only packerized the base when uptime and scale demanded it, so nodes come up in a minute. Charity's message on best practice is "Dude, it's contextual," since over-engineering early can kill a company as surely as neglecting things later. Pete adds that a launch date moved up to re:Invent forced choices, and the important part is going back to pay down the tech debt. Matty says you make a decision about acceptable risk with your eyes open, and revisit it as the business changes.

Decentralize Accountability

Asked what reduces operational risk, Charity says decentralizing accountability. In Charity's words, "your operations engineers honestly are not responsible for your reliability." Software engineers should own their services end to end, and ops should be in the first design meeting asking how it scales and how it's instrumented. The toss-it-over-the-wall model fails because "the feedback isn't effective if it isn't immediate and if it isn't somewhat painful." The goal is not never going down but limping along in a degraded state when a component fails, which Bridget calls circuit breakers and continuous partial failure.

Pete adds a security angle. Threat Stack monitors what systems do, and connections to Russia or China turned out to be apt repositories served over anycast. Vendoring cuts those random connections so an odd connection stands out. Matty warns you can overcorrect: if doing the right thing is hard, people will route around it. Matty quotes Sascha Bates that "if you treat your employees like children, they're going to behave like children." Pete says people will route everything over port 80 if it's the only port open, and Bridget recalls finding Comcast IPs in a production MongoDB security group that someone had added from home.

How to Vendor

Pete's starting point is an object store like S3. The Ruby gem deb-s3 pushes Debian packages to S3, which is basically apt repos on the cheap. Charity suggests wrapping package installs in your image build so they stash a copy in S3, and Bridget says to make it a step in the CI job, not a 42-item checklist. Pete points to paid options like PackageCloud, Artifactory and Nexus, and says "open source is only free if your time is worth nothing." Bridget suggests not depending only on public Docker Hub, having been paged more by Docker going down than by Bridget's own stuff. Charity says that if it's just caching package files, "that's seriously one of the easiest problems in computer science."

Pete raises verifying that the packages you get are the ones you expect and treads lightly around GPG. Deb-s3, Artifactory and PackageCloud all make signing easy. Charity has built Debian packages at every job, and says the packages that have to be built become the seed of the local repo. Matty says the tension is trust: you don't want to build everything, but vendors mean trusting someone else.

Open Source and Understanding What You Run

Charity says an agent that isn't open source is a non-starter if it runs on Charity's own hosts. When you get a stack trace, you need to read the code that generated it. Trevor likes walking traditional closed-source shops through Chef cookbooks and answering "what does this action do" with "you tell me." Matty adds that if a vendor goes out of business, understanding how it worked matters. Pete's example is an engineer who wanted to gem install Sensu plugins, when Pete preferred to bring them into a cookbook. A week later a chunk of the code turned out to be dead code that would have confused them if they had just installed it. Bridget says you want a good picture of what normal looks like before 3 a.m., and Pete says every dependency adds risk that some organizations can absorb.

Charity offers a cautionary note from repeatedly moving the source of truth from an untrusted repo to GitHub, "and then you have 2 problems."

Managing Up

Matty asks about bosses who want someone to blame or sue. Pete says you sometimes have to play the game and call it a security issue, recruiting the biggest curmudgeon to hammer on it. Matty says SLAs are a lawyer's favorite thing to say, since they're made up and never apply. Charity says it's about audiences. Developers trust you when you give the dirty details and take ownership, while executives report to people who don't want to understand software. Translate for them: paying for something and getting an SLA won't reduce outages, and probably means more of them because "we won't be able to debug them." Charity says to translate that into corporate-ese, but it will build a better product.

Who Owns Your Availability

Pete's closing point is that everyone's stage, risk tolerance and budget differ, and "you don't have to listen to what talking heads on the internet say" about vendoring every dependency. Use each crack in the mortar as a chance to make things a little more durable. Charity says the answer is "it's you, but not you personally": a successful culture is one where every person feels they own it, and it's not some other team's, a DBA's or a vendor's problem. Bridget notes the subtlety in English: "is you singular or you plural?" And the answer is yes.

• kik, left-pad, and npm
• Who owns my availability?
• deb-s3
• T-shirts and mugs

Community Stuff

Upcoming conferences

• DevOpsDays Rockies April 21st - 22nd - ADO listeners, save 10% off regular price with the discount code ADO2016
• DevOpsDays Atlanta April 26-27 - ADO listeners, save 20% off regular price with discount code ADO2016
• DevOpsDays Seattle May 12-13 - ADO listeners, get 15% off with the discount code ADO2016

Open CFPs

• DOD Vancouver and MSP CFP and Abstractions open until March 31
• DOD Washington DC open until April 15
• DOD Salt Lake City open until April 19
• DOD Amsterdam open until May 30
• CFP for That Conference Opens March 1- 31

What a Personal Brand Is

Michael says a personal brand is "what people say about you when they go to lunch and you're not there." You don't control it, and you can't compile it or run tests on it, but you can influence it. Andrea puts it as what colleagues would tell someone about how to get something done through you, whether you are collaborative or prefer to work alone. Asked to distinguish it from reputation, Michael says they're very similar, and Andrea adds that reputation is built from past experience and shapes how others play back your brand. Michael's point to technical people is that "everybody gets a brand," and it isn't only for sales and marketing.

Matty says it makes you a more effective change agent, and asks whether it's unfair that Matt gets listened to more because people think Matt is an expert. Michael's answer is a story about a colleague who dictated a policy that Michael complied with outwardly but didn't follow. If people feel you're a jerk or that they aren't heard, the parts of the organization most resistant to change win, because they can say you live in an ivory tower. Matty ties it to compliance versus commitment, and Michael compares it to going upstream or downstream in a river.

Fitting the Culture, and the Technical Person's Blind Spot

Andrea says how you get things done is as important as what gets done, and you have to manage up, down and laterally. Andrea stresses congruence with the company's culture, and describes Beam Suntory as fast-moving, which suits Andrea as a fixer. Michael says technical people start out with no social expectations, get rewarded for being loud and right, and are often surrounded by managers who cover for their awkwardness, so "behind every technical person who thinks that they're immune from those things is a manager getting a big, fat bonus."

Andrea describes feeling the opposite burden, leading a social media team, and tells of a Predictive Index personality test in which Andrea was the second most extroverted person at the company, behind the woman who runs the distillery in Clermont, Kentucky. Andrea suggests IT people might learn from marketing counterparts how much the how matters.

The Go-To Fixer and Internal Versus External Brand

Trevor asks about the person whose brand is the one you throw every problem at. Michael says they secretly like it, and "you cannot get out of your real brand without looking fake," so if that is who you are, embrace it, maybe ask for a raise, and if you don't want it, make changes at the action level. Andrea describes a colleague who broadened the role by swinging by Andrea's desk and asking what was important to Andrea, until the colleague was working across marketing, sales and legal.

Matty notes that in consulting and customer-facing work, Matty and Trevor think about external brand more than internal, whereas a big Twitter following retweeting Michael's Ansible opinions wouldn't help Michael inside NCR. Where your brand is consumed matters, though Matty doesn't think it is either/or.

Finding Your Brand: Data and Cake

Trevor asks how you work out your brand. Andrea says you need data: self-awareness, feedback from people you trust, and tests like the Predictive Index or the diagnostic from Sally Hogshead's How the World Sees You. Michael brings up Branding Pays by Karen Kang, where a brand is cake and icing. The cake is your rational value, like being a Chef expert who is good at sales. The icing is how people feel when you're around, and people go for both. Andrea sums it up as "the what and the how." Trevor points out you can be great at .NET and still be someone nobody wants to work with.

Michael's caution is that if you're the jerk, nobody tells you, so self-awareness has to be iterative. You find one person who will tell you and show them you'll change. Matty links to a blog post called Empathy Is CI for the Soul and tells of a colleague who asked whether Matty wanted direct feedback after a proof of concept. It was hard to read, and part of it was that Matty talks too much, which the other three on the call agreed with.

Authenticity, Results, and Humility

Matty raises imposter syndrome and the worry of seeming to brag. Michael thinks the branding metaphor is unfortunate, because people picture someone full of themselves. What Michael wants people to say is the truth, and "the truth only comes about by results." Michael has been working on going from the person who is always right and always talking to one who listens, lets others have the idea and facilitates. Andrea adds that being authentic means you needn't fear being called braggy, and describes doing a guest lecture at a university about once a quarter and not keeping separate work and personal social accounts.

Michael tells three project stories. The first involves TeamCity, which Michael introduced years ago, feeling a duty to help anyone get on it, even meeting people at 2 a.m., and it became what hospitality at NCR uses for continuous integration. On another, a multi-product initiative, Michael ignored a lack of buy-in and pushed through on technical prowess, leaving half the people unhappy, and it felt like swimming upstream. For the configuration tool search, Michael wanted something people would adopt on their own merit, and when Michael admitted to Matty not knowing how they'd decide, Matty said it would be 45 days. Michael hung up thinking it was impossible. Chef has been like TeamCity, with people coming to Michael out of the blue with Test Kitchen questions.

The Fear of Being Wrong in Public

Trevor says the terrifying part is looking vulnerable in community Slacks and GitHub, where Trevor is a Chef expert for a consultancy. Even at 90% certain, Trevor fears someone will call the answer wrong and cost credibility. Matty has struggled with it too. Joshua Timberman told Matty that Chef is too big for anyone to know everything, and that Adam gets things wrong too. If you answer only when you're 100 billion percent sure, you damage your brand because you stop talking. Andrea says being an expert means being endlessly hungry to learn, and Michael tells Trevor the cake isn't the whole of Trevor's value.

Matty says as your knowledge grows you raise the bar for what counts as table stakes, and tells of being corrected in a Slack channel, recoiling, and then realizing it was an opportunity. There was some "writing to the lurkers" happening, and beginners won't have to say the stupid stuff later because an advanced person said it for them. Trevor says answers come freely in a room but the public setting raises the discomfort. Michael says people "really hate a know-it-all," they feel belittled, and Michael has been trying to clamp down on the urge to answer every question.

Practical Tips

Matty's first tip is to share what you know and bring others along instead of reciting what you've read, so you're cemented as the person who knows about that thing. Andrea suggests guest lecturing, industry events where the real value is networking, and being the same person on and off the clock. Matty says if you don't like public speaking, that's fine, and if you do, anything you have to say is valuable, and that meetups and DevOpsDays hallway tracks can connect you with people who have done the migration you're considering. Andrea says it's okay to be deliberate about networking, for example setting a goal of three unprompted hallway conversations a week, as long as you're genuinely interested in the other person. For the virtual version, Matty suggests asking on Stack Exchange or the HangOps Slack before calling the vendor.

Thought Leaders and Opinion-Havers

Matty admits the title may sound like snake oil, and says you shouldn't decide to become the foremost authority on testing Go code so you can keynote GopherCon. You accentuate something you already believe. Michael says people know who the fakes are, and imagines the business dinner where their name comes up. Andrea says "real thought leaders don't consistently call themselves thought leaders." They are "humble enough to know that they need other thoughts to lead any thought of their own." When people call Andrea a guru, the reply is that Andrea is passionate and excited to learn, along with a suggestion to look at the data together. Matty's version: "I would say I'm an opinion-haver." Trevor closes on the difference between wanting to be a master at something and wanting to talk about being one.

Empathy is CI For The Soul by David Shackelford

Community Stuff

Upcoming conferences

• DevOpsDays Rockies April 21st - 22nd - ADO listeners, save 10% off regular price with the discount code ADO2016
• DevOpsDays Atlanta April 26-27 - ADO listeneres, save 20% off regular price with discount code ADO2016
• DevOpsDays Seattle May 12-13 - ADO listeners, get 15% off with the discount code ADO2016

Open CFPs

• DOD Vancouver and MSP CFP and Abstractions open until March 31
• DOD Washington DC open until April 15
• DOD Salt Lake City open until April 19
• DOD Amsterdam open until May 30
• CFP for That Conference Opens March 1- 31

Check Outs

Andrea

• Maker’s Mark whisky
• How the World Sees You: Discover Your Highest Value Through the Science of Fascination by Sally Hogshead
• Setting the Table: The Transforming Power of Hospitality in Business by Danny Meyer
• The Life-Changing Magic of Tidying Up: The Japanese Art of Decluttering and Organizing by Marie Kondo

Michael

• Branding Pays - a great book about personal branding
• Mr Money Mustache - frugality
• You Need a Budget - budgeting

Trevor

• Venture Brothers Season 6
• WMF 5.0 RTM again
• Hangops Slack

Matt

• How Google Works
• Visual Studio Code
• The Newsroom on HBO GO, Amazon Prime
• Someone tell me how to use OmniGraffle (future checkout?)

How GitLab Started

In 2011, Job says, a PHP developer in Ukraine named Dmitry wanted the company to move to Git, but it wouldn't allow code on GitHub off-premises and there was no good alternative. So Dmitry built one in Ruby on Rails, working through the night after the day job. Job likes to mention that Dmitry's house had no running water, so whenever water was needed, Dmitry walked about 100 meters to a well with a bucket, and kept developing GitLab in between. Matty wonders what water-bucket-driven development did to the early commit patterns. Dmitry put the project on GitHub, where the programmers were, and it gained traction.

Around 2013, Sid, a Dutch guy whose real name is Sietse, thought it would make a good SaaS and told Dmitry that gitlab.com would launch without Dmitry's involvement. Dmitry said go for it. A few months later Dmitry tweeted about wanting to work on GitLab full time, Sid saw it, and they started a company together.

Omnibus and Time to First Delight

Matty's first encounter with GitLab was for a customer who couldn't put code on GitHub.com and was stuck on an ancient version of AccuRev, and the Omnibus install, the Chef packaging, won Matty over. Job says GitLab hadn't always used Omnibus. Before that, installing a Rails app meant a manual with at least 12 involved steps, and switching to a one-command install made downloads go up "like 1,000%." Job credits a good part of the project's success to it. There is now a package repository so you can just apt-get install gitlab. Matty calls it time to first delight, and Job says for a developer-oriented product, "it has to be extremely easy to use," installation included.

What an Open Company Means

Job says GitLab does in the open everything it reasonably can where the community benefits. All development is public, from an idea or bug report through code review, merge and release, on the public gitlab.com instance. Then they opened the company handbook, a website of markdown files at about.gitlab.com/handbook where every page links to its file in the repository, and later support and operations. Product direction is open too. It lives on a direction page, not called a roadmap because they don't want to commit to dates, and in the public issue tracker where all of Job's work happens and feedback from customers and the community arrives.

Matty argues that the list of things you can't share is shorter than you think, and asks how to move an organization that way. Job says there was a lot of internal resistance, including Job's own. Customers rarely want to be named, so a support ticket becomes a public issue with the name sanitized and a link to the private ticket, labeled as a big or medium customer. It adds some process, "but the positives massively outweigh" the negatives. Being open also means saying in public that you don't think a feature is a good idea and being open to hearing why you're wrong: "We don't want to be right. We don't want to be authoritative. We want to build a really good product." Matty says the worst feeling is being ignored, and it's better to hear no in a discussion. GitLab doesn't share revenue or salaries, which Job says wouldn't be immediately valuable to the community, unlike the things they invite people to contribute to.

Outages in the Open

GitLab.com is free so they have a good place to load test, but it runs as production, with the team's own work on it. In 2014 it had a brownout of many hours, when there were about six people, all engineers in Europe, and part of it happened while they slept. They opened a Google Doc to discuss it and then decided to share it, put a link on Hacker News, and got a positive response. People said they had more faith in them after seeing the team work hard and disclose what was happening. It wasn't the moment they opened everything, but it led up to it. The site later carried a message saying it was slow and had downtime but the data was safe.

Job loves the GitLab status Twitter account, run by the DevOps engineers, who share their emotions, and says "I think it humanizes it." Matty read one saying that because GitLab.com has three single points of failure, they expect it to be unavailable at some point, and Job says they are now down to one. Matty connects this to HugOps and to blameless culture, recalling Charity Majors telling an engineer who had gone eight months without breaking production to step it up. Job says being visible online has never been a regret: "There has not been a single situation that we regretted being very present online."

Growth and the Release Train

Job joined at the beginning of 2014 with six people, four of them engineers. A year later there were nine and they had been through Y Combinator, and now there are about 50, about half engineers. Dmitry released the first version on the 22nd of the month and GitLab still ships on the 22nd every month. Job calls it the release train and announces in Slack, "choo choo, the release train is going." They've done it 51 times and are going for 52 without fault.

Dogfooding and Saying No

GitLab tries to use GitLab for everything, and when it hits a wall it builds the feature. Job's example is folding an external feedback tracker into the internal issue tracker, which gained a voting feature but then had thousands of issues instead of hundreds, so they look for ways to improve the product for that. Matty asks when a vendor should stop and integrate with something else. Job's example of a firm no is permission management on specific directories, which SVN migrants ask for. Because a clone contains the whole history, "we're not even going to try this because this is simply not the way Git works." Matty says that's cruel empathy: if your old way worked, you wouldn't be shopping.

Community Contributors

Job says there are about 1,000 contributors, from a single commit to hundreds. Some of the best features came from the community, like the button to merge when the build succeeds, contributed by someone who was then offered an internship and is now a developer, and the fuzzy file finder. Customers contribute too. CERN, a customer, built several Enterprise Edition authentication features and worked in the open on issues and merge requests.

This year GitLab has at least one developer whose full-time job is to coach incoming community merge requests, finishing abandoned ones if needed, and an "up for grabs" label for small, quick issues that a first-time contributor with a little Ruby or JavaScript can tackle. Matty notes the parallel with the Phil Dibowitz episode about starting in open source.

Relevant Links

• GitLab Operations
• GitLab Status Twitter Account
• "Because http://GitLab.com has 3 single points of failure we expect the service to be unavailable at some point."
• How GitLab learned to be open

Check Outs

Job

• iTerm2, version 3 Beta
• relay.fm

Matt

• http://10x.engineer/
• flowstate - $9.99 in Mac App Store

What OpenStack Is

JJ's 50,000-foot view is that OpenStack is a private cloud you build yourself, from database as a service and compute to imaging, object storage and even container clusters. It started between Rackspace and NASA, with NASA's front end to Nova for spinning up VMs and Rackspace's Swift for object storage. It has since grown to about 19 projects on a six-month release cadence, which JJ says is a challenge for operators, and you can pick and choose which ones you use. The core ones are Keystone for identity, Glance for images and Nova, with Neutron the subject of debate about whether it counts as core. JJ says OpenStack recently got nonprofit status in the US under the same tax code as the NFL and churches, so contributing can be described to your employer as charitable work, which Matty calls joining the Church of OpenStack.

Why Not VMware or EC2

JJ says if you care about owning your data in your data center, OpenStack is an open source way to build a cloud instead of paying a VMware tax. At scale, VMware "gets prohibitively expensive around the 10,000 hypervisor mark," and JJ would rather spend that on hiring an engineer you can train than on a license and support contract. It is also good for dev and QA, where a machine sitting idle on EC2 for a year costs $1,200, and reclaimed hardware can run the same APIs as EC2 locally and get more life out of its depreciation.

The Hard Part Is the Mindset

According to JJ, "9 out of 10 times, it's teaching a company to go to the cloud." At a previous employer, they spent money and effort on an OpenStack infrastructure, but the idea that a misbehaving machine gets rebuilt from scratch never took hold, since the company couldn't accept ephemeral machines. JJ's warning is "OpenStack is not free VMware." Handing someone who has spent their career on vSphere an OpenStack cloud is like putting a lifelong Windows user in front of a FreeBSD box: they'll get by, but with a long learning curve. Matty compares it to Windows being API-based and Unix file-based.

The Microsoft footprint is surprisingly high, JJ says. The main hypervisors are KVM and QEMU, there are Hyper-V ports, JJ knows of production Windows 2012 R2 boxes, and one company, cloudbase.it, built its business on Windows images for OpenStack clouds.

Operators and OSOps

JJ says OpenStack has three camps: developers who commit code to OpenStack, users JJ prefers to call consumers who just want an API endpoint for a Test Kitchen instance or a Redis server, and operators who run the clouds. Operators have been underrepresented for a long time. The Foundation responded with mid-cycle operator meetups, the next in Manchester, UK, the first outside the US. Operators also usually can't get an ATC, the Active Technical Contributor status that earns a free summit ticket worth $600 or $700 for anyone who has contributed in the last 12 to 18 months, because their bash, Python and Ruby scripts don't go into official projects. OSOps is a place for operators to share those tools, with the goal of getting them ATC someday. JJ says the Foundation is happy with how it has taken off, and is considering making it a core project, since you need people to run these clouds.

Getting Started

JJ says to run away from DevStack, which "has grown into a monstrosity" and won't teach you what you need. First figure out what you want to build, because saying you want an OpenStack cloud gets you choice paralysis. Projects like OpenStack Chef and OpenStack Model T, along with some Puppet and Ansible builds, let you build small clouds to learn. Commercial options include Mirantis and Blue Box, now IBM, which are useful, but you are handing off understanding of the cloud to a third party, no different from asking an MSP. JJ comes from the camp that if you want full control, you should understand how the whole thing works, though the engineering time may be cost prohibitive, JJ admits.

OpenStack fits with other technologies: apps.openstack.org has a one-button push to pull a Glance image and build a Kubernetes cluster, and a project called Magnum uses Heat to spin up CoreOS machines with Fleet and etcd and points your API endpoint at the cluster so you can use Docker as usual.

Running for the Board

JJ is running for the OpenStack Board of Directors, and Matty says JJ would get a vote from Matty if Matty had one. JJ's argument is that board members have been removed from day-to-day OpenStack. JJ uses a story about pioneers, town planners and city builders. CIO magazines and Foundation blog posts make OpenStack sound like it is in the city-building phase, but JJ believes "we are just getting out of the pioneering phase and just getting into the town planning phase." JJ sees apps.openstack.org as the general store, a milestone, but says features get pushed into releases before operators have adopted the previous ones. JJ's example is that people are still arguing about basic layer 2 and layer 3 networking while vendors fund edge routers inside SDNs. JJ wants a seat to ask why, and to see "how the sausage is made." Who has the right to vote isn't clear to JJ either. JJ had it last year but not the year before, which is how Matty got to Chicago politics.

OpenStack Model T and the Chef Project

JJ is proud of OpenStack Model T, an opinionated build of OpenStack in one Chef cookbook, which automates the long OpenStack install guide, with RabbitMQ as the queue and Ubuntu as the base OS. Run one recipe on a controller and one on each compute node you add, and you have a horizontally scalable cloud on reclaimed hardware. JJ hopes to give a ChefConf 2016 talk on a reference architecture bootstrapped with a cookbook called Pixie Dust and tested with Test Kitchen on bare metal.

The OpenStack Chef project has cookbooks for all the major projects, and a small team that needs more help. It is in the middle of a refactor because the last two cycles added everything including the kitchen sink, which JJ thinks scared people off. With a 16-gig MacBook Pro you can build a multi-node OpenStack cluster with Chef Provisioning and Vagrant, and JJ used the same approach on a donated 96-gig quad-core Xeon to build a production-ready all-in-one cloud in about 45 minutes, which is still running. JJ's parting message is that operators are trying to come together, and if you have a tool, drop it into OSOps and, once ATC is possible, you'll get it.

• OpenStack OSOps
• Check out this tool library for OpenStack operators
• OpenStack-model-t
• Using Automation to build an OpenStack Cloud
• OpenStack-Chef project

Check Outs

JJ

• This War of Mine
• Labyrinth Black Ale

Matt

• Asphalt 8

Using DevOps to Sell DevOps

Michael says the Seattle training was built around value stream mapping and Kanban, so the sales engineers would have some DevOps fundamentals as tools for conversations with customers. Trevor asks what value stream mapping is, and Michael explains that you draw a current state map of how a piece of work flows through a process and where time is spent, then build a future state map that removes the waste. It comes from lean manufacturing, and Gene Kim is a proponent of it in The Phoenix Project. Michael has done enterprise software sales for eight years, mostly on the presales side at BMC, Instratius (which Dell acquired) and Chef, and now manages Chef's East presales team. Michael wants to be less of a demo jockey and bring more value to the conversation.

Five Kinds of Salesperson

Bridget says the golf-and-wine stereotype doesn't match the field people encountered at Pivotal. Matty introduces The Challenger Sale, which describes five profiles: the hard worker, who makes the calls and follows the process, the lone wolf, who ignores the process and closes deals with a Rolodex, the relationship builder, the problem solver and the challenger. The problem solver says "tell me where it hurts and I will sell you the Band-Aid," while the challenger teaches you something about your business you didn't know and acts as a trusted advisor.

Bridget pushes back that you need a relationship before you can challenge anyone. Matty's answer is that the relationship builder means you buy from me because you like me, and Michael adds that many CIOs buy from the same salesperson they have bought from for 20 years. Trevor, as a principal consultant, lands in the problem solver role, and says Matty is probably half problem solver and half challenger. Michael says you have to wear different hats depending on the customer, since some know their problem and some have read about DevOps and called the DevOps company to buy it.

Eight Units of DevOps

That leads to a recurring question: "do you want to sell them 8 units of DevOps today, or do you want to sell them 4 units of DevOps over the next 15 years?" Bridget says you sometimes have to give people what they need, not what they want. Matty explains that with a subscription model, selling a customer something they aren't ready for creates a churn customer who will cancel in a year.

Selling in an Open Source World

Michael contrasts the proprietary days. The customer knew little about the software, the documentation was behind a login, and a $2 million deal meant convincing everyone through executive relationships. In the open source world, the customer is often smarter about the software than Michael is and asks why they should pay Michael. The sale runs through the individual contributors who sell it upward to the economic buyer. Bridget adds that if those people can't look at your code on GitHub they will veto it, and Michael says you're also selling the community, which gives customers other people to share the pain with, where in the proprietary world you're "always angry at the vendor for not giving you what you need." Bridget adds that customers can choose among vendors, since other companies also help with Chef and commercial Cloud Foundry.

Matty describes r/sysadmin complaints about vendors' cold calls, with SolarWinds getting the worst reputation, and the flip side: when you do your own discovery, there's a lot you don't know to look for, which is the advantage of a trusted advisor. Michael says one reason for leaving BMC was the constant wondering "what are you trying to hide" when documentation was behind a paywall, and recalls an executive saying the customer shouldn't be able to download the software because it's enterprise software and complicated. Bridget says that at Pivotal, missing public information is usually because nobody has had time, not malice, and describes a ThoughtWorks person who found an error in Pivotal's docs and was pointed to the GitHub repo to submit a PR.

Good Vendor Relationships

Bridget says that as a Chef customer at Drama Fever, Bridget happened to run into Julian Dunn, who was doing product management for the thing Bridget was unhappy about, and who told Bridget where it was on the roadmap, with no secret handshake. Matty's example is a rep from Serena Software on a small deal of about $50K, who talked about the roadmap and told Matty that Matty wasn't ready for something yet, so they'd get Matty there. It paid off: they later partnered in a big way, and Matty says a hard sell on the problem as described would have gotten them barely anything.

Proof of Experience

Michael says Chef used to run proofs of concept the traditional way, with experts setting up the tool, writing the customer's use case and demoing that the software solves the problem. They realized that was broken, and now "it's more about can the customer solve their use cases with the software." Matty recalls the Fast search POC at Apartments.com, six weeks with the vendor's engineer building everything. If Matty came in and wrote Chef code to install WebLogic, "it proves that Matt can write Chef code," which doesn't help you because Matty doesn't work for you. Matty calls it a proof of experience, a time machine to see what it will be like, and says it is physically and emotionally exhausting and Matty's favorite part of the job. The sign that it is working: "when the stickers go on the laptop, you know the deal's on its way."

Michael says the point is to challenge expectations in enterprises that say they could never work that way, by having people do it. Bridget adds that if only one part of the organization is in the room, the parts that are always at war aren't, and you sell them the units and they don't use them, so you have to get everyone in the room, since you can't just buy a tool. Michael quotes Adam Jacob: the tool reinforces the culture and the culture reinforces the tool.

Everyone Is in Sales

Matty quotes Nathan Harvey: "everyone's in sales in your organization." Bridget found a 2014 tweet of Bridget's own for a presentation to a sales organization at a quarterly business review, "DevOps is culture, princess. Anyone who tells you differently is trying to sell something," and Andrew Clay Shafer's reply was "everyone is selling." Michael brings up Daniel Pink's To Sell Is Human, which says one in nine Americans work in sales and the other eight are selling something too. Michael says the presales team teaches value stream mapping, Kanban and pair programming during the POC, and then at the end tells customers that what they just did is foundational DevOps, and that they had no idea the presales team had done "the DevOps on them." Bridget adds that they did the DevOps for themselves.

Closing thoughts: Matty, who never expected to be good at sales, doesn't think wanting to change the world and being a good salesperson are mutually exclusive, and asks listeners to skip the assumption that everyone is selling snake oil: "Assume positive intent." Bridget, who as a sysadmin was lied to about IOPS, is humbled by how much customers trust them, and says sometimes the right answer is to stop hating each other and talk, which is what Bridget calls "DevOps therapy."

This was a special co-production with The Goat Farm. Don't forget to subscribe to their Enterprise DevOps podcast on iTunes or Stitcher!

• The Challenger Sale In Less Than 10 Minutes
• “To Sell Is Human” - by Daniel Pink
• Value Stream Mapping

Check Outs

Michael

• "I’ve discovered that tweeting something inflammatory about Docker will give you a really popular tweet."

Bridget

• PetFinder.com - searching for that adoptable pet of your dreams

Trevor

• Bryan Fuller co-producing the new Star Trek TV series
• posh-git

Matt

• git-blame-someone-else
• New DreamWorks/Netflix Voltron series title announced - Voltron: Legendary Defender
• Snarky Agile Tees

Upcoming Events

• DevOpsDays Rockies is offering a 10% discount for ADO listners with the code ADO2016

Upcoming CFP's

• DevOpsDays Rockies and DevOpsDays Seattle CFP open until February 28th
• ChefConf CFP open until February 29
• DevOpsDays Atlanta CFP open until March 1
• Dockercon CFP open until March 18
• Platform/Spring One CFP open until March 24
• DevOpsDays Vancouver and DevOpsDays Minneapolis CFP and Abstractions open until March 31
• DevOpsDays Washington, D.C. open until April 15
• DevOpsDays Salt Lake City open until April 19
• DevOpsDays Amsterdam open until May 30

Where Chocolatey Came From

Rob was moving from company to company and repeating the same silent-installer work each time, and wanted to make the concept more global so the wheel wouldn't need restarting, including when working outside the company firewall. The goal was to be able to sit down to pair with someone who didn't have Notepad++ and fix it quickly instead of doing "the nice long yak shave." At the time Rob was on the NuGet team, which had decided not to approach machine package management. They joked that if they ever built one they'd call it Chocolatey NuGet, since it wasn't vanilla NuGet packages, and the name stuck.

The Community Feed and Trust

Rob's one clarification is that Chocolatey is a decentralized package management tool, and in production "you definitely don't want to be depending on the community feed." There is little control and very low trust there. Trust is something they are building, and package signing is the next big step, giving traceability from who you think created a package to who actually did. Companies can and do use Chocolatey by creating their own packages, hosting them on an internal repository, and never touching the internet. Community feed packages typically point to a binary at an official distribution point, with a checksum (many packages have one, some still don't) and the silent arguments to install or upgrade it.

Underneath, Chocolatey works through the NuGet packaging framework plus automation scripts, which are currently PowerShell, with Script.cs support to come. Script.cs is a C# scripting tool with a REPL that runs on Mono, which Trevor finds exciting.

The Rewrite

Chocolatey started as a command line app written entirely in PowerShell, which meant starting PowerShell on every command and paying a one to two second startup penalty. It had about 20-some thousand lines of code and became interesting to maintain. For maintainability, speed and the possibility of going cross-platform, Rob decided to take the pain of a rewrite in C#. It was in beta for a while and came out that March. Rob says the goal is to make Chocolatey look like what you would expect from a Linux package manager, and that there is more work to do.

Hosting Your Own Packages

Trevor asks how a company mirrors community packages when they point at external download sites, as when Trevor's Notepad++ install broke. Rob says you currently have to edit each package, a process called repackaging. The business version coming next year will make that a single command that fetches the package and its downloads, puts them internally, and rewrites the package to point there. The paid versions will also add virus checking and, for pro users still on the community feed, an alternate private CDN so a vendor's 404 doesn't break the install.

Moderation, the Verifier, and the Validator

Rob says a free service still costs money, and after three or four years of running the site, it came time to decide whether to keep pouring money into it or find a business model. Rob ran a Kickstarter last year, which succeeded. Moderation was introduced in October 2014, and before that a package went live the moment it was uploaded. It was popular but they probably turned it on too early, before the infrastructure automation was ready. Some maintainers push up to 200 packages at a time, so automation on one side and human review on the other doesn't work very well.

A couple of weeks earlier they released the verifier, which checks existing packages every two weeks to see if they still install, and emails the maintainer if not. On new submissions it runs headlessly through Vagrant, installs Chocolatey, sets up a sandbox, installs the package, and checks the uninstall too. Chocolatey takes a registry snapshot on install so it can uninstall automatically, which means about 80% of packages need no uninstall script. The validator checks quality and consistency against about 35 rules, such as naming guidelines, where an icon is hosted, and use of Start-Process, and posts notes on the package page. Only when a package installs cleanly and passes validation does a human reviewer step in, mostly to read the install script and check that the download comes from the official distribution point. Rob says the messaging is not being reported back to users yet, because they want it short and rock solid.

Trevor asks about malicious packages. Rob says they haven't found one, but "it only takes one." They have seen packages that use SourceForge, which pulls down malware. To see more of what an install touches, they plan to add Turbo, a tool formerly called Spoon that can diff a container to show every registry key and file touched.

Compared With Linux Package Managers

Rob says Chocolatey lacks virtual packages, so a package can't depend on "a PDF reader" and have Adobe or Sumatra satisfy it, and it lacks concepts like replaces or conflicts. It also has no package indexes, so it hits remote systems whenever you query. Checking for upgrades took about four minutes in the old PowerShell version and about 40 seconds in the C# version, which is still too long, and with indexes it should take under a second. It has a CLI and a GUI, called Chocolatey GUI, which Trevor suggests could be renamed.

Growing Pains and Getting Involved

Microsoft's OneGet is a package manager aggregator that talks to other package managers, and the Chocolatey provider for it is unfinished, since the few volunteers working on it are stretched thin. Rob wants help from people who know PowerShell and C#, and would like it done in the first half of the year. Rob is also dealing with growth: downloads went from about 5 million total before moderation to, by Rob's account, another 20 to 25 million in the following year, and the site handles about a terabyte a day and 4 to 7 million requests. Approvals have slowed, and "we've sort of failed, that we didn't get the automation in place as quickly as we needed to." Rob works on stability problems on personal time, sometimes taking PTO, because as the guy behind Chocolatey, Rob hears about outages.

To get involved, Rob points to the mailing list and Gitter, taking over a package whose maintainer has disappeared, or the Up for Grabs items, which are often small and documentation-related. Rob says a longtime user once said "once you go chocolatey, you don't go back." Puppet has a Chocolatey provider and Chef has a cookbook, and there is more coming for offline use. Rob also likes the open source business model, since the paid version keeps driving features into the free one. Rob's closing plea is for better documentation, ending with: "Chocolatey, it's awesome. Use it."

Check-Outs

Rob's pick is Turbo, and another pick slipped Rob's mind, so Trevor jokes it will be posted as Rob's latte thoughts. Trevor recommends playing with DSC and Windows Management Framework 5, Agents of S.H.I.E.L.D., and David Bowie's new video Blackstar. Rob adds that both Puppet and Chef have DSC providers, and that it is a sign of Microsoft doing things differently and more openly.

• Chocolatey.org
• Chocolatey Mailing List
• Chocolatey on Gitter

Phil's Background

Phil has spent a couple of years building a pattern at Facebook for teams to own their own tiers of machines. Phil's team wrote low-level core cookbooks that expose an API on attributes, so setting a sysctl or adding a cron job is a variable assignment in your own cookbook, with no need to understand every kernel tunable or every other cron job on the system. They deliberately used templates and notifications so that deleting those lines from a cookbook cleans everything up. It took a couple of years to get everyone migrated. Along the way Phil got deeply involved in the Chef community as a maintainer, most recently adding multi-package support, and mentions internal Facebook tools people have found useful, like Taste Tester and Grocery Delivery.

Phil will hit five years at Facebook in November. Before that came Google, working on Gmail infrastructure, and before that at Ticketmaster, where Puppet and Chef didn't exist yet, so a group of them wrote their own configuration management system called Spine. It was very cool for its day, Phil says, but it never had a community and wasn't maintained, and the industry had moved on by the time of Phil's return to it.

Open Source Because It's the Faster Way

Matty describes the odd alternate universe at ChefConf, with Mark Russinovich and Jeffrey Snover on stage talking about running Linux on Azure, and says it feels like Michael Corleone: every time the Windows chapter looks closed, they pull Matty back in. Matty asks how open source has evolved over Phil's career.

Phil got into the industry young by hacking on open source as a kid, and when interviewing at Facebook wanted to be able to open source what they built. Phil sees a shift since the late '90s, as Linux, Apache and the browsers showed open source would not only work but be better. What changed is that companies no longer do it for moral reasons: "the best way to move fast is to let everyone else help you move fast." Phil's example is the HipHop PHP to C compiler, which Facebook released and tried to build a real community around, including engaging the Zend community, so a lot of the work now gets done for them. It is also why nobody at Facebook wanted to write a new config management system when good ones were being maintained.

On Microsoft, Phil says it was lip service for a long time, but thinks large parts of the company have bought in and finds it exciting. Like Chrome and Firefox, each side makes the other better, and Phil doesn't expect Linux and Windows to merge. Matty adds the open source that doesn't accept PRs as the lip-service version.

Plumbing Isn't Your Competitive Advantage

Matty says enterprises used to keep quiet about how they worked, so the only stories were from Facebook, Netflix and Etsy, which is where the unicorn idea came from, and clients would say Matty was the fifth person that day to tell them they should be like Netflix. Matty credits the DevOps Enterprise Summit for getting companies like Target, Nordstrom and GE to talk. Most of what gets shared is plumbing, not competitive advantage. Netflix doesn't open source its video encoding, and if someone is going to disrupt Facebook it won't be by being better at building 300,000 servers fast. Phil calls it infrastructure scaffolding.

DevOps Isn't New

Phil wants to comment on the history. As a junior admin in the '90s, Phil asked a mentor what made a senior sysadmin, and the answer was being able to read the code of the applications you deploy and to bridge the gap with developers and DBAs. "That's what DevOps is," Phil says, and "it drives me nuts when people talk about, oh, look at this new idea that we just came up with, and it's never been new." Matty quotes a friend who has been doing DevOps for a long time, "back when I just called it work," and suggests the current wave corrects a pendulum swing toward admins who only care about the box. Phil's take is "those people are just bad at their job." Matty says the problem is that there are a lot of them.