John Willis

John Willis introduces the presentation as "DevOps 2020 Re:Think!" He says it was a collaboration with several coworkers and collaborators: Jay Bloom, Andrew Clay Shafer, and Kevin Behr. He wants to credit them for some of the slides. The talk is called DevOps 2020 because DevOps has done a good job over the previous 10 years, and the question now is what the community should do at the start of the next decade. He says he will focus on three areas: organizational conversations, organizational design, and what his team is calling the three economies.

He introduces his Red Hat Global Transformation Office team. He started at Red Hat in October 2019. Andrew Clay Shafer is his boss; Kevin Behr is on the team; Willis is also there; and Jay Bloom, who is pursuing a PhD in transition design, contributed many of the ideas about how design and transition design apply. Willis notes that he, Kevin, and Andrew have written books: Willis co-authored The DevOps Handbook and Beyond the Phoenix Project; Kevin co-authored The Phoenix Project; and Andrew co-wrote Web Operations and material connected to site reliability engineering.

Willis looks back at the previous decade of DevOps. He jokes about the "unicorn" image that, as he later learned, was originally created by Patrick Debois. He says the DevOps Enterprise Summit community has accomplished a great deal, but it has been a struggle. Early enterprise conversations questioned whether the enterprise could do DevOps, then whether security practices could apply around DevOps. By 2020, he believes most people have the memo. But digital transformation has been rising as a modern discussion, even though it has existed in some form forever. He points to reports saying that 70% of digital transformations fail, and says the next 10 years may require the community to take on digital transformation in a bigger way.

He says the people at the conference have generally been working on these problems already, but the community still needs to educate others. Andrew Clay Shafer has a model of five failures. Leadership prevents change, whether through governance, risk, general business behavior, or other constraints. Product builds things that do not matter; Willis says organizations still have not fully absorbed Eric Ries's Lean Startup message. Development often builds the wrong things. Architecture is often not involved in design decisions or designs the wrong way, so the organization builds things wrong. Operations still has a split mindset around incidents, outages, service management, DevOps, and newer incident-management ideas.

Willis says DevOps has done a tremendous job improving commerce and people's lives, and the community should take that seriously. But in 2020, the question is what DevOps will do for the next decade. If the community is still talking about GitOps and CI/CD five years later, it will probably have failed, because the digital transformation discussion will have overtaken it. Continuous delivery was a novel conversation five years earlier; by 2020 it is table stakes. The goal is to make new conversations into the next table stakes.

The first area is organizational conversations. Willis has been doing this work for three or four years under different names. He goes into large organizations and speaks to hundreds of people. Usually he enters through the CIO level, after an internal champion recommends him. He then convinces the CIO to let him have conversations with people at the edge: the people who put their fingers on the keyboard. He is more interested in bottom-up conversations with the people doing the work than in top-down leadership discussions. He has done this with large banks and insurance companies.

From that work, Willis developed the statement: "You can't Lean, Agile, SAFe, or even DevOps your way out of a bad organizational culture." Lean, Agile, SAFe, and DevOps are useful frameworks, patterns, and practice tools. But if an organization does not get to the bottom of how things really work, or have real conversations with people doing the work, these frameworks can create false truths.

Willis says he has a presentation about the seven deadly sins of DevOps. He will not go into the full detail here, but the patterns he finds in conversations often funnel into what he calls security and compliance theater. In other words, the audits are basically nonsense. He says he has full presentations on this and related work in automated governance and automated cloud governance.

He then tells the Abraham Wald aircraft story, which he heard Sydney Dekker tell at DevOps Enterprise Summit. During World War II, scientists, mathematicians, and statisticians studied the bullet holes on fighter planes that returned from missions. At one point Wald realized they had it wrong: they were repairing where the bullet holes were, but those were the planes that came back. They needed to look where bullet holes were absent, because planes hit in those places did not return. Willis describes this as an early definition of survivor bias. He says Dekker's lesson is that organizations should not only look at the absence of negatives; they should look for the presence of capacity, the things that go right. Willis uses this idea in organizational-conversation work.

Willis then references Eliyahu Goldratt, author of The Goal, which inspired The Phoenix Project. Goldratt also created an audio-only project called Beyond the Goal. In it, he discusses complex systems and complex adaptive systems. Willis describes two systems, A and B. Most people might say B is more complicated, but a physicist or someone who understands complex systems might say A, because it allows more degrees of freedom.

When Willis talks to customers and works through the CIO but speaks to people at the edge, people often first give him system B answers. They say things like "that works" or "my CMDB is fine." Willis needs to get beyond those simple answers to the truth. He uses the "this is fine" dog cartoon: people initially say everything is fine, but once trust and psychological safety are created, they tell the real stories about fantastic workarounds and the places that are actually on fire. Those are the system A discussions he wants.

He uses the Equifax breach as an example of the difference between system A and system B answers. In 2017, the breach involved Apache Struts 2 and a Jakarta parsing module. A curl command against a system with that vulnerable library could compromise the system. When the breach was summarized, the CEO said the problem was a single person who failed to deploy the patch. Willis calls that a system B answer. But the 2018 Congressional oversight report described many complex system problems. The chief security officer reported to the chief legal officer. When asked why he did not notify the CEO of the breach, he said he did not think about it. Willis connects that to the reporting structure. The perimeter intrusion detection systems had certificates expired for 18 months. Willis says he looks for those complicated, honest answers and discussions.

The second area for the 2020 rethink is organizational design. Willis gets much of this from Jay Bloom's work on transition design and design thinking in transformation. He starts with the evolution from agricultural economy to industrial economy to knowledge economy. The world is now in a knowledge economy, but when people talk about Lean and Toyota Production System, they are still struggling with how to map what works well in an industrial economy onto a knowledge economy. Knowledge work is still partly art.

He introduces the five elements: leadership, product, development, architecture, and operations. He also introduces design, engineering, scale, and differentiation. He says roads in Lean, Agile, and DevOps often lead back to Toyota. Toyota's four V's of learning can be used to understand economic value and balance theory: variety, variability, velocity, and visibility. Variety balances market demands and operational efficiency. Variability manages inconsistency to reduce cost and improve quality. Velocity concerns steady flow through the system. Visibility creates transparency for learning and improvement.

Willis says the five elements and the four V's have different motivations. Developers and product often want speed and differentiation. Operations and architecture often want resilience. The conflict is not simply that one side is good and one side is bad; it is a set of economic motivations and trade-offs.

He then discusses variety as a way to balance market demands and operational efficiency. He quotes Alicia Juarrero on constraints enabling freedom: by curtailing possible variation in component behavior, context-dependent constraints can paradoxically create new freedoms for the overall system. Willis summarizes this as certain types of governance enabling freedom for the system.

Next he discusses Garrett Hardin's tragedy of the commons. Self-interest can lead users to behave contrary to the common good by depleting or spoiling a shared resource through collective action. Willis summarizes this as consumables needing to be managed to preserve the system: too many cows consume all the grass and the field collapses.

He then references Ross Ashby's Law of Requisite Variety: for a system to be stable, the number of states in the control mechanism must be greater than or equal to the number of states in the system being controlled. A stable system's controls must match or exceed the controlled system. He also cites Don Reinertsen on cost of delay: any prioritization decision is a decision to serve one job and delay another. Without all the details of Reinertsen's book, Willis summarizes the practical lesson as focusing on high-value, high-probability items in the backlog. The common theme is economic balance and how to make trade-offs and decisions.

Willis summarizes variety using constraints, consumables, Ashby's Law, and cost. Constraints enable freedoms. Consumables must be managed to preserve the system. Control variety must be adequate to the controlled system. Cost of delay should guide prioritization.

He then turns to variability, or variation. He quotes an unknown author: misunderstanding variation is the root cause of knee-jerk reactions, over-control, micromanagement, and tampering. He cites Deming on operational definitions in collecting data. Without operational definitions, data is suspect. Change the definition and the data changes. Without written definitions, different opinions among those collecting the data create muddled data. Willis says the bottom line is that the community quotes Deming constantly but rarely listens to him. He asks whether organizations are really doing operational research and applying the science: statistical process control, the system of profound knowledge, Deming and Shewhart's plan-do-check-act or plan-do-study-act thinking.

Another place to look for variance and opportunity is Genichi Taguchi and the Taguchi loss function. Willis quotes the idea that cost is more important than quality, but quality is the best way to reduce cost. He says this means finding the edges of variability. It is not only about how tight tolerance levels are, but how far they can be stretched. Hidden value is at the corners. He also references the Red Queen effect from Alice in Wonderland: in 2020, if the community is running in the same place, it is losing.

He summarizes variability through statistical process control, tolerance and Taguchi, and the Red Queen effect. He says the commonality is math, engineering, and statistics. The next age of DevOps needs to stop knee-jerk reactions, such as reacting to a breach by hiring 100 security professionals because a finger-in-the-wind answer feels right. Industrialization and power plants have used this knowledge for a century. There is 100 years of engineering in front of the software community that can be applied more seriously.

Willis cites a quote about Walter Shewhart: Deming said in 1980 that it would be 50 years before people understood the real value of what Shewhart was saying. Shewhart created much of the genesis of Deming's work, statistical process control, and plan-do-study-act. Willis says there is still a lot of useful information in industrial engineering and operations research. The community should stop merely quoting it and start looking at the real science to make breakthroughs. He is not trivializing the tremendous work people have already done, but he is saying that organizations often stay at the level of platitudes. When someone says they will never get management to understand Taguchi, Willis replies that Toyota got management to do it and decimated a market for 50 years.

Willis ends with platforms and the three economies, an idea discussed internally and written about by Jay Bloom. Most infrastructure, scale, and DevOps discussions have been bound by two economies: differentiation and scale. That can be called development and operations, or infrastructure and development. It has been a bimodal discussion. The differentiation economy emphasizes velocity, novelty, niche fit, experimentation, and incubation: the things expected from differentiation-oriented development. The scale economy emphasizes regulation, reduction, resilience, reuse, and consolidation: the things expected from operations or infrastructure. Willis says those two economies are understood fairly well.

He then describes conversations with Mark Burgess, who wrote the foreword to the SRE book. They talked about SRE and how Google built infrastructure over 10 or 15 years. Google began with Borg, turned that into Omega, and ultimately the world sees those ideas in Kubernetes. Burgess said one brilliant thing Google did was make a non-deterministic infrastructure look deterministic to developers. Developers did not know the particulars of virtualization or storage. They had APIs and interfaces, and in many cases could not know the underlying details. They created applications and services through bounded interfaces.

Jay Bloom would call that middle layer a scope economy. Google did not call it that, but Willis sees it as a clutch between scale and differentiation. It is not just a platform; it is an interface and an abstraction that lets developers get their best value and infrastructure get its best value. Differentiation and scale both press toward the middle, and the scope economy creates adoption, control, and the balance both sides need. Differentiation enables velocity, variability, and variety. Scale controls velocity, variability, and variety. Scope gives the best of both worlds.

Willis says he believes platforms are one of the most important issues in 2020. He acknowledges that this is self-serving because he works for Red Hat, which sells OpenShift, but he says he believed this before joining Red Hat. Stephen O'Grady said developers are the kingmakers; Willis says that in 2020, if leaders are not thinking about the next decade of platforms, what those platforms will look like, how they will be used, and what strengths the organization has to use them, they did not get the memo and will probably lose. The new way forward is platforms: platforms are the new kingmakers.

He cautions the audience not to get lost in marketing hype, including hype from his own company. The lesson is what Mark Burgess said about Google's brilliance: Google created an abstraction that allowed developers to be divorced from infrastructure and work through APIs, interfaces, and well-documented boundaries. Enterprises need to get there, though it is a long haul. Google may have had one infrastructure and relatively few applications; banks may have 20 or 30 lines of business and thousands or even 10,000 services. It is a harder ask, but enterprises still have to get there.

To get there, Willis says organizations must stop thinking of a platform only as a differentiation economy, a platform-as-a-service, self-service, or, worse, a container system that manages clusters. They should think of it as a scope economy and a platform as an interface. The platform enables what the differentiation economy and scale economy need, collapsing them in the scope economy. That is how organizations can start looking more like Google.

Willis says he often tells large organizations to calm down because they are not Google; they are a bank or healthcare company reading what infrastructure is supposed to look like. Later in the same presentation, he says they can be like Google if they understand that Google did not really create a PaaS; it created a platform as an interface. As service mesh, Istio, Envoy, and related technologies mature, the platform becomes an experience. He has seen smart people who used to work for software companies take VP of engineering jobs in large Global 1000 companies, including shoe companies and banks, because those enterprises know they need people who can carry them through the next 10 years. Willis says that new generation and new way of thinking is a scope economy based on platform as interface.

His final ask is that people help push the conversation about organizational conversations, about what has been learned from Toyota and operations management, and about whether the community is applying the right science. He thinks the industry often has a lot of platitudes and quotes. He wants the three areas--organizational conversations, organizational design informed by real operations science, and platforms--bound together into the next-generation discussion. He invites anyone who wants to have that conversation to reach out and help drive it. He closes by giving his name, Twitter handle @botchagalupe, and his Red Hat email address.