Bryan Finster

Bryan Finster: Hi, everyone. I hope you are enjoying the virtual conference so far. I am Bryan Finster. I am a value stream architect for Defense Unicorns. I am also a contributor to the project I want to talk to you about today, MinimumCD.org, and a board advisor to the Value Stream Management Consortium.

I have been solving business problems with code for more than 20 years and looking for ways to make that less painful for myself and others I work with. Today I would like to introduce you to the Minimum CD project. This is an open source project several of us are working on to help others in the community on the journey to continuous delivery.

All of us are here to find ways to improve the outcomes for ourselves and to help others do the same. A common approach to this is enterprise transformation. Also common is for that transformation to focus on agile process: we need an agile mindset, or we need to be better at Scrum, or what we really need is a scaling framework to make sure we are all being agile in exactly the same way.

I am not a big fan of transformation. I think what we are really after here is the continuous improvement of our outcomes for ourselves and our customers.

In 2018, Accelerate reported that continuous delivery improves both delivery performance and quality, and also helps improve culture and reduce burnout and deployment pain. As a longtime developer, all of those things are very important to me. And I know that these things are true. I and others in this community have experienced this, but only when the practices of CD are taken seriously. When they are, they become enabling constraints to shine a spotlight on problems throughout the value stream so that we can relentlessly improve them. The result is better outcomes for everyone.

People read this and what they see is: oh, we need continuous delivery, so we need CD. Do CD. But instead of digging in and understanding what this means, and having CD become an enabler of improvement, they try to force CD to work without changing anything else in their environment. And then problems begin. I want to talk to you a little bit about some of the problems that I have seen, and others have seen, and others have reported, and how CD actually should be used to make these things better.

The first thing I want to talk to you about is credit card continuous delivery. This is where we buy all the tools, and now we are doing continuous delivery because we have the continuous delivery tools. And then: you get a pipeline, and you get a pipeline, and everybody gets pipelines. The results are not optimal. We have new capabilities without also growing the knowledge to use them safely, and then people blame CD as the problem.

I have literally had people tell me that CD has been banned in the organization. In a previous organization, I saw this firsthand, where someone decided that all teams would be CD by a set date. This leader decided that if we just met that date, then we met our goal. But CD is a workflow in an operating model, not just the tools. The teams that were early on and went through the learning and what this meant saw good outcomes. The teams that were just handed pipelines did not. We see this as: CD will not work here; obviously, it is too risky; you are not mature enough. And then CD is banned in that organization and nothing ever improves.

Another interesting one is that CD means we can deliver on command, on demand. If you read the definition of continuous delivery versus continuous deployment, that is what it says: CD means we can deliver on demand. But when we only deliver once a month because we say, well, our customers just do not want change more frequently than that, we do not really see the outcomes we are looking for. The outcome we do see is automated stagnation. Nothing improved except that we automated the current process we have for doing build and deploy.

People say, well, we have automated delivering. Why are we not seeing these promised improvements? CD is just another buzzword. Well, because the behavior stayed the same. We did not have to change anything. Our batch sizes are still too large. Because of that, our feedback remains slow. We have long delays in getting quality feedback. We have long delays in getting product feedback: was that even the right thing to deliver? Quality processes did not have to improve, so the risk remains high. We still have so much drama. Communication did not improve. Really, what we are looking at is the supply chain of communication, and we are trying to optimize that communication flow through the value stream. But when we are only delivering once a month or once a quarter, none of these things have to improve.

Last year at the dockside bar on Gather, we were sitting around on day one of the conference and talking about these things over lunch, drowning our sorrows a little bit and seeing the problems that people were having. How can we make this better? We said we had two options: we could just sit here and gripe about it at the bar, or we could try to fix it. So we decided to choose number two.

We set ourselves some goals. Number one, we needed a minimum set of measurable behaviors. They had to be absolute minimums. We did not want any fluff. We did not want to write an entire book, because there is already a book about these things. We wanted to talk about the things that really counted to start seeing the outcomes promised by Accelerate.

They needed to be true in every context. This was also important. We did not want to just come up with a list of things that works if you are delivering to the cloud with a web application. We wanted them to be true on bare metal, on the edge. We wanted them to be true in air-gapped environments where you literally have a sneakernet deployment method. And we wanted these to be enabling constraints. We wanted these to be problems that, when you solve them, reduce pain and improve outcomes. We were pretty relentless about this. We hammered away at these things.

You might ask, why did we work so hard on this? Because we think everywhere should be an awesome place to work. We think that if everywhere is an awesome place to work, that elevates the entire industry. Selfishly, when we go to another organization, it is already an awesome place to work. We do not have to start over from scratch. We know for a fact that if you deploy more, you do sleep better. Everybody should have that outcome.

Three days later, we published the website, MinimumCD.org. We asked people at the conference to review and to give us feedback, and if they agreed, sign on. If they did not agree, how could we make it better so that it gets closer to what we are really after? A week later, we received a contribution. We were all very excited about it. It was a small language change, some enhancements to the phrasing, and a signature from the co-author of Continuous Delivery, Dave Farley. That was super exciting.

Today, where are we at? We have 39 contributors from all over the globe. We have 133 stars on GitHub and 124 signatures, and more keep happening all the time.

Let us talk a little bit about what these things are that we came up with and why some of them are so important. First, use continuous integration. This is foundational. You cannot do CD without CI. You will sometimes see CD written CI/CD because they are so tightly coupled.

Second, the application pipeline is the only way to deploy to any environment. We need one flow all the time. The pipeline decides the releasability of changes. Its verdict is definitive. Very important: we do not need any approvals outside the pipeline.

Artifacts created by the pipeline always meet the organization's definition of deployable. That is going to be different for every organization and probably for every application. We need an immutable artifact: no human changes after commit. If there is some failure in the pipeline that requires something to change, we change it in version control. We do not change it further down the pipeline because that creates snowflake environments.

All future work stops when the pipeline is red. This is a behavior thing: do we work this way? It is very important that we have a flow to validate our changes and to deploy. If we cannot do either one of those things, nothing else matters. We need to stop future work until we can. We need production-like test environments. We need to roll back on demand. And the application configuration deploys as the artifact. Everything originates from version control.

I do not have time to talk about all of these. I will touch on some of the things that I think are very important. I encourage you to go to the website and read some more about these, and we continue to expand on why some of these things are important.

First I want to talk about continuous integration. We define what these things mean on the website, so there are no misconceptions here. What this means is that tested changes integrate into the trunk at least daily. If I make a change today, it integrates to the trunk today and it is tested, not waiting to be tested.

Changes are, to the best of our knowledge, deliverable. There is only so much testing we can do during the CI portion of this flow, but we need to be 85% sure that we can go to production and not break anything. Does this mean it needs to be feature complete? Not at all. What it does mean is that whatever change I am making today is validated with tests and does not break anything.

Also, fixing a broken build is the highest priority. This is a very important behavior thing. What this means is that if I cannot validate that the change I just made is non-breaking because the build is broken, then I cannot do my work. We as a team need to make sure that we have the ability to validate our changes at all times.

I also think it is very important to look at this from the point of view of an individual on the team. An individual did not break the build. The team broke the build because an individual was allowed to make a change that broke the build. This is a continuous improvement thing. As a team, how are we able to break the build, and what pre-checks can we make to reject bad changes from the pipeline to keep the pipeline green at all times?

Then ask: why can we not do this? That simple question uncovers so many problems. Communication: how well do we talk to each other within the team and outside the team? Do we have the right information coming in about the product goals we are after and what it is we are actually trying to accomplish? Are we breaking down work effectively? Are we able to do evolutionary development so that a change is backward compatible for the pipeline as we go forward? We are continually improving things, but we are not doing things in a way where, for example, deploying a database change breaks existing things. Is our testing up to par? Do we need to improve that? How is our teamwork doing? All of these things show up when we focus on CI.

I have seen this repeatedly. These are very common problems, one or more of these on every single team. CI very effectively uncovers this. It takes a team working closely together to integrate tested working code that frequently. It requires clarity of purpose, improved testing, and minimizing handoffs. CI teaches us how to shrink batch sizes so that we can get faster feedback and reduce the risk of delivering incorrect or defective results.

I have seen this: when you are doing this even at a medium level, suddenly things around planning poker or story pointing no longer have value because we have learned to break things down so small that everything takes a day or two to get done, and we have much higher clarity.

CI is the beating heart of our improvement process. It is the thing that drives everything else, and this is where we should always start. Overcoming these things makes things suck so much less. I guarantee you that the clarity of purpose, the teamwork, the understanding that we are doing the right thing, the rapid feedback, all of these things give us so much more confidence. On the first team where we were doing this, an unexpected outcome was just higher morale for ourselves and for the people receiving our software. It was amazing.

The next one I want to talk to you about is one path to the environment. We do not release to a test environment using a different method than production because we need confidence that our production delivery will not fail. I have seen people say that they are using continuous delivery, but they are releasing to a test environment from a branch called test or releasing to another integration environment from a branch called integration. That is not continuous delivery because we are not validating the entire process.

We need to release to every environment from the same place that we are going to release to production, using the same exact method, so that we are validating everything. We are validating the quality of our delivery process, not just the code that was written.

If we only have one method to deliver, then we also must optimize that for delivering in emergencies, because we only have one method to deliver when there is an emergency. We do not have a hotfix process that is separate and different from our process for delivering features. That is really the secret of continuous delivery: in CD, everything is always a hotfix. The fact that we are delivering features using our hotfix process means that when it is a hotfix, it is muscle memory. It is something that we know how to do. We do not have to invent something new. We have something that is very safe. We have something that is fast, and it just removes the drama from delivery.

The next thing I want to talk about is that the pipeline decides releasable. We cannot inspect in quality by reviewing work at the end. Instead, we must build quality into our system of delivery. To take this constraint seriously, we need to evaluate how we decide things are deliverable. Is it truly an objective assessment? Is it not based on emotion, someone's feelings, or a VP saying, I do not know, you did not say the right words?

If it is objective, we can automate it. We are also forced to take a hard look at how we approach security and compliance. It is simply theater to do occasional checks for these after delivery. We need to embed our requirements into the pipeline so that every change, every commit, every small thing is not only functional, but as secure and compliant as we can make it, and stays that way. We make security and compliance automated safety nets in the process, not discoveries after delivery.

By doing this, yes, we are functional. We definitely need to be checking to be sure we are performant, but we also know we are safe, and that is incredibly important. With the pipeline doing this, we can do this every single day and always know we are doing the right thing.

So is this another maturity model? Are we gatekeeping and telling people, you are just not good enough for CD, or you have to be this tall for continuous delivery? No. That is not what we are trying to go after at all. This is an improvement journey. It is a list of challenges that we need to solve to improve everything around us, not just the application we are building, but the entire organization and our way of working. Yes, we can do these things, or no, we cannot do them yet. It is just a binary thing. Let us start hammering away at these things.

Going on this journey will uncover problems that you may be numb to or did not even know were problems in the first place. We can choose not to fix them, but at least they will no longer be hidden. We have lifted up the rug. Honestly, as soon as you see them, you cannot unsee them. That pain will nag at you and you will want to make them better, and in doing so you will improve everything around you.

So where do we start? We start with: why can we not deliver today's changes today? I literally wrote this on a piece of paper and put it up in the team area on the team I was on at the time. Hey guys, why can we not do this? We just started hammering away at the problems until we could deliver multiple times a day with high morale, high confidence, and high-value change.

Start with the roadmap. Define releasable in your context. That context may be your entire organization, but honestly, it is probably going to be for this particular application, this one value stream. What does releasable mean in this context? Fall passionately in love with testing. I do not mean that everybody needs to learn how to unit test. I mean everybody needs to understand what is required to test everything fully before it goes to version control, so that the pipeline can validate that we are releasable. This takes time, and it needs to become your primary focus.

Start with continuous integration. As I said, that is the beating heart of the improvement. You will find so many issues early on and see improvement early on that will make things better. Automate manual validation of releasable. You are doing manual validation at some point, most likely. We need to look at every single thing we are doing manually as a defect to be resolved, not a feature. If we have any manual validation of the pipeline, that manual validation should be happening in an emergency. If it is not, then we are not using a safe method to deliver in an emergency.

Improve team structure to remove handoffs. Do value stream maps. Look at how information and work flow through the value stream. Look at those handoffs. Every single one of those handoffs is a problem to be solved. Even a code review, if I write code and then wait and do a handoff for somebody else to code review, is a problem that causes waste and loss of context. We can do that better. How do we improve that to remove those handoffs?

Relentlessly improve. Look at pain and make it go away. Jez Humble says that if something hurts, do it more frequently. I have done that recently. I had to go and improve several tests on an application I am working on because they just hurt, and the pain was causing us to not want to do that very frequently, which meant that we had to fix it so that we can be safe.

Recently I asked Dave Farley if he could give us some feedback on what he thinks about this project. This is what I got back from him. He says it is a clear, focused, no-holds-barred statement of what it takes to achieve CD, and that was our goal. He says if you can do what Minimum CD says, you will be doing a better job. It gives us a clear, simple focus on the essentials of CD that can help teams understand what really matters to build better software faster. We want better software. Everybody wants it faster. And it makes us better lives.

So we need help. First, contribute stories to the website. What have you done in your organization to get closer, and what outcomes have you seen? Find better ways to do things, and then contribute that information to the website. We have a section that is all about beyond the minimums. These are good practices that are recommended, that may not work in all contexts or may not just be the absolute minimums, but absolutely are good ideas.

Then share them. Help everybody else in the community. Pay it forward. We are doing this to help people. None of us are getting paid for this. Spread the word. We need translations. We are currently translated into five languages: Spanish, French, Italian, Portuguese, and Finnish. Somebody I know is currently working on German. It does not take a lot of effort. It is just the one page. We have not translated out the entire website. So please help us get it out there for everybody else in the world.

With that, you can reach me on LinkedIn as Bryan Finster and on Twitter as Bryan Finster. I also have a series of blogs at blog.bryanfinster.com. Again, this is not my project. I am just representing the community that is working on this. Test, commit, repeat. Go out and look at MinimumCD.org. Give us feedback if you find a problem or an issue, or if you have a suggestion. Open an issue. Join the Slack org; there is a link out there to do that as well. Join in the conversation and help out.

Thanks very much, and enjoy the rest of the conference.