Gunther Lenz

Hi, my name is Gunther Lenz. I'm VP for software R&D at BD Biosciences, based in San Jose, California, and I'm super excited today to talk to you about our experience doing a digital transformation Phoenix style in the medical device environment.

Before jumping into the content and the slides, I really wanted to put some disclaimers up front.

First of all, digital transformation is a journey. It's not something that happens overnight. It's not something where you can just flip a switch and it's done. It needs the engagement and the commitment of the entire organization because you literally will touch everything in the organization, from product, processes, to the people. Keep that in mind if you start that journey.

The second point I wanted to make is, while in my presentation I'll present to you some of the concepts and the frameworks that we are using and that we found helpful in our journey, those are just frameworks, and they are guidance. They need to be adapted to your environment. They are helpful, but they're not really silver bullets. There's no silver bullet in digital transformation, like anywhere else in software engineering. In the appendix, I actually have a collection of all the links in case you want to learn or research a little bit more of anything that you hear today. It's going to be available there because it's only 25 minutes and it's a huge topic, so stay with me there.

Lastly, I wanted to just say that DevOps, or SRE, or platform engineering, is really the basic capability of the organization in a way. These capabilities build the road on which the digital transformation is driving. The better the road is, the faster that digital transformation can move forward. Keep that in mind when you start your digital transformation. With that, let's dive into the presentation.

Let me give you a quick overview of what our software does here at BD Biosciences. We focus on flow cytometry. Flow cytometry is really used in clinical environments or medical research for diagnostics and for research on cancer treatments, as well as vaccine research, for example for COVID-19. You see the instrument on the left. This is our latest and greatest instrument to market, and the instrument produces data, and that is then transferred to the software application that you see in the lower right corner there. You see there's a lot of analytics going on for the researchers. These are all cells, and you get density charts and dot plots, and you can sort on different parameters, some spectral analysis going on. On the right-hand side you even see images of the cells that go through the flow cytometer, and these images are acquired, ten thousand images per second, without a camera. This is a first-of-all technology that was featured on the Science magazine front page earlier this year.

So that's the software. I listed the major symptoms that we saw that made us start our digital transformation. First of all, we are really challenged with keeping up with the business requirements for new capabilities, new features, and new software releases. In addition to that, we saw incompatibilities, inconsistencies, and duplication within our software.

Next, at BD, we are a medical device company, and we are planning for instruments for release for one, two, or three years in advance. Obviously, naturally, that results in long front end and long tail end for releases. It's almost impossible. Think about it: we are planning for three years, let's say, or two years, and you're trying to define detailed requirements. We do effort estimates on those requirements, so it's really tough to get all that right, since this is a very complex system that we are working on. It's not a shopping cart for an e-commerce application. This is first-of-all technology that is highly complex, that includes software, firmware, hardware, everything kind of together. It was really tough to do that and to see those long release cycles, particularly those long tail ends.

Those long tail ends also come because we found defects just before release. We had those release cycles for a year or two on the software, and then at the end, when we do the system tests and verification, then we find a lot of the defects, and that then obviously delays the release even further. Even worse than that, I think, is that we get late customer feedback. Customers really don't necessarily see the product before it is very, very close to release. If there is a finding and there's something that we want to change, then that delays the release for even longer.

Also, from a technical side and from a tech-debt side, we look at our complexity and we find that we have higher accidental complexity than we really would want to have. We found that that is also something we need to address. From my perspective, one of the major reasons here is also that the software organization is not scalable as it is. Let's say we want to accelerate a release of software. If I hire two scrum teams, will that change our release date? Will that have a linear effect on getting our release out earlier? Currently the answer is really no, and it's a combination of different things why that's not the case. It's people, process, product, all of them, and architecture obviously as well.

The last symptom is really around that we're successful and we're good, so we don't feel the need for change as much. We're not as hungry for change as we should be.

All these symptoms together, we then tried to find: okay, we see that, but those are not really the root causes. What are the root causes for a lot of these symptoms that we are seeing and the root causes that we then want to solve?

The root causes here are really four major points. The first one is a siloed, hardware-centric program view of software. I mentioned it already in our planning cycle and release cycle. Everything is really based on an instrument and on hardware, and that doesn't work for software very well. Software is inherently different than hardware. The software innovation cycle is much shorter. Software has a high fixed cost to get it out, but it has economies of scale. You can distribute it for free, basically, so low marginal cost to get it out. You can also capture economies of scope easily if you do it right and share components and modules between different software products.

The second point is, we have large batch processing for our releases. We get a really big backlog and walk through that. It's a very detailed backlog too. It's not a problem statement. It's really more on a solution-feature basis, where we roll through that backlog.

Then we have highly coupled software, and there are point solutions. That is definitely something that we need to change, and that is also part of the reasons why the software organization is not necessarily that scalable. Then we focus a lot on short-term features, because the priorities are always on one instrument and not on the software, if you think of a platform also.

These are the root causes that we identified and that we wanted to solve. It became inherently clear to me that we need some outside perspective to make that happen. We partnered up with some strategic partners to accelerate our change in that area and help us really make good choices on the engineering side.

We engaged Thoughtworks. It's a consulting company. They're very well known in the software world. They have extremely good technical talent. They are very state of the art in best practices, architecture, and those types of topics. They also have a lot of publications on those topics, and they also work in an agile fashion. We brought them in to make sure we get this outside perspective. They work with our teams. They do not solve our problems, but they help the teams. They bring this outside perspective, with the good technical talent that they have, to help us solve and implement the solutions to those problems.

Because we are a medical device company, we also partnered up with Software CPR. Software CPR is a small boutique consulting company and helps clients with being a software organization in the medical device environment and staying agile. They really help us look at our processes that we have, making sure that we are regulatory compliant, but at the same time efficient. We really want to be lean and agile, and they are helping us to do that.

With all that said, the digital transformation that we underwent really fits perfectly in the concepts that are represented by Gene Kim in The Phoenix Project. If you think about it, the problems that I just stated and the root causes, we see the symptoms. We have those long front and tail ends. That is just the left-to-right flow; you have to look at it. How do we optimize that? How do we make sure that our throughput is there? How do we make sure there's no bottlenecks? Then we want to get faster customer feedback, so amplify the feedback loop. At the end of the day, we really want to optimize on what we are doing. We want to have continuous improvement at our core. Always look to get better in what we are doing.

What I learned from The Phoenix Project is particularly that these concepts that we see in manufacturing and lean manufacturing, and how they translate into software, fit perfectly in the digital transformation journey that we are undergoing. More interestingly, I published a book with a colleague together in 2008 on software factories, and we talked about similar concepts. We looked more at product line engineering, but if you think about it, platform engineering is the same concept and it's the same thought process as well. All that comes together in this digital transformation that we are undergoing. That's why I call it digital transformation Phoenix style.

Now let's look at the First Way: flow and systems thinking. We firstly wanted to know, where are we as an organization in this journey of digital transformation, digital fluency, agile, all those types of things? With that, we wanted to say, okay, let us pick some focus areas of improvement. We can't do everything at once. We want to boil the ocean, but we want to do it one spoon at a time. Let's identify which focus areas give us the biggest bang for the buck for our transformation.

Then we did a light value stream mapping. We focused here on the new product development, but we wanted to know what is the left-to-right flow for those new products that we develop and identify how we can streamline that. Then, bringing in the software portfolio view from the perspective of more platform engineering thinking, we wanted to see what we can do there to help us in our journey going forward.

The best model that I could find was a digital fluency model that's published by Thoughtworks. You see the link there. There's a lot of information there. You can actually define your own digital fluency model for your organization if you want, and there's a lot of information on these capabilities, as well as the fluency levels and how to get from one to the other.

In the first column, you see the digital fluency level. It ranges from awareness, which is the lowest, to strengthening, which is the highest. Then we have capabilities: frictionless operating model, platform engineering, experience design and product capability, intelligence-driven decision making, and engineering culture and delivery model. This is the framework that we put ourselves into.

Before doing that, I wanted to say there's no good or bad. There's no right or wrong. Each organization should decide where they want to be for each of these capabilities and digital fluency levels. It doesn't mean that everybody has to be strengthening ongoing in all the different capabilities. That's not necessary and it's probably overkill for organizations.

We did that exercise and we decided where we think we were when we started the journey. That is the red dots, and we connected them by the line. You can see some executing, some focusing, some bordering on awareness. We said, okay, the biggest bang for the buck for focusing our efforts to improve will be platform engineering and engineering culture and delivery model. We said, okay, these two are the ones that we attack and that we will work on to improve. We actually think that if we work on those two, we will lift all the other ones up as well. That's just natural interdependency, and lifting one will actually impact the other ones positively as well, potentially.

Then we looked at the light value stream mapping for a new product here. We threw that out here, and you can see we overlaid the functions from the different organizations. Those are the red boxes. Then we mapped out our process. You can see that we have a long pre-work phase here, that is discovery. You can only imagine if a program takes one, two, three years, that's a long pre-work because we do really upfront requirements analysis. We do design. We do even estimates on that. We take that then and hand that over to the development teams, and then the development team runs in their sprints and runs for a while until it hands that over when you want to release the software a year or two years later.

Then SQE and test automation run FCAT, which is feature completion and acceptance test. There's a hidden step that nowhere shows up in an official process map, and those are the stabilization sprints. Before FCAT, before even we can run FCAT, we have to do stabilization of the software, and that stabilization actually goes for weeks, potentially months. For large projects, it could go up to six months, where we really look at fixing defects so FCAT can actually eventually ensue. Once that succeeds, we hand that over to quality and systems function. They do verification and validation. They will potentially find some issues, send it back to development, we go through FCAT again and go to V&V again. We circle through that a few times, all that adding obviously to the long tail that we have before eventually we hand this whole thing off to service and operations.

You can see that the problems are the large batch processing and long Scrumerfall release cycles. We have way too many handoffs, and we can see some dark agile attributes popping up in our value stream map here.

Then from a portfolio perspective, I mentioned already everything is run by funding, and program management is done by an instrument pretty much. There are some exceptions on software that doesn't have an instrument, but it's the same concept. Even there we have those stovepipe isolations of the software. It's prioritized where its work is done in its own environment, but not really overarching over the portfolio itself. That's the siloed, hardware-centric program view of software. You can also see that Conway's Law is real. Just to reiterate, Conway's Law pretty much says your system will look like your communication structure in the organization, and that's what we have exactly here.

Then we have a lack of platform engineering. We have all those different software products. They do pretty much similar things in many cases, but we don't have anything that holds them together. We don't have platform engineering. We don't have enablement to make those teams themselves more efficient and to have structured reuse in that as well.

The improvements that we did resulted fast in some benefits. We really focused on test strategy and looked at our test pyramid. There's a link here that might be useful for you if you are interested in that. We focused on test automation and a more stringent definition of done and hard gates for check-in, so we find issues earlier in the process and don't take too many shortcuts. Then you can see some numbers for improvements that we achieved in implementing this First Way in the first probably six to nine months.

This brings us to the Second Way: amplify feedback loops. What we're doing here is, we apply an inverse Conway maneuver. We reduce the batch size and increase the feedback that we get. In general, we shift left, so we have more testing done at the time of development rather than waiting to the end.

Inverse Conway maneuver: what we want to do is create an organizational structure that looks like the system that we want to develop, or want to get to. What that means is that we change to a product team. Instead of all these functions isolated and having a lot of handoffs, we collect them and create a product team. That product team has all the functions that it needs to own a customer problem down to the development, deployment, and potentially operations as well. These teams own those problems. They define the solutions in an iterative way, and they make those decisions by themselves. The structure reflects what we want to achieve from a system perspective.

Then we want to accelerate the feedback cycles. Now we have those different product teams. They own the customer problem and the feature scope down to the release. They now can run in shorter iterations to release every one to, or maximum three, months. You can see that visualized here. It's basically becoming a release train. At any point we could say, hey, R3 is good enough for an external release, and then we hand it over to V&V, but the software is already ready for V&V. They don't have to circle through FCAT a couple of times. It gives much more flexibility in defining an MVP and releasing MVPs and software independent of an instrument release cycle.

So here it is: the autonomous product teams. We have frequent internal as well as external feedback because we release the software capabilities much faster and much more iteratively. We also focus on increasing software delivery and process automation in that, so we spend less time on manual work and really automate a lot of that.

Then we want to do shift left. Shift left is really how we create this scalable organization that I talked about earlier. That is through modularizing our software, and now the autonomous teams own those modules. We can much easier scale the organization up. Because of the modularization, the complexity is reduced, and each of these modules has very well-defined interfaces for the dependencies to other teams. It's much easier to onboard a new team or two to put into that mix.

Then we empower the teams and we have them loosely coupled and highly aligned, so they walk into the same direction. The alignment is through our goals and the alignment via the Lean Value Tree mapping that we do. You find links here too to do some reading if you want to learn more about the Lean Value Tree goals that we do there. Then being able to decouple the software from the instrument here as well. This really hopefully helps us to release a smaller MVP and obviously deliver software much more frequently. In the process, we also focus on a lot of DevOps improvements there to enable the developers to be more efficient as well. Because we have frequent software deliveries and releases, we increase the quality because we shift the entire testing much more left. We get feedback much earlier, so we can react much earlier to that.

Let's finish up with the Third Way: experimentation and learning. That continues our journey in enabling software developers and improving our customer experience. We continue the journey of creating a scalable software organization, and, importantly, also redefine a governance model. Now that we have all these autonomous teams running around, you want to make sure everybody walks in the same direction.

First, enable software developers and improve customer experience. I mentioned that already. We introduced a platform engineering team, and that platform engineering team really has two main goals. On one hand, they are looking at whether there is a customer experience across the product portfolio that would create value for the customers, and we could focus on those. This is really more on the value side. The other side of that, and it also falls in that team, is around how we can enable software to be more efficient: enabling emergent reuse, which is really just evolutionary architecture and evolving our system to have less duplication. We can also look at additional process automation to bring our releases faster to market and then have the developers themselves be more efficient in the process.

We also continue our journey of creating a scalable software organization. What is really important here is we have those autonomous teams defining what their role and responsibility is, and we did that by introducing a playbook. These are just two aspects of that. One is the leadership team, and then we have many, many of those autonomous teams. They have their templates on roles and responsibilities, what they need to define, and how they measure themselves against them, and they report out on that.

Here's the governance template just as an example. In the lower left corner there are the roles and responsibilities, the accountables, and then the right-hand side is what they are doing and the activities for the governance that this team plans on doing.

All that brings us to the cumulative results for the last year. We look at 2022. In the first half, we really focused on the fundamentals, building the road for our digital transformation train to run a bit faster, and you can see some of the work that we do there. In the second half of the year, we really focused on onboarding a pilot team to roll out all these changes that I mentioned in the Second and the Third Way, helping us out there. We did that, and our plan is to roll those changes out to the rest of the new product development teams over the next three to six months.

With that, that brings me to the end. Thank you very much. If you have any questions, reach out to me. I'm always happy to talk about this. I'm very passionate about it. I hope you found it useful. Thank you.