Peter Fassbinder

Welcome, everybody. I'm quite happy to be here and to share our experiences on applying DevOps in industrial environments. And the topic I'm going to talk about, from milestones to continuous quality assurance, is a very crucial aspect in this context.

Before I start, just quickly a few words on myself. My name is Peter Fassbinder. I'm a principal expert for PLM Process Innovation with Siemens Advanta Consulting. I have considerable experience, more than 15 years in Lean and Agile, and now for more than five years, I'm almost exclusively dedicating my time to the topics of continuous delivery and DevOps, and especially to the question: how can they be applied in industrial or regulated environments?

I'm also quite active in the external community, so I'm always happy to share my experiences at conferences like this.

When we want to move in this direction, a company like ours or other industrial organizations, we really need to address what's been shown on this picture: stop thinking projects; start thinking continuous value stream change.

And this is a significant move. You can see this if you really look at where a typical industrial organization is coming from and where we want to move towards this more frequent deployment.

A typical industrial organization is coming from a project-based approach, with projects running anytime between maybe six months, maybe one year, maybe even more, up to one and a half years or maybe longer, in between releases of new versions.

Now we want to move toward these more frequent deployments. Even if you're not talking about daily updates, but, for example, talking about product enhancements with every sprint, every two weeks, for example, or maybe even only once per month, we're always talking about at least an order of magnitude more frequent than where we're coming from.

And this is a tremendous and significant change.

What is also important, especially for the rest of the presentation now: I'm talking about this continuous value stream, which is shown on the right-hand side of the lower arrow here. On the left-hand side, we have something I called Lean Startup, which means we already have released a minimum marketable product into a productive system. We had, hopefully, a successful initial market launch, and now we are addressing the question: how can we implement frequent deployments, updates, enhancements of this minimum marketable product while it's already in production?

And this is the basis that we assume for everything we're going to present when we talk about this continuous quality assurance flow.

Moving in this direction towards more frequent deployment, as already pointed out, has a significant impact on a lot of the topics and areas within an industrial or regulated organization. This ranges from the toolchain, from product architectures on the technology side, from new and modified processes, responsibilities, organizational changes, potentially, all the way up to potentially redefining your business model, your relationship to your customer partners.

What I'm talking about in this presentation, on this continuous quality assurance approach, is one topic within the processes box here. But it's a very important topic. And it's a topic on how can we ensure and move towards a more continuous quality assurance approach in this so-called more continuous world of continuous, frequent updates of the live system.

First, I would like to show you a few slides on the core idea, the core concept behind this continuous quality assurance that we developed in the last couple of years, and by now have implemented successfully in several organizations. Then I'm going to dive a bit deeper into how this can be managed efficiently in practice.

This is a very important slide, and this kind of contains the key idea of the continuous quality assurance approach. It's all based on the green-to-green mantra, which I hope you're all familiar with.

On the lower part of this picture, labeled continuous integration, that's again something where I'm pretty sure you're quite familiar with it. It shows, basically, the shift from a project-based approach to a more continuous value stream for the software.

In the past, software was somehow created over the course of a couple of months and verified and validated. And eventually, at a date towards the end of the project, you had running software and implemented it into the productive system.

Today, with a continuous DevOps green-to-green approach on the software side, this looks very different. You already have a running system up in production, and now you want to implement a very small change on a frequent basis into the productive system.

So for every change you implement, there are two things you need to do. First thing is, of course, you need to identify whether it's doing what it's supposed to do, for example, some functional testing.

But what is much more important is actually to ensure that this change does not break the running software, that the software system that is in production after you integrated the change is still up and running, is still green. This is the green-to-green approach.

So this has the uppermost priority in this continuous integration, continuous delivery approach. And this is something where, of course, in the past decade, tremendous progress has been made on the toolchain, on test automation, on the continuous delivery pipeline, that ideally, at the push of a button, every developer, every team, with every code change, can immediately identify and verify whether the overall system is still green.

So this is great, and this is, of course, something you also need in an industrial environment. But it's not sufficient.

Even if you have a perfectly running, fully automated toolchain, and with an instant, at a push of a button, you can identify whether your software system is still running after you implement a software change, you're not in a position to really deploy this into the productive system.

Why? Because if you want to deploy software in an industrial environment, you need to deliver a lot more than functional software. It's not sufficient to have software alone, even if it's flawless, with the highest quality. You need additional things to deliver.

And I like to use the term non-software artifact for it. This comes from a variety of sources and requirements.

First of all, there might be, for example, user documentation that the end user needs, maybe some frequently asked questions, maybe some online help. You could also potentially have some customer service personnel that require some artifacts, some installation guidelines, and so on.

And you also have some documents or artifacts that you really require for yourself or by regulations. We're talking here about maybe patent applications, or export control, open source, risk assessment, and all things like that.

So there are a lot of, as I pointed out, I like to call them non-software artifacts that also need to be available. They need to be in a valid state that is actually matching the software you want to implement. And this needs to be available, verified, validated at the point in time you want to deliver software, deploy it into the productive system. It's mandatory to have them. You're not allowed to deploy the software without those artifacts.

If you're looking now again at the project-based approach, the mechanism to ensure that before you deploy, you release something after a large project, was exactly the so-called milestone or quality gate approach.

Now we want to move towards a continuous value stream. It's not possible anymore to apply this milestone quality gate approach with every deployment because it's just too heavy. It wouldn't be feasible, it wouldn't be efficient to really manage this with this approach.

So you really need to redefine the way you look at this quality, addressing those topics, and how you release it.

What we now applied is actually we said we're going to take this green-to-green approach, which has proven to be very successful to ensure the quality of the software, and we also take it in the same way, and we ensure by this the quality of the non-software artifacts.

We label this continuous conformance, and it basically includes two things.

First of all, you need to make sure that you identify which of the non-software artifacts is affected by a certain change you want to implement, meaning you need to identify which of the non-software artifacts requires an update. Then you need to ensure that this update is completed and done before you deploy it into production.

And this is again shown on a high level on the yellow part of this picture. What we want to achieve is a continuous, or you could say frequent, deployment into the productive system, and we need two equally important streams to ensure this.

We need the continuous integration stream that ensures the quality of the software as such itself. But in addition, we also need the continuous conformance stream that ensures the quality of all corresponding non-software artifacts.

Both are equally important. Both need to be green. And only if both are green for a specific change or new feature can we implement this into the productive system.

An interesting thing in here, which is shown in blue, is that when we started implementing this approach, we identified that quite a number of so-called quality criteria, which were in the past covered by these milestone checklists, do not need to be tied to this high-frequency continuous deployment schema.

Examples are, for example, something like a test strategy or marketing flyers and things like that. Of course, they also need to be kept up to date. They need to be addressed. They need to be managed. But they don't really need to be tied to these individual deployment decisions.

And this has two effects. First of all, it makes a continuous conformance stream, the high-frequency stream, the yellow part in this picture, leaner. There are fewer artifacts we need to take care of in these high-frequency activities.

But on the other hand, we must ensure that we don't forget those artifacts. We need to define appropriate mechanisms to handle them more on an organizational program level.

All this taken together is what we call the continuous quality assurance approach.

Now, for the rest of the presentation, I will dive deeper into this continuous conformance stream because this is what is really the novel part in our approach.

If we dive a little bit deeper into it, another thing we covered is that it's not sufficient to only apply the green-to-green mantra. We also need to think about applying the shift-left mantra also to these non-software artifacts.

Why is this the case? I can explain here briefly.

Initially, we thought we want to achieve these frequent deployments. We have, for example, a two-week sprint. We have a continuous integration toolchain already implemented multiple times per day. This is running. So we have a lot of small cycles where we ensure the quality of the software, and by the end of a sprint, we can, from a software point of view, be in a position to deploy into the productive system.

Now, our initial thought was, well, if we have potentially two weeks' time, at the beginning of the sprint we can identify which of the non-software artifacts are affected by the changes we want to deploy. Then we have two weeks' time to address them and to update the corresponding non-software artifacts that are affected. And then by the end of the sprint, we have them ready.

But we very quickly realized that this is not possible for two reasons.

First of all, some of these non-software artifacts, for example patent applications, if they require an update, this takes a long time, more than a sprint typically. So it means we need to start earlier, so we have completed this update by the time we want to deploy.

The second and more important reason for doing these things earlier is that a lot of these non-software artifacts are actually input for development. And then, of course, if you say a certain change requires an update of this, you need to ensure that the update is being completed before you actually start development.

And this is the reason why we said, in addition to applying the green-to-green mantra, we also need to think about appropriate ways to apply the shift-left mantra also to the non-software artifacts.

Very important: this does not mean that we do a huge upfront, 12-months-ahead planning analysis and update of these non-software artifacts. It rather means we need to do this earlier, one, two steps or sprints ahead of the software development, and on a rolling basis. Just sufficiently early so the development team always has sufficient backlog items that are ready to develop in their continuous integration chain.

And if we dive even a little bit deeper, again, the world turns out to be more complex, as always.

We also identified fairly soon that these non-software artifacts can be quite different from the requirement when you need to do the analysis and when you need to have the update completed, if an update is completed.

Because these non-software artifacts are from very different sources, very different stakeholders, they are of a very different nature, this turns out they cannot be all treated the same.

We identified four different types, which are shown schematically on this slide.

Type number one is, for example, patent applications, that really require a long time to do, always under the premise if an update is required. So this should be done really early, and it also needs to be ensured that this is being completed early enough. If you're applying, for example, something like a scaled Agile framework, you have, for example, the PI planning, and in this case, we really recommend that this should be done before the PI planning.

Type number two, where actually most of the non-software artifacts fall into, are non-software artifacts where the update of the artifact is required before development starts, which means before the sprint starts in this picture, because it is an input for development. For example, architecture documents, risk assessment, UX specifications, and so on.

Type number three is something, for example, user documentation. In order to update this non-software artifact, you really need to have the software change already implemented because, for example, some screenshots are required. And of course, in this case, you cannot update it before development starts, but you really have to update it during the sprint and make sure that this is being completed before you want to put this into production.

And type number four depicted in here is, for example, open source, where you're only, during the course of really detailed design or potentially development, identifying whether an update is required. And then also, even the analysis can only be done in parallel to the development.

So this is getting a bit more complex, and we introduced something like a ready-to-development entry quality gate, which is a lot more than a typical definition of ready of a backlog item. And it actually means that all these green tick marks for every change, they need to be completed. These analysis and update steps need to be done before actually the team is allowed to put them into the sprint planning, in order to make sure that by the end of the sprint, when really from a software point of view, everything is done, everything is fine, everything is verified and validated, there's nothing left over on the non-software artifact point that stops us from deploying this into production.

What is very important if you move in this direction is that you find a way to really efficiently manage this continuous conformance, and this is really key to success. This was something we already looked into from the very beginning.

And this is something like a real analysis we did quite early, where we looked at two teams, we looked at their features, and we went through all these features and through all the continuous conformance criteria, which equal non-software artifacts. And we looked at them and we said, for which change feature do you need an update of which non-software artifacts? And they are marked in blue in here.

What you can see in here is something that we already assumed, but we were quite happy when it was confirmed. That is, typically only about 10% to 20% of all possible non-software artifact updates are really required.

Why? Because you have a lot of changes where you don't need to update a lot of the non-software artifacts, or maybe for some changes, you don't need to update even none of them. And this is, of course, what makes this whole continuous conformance, continuous quality assurance so attractive: that we really say we very efficiently want to identify which non-software artifacts must be updated. And it's typically only a small set of all possible ones.

Then we really want to make sure that we can identify and track these tasks in the most efficient way, in order to make sure that they are completed, and we have a clear transparency on this at the point in time we want to deploy.

And this is the second thing that was very clear to us from the very beginning: that we cannot do this manually or in any Excel sheet, but we really need a toolchain. We need to do this, and ideally, we should integrate it into the toolchain, which is typically already existing in an organization for the software and the software-related artifacts.

By now, we have done this for three different commercial off-the-shelf toolchains: for Rational Team Concert, Jira, and Azure DevOps.

Why those three? Because in those organizations where we have implemented this approach so far, we have come across those application lifecycle management toolchains which were already in place and which were taking care of the features, of the backlog items, of the user stories, of the test cases, and so on.

And what we did now is to integrate the continuous conformance approach in this toolchain, as again shown schematically on this slide. On the left-hand side, you can see a typical requirements breakdown structure, where you have, for example, a feature and probably already a number of user stories underneath it. And from a software point of view, it's very clear: you have to complete all the user stories, and then the feature is complete and you can implement it.

And now we say on the same level as the user stories, we want to attach so-called continuous conformance stories in those cases where an update of a non-software artifact is required.

So we could have, for a specific feature, anything from zero continuous conformance stories, to over a handful, to a whole lot, really depending on the impact this change has on the different non-software artifacts.

And now what is also shown in here with some screenshots is that, of course, in this tool, we have to do these two steps. The first step is analysis: do we really need, or which of these non-software artifacts need to be updated for this specific change?

Then we need to track this, to trace this, and to update it. And in all those cases, it looks different in the different toolchains, but from the core idea, the implementation is the same.

We have created templates with a predefined list of these continuous conformance criteria, or you could say non-software artifacts. And in the analysis step, very easily just by clicking on those, you say, well, those need to be updated for a specific feature or change, and the others not.

For those that are updated, these continuous conformance stories are then created. And then, of course, in these toolchains, you can easily track the status of them. You can create dashboards and really monitor the progress.

What we really want to have in here now is that we say we have one integrated toolchain where we have the software tasks, we have the non-software artifacts. They are all linked on the feature or change items you want to deploy. Ideally, we have two or a combined dashboard where you really can say whether the software and the non-software parts are what is still open, what is closed. And if everything is closed, if everything is green, you are in a position to really deploy this feature.

So coming towards the end of my presentation, what I've shown you is how we really apply this green-to-green and shift-left mantras also to non-software artifacts in order to create a more continuous quality assurance approach.

We talked about the deployment strategy, about continuous conformance criteria. I also showed you about the importance of integrating this into the toolchain, of creating dashboards.

What I have not shown due to time constraints is that, of course, you also need to think about how this can be implemented into your processes, in which meetings you can do the analysis, the tracking, what are the changes, modifications to the roles and responsibilities.

And what is also a very important point, because even the continuous quality assurance approach just by itself is such a significant change on the way you're typically working, is that you should really have a CEO-driven, high-level vision towards this more continuous world. You need to have a suitable change management and coaching in order to drive this forward across your organization.

As a very high-level summary, you could say, of course, whether you're an industrial, regulated, or whatever company, if you have products, your customers would be very happy if they would get better during the course of their lifetime every day they use them. But it needs to be made sure that it's done safely, securely, and with the highest, in our case, industrial quality.

Thank you very much for your attention.

I brought two further links with me, two further sources where you can dive a little bit deeper into it. The first one is a white paper I've created some time ago that not only touches the continuous quality assurance topic, but also some other topics that are relevant if you want to apply DevOps in industrial environments. You can download it via the link or the QR code.

And also, what I just presented in the past 25 minutes, from milestones to a continuous quality assurance flow, introducing the continuous conformance concept, this has also been an article that is very soon published in the Spring channel of the DevOps Enterprise Journal.

So as I pointed out earlier, this should not be a one-way exchange. I'm really very happy to discuss these topics further, to address your questions, to get your feedback, to please challenge me, and also provide me with your experiences.

What are your experiences with this topic? How do you address quality in your organizations, in your company?

Please feel free to contact me either directly or via LinkedIn. I'm really happy to exchange any thoughts.

So again, thank you very much for your attention, and I wish you all the success if you want to apply similar approaches.