Suzette Johnson

I'm Suzette Johnson from Northrop Grumman.

Robin Yeman

And Robin from Lockheed.

Suzette Johnson

And you may recognize our companies. If not, we are in the aero defense industry, and we build very complex systems for the Department of Defense, and we are competimates. And what that means is sometimes we compete, and sometimes we are partners. So today, we're coming as partners with a common message and a common set of experiences that we want to share with you.

Robin Yeman

Absolutely. So this is Suzette and I at maybe Interop.

Suzette Johnson

Interop.

Robin Yeman

But we tend to partner pretty frequently when we're speaking because most of the projects we work on are so similar. So take us to the reason we're here today.

Suzette Johnson

We're going to present Industrial DevOps.

Robin Yeman

Industrial DevOps. Now we couldn't really do something that we build, actually. So we had to do something that was similar in nature. So we partnered with a larger group in order to build something that would be similar. But we'll try to give you some examples from our environment as well.

Suzette Johnson

We want to talk about the discussion on how we build products, not just the software. So a lot of the folks that we work with have this perception that we're just going to be talking about software, and everything that we build, from the F-35 to the B-2 bomber and everything in between, requires software, hardware, and firmware. So we're really passionate about making sure that we can look at the entire value stream and how we can optimize that delivery.

Robin Yeman

Great. So what we did is we took a scenario, something that we could actually talk about and actually get through our external release process. So we chose this autonomous vehicle, this car, and it has a lot of similarities. Of course, it has different capabilities, which are the things that we can't talk about, but we can in this scenario. So we have an autonomous vehicle, and we're looking at the software, firmware, and hardware as part of this scenario.

And what does it mean to apply DevOps principles into this scenario? What does it look like? How could we start evolving this beyond software? Because one of the things we recognized is if we built all our efficiencies only in software, there were other parts of the value stream that were being ignored. So we want to take a more holistic or systems perspective.

Awesome. So we're going to talk to you, and we did. We took this from, or it's very similar in nature to, Scaled Agile because we've been working a lot with those folks as well, hence the scaling part. So we're going to look at collision avoidance and increase obstacle detection systems by 50%, which would be something similar that we would do, let's say, when we're building automated drones or autonomous drones. So the enhancement of our collision avoidance capability is going to be handled through three things: sensor firmware, a control system with software, and a user interface.

This would be very similar in nature to how we would break down an existing product. So we discussed two key epics, large cases, in order to demonstrate that actually Agile and DevOps works really well for the entire value stream, not just for the software portion. So we're looking at both the camera and updating the camera in future vehicles. And Suzette's going to walk you through the first set of practices.

Suzette Johnson

What we'll share with you at the end are where we've been publishing this. So the first thing that we did is we looked at what are the principles. What principles in this type of scenario would we want to implement or to align with in this situation? So we came up with eight of those. And what we're going to do in the next set of slides, the next set of eight slides, we'll walk through each of these principles and then how it's applied against the scenario.

Okay, so the first one being looking across the whole value stream. Again, this is where it became really important because what we noticed as we were applying Agile and DevOps principles and practices in software, there was the rest of the value stream that was being ignored. So we wanted to take this perspective of, let's look holistically. So with any of our transitions, one of the first things that you probably look at, even in your organization, is what are the value streams, and how do we want to get down into what we're calling the streamlet level in terms of delivering and defining these larger epics.

So in this situation, we looked at the vehicle as being the entire value stream. Then we looked at the safety and security as being a sub-component, and then zeroing in on collision avoidance. So that's what we're calling our value streamlet. So with collision avoidance, we have two areas that we're focusing on. Again, the software and the software updates to vehicles that are already in production. So we want to improve our collision avoidance system, and there's a set of vehicles that are already in production, and we'll make those changes, and we can actually field those. But then we have another situation: we want to make those improvements, but we want to make those improvements to the new set of vehicles that are going to be released. They're still in development. So looking at a value stream perspective, we're going to look at the software, but also the firmware and the hardware around that situation that we're wanting to improve.

So what do we do next? Well, we think about the value stream and the streamlet. We think about collision avoidance, and then we always start with the vision. What is the vision that we're after? And then what do we want to do? And we start taking a year look, a year-long perspective, again, both on the side of cars that are in production and with the vehicles about to go into production.

And from that perspective, we want to enhance and improve the camera. So with vehicles that are already in production, we're just going to make software changes that improve the collision avoidance and reaction into the system. But for the ones that are going to be released, we're actually going to do hardware-significant changes. So we're going to take a year-long perspective and think about both of those types of epics and what do we need to do both yearly, and then how do we start taking these quarterly views?

And the quarterly view is really important because we want alignment. We want to make sure the software teams are also talking with firmware, also talking with hardware as part of this collaborative effort. And then, of course, the more familiar that we get into are the iteration plans and the daily plans.

The other thing around this is because hardware is unique. It's not, of course, exactly like software. There's some challenges around that. Some of that's the longer lead items, which is also why we need to take a year-long perspective. And with those longer lead items, we have to get those scheduled, and then we also have to have targeted milestones of when we might be receiving equipment or how we'll be working with our suppliers. So that's why these multiple horizons of planning become really important and significant for these larger solutions that we're developing.

So once we're taking that perspective, then what we want to do is, as we're thinking about each epic, and think of an epic as something that's going to take six months to a year to create, a feature would be something we would develop within a quarter, and then stories are iteration, two-week-level type of backlog items. So what we want to do, though, with each of these, starting at the epic level, is think about what would we actually demonstrate.

So what you're seeing here is on the software side. It is looking at being able to, with that epic, drive the vehicle through multiple scenarios. And we can actually be able to test it, demonstrate, and show that we've hit those outcomes that we were expecting in terms of improvements.

On the hardware side, it might be looking at some prototypes. We'd have to be testing in some prototypes and getting regular visibility of progress against the prototypes and improvements that we're making on the hardware side. There might also, on the hardware side, be some research items that we have to do, or certain designs as well.

And then most importantly is architecture. So we understand at the software level and at the lowest level, architecture emerges. We get that. But also when you're building really large systems, what we want to do is think about architecting for modularity, and we have to think about that for the bigger system a little bit ahead of development. So here is our architecture, and with this scenario, we focus primarily on not just the improvements in the sensor management and the light detection, also the reactions and how it integrates both with the braking system, the sensor management, and overall vehicle control.

So as we're going through and we're planning across those multiple horizons and we're looking at the different backlog items, we can also then align it with our architecture and seeing where the impacts are in the architecture. So what does this start to mean? That means as the teams are planning together, we have to start defining the dependencies in the architecture and where they exist. Because especially on the epic two, where there's hardware components, we may actually start talking about modeling and modeling and simulation because some of the hardware might not be ready. So that would be part of how we're looking at this whole picture, from our architecture to our modeling and the modeling simulation approach that we're taking.

Robin Yeman

Okay, so I'm going to take the next couple. Again, many of the systems we build, very similar. So in this case, iteration and batch size is still critical. How many of you guys have had hardware folks on your team or have built hardware within your programs? How many of them come back and say, "I can't actually get anything done in a week or two weeks"? Yeah. It's not possible. And the reasons they tend to come back with that, I think, is that we're really looking at things like how to break down our work. I think that that's where the struggle is.

So Suzette and I are both very passionate about systems engineering and how to actually decompose your work because it enables this ability to do what we're talking about here, which is iterating and reducing batch size. So the first epic that we're talking about is that software epic in this case. And so I can start delivering some upgraded capabilities or evaluate those simply by injecting them into the system.

And we will be using, and we are for many of our systems, things like a digital twin. So digital twins do emulation simulation. So I can take a cyber-physical system and recreate it in a digital environment. By recreating that cyber-physical system in a digital environment, I can start buying down risk in all different kinds of areas in much shorter cycles. So we're going to do that with our software to make sure that we don't impact any of our current users, which would be critical.

And then we're going to iterate on our new camera. Now, just like Suzette was talking about, we can do some 3D modeling. We can leverage our digital twin, again, to determine the right camera to buy, the right way that it's going to interact with our vehicle, and we can do small demonstrations. This is, again, something that's been around for years. I keep telling the hardware guys who tell me they can't get anything done in less than two weeks, I said, "Actually, software did what software does, which is copy and paste this from you guys in the first place." Because if you're looking at it, it looks a whole lot like lean manufacturing, right?

So that's what we're looking at for our car when we've decomposed that down to get you something out and using that objective evidence that Suzette talked about earlier.

Next thing we want to do, very similar, is apply cadence and synchronization. The difference is not all of them have to be on the exact same. We want the integration points to line up, but if I do have a long lead item when purchasing, let's say, custom hardware, or let's say for something that we would build, maybe I've got to look at satellite panels and I need to understand how they impact with radiation. I have a long lead item.

So while I want to make sure that my integration points and my teams are all working on the same cadence, you could see here that I might give myself a little longer with that hardware. And firmware, I might say, well, those FPGAs or ASICs, I might give myself a little bit longer than I would with software. And when I get down to software, I may give myself more time, a longer time on, let's say, real-time embedded versus software that doesn't actually have any of those timing constraints.

So this is how all of those teams can actually be on the exact same value stream and be delivering objective evidence against the capabilities. Now, SAFe usually refers to this as the system sprinting as opposed to the team sprinting, and we just built upon that and said, "Hey, we want our car to sprint. We want to get objective evidence for that capability out soonest."

Now, the next thing that we talked about in our epic is really looking at how quickly you can integrate. So I've got teams that will never do continuous integration. I know not everybody loves the term continuous, but they'll never do continuous integration. F-16 would be a good example of that. It's built on an Ada baseline, and believe it or not, there's not a lot of automated test tools out there that support Ada. So while originally, they were integrating every three or four months, we got them down to integrating every couple of days. That's as close as they're ever going to get, but quite honestly, that's an amazing feat.

So that's why we talked about it with our vehicle, too. We're going to leverage the digital twin, and we'll allow our teams to be able to visualize updates to the code, we'll allow them to do all of the sensor management, things of that nature. But maybe for some of the real-time embedded, they're not integrating every day, or maybe they are. It depends on the environment and tools that we have available.

The new camera, that's going to be a much longer integration item. So as we're looking at that new camera device, we want to understand how it's going to fit with the car. We need to understand how it works with our current APIs. Any kind of mechanical issues that could possibly happen when we attach it to the car, and all of that's going to have to be prototyped. So that lead time, they're not going to integrate every day, but maybe they could integrate with prototypes once a week, or maybe they could integrate every few weeks, or maybe they can integrate quarterly. The bottom line is, the faster I can integrate whatever I'm building, the less risk I have.

So I'm not trying to get away from getting that fast feedback, and that's kind of the thing that we've run into with a lot of our larger systems. So that's why we called it continuous. I would tell you, if you come up with a better term, let me know, because I have heard that bashed a little bit here and there.

Test-driven. So be test-driven. Begin with the end in mind. This also frequently comes up with our hardware engineers. They're like, "We can't start with the test." I'm like, "Well, why is that? If I don't know how I can prove what I'm building, then how am I building it?" And what happens, you probably have seen this with some of your teams, is they keep building until they've got something done, and it could be that it's exactly what they need, but usually there's more in there. It's like Prego, it's in there. So it's not quite as clean. I probably built some extra things that I may not have needed.

So with our approach, we're going to have them use acceptance test-driven development. So we're going to start looking at how I'm going to validate and test this work before I even begin updating my software or my hardware. And using things like MBSE is going to allow me to test that model out, looking at how I'm going to do validation, even real validation. So in person, I want to make sure that I actually have two cars and I can determine what that collision braking distance looks like. So beginning with how I'm going to validate that versus trying it afterwards or just testing it afterwards is going to make a better product. And we didn't invent this. Actually, Joe Justice showed us exactly how to do that with building a vehicle, if you guys are familiar with that.

Suzette Johnson

Yeah, so a couple items. So first of all, the importance is going back to the why. Remember in one of the earlier slides, we said why are we doing this? And some of the points we touched on was the need to burn down risk and looking at flow and feedback across the whole value stream, not just in software.

We have a couple more things that we want to talk about. One is our contributors, and Harry is actually here with us. Josh is somewhere around. I'm not sure the others. I haven't crossed paths with them yet at the conference. But we appreciate their input in helping to write the paper.

Robin Yeman

Actually, we got a couple more. So Steve Mayner's actually here.

Suzette Johnson

Oh, you're right.

Robin Yeman

And so is Dean Leffingwell. So we have a team with a lot of passion around DevOps, and we got some really good input from all these guys.

Suzette Johnson

Yeah, so help that we're looking for. As I mentioned at the beginning, we had those eight principles. So year one, what we did, the group of us, is to figure out what were the eight principles that we wanted to talk about. And we defined what those eight principles are and why they were important as part of what we're calling Industrial DevOps or DevOps for large cyber-physical solutions.

This past year, we did another article, which was taking those eight principles and then putting them into the scenario with the autonomous car so we could start looking at how would you actually implement those principles. So next year, we want to write the next paper. And here are some things that we are thinking about, and we want you to help us figure out what that next paper should be.

So based on what you're hearing, thinking about, well, that's really great, but it'd be really helpful if, and then you help us figure out what that if is. Is it more metrics? Is it more systems engineering? Is it how we actually trained and we get the people up to speed and the skills and the technologies needed to do this?

Robin Yeman

Yeah, and any kind of what are all the aspects of this value stream, the content? So things that we're being impacted with right now, just as a side note, is the government's making massive changes in the acquisition policy. So we've been working closely with them as they do that. Why? Because we need to get capabilities out to the war fighter quicker.

So when I look at this, I want to build useful content that will enable our programs who are building capabilities for the war fighter to be able to get something out there quicker. And things that I think would be incredibly helpful is, okay, so how do I decompose a system where I can show vertical slices? That's one way. So we want to look at maybe there's some additional systems engineering we need in that content. Another thought is how do I do test automation? Are there other practices? So is that something that would be valuable and useful to the greater community? Can we build some of that?

Could we look at improving how we get those cross-functional teams? Or do they need to be completely cross-functional? So that might be a place where we could expound upon this. Or should we try to put more systems through the paper? So the first year, as Suzette said, we created the principles, kind of walked through it. Second year, we actually took that and applied it to a vehicle. Maybe we need to look at applying it to something else.

I keep bugging her about this awesome CubeSat that's out on GitHub that I'm like, oh, we could build the CubeSat, we could 3D print it, and we could update the real time with Arduino. I'm all in. So do we want to push something else through the system? And the reason I'm excited about that is because I actually work for LM Space. So it's something that I think would be near and dear to a lot of my team's hearts.

So I think there's a bunch, and how do we want them to give us that data? Like either email or do we want to crowdsource it?

Suzette Johnson

Or connect on LinkedIn.

Robin Yeman

Or LinkedIn.

Suzette Johnson

Also if there's anybody who does have passion around helping to build this content, you want to join our team, we're always looking for help. That would be awesome, too.

Robin Yeman

So here are the papers, and they're actually here. They're printed here. These are the two ones we did, defining the principles and applying them. And I think that you can access them both on the website as well as get a published copy here, but I'm not entirely sure about the published copy. So we probably have to go look at that.

These things, I believe, are going to add value to the ways we're working. When Gene talks about the way, one of the best things that we've seen is the huge transition in how people are delivering software, like the unicorns. The thing that I think would really, really help is taking that way of building our digital products and adding that to how I can do physical capabilities, real live physical things that you can see and touch. So how do we apply Industrial DevOps to building ships? Or how do we apply Industrial DevOps to building radars? How can I get capabilities out the door quicker, no matter what domain you're in, whether it's in the government domain or the commercial domain, because we think the life cycle's getting shorter all the time.

With that, we actually have a few minutes for questions. We probably went a little fast. So does anybody have any questions about this? Any interest? Yep?

Q&A

Audience

So how will you get the DCSA security process?

Robin Yeman

It's a good question. Might be another piece for the paper. How do you get security aligned with these principles? So while we're giving faster capabilities, how do I make sure that I'm building that security both into my software and the hardware? And you're right, that is also something that we've been wrestling with because a lot of folks will say, "We're building safety-critical systems. How do I get a safety-critical system out the door?" Now, the answer to that question is we haven't resolved it yet, but it does seem like a very good option for the next generation of this paper as well.

Suzette Johnson

Yeah. I was just going to say, in general, we build the security and the security requirements in, but when it comes to external entities that need to work with us, then that's a collaboration opportunity.

Robin Yeman

Yeah. There's going to always be flight qualification. There's always going to be external folks that are accrediting our work. So that makes sense.

Suzette Johnson

Yeah. And in some cases, they have engaged, depending on what the system is and who the entity is that is securing and reviewing the software. In some cases, they're more engaged, and they understand what we're trying to do. So that helps. In some cases, there are, like I said, still opportunities.

Robin Yeman

So within Lockheed, I can tell you a couple of the systems that we've actually been applying some of this to. So fleet ballistic missile would be a good example. That's across the U.S. So we have them in Sunnyvale, California, as well as down in Florida, and they have been doing a lot of this for their hardware.

Aegis, which is one of our biggest agile teams, there's about 190 teams that are building that combat system. We also have F-35, just portions of it, but you're going to see over the years that F-35's going to transition much more into this type of capability. F-22 actually has already done a few presentations about how they integrated the software, the hardware, and the firmware cycles. They're not perfect yet, but you can see that they're also building out content. So a lot of this work is very relevant to what we've been doing.

All right. I think we're good.

Suzette Johnson

I think so.

Robin Yeman

Okay. Thanks, guys.