Can the National Imperative for the Department of Defense Accelerate DevOps Adoption
For the first time in decades the US department of Defense is managing adversity on two fronts. Our physical military might enabled us to support global order in the aftermath of World War 2. However fast forward to 2022 and the world has changed, we live in a digital ecosystem where physical military systems are all powered by software. Many papers propose we are behind our adversaries in modern software development which can impact our military advantage during a time of global unrest.
The Department of Defense (DoD) is doubling down on DevSecOps with more than 20 software factories across the DoD. Over the last 3 years we have experienced more change across the DoD in how we build systems than we have in the last 25 years. I believe the National Imperative to enable security across the globe could enable the DoD to leapfrog the commercial industry in software development. I am proposing to co-presentation with one of the government SW Factory Leads from Platform One.
Chapters
Full transcript
The complete talk, organized by section.
Robin Yeman
Robin Yeman: Hi, my name is Robin Yeman, and I am here to talk about Agile and DevOps within the Department of Defense. I would like to introduce you to a friend of mine who is going to partner with me on this presentation. Duong?
Duong Hang
Duong Hang: Hi, everyone. My name is Duong Hang, and I'm delighted to be here. I appreciate Robin inviting me to have this great conversation and talk to all of you about this.
Robin Yeman
Robin Yeman: I'm going to just go over this abstract really quick. At a high level, for the first time in decades, I want to tell you that we're experiencing adversaries on two fronts. It really means that we have to double down and move at the speed of relevance. We have to go faster.
I want to talk to you about what I've witnessed within the Department of Defense. I personally have worked within the government ecosystem for the last 26 years, and what I've seen in the last five I think is pretty impressive.
The first thing I want to do is anchor you in Kotter's Eight Steps for Change. John Kotter wrote a book, and when you take a look at it, he talks about these eight steps that allow us to make change. The first one is creating a sense of urgency. We want to make sure that there's a reason for the change.
From there, he talks about forming coalitions: working groups of people that can help collaborate and move the change forward, because change isn't easy. Then we want to talk about creating a vision. We need line of sight to those common business objectives for everybody to be able to take a look so that we've got at least a common goal, a common path.
Number four is communicating that vision wide and far, probably farther than you would think. From there, we want to remove obstacles to change and then look at generating short-term wins. From there, we're going to build on that change and then make it stick. These are hard things to do. One of the things that I've noticed is, as we're making this change for software delivery within the DoD, it looks like whether intentionally or unintentionally, we're actually following these steps.
Duong Hang
Duong Hang: Hey, Robin. I just want to add, I think one of the challenges, and you're right, a lot of these organizations, especially software factories, have been following some of these steps. Oftentimes what happens is you're doing bits and pieces of each step and not necessarily having a concerted effort where you're trying to build upon each one. If you don't do that, sometimes you will regress in some ways. Oftentimes you have to cycle back and revisit some of these other steps previously because there are things that have changed or matured, and you need to continuously make those adjustments over time. So take this as an agile approach versus a waterfall step approach.
Robin Yeman
Robin Yeman: Absolutely. That's a great point. I think you're going to see this throughout, that we are revisiting iterative and incremental.
In 2018, James Mattis came out and talked about the need for urgency. In order to keep pace, the department needs to transition their culture, which is huge. The DoD is one of the largest employers in the world. He talks about success not going to the country that develops the tech first, but rather the one that best adapts and integrates it. The downside is he talks that our current bureaucratic processes are insufficient or unresponsive to being able to make that change. He's trying to set us up to create a culture or a climate for change. The reason that's so important is because our adversaries are moving forward, and we need to make or keep our competitive advantage.
Duong Hang
Duong Hang: Absolutely. One thing I want to add, too, is that in the past, especially on the DoD front, the U.S.'s superiority is because we were able to build hardware better and faster because we had the money to do so. Now that software is sort of a great leveler and everyone has access to that capability, now you have to find ways to actually be competitive. It's not trying to build something that no one else has. It's actually taking, adopting what's out there already, moving faster, and it's the speed in which you integrate, like Secretary Mattis talked about, that is the competitive advantage. No longer is it like, I keep secrets of how I did something and it's the formula, but actually having a culture and the frameworks to actually build quickly and change quickly. That's how you stay competitive.
Robin Yeman
Robin Yeman: Absolutely. Thank you for that.
The next step we talk about is forming a coalition. The one here that I'm describing is the Defense Innovation Board, and they were started in 2016 to bring technical innovation and best practices in Silicon Valley to the military. I know there's a number of others. Duong, do you have any thoughts on what you've seen in this area?
Duong Hang
Duong Hang: There's been a lot of changes throughout. I've been focusing a lot within the Air Force myself. One of the things that we've seen happen is there's been a standup of the DSOG working group that is basically a set of leaderships from each of the military services coming together and really looking at how do we do security across the services so that we can actually share and inherit each other's security posture, versus trying to build the same thing over and over again and then basically having to check other people's work. There's this really great movement to collaborate across the DoD and actually share the results of the work that everyone does really well.
Robin Yeman
Robin Yeman: Awesome. The next step within Kotter's change is creating a vision, and we saw this happen, or at least begin to happen, in 2019. The Defense Innovation Board came out with a terrific article that talked about software is never done. From there, they also reported on some key themes and lines of effort. Their key themes are speed and cycle time matter, and we know that; software is about people, and that I sometimes think we forget; and software is different than hardware.
Some of the lines of effort that we need to do to make this change: refactor the statutes and the process, which is what I think the acquisition department's working on; creating a digital infrastructure; building new paths for digital talent; and changing the practice, so DevSecOps.
We've done a great job of communicating the vision. I'm showing you the initial Defense Science Board report in 2018 that came out. Shortly thereafter, Eric Schmidt posted a report that he delivered to the House Armed Services Committee. Then we got to the DIB, which is May 3rd, 2019, of software never being done. From there, the DoD jumped into building a digital modernization strategy. Then one of my favorites, Dr. Roper, came out with "Take the Red Pill," so what do we need to do to get to DevSecOps? Just recently, we've seen the software modernization strategy get published. Duong, which ones have you seen or found influential in this communication?
Duong Hang
Duong Hang: I think the main one that resonated with me quite a bit was when Dr. Roper's "Take the Red Pill" document came out. It was part of a whole orchestration of change that was happening, especially within the Air Force, to really push along how do we do software better and make that be the competitive edge for the DoD. That's already been happening in the commercial sector for so long, and now the DoD is just like anything else. We're catching up to what the commercial sector is doing. I think that was a very seminal document in terms of really outlining a vision for how the Air Force and the DoD can actually modernize and take advantage of the fact that the digital environment that we're in is a new space for competition and superiority.
Robin Yeman
Robin Yeman: Awesome. Next step in Kotter's change is removing obstacles, and it was very important to start looking at the acquisition approach. The weapon acquisition approach within the government is long and too slow. In 2019, they developed a new software acquisition pathway that allows people to iterate and incrementally deliver MVPs, minimum viable product. Then they added a new term called MVCR, which is minimum viable capability release, because they wanted a mechanism to show when it was going to be deployed to operations. But there's still more work to be done. What kind of things do you think, Duong, we could look at?
Duong Hang
Duong Hang: I'm going to challenge that definition of MVCR and MVP, to be honest with you. I was never a fan of that statement because it really started trying to gel, or it was trying to pull back the idea of DevSecOps and Agile and pull it back to the waterfall model, because it's basically separating out the differences between actually building something and actually getting it in the hands of users.
Really, from an Agile perspective, it's about getting feedback quickly, and the real feedback only comes when you actually get the products in the hands of your users. In the industry, I think they never came up with the term MVCR. That was the DoD term. But really MVP is what it needs to stay as: the minimum viable product for minimum viable learning. In order to truly learn, you have to get something into the hands of users. It doesn't have to be something that you build where it's in the system, it's software. It could be just a mockup, it could be a prototype, it could be just a presentation, a briefing. The point is that you want to have minimum viable learning. A minimum viable capability release is a little too late at that point.
Robin Yeman
Robin Yeman: That's a good point. I think in any step or any time we want to make change, people sometimes straddle the fence before they make the jump. I agree. Definitely focusing more on the MVP.
The next step in making change happen is creating short-term wins, and I'd be remiss to not talk about the great story of Kessel Run at the U.S. Air Force's digital journey. The Jigsaw success story really talked about that air tanker fuel application that was very manual and took many hours to actually update daily, to taking something where they were able to automate that air tanker solution. From my understanding, they were able to build that solution in months and actually receive return on investment, like pay for that solution, in weeks after it was deployed. It was a great news story.
Duong Hang
Duong Hang: Right. Obviously they were the poster child for the Air Force's foray into modern software development and just modern approaches to how we actually improve our capabilities. I was personally inspired by their work, like many, and that led me to where I am today with Platform One. I think one of the takeaways of a short-term win is that you have to use that as a means to justify further investment and time and people's energy. Then also in terms of the value definition, you can't just limit it to, "Hey, well, I saved some money," because that doesn't resonate with a lot of people. It's really about saving time, even to people at the lowest levels. Oftentimes we try to say we save time for maybe the mid-level management or senior-level management, and it's really about saving time for the people that are at the lowest level, because typically that's where most of the workforce is at. If we can do that, we can actually take that and translate it into other places based on that win.
Robin Yeman
Robin Yeman: Awesome. The next step we're going to talk about is really how to sustain that change. Since you know so much more about it than I do, Duong, I'm going to have you talk a little bit about these software factories that have just exploded across the world.
Duong Hang
Duong Hang: Thanks, Robin. We were all in the middle of it. I've been with the DoD for almost 20 years now. I've never seen anything like this, and actually, it's been really encouraging to see that even though the explosion of these "software factories" has come out, I think the key thing, and there are some concerns from senior leadership that, "Hey, do we have too many? What are they doing?" But at the same time, I think that's just sort of the concerns at the senior levels.
If you look at it from another lens, there is a desire to make things better at the grassroots level. Maybe there are lower-level managers that are empowering people to get together, build something that they think is what's needed for the defense of our nation, and they're doing it. They're actually not just doing it, but building a group, building an identity with the logos. I think this is such a positive development in the last few years within the DoD, that people actually are wanting to make things better, and they're not necessarily waiting for senior leaders to actually get at it. This is that representation of that.
If you look at this map here on the slide, they come from all places across the U.S. I visited a couple of them. They have very different cultures, and I think part of that is because they have a different mission set. We talk about why do we have too many, and there was a recent article in Breaking Defense talking about the Deputy Secretary of Defense wanting to maybe reduce it to a smaller number, maybe even one. The question is, why do we need many versus one? What's that really mean?
In my opinion, and a lot of things that I've read, software development and developing capability from an Agile perspective means delivering value. The way to deliver value is to actually bring the products you build to the user. Each user, every user, is going to be different. There are groups of users that are similar, so we can identify those groups, but they're going to be different groups. In order to make sure you deliver value, you're going to have to deliver software that's unique to that set of users, which means you need to have a custom-made software factory to build that.
I don't believe personally that we should have just one software factory. There's going to be a spectrum of them. They're going to have niches, and they're going to really get after the value that's needed for the user. Some people question, why do we have so many military services? Why can't we just have one military service and it does everything? I think if you actually spend some time in the military and the DoD, it's really about having a set of capabilities tailored to the domain they fight and then bringing it together when we actually do the fight to actually see profound effects across those domains. That's the same thing, I think, with these software factories as well.
Robin Yeman
Robin Yeman: That's excellent. I'm super impressed by the changes that have been made, and I really appreciate your opinions on these because I do think that we're going to come back, we're going to iterate, we're going to incrementally make change.
Duong Hang
Duong Hang: Absolutely.
You're mute, Robin.
Robin Yeman
Robin Yeman: The last one we're going to talk about is all about Platform One. Can you tell us a little bit about what Platform One is, what you've seen so far, and maybe what the future looks like?
Duong Hang
Duong Hang: Platform One is an Air Force organization that has an enterprise DoD and federal, and even commercial sector impacts. What we are is an organization that builds DevSecOps capabilities, and we also can provide a managed platform-as-a-service capability, which means we take software pieces, we build, we add to it, and then we build one of those software pieces for people so they can build on top of that, build applications that their users are going to use to get some kind of value.
The organization started about two years ago. Since then, we've grown quite a bit, we've matured quite a bit, and we've been providing capabilities for a lot of Air Force customers out there, Navy, the Marine Corps, the Army. We partner with them. We've also had a lot of anecdotal stories about the commercial sector actually using a lot of our capabilities today, such as some of the software containers that we bring in from open source and the commercial. We harden it, we scan it for vulnerabilities, and then we provide it as a more secured, trustworthy source of software that you can build on top of. A lot of organizations have used that.
We also have a capability called Big Bang, which is basically a set of software that configures all those containers in a way that allows you to build a pipeline for your software factory to build your applications. We've provided that as part of a suite of free services and capabilities as well. Finally, we have this thing called Party Bus, which is essentially taking all those things I talked about, the containers in Iron Bank, the Big Bang code, and integrating it, and then delivering a managed service, a place, an environment where software developers that don't want to find software to build off of or put it together, they just want to build their application, they can go there and actually build it and try it. That's something that I've been so fortunate to be a part of, and that's kind of what we do today.
Robin Yeman
Robin Yeman: I'm a big fan. One thing I'd like to ask you just a little bit more about, because I don't think that folks maybe in the commercial sector have had this to deal with as much, is continuous ATO. I've been told in many cases it's more expensive to get authority to operate for a system than it is to build and maintain it. Do you have any words of wisdom on that?
Duong Hang
Duong Hang: I don't know about wisdom, but a lot of heartache and scars to show it. I think there are a lot of anecdotes about sometimes it actually costs a lot more, sometimes double, to actually get something approved for use rather than the cost of actually developing it in the first place. I'm not going to name any names, but there's definitely a lot of anecdotal evidence that it does cost a lot of money.
I get it. In the DoD, in the military, obviously security is really important, just like a lot of the commercial sectors in banking and so forth. But we can't make security so important that we're not able to deliver value. At the end of the day, it has to be a balance between the two.
The idea of an ATO originally, before we go into cATO, is the fact that we have a waterfall process where there are people that check all the security controls, all the things that you need for security, which is not just making sure that the software has all the pieces and parts that people can't hack into, but also that you have an environment where it runs that's safe and secure. The people that actually manage it and run it are actually approved to do so. But they made those checks at the very end of the development process.
In Agile and DevSecOps in particular, you build a lot of that security in. A continuous ATO means you have a process in place from development all the way to operations where you are continuously checking for how things are being protected in the process, not the product being outputted of the process. That's one of the key differences: if you can ensure that the process of developing something, a product, is secured, then you have a much better assurance that the product at the end of it is secure, and you're doing it on a continuous basis. That's some thoughts of what the major changes are from a cATO versus an ATO perspective.
Robin Yeman
Robin Yeman: Thank you. I'm going to ask this crazy question here: could the DoD leapfrog commercial? Whether it's been intentional or unintentional, the Department of Defense has been following along Kotter's model for making change. We have seen differences in how we build systems. We've seen more momentum than I've witnessed in the last 25 years. Based upon your experience, do you think, A, Platform One, could it be a game changer? And B, do we even want to leapfrog?
Duong Hang
Duong Hang: I love it. First of all, the graphic is real cute, Robin. You picked a really good one. I think this is a very thought-provoking, if not provoking, question. I know it may trigger some people out there. But I think really the answer is it's not necessarily a leapfrogging. I think it's just a continuous evolution of how we as the government and with private sector are basically co-evolving together so that we're actually building on top of each other's work.
Really, I think the main point is that we want to make sure that we are collaborating. We have different objectives and maybe concerns, but I think that it's not a bad thing. In some areas they are kind of conflicting or there's friction, but I think you can also take a different lens: let's use our differences as a way to complement each other.
For the Defense Department, obviously, I mentioned security is very important. We've had a lot of other organizations in the commercial sector where security is important to them. I mentioned just a couple of sectors: investments or banking, or even healthcare. Those are important, and so there's an alignment there where we can actually partner and provide capabilities today that the private sector maybe is not interested in doing or where there's no viable commercial model for it.
One of the things that I think is important to think about is what are those natural pivot points that we can either work together or maybe walk away from each other and focus our innovations and then come back together? How do we do that? How do we make sure that we have a vibrant ecosystem of large businesses, small businesses, businesses in the Defense Department, businesses that have nothing to do with defense? How do we then look for the best, expose the innovations out there, and then be able to share it across those different sectors to make each other's business or defense much better?
Robin Yeman
Robin Yeman: Excellent. Excellent answer. Thank you so much, Duong, for participating in this discussion with me. I think that our stakeholders are going to be really interested, and I can't wait to see what the government has in store for the next couple of years.
Duong Hang
Duong Hang: Yeah, I'm super excited. It's a really awesome time to be here.