Host Intro (Gene Kim)

I love this next talk for so many reasons. Scott Vigil attended DevOps Enterprise Summit here in 2019 with his friend Mike Snyder from [unclear], and I actually got to meet him earlier this year when I was in April. Not only that, I got to finally learn about what he and his team does, and my reaction was, holy cow, Scott needs to present here this week.

Scott Vigil is flight director for the PRC2 system, our Personnel Recovery Command and Control. He's been part of this project for 10 years. So if you've seen Top Gun: Maverick, by the way, anyone here see Top Gun: Maverick? Okay. So if you've seen that movie, you've always seen the system in action, because when Maverick and Rooster gets shot down, this is the system that gets activated.

The story behind this application is so cool because it actually helps people make it back home alive. So without further ado, here's Scott.

Scott Vigil

Hello, everyone. Excited to be here to share our success story with y'all concerning some software that we're doing in the United States Air Force. The DoD typically has some bad rap about not being able to be fast and be able to do some agile things, and I'm proud to stand here before you and tell you that we can go fast. We can do good things inside the DoD.

Just a little bit of background about me and my team and some of the things that we do. We are part of the United States Air Force software enterprise. This enterprise is made up of roughly 3,600 software developers, engineers, and support staff spread across three installations, or Air Force bases, in the United States: one located at Hill Air Force Base, Utah; one at Tinker Air Force Base in Oklahoma; and Warner Robins, Georgia.

From all these bases and all this software enterprise, we work on some really cool stuff and we provide air support framework for many of the aircraft in the Air Force. We do ground support. We do simulation. We do training. We do testing. A lot of really cool stuff that's done in benefit of the warfighter and what their needs are.

Our team personally is from Hill Air Force Base. We're part of the 309th Software Engineering Group at Hill Air Force Base. We're comprised of about 2,000 software engineers and support groups. We're divided into squadrons. There are six separate squadrons located in our group. We have three support groups that help out there, and we work on some really cool stuff.

We do flight supports for some of my favorite aircraft in the fleet. We do flight control modules for the A-10, for the F-16, for the F-22, and the new F-35 workloads. We do weapon systems. We're deeply rooted into the GBSD, or the new Sentinel project; that workload resides at our base. We also do mission planning type work for the pilots as they're flying. We do range systems, threat analysis, so they can train. We also have a space aspect where our teams manage the military satellite communications, as well as our favorite, the GPS system that we all use for our own phones and our devices. That software is done through our teams.

My team in particular, we're known as Personnel Recovery Command and Control. We're a team of about 30 folks. We have software developers, system administrators, information assurance officers, logisticians, and we have configuration managers that are part of us. We have oversight from Air Combat Command, who's a group out of Hanscom Air Force Base in the Boston area, and we take our requirements from them and we build the applications that they've asked for us to do.

So a little bit about what our project is. If you notice, I brought a prop. When an individual from any branch of the military is deployed to go anywhere in the world that puts them at a medium to high risk of isolation, meaning that they're going to be in enemy territory, there's a chance that they might become injured, they might become isolated, they might take upon them enemy attack or warfare, they'll carry something like this with them.

This is known as a 406-megahertz distress beacon. What will happen with this is, if an individual comes upon any kind of threat or a need, they will activate this radio beacon and they'll press the button. This button will then kick into motion a series of events that will start to initiate a recovery.

Our web application, one of them, is the database that manages all of these beacons. We know who owns each one of these. We know if they're active or not active. There are several types. Also, there's handheld radios. They could be in an aircraft. They could be on a tank or inside a Humvee, or particularly maybe even on a boat or a ship.

And once they register the device in our applications, we now know, if something happened to one of these individuals, who it is, where they're at, what's going on.

Just a little quick anecdote about this particular device. I had to have it driven here. One of my colleagues drove it and brought it to me yesterday. And as I was unpacking it, there was a note in the box that says, warning: do not push the [emergency] button. It will initiate a rescue. So I've heard. Rest assured the power is off. I will not be pressing any buttons on this device, for we do not want any of the Apaches nearby at Nellis Air Force Base to show up here unexpectedly, right? So anyway, I just thought maybe you'd enjoy seeing that.

This is the first of our applications. It's known as the Joint SARSAT Electronic Tracking System, and we track all of these beacons that every branch in the military uses.

So one button gets pushed. What happens? Well, what happens is the individual then needs, we need to be able to say, do we know who this person is? Are we communicating with the person that truly is in danger, that truly needs our help?

Our next application, which is known as Personnel Recovery Mission Software, contains what's known as the isolated personnel report. What this report does is it gathers information about individuals so that as they're downrange, if something were to happen to them, we could positively identify who they are.

So the report contains information like how tall is the person? How much do they weigh? What's their hair color, their eye color? Do they have any kind of birthmarks or scars or tattoos, anything like that that could identify them? Because typically what will happen if somebody gets lost downrange or gets separated downrange, they lose all of their identifying clothing, right? So we need to be able to positively identify this person.

We also have inside that application what's known as the word of the day or duress signal, and they are personal information about these individuals so that rescuers, when they actually have communications with them, can ask leading questions that will help positively identify who those individuals are.

So now we know who the person is, but what do we do next? Well, the next thing that happens is an actual rescue goes into place and they actually act upon this person actually needs help.

So what happens is from our next application that we develop, it's called Personnel Recovery Mission Manager. The button's been pushed, and the application will automatically receive a notice, drop a pin on a map. From the pin that's on the map, we gather a lot of information. We have API calls that pull in tons of data into our application. We gather, what's the terrain in the area where the individual's isolated? What do we know about that? What do we know about the weather going on currently in the area? And most importantly, what do we know about possible enemies that might be nearby and threats that are nearby? And are there any allied assets that are there to help perform the rescue efforts?

If you can imagine, this has to happen in a very, very fast, timely fashion. The applications have to be responsive. They have to have high uptime, have to have high availability, because just like if an individual was hiking in the woods and they got lost, time is of the essence. I like to always kind of refer to our projects as sort of like the search and rescue effort for the military, for all active duty people who are actually fighting for our freedoms and actually on the front lines.

Our team has kind of adopted the motto from the paratroopers, that is, "the things we do, that others may live." We take pride in what we develop and what our applications are for. When you traditionally hear about the DoD, you think of the big machines, the aircraft, the boats, the tanks. Our project is directly related, like Gene mentioned, about the people, the rescue, the person in the seat. And we want to ensure that those people have some safety, that they can return back home to us.

So just to hit that word DoD: web applications developed by the Air Force but used across all branches of the DoD. We're Personnel Recovery Mission Software, Joint SARSAT Electronic Tracking System, and Personnel Recovery Mission Manager. And then we also have operational control over our live environments, which are currently cloud-based through a military cloud service.

So the fun part. What happened? Well, many, many years ago, we were doing software development for this application and, just like you've always heard about government development, we are very, very steeped into the waterfall process. I mean, we were hooked. We were good at it. We were doing this traditional waterfall process like nobody else. We did all the things, right? We did all that heavy up-front contract negotiation with our customer. We established heavy requirements early on. We went through the preliminary design reviews, critical design reviews, requirements documentation. We did the cost. We did the scheduling. We set up for schedule variants and cost variants and planned for our lengthy tests at the very end, sometimes lasting over six weeks to go through all of the applications.

This was very, very slow and very, very expensive. However, we were good at it. We got really good at it and we were able to deliver to our customer in approximately about a two-year time frame all the requirements they needed, and came back with hardly any failures, right? We hadn't hardly any failures. So we were success at that point, we thought.

Well, we learned that there were better ways, and this is kind of what leads me into our agile DevOps practices. We knew and we'd heard about agile, but just in title, just in name. We really didn't know a lot about it. So one of the very first things we did was we decided we needed to become educated. What is this agile thing? How do we learn about it?

So we started taking training and we started learning about all the different agile methodologies, from Scrum to SAFe to Kanban to XP. We did all the trainings. We did all the research. We started attending conferences just like this one. We were at the 2019 DevOps Summit, where we met a lot of you. We talked to a lot of you, gathered a lot of good information.

We actually also went to our neighbors that were just nearby us at Hill Air Force Base, found some technical professionals that were close to us and said, what are your lessons learned? What are the things you've done that have helped you get better, faster, stronger? So we took all those things and we decided, all right, we're going to adapt and tailor an agile process that fits our team, that meets the needs of the United States Air Force, that meets the needs of our product, and also meets the needs of our staff.

So what we did was, we took a little bit of everything. We made our own sausage in our own flair. We adopted test-driven development as one of the first things, shifting that test pattern way to the left. We did pair programming. Part of the pair programming effort we did, which was very controversial with our teams at the time, is we tore down the cubicle walls and we created collaborative workspaces. This was a major shift for our teams, to be able to hear one another talking and to hear one another actually working, that they couldn't wrap their heads around. How do I focus? How do I continue to do stuff, right?

We built in test automation into our process, and then we started to do continuous integration and continuous delivery, and we accelerated our fielding approvals from our end user perspectives.

So where do we go? What do we do? Well, let me jump back real quick. I want to tell one quick other story about where we're at.

So I mentioned earlier we talked about that testing cycle that originally took us about six weeks to accomplish. When we first started down our DevOps path, we thought, okay, well, we've got to build a pipeline. We've got to build some automation. We've got to build this into what we're doing. And we thought, wow, we have come such a long way because we've automated our tests. We've taken this six-week process and we've shortened it down to about 13 hours.

Our team could publish code to the pipeline, run the tests overnight, we can come back tomorrow and we could say, hey, what passed? What didn't pass? How good did we do? We thought we were cruising. We thought we were full speed ahead. We thought we were just [unclear], and we realized real fast that that wasn't fast enough. We needed the ability for our developers to fail fast so they could fix fast. And so we knew that testing had to be improved.

Also, what we needed to do was we realized that because of our application's sensitive nature, we have roughly two million records within that PRMS application. We need to protect that data. The data is very critical. And so we needed to make sure that we weren't doing anything so fast that we introduced vulnerabilities to the applications.

So we started getting better with our pipeline. We started doing faster things in our pipeline. We adapted security scanning that does dynamic and static scans of our code, and we were able to use technology, newer methods, to decrease our test times. And to make a long story short, we have decreased our test time from that six-plus-week effort to now under 24 minutes full execution. So that's a win in my book.

What this has done is this has given our team the ability to be creative, to put out new abilities. We're capable now of, we work in two-week sprint cycles. We deliver to our customer every two weeks. We could, if our customer would allow us to give it to him on demand, we could give them code as soon as we finish the code, but what they've told us is, wait a minute, that's a little too fast for us. We can't keep up with your cadence. So let's stick to a two-week delivery cycle.

We're super proud about that as well. We feel like, hey, when they're ready for us to step on the gas, we're prepared to be able to step on the gas. And this last May, so we started our agile journey in about the fall time frame of 2017, and May of this year we surpassed our 100th sprint, culminating in our 100th delivery to our customer with zero rejects.

So we've been providing capability to the warfighter, to our customers, for close to four years now. We continue to set the pace. We continue to set the bar. We're constantly looking for the opportunity to try to do something better.

As we talk about things of like, what can we do? What can we do more? How do we get better? Things that I want to talk about and talk to the community here about is, how do we continue to do a CI/CD situation where we're air-gapped, our development is in a classified space, but we release in an, I'm sorry, we develop it in unclassified space, release in a classified space. We have to physically take the code and move it from one place to the next. Cloud computing, one thing.

It happened to all of us. Spring of 2020, COVID hit all of us. And because we had just recently tore down all the cubicle walls and we got this collaboration going, COVID nailed us and we sent every one of our people home, and all of our developers currently are working remotely. They telework.

This is another really cool story. We had an on-premise development environment. And in that COVID timeframe, we were forced to adapt and become agile. We pushed our whole development environment into a cloud space within two days. Our system administration team was able to stand up the entire dev environment, and the team was able to connect remotely from home and continue delivering, so that 100th delivery actually occurred while our team was all isolated from one another working at home.

As a whole, the Air Force, we're talking a lot about things. How do we get better, like simulation for aircraft testing or model-based systems engineering 2.0. How do we do tests on aircraft? I know that that's a problem for my colleagues as well. AI and machine learning, future research and development.

Personally for me and for my team, one of the things that we've tried to take pride in and what we do is we want to think that we're leading edge. We're on the front edge. We're pushing the boundaries within the Air Force and we don't want to be laggards ever again. We want to be early adopters. So when the new things are coming out, when there's new technologies or there's new philosophies or there's new things, we want those things. We want to be in touch with you so that you could help us remain out in front as things happen.

Okay, I'm about done. Here's my contact information. That's my business card. It's a digital business card. You can just scan the QR code there. Please reach out to me. I'd love to talk to any one of you, my team, tell you more in depth of the story. Twenty minutes really doesn't give you the full picture, but you get a flavor of what we've done and just share the pride in what we do.

And this last slide might choke me up a tiny bit, but this is really the reason why we do what we do and why we have a passion for what it is we do. And it's so that the warfighters protecting our freedoms as Americans have the tools necessary to come home to their loved ones safely. That's what we care about. That's what we want to do. That's our project. So thank you. Thank you for your time.

Thanks, buddy. Thank you, guys.