John Schreiner

Just a couple things before I get into this. I am an active-duty Marine, so any opinions I express are my own. They're not policy for the Marine Corps, the DoD, and they don't speak to my employer or anything I happen to be associated with.

Task Force Phoenix: I'll talk about what it actually is, but I wanted to start with the problem we were actually solving. I was one of the individuals associated with this; this wasn't completely my thought, dream, or idea, and not completely my intellectual capital. We knew we wanted to bring DevOps to the Marine Corps, and we knew the only way to succeed on the modern battlefield would be to deliver capability at speed. Our assumption was that we need DevOps to do that. Before we even got started, we knew we didn't have either the organizational network or the physical network to get that done. You've probably seen the "fix our computers" memes on social media; that's something we've been struggling with in the Marine Corps for a long time. So we knew we needed to build the right people network and the right physical network.

To do that, we came up with something called Task Force Phoenix. If you're not familiar with the military, the general idea is that service components like the Marine Corps, Army, Air Force, and Navy man, train, and equip service members to fulfill missions needed by a geographic combatant commander. When you put those forces together, they normally do so under a task force. That task force is not a standing Marine Corps thing or a standing Army thing; it is ad hoc or temporary in nature. We used the vernacular the military is used to hearing, so we came up with a task force.

The reason we needed to be a task force was because we had different silos inside the military. Crazy to hear, but there are silos inside the military. We couldn't deliver capability inside a single silo. Requirements feed budgets, another silo; budgets feed compliance, which tells us what we're allowed to do; compliance feeds acquisition, and then acquisition feeds operations. All these separate silos are generally controlled by general officers of varying numbers of stars, with essentially their own and different value stream, because the acquisitions general doesn't have the same value as the operations general.

So we said we would come up with a task force and stripe across all these silos to deliver capability. That's the cross-organizational part. We're used to cross-functional teams, but we did cross-functional and cross-organizational teams, somewhat like a matrix organization in function, but a little bit different. The last bullet is really important: we came up with a way to communicate what we were accomplishing so each general officer or different organization could say, "This is what's being accomplished. I'm agreeing that my people will work toward someone else's goal inside their part of the silo." We gave them a narrative to say we were seeing success and accomplishing things.

Here's the approach we took. The first one was we started at the top. We had general officer buy-in, and our general officer on the operations side said, "I am the head of the task force. Anytime you have a problem, anytime somebody says, 'I don't know if I agree with this,' feel free to throw my two stars on the table. I am endorsing this." This was tested in many different opportunities.

To give you an example, anytime any member of the task force needed to talk to the general, it was a same-day answer. If you imagine a two-star general like a CEO, imagine saying one day, "I need to talk to the CEO," and you're on the schedule the same day. Depending on your org, that may or may not be crazy. Given that the Marine Corps is about 200,000 folks, still an awful large number of individuals, being able to talk to the CEO of network operations was a big deal. When we said, "We need more advocacy," he stood up daily scrum meetings for about three months.

That was the type of buy-in: use my rank anytime you want, they are in my office same day, and I will make a daily standing meeting if it needs to happen. I did not necessarily have the right appreciation for how rare that was, but it was a huge deal to have that level of advocacy. Possibly the most impressive thing was we had a general standing rule of no more work. I did not produce a single PowerPoint as a field-grade officer for the general the entire time I worked on this project, which was about a year and a half. If you tell another field-grade officer they don't have to produce a PowerPoint slide for a year and a half, you would probably see tears, mostly of joy.

The next thing we did was a value stream mapping exercise, mainly for the matrix organizations. It was clear that since the two-star owned operations, he understood where his value streams were; that did not need to be communicated. It's his task force, and they're going to do what he wants. What we needed to do was value-stream map for all the other matrix organizations to show them, "We all love the Marine Corps. We want to give the Marines what they need, but we need to show that it is in your best interest to get on board with what we're looking to do." We took delivery of capability from years to months by doing a couple things differently. We did not stand up extra authorities. We didn't do anything crazy, illegal, immoral, or unethical. We did things differently under a different construct and took significant time off.

The next thing was Chief Explaining Officer, borrowed from somebody who came on the podcast. While a lot of work was being done, and while we had a constant drumbeat of selling and getting work done, we also deliberately explained. We did not make PowerPoint slides; we made a very deliberate push to explain. There were times I sat down with the people who did commands on routers and switches, who didn't understand why we were doing what we were doing, didn't understand where it fit in the grand scheme of things, and generally thought from a routing and switching standpoint that what we were doing was not a good idea. I sat down and explained to them for four hours: the construct, the model, and then the engineering level of why routing traffic this way in the grand scheme of things would be best for the enterprise.

That is one example. We did that dozens of times. We didn't just do it internal to the teams. We did it for managers of matrixed resources, senior leaders who wanted to understand what we were doing, and other DoD entities. At first, Marines brand something and our PR campaigns are generally second to none, so there got to be a lot of bluster around the task force. We intentionally left it somewhat vague so they would have to come talk to us about it. We took the time to talk to other DoD entities and explain. Once we started moving really quickly, especially around cloud migration, they started raising eyebrows and saying, "This is interesting. I like what these guys are doing."

Then we picked out key individuals, which we called the coalition of the willing, and they all met in my basement. Representatives from each matrix org were not star generals or senior SES civilians; they were generally majors, captains, and lieutenant colonels. We wrote out what we were going to do, how we were going to do it over institutional reticence, and how we were going to be good teammates for each other. One of the biggest things, and I know Gene and I have talked about this a lot, is keeping the attitude right. We all want to win, but there is institutional scar tissue and long-held hatreds between organizations. We made a deliberate push: how am I going to support people from organizations that we generally did not work very well with previously? It took a decent amount to get them to trust us, but I was appreciative that people were on board and willing to do the plan as we put it out there.

The next thing we did was agile training. Depending on the room, agile may or may not be a happy word, and we generally used SAFe, which I know will get me in trouble with some presenters. But we got everybody together, got common training and a general common understanding, and no one was left out. There were people saying, "I haven't been trained, so I don't understand what you all are saying." We spent a decent amount of money making sure everybody got training, from the most senior folks down to the most junior folks, so everybody could at least speak a similar language.

The last approach was standing up a podcast, the Phoenix Cast, a podcast about cybersecurity, technology, and innovation issues in the military. I am one of the co-hosts. The aim was to have a larger microphone. Starting a podcast and getting people to listen is a large undertaking, but it was massively successful. Every target audience we identified ahead of time, who we wanted to influence, who we wanted to educate, and the word we wanted to get out: check, check, and triple check.

Let's talk results. Bandwidth-wise, we saw measurable bandwidth, over 10x increase in less than 18 months. This was not imaginary; I didn't take one interface from one gig to ten gigs ten times. This was 10x observed throughput increase. That is massive for what the Marine Corps is able to do. Our databases were able to reliably replicate where they were not able to previously, so we made huge gains on the bandwidth side.

Next, we got architecture changes in place. Outside the military, it might be hard to understand why this is difficult, but we got the entire Marine Corps to sign off on three different architectures: a network architecture, a sensor architecture, and a cloud architecture. That is a huge feat. Every enterprise architect had to get together in a room, listen to our presentation, and sign off. To the best of my knowledge, the Marine Corps previously had a network reference architecture and had never had the other two we delivered. We updated one and delivered two new ones that never existed. A lot of that had to do with who was in support. Previously people would say, "This is a great idea, but we should improve it," and it never got produced or bought off because somebody wanted something better. With enough buy-in, we were able to push through something not perfect, but at least something to start from and deviate from.

In 21 years in the Marine Corps, it was the first time I saw work visible. Through agile, we got work visible and reached the point where we were making opportunity-cost decisions. We planned out our sprints, and when leadership deviated, we said, "Here is what is moving to the backlog." Senior leadership was involved in grooming and prioritizing the backlog on a regular basis. These were significant wins from an institution that was not even doing agile six months before the task force.

We were the fastest service to migrate to Office 365. It went incredibly quickly. We were not the first to start; several entities started before us. But we had great timing between the 10x increase and delivery of the capability. It was also the first time I saw us making decisions based on numbers. We were not randomly migrating people. With a lot of partnership through Microsoft, we measured active users and the percentage of users that, if migrated, would actually be active. We had strong metrics and made migration decisions based on numbers, not based on "this unit will go at this time because I told this general, I don't care about reality, just make it happen." In some cases that was painful because we said no to people and had numbers to back it up. In other cases, we went exceptionally faster than expected. This timed with COVID, which was massively successful because we essentially migrated everybody right before everything shut down. The Marine Corps was in an amazing place, and people were scratching their heads about how that happened.

The next result does not look great on paper: we did the first lift-and-shift migration of a Marine Corps application. This room may not be super excited about that. However, has your chaos engineering team ever run a flood through your server in the middle of a migration? That is what we did. We actually migrated a day early because the data center housing that application flooded. Leadership freaked, because in the military when we say we are going to do a thing at a time and we do it a day early, people are ready to make heads roll. But the database had migrated significantly ahead of when we expected, because of the bandwidth. It flooded, we did checks to make sure it was good, and then we said, "We are going to migrate to restore services." Huge win, and I don't think it would have been possible without the other task force efforts.

The other thing I had never seen was acquisition crossover delivery. In the military, you have a requirement, and the acquisitions team says it will fulfill that requirement with this widget in this way and sustain it. If you have rifles, cars, tanks, servers, or even servers that work together, these are all different acquisition projects and generally set up to succeed on their own. If a server requires a switch, and you have two server projects, even though you only need one switch for both, you buy two switches because each project needs that to happen inside the project. We were able to convince some acquisitions teams not to buy redundant or unnecessary gear, which on paper does not sound difficult but is incredibly challenging. Then we were able to time acquisitions delivery based on other acquisitions requirements and deliveries, and deliver architecture changes where we needed them, where traditionally we would say, "What is your sustainment life cycle? You will be refreshed on this date and not a minute sooner." These were relatively huge military-wise deals that had not happened previously.

Those are the results. I will tell you, mistakes were made. The first problem was the frozen middle. We had senior leadership on board and junior field-grade officers bought in and ready to go, but there was a frozen middle between the most senior and middle-level managers where we had significant challenges. If there was a community we missed, or we had to try a different way, traditional lists were not particularly successful.

The next failure was conceptual understanding. I think we did a good job explaining things, but it was not always understood by everyone, and not always understood at the level they needed to understand, and there were missteps as a result.

The next is the book I talk about all the time, Greg McKeown's Essentialism: The Disciplined Pursuit of Less. In some cases I heard senior leadership say, "This is great. I understand what you're saying about needing to do less to deliver results faster. We are not going to do that." Maybe we need more conditioning or to get people to read more Essentialism. It is very antithetical to the Marine Corps to do less. We always want to do, do, do, and sometimes we don't understand the second- and third-order effects of that.

Next was agreement on the problem we were solving. We communicated it and put it out there, but sometimes people hear you and think, "You're a senior officer," or "the general said we're doing this, so we're doing it," but they didn't actually agree these were the right problems to solve. Because they interpreted them as unchangeable, they didn't talk about it. We got way down the line before we figured out they didn't even think the problem we were solving was the right problem. It is antithetical to the military because we wear rank on our collar. Everyone knows it; there is no hiding it. My subordinates are not allowed to call me John, even subordinates from other services. Rank makes it hostile to clear communication and to ideas coming out.

Next was communication on what agile would provide. In some cases agile was distilled down to "you're going to put it in a sprint and we're not ever going to be able to change it." The conceptual understanding was there, but not the actual understanding. Worker bees were told they should buy in because they would get stability. Then when we started changing things on sprint plans, people freaked out. We could have done a better job explaining that stability was not the end goal and communicating with middle management.

The next difficult thing was implementing meaningful measurements. We were able to do that for some things, but meaningful measurements for other things were incredibly difficult. Even if you understand the right thing to measure, do you have the capability to measure it reliably, where you need to do it, when you need to do it? The DoD sometimes presents a weird footprint for where it goes out. Getting meaningful measurements and using them operationally was incredibly challenging, and something we aspire to get better at but are still not quite there.

The next thing is siloed product delivery. While we had some acquisition wins, others were incredibly painful. Why are we buying another router here when we know we don't need it, and we have a reference architecture that says we don't need it? The plan that was agreed upon, briefed, purchased, and already contracted was unable to be changed. Now you are either spending taxpayer dollars and not using something, or doing suboptimal engineering. These are difficult challenges that we continue to work through today. I don't think there are great answers, because even in the two-switch example: what is the refresh interval, how do you buy half a switch, and if someone comes on, how do you take that money and split it a different way? It is difficult in the way we purchase things in the DoD.

Where are we now? In classic Marine Corps fashion, I have PCS'd, or moved on, from the duties that started the task force. The task force before was under a two-star general in charge of operations. It has now transitioned to being under a three-star advocate, which is beneficial in some ways, but there is also the problem of focus because a three-star general has an awful lot under their umbrella of responsibilities. There is a little drift of focus, and we as service members will have to be extra diligent to keep it on track. The task force has gotten bigger and added more, including talent management. I look forward to seeing how the task force manages that. With that, I will answer any questions you have for me.