Jeffery Payne

My name is Jeff Payne. I'll introduce myself in a bit. I have to apologize for Ken Johns, who was supposed to be here with me. We were going to tag-team this and beat each other up over these lessons learned, but unfortunately duty called and he got pulled away and could not be in attendance. He does send his condolences to everybody.

I'm going to talk today a little bit about lessons learned in federal healthcare. It looks like a popular topic today. I entitled it "From HealthCare.gov to the Battlefield" because during the last 10 years, both Ken and I have been involved in a wide variety of healthcare-oriented programs in the federal government, and a lot of lessons learned in those programs. We're going to talk a little bit about some of those and hopefully give you some ideas, if you're in the federal space, how you can deal with some of the challenges that are there in the federal world.

So just a little bit about me. I've been building and testing software for 30-plus years, have run a couple of software consulting companies in the DC area. My most recent company is Coveros. I started it 15 years ago. We are a pure-play DevOps/DevSecOps consulting company. We help all the way from the top enterprise-level coaching and strategy all the way down to hands-on CI/CD implementation and integration of application security and testing into that process. That's all we do at Coveros, and we work mostly on the commercial side. But as I always say, if you live in DC you cannot avoid the federal government. They are everywhere, and we sometimes get called in when projects are in trouble and we help get them out of trouble if there's a DevOps challenge there.

I'm going to jump right past the transformation slide and get right into the meat of the discussion, and that is lessons learned. I'm going to try to give you some stories as part of these lessons learned because I always feel like stories really hit home with what the challenges really are.

01Lesson 1: Our customer is not the customer

My first lesson learned is that oftentimes in the programs we've been involved with, our customer is not the end customer. We see this in commercial as well, but not anywhere near as often as we see it in the federal government.

I'll give you an example. A recent customer provides a large-scale operational medicine system that's used worldwide in DoD. This system tracks all of the operational medicine supplies, people, mobile hospitals, blood supply - everything you can think of associated with both conflict or wartime and peacetime humanitarian efforts around healthcare.

As those of you that know DoD know, the world is broken down into these things called combatant commands. They're different aspects of the world, and they're all administered and run by various people who own those particular regions and control everything in those regions, and pass things back and forth when troops, or in this case surgeons and medical staff, cross boundaries in different regions of the world.

So the system gets deployed into combatant commands and gets used out in the field to track what's going on logistically with all the operational medicine types of things that are part of the process. Unfortunately, those of us who are involved in the development of these systems don't have direct access to the entire customer base. Those involved in the application development before we got involved didn't even really understand this. They didn't realize this because there are two pieces to the system.

There's the strategic piece of the system that sits at headquarters, and it's used by the generals, the admirals, the high-level ranking officials who are strategically trying to decide what to do. When things are happening in the world - as you can imagine, as the Russia-Ukraine war is happening - they're making decisions about where medical supplies and medical aid need to live and reside, and that's moving around every day.

In this program and a lot of other programs, we have an intermediate organization whose job it is to help facilitate and understand what the customer needs. But unfortunately, the customer in their perception were those strategic commanders who are making these high-level decisions. If you dig into the system, what you find out is that the data that populates the system comes from the warfighters. It comes from the people in the field, down in the trenches, in the combatant commands. They have to provide information on blood supply, current logistics, personnel, what they're seeing in terms of patients - all that has to be entered into a system in order for the strategic decisions to be made properly.

Unfortunately, those people aren't part of the customer base as looked at by the intermediaries that we deal with. It wasn't really until we got involved and said, wait a second, who's talking to the end customer - the surgeon down in the field, the MASH unit if you will - about what's going on, and are they entering all their data or not? Well, as you can imagine, what we found out is they weren't entering their data. When we were able to finally get in touch with and start collaborating with these end customers, we found out that they were not being supported at all.

So we didn't have direct customer access. That's very common. Intermediaries are often in place in these systems, and they may not even know how the system works or who really all the customers are. Of course, if we can't understand our customers, we don't know how to help them do the job better.

What do we do typically in this scenario? We've seen it many times. One thing is we try, as much as possible, to instrument or provide some sort of monitoring capability in the application so that we can have the software collect some information. Obviously, there are performance constraints to that, and other constraints sometimes too, but we want as much as possible to use technology to keep track. Even simple things - if the software just tells you how often those in the field update the data, that's a useful piece of information. You can tell whether this data is real time or if it's weeks old, which we found out is often the case.

The other thing we did was we started to work with the intermediaries to get them to understand that we needed them to be involved and engaged on a more regular basis. In this particular program, the intermediaries were called the product owners. They were the product owners for the agile and DevOps teams, but we never saw them. The expectation wasn't that they were involved day to day, and that was a challenge. So we had to fix that. We had to set expectations that if you're going to represent the customer, we need you representing the customer every sprint, not every quarter, not every year.

Last but not least, we started to strategize about how, as field units moved out of the field and back here to the US, we could grab those users as they come back. Now they're not in a pressure situation. Let's grab them there and understand what their needs are in the field when they have time to actually collaborate with us and communicate with us. Those are some of the things we've done to bridge this gap.

02Lesson 2: Create delivery pipelines using agile

The second thing I've seen - and you'll see this in commercial as well, but more so in the government - is that oftentimes our teams don't use agile to create their delivery pipelines, their CI/CD pipelines. What I mean by that is, instead of working very closely with the development teams and trying to help make sure that what gets delivered is usable, delivering things in small chunks, iterating, and treating the teams as their customers, these DevOps teams will disappear for a year, heaven forbid a couple of years, without giving any feedback or any technology to their teams.

Oftentimes this happens for one simple reason: the way the government contracts things. A lot of times DevOps capability or CI/CD capability is set up as a separate contract, and therefore those people don't really feel like they're integrated in with the rest of the organization. Nobody really told them they should be working with these other teams, these other contractors and others, to make this all work.

We've seen examples where a team will come back after a year and what they provide not only doesn't help the teams we're working with, but actually doesn't provide any value to any of the development teams because they haven't really been treating the development teams as their customers. So what you've created - and the singer on Tuesday night said this great when he was talking about silos - you create another silo called DevOps. The whole idea behind DevOps, ironically, is that we tear down silos. We don't build new ones. But yet now DevOps in some organizations is a silo, and we see this a lot in federal programs and federal systems.

What do we typically do about that? We had this problem in a large-scale healthcare system a couple of years ago. We were on the development side. We were actually responsible for the CI and the vendor-produced pipeline, which then fed into an overall pipeline produced by a platform engineering team staffed by another contractor.

What we did was we reached out and started to pull this team into our sprints, into our application development process. We got them to understand: you're part of this cross-functional team, and you need to be iteratively providing us capability so that we can provide capability out into the field in an incremental manner. We were able to pull them in and get them actually engaging and working with us every day.

We were tracking CI/CD needs in pipelines. We encourage teams to do that and work with the platform team to figure out how to best get those things done in the time they need it.

We focused first before tools. We really tried to get the DevOps teams and our teams to understand: don't start with tools. I think you've all heard that during this conference. We see that in the commercial world as well, but in the government it just seems like they think the easiest way to solve a problem is, let's go acquire a tool. We know that doesn't work.

So we started to work with the development teams and the platform teams to understand the delivery process that we need to put in place, and then leverage value stream mapping and value stream analysis to start laying out what the process should be, looking at what it is currently, and how we move from what we call as-is to to-be over time. The other thing about value streaming is it is a great way to identify barriers, roadblocks, and the true things that are slowing you down. You'll find all sorts of fun things when you do that. That process in and of itself is extremely valuable.

03Lesson 3: Socialize DevOps up the chain

Lesson three is about getting people up the chain to understand what DevOps is all about. In the federal government, and particularly military, you have what I will politely call old-school executives. I'm old, so they must really be old. They don't understand software. They didn't grow up building software like many of us did. They don't understand how it works. They don't understand why we want to change this process. "What was so bad about the way it used to work? It seemed to work pretty good for me. I told people what to do and they built it, and that was it." I've heard that many times.

They'll put up with agile and DevOps and say, yeah, go do that stuff, but don't forget, I still want my requirements document, and I want it up front. I want my test plans, I want them up front. I want my design, I want it up front. Then they're mystified when you say, well, that's not how the process works. They're like, what do you mean? I need this stuff. I need assurance you're going to do it right. So this is obviously an education process.

Also, they're scared about automation. They don't trust automation because they don't understand it. They don't believe that you can do some of the things discussed this morning around governance using automation. I actually had a senior exec say to me one time, when we were talking about a technology problem: "Gosh, can't you solve this with software? I mean, can't you automate this or something?" This particular issue was really not something software was going to solve. I said, I'm sorry sir, this isn't really something that software's going to solve. There are other ways we can solve it, but it's not through code. He looked at me funny and said, "Gosh, I thought you could just sprinkle some magic software dust over this problem and it would go away." I was like, if you think this is magic, we've got to have a lot more conversations here, because that's just not the way it works.

So what are some of the things you can do? First, obviously, you've heard it: you've got to repeat things again and again and again, and get people to understand how it really works and what's possible and not possible. The why message you have to repeat consistently. Why is this important? Why are we doing it like this? Why is there value and benefit to the mission in doing these things? You have to repeat that constantly.

One good thing is that there are starting to be agile and DevOps practices and guidance and documents created across the federal government that tell and direct programs about how to use agile and DevOps to be successful. So you can at least point to things that are being written by the government themselves and say, hey, this is now an approved approach of doing things. We need to do those kinds of things.

In terms of automating some of this stuff, I've had good success sitting down and getting agreement on what the automation is going to provide, and then letting the executives essentially audit or review the controls and the governance that we put in place to show them that they're getting the same level of value and protection in the automation that they were manually. They're just getting it way faster.

A great example: we got one of our customers to the point where they were continuously delivering applications into production - well, they could deliver into production every two weeks. That moved down from what it was a year earlier in the program. They could deliver every two weeks. They have a change management process that has not yet been automated and improved. That takes eight weeks. So it takes four times as long to approve a change that you could build, test, and deliver in two weeks.

Those kinds of inefficiencies make no sense. If you start value streaming, you'll find those kinds of problems and you can start to address them. You can smooth out your delivery process, because a lot of times it's those kinds of processes that are slowing us down.

Document signatures are another example. It takes, in one organization we're working with, at least two months, if not three months, to get a document approved and signed. I laugh because there are some documents in that org that will never be signed because they reorg faster than the signature process. We've gone through three iterations where it gets partway through the signature process and then they say, oh, well, the organization doesn't look like this anymore. You have to fix what the organization looks like in this document and resubmit it. We've done that three times. It's taken almost a year and it's still not done. It will never be done unless that process changes. We're working on fixing that as well.

One thing that's interesting is I've never been a fan of mandates telling people this is the way you do it. I've always been an agile person. I believe in carrots versus sticks. You heard it this morning. I fought pretty vehemently when some of our sponsors said, we should just go get the general to sign an executive order that says thou shalt do this. I'm like, that's not the way to get people to change, that's compliance, yada yada.

Then I had an old-school exec who actually had a good piece of advice. He came up to me, put his arm around me, and said, "Son" - I haven't been called that in 30 years - "let me tell you something. The military is trained to follow orders. They've been training their whole life to follow orders. If you want them to do something, you give them an order. Give them an order and they'll do it. If you don't give them an order, they won't do it." I was like, wow, you're probably right. So we started going to get orders, and it does make a difference. They will march to that. It doesn't equate in my mind from what I'm used to in the commercial world, but in that setting it works.

04Lesson 4: Downstream hurdles are hard to overcome

Number four: downstream hurdles are hard to overcome. You heard it throughout some of the other federal projects that were described earlier. There are downstream testing and cyber and other types of requirements that you have to go through that slow down your process tremendously. It's very hard to get through those things quickly. If you're dealing with DoD, each of the services has their own testing process and they don't trust anything you've done.

They don't care what you've done. In fact, some of them want your source code. They want to start from scratch and compile it to make sure they believe that it compiled properly. They don't trust anything you've done, so they're going to redo everything you've done, and that totally slows down this process. It means you're not going to do continuous deployments, obviously, so we're really in a continuous delivery world. They don't trust the cloud in a lot of places either, so you can't even use those kinds of systems.

What have we done? The best we've been able to do is work with these external testing agencies to try to make what we're doing transparent and visible to them. We're saying, look, we're an open book. We're going to show you everything we've done. We're going to give you our tests. We'll give you our harnesses. You can rerun our tests; don't write your own. Or better yet, come to our sprints, come to our systems, work with us day to day, and learn that we're doing the right things. So when it's your turn, you can decide how much more you really think is appropriate.

One thing that's interesting is I thought out of the box that was the solution. I expected one group might push back because they were going to not be able to do things anymore, but it was a different group that didn't like this model. Guess who it was? The development teams. They didn't want them in their sprints. They didn't want them to know what they were doing. They didn't want them to do the testing they had done. I was like, whoa, now we have a real trust problem because that's where it belongs.

So we've been working to get the development organizations to understand: this is a good thing. It's not a bad thing for them to be there. Build a relationship, build trust. It's going to accelerate this process. But that was not something that they really thought was something they wanted.

05Lesson 5: Silos are bigger and badder

Lesson five: silos. I call them bigger and badder in the federal government. These are ingrained in culture. The organization is very typically, in agencies, very hierarchical. And of course they award contracts by silo. They have a team that helps plan. They have a team that helps design and develop. They have a team that does independent tests. Sometimes they have a team that does DevOps and deployments. They're all separate contracts and they all are competitors. They probably all bid on all this stuff and are mad that they didn't win other pieces.

It's very difficult because you are trying to get them to work together across silos, and they don't want to. Better yet, a lot of times the contracts aren't written to allow them to work together. The contracting is where this change has to start.

What are we doing about it? We've been working on what we call working agreements: going and saying, look, we need to work cross-functionally. We need to work together. Let's look at all our contracts. Let's figure out how we can work together to achieve what we're trying to do, which is work cross-functionally and not have gaps and other things in process, but make sure that you still meet your contract requirements. Get working agreements described.

Really try to focus on what they call in the military or in federal government a badgeless culture, getting everybody focused on the mission, not their particular piece of the puzzle. Last but not least, push it back. We've started working with acquisition to understand that the language has to be put in these contracts so that everybody's working together, and that we aren't putting things in there that are hurdles for them to work together, because that just makes the whole process harder.

Same with tools. If you let all the vendors pick their own tools, you're not going to get any kind of consistency and you're going to have to figure out how to integrate it. So what we've gone to is the model where it says, you can bring your own tools, but ultimately everything has to end up here. So you're either going to figure out how to port that into here, or you're going to do that manually or automated.

06Lesson 6: It can be done

Last but not least, and I'm wrapping up, it can be done. Here are a few examples of things we've been able to achieve with some of our healthcare systems over the last 10-odd years that we've been working on these kinds of things.

Just so you know, we're not just a bunch of coaches. We do some pretty hardcore DevOps tooling. This shows one mostly open-source stack that we put together to do a whole variety of infrastructure and automation things, security test automation, and governance as code for one of our customers.

The biggest challenge we really see is there is typically a disconnect between federal staff and the contractors in most places we go. They don't trust each other. They don't know each other. They don't even really know how to work together. They want to be in their own silos, and that causes problems.

I'm out of time, but I'm happy to stick around afterwards and talk to anybody if anybody's looking for any DevOps help, particularly if it's complicated. We focus mostly on highly regulated, compliant, embedded, legacy types of systems. Just let me know and happy to talk to you. Thanks.