Robin Yeman

Right now we're still headquartered in Bethesda, Maryland. That hasn't changed. We've got just over 100,000 employees building pretty much everything from submarines to satellites to intelligence systems. It doesn't matter.

We've got five lines of business. Each one in their own right could probably be a company. So we've got Aeronautics; Missiles and Fire Control; what we call RMS, which is Rotary and Mission Systems. It's really the integration of the Sikorsky helicopter and our surface ship stuff. I don't totally get the link there, but it's what we have. Space; and then we've got internal EBS.

We began our agile journey in about 2002, and it's not because we decided that we wanted to be on the bleeding edge, because that's not usually our jam. But the intelligence community started requiring agile pretty early, so we got RFPs, or requests for proposal.

We began our DevOps journey in 2012. I remember the day that I came back and I presented at our Systems Engineering and Architecture internal conference, and I said, "We need to go to DevOps." And they were like, "What is that, and why would we need to do that, Robin?" So, lo and behold, persistence over the years made able to get them to move.

This year, a couple things I'm pretty proud of. Well, the one thing I'm proud of that we've been able to accomplish is building a software dojo. In 2015, Target did their presentation about their software dojo, and the first thing I said is, "I want one of those." It only took me three, four more years to get it, right? But on October 24th, which is tomorrow, it will be officially done, so I'm pretty excited about that.

What I was going to tell you is we've had a lot of impact this year, and the reason for the impact really came about from something we call the DoD Science Board report. I'm going to tell you guys a little bit about it, but it's really why the DoD, the federal government, the intelligence communities, all of those things are going to be moving to DevOps in a big way fast.

Just a little bit about the landscape. Here you can see we've got our defense strategy and the key members in that defense strategy and how they put together what we're going to build: what does the nation need? They've hired somebody called Jeff Boleng. He came out of Carnegie Mellon or SEI, and he's going to lead software for the DoD. The reason is because one of the highest risk areas, or the highest risk area for the DoD and cost overruns, happens to live in software. So this year they were like, "We have got to get a handle on it."

Here is a summary again of the defense strategy and some things that were said during that timeframe. I'm not going to read it to you because I think you guys can read it, but I do highly recommend that you go out here and get these reports if you have any interest in development or government contracting or anything like that.

Again, some other hugely important people in here. You've got Ellen Lord, and then Jeff Boleng. He's the one that I told you has just come on board. One of the key things that they're all saying is software is the thread through our programs. It's the important piece. A lot of things that they mentioned I thought were just brilliant because the commercial industry got it a long time ago.

Agreed, we're at the inflection point. Just like McChrystal's presentation last night, he was kind of talking about where we were. We are at this inflection point, and the government knows it too.

There's a DoD software task force. MITRE helped this, and this is the presentation that they put out, which is the DoD Science Board report. But I'm more interested in telling you what's in it. They gave about seven recommendations that are really going to impact how we deliver product services, capabilities for the war fighter, how we get things faster.

One of the also interesting things, if you guys do pick up this report, is they actually put a picture of the F-35 in there, and they pointed to it with an arrow to say, "This would be a good one to transition." So I think they thought we didn't get the hint before, and they thought if they put it in pictures, we would be all over it. I'm just saying.

What did they say? What is this magic? Some of these, actually many of these, I am sure that you guys already knew were important. You're already doing.

One, they interviewed a variety of companies in Silicon Valley, and one of the things that they found out over and over again is programs, projects, value streams, whatever, that are successful have a software factory: people, process, and tools to be able to deliver business outcomes. Cool.

Continuous iterative development. Before we were doing our traditional waterfall, do all the requirements. Here they're saying, you know what? They're not calling it agile, but they want continuous iterative development so that they've got transparency. Terrific.

My favorite: risk reduction for new metrics. We'll go into detail in each of these in just a few minutes. But one of the things after this one, I asked my boss if I could get a T-shirt that had the word SLOC in it and a circle and an X in it, because we've been measuring SLOC for as long as I can remember. I've been telling them it's a bad idea for as long as I can remember, and I'm really excited that they were like, "Yeah, wrong metric. Great vanity metric. That's about it."

They made the point to say it's not just new programs they want to transition. They said any program: current, legacy, in development, production, or sustainment, they want us to transition.

Investing in our workforce. Right now there is an actual shortage of software engineers, at least in our space, and I believe for every space. But it's really noticeable in the government space because a lot of them need to be cleared as well. Finding a top secret cleared software engineer with a background that has DevOps and Agile skills is like finding a unicorn. We are just very short.

They also mentioned that software was immortal. Again, I work at Lockheed, and I can tell you for the last 25 years I've been hearing, "It's just software, right?" because we really believe in the hardware. Afterwards, I had to send a note out to all of our leadership to say, "Actually, you know what? I heard that hardware is nothing more than a mechanism to deliver software." Making friends and influencing people.

Then they said we should be moving to IV&V, use machine learning for IV&V. IV&V for us is integration, verification, and validation.

How are we going to do this? The other thing I didn't tell you is this report came out in February of 2018. In August of 2018 this report was actually -- well, the report itself wasn't, but there was actually legislation passed that said all government folks that are working with government dollars have to transition within the next 18 months to leverage the recommendations in the DoD Science Board report or explain why not, which will be quite a challenge. Apparently we didn't move fast enough, so they passed legislation.

What are we doing? Well, we've been working on this software factory for a long time, and actually we have multiple and they're in different business areas. One of the things that we're working on is trying to standardize that, but really standardize it at the data layer. As much as everybody wants to say we're going to use the same tools across the entire platform, by the time I switch any set of groups to a specific tool, I guarantee that the next tool will be out. So I really want to make sure that we leverage standardizing at the data layer and integrating those tools because I think we're at a place where we have to assume that by next year there'll be something amazing that I haven't even heard of or thought of yet.

Also, we've been able to take that software factory, like I said, and put it in the software dojo. Maybe next year I'll be able to tell you about results. I'm really hoping so.

Here's the example. This is just one example, one instantiation. There's many. I did put tools in here. I wouldn't want to get into a tools war because actually, I'm not that concerned about any given one. It's really that layer that lets me integrate them together that's my biggest push. Years ago we used something called App Hub, but you had to script it out and it was a lot of work to do that. So now I use Tasktop pretty much in this pipeline because it makes short work of being able to integrate these toolchains in and out. But anything that would allow us to do this type of activity.

The other thing that we have to be very careful of, and I think you guys do too, is as we grow we have to actually secure our factory. One of the things that I spend a lot of time with my teams on is how to build things like hacker stories. Before, we used to think, "Test-driven development. Yes, perfect." Now I need to do security-driven development as well. It can't just be, "Oh, I compile." It's got to be, "Oh, how would this be exploited? How can it be exploited?" In every step of that delivery pipeline, integrating things like fuzz testing, which injects errors, problems into your code, security vulnerabilities, penetration testing: these have to be a part of my delivery pipeline. Otherwise I can't actually deliver the right software factory to the government space because most things that we build require a lot of security.

Continuous iterative development. I stole this picture lock, stock, and barrel from the DoD Science Board report, and they said, "Hey, here's waterfall up at the top." Then we got down to agile, but somehow we just couldn't get it all the way out the door, and now this is how they refer to agile and DevOps: small single-piece, low-bite-size batches across the piece. That's not how we've worked historically. This will be a new thing for a lot of our government customers.

Metrics. Again, I'm getting the T-shirt for SLOC even if they won't let me wear it to work, because it has to go. They're always like, "Oh, well, so-and-so had that much SLOC." I'm like, "Oh, that's brilliant. Were they writing in Ada? Were they writing in .NET? Did they have Node.js? What was this magical SLOC that you're telling me about?" I wouldn't even mind if you measured it if you didn't call it a productivity metric. That would be okay.

I saw a great presentation yesterday with the Dilbert cartoon. Probably you guys have seen this cartoon before where in Dilbert he goes, "I'm going to go code me a minivan." That's exactly what you're incentivizing people to do.

So what are we looking at? Velocity over time. Velocity over time is goodness because it allows me to look at some predictability data. Cumulative flow. Love cumulative flow. Burndown. They called these specifically out in the report. And then cycle time and lead time. Cycle time, lead time, mean time to repair, and then as I secure my factory, mean time to detect, mean time to attack, mean time to get back to a known state. All of these things are excellent metrics, all of them better than some of our historicals.

We have a lot of programs already doing this: F-16, F-22, F-35, Aegis, Orion. All of them are on their journey. All of them have some implementation of agile, and some of them DevOps. In the different areas, I've got things like number of teams, the practices they've implemented, the tools, metrics, and where they're at in agile architecture and automation. These are all different.

F-16 has made some really good progress. One of the things that they've been able to do is improve collaboration between the agile teams and program teams by scaling agile with SAFe. F-22 has been doing some really cool stuff in product-centric cross-functional teams. They've got work cells that use pair programming for instantaneous product review. They've got a more modern tool stack and pipelines, and they're really focused on user-centric outcomes. F-35 has been working on PI planning, work decomposition, tracking, and the communication with stakeholders. A lot of the work for them is about planning, communication, and stakeholder involvement. Aegis has a huge scale. Their single greatest benefit was improving agile planning and execution by moving to a flow-based system to manage scaled agile. Orion also has had a lot of benefit from communication. Release-planning events revealed dependencies and resolved issues ahead of production. That was a big deal for them.

Rethinking software sustainment. This one is interesting to me because I'm starting to think about platforms, design patterns, telemetry, and the factory. If software really is immortal, and if it's not just a thing that we build once and throw over the wall, then the sustainment model has to change as well.

Investing in the workforce. We've been doing a variety of things. We have communities of excellence. The DevOps community of excellence only has about 500 members in it right now. I have about 1,800 members in the agile community of practice because it's been out there a long time. It's helpful; it's just not the end. It's a good mechanism for communication.

Team safaris have been really helpful, which is when a team is doing something really well, bringing another team to watch as opposed to telling them. It seems to provide a lot more context and tacit knowledge.

Pairing took a long time for us because I had a number of customers that said, "I am not paying two people to do one job," until they really started seeing the benefits of how much faster work could get done with this.

We have an improved tuition strategy. Before, you could only go to school for your master's or your PhD, and it had to be in these specific fields. Now they're funding things like certifications, and they're even providing funding for things like meetups and stuff like that.

We have a variety of hackathons. Right now, our current hackathon is a machine learning hackathon using the Sphero robots. We're using the Robot Framework, and we have different teams at eight different sites across Lockheed Martin, and then Shark Tanks. We're doing a lot to invest in the workforce, but again, we've got a lot more to go.

Machine learning. Again, we have a community of excellence for machine learning. We're doing a lot of work in this area for different things: autonomous vehicles, autonomous drones, all of that stuff. This one where the DoD Science Board said leveraging machine learning for IV&V, this is the only problem we've seen, and I don't know exactly what the result or how we could maybe fix this is. Currently it seems like it's not quite mature enough to do the job.

Here's what I'm referring to, and maybe it's just because of how we're doing it. What they've found while they're doing this integration, verification, and validation and trying to do machine learning for it, is that the software is learning something different in the development environment than it learns in the test environment, than it learns in ops. They don't know what that difference is, which is making it difficult. One could argue if we didn't have snowflake environments, this would go away, but that currently seems to be a barrier in this space for us.

I got done early, which is awesome because usually I talk too much. But it's cool because you guys will ask me great questions. That's my presentation. Next year I can tell you what the impact of this DoD Science Board report is. Also, I can tell you that I support the NDIA working group to help build artifacts and things to enable our government counterparts to make this transition.

Q&A

Audience: Right now I work at Silicon Valley, but before that I was in Army acquisition. The first question that always comes to my mind when I look at this is, I remember my Army program getting an award from the Army not all that terribly long ago, five years ago, for being the first Army software program to implement Agile with our two-year software release cycle that we achieved. I remember back joining Silicon Valley, and I have to laugh at it. But really what we got the award for when I look back on it, we weren't really getting the award -- well, yes, we did get the award for reducing the software release time down to two years, but really we got the award for changing our contracting strategy to support it. When you look at trying to go as far agile as this DoD Science Board is trying to say, my first instinct just says, do you actually get the sense that as a contractor, I often felt that the way my contract was written did not actually reflect what the government seemed to expect me to do when it came to implementing agile.

Robin Yeman: Yes. There's still some communications things that we're going to have to work on. One of the things that I would recommend, and I've even recommended it to people that were here, is the DAU, or the Defense Acquisition University, is a great program. However, even though they're making this change, they've only got one class in DevOps. One. There's a lot of work, and they could really leverage a lot of that experience from the commercial space to bring that into Defense Acquisition University, or even maybe take advantage of things like MOOCs or other online training. But there's still a lot of knowledge that has to be given on the customer side too. Otherwise, we're speaking two different languages.

Audience: The other question I had was on the sustainment side. Do you think we're actually seeing a shift in the defense procurement culture? Certainly I know that every program in the Army was structured, and I believe that the Air Force and the Navy were doing it basically the same way: specifically, there was a difference between the acquisition branch and the sustainment branch of the same program.

Robin Yeman: Yes. I volunteer a lot for what's called the Government Acquisition Organization, or the GAO, and the big conversation that we've been having is the color of money, which is exactly what you're talking about. This problem has to be resolved. It hasn't. Because once I start doing things like DevOps and I'm getting into continuous delivery, when does sustainment really start? What does that look like? And if software really is immortal--

Audience: Yeah. No, we had to do some very interesting gymnastics around the color of money to do anything even vaguely agile.

Robin Yeman: Yes. That's definitely a barrier. Now, if I may, you ought to look up the DSB report because there were, I think, seven areas that they were asked to look at, and the two questions you had were two of them.

Audience: I lead a center of excellence within my organization, and we've done things like do success stories and kind of ingrained ourselves on video crews and really selling and evangelizing. But you mentioned team safaris as well. Can you expand a bit more on that? What's the structure and how do you do it?

Robin Yeman: What I've found is it's really hard to tell people what they're doing right or what they're doing wrong, because everything they look at is with your lens. I got this from my friend Suzette Johnson. I didn't invent this. I was explaining to her how difficult it was to demonstrate. So what I did was take her idea.

We've got teams, like I have lots of teams, let's say in Moorestown, New Jersey. I have lots of teams in Denver. If there's a team, especially in the same facility, doing something exceptionally well and their coaches are saying, or the other coaches are saying this is a problem area, we'll bring in the other team to visit. Just to say, you know what, look at how this team runs a daily standup or maybe sit in on an inspect and adapt workshop.

The one most interesting thing I've found is we're really good at picking out what somebody else is doing wrong. So it usually turns into quite a good two-way feedback because they're like, "Well, I don't know why they're doing that." Basically we literally just go walk over and watch them. Then it also does something interesting too, which is they start talking to each other. You would think that would be a normal thing if we were all in one facility, but it's not. If you're on a different team in a different program, we almost think that you're in Jupiter. So it's provided a lot of benefit from there too.

Audience: You showed different teams doing slightly different things. Can you talk about why that is and why you gave the freedom for those teams to choose as such?

Robin Yeman: I think that it's going to be really important, even though we want alignment, to ensure that teams can have freedom to implement what works for them. I come from a culture where we try to make one size fit all, all the time, and it's just not true. It's a difference between working with my multimodal biometrics team that's got maybe 12 to 14 teams in three different locations versus working, let's say, with Aegis, where they're all either in Moorestown or Syracuse. It's different. They have different problems.

The things that I want to standardize on are things like cadence and synchronization, maybe focusing on business outcomes or objectives and key results, and maybe some of the best practices like that. But other than that, I truly believe for each of your programs, you kind of have to build your own lightsaber. I think it's more important to let the teams own that work and do it the way they think would be best. It would be like me telling another coach to operate this way. Well, you can't operate exactly how I operate. I'm different. What works for me isn't going to work for everybody.

Audience: Just to follow on from that, say you're dealing with some teams, and especially in government organizations you've had people who've been there for years, and you come up against the people who say, "Look, we've done it this way. We don't want to change," and they're the people who want to hold everyone else back. How do you manage them and get them on board with your dream so that everyone's pulling in the same direction?

Robin Yeman: I don't always make progress. Sometimes I totally have run into that. But in other cases, and this is maybe manipulation, I don't know, I usually approach every team as, "I know this won't work for you. I know that you're special and you're unique and your problem isn't like anybody else's," because that's how I know I'm at Lockheed if that's what people tell me.

I start by that and I'm like, "So if we really want to get people off your back because they think there's something to this, how about we just pilot it and you can show them how none of this works? Go to grins and giggles, just follow it verbatim, let's see what we get." Which overcomes the barrier to, no, it won't work, because they can prove that it doesn't.

Once that happens, once that initial barrier goes, you'd be amazed at how actually amazingly smart the folks we have on our teams are and how they can make it work. Once you take away, "You know what? I know this can't work in this environment because you're special," all of that kind of argument seems to go away. That being said, I've still had teams that were just like, "No." And that's fine. It's not about being DevOps or Agile or anything. It's about delivering better business outcomes. However you get there is totally fine with me.

All right. My time is up. Hope that was helpful, and I really do appreciate you guys coming because, like I said, there's some great speakers in this block. All right. Thanks.