Suzette Johnson

Good morning. Thank you for being here with us as we share our Agile DevOps journey with you.

While Robin and I may work for competitors, we also collaborate and work together as teammates often to build systems and deliver value to our customers. This is one of those times where we want to come together and collaborate again and share our experience.

With some of these challenges and lessons learned that we're going to share with you, I'm sure in some cases you'll resonate with these experiences. In some cases, it's nothing new. You might have seen this also in large organizations. Hopefully, there will be some things that you can take away and help move along in your own Agile journey.

I am Suzette Johnson. I work for Northrop Grumman. I am, by trade, a systems engineer. If you're not familiar with what systems engineering is, that might be an unfamiliar term unless you're from the DoD world.

Systems engineering: we build really big systems. Someone is pulling together the large system requirements, helping to understand what our system needs to interface with, what kinds of data we're collecting. My job is to pull together that information and make sure that our Agile teams understand that big picture that they're building against.

I am located in the Baltimore, DC area. That's where I sit. Although, that's where my office is. I'm not there very often. I'm usually out supporting our programs. In my role now as the Agile Center of Excellence lead at Northrop, I go out and help our programs get started. I go out and meet with our customers and just kind of help them along at their Agile DevOps transformations.

Robin Yeman

I'm Robin Yeman, and again, Lockheed Fellow. Similar to Suzette, I've been working at Lockheed over 20 years. I've been doing Agile for probably the last 15.

For those of you who aren't very familiar with Lockheed and Northrop Grumman, we build really big systems. We're government contractors. We don't actually work directly for the government, but we build systems and service for the government.

Those systems, just like Gene was mentioning, things like bombers and planes, radar and satellites, submarines and battleships. We've also got a whole range of IT systems that we build. Things like data analytics, multimodal biometric systems, and even cyber defense systems to protect against things like WannaCry, which just came out and made us all want to cry.

Suzette Johnson

What we're going to do is take you along our journey. We're starting here with my first adventure, and then Robin will talk about her first Agile adventure early on.

It's really important to note, if you're not familiar with companies like Northrop Grumman and Lockheed, they're really big companies. You could have anywhere from 40,000 to 60,000, 80,000 engineers. We have a lot of different levels of maturity that we're facing.

My Agile journey actually began in 2005. I was working as a systems engineer on a large data collection system, and this large system was responsible for collecting data, processing data, and providing, eventually, analytics to make better decisions that the warfighter would use, that would provide safety to others, whoever those others might be.

My team was 125 people, but we didn't start there right with 125 people. In May 2005, we had four of us, and we were getting started. As part of that, because of the quick-react responsibility that we had, we were trying to figure out what the engineering process should be, and that was my job, was to define that engineering process.

I wasn't sure what it was in 2005. I knew the answer wasn't waterfall. So I just interviewed the really smart engineers, those that are always thinking innovatively, have new ideas. I went to this guy and I was like, "I need something. I'm supposed to write this engineering process, but I know we need to deliver, whether it's every few months, every couple of weeks, whatever that timeframe is, we're going to have to be ready to deliver something."

He whiteboarded this Agile process, and I'm like, "I don't know what it is, but we have this problem space, and Agile seems like a good framework to use to meet this problem space." So he and I went over to our technical lead and said, "Hey, we've got this new idea of how we think we need to implement in terms of a process and a framework."

Fortunately, our tech lead gave the thumbs up. He's like, "I don't know exactly what it is either, but hey, let's go for it. It seems to meet the needs of our environment."

So that's what we did. Actually, none of us had any experience with it. That was in May, and by October, we had 125 people on my team.

The significance of that was not only were we transforming 125 people starting on this Agile journey, but we were part of a larger systems environment. We were only building one component that had to integrate regularly with other components, and there were roughly anywhere from 500 to 800 people building those other components.

This was pretty significant because even while we were trying to be Agile, it was: how do we start integrating at scale? In 2005, this was not commonplace. Just even how to set up your Agile teams, how to have product owners for each of those teams. Then we had to have a chief product owner. And we had some challenges at first.

One of the first things we did was give everybody a book to read, Ken Schwaber's Agile Software Development book. We handed that out and said, "Hey, read that book. We're doing Agile." And we had some lessons learned out of that one: not everybody read the book.

So we thought, "Well, maybe we should actually do some training on this." We had Ken Schwaber come in, and he trained our team. But most importantly, not only did he help train us, but he also stayed for a day of consulting and helping us figure out a transition roadmap, which I have been able to reuse over and over again for the programs that I support.

Some of the challenges as we were starting to go through this journey was really focusing on continuous integration, automated testing. We were mostly doing manual testing at that time. Even once we got it figured out, just thinking about, in 2005, figuring out how to do automated testing and continuous integration within 125 people, but then how to start streamlining that over into your large systems integration environment.

Depending on your background, you might not know exactly what that means. But that large systems integration environment is where all these components are trying to take their pieces of the puzzle and trying to integrate them regularly.

But in 2005, we didn't have that all figured out. Our pieces of the puzzle worked really well in our environment, but then you take those puzzle pieces and you move them over to another environment, and all of a sudden those pieces that fit so well, they need a little bit of fine-tuning and a little bit of reshaping to actually work.

So what we did was we first had a "make it work" team. Isn't that terrible? Set up a "make it work" team because your software is not working. So we started that way, and we're like, "Okay, it's time to go into integration, so let's send that make it work team over to integration and have them start working on integrating."

It was a real challenge. But then we started to learn from that and learn how to build our interfaces such that we could test more frequently, and then how to build those integration and testing practices all the way from not just our 125-person product team, but over to that larger systems environment.

Some of our lessons learned from that was the importance of getting people trained, which you may resonate with that as well and seen the importance of that. The consulting with Ken Schwaber and figuring out our approach for improving and for transformation. Also thinking about the continuous integration, automated testing. Again, this was 2005, so these were some pretty early ideas, especially at scale.

Part of that too was looking at my role, which was systems engineering. So how did systems engineering fit into this? Because if you have your development teams, they're off developing. How do I get these big requirements as part of this process? Because everybody in the value stream is part of that framework, and that was our approach to starting to figure this out.

In the end, our customer was very pleased with the results they were getting. We were mostly within schedule and within budget. We were delivering results. What was happening at that time is more and more people within Northrop started to take interest in our program because it was a large Agile program at scale, so it was drawing a lot of attention. We started to share those lessons learned, and that's how our grassroots effort started to emerge.

Robin Yeman

Awesome. Very similar to Suzette, I was in the same boat. Pretty close to the same neck of the woods as well. I was asked to lead a solution for a large asset management system for one of our intelligence customers. We had said that we would do it Agile because the customer required it, which is interesting in its own right. At the time, we're thinking what Agile is or what it should be.

Asset management's critical for a number of reasons. One, security. If classified information gets in the wrong hands, we can have lives cost. Two, we need a lot of transparency for asset management. Why do we need that? Well, because me and you gave our customer, or they gave the government, a large amount of money. So we want to see how our investment's being managed. All of this was a really big deal at the time.

I went ahead and tried to learn everything I could about Agile. I didn't even know what it was. I read books from Ron Jeffries on XP. I searched the web. Crazy enough, this was actually only about a year after the Manifesto was written, so there wasn't a lot of information out there. But what I read was cool. It was awesome.

Downside, it didn't solve some of my problems. Everything that we build is huge, and like Suzette was saying, first thing I was like, "Well, how do I work with multiple teams? I don't have a single team. I've got 85 to 100 people. How do I do that? It says that we only need seven plus or minus two."

I wanted to know how I handled, I'm not sure in the commercial space if you guys run into this, but how I handled having to have classic control gates. In the government, they said, "I want Agile. And oh, by the way, I also want PDR, CDR, TRR," all kinds of these phase gates, because they didn't know as much either.

And three, what happens if we really did contract for a set of requirements? It's pretty often that if the government says that they want a multimodal biometric system, they really do want all three modes of the biometric system. They don't usually change their mind on that. So how do I handle that?

Well, I did what anybody else would do. I made it up.

So we go ahead, we make it up. I'm like, "Wow, we need multiple teams." Integrated product team, we've done that before. We can do that. We got this.

One-month sprints. Why? Because somebody read it, and it must be true if it's written down. So we'll have 30-day sprints, which actually turned out to be quite long for us.

We had a separate systems engineering team like Suzette was talking about, and a separate test engineering, because why? I needed somebody that just concentrated on those phase gates. I don't know if they were super important, but had to make sure I got all of those artifacts completed.

We set up things like continuous integration, a little bit of automated tests, but not too much. We didn't have retrospectives. I'm pretty sure we thought we had real work to do, so we thought, "Eh, we'll just skip those." Yeah, obviously an issue.

If I went back, wouldn't do it the exact same way, but I learned a lot.

Now, what was the outcome? The customer was thrilled. They actually got the system they wanted, so that was goodness. They became known in their area as this leading-edge, new development methodology. I didn't even realize we're on the front end. So they became really well-known.

I became known as a subject matter expert. Pretty much if you can sell something, then you are the SME. And by the time 2004, 2005 came along, everybody and their brother said, "Hey, we're going to do Agile. Robin's the SME." I can tell you how I broke it, but Robin's the SME.

I made a lot of long-term friends. But the coolest thing was we got to define the art of the possible. The government became known, together with us, for everything that they're not usually known for. They got to be known for being fast and engaged and flexible, and that's always not the case.

Lessons learned for us: communicate both success and failure. I'm type A. I don't usually like to communicate failure too much. But you'd be amazed if you allow people to engage in your project, how much they want to share, how much they want to help you. So do share both.

I am a control freak, type A, like I said. I really needed to learn to begin and end with the team. I used to have this manager who said, "Robin, it's great that you're moving forward. Have you looked behind you? Is anybody there?" I was like, "Oh, shoot. Got to bring them with me." So bring the team. I would've never seen the amazing solutions I saw. And include everybody on the teams. Remember, we had a separate systems engineering and test, and now I learned, hey, we need to put them all together.

The last one was really interesting: measure and review your results regularly. Oh, that retrospective that we didn't actually need. So why was that? It was interesting that we would try to solve the problem that wasn't actually the problem until I took a closer look at those metrics. "This needs to go faster." Eh, that's not really where your bottleneck is. So pretty similar to Suzette's.

Now, we're going to tell you a little bit more about our journey because it's fast-forward time. So Suzette, back to you.

Suzette Johnson

Right. So the grassroots had started to emerge. We were sharing our successes and some of our challenges, especially scaling and figuring out how to go across that value stream, how to pull in systems engineering, and how to bring in, again, that larger systems integration and test environment, not just the local product development integration and test that we're mostly used to.

There's a couple things that really made a difference. The first thing I did is I was trying to figure out how does systems engineering fit into this? Because that was my team. We were trying to figure out how do these formal requirements, so when you're talking about PDR and CDR, that's your preliminary design reviews, your critical design reviews, test readiness reviews, and all these different milestones that had to fit in with our Agile framework. And we didn't have all the answers in 2005.

What I did is I went to one of our VPs. He knew who I was. He had been part of my interview process when I went to Northrop. But think about it, I was just a person on a program. But I had a problem I needed to solve again, and that problem, I wanted to have these every-other-Thursday meetings, and I wanted to pull in engineers from the company to help start figuring out how do we solve these problems and bring these components into our Agile framework.

I went and asked the VP, I was like, "Can I start these meetings? Do I need to ask permission to have this meeting?" He said something to me that was really powerful that I have thought about many times over the years, every time I wasn't really sure if I had permission to take that next step. What he told me was, "If what you're doing is in the best interest of our company and for our customers and program, then you don't need to ask permission. But what I do want you to do is keep me informed so that I know what you're doing. And when the time is ready and you want to present an idea, just let me know and I can make that happen for you."

That was really powerful to me because all of a sudden, I had this freedom I hadn't had before where I felt empowered to start doing these things. As I left that office that day, as I was about to walk out that door, I paused and I said, "Oh, and by the way, will you feed them so they actually show up for these meetings?"

But they did. I don't know if you have that experience, but that really helped to get these guys in. We started what we call now the Agile Community of Practice. So it all started with that moment in time.

Now we have over 1,000 people as part of our Agile Community of Practice. We still meet every other Thursday. We don't provide lunch for that many people anymore, and they are across the country, sometimes even globally, as we include our global offices into that.

That's been a real exciting time, and I love the community of practice because that is where people get to share ideas. They share their lessons learned. There's no judging, and you have that freedom.

But something else happened with that as we were starting to figure out not only the Agile framework and starting to pull in DevOps. Again, I was still working on this program, and all of a sudden, people, they were like, "Okay, you've convinced me. I've heard the success. We want to start transitioning to Agile in our program. Can you come help us?" I'm thinking, "Well, how am I supposed to do this? I'm on a program full-time."

So again, I ventured out, and what I started doing was interviewing engineering directors that had budget to hire me. I said, "We have this need now in the company, and the need is that we need now a supporting infrastructure. As people are starting to move towards Agile and towards..." We really didn't have the concept of DevOps at that time. That was around 2009 or '10.

I said, "But we need a supporting infrastructure. People are asking me to come help train or coach and provide support, but I don't have the mechanism for that. Can you support me?" So they did, and that is how our Agile Center of Excellence got started, was that one individual who was willing, again, to support an idea. We had a problem space, and here was the next solution for us as we were scaling.

Our Center of Excellence has been very instrumental in providing the training across the company. We have trained thousands, in excess of 10,000 people within the past three to five years. We provide the coaching to get you started, because if you're learning a new skill, it's really helpful to have someone coach you and support you as you're learning that new skill. We can get you up and running faster because of that, and also the supporting artifacts.

So our Agile framework is, we can modify it, but in our Agile framework, we've talked about some of these traditional DoD acquisition type of milestones that we are required by contract to implement. And so that is kind of where we were starting, where we've kind of ended up with our Agile Center of Excellence and these improvements.

And that same team, remember we had the Make It Work team that was helping integrate our software? Well, they learned, too. Through our continued efforts with their commitment to their integration and their testing, that same team now runs over 10,000 tests a day, somewhere like 10,000 to 15,000 full regression tests every day at the lowest level.

Really seeing, again, these results and communicating these results, it goes back to the art of the possible. But the reason that these things happened is because of the continued dedication to continuous improvement. So sometimes we do things really well. Sometimes we have some challenges and maybe don't do things so well, and we have to learn from it. But the important thing is to make sure we learn from it and we continue to improve.

Robin Yeman

Again, similar to Suzette, that's how we met.

Fifteen years later, so where are we? The problems have gotten bigger. The teams have gotten bigger. I get to work across a number of lines of business, and I feel very privileged to do so.

I spend time with Aeronautics. We've got the integrated fighter jet team, which is composed of F-16, F-22, and parts of the F-35, and they are doing absolutely wonderful with their Agile transformation. I even had one team that went on for the F-16 to deliver 50% ahead of schedule, below cost, record quality levels hit in hardware integration. But the coolest part was one of the engineers who'd been there for 30 years and was pretty sure that he couldn't learn anything new told me he'd never had so much fun. So that was awesome. He had a great time.

I work for Space Systems. So in Space Systems, in Denver and Houston, we've got over 45 teams building the next-generation spaceflight. They're completely Agile. They get together quarterly to do release planning, and last August, they met their first flight milestone, so awesome. Every time I say continuous integration and people say, "I'm not sure you can do that," I'm like, "Yeah, so Orion can." So pretty cool.

I support Rotary and Mission Systems. We just got a new name because we bought Sikorsky. Rotary and Mission Systems has Aegis in it. Aegis is actually one of our largest Agile teams. Aegis is a battleship, and it has over 142 Agile teams that are delivering capabilities to multiple customers every single day. While all of Aegis is not under continuous integration, we have huge pieces of it that are, and we continue to get better all the time. So it's usually pretty exciting to see my teams.

What else? I lead the community of practice. So just like Suzette, we've got a DevOps community of practice. This week I had the pleasure of being able to have a group come in. They're working on one of the submarines and explain their continuous delivery pipeline using Ansible and Docker and Kubernetes. So almost as cool as Netflix, except they're building a submarine, so that's pretty cool. Love to meet these guys.

I continue to work with my customers and understand my customer goals. Usually try to make sure practices marry up with their goals. It's all about the business outcome for us.

I got my Fellow. I'm one of the Lockheed Martin Fellows, and that was pretty cool because I tell you, I spent all this time spending... Thank you. The "I'm not worthy." I just meet so many really brilliant people that I'm just amazed that I get to hang out with them.

We have the most active community of practice within the company. We have several book clubs, and I am trying to talk them into the Target Dojo. I keep telling them it'd be called the LM Hangar. So next year, I'll come back, and you tell me if I can get there. But that's the goal because I love this dojo idea.

What's the lessons learned? Build more champions than we need. Me and Suzette both spend an awful lot of time on the road. We have not nearly enough to really be able to continue to make a dent in these organizations.

Look both internally and externally for new ideas. We have a tendency for some of our engineers to just continue to look down. They've worked on programs for the last 30 years. We try to get them to go to meetups and to conferences, things like that, because we kind of forget that some of the answers come externally, hence working with our competimates, which is a great idea.

Really build your infrastructure. For us, we tend to shortchange it too much, and in many cases, I'm constantly saying, "Hey, give the team the environment they need to get the job done." That'd be cool. I think they said something like that in the Manifesto.

And then make sure you're not talking past each other, because we are. We do all the time. I've had the opportunity to be a program manager and a subcontracts program manager and a test engineer in software and systems. One of the reasons I've done that is so I can learn the language, because every time I talk to people about why we should do something, I have to use their language in order to actually get them to understand what I'm talking about.

So big themes that we've seen. Working across the organization, so both Suzette and I are doing that. Really focus on your teams, and you guys already knew that because I've seen lots of cool presentations so far. Really make sure that you put that money into your infrastructure. We still keep working on it. And measure. That whole retrospective thing, I think that's probably a good idea.

Up to you.

Suzette Johnson

Okay. All right. So taking those four themes and sharing our summary of those in the next slide of some of the takeaways as you are maturing your Agile journey. I know we're never done. I think that's the biggest lesson learned that we've had is we are never done yet.

So while we primarily started all of this with these large software-intensive systems, as we work across the organization, we're finding how do we apply these DevOps and Agile principles with some modification of practices in software, firmware, and hardware environments? And what does it actually mean in a hardware environment where some things have some flexibility of change and some things don't have that flexibility of change because of the amount of cost associated with hardware?

As we think about working across the organization and working across these domains, I would be interested also in talking, I know I've already had a couple conversations yesterday with others that are working and looking at Agile and DevOps principles in hardware. I think that would be very valuable if we could collaborate. And I have a couple other points of contact that I've been trying to figure out: how do we mature this practice, and what can we learn from each other?

The importance of cultivating teams, because as Robin mentioned, it is really hard to grow more leaders in this area. So when you have a company of 65,000, 100,000, whatever it might be, to continue to grow leaders and to have the discipline and the sense of urgency to carry that as you scale.

I think the cultivating teams and building that trust and aligning incentives is very important. Some things that we look at also in our performance measures of how are we performing and how are we measuring the performance of our Agile teams, not just the individual contributor.

Again, building the infrastructure and that integrated tool set is important to get them up and running. And for us, we don't have just one solution. We have to have multiple solutions. Some low-cost open source solutions to the bigger dollars for the very large type of multimillion-dollar, $300 million, $500 million programs require a different type of infrastructure.

And some takeaways with measuring progress and success and defining what does success look like. When I go out and work with teams, there's always a few questions that I like to ask. One is, what problem are you trying to solve with Agile and DevOps? What is it that you're after? What does success look like for you?

Most importantly, because with large organizations, sometimes people say, "Well, what level of Agile maturity are you within your organization?" And I say, "Well, that just depends on where you are in the organization." It's huge. We have some people just beginning that Agile journey. We have some that are very mature. But regardless, whatever team I'm working with and whatever their journey is taking them, we always want to ask, and I always focus on, are we better today than we were yesterday? Because we're never finished. We will continue to learn and evolve.

Robin Yeman

The only thing I would ask is this: how do I actually tell the test pilot for the F-16 that fail fast is a good thing? I don't know.

Suzette Johnson

Learn early.

Robin Yeman

We need some better... Yeah.

Suzette Johnson

Yeah. We've got to work on the nomenclature. So if you got an answer, you let me know. Right. Thanks, guys.

Robin Yeman

Thanks.