Suzette Johnson

I'm Suzette Johnson from Northrop Grumman.

Robin Yeman

And Robin Yeman from Lockheed.

Suzette Johnson

And yes, we do work for competitors, and sometimes that's kind of confusing. But what people don't realize is that we are also teammates at times, where we work together to build systems and build solutions for our government customers. So that's the stance we're taking today.

Robin Yeman

Mm-hmm.

Suzette Johnson

Coming together as a team, sharing commonalities that we have found along our journey.

Based on the stories we've already heard this morning, it is obvious that you will resonate with some of the stories that we share, and hopefully there'll be some new takeaways that you can take on your journey.

And most importantly, one of the things that we'll be sharing is how DevOps is applied in some very unique types of environments.

Robin Yeman

Awesome. All right. Let's see if I can figure out how to do the clicker.

Suzette Johnson

So here we are.

Robin Yeman

Here we are.

Suzette Johnson

Continuing our journey. So let's go ahead and jump right in.

All right. So we want to give a little bit of background about our companies, because it is very different than many of the companies you might be working for, the type of space that you might be working in.

So I work for Northrop, and in 2016, we had approximately 65,000 people, and we did over $24 billion in sales, as compared to Lockheed, who had approximately 90,000 employees and did approximately $40 billion in sales. Slightly over that, I believe.

So what kind of systems do we build? Well, here's a picture of some of those systems. We have autonomous systems, unmanned systems, missile defense, cyber solutions, space exploration.

And one thing that makes these very unique is the fact that it's not just software, but it's software, firmware, embedded systems, hardware, lots of hardware integration, a lot of security concerns. In some cases, we only have one shot at getting it right when we deploy, right? So a very unique space for us.

Robin Yeman

Absolutely. All right. So just like Suzette said, same thing: Lockheed, very large. We've got five main sectors. We've got space systems, aeronautics, rotary mission systems, because we acquired Sikorsky this year, and missiles and fire control. And then I said five, but the fifth one is really internal. We call that enterprise business operations.

But I have a little bit of video here because I thought they could explain it better than me.

Technology that enables cars to drive themselves.

Routine flights to Mars.

Fusion reactors that produce limitless energy.

To most people, they're pure science fiction from the world of tomorrow, like something out of a movie.

But at Lockheed Martin, we live on the cutting edge of physics, material science, technology, and engineering. We obsess over things most people only imagine.

We're at the forefront of the science that makes them real, and they're available when it matters most.

It's why we're always thinking about new ways to prevent the unthinkable and building the most advanced fighter the world has ever seen.

It's how we redefined what a combat ship looks like.

Why we're harnessing the tides to generate electricity and protecting our most valuable resources.

And while we don't know what's going to change the world next, we're probably already working on it.

Thanks. Yeah, so we pretty much build everything.

Suzette Johnson

Right.

Robin Yeman

Everybody's like, "Oh, yeah, you got the fighter jet." Yeah, we do have the fighter jet, but we've got quite a bit more.

So I'm going to let you go to the next one.

Suzette Johnson

Okay. All right, so what we're going to do is take you along a journey. We're going to start with where we were over a decade ago, and then as we go through this journey, we'll fast-forward to where we are today.

So let's start at the beginning.

My first adventure started in 2005. I was a systems engineer on this new product development team, and this team was responsible for developing a system that was going to collect, process, and disseminate information near real time in physical situation awareness and provide that information to our analysts and to the warfighter.

Part of the things that made this complex was the security around it. We had to build security in. It had to be tamper-proof. And also, based on what you saw, these are really big systems that we're building. So our system was going to have to interface with a lot of other systems. That meant our ability to be able to take in different data and different types of data format, and then being able to send that information out in a meaningful way.

So in 2005, as this product team was being stood up, I was given the task of defining our engineering process. And I wasn't really sure what that engineering process should be, because I knew with the mission, we were going to have to deliver things quickly, in three months, sometimes in a matter of weeks. So the question was: what kind of framework are we going to use to be able to develop systems that quickly?

So I knew the traditional model wasn't the answer, but I didn't have the answer. So the thing that I knew best was I just went around and talked to what I call the really smart, innovative engineers who are always thinking up new stuff. And I asked him, "I've got this problem. What do you think should be our engineering process? That traditional model, I don't think that's going to work for us."

And that was my first introduction to Agile, as he's whiteboarding this Agile process and the importance of regular validation, the improved learning that we were going to have to make sure we're building the right thing, and getting that information out.

So we went to our technical lead because we had never done this before, and we asked him, "What do you think? This is the framework that we want to use."

And it was interesting because we had a slightly different mindset than you might experience in different areas, is that we felt it was more risky not to change than to keep doing things the same way. So we took that risk, and we decided to go with it. And that started a very interesting learning journey.

We went from October of 2005 from five people to October of that year with 125 people, and all of a sudden we were Agile.

So how did we do that, right? So the first thing we did is we gave everybody Ken Schwaber's book on agile software development with Scrum, right? And everybody read the book, and we found out that's really not helping. Great book, but it wasn't providing the framework, the training that we needed, because some people never actually read the book.

Robin Yeman

They just said they did.

Suzette Johnson

Right.

Robin Yeman

It looked good laying on the desk.

Suzette Johnson

So the next step was we actually invited Ken Schwaber into that program to provide our training for us, that initial step, and that was huge in developing a common foundation, a common language, a common mindset around how we were going to collaborate to build the system, which was critical.

So we get on board from at least that perspective, but then we start encountering all these other problems, right? So 125 people, right? We had approximately 10 small Agile teams, and we're each building our piece of that product. I'm sure you are probably already knowing what comes next, right? And that is the integration challenges.

So we're like, okay, every time one team makes a change, things are breaking. So we start thinking about... We get together, and we think, "What's out there? What are some other things we can do to make this better?" So that's where we start with continuous integration.

And continuous integration, at ASI, especially at that time, was not trivial. So we start down the continuous integration path, and we start talking about the importance of test early, test often, automate. Excellent. This is 2006, and we're on our way.

But remember, this is a big system, so our product had to go integrate in a much larger integration environment to be tested with all those other systems before it could be deployed. That made another interesting adventure for us because, well, it worked in our environment. I don't know why it's not working over there.

So we actually had a small team that would walk with our software over to that space and help integrate our software to make it work. We called them the "make it work" team because that was their task.

Robin Yeman

I like it. It's awesome.

Suzette Johnson

Journey. Right. So we realized that wasn't really the answer, right?

So we then had that next problem of not just integrating in our product suite, but how do we integrate across those other systems, and how do we start to flow? Because what happened is every time we tackle one part of the value stream, the next part of the value stream starts to become a bottleneck.

The importance about that, though, was we were never satisfied with just good enough, right? We're always continuing to evolve, always continuing to learn. And that same team now runs over 15,000 automated tests a day at the lowest level and full product tests, right? So that was their commitment to continuous improvement upon their journey.

The other thing that we had to learn with all of that is making sure that everybody was trained. The small steps of improvement, building quality in wasn't something we could tack on at the end. And most importantly, being clear of what our goals are.

So with every change that we went through, every testing change and improvement we wanted to make, we actually had improvement goals so that we could kind of measure ourselves about what we said we wanted to do against where we were headed.

Robin Yeman

All right. So I have a very similar journey to Suzette. A little different.

In 2002, I had a customer require us to do Agile. Now, I don't know about you guys and your companies, but if it's in the RFP, yeah, we'll do that. We just didn't really know what that was.

So I had a short turnaround to figure that out, which was good. It was actually only a year after the Manifesto was written, and it was to build a large asset management system. The team was only about 80 folks. It was a midsize application.

We had to develop continuous integration. All of that was, I mean, it's nothing like what it looks like today. Back then, I can tell you I had daily stand-ups, and I had teams that were doing some integration, but like Suzette said, breaking each other every other day.

We had separate systems engineering and software teams, so a little less cross-functional than you would've liked.

In order to make that transition, did everything we could. Read books, just like you guys. We invited people in to speak, but it was pretty new. There wasn't a lot out there, and actually at the time, everything you read was about the team of eight.

So all of the things that I had to try to figure out were, how do I make the thing with a team of eight work with a team of 80? So we kind of had to make it up as we went along. And some of it was good, and some of it was probably less than good.

But in the end, we had multiple 100% award fees. That customer was very happy, and we went on to do a lot more Agile projects. Again, I'm not sure about your company, but if you can spell something, then you pretty much are the subject matter expert.

So from 2002, they're like, "Oh, Robin did Agile. She's the SME."

Suzette Johnson

Yeah.

Robin Yeman

And so, yeah, I got to be the SME. I've had a great time, and since then we've had a number of programs.

So I spend time working with Aegis. We've got 142 teams building the combat system. You saw the little handle.

I am now working a lot with F-35, so we're building a DevOps pipeline so that we can automate the entire delivery stream.

I spend quite a bit of time with Orion out in Colorado. Again, we've got about 42 Agile teams across Houston and Denver. So pretty much all lines of business, and we're building all different types of systems.

Suzette Johnson

Okay. So let's talk about what happens when you have these small steps of success that you continue to build on. People start hearing about it. They start seeing what other teams have done. They become interested.

We actually started an Agile community of practice in 2006, where these teams that were starting to become more interested in Agile, at least they had a forum to go to, to exchange ideas, lessons learned, and to just hear from each other about some of the challenges they were facing and how they overcame them.

Our Agile community of practice is still strong with over 1,000 members, and we still meet every other Thursday to share these ideas, and it's still growing. So wasn't sure if initially it was going to last, but it definitely has lasted and continues to grow. So we're excited about that.

But within the organization, as more programs and more teams start hearing about these successes, they want to start transforming. They want to start adopting some of these practices as well because they want some of the same results.

So what happens is in the organization, now we need to start being prepared to provide the infrastructure that is necessary for change. It's one thing when you have a couple small teams or small programs that are moving towards Agile and these DevOps principles, but when you start seeing that grow across the organization, you have to think about, how are we building success, and how are we building that infrastructure so our engineers and our managers can be successful in this new environment? Because now things are changing. How we are starting to report progress has to start changing.

So that's when we set up a center of excellence, and you probably have something similar in your company, or maybe it's called something slightly different. But the center of excellence is really providing a team that helps develop that infrastructure of what are the auditable processes or procedures that we have in the company.

And those auditable procedures, at least in our company, are really important. Because, again, because we are government contractors, our audits, we take those very seriously. So we want to make sure our auditable procedures, as we review them, are actually enablers, or at least at a minimum, not impediments to moving forward. So that's one thing that this team will do.

The other thing that we start to do is building the guidance. Because in a large organization, you don't want people to have to keep inventing things over and over and over. So what is the common framework we can put in place to help teams get up and running quickly?

So we'll put that framework in place of those processes, knowing that you're able to modify them to meet your product needs or your unique customer needs, because sometimes our customers are very much integrated as part of our program, so we have to have some flexibility to tailor that according to their needs as well.

And then the training. That's huge. We learned that early on, that we can't just tell people, or people can't just say, "Oh, I want to go do these Agile and DevOps things." What are the resources we're providing in terms of training?

But training starts to look different over the years as you start to scale. Initially, it was training at the engineering level, training the engineers. And then all of a sudden we start to realize, well, it's not just the engineering level, we need to start training the management level as well because their life is changing a little bit.

And then we realized, well, it's not just engineering and it's not just management, but how do we start training our leadership as well so they understand, one, what is this Agile and DevOps thing that we're talking about? What is the perceived or the hoped-for impact to the organization, and what are their either responsibilities or how will they participate with us through this journey, and what kind of support do we need? Because we do need their leadership support.

Because as an organization, if you're only doing it at the grassroots level, you don't always have the authority to make changes in the organization that need to be made. Sometimes it's leadership that you have to look to to help make those changes. So we depend on them to help with those types of changes.

So we train at all levels of the organization. We have trained, probably over the past five years, over 15,000 people across the different classes that we're offering.

But we're also noticing as we continue to grow, because what I found is as you build this infrastructure and you start making it easier to adopt these principles and practices, that more people want to adopt these principles and practices because you're making it easier for them. So then you have to start thinking, well, how do we meet the needs of the global community, distributed teams? How do we do virtual training? So it starts shaping even your training as you go along this journey.

The other important thing that we learned is the coaching. So coaching, again, at all levels. But the coaches are very important because they can help get the programs up and running. They know what resources are available in the company.

So in a large company, sometimes it's hard to always find things and know exactly who to go to. So our coaches help get these teams up and running.

So we're contract-based, so we bid on contracts. We win the contract, and then at award, you want to be able to hit the ground running. So you want to have this infrastructure in place to be able to do that.

And then importantly is the infrastructure. And at this point, I am talking about the tooling for your DevOps and delivery pipeline. And that's, again, in a large company, it's not like you can only have one solution. So we have to figure out what are different solutions that meet the majority of our needs in the company.

Robin Yeman

All right. So near and dear to my heart, we've spent a lot of time training the engineers. Quite a bit of time now training program management, finance, contracts, everybody. So we've included everybody.

But we're growing so fast that we can't bring in the right talent. I had the opportunity to take my son to look for new colleges. He was graduating maybe two years ago because he's in his sophomore year now, and he's going to school for computer science. Interestingly enough, the computer science curriculum looks exactly the same now as it did 25 years ago when I took it. So I'm a little concerned that potentially we're solving last millennium's problem, not coming up.

We've really got to build those. So we bring in a lot of new engineers, but they don't have a lot of diversity. They don't have background in, we can build it, but just because you can build it doesn't mean you should. They don't understand things like make-buy and whether you should go ahead and do that.

We have a lot of legacy technology that we're having to bring them up, and then at the same time, do transition. So transition from what we were to what we want to be.

So what actions are we taking? Well, engaging with universities. We really are spending a lot of time because we need to build that next generation of engineers, that cross-functional, T-shaped engineer that can really solve problems.

We mentor the new employees. So we've got a lot of mentorship programs, both ad hoc and formal.

Just like Suzette was saying, we have a very large community of practice. So we have an Agile community of practice. We actually have a separate DevOps community of practice. They're not one and the same. But they're both very large. We have, again, well over 1,200, 1,300 members for the Agile community of practice and about 800, because DevOps came a little later for us, in the DevOps community of practice.

We do a lot of things like book clubs. We just got done reading The DevOps Handbook, which is one of my favorite books. So everybody's learning as we're going.

We are working a lot with small businesses. The government requires it for most of our contracts, but the cool thing is, is they actually come in with those lean, T-shaped engineers, because we haven't tamed them yet and stovepiped them into their roles, which is good, because we're really large, and we love to do that.

So what kind of outcomes? Right now I'm working with the F-35 pilot training devices team, and they are fabulous. They are so excited. Before I started working with them, they said it was boring, a daily grind. Everybody was just bugging them about whether things would work or not, a lot of failures. But now the team is having fun.

They asked me last week, they said, "So what do we do next?" And I go, "What I really want to know is what you think we should do next. I want to be the enabler to make it so that you can do it, but I think you guys have the domain and experience to tell me what really needs to happen."

We're getting much more engaged customers. Engaged customers, engaged government supporters. Working last week just with PARCA, who called and said, "Hey, can you walk us through again how you would do what we call an IBR, which is an initial baseline review with Agile?"

Our products are getting better and better, more innovative. We're building security into the pipeline. So it's not just DevOps, it's really whatever you want to call it: DevOpsSec, DevSecOps, whatever. But we're building it into the pipeline because that's critical for us.

And really getting our leadership to focus on those business outcomes as opposed to the inputs.

One of the things that we've started doing is measuring employee happiness. And I can tell you, I was over in Australia, and they said, "Well, why do we care?" I said, "Well, because they're actually the people that are building our products, and it's very expensive to hire." There's all kinds of reasons why we care.

I try heavily to get our teams to go to meetups. Again, I'm not sure about your companies, but we tend to look too much internally and not realize how much is going on externally. Or everybody wants to go to a class, send you to a four-day class, which is cool. Love going to classes. However, I learn a lot at meetups, so highly recommend it. Consistently exposing people to new ideas.

So just read a book called Scaling Up. It's not an Agile book at all, but it's very interesting to see that a lot of the same practices are in play. They talk about daily huddles, and they talk about timeboxing and multiple horizons of planning and line of sight to business objectives.

And then we're really spending time on ensuring that our employees feel like they are psychologically safe, which I don't think we did very much before. But if everybody's afraid to fail, then they won't actually extend themselves. And everything that we're building requires us to think two steps ahead. If you never fail, it pretty much means we're never extending ourselves.

Suzette Johnson

Right. So some of our lessons that we want to share from our journey to your journey is, one is understanding that our DevOps principles, as we've been telling this story, we're not just talking about the software, but how we're extending those principles across the value stream to touch other parts, such as software, firmware.

What does it look like to integrate early and often at a systems level?

Also, the importance of problem-solving. So our journey has been well over a decade, but we are still focused on working with our organizations and with our programs to solve problems, whatever that next problem is, because it seems like as soon as you get one part of it resolved and understood, there's another part that comes up that you need to start investigating and exploring because you're never done that learning process.

Have courage. Being able to speak up. I know not everybody's always willing to speak up, and I think that's one of the important things, that if you see something that isn't working or that could be improved, is to be able to have the courage to say something and then figure out together how we can make it better.

Robin Yeman

Yeah. I was just looking at things like security. So how do we build blockchain technology into the Internet of Things? Can we do that? And people are like, "Robin, is that your job?" And I said, "So if not me, then who?"

Suzette Johnson

Right.

Robin Yeman

So every time somebody says, "Hey, do you really need to do that?" You absolutely do. Because we don't even know who our competitors are going to be. They say by 2020, half of the companies that are in the S&P 500 won't be there. So I don't want us to get so complacent that we're not looking.

Suzette Johnson

Yeah, and I think it's real important too, that as you're exploring new ideas, sometimes people want us to look at it, "Well, we can't do that because..." And I always say, "Well, what if we just owned the world for a moment? How could we do it if we had full control over things, and what would that look like? And then are there pieces of that that we can actually go and implement?"

And I think also, because this has been pretty exciting over the years, getting more and more training, more programs, more coaching, and seeing this transition happen from the grassroots up has been real exciting. But the real important thing is we can't lose focus of what the goal is. And the goal is, are we delivering value to our customers in an affordable way?

So with every one of our initiatives, we need to keep that focus, as well as are these things actually making us better and delivering on the commitments that we've made?

Robin Yeman

And the last one definitely is to have fun. I love going to see the teams that are actually having a good time. And magically, they also always seem to be the ones that are delivering the most.

So, had one team down in Fort Worth, and they had played with Lego Mindstorms. Very cool. Lego Mindstorms. And somehow had set it up so that they could send Nerf rockets into the back of people's heads when the build failed.

Now you might ask what charge number they used for that, but they were having a really good time. It was just fun to be there. So I think the more we can try to build that into our business, the better.

Okay. So we'd welcome more discussions on how to build more talent. Again, I'm really passionate about trying to work with universities to build that next generation of engineers because I'm sure that what I learned 25 years ago was good, but I know there's much better now.

We want to build that lifelong learning in. Keep reading, keep experiencing, keep meeting with people. Work with your competitors. Work with your teammates. Work with people you haven't even learned about yet.

We want to keep growing the organization. Keep working. Again, I would love to be able to say, when all is said and done, we actually made an impact in the government space. I think commercial space is awesome, but wouldn't it be great if they could say, "Hey, you know what? You actually enabled us to get more capabilities to more people and didn't increase our tax rate." Wouldn't that be awesome?

And then continue to expand and mature our DevOps principles in software, systems, hardware, firmware, and again, we've got so many sensors, the Internet of Things. How do I build that security in there so I can make sure I'm delivering constant value that is secure and safe for the country?

Suzette Johnson

Yep. All right. So thank you for your time, and hopefully you have some takeaways. And if you would like to share your experiences with us, we would love to talk with you more over the next couple days. Thank you.

Robin Yeman

Thank you very much.