DevOps: New Methods Require New Skills
I'll talk about successes we've had in transforming towards DevOps and how we're bridging the gap between our existing skillset and the engineering skills needed for DevOps.
Chapters
Full transcript
The complete talk, organized by section.
Terri Potts
I'm Terri Potts, and I'm the technical director of the software organization in the Information, Intelligence, and Services portion of Raytheon Company.
So let me talk a little bit about Raytheon. You may have heard about it. We started way back in the early part of the 20th century, and we built some systems that you've probably heard of. In the early days, we had a branch called Amana that built the very first microwave ovens, discovered when an engineer who was working on microwaves walked by a microwave with a Hershey's bar in his pocket, noticed it melted, and decided maybe there would be some use for this technology other than what they were originally building it for.
We also do some other things that I'm sure you're familiar with, including a lot of missile systems. But the division that I work for, Information, Intelligence, and Services, we do a lot of things like satellite ground stations for things like weather systems that measure sea surface temperatures, the ice up in the polar regions, things like the aerosols in the air, all of that sort of thing.
We also build systems for our troops out in the field to make sure that they know where their troop mates are so that, in a firefight, they don't shoot each other, which is a really positive thing. That uses a little Google Glass kind of device on their helmet.
So that's great stuff. But we build a lot of these big systems. We've been around for a long time. A lot of our software is... well, almost all of our software is legacy. It's very controlled by our government customer.
And that actually is a big part of our story, because we build systems for our government customer. They very much control what our organizations look like and what our processes look like. We don't always get to define what our processes look like.
So of course, we're at a DevOps conference, so most of you know that DevOps is about getting the development and the ops organizations together. And sometimes we even talk about DevSecOps: getting the development organization, and the security organization, and the ops organizations all talking together.
But at Raytheon, this gets a lot more interesting because our organizations mirror our customer organizations, and our customer organizations are quite a bit more stovepiped than that. So they still have a systems engineering organization, a software organization, a platform or infrastructure organization, hardware organizations, and security. So they're way more stovepiped than most commercial houses.
So ours looks more like this: systems, security, hardware, infrastructure, software, integration, test, SecOps, CM Ops.
So this is a challenge for us, and so we've had to be even more creative than a lot of organizations in how we transition.
A couple of years ago, when I really started trying to move our organization in IIS toward DevOps, I knew that I wasn't going to be able to get to that nice DevOps construct and break down all those silos right away. So I started looking at how we can start building bridges until we can all be on one island together.
And so we start talking about that: how do we build these bridges? And I'm the tech director of that software organization, so I started by reaching across the aisle to my counterparts, the tech directors in my sister organizations in systems engineering and what we call infrastructure. Those are the folks in our organization that build up all of the environments and do all of the deployments out into... well, we often have many environments besides the environment that the developers work on and the integrators work on. Then we have multiple levels of tests that our customers want to see, and then eventually to ops.
So we have some programs that could have 20 different kinds of environments. Those infrastructure folks build all those environments, deploy to them, and keep them all up to date.
So I reached out to those tech directors, and we got together in person and built roadmaps that complemented each other and worked together, and made sure that we were moving forward together with our roadmaps so that, even though we weren't quite on the same island yet, we were communicating and we were talking together.
Now, on the flight out here, one of my colleagues sat with me. He's here at the conference. And he said, "Wow, that kind of looks like the bridge on my program." Because that's the other challenge: every program in Raytheon kind of gets to decide with their customer. So the Air Force customer is different than the Army customer, is different than the Marine Corps customer or NASA, NOAA. So every one of them is going along on their process transformation at a little different speed, right?
So his customer is in one particular place right now, and he said, "Yeah, the bridges that I'm trying to build look kind of like that, but we've got some landmines hanging off underneath in some of those places, and we've got some holes in the bridge in some other places, but we are trying to build that bridge."
And I would say that that happens, but just keep building that bridge and keep reinforcing it. And that can happen either at my level, when you're senior in the organization, but I would say that early in my career, when I was just a worker on the program, I was starting this. And you can do this at any point along the way.
I would say that the more places that you build bridges across to the other organizations, the stronger that that's going to be and the more it's going to work for you.
Another thing that we're doing in order to really make this transition is mentoring. In my case, I came back from a conference a couple of years ago, right after Gene released The Phoenix Project, and I pretty much twisted the arms of the software leadership team with all the department managers. I said, "Look, you guys, this is really important. This is coming. We have to get on board with DevOps, and I want you all to learn about it."
And they kind of moaned and groaned. But we said, "We're going to buy the books, and I'm going to run a study group, and we're going to study this book, The Phoenix Project."
And these guys, they've all been around for probably close to 30 years, and they're like, "A study group? We haven't done that in 20 years."
But we ran through the book, and they all participated, more or less. And some of them I'm really proud of. They actually turned around, bought the book for their direct reports, their group leads or section managers, and then ran a study group with those folks that are off on programs trying to do the same thing. So then in turn, they're trying to teach those folks what DevOps means and to spread the word there.
And we're doing some other things. It's books, and when I'm in the staff meeting, I'm always talking about the latest thing. So we'll be talking about The DevOps Handbook that's just come out. When Leading the Transformation came out, I talked about that. I've talked a lot about Jez Humble's books.
So all of these things, I'm always pushing it out to those guys. I'm pushing it out to all. I personally mentor about a dozen people in our organization. We do have a large organization, about 2,000 software engineers spread from California to Massachusetts.
So I mentor quite a few of those myself and really try to get them reading some things. We personally have conversations about how to make transformation on their programs, et cetera.
Another thing that I do in my organization is just trying to communicate the vision. We have sites in, is it nine, Rusty, locations from California to Massachusetts, and I try to hit all of those at least once a year. Talk about what's in the roadmap. Talk about not only DevOps technologies, but other key technologies for our software organization and what's important, what are priorities for us right now, and what we need to do.
And I do this in two forms: either in all-hands meetings, where we get together everybody in the software organization at that particular site, or I sit down with a group of maybe 10 or so of the leaders and we have lunch and we talk about things that are important to them or challenges that are important. And I try to pass on the vision of what we're trying to do in the organization.
I did talk quite a bit already about the study groups. I do think that this is important, especially these days. We had a conversation a little earlier today about taking time out, because Heather talked early this morning about how they have their DevOps Days in Target, and that's not quite as easy when you're talking about a government contractor, because we don't have much funding to get people together. Because pretty much every hour that people are in the office, they have to be on some government contract. They have to be charging some government contract, and that goes to taxpayers' dollars.
And so we don't have a whole lot of money. We don't get to just charge to send people to those sorts of things because we want to keep our rates low and keep how much we pass on to the taxpayer fairly low.
And so a lot of things like these study groups have to be done on our lunch hour or on our personal time. But this is important for our continuing education and to grow as professionals.
The next thing that I'd like to talk about is expert advice. I suspect you all know it's not easy these days to hire DevOps experts. They're not easy to find.
Again, a couple of years ago, when I started looking at trying to really grow this skill and capability in our organization, we looked around. Certainly, I would say that there was nobody in our organization that we could point to to say, "Hey, this is a DevOps expert." We had certainly people with skills in automated test or in Agile or Scaled Agile or some of those things. But we really decided we wanted to grow these kinds of skills.
And so what we did was we just went out and looked and said, you know what? We're going to hire somebody that is really bright, that has a really solid track record of learning, and being passionate, and wanting to grow in the organization.
And so we hired Brad, and we brought him into the organization, and I sort of pointed him in the right direction, gave him a lot of resources, and spent some time mentoring him and guiding him. And he just took off.
Because it's not that DevOps is rocket science. We do rocket science at Raytheon. And it's not rocket science.
So we were able to hire him, and he's done great. He's been with us for two years now, and he has really taken off. He started out by mapping a deployment architecture and saying, "What would this look like?"
And it was great because, at the time, our executive leadership thought that automation meant, hey, we just have to buy Chef, and then everything will be automated. And we don't have to really do anything. We just buy Chef, and then a month later, it'll take us 30 seconds to deploy everything.
And so he started out by drawing some diagrams, some really nice architecture diagrams that said, look, here are all the things along our big, long process that would need to get automated, and the kinds of tests that do that, and who in our organization, whether it's systems, software, integration, test, hardware, infrastructure, that is responsible for those things and what those look like.
And then we started having him go out and have engagements on programs, and he would do Six Sigma projects on these programs and start looking for what's your tough spot? What's causing the most pain right now? And he would help them figure out what was causing the most pain, and then work down that pain point, and get that part automated, and get that part a little easier to move past.
And then he started having engagements on proposals, because we do all of our work where we submit proposals against requests for proposals against our competitors, Lockheed and Boeing, et cetera.
And so he would go out and help them formulate the sections of the proposals that talk about DevOps, and how we would do DevOps, what that would look like, and that sort of thing.
And he also does some engagements on programs. Right now, he's working on the next-generation GPS satellite ground station, helping them set up their DevOps automation infrastructure and how all of that's working right now.
I would say that something that's really important, and we're growing more experts in this area as well, but I would say that something that's really important about those experts when you do try to hire them and bring them in and really try to round them out is that it's important to protect them from getting sucked into any one place.
They're really an organizational asset that you want to have available to all of the organization. So that's a challenge for us, and any time he goes and does an engagement on a program, we always have to have this sort of upfront conversation about, yes, we want him to come help you, but you have to know that it's for a limited time, and then he's going to come back into the greater organization so that he can help others in the future.
And we also set up a technology working group. This is a group that works across the organization. It's virtual, and we meet monthly by WebEx, essentially. And Brad runs this group, and he gets us together, and he usually has somebody present some lessons learned or some great thing that they're doing on their particular program.
And so this is just a way to get people sharing their experiences and things that they're doing with Chef or Puppet, or how they're moving toward automated test, or how are they making the changes that they need to make on their program to move toward DevOps or any of the other important technologies for Raytheon.
And we've created a repository where there are presentations and links to websites and great things like, one of my favorites is the XebiaLabs Periodic Table of DevOps that we reference all the time. There's so much great information out there. So we have a repository that we can point everybody to, and that's really helpful.
Internal investment. So there's some level of internal investment that we are able to do. Every other year, Raytheon holds an internal symposium. We've had an Agile track at that symposium for a long time, but this past year, it was an Agile DevOps track, so that was good, where we got to share lots of information with folks from across the company.
We also are doing some other things, like some small research and development, IRADs, to try to help set up some frameworks and infrastructure that programs can pick up and get using a little more easily. Perhaps things like Chef reuse libraries, Puppet reuse libraries, things like that. Occasionally, some white papers and various things.
A few years ago, we had what was called the Aspen Center. That was very much a concept like the Target Dojo concept that Heather talked about a little earlier today.
Finally, I think the other thing that is important at Raytheon, I don't know whether you guys have the same issue at your organization, but at Raytheon, we tend to be very insular. Perhaps it's the nature of working with classified programs.
When you do a lot of classified work, you tend not to talk about it outside of your organization. You tend not to talk to other people about it. So it turns out that often people stop looking outside the organization.
And so I've found that it's really important to get people to look up from their desks, from their programs, and look outside of the organization.
So this is a great start that you're here at DevOps Enterprise Summit, and I would encourage you to keep looking outside. There's lots of other good resources I've found in Denver. Certainly, the DevOps meetup, the Agile meetups are really great.
For me, building the community of both the getting to know the other speakers at the DevOps Enterprise Summit and working with the DevOps Enterprise Forum for the last couple of years, working on those papers, has really helped me reach out and get to know a lot of people in the community.
Heather and Scott Prugh and just all of those great folks that I feel very privileged to work with and get to know, and I feel like I can pick up the phone and call them and say, "How did you work through this problem?" And, "How did you get past this?" Or, "Wow, I really like those slides. Do you mind sending me a copy of those slides?" Or, "Where can I get a copy of those?"
And that's really helped, and it's just so easy when you're working with the same people over and over again to forget to get those creative ideas. So I'd say looking outside the organization is super important.
And that's my last slide. So I think I have two minutes for questions if there are any questions out there.
No? We're all good. All right. Thank you.