Host Intro (Gene Kim)

If you had told me 10 years ago that the next speaker would be sharing publicly her experience report of elevating developer productivity in her organization, I would have laughed at you, because the business of this organization is quite literally secrecy and security.

So Virginia Laurenzano has been a technical director at the National Security Agency for five years, heading up the DevX initiative, which she created in 2018. She is a cryptomathematician by training, but gravitated towards solving the problems that plague developers in their daily work, preventing them from achieving their mission. She will tell the amazing story of how they're changing the way software is created and delivered throughout the agency.

It turns out that Virginia and I were sitting in the same room at the 2017 DevOps Enterprise Summit, marveling at the work of Kurt Hockade at the Joint Warfare Analysis Center, who works very closely with the intelligence community. So I am so delighted that Virginia will be sharing her amazing achievements, and that she was recently detailed to US Cyber Command, MARFORCYBER, our Marine Corps Cyber Force. This is part of a rotation program that candidates for advancements go through. So here's Virginia.

Virginia Laurenzano

I'm Virginia Laurenzano, and I'll be talking to you about improving the developer experience at the National Security Agency, also known as DevX.

I so wish I could be with you in person. I imagine most of you don't envision someone like me, small, nerdy, feminine, when you imagine an NSA employee. Hopefully, we'll be together next year.

To set the stage for who DevX serves, I'd like to share some basics about NSA. Yes, security is our middle name. And while there are a lot of details about our work I cannot share, there are plenty I can.

NSA is responsible for providing both signals intelligence, or SIGINT, to United States policymakers and military forces. We are also charged with protecting the networks and sensitive information on national security systems, known as cybersecurity.

Behind these two missions are developers, systems and network administrators, and other technical folks. Founded in 1952, NSA is both a member of the US Intelligence Community, IC, and the Department of Defense. With over 40,000 affiliates, we're the largest member of the IC. And based on our budget and size, if we could give you all the numbers, we would be in the top 10% of Fortune 500 companies. And we're the largest employer of mathematicians in the United States, like myself.

So, who am I? Well, personally, I try to honor Gandhi's advice and be the change I wish to see in the world. And I'm never satisfied with the answer, we've always done it this way. Professionally, I'm a career civil servant with over 20 years at the NSA. Given the size and scope of our agency, it's been fairly easy for me to move around and try different jobs.

I started out in NSA's oldest training program, the Cryptologic Mathematician Program, choosing to stay in cryptanalysis after my three-year rotational period was up, focusing on the automation of algorithms. I might have stayed focused on automation, but a beloved mentor boss pushed me into technical leadership, saying, "You can do things that others can't. You can explain technical concepts to leadership. We have enough programmers. Go do that." And I did.

As a cryptanalysis technical team lead, I revolutionized how 150 employees perform their daily work in support of NSA's SIGINT mission. Then I joined the Analysis Fellows Program, one of our mid-career rotational programs, to learn what life is like for an NSA intelligence analyst. After graduating, I set out to become a technical director.

Throughout my early career, I found myself and others desperately wanting to use modern software tools and stuck instead with Make and putting the versions into the names of the programs. It was bad.

Five years ago, I became the technical director for the Processing Solutions and Services Division. And that group was tasked to make the dream of modern software development tools a reality for some NSA analyst developers. Currently, I'm detailed to US Cyber Command, Marine Corps Cyber Force, as my last professional requirement before seeking a Defense Intelligence Senior Leadership position.

What follows is about my five years as a technical director, founding and scaling DevX, and some of the bureaucracy hacks we leveraged along the way.

All right. I've got my dream job, but I don't know anything about agile practices or containers. How am I supposed to technically advise teams working on this stuff? I'm a math dork whose skills include translating conversations from manager to intelligence analyst to cryptomathematician.

So I dove in. I became versed in agile practices, and I confirmed my teams were agile. And it wasn't enough. We were still failing to deliver meaningful solutions to our customers.

Okay, so I started looking for more. My boss suggested reading The Phoenix Project, and I haven't looked back. DevOps practitioners taught me the language to describe the problems I was seeing in my teams and had me wanting to have that at NSA.

Others were coming to this conclusion, too. NSA formed an internal, large government working group to admire the problem. Now, during this time, I also found myself asking, "Don't they know this needs to be fixed? Don't they know this isn't working?" And slowly realizing there might not be a they.

While no actions came of it, I did get to know other champions in the software development space. And at the same time, I brought my new DevOps admiration to work and started a technical exchange meeting, inviting the over 150 practitioners in my office to come to the table and to talk.

One outspoken contractor took me at my word when I said, "If I suggest something that you think is crazy, please tell me." And he did. "Virginia, you want to adopt these practices, but we do not have the tools."

All right. I'm a technical director. I can be a detective. Was it really that bad, or was he exaggerating? Yeah, he was right. It wasn't good.

Development teams were rolling their own software development ecosystems, spending considerable time and money on stuff that wasn't their mission product. What enterprise-accessible tools that there were lacked both the resources, people, money, hardware to support their user base.

Okay. And the teams that were supporting either just the tool for their team or for the enterprise were making custom solutions. In the case of FOSS tools, they were changing the source code, and then they weren't pushing those changes into any internal repository or to the outside, which meant with every new version, the changes had to be made again.

The teams that were supporting COTS tools were making custom one-off integrators to make the COTS tool do whatever it is we have to do at NSA: RBAC, ABAC, portion marking. And again, they weren't sharing it with other people, and they weren't talking to the COTS companies to say, "Hey, could you make some changes?" So yeah, this was not sustainable.

Okay. The world is messy, and I cannot fix NSA's complicated, disconnected-from-the-Internet system. But maybe I can help my developers. Yeah, I can help a few more developers.

Using those DevOps technical exchange meetings, my growing network of software development advocates, and interviews with developers, I learned some of the things developers want and also need.

What are those things? Well, they want modern development tools so they can focus on SIGINT problems or cybersecurity problems. They want reliable collaboration work management tools. They want a centralized place to share information with each other on best practices.

That all seems reasonable. Why is it so hard? Well, again, NSA. We mainly work disconnected from the Internet, and that makes things complicated. Classified development needs proper marking and access controls. We actually write software that is protected by certain caveats. And then there's cultural barriers. Many teams have this attitude of, "Well, I can't trust another team to support a tool for me."

All right. Enter a bureaucracy hack. We're big. We have funds. We have a mandate to work with industry. Let's use this bureaucracy to our advantage. We worked within NSA channels to form and foster a public-private partnership, bringing those classification marking abilities or project labels, role-based access controls, RBAC, and attribute-based access controls, ABAC, into the core feature set of a COTS product.

Had to remind ourselves, can't lose hope. These things take time. We're talking government contracts. We're talking adoption of big software changes. It took three years to go from good idea to people banging on the keyboard could actually use it. But it was worth it. Even today, my teammates and I still get thank you notes, both on the computer and handwritten ones, for making this happen. And those thank you notes drive our desire for change.

Bureaucracy hack number two. DevX members weren't standing still while we were waiting for these software changes. Not that we existed yet, but we weren't standing still.

So, thanks to my DevOps learnings, I recognized that my organization needed to change a failure-averse culture to make good use of the tools that were coming down the pipe. But how? Well, as a civil servant with job security, I encourage folks to model failure and getting back up again.

I personally did it publicly, trying to scale postmortems within my larger organization, sharing my reading list, fueling heated strategy discussion meetings. More than once, I earned negative attention: "You don't understand" lectures, indifference to recommended books, and the admiration of junior colleagues who still to this day talk about some of the books that we would share later with you on our internal social media platforms. What I kept learning was that failure and getting up again, the perception of that activity is in the eye of the beholder.

Our next bureaucracy hack. By this time, my network of software development advocates had connected me with this guy named Jacob, and together we pitched and founded DevX.

Our DevX pitches were variations on learning from mistakes and reading the room. The initial pitch focused on duplicative efforts and was unintentionally shared with the leaders of those efforts, and it was seen as a condemnation of them and their teams. It did not go over well. And instead of giving up, we went back to the drawing board.

We landed on capturing key metrics from FAANG to show senior leaders how comparably large organizations save money and exponentially benefit from centralized support of foundational software development tools. Calculated productivity acceleration is possible via centralized tools. We showed how our part of NSA, including the teams named in the initial pitch, could benefit from centralized tools. And that worked. We had laid the groundwork for DevX.

Now, for those of you wondering, because I haven't given any timelines, DevX went from idea to money to people in less than 18 months. By government standards, that's nearly instant. And Jacob had coined our motto that I alluded to earlier: "There is no they. We are they."

So what exactly were we trying to do with DevX? Unofficially, make it suck less to develop at NSA. Officially, we set out to improve the developer experience by delivering tools and services to increase productivity. We set lofty goals, like DevX speaks for the developers. We elevate their needs. Long term, DevX wants to help senior leadership understand that NSA is a software agency focused on both SIGINT and cybersecurity.

How did we start off that lofty climb? Well, we looked to industry leaders like Amazon, Microsoft, Target, and we attended conferences like this one. We also read or listened to books and podcasts and talks. We immersed ourselves like we were starting a startup.

And this big quote here is from one of our systems engineers, and I'd like to read it to you. "I have never before worked with civilians who did the type of external research DevX members did. Plugging into the latest technology information, reaching out and working with external companies, attending and actively participating in conferences, reading technology and business books, and bringing that all back inside, making changes happen, not accepting the status quo or the fact that a policy is in the way, evangelizing and making it part of the culture of your teams while building it into the larger fabric of the enterprise."

What we were doing at the start of DevX was exciting, but it didn't feel that crazy. But it looks revolutionary now that I captured that note. And just like thank-you notes for that COTS product a few slides ago, we keep quotes like this around to remind ourselves, both as a program and civil servants, how far we've come.

All right, on to the work of being DevX. We have people, we have money, we have OKRs and a vision. Many of those brownfield, enterprise-accessible, formerly under-resourced tools I mentioned earlier. So what next?

We set an early goal to cultivate relationships with users so we can effectively speak for them. And we identified a particular relationship in need of repair. We made the connections, and we set up a ride-along for a peek into the day in the life. Wasn't meant to be. Day of the meeting, a DevX tool critical to those users' daily work degraded. Classic.

So instead of the ride-along, we got to repair our relationship during a period of service degradation. Our secret? Talk. Make human-to-human connections, both with your teams and your customers. Share laughs, maybe a donut. Talk some more. Coordinate messaging. Present a unified front.

Instead of standing back, DevX managers sat alongside the support teams, fielding questions, leaving the engineers time to try, test, refine, try again. DevX leaders wrote and refined communications plans, practiced talking to customers, coordinating messages, telling the truth clearly for various levels of leadership. We modeled the communications patterns that our teams would adopt.

DevX engineers used this time to build our monitoring muscles so that we could see and react to failure signals earlier next time. Maybe so early that no one else would know about it. Now, as you see from the title of this slide, we were rewarded. NSA's four-star general praised our communication skills during this event. We earned challenge coins, which in the DoD are a way of rewarding collaborators and marking participation in events.

On to our next hack. Don't wait for permission when there is no policy, law, or governance forbidding an action. Civil service necessitates action. Ask yourself, how can we change this existing process to suit our mandate?

As an example, NSA has an evaluation process for some software purchases. DevX added rigor for the tools in our purview, requiring defensible evidence. We did not ask if this change was acceptable.

Now, the expectation that people coming into this process have is, "Well, I want the tool. I have money, so you're going to say yes." In reality, when we asked for proof of need, it was difficult to accept.

And some of our senior leaders came to us and said, nicely, but basically, "Who gave you the right to make teams justify their acquisition requests?" And we pointed to the DevX founding pitch. NSA literally gave us money and resources to reduce duplication of effort in this space. Scrutinizing requests is one of the ways we're reducing potential duplication.

It got to the point where one particular team went to senior leadership demanding DevX fall in line and approve their request. Instead, our CIO approved of DevX's processes, and that team quietly adopted the centralized tools. For that one team, the savings included license costs, compute costs, and the salaries for up to four engineers, all because we asked, "What is it you need to do or need to protect that our tools don't do or don't protect?"

So just exactly how did DevX enable different software teams requiring controlled access to their software projects to share the same space? By supporting RBAC, ABAC, and project labels across our suite of CI/CD, source code and binary repositories, collaboration, and workflow management tools.

How did we make this process sustainable? Well, first, DevX teams intentionally do not change local copies of FOSS products. Instead, we either write modules that sit outside of the products that we use, or we leverage public-private partnerships to have companies support those features and commercial offerings for us.

In other words, we don't change the inner workings of FOSS tools anymore, and we make modular changes to COTS tools that we can share with others. As a result, DevX is able to provide COTS and FOSS tools and ensure that only those who should access an artifact can access that artifact. And teams are able to store classified, need-to-know artifacts in shared tools, leaving teams able to focus on their mission-enabling software.

A few weeks ago, Gene asked if I could share the kinds of projects that teams are making using our tools, and I can't. He also asked if I could share any numbers, and I can. Basically, though DevX is small, we are mighty.

We have under 100 people supporting a whole suite of software development ecosystem tools, and we've enabled 250 builds per day, about 3,000 active ChatOps users per day, 4,000 and growing active CI/CD users per day, 20,000 active workflow collaboration users per day, over 100,000 software projects, and more than five million software builds. To all of my team members out there, I am very proud of what we've been able to accomplish.

Now, no DOES talk would be complete without a discussion of challenges. For DevX and larger NSA, blameless postmortems are both a challenge and a success. We hear a lot of reasons why we probably shouldn't do them at NSA. Internally, we've done them for our products to great praise from users.

Some of the challenges we've encountered are comments like, "Everything I do is close-hold." So we ask that those people focus on the why and the how, but not the what. "Everything my team does is unique." You're not the only person using that technology. Talk about your technology stack. We have a culture of hiding imperfections. I personally remind participants at every event, "It's okay. We're all human. We're not supposed to be perfect." I've seen participants lay blame during and after meetings. We continue to model blameless and show how word choice can greatly affect the perception of what you're saying. And unfortunately, right now, it's a personality-driven practice. So one of my career goals is to normalize blameless postmortems at the NSA.

Some of our recent successes include identifying and correcting a bug in one of my division's most used tools, getting teams outside of DevX to participate in large cross-organizational postmortems, reviewing communications surrounding events, not just on the technical components, because how you share information with your technical workforce can be even more important than what you had to change. And finally, identifying unadvertised policies impacting all software developers at NSA. As a result, informed developers are empowered to contribute to the next generation of solutions and policies.

Another challenge where we could use your help. Many middle managers still view COTS and FOSS tool support as a sustain activity. We haven't yet helped them understand the need for creativity and experimentation in the space. So we're interested to hear what metrics you've captured for your various levels of leadership to help paint that picture.

How did you help your management understand things like the tools their employees need to do their job, that software powers their mission, and that an evolving software development ecosystem is critical to their overall success?

As I mentioned earlier, DevX team members have read a lot, and I wanted to share with you our reading list. Having professionally and personally benefited from seeing the resources that others used, we wanted to share ours with you. To all the authors and contributors of these books, thank you for sharing your insights. You are helping DevX members grow as leaders and helping us improve the developer experience at NSA.

And finally, DevX members will be here on this channel for questions and can be reached after the talk through mediarelations@nsa.gov. Thank you for coming.