Host Intro (Gene Kim)

For years, I've mentioned that one of our goals within the DevOps Enterprise community is to have business leaders co-presenting with their technology leadership counterparts at this conference to validate that the work of this community matters to people who matter. As Maya Leibman from American Airlines said earlier this morning, this year has been a whole lot of 2020, but without doubt, one of the high points for me was meeting the next speaker, and I was so grateful when she told me she was willing to present here.

Kimberly Johnson is executive vice president and chief operating officer for Fannie Mae, which is in the Fortune 25 list of companies. Their mission includes making homeownership and affordable housing available for all Americans. For decades, she has earned the reputation of being asked to solve the toughest problems facing the organization, and I think the story of how she ended up being responsible for the technology function is profound, which is changing how technology's integrated into business strategy and operations.

I can't overstate how much I've learned from her in every interaction I've had with her, and I've asked her to help teach this community of what we need to know in order to help our organizations win. She'll be co-presenting with Chris Porter, SVP and chief information security officer, Tim Judge, VP of Climate Impact, and Ramon Richards, SVP of Integrated Technology Solutions. So here's Kimberly Johnson and team.

Kimberly H. Johnson

Welcome. I am really excited to join you today. Here at the DevOps Enterprise Summit, we're looking forward to providing some new insights, challenging ourselves to think differently, making some connections, and of course, sharing some stories about the journeys that we've all been on along the way. So I looked through the list of attendees, and there are some really amazing companies here. I'm sure as you looked through the list, you were wondering, why is Fannie Mae at a DevOps summit? And some of you were probably even asking, "Well, what is Fannie Mae anyway?"

Well, I'm going to spend a little time this morning talking about the role of Fannie Mae, how DevOps has helped us advance some solutions in the face of a rapidly changing environment, but we're also going to talk about the Fannie Mae version of DevOps, which is really DevSecOps, and how we incorporate security into everything we do. I'm going to pull in some of my Fannie Mae colleagues along the way to help tell these stories, and we're really going to drive home the theme about connecting business and technology. So first, what is Fannie Mae?

Well, we like to describe ourselves as a secondary mortgage market company, but those words don't mean a lot to most people. The gist of it is banks lend money to people so they can buy homes, and we buy those loans from those banks so that they can have more money to lend more people to buy more homes. It provides liquidity to the entire mortgage market. What we do with all those loans we buy, we package them up, we put them into securities, and we sell them into the capital markets. Now, this is great for homeowners. It means their houses can be more affordable. It makes it cheaper for them to borrow.

Instead of borrowing money at the rate that any person might be able to access for the risk that you might get for buying a home, we take thousands of borrowers, we package them together, we put a guarantee on those cash flows, and that gives a really reliable security that investors can buy. And they put billions and trillions into the market to help support housing for everyone in America. And everyone in America is a really important theme for us. We believe in helping all mortgages at all times. Fannie Mae, in its size and scale, is something to really wonder.

We have a $3.5 trillion balance sheet, and housing is up to 15%, sometimes 18%, of our annual GDP. So the bottom line is housing is important to the economy in America. Now, this is great in terms of scale when it comes to us delivering customers solutions, but it's tough when you think about change. Changing something that's so deeply embedded in the economy of the country means you have to be very careful. It's created a risk tolerance for us that doesn't necessarily help drive innovation, speed, adaptability, the things we're looking to get out of a fast-moving experience.

When we started looking at the way that customers interact with us and the experience that they get in getting a mortgage, we realized we really have to start evolving, and have to start evolving fast. Customer experience is really changing. They're expecting to be able to buy things from their phones with their fingertips in moments, and we still were working through a process that was full of paper and forms and signatures and faxes. By God, faxes.

So we've been spending the last part of our digital transformation trying to digitize the front end, but also trying to digitize the entire process, making things faster, easier, simpler for the lenders and for the investors who are part of our mortgage ecosystem. So before I launch into our DevOps story, I want to tell you a little bit about myself and my journey at Fannie Mae.

So I had Gene Kim earlier this year come and talk to my entire company about DevOps, and as he began to learn more about Fannie Mae and how we've come to value sort of speed, agility, locality, simplicity, flow, the things that are really core, he asked me a really important question. He said, "Kimberly, I love that you're involved with this technology, but you've told me your story. You've been here at Fannie Mae for 14 years. You've been in capital markets, and multifamily, and single family, and risk, and why are you the COO? I don't understand how you're leading the technology team."

And I have to say, I took that as a compliment to, like, the "what makes you qualified to do this job?" question. That can come off a little prickly, but Gene meant it in the nicest of ways, and I had a really ready answer. For us, it's not about having the best ability to code or knowing how to provision servers, the leadership of our technology team is about bringing together business and tech. We recognize at this moment that it's not going to be about us and them. There's only going to be one business in the future, and it will all be powered through technology.

Fusing those things together is going to be what makes us compete and win in the future. I have to say, when I started in my role, we had a long history between the business and the technology teams. The trust was pretty low, and I would say the empathy was pretty low, too. We had a history of taking too long, costing too much, and not quite getting the mark when it came to delivery. And so there was blame on both sides of that equation, but it wasn't a harmonious co-location of people working together towards the same goals.

Number one thing we did when we started off on our DevOps journey was make sure that we were actually connecting our customers, our business people and our technologists, so that we could all be going after the same thing. I brought my entire management team along with me. We really went all the way to the experts. I had people like George Westerman and Jeanne Ross from MIT coming in, talking to our board of directors about what it means to be digital and why. We had them doing classes with us at MIT, talking about how to bring digital into the world and the difference between digitizing and a digital transformation.

And lastly, of course, we had the illustrious Gene Kim. We brought him in to talk to our whole company, not just our tech teams. We wanted everyone to understand what it meant to be on a DevOps journey. And believe it or not, it was riveting. He was explaining to people of all levels and all businesses why we needed to tackle technical debt, and they were just rapt, listening, understanding and getting to the point. So this has been something for us that has been not just a technology journey, but a company-wide journey, and that's why it's starting to stick.

So we've now aligned our DevOps journey and our cloud journey together, and as we're putting things in cloud, we're shifting our teams with more DevOps tools and practices so that we can actually become that future that we had that vision of when we started this a year ago. Another funny thing that you want to focus on, once you get your alignment, you know where you're going, and you get all your foundational pieces right, and you have everything set, you get everyone on board, and the first question becomes, "Okay, great. Is it working? How about now? Is it working? Is it working yet? Are we there yet? Are we done?"

I found that as soon as we got ourselves launched, we got funded, we got moving, we got people actually aligned, hands on keyboards coding. We got our CI/CD pipeline in place. We put our microservices architecture. We had our streaming data platform. We were off to the races. And not three months later, from the top of the house, from the board to the management committee, everybody said, "How about now? Is it working? How do you know it's working? Where's it going to show up? How will we know?" And that is a really good question. How will we know? For us, it's about metrics. And metrics have been a journey, too.

We started off trying to measure a handful of things, and what you find is you get what you measure, but that means you have to set your measurements right. And that's been a continual evolution, a process of continuous improvement on measuring the right things to get the right results. And so I would say that you've got to put in a lot of elbow grease to get the right metrics. For us, it has been a learning experience. We focus on things like lead time and mean time to restore and even on the software failure deployment rate, some of the key DevOps metrics. And those have been so volatile.

You go and you change something and you get it just right, but it doesn't change the behemoth of the company and the way you measure everything. It only changes in one place. So how do you get your metrics to start moving in the right direction and people can see the progress that you're making? So we decided, oh, well, we'll split out the metrics, the old world and the new world, and we'll compare, and that'll help people understand the value of what we're doing with DevOps when they can see the difference between what we measure in the new world and the old.

And then I'd say, lastly, we had to do a lot of work breaking down our metrics into things that people could understand. We said, "You know what? Every metric isn't the same." There's metrics that measure inputs. Are we doing the right things? Do we have the right guideposts? Are we following the right practices? We had measures around execution. Are we living the plan? Are we doing what we said we're going to do? And then we had measures around outcomes. Is it working? And we helped everybody understand. You can measure your input metrics really quickly.

You can measure your execution metrics along the way, but those output metrics take a while to move. And that started to help, and we stopped having to answer, "Are we there yet? Are we there yet? Are we there yet?" All right. I guess the last thing I would say is just about communication. Oh my goodness, everybody says it takes 10 times. 10 times is a very low underestimate of how long it takes for things to really sink in, especially when you're talking about change of this magnitude. For us, we anchored on agility. We want to be able to do that quick responsiveness.

We don't want to fall into the same traps we did before, around not being able to integrate new risk management techniques. We want to be able to move fast, and agility has been the underpinning for us. And it's been a year of telling everybody every time anybody asks, "Why are we doing this again?" It's for the agility. We found that if we focus on both agility and efficiency, you can get to some of those results that your CFOs really like. For us, it's focusing on retirements, turning things off. Fastest way to save money, turning things off. Fastest way to spend money, build everything twice.

So there's a really important trade-off about creating the new, turning off the old. And very last, I would say scale. We spent a year getting our DevOps journey right, and we can finally see some of these things coming together. But you find that getting it right in one place is a wonderful victory. But getting it right everywhere is a hugely daunting task.

And so we're on the journey too of figuring out how to take those small nuggets of really great productivity that we've developed by putting our DevOps pipelines in place and having our teams smaller and autonomously designed, so that they can actually deliver on a more continuous basis, has been terrific for morale, terrific for productivity, terrific for results. But getting that from one or two teams to hundreds and hundreds of teams, now that's the question that we really want to answer. And I'm excited to have some of my friends tell you about how we're doing it.

Ramon Richards

So with that, I'm going to bring on Ramon Richards. He heads up our development teams. He leads our integrated technology solutions group. Ramon, my first question for you, can you please share how that COVID-19 reaction and everything we had to do to bring forbearance to borrowers has translated into technology and operational challenges for us, and how DevOps helped us address those challenges? So the impact of the pandemic required Fannie Mae to quickly figure out, how do we deliver new solutions for our customers in a matter of weeks to ensure we provided the help they needed, before they were adversely impacted by the crisis?

And so, while we're used to delivering for customers, the speed in which we had to turn around to respond to this crisis was new for us. But our confidence was high. And our confidence was high because we had made an investment in DevOps in our servicing area. And we were confident we could deliver the solution, but there were a number of challenges we had to immediately respond to. The first challenge was finding the capacity to take on this body of work. We had a number of high-priority items we were already focused on. So we were able to identify the teams that we would reallocate to focus on this high-priority work.

One of the challenges in the past when you reallocate teams is that you have development work that they already have in flight that you have to put on hold. But because of our commitment to the DevOps practices, we have been delivering production releases on a regular basis. We have a continuous delivery process. So it minimized the work that we had to put on hold, and it allowed us to reallocate teams but still deliver value to our business partners. The second challenge was we were still maturing along the DevOps curve. So we hadn't fully arrived yet. But we knew we needed these teams to quickly be high-performing.

So what we did was we assessed the talent that we had on the teams. We were able to swap in some more seasoned DevOps experts where we needed to, and we also partnered our teams with some coaches who were very seasoned in the DevOps space as well. That helped us accelerate our maturity in a short period of time and positioned us to deliver the capabilities we needed in a matter of weeks.

Kimberly H. Johnson

Ramon, thank you for that. That makes a lot of sense. Does DevOps help us more broadly in servicing loans beyond just what we're dealing with in COVID-19?

Ramon Richards

So there are certainly benefits beyond just the crisis that we were responding to. So the adoption of DevOps has allowed us to think and operate differently with all of our partners within the servicing space. So one of the important concepts is integrating our product management teams, our development resources, our site reliability engineers, as well as our partners like architecture and information security, where we're all at the table together, which reduces handoffs, reduces communication, and allows us to make decisions in a timely manner. It's changed the way we think about our testing.

We shift a lot of our testing earlier in the process. We drive automation as we are developing our solution, which speeds up our ability to deliver a quality product. And then we have a continuous build process. So on a daily basis, we are building the software that we are creating to ensure that it is high quality, and that we are learning as we go, and we're adapting pretty quickly. So a lot of those benefits played out as we executed to turn around a solution quickly, but it's also benefiting other important work that we're doing in the servicing space and really driving us to be more agile in delivering with a faster time to market.

Excellent. Well, fast cycles and quick learning loops. That sounds really good. Ramon, if you had to summarize the big takeaways for us when we think about COVID and DevOps, what would you say would be the things you want people to know? I think one of the things that's really important is that the team that is focused on your delivery and are important stakeholders in how you get the job done, that they all are operating with the same mindset. And they all understand the principles and the processes that are part of the DevOps process. That it is about collaboration, early engagement in the process.

It's about automating everything from start to finish. It is about timely decision-making. And so, we have found through the experiences we've had, how effective DevOps has been in allowing us to deliver. And the next challenge for us now is to continue to scale these practices, continuing to build that understanding, the mindset, the understanding of how you leverage the tools across the organization. Thank you, Ramon. That's terrific. All right, so moving into our next

Kimberly H. Johnson

segment. We just finished up how Fannie Mae's used DevOps to be able to have more agility in managing some near-term things that have emerged, like COVID. But we've also been looking at things that emerge over a longer time horizon. You remember Fannie Mae makes loans, 30-year loans. All of our loans are backed by collateral: houses. And so as we see more weather events and climate change, there's a lot more risk of things like wildfires, hurricanes, things that damage homeowners, things that have risk for loans that last 30 years that are backed by properties. So with that, I'm really excited today to introduce you to Tim Judge.

He's our top officer for climate risk, and we're going to spend a little bit of time today talking about how DevOps has also helped us with some of our longer range goals. So Tim, it seems like we see these types of new weather events happening more and more frequently. Now, what's Fannie Mae doing to stand up capabilities to be able to handle this, and how can we respond?

Tim Judge

So, the first thing I would say is the intensity and frequency of severe weather events are certainly increasing. And so when we look at the program, we knew we had to stand up a more holistic program around climate. Now, the first challenge there is very similar to digital transformation is, how do you address day-to-day severe weather events while still building a longer term program? At Fannie Mae, I was very lucky to have already a disaster response recovery team that Fannie Mae had already stood up that was able to take care of things on the ground when natural disasters occurred and help communities.

So that really allowed me to focus more on that medium term and longer term build-out. And then when you look at the longer term build-out, it's really about breaking that problem down. So when we took over the climate space, the first thing we did is say, "Where do we want to focus our time and our energy first to bring value?" And we found, as you break down the portfolio, we have, like we said, $3.5 trillion of balance sheet, and that's about 18 million loans, right? So you can't tackle that all at once. So we broke it down, and we said, what part of the portfolio is most at risk, and what risks are we really going to focus on first?

For instance, are we going to focus on flood versus wildfire versus hail and severe storms? And then the other important thing is we brought that plan to the board and made sure the board understood what was our short-term delivery, where they would see quick value out of the climate program, but then also make sure they understood the longer term transformation journey that we're going to have to be on. Because I think that was a really important part of the socialization, is giving people quick value, but also making sure that they understand that it's going to be a significant journey.

Kimberly H. Johnson

Awesome. Thank you, Tim. That makes a lot of sense. And Fannie Mae's been at the heart of this digital transformation. We've been talking about this as we go through this conversation. How has that digital transformation, and DevOps specifically, helped you guys as you've been building out this team that's facing all this really interesting and evolving climate risk?

Tim Judge

Yeah, so I'm a little bit different than most business folks in that I came recently from the COO, so DevOps is kind of in my blood about me wanting to be a sponsor for that. But I would say a couple things about DevOps is, for me, tackling, like I said, a huge climate issue, one of the things that really helps me is the speed and agility to quickly experiment. The climate world is changing so fast, the modeling of these climate events is really evolving very quickly. So I have to have an environment that I can quickly churn and quickly look at different models, and DevOps gives me that.

The second part of it is there's a big regulatory part of climate, and the expectations from regulators and investors on disclosure is only heightening. That gets to the risk side, right? I want to be able to give really good disclosures to the industry, which means I need to deliver high quality metrics on a regular basis. And then the other thing I think we've talked about a little bit before, also on DevOps, is scale. Likely for the climate, I'm running those 18 million loans. I'm not running them once and looking at them. I'm going to have to look at 18 million loans and what climate does to them over 30 years.

So you can imagine the scale that that has today in terms of data footprint and in terms of compute. All those things aren't possible if we were still living in the old world of the basic infrastructure that we used to have. DevOps gives us that ability to be really nimble. The other thing I think it really does right now, now is almost a perfect time for DevOps because with COVID, a lot of the industry is really challenged on budgets. We talked about the CFO wanting to see something. Well, with DevOps, you get to walk into the CFO and say, "Hey, here's some quick value that I can deliver.

Here's a lower risk program, so you're putting your money in a lower risk area." And then finally, it's more responsive. So I don't have to tell you, "Here's my three-year plan," and stick to it. What I'm really telling you is, "Here's my plan every single quarter, and you can know that I can respond as needed."

Kimberly H. Johnson

Excellent. Thanks, Tim. That was really insightful. So we've talked a little bit about COVID-19, which was sort of a near-term, fast-moving crisis. We talked a little bit about climate change and how we're tackling a sort of longer term, more drawn-out crisis. And as I think about how we're putting all this together, it reminds me that the one thing that underpins it all is security. For Fannie Mae, safety and soundness is part of our mission, and we're a company that has gone through a crisis before. I think I mentioned at the beginning, we have a $3+ trillion dollar balance sheet. We help finance more than one in four homes in America.

We have a really important mission that we just can't put in jeopardy. So our risk tolerance has been somewhat low over the years, and that makes all of this change really, really challenging, and it makes security very, very important. So with that, I'm asking our CISO, Chris Porter, to join us, and we're going to ask him some questions about how we're managing through this evolution, this transformation that we're making, and how we're incorporating DevOps into everything we do in a safe and secure way. So Chris, it sounds like infrastructure is a really important part of security.

Can you tell us a little bit more about how security plays into the role of test-and-learn in this environment?

Christopher Porter

Yeah, certainly. I think about this in a couple of ways. One is about culture and changing the way that security communicates with our development teams, and the other is about how we integrate security tools. When it comes to culture, we've had to change quite a bit, ourselves. The old way of security is that we did our own tests. The app was handed off to the security team. We ran our own tools against the apps. We collected the results, the vulnerabilities, and then we handed these back to the dev teams, and this made for a completely inefficient process. Nobody liked it.

By the time the code was sent to us, it was ready for production, and now they were waiting on us to do our tests. And we would take our sweet time, further delaying the delivery, and then we would hand them a giant spreadsheet of vulnerabilities and said, "Good luck. We need you to fix all these things." So we had to figure out how to move left in this process. How do we shift left in the development process? And so we did this by relinquishing the control over our precious security tools, making them much more self-service, API-based, integrating them into Jira and Jenkins and allowing developers to run these themselves.

So it took a lot of time. We had to train developers on how to use and run the tools. We had to teach them what the results meant. We also had to change our nomenclature. We stopped talking about vulnerabilities. We started calling them defects, because ultimately, security vulnerabilities are just another kind of code defect. And then we need to continuously get better and fully integrate all of our tool sets within the CI/CD pipeline. We needed to integrate all the security tools like static analysis and dynamic analysis, configuration management checks, compliance checks.

All needed to be in the pipeline so that every time that code was checked in, we were running a test. Every time that we were doing a delivery, we were running a test. And this makes it easier for developers to know what they need to do, right? You run a test, it fails. "Oh, I have to fix something." I call this the paved road. If you follow the paved road and you use the CI/CD pipeline, which has all the checks and integrated into that pipeline, then it's going to be much easier for you to deploy code. And we treat it like an Andon line, right? If the test doesn't pass, it breaks the line.

It has to go get fixed, and then delivery can continue once it's fixed. So if you follow the pipeline. If you don't follow the pipeline, then it's the rocky road. It's slow. There are lots of potholes. You get pulled over by the locals. No one wants to go that route. And it'll take you much longer time to deliver value to the customer. And a little note to Gene Kim and the Unicorn Project, it takes you longer to get joy to the developer as well.

Kimberly H. Johnson

Excellent. Well, I think that's terrific, the idea of automating everything, especially those tests. That's so important. And you gave us some really good tips there. I got one final question for you. It's about mindset. From the CISO perspective, what mindset do you think developers need in order for all this to come together successfully?

Christopher Porter

I actually think there's a mindset change that has to be made by both the development teams and the security teams. And I saw this tweet recently that I think really drives this home. It's from Richard Seiersen, who I think is with a company called Soluble. And he said that DevSecOps is an observation, not a title. And it's we don't care who does the work or where it sits in the organization, it's just how security is done in modern organizations. So on the security side, there's always been this mentality that you must protect the developers from themselves. But when you move to a DevOps model, you move to the you build it, you own it.

Those full stack developers operating there, well, you know what else is part of the full stack? Security. Security is part of the stack. So there's got to be this kind of shared accountability for security between the DevOps teams and the security teams. And then on the security side, we have to do our part. We've got to deputize those DevOps folks. We've got to help them. We've got to train them. We've got to build programs like security champions, where on every single DevOps team, there's somebody, at least one person, that's accountable for making sure that security is implemented. On the security side, we can't scale, right?

There's a supply-demand issue in our industry for security folks. The demand is far outstripping the supply. So we've got to do our best to bake it in and then train the DevOps teams on how to own it and implement it the way it needs to be done to appropriately manage risk.

Kimberly H. Johnson

Awesome. Thank you for that, Chris. That was really terrific. So today, we had a full conversation around how DevOps can help us respond quickly to the evolving world around us. Here at Fannie Mae, we are preparing for today, for tomorrow, for 20, 50 years from now. We're rebuilding our infrastructure in a brand-new way to meet all the challenges that we see ahead. And again, we looked at some fast-moving things like COVID, some slower-moving things like climate, some really essential foundational things like security, and we shared a few of the stories with you. So our big learning is that DevOps is about how we do everything.

Business, technology, completely intertwined. It's how we get it done. And the lesson that we've learned isn't so much like you've got to design the way that you do things the right way. It's around the change management that's required to get everybody to adopt the new way of doing things. And for us, change management's been around communication, making sure that you have alignment of vision, around metrics, making sure that you're executing as you expect. It's been around scale, getting it right once, and then being able to scale it up, and around continuous learning so that you can get better and better.

Nobody gets this right the first time. So those are the lessons we've learned on our DevOps journey, and I hope you've enjoyed what we've shared with you today, and we look forward to many more DevOps conversations in the future.