Log in to watch

Log in or create a free account to watch this video.

Log in
Las Vegas 2022
Share

Driving Continuous Improvement of Northwestern Mutual’s SDLC

Northwestern Mutual has done many transformations previously, but needed to accelerate the business through a large scale enterprise change program. Northwestern Mutual embarked on a holistic program to address the process and tools that are used by our software delivery teams. Through this journey, Northwestern Mutual used a unique approach (using JIRA) to disseminate information and tasks to our over 350+ teams. Using the principles of DORA, Northwestern Mutual is driving incremental change in the way we develop, test, release, and operate software.

Chapters

Full transcript

The complete talk, organized by section.

Jason Cowdy

All right. Thanks, everybody, for coming in and having a seat. My name is Jason Cowdy. I am a principal engineer over with Northwestern Mutual. I am on our Enterprise Shared Services team. Today we are going to talk a little bit about our SDLC journey that our company has been going through over the last year.

No. Try again. Better. All right. Technical difficulties for a second. I do not know if you can run us one of the mics from the backup. We will get it.

Otherwise, I will introduce you to Steph. We have Stephanie Michiels, who is a senior director of product, Infrastructure and Cloud Services. It is a tough one. It is a long title, and she does much better than I do. Uma, if you want to go next.

Uma Vandegrift

Everyone, I am Uma, and I lead Shared Services. We are excited to share our journey with you all. First time presenting at this conference, and we are excited for all the conversations.

For those of you who are not familiar with Northwestern Mutual, Northwestern Mutual is an insurance and financial services company. We are headquartered in Milwaukee, and we have been around for 165 years. We have close to 8,000 employees, so we are a large enterprise.

With that large enterprise, it is probably no surprise that our technology is very complex. We have everything you can imagine on the tech side. We have mainframe. We have AWS. We have everything in between. We had similar complexity with our processes. We have some teams that run SAFe. We have some teams that are waterfall. We have everything you can imagine, and it seems like every other week I hear about a new process that someone is following, and we are just extremely, extremely vast.

We have over 350 agile teams and over 4,000 technologists. With that much complexity, maybe it was not a surprise that we found ourselves contemplating this: What do you do when you failed again?

For us at Northwestern Mutual, we have gone through many, many iterations of change, and we took a second to think about this. Yes, we might not be where we want to be, but is this statement true? Have we really failed again?

We realized that we needed to take some very active steps to reframe that way of thinking. We stopped thinking of ourselves as going through a binary thing of failing or succeeding. We stopped thinking of going through these vast pendulum swings of, hey, we are going to do this, and now we are pulling in completely the other way. We started thinking of this as a continuation, a continuation of some truly excellent work that had been going on for quite some time. So we embarked on a new SDLC journey.

We were extremely intentional when considering rebooting our SDLC initiative, and we had the excitement and the right mindset around going on this journey and building on what we had already done.

This time, we knew we needed a lot of flexibility to help us move forward. We knew that we could not box ourselves in, but we also understood that our enterprise needed a similar mindset to be able to move forward. So we cheated a little bit. We stole some of this from the Agile Manifesto, at least the format, so hopefully it is familiar to most of you.

We have these four values that we anchored around. We valued automation, streamlining our process, solving engineering pain points, and maybe my favorite, standardization. These were four values that everyone could buy in on and that felt real enough to us. They underpinned almost all of the work that we did, and they allowed us to gain the trust of the enterprise that we needed. These are what helped us reframe and move toward that journey.

To make that a little bit more real and concrete, we anchored these four values to some tangible moonshot goals that Jason will talk about.

Jason Cowdy

Thanks, Uma. All right. Up here on the left side of our slide, you can kind of see these four moonshot ideas that we came up with. These moonshots are really the embodiment of those values that Uma just showed us on that last slide.

These are our biggest bets for our company and for our team, where we think we can move the needle for everybody across the organization, no matter where they are in that technology landscape.

The cool thing about these is that it is kind of a good mix in there. We have some very strategic enterprise initiatives in there, but we also match those up with some very grassroots engineering pain points that we wanted to solve as well.

I am guessing a lot of you are sitting out there and thinking, oh, we have been doing these all the time. These are not a big deal. But for us at Northwestern Mutual, these are huge moonshots, and these are very difficult things to do. We are really excited about trying to solve them.

Our first moonshot is for every new engineer that joins Northwestern Mutual: on day one, when you show up, you are going to be coding, committing that code, and deploying that code. You are going to have a working application by the end of the day.

Behind that, there is a little bit more work that goes into that. When you show up, we need to make sure that we have a laptop ready for you with all your software installed, all the same defaults, all the configurations, so you do not need to do that. In addition to that, we also need to make sure that you have access to all the systems and all the tools and all the applications that you are going to need in order to do your job.

Our goal really there was driving down our time that it takes to onboard a new engineer. In the past, it is taking anywhere from a month to a couple months to get somebody to the point where they are truly productive. Our goal there is to drive that down, and within your first week, when you move over to your team, you are ready to go and you have all the skills that you need in order to start contributing.

Our next moonshot was increasing our deploy frequency. We were looking at our DORA metrics, and we decided to focus on just the one to start with: increasing that deploy frequency. A piece of that that we also realized really quickly is that we needed to pair that up over on our agile side. A lot of our teams needed work making sure that their backlogs were in a very healthy state, that they had work that was pointed, groomed, and had good requirements to it that could also support that deploy frequency when we saw that improvement.

Along with that, we made a pretty stark realization with the help of some friends: you cannot just focus on velocity. There are other components of that. You need to also worry about stability, and we need to keep those in a balance. We also need to think about how we do that securely at the same time.

If any of you are paying attention this morning, we had a couple talks, and those last two on there should look awfully familiar. The folks over at American Airlines and PNC had a very similar story that they told.

Our first one in there is our 100% automated guardrails to production. Really, our goal there is that any engineer, when they are making a change through our pipelines, can go from their commit all the way to production through the pipeline without any human intervention, without any handoffs, without any roadblocks or check-ins.

The way that we do that is we basically watch our pipelines, and we have a thing called our pipeline enforcer that is our automatic guardrail there. The cool part is we are able to catch issues early before they ever even make it into a non-prod environment, let alone a prod environment. It will alert you that something is wrong, whether it is a compliance issue, a security issue, or something else.

The really nice part about that is that we also paired it up with Slack. As I push my code and my merge request goes through and the pipeline starts, I get a Slack notification from our bot that says, hey, there is a problem over there. With that notification, you also get a link to our documentation that says, here is how to fix the problem too. I had to laugh a little bit because right before we were here, sitting in the speaker lounge, I got one of those notifications because my team was doing something in the pipeline that I was not happy about.

The last one in there is probably the one that I am most excited about, and that is building an opinionated platform in the way that we deliver and deploy software.

I would say in the past, we have had a little bit of the wild west of software engineering. It was use whatever you need, do it whatever way you need to in order to deliver the feature, deliver the application, or get the job done.

That led to a couple different problems. When we looked at our lead time, just going from idea to something running in production was taking way too long. That is because, again, every engineer out there had to reinvent the universe, had to be the expert in every single system and every single decision just to get to the point where they could start writing the code to deliver business value.

On the flip side, once you did get to production, you got another problem, and that is those day-two operations: the care and feeding of that application, the patching, everything to keep it up to date and compliant. Because everybody did it their own way, it is super hard to automate those solutions.

Our goal there is basically to go from decreasing our lead time from weeks or months to deploy an application down to days, and from getting code running in our clusters from days down to minutes. On the flip side of that as well, in those day-two operations, we really want to drive down the amount of time that our engineers spend doing that toil work and let them have more fun and work on more value-add.

The way that we are going to do that is through the two there over on the right side. The first one is called Ideal Pipelines. For any engineer, for applications, they are going to have a single include for their pipeline that includes all of the required stages that they need in order to deliver an application. That is going to include all their building, all their scanning, all their linting, all their testing, and even through to deploy.

The tricky part about that is we know we will never get it exactly right and cover exactly every single use case out there, because that is impossible. So we also need to look at extensibility: how would I add my test cases that are specific to my application here in this job?

The last one on there, and I had to laugh a little bit with the last presentation from Bill, he stole my line: golden paths. I am really excited about this. The idea is, whenever an engineer starts a new project, they are going to go to our developer portal and they are going to go to our vending machine UI and pick their pattern. After picking the pattern, they are going to answer a few simple questions: What are you building? Where does it go? What type of thing is this? Then you are going to hit the button, and in the background we are going and automating everything we possibly can, from creating your repo, to setting the environment variables on it, to filling it with code, to kicking off your pipeline, to setting up alerting and monitoring, and everything that a human would have to do initially, we are going to try to automate.

All right. It has only been about six or eight months since we started this journey, but I think there are some really good lessons that we have come away with and that we want to share today.

I am going to take the first one here. I think our first lesson that we learned is that we needed to be very engineering-focused. In our previous SDLC attempts or our programs, they were very top-down. It was very process-focused. I think if you went and talked to any of our engineers, they would kind of say, yes, it was not really for me, not really involved there. I know it exists, but it does not really affect what I do.

With this new approach, we decided to be extremely intentional about meeting our people where they are at, especially our engineers. Like Jason Cox said our first day here, we really needed to listen to them, to hear their pain points, and to have empathy for what they are going through and how they deliver software.

Over on the right side of the screen, I am not going to go through them, but those are a bunch of the pain points that our engineers brought to us. If you read through those, you can actually see those are a lot of the inputs that drove some of those grassroots engineering moonshots that we are going through today.

The other part of this that I am really excited about as well is our rotation model that we took on our SDLC team. We invite engineers and actually some non-engineers as well to join our SDLC team for a six-month rotation. They are embedded on our team. They are 100% a member of the team while they are there, and they kind of have two jobs.

They help us build all of the moonshots that we just showed you and help deliver those. In addition to that, they also act as a representative of their organization. That kind of works two ways. They get to represent their organization. They are going to tell us, hey, this is not going to work the way we work over here; what you are building is not going to work, we need to do something differently. It also works the other way around. As a member of our team, they get to learn about how we operate over in the SDLC, and then when they go back to their team, they are kind of our inside person over there. They are the expert, and their team looks to them as well.

For our next lesson learned, I am going to hand it over to Steph, and she is going to talk about some of our platform learnings.

Stephanie Michiels

All right. The second key lesson that we really took away with and reflected on in our journey so far has been our internal platform approach within our Infrastructure and Cloud Services. I have the privilege of representing our Infrastructure and Cloud Services engineering team.

We talked about, hey, our SDLC journey has really started about eight months ago. But really, we were also going through a journey on the Infrastructure and Cloud engineering team side, and that was really to think about ourselves through the lens of a product mindset. Again, it started a couple years ago, where we really started to say, hey, what are the things that we need to do from a platform to really unlock the power and the value for our developers? It was really a pivotal mindset change that we went through.

What we did as a team is we really asked our developers, what are your pain points? Jason talked about it on the SDLC, on the application side. We cared about velocity and speed, and on our side we cared about the stability and the security.

We thought we were doing all the right things. We have really talented engineers. We were doing a lot of security hardening for a long time to make sure, for example, our cloud environments were secure when we gave them to developers.

But it really actually was not until we started to pivot into this product-centric approach and really thinking about ourselves as a platform, when we asked our developers, what do you need? It was kind of shocking. One of the things that they told us right off the bat was: managing your infrastructure can be hard. The day-two operations, the patching of what you give us, is really hard. Please, can you make it easier for us to patch our systems?

It was a moment in time when we said, hey, just by asking that question, just by thinking through it from a product-centric lens, it really changed what we prioritize and the way that we did engineering work. This was a big lesson for us.

Then, ironically, which was awesome, the State of DevOps Report, the 2020 report, came out. We were like, hey, look, this is awesome. No, this is not just me trying to be lazy and not create a slide. When this came out, this was part of the journey that we had been going on. We really found it is things like this, talking through from an industry perspective, really understanding what we are doing and the similarities with other companies, that has been really helpful for us.

The other thing we would say is that mindset change on our side, and the way that we set up this program, we cannot emphasize enough. I know there is a ton of collective experience in this room and at this conference. We have all been through several organizational changes and transformations and agile journeys and cloud journeys.

The trust and the teamwork was paramount for us to be successful. It was such a game changer for us. We were lucky for this one. The stars kind of aligned, where from an infrastructure side, we had started leaning in toward this product mindset. The SDLC program led very inclusively with how we were going to build toward common outcomes, and it has really allowed us to kind of swarm together for the greater outcomes that Jason talked about as a team. We have engineers that float in and out of this effort, and it has really been, again, a game changer for how we think about success.

With that, we are going to leave you with a few takeaways. One, we would say always learn from the past. Retros are awesome. We are always learning. We are never failing. We are always leaning forward with something greater and better. I think we see that often. We live with a mentality of, let us sprint toward our next outcome and goal.

At the same time, set your outcomes big, because your engineers are awesome and they can meet them. We find that over and over again. If you set your moonshots, if you set your big goals, people will do amazing things.

We talked about it on the internal platform approach. I think we have all kind of heard that at this conference so far this year. It is really, really important for us to think about our internal platforms and have the product maturity and make sure that we are doing the right things for our developers.

Jason talked about fix engineering pain points. Do not fix what you think you know. Get in there with engineers. Make sure you know what they really need to be successful.

The last corner: foundational teamwork and trust. There have been a number of presentations. Some of the stuff we are talking about is complex. You have different teams from different organizations working on them. If you do not have trust in each other that you are going toward the common goal, you have the potential of really failing and not going really far. Again, just working together has been one of the most foundational things that has allowed us to make sure that we trust where we are going together.

With that, we just say thank you for listening to a bit about our journey. This is, I guess, the start, or middle, or ongoing, however you look at it. Super grateful to talk. We would love to hear any questions and future collaborations. So open it.