Host Intro (Gene Kim)

I'm super excited about the next two sessions. Over the years, we ask the community their top obstacles to things they want to achieve, and almost every year, when they talk about the obstacles in their way, it's audit that strikes the most fear, dread, and frustration because of the special powers auditors have to generate findings that are seen at the highest levels of the organization and their use of sometimes decades-old audit practices.

I am personally grateful to all the work that Clarissa Lucas has done. She is Director of Technology Audit at Nationwide Insurance, based in the U.S., one of the largest mutual insurance companies. Over the past two years, for reasons I'm just now starting to understand, she's presented at this conference with very specific and useful advice about people who have to work with audit. Amazingly, she and her team have come up with a very specific technique on overcoming issues specifically around segregation of duties, change approvals, and so forth.

In the past, she's presented with fellow auditors on her team, but earlier this year, to my utter shock and surprise, she planned on co-presenting with someone in technology leadership at Nationwide, which, to put it lightly, is just something that is not normally done, at least not in polite company. She's presenting with Tod Bickley, Associate VP of Information Risk Management, responsible for the identity and access management systems. It is a shared service, which is so important because so many major applications rely upon it. They describe a startling audit engagement model. In fact, it's startling, and I say this based on decades of seeing talks from ISACA and the Institute of Internal Auditors communities. Here is Clarissa and Tod to talk about what they did, why they did it, and the value they created.

Clarissa Lucas

Welcome, everyone. Let's kick this off with a quick question. How many of you brought your auditor with you today? Just me. Oh, Foundation writers did? Oh, he works with us. All right, let's try another question: raise your hand or make some noise if you've ever been dreading the auditors coming, or you've cringed when you heard the word audit or auditor. Okay. As much as my heart is breaking right now, it's all right. You're in the right place.

As Gene mentioned, I'm Clarissa Lucas, and I'm here with Tod Bickley. We are going to walk you through Nationwide's journey to a better audit experience through auditing with agility. We'll explain the benefits you can experience when an audit is performed this way, and then give you actionable takeaways you can take back to your organizations to influence a better audit experience.

I'm a technology auditor at Nationwide. I've been there for ten years. I've got about fifteen years of audit and risk management experience helping management and auditors work together toward that better audit experience with more value for your organization. That's something I am incredibly excited about and passionate about, so I'm really excited to be here with all of you live in Las Vegas.

Tod Bickley

Thanks, Clarissa. My name is Tod Bickley. I lead the identity and access management capability at Nationwide. When we say IAM, we're talking about authentication, authorization, single sign-on, identity governance, MFA, all the things. I've been at Nationwide about twenty years. I started on the IAM team as an engineer in the early 2000s. After a couple of years, I looked around at my fellow engineers and said, you know, I'm just not as smart as these people, so I became a manager.

I started leading different teams in infrastructure and application teams. In 2019, I came back to the IAM team as its leader and came back to several things that were going on. We were having problems with flow. We were having problems getting work through the system, and that was my charge to help do that. We also had audit issues that kept recurring that we weren't getting our arms around.

Nationwide is a U.S.-based company. I think we're 80 in the Fortune 100. We're an interesting, unique company: a large property and casualty company and also a large financial services company. Up until about fifteen years ago, our financial services company was publicly traded and the property and casualty side was not, so the technology evolved differently over the years, which adds complexity. About fifteen years ago, the companies came back together. We're now a mutual, privately held, owned by our members. We're number one in 457 plans, number one in several property and casualty areas, and in pretty much every other place you're going to see us in the top ten. We're also big in our communities: avid supporters of Nationwide Children's Hospital, hunger relief, United Way, Feeding America. We take real pride in contributing to the communities where we have businesses and where our associates live.

Let's back up to 2019, the before times, when they asked me to come back to identity and access management. I said absolutely because I love IAM for two reasons. First, IAM, in my subjective opinion, is the foundation for zero trust architectures. To be at a conference in Vegas and not hear the word zero trust for a cyber guy in the last day and a half is shocking to me, because usually I hear it 700 times a day out here. The second thing I love is that it's a set of technologies that touches every associate, every member, every customer, every business partner, every day, and I love being involved with something that has that impact.

When we took the role over in 2019, common things were causing issues in the team. We had no flow of work. The teams weren't organized effectively. There was no priority. Everybody was chasing things individually. My boss had just come from the software side of the house and had implemented agile, product, and DevOps methodologies across the app teams, but we'd never done it in infrastructure. He said, why don't we look at putting product models here? I said, well, it can't get any worse, so let's give it a try.

Within the first few months, we started doing all the standard DevOps and agile things application development teams had been doing for years. We reorganized into product teams. We had plan, build, and run on the same teams. We created product management and product owner models. We started getting all demand backlogs into Jira, doing prioritization, unit costs, sprint deliveries, and all those cool DevOps-y things. The team was focused. Engineers were only in meetings in the morning and then actually doing work all day instead of sitting in meetings figuring out what to work on.

In the middle of a large technology transformation, I got a call from my auditor friend Clarissa. She said, hey Tod, congratulations on your new role. It's time for your biannual IAM audit. I said, awesome: I'm in the middle of a multiyear, multimillion-dollar technology transformation and a huge product reorganization. The first thing I want to do on top of that is an audit.

Seriously, it was fine for a couple reasons. Audit work, when you look at it, is just demand into your team like any other set of demand. It's evidence asking. In the IAM team, we have tons of controls, so we're generating evidence that Clarissa's team has to validate. It's meetings to go through evidence and meetings to review results. If you think about it, it's just like demand you get from any other infrastructure or application team. I said, great, Clarissa, love to have your team come in, let's get this done, but we cannot do this in a waterfall method. The teams are all agile. We're delivering in sprints. We need to think agile in terms of this. Clarissa said, that's awesome, because I've been thinking about agile and reading about agile for the last year or so and would love to kick the tires on how we start to do agile deliveries with audits within your team.

Clarissa Lucas

That's great. That's absolutely how it happened. But it wasn't quite that simple or easy. Both teams, the auditors and the IAM team, faced challenges when figuring out how to change our ways of working and not bring a waterfall audit to the IAM team.

The first challenge the auditors faced was fear that we would violate the professional auditing standards we're held to. To make a long story short, there was no violation of those standards. We were able to modify our ways of working and still comply with them, but walking into this we didn't realize that. We also didn't have much experience or knowledge in agile or DevOps ways of working, which created some hiccups. And there were major cultural and procedural changes from an audit perspective that we needed to overcome to change our ways of working.

Tod Bickley

We had challenges too. First, the sprint approach and aligning on how we were going to deliver results was new for the auditors, so we had to work with them to help understand the agile methodologies. Next time, I think we're going to put a bigger emphasis on upfront planning. This was a result of the situation: the call came, we were in a transformation, and we didn't have a lot of time to do planning. Next time, which is coming up here in about six months, we'll sit down for a day or so and say, how do we want to lay these controls out across our sprints?

The third was that auditors knew about agile and read about agile, but they hadn't done anything with it before, so it was really about helping them through those agile processes. One thing I've loved over the last couple days is empathy for people around you. You can have empathy for your auditors too, because they are people and they really do want to help you and the company make sure you're staying secure.

Clarissa Lucas

Definitely. Let's take a quick step back and set the stage with context about where we started. For as long as I've been an auditor, and even well before that, we've used a waterfall approach to conduct audits. Similar to waterfall in software development, a waterfall audit is a very gated, phased approach with dependencies. We plan the entire audit, identify key risks and key controls, write testing procedures, get a request list together, get that approved, and then move to fieldwork. Fieldwork is where we execute everything we planned: test the controls, document evidence and work papers, get those approved, and then move to reporting. Reporting is when we finally deliver results to our clients.

Based on the reactions I got at the beginning, I'm guessing you're familiar with the challenges that approach presents in software development and during an audit. I won't spend a ton of time on that, but I want to touch on a couple. The first is applying the same rigid audit approach to every situation. Waterfall worked really well in the past because risks were pretty static. They didn't change. We could plan upfront, take time in the other phases, and deliver results at the end without much changing. There are still situations where that makes sense. But thinking we can apply it in every situation in today's environment, where risks are changing with a velocity I've never seen before, doesn't really work.

The second major challenge is feedback. We hold feedback to the end. Not only do we wait until reporting to deliver feedback to Tod and his team, but we also wait to solicit feedback from him. I love that we get feedback, but waiting to the end isn't the best. When Tod says, hey, it would have been better if you had done this, we missed the opportunity to pivot during the audit. The best we can do is, a few years later when we audit him again, hope we've addressed that feedback.

On this experiment, we had encouragement and coaching from Tod and his team, and we modified our ways of working to address those challenges. It wasn't just at Nationwide that we knew we needed to change from waterfall. The entire internal audit profession has been working toward other approaches, and a lot of organizations have moved to what's called agile auditing. Agile auditing is a sprint-based delivery model for an audit. You divide the audit timeline into sprint time boxes and deliver iteratively at the end of each sprint. This is great because you're doing feedback more frequently, but it still applies the same rigid framework to every situation without taking into account the team, delivery model, environment, and other factors. It made progress, but still assumes one size fits all.

What we did was a sprint-based delivery model because Tod and his team deliver in sprints, but we also applied the three ways of DevOps, creating a customized approach. The first way, flow and systems thinking, meant we came together as one team. It wasn't auditors and Tod's team on opposite sides of the table. We were together, collectively trying to provide assurance over the key controls that mattered to the organization. That was great because we were no longer getting in each other's way. We were focused on one goal together.

The second way is feedback loops. We addressed this by delivering in sprints every thirty days. Instead of waiting until the end of the audit, a few months later, and handing all findings and assurance to Tod, we delivered every thirty days. That enabled his team to start working on findings and gaps from the beginning. We could say: these are the controls, these are the most important controls your team absolutely has to get right, and either they're working so go get sleep tonight, or they're not working the way you want them to, so start working on those. By the time we got to the final audit report, a lot of progress had been made.

We also intentionally solicited feedback throughout the audit. At the end of each thirty-day sprint, we had a retrospective with the entire team. In the first retrospective, we got feedback on meeting frequency. At that point we were meeting twice a week: Tuesday working sessions for control walkthroughs, document requests, questions and answers; Friday to talk through results. In that first retrospective, we got feedback that instead of meeting twice a week, they wanted to meet with us daily. Let that sink in: management wanted more frequent meetings with their auditors. I had to play it back in the retrospective to make sure I heard it correctly, but it was an actual request. We moved to daily meetings. They turned into stand-ups, only fifteen minutes, and we got clarity on requests. Tod's team's questions and my team's questions were answered the same day instead of waiting days or weeks.

Finally, the third way is continuous learning and experimentation. We built this throughout the audit from day one: let's experiment and change from waterfall, try adding agility and DevOps concepts to this audit. We knew it would be an experiment and clunky. That mindset came in handy, especially in the reporting phase. While we delivered in sprints, we still had a final report at the end. We delivered interim reports every thirty days to get results into Tod's hands, and then compiled all of that in a final report with our overall audit opinion. Where we went wrong was the auditors went back to our desks and did that in a silo. That was not how we had conducted the rest of the audit. We had been doing everything together.

When we delivered that report to Tod and his team, it was a surprise, and it was not the kind of surprise like chocolate or champagne. It was audit findings and an audit report. Nobody loves a surprise audit report. It got clunky and challenging, but because we had built continuous learning, experimentation, and trust throughout the audit, we came out stronger on the other side and worked through it.

We also implemented making work visible, self-organizing teams, and greater collaboration. The benefits included greater collaboration and engagement within both teams and the collective team; focus on the areas of greatest priority and highest risk to the organization; successful adaptation to change; greater buy-in; more timely communication of results; and reduced wasted time.

What did it actually look like? We got faster. The amount of calendar days we spent auditing Tod's team reduced by ten and a half percent. We also increased the speed with which we delivered gaps to Tod and his team, from when we identified them to when we got them in his hands so he could do something with them. In addition to getting faster, we got better. Because we delivered results sooner, when we got to the final report showing everything we identified together throughout the audit, we could show progress. Instead of saying, Tod, here are a bunch of findings, good luck, or telling stakeholders to look at all the stuff Tod has to do, the story was: look at all the assurance we collectively provided, here are the things that need improvement, and look at all the progress Tod and his team have already made. Completely different story.

We also got more coverage. We covered more identity domains in the 2021 audit than in 2019, when we did waterfall. We sent a survey at the end of the audit to Tod and his team to identify what went well and what we could improve going forward. We collected feedback throughout, then wrapped it up with a bow. The satisfaction rating Tod and his team gave us improved two levels from 2019 to 2021.

Tod Bickley

How can you get there? Partner with your auditors and help them through the learning curve. If you have a way of working that is very DevOps and agile, partner with them and take that journey. Demonstrate how to run effective stand-ups. That was one of the first things we started, because the biweeklies didn't work for us, so we went to daily stand-ups. Teach them how to use the tools you're using, whether it's Jira, whatever you use for backlog, scrum, Kanban, or whatever method. Bring them into your team. Keep an open mind, have empathy, and encourage your teams to participate with the auditors as one team. When you look at it, they are just here to help the company and help protect you from things that are going on.

Clarissa Lucas

We all have the same objective. We're here to help the organization make sure it can achieve its objectives, and management is trying to achieve those objectives. Instead of working against each other, let's work together.

We have some help we're looking for. As much as it will break my heart, I do need to hear some of these audit horror stories. Connect with me and let me know your worst: what went wrong? I'd also love to know if you love your audit experience and just forgot to bring your auditors with you today. What went really well? What are some things I can learn from you and take back and experiment with at Nationwide?

Tod Bickley

Mine is actually away from the audit thing. We're on a journey to eradicate passwords from our environment, both for our associates and our members and business partners. If you're on a passwordless journey, I would love to hear what you're doing, your plans, and blockers, because we are hot and heavy into moving forward with that right now.

Clarissa Lucas

I appreciate all of you hanging out with the auditor in the room today. Next year, bring your auditors. Teach them about DevOps. I really appreciate it and look forward to connecting with each of you afterward.

Tod Bickley

Thanks, everyone.