Host Intro (Gene Kim)

[00:00:14.260] Hello, and welcome back to the day two general session talks.

[00:00:19.060] Over the years, we asked the community about their top obstacles to things that they want to achieve. And almost every year, it is audit that strikes the most fear, dread, and frustration, probably because of the special power they have to generate findings that are seen at the highest levels of the organization, and their use of sometimes decades-old audit practices.

[00:00:38.200] I am personally grateful for all the work that Clarissa Lucas has done. She is Director of Technology Audit at Nationwide Insurance, based in the United States, and it is one of the largest mutual insurance companies.

[00:00:51.940] Over the past two years, for reasons I don't quite fully understand, she has presented at this conference with incredibly specific and useful advice to people who work with audit. Amazingly, she and her team have shared very specific techniques on overcoming audit issues concerning separation of duties and change approvals.

[00:01:12.420] In years past, she presented with auditors on her team. But this year, to my utter surprise and shock, she had planned on co-presenting with someone in technology leadership at Nationwide, which, to put it lightly, is just simply not normally done.

[00:01:29.040] She is presenting with Tod Bickley, an Associate Vice President of Information Risk Management, responsible for the identity and access management systems. This is a shared service, which is such an important security control because so many major applications rely upon it.

[00:01:46.050] They describe one of the most startling audit engagement models I have ever seen. In fact, it's one of the most startling presentations I've ever seen, period. I genuinely believe that they are on the frontier of revolutionizing internal audit practices for the entire profession. And I don't say this lightly, as I've seen decades of audit talks from the ISACA and IIA communities. So here is Clarissa and Tod to talk about what they did, why they did it, and the value they created.

Clarissa Lucas

[00:02:16.320] What comes to your mind when you find out that the auditors are coming to do a review of your processes? Go ahead and put those thoughts into the chat.

[00:02:24.980] Now, I imagine most of you aren't jumping for joy, and a lot of you probably aren't typing things into the chat like, "We're having a great time," or, "Enjoyable."

[00:02:34.420] What if I told you that it's not only possible to enjoy a highly collaborative audit, but that you also have the tools to get there? I'm Clarissa Lucas, and I'm here with Tod Bickley.

[00:02:44.340] Together, we'll guide you through Nationwide's journey to agile auditing, where you'll learn about the benefits you can experience in an audit performed with agility. You'll also learn valuable insights on how you can work together with your auditors towards a collective value-added outcome, and have fun along the way.

[00:03:01.800] I currently lead Nationwide's technology audit team. My team and I help Nationwide achieve its objectives by providing assurance on key risks and controls. We also help management see around the corner to provide advice on emerging risks.

[00:03:15.500] I've been with Nationwide for about nine years, and in my current role for a little over three. Before taking on this adventure to leading the technology audit team, I've been in various roles in audit, risk management, and compliance. Outside of work, I chauffeur my eight-year-old son to hockey practices, and I enjoy pretty much everything Star Wars with my husband and my son. Tod?

Tod Bickley

[00:03:38.040] Thank you, Clarissa. Hi, my name is Tod Bickley. I'm currently the Identity and Access Management Product Owner for Nationwide and was formerly the Identity and Access Management Product Manager. We'll talk about that in a few minutes.

[00:03:51.700] I've been at Nationwide for a little over 20 years now, and I've covered a spectrum of technologies here. I did start off on the IAM team, actually as an engineer, and I've sort of done full circle here over the last few years and came back to lead the IAM team, because I love the IAM technologies.

[00:04:07.760] The reason I love IAM technologies is because, as the world moves to a zero trust model to secure our associates' data and our members' data, making sure people have the right access to the right things is critically important, and that's what IAM does. Given that it's critically important, the controls that we use to manage that are also critically important.

[00:04:31.340] Our audit partners, like Clarissa, also help us manage our control hygiene, to make sure that we are doing things the right way. So having a good, beneficial relationship, making them feel part of our team when they come in to do these audits, is really critical to our success. I appreciate Clarissa invited me to this session, and I look forward to talking to everybody about how we made this happen.

[00:05:00.460] Who is Nationwide? Nationwide is a U.S.-based company. We don't sell any products or services outside of the USA. Historically, we were an insurance company with a financial background. And now really, we're actually a pretty large financial services company.

[00:05:17.140] You can see a couple of stats up there, but we're number one in state-sponsored 457 plans. We're number one in selling agriculture insurance to farms and ranches, number two in corporate life, and overall, we're the number eight overall property and casualty insurer in the U.S. We have about 28,000 U.S.-based associates.

[00:05:40.640] In terms of where we are with rankings across, we are number 25 out of 100 for Fortune's Best Places to Work. We're number 50 out of 100 for Best Workplaces for Diversity, and community involvement is a very big priority for Nationwide. We're a big sponsor of Children's Hospital in Columbus, Ohio. In fact, it's Nationwide Children's Hospital.

[00:06:06.900] Volunteering, giving back to our local communities, whether it's in Columbus or a lot of the other satellite cities that we're in, is very important. It's actually one of the things that is one of our big objectives every year. It's one of the reasons that attracts a lot of people to this company, because not only do you do good work to protect people's futures, you're doing work in your community to help everybody, whether you're a Nationwide consumer or not.

[00:06:32.388] Let's talk a little bit about what we've done in terms of our product model journey for IAM. Identity and Access Management encompasses all of your standard IAM technologies. We've got single sign-on and multi-factor. We house all the authentication repositories, all the authorization repositories, all those standard things.

[00:06:52.608] We started a product model journey really late in 2019, with a focus on implementing all those great agile, DevOps, product practices that our application development teams have been using for years. We were trying to really solve this classic IT problem of getting work to flow efficiently to the teams, getting work to flow efficiently through the teams, and getting work to actually execute efficiently to deliver outcomes and values for our consumers.

[00:07:19.928] We wanted to understand our unit cost. We wanted to have people in standard roles. We wanted to be able to use stand-ups at minimal times so people could focus on doing work during the day. Just all those great things: deliver things in sprints, break up work into consumable chunks, think about MVPs, just all the great things that Agile and DevOps have done for our development communities forever. We wanted to start to inject those practices into our infrastructure teams.

[00:07:49.588] When we looked at identity and access management and our product areas, there's really two specific functions we broke in. We've got our technology product management function, which is really our infrastructure teams, and they focus a lot on the hands-on work. They do the hands-on keyboard work. They're configuring and programming and supporting and operating all the systems. They really keep everything, sort of the heartbeat of the systems, going every day.

[00:08:14.248] On the risk side, we have the product ownership. The product owners work closely with the product management team, to help set the work, to help groom backlogs, to help bring work into the backlogs, to help the product management teams understand what's important so they can focus on actually doing the hands-on keyboard work. We do a lot of the strategy work, and we set that forward for everybody to execute.

[00:08:41.908] We really started taking these product principles in late 2019 and early 2020, and defining them within the infrastructure teams, and embracing all these agile methodologies.

[00:08:54.768] About five or so months into the journey, we have a biannual identity and access management audit that Clarissa and her team bring forward. Clarissa came and said, "Hey, it's time." We said, "Great, Clarissa. We love working with you. We love the way you guys look at things differently than we do, but we're really in the middle of this journey. If you're going to come in and work with us, you're going to have to work with us through our new agile processes. We've got the team really excited about the sprints and the backlogs and the fact that we've got single front doors and we're using Jira to manage all of our demand. We can't come in and do a waterfall delivery of all of our agile work."

[00:09:34.528] Because, as most of you know, when audit teams come in, they're really just generating demand into your team. We're producing evidence, we're having meetings, we're reviewing things. It's really just like a development request or a request from other teams. So we said, "You have to do this." And Clarissa, being a great partner, said, "You know what? We've been talking about agile audit within the industry and really here at Nationwide. This is an excellent chance for me to rally our team around doing this."

[00:10:03.188] So we brought them into the team, and really, they just became part of our IAM team for the three months or so that we did the audit. They attended our stand-ups, they came to our sprint reviews, they attended some backlog grooming sessions. They used our flow processes that we defined for our other folks who were bringing demand into the team, and they really just functioned like everything else.

[00:10:33.208] I think it changed a little bit about the audit outputs, because what we didn't do is we didn't get a big chunk of work or a big chunk of what happened with your audit at the end. What we saw was we saw our audit now, not only was the work broken up, but our audit results were also delivered in incremental pieces, and it was much easier for us to consume those things instead of just a big hunk of work that was dropped on our plate at the end. So it worked really well and appreciate them being able to engage.

Clarissa Lucas

[00:11:03.388] But we thought it wasn't quite that simple. Where we typically do our work, yes, we were totally on board, but this is a huge change for how we do our work.

[00:11:15.088] The way that audits have been performed for years has been pretty consistent, using that waterfall approach to do an audit. We plan out the entire audit, figure out what are all those key controls and risks that we want to cover. We go through our approval gates and get everything approved before we move to testing and fieldwork. Once we move to fieldwork, then we test all of our controls. Once we finalize the testing on that last control in scope, then we move to the reporting phase. Then we can communicate our key results to you and your team.

[00:11:48.668] This has worked really well, but we know that there's always room for improvement. With your team's encouragement and guidance, we were definitely able to take that leap and make that drastic change in the way that we were doing our work.

[00:12:07.088] Some of the things that were really helpful that we implemented in this were adopting some agile concepts. We anchored throughout the engagement to the four key agile values, and we made one slight modification along the way. We modified the value on the bottom left of the screen that typically emphasizes working software over comprehensive documentation.

[00:12:30.108] Valuing working software doesn't really speak to the team, didn't really speak to the audit team or Tod's team in this instance, because software wasn't our deliverable. Instead, our deliverable for the audit was actionable insights. Those include assurance that key controls are designed and operating effectively, findings articulating a risk or a control gap, and insights on how to improve the effectiveness or efficiency in that process under review.

[00:12:57.038] Please don't get me wrong, we auditors, including myself, still love documentation. We value both documentation and those actionable insights. But the key difference when we changed our way of working with Tod and his team is that we made sure we didn't lose sight of that collective outcome of delivering actionable insights for the sake of dotting all of our I's and crossing all of our T's in our work papers.

[00:13:18.118] Nobody sees those work papers except the auditors and the people who audit the auditors. They are incredibly important to support those conclusions that we reach, but that audience is really limited. On the other hand, those results that we communicate, the assurance over the controls, the control gaps, and those opportunities for enhancement, has a much larger audience and a greater impact. So that's where we focus most of our attention.

[00:13:43.998] We anchor back to these agile values, as well as the principles outlined in the Agile Manifesto, and successfully implemented a number of those agile concepts, like self-organizing teams, prioritizing our customers' needs, fostering a collaborative environment, and delivering results frequently.

[00:14:03.158] One of the things that we did with self-organizing teams was we further expanded the team beyond just Tod's team and my team to include someone who's really well-versed in agile auditing standards. They helped provide those insights along the way to keep us within those guardrails while we adapted agility and were still able to comply with auditing standards.

[00:14:27.118] We also intentionally prioritized our customers' needs and fostered that collaborative environment by planning the engagement scope together. I just explained the traditional approach to an audit where we would talk to Tod's team, get a high-level understanding of what they did, what they were accountable for, go back to our desks, develop our scope, and then come back and present it to Tod and his team: here's what we're going to do in the audit.

[00:14:54.398] We didn't take that approach this time. Together we worked with Tod's team to identify those key risks and controls relevant to the area that we were reviewing. Collectively, we came up with the most effective way to test each of those controls together. Who better to tell us how to perform a control and where that documentation is and what it looks like than Tod's team, who lives and breathes identity and access management every day?

[00:15:23.808] Another element of the Agile Manifesto that we incorporated was delivering results frequently. We delivered our results every 30 days. Instead of waiting until the end of the audit, we issued interim audit reports at the end of every 30-day sprint. This enabled Tod to begin addressing any findings earlier in the process, which reduced risk exposures much sooner.

[00:15:50.078] At last year's DevOps Enterprise Summit, I joked about taking a page out of your book. This year I took it straight out of The DevOps Handbook. We incorporated the Three Ways of DevOps into this audit as well.

[00:16:00.978] The First Way was, Tod described this when he talked about how we became one team. We also incorporated the Second Way, which is feedback loops, throughout the audit by intentionally soliciting feedback from the entire team. Again, not just Tod's team, not my team, but all of us, including executive leadership. We held retrospective reviews at the end of each sprint to identify what went well and what we wanted to focus on improving in that next sprint.

[00:16:28.158] At the end of the entire audit, we sent surveys to Tod and his team to collect additional feedback. Most importantly, because we worked so closely together during the audit, we exchanged real-time feedback too. Rather than waiting until the end of the sprint or the end of the audit, we got real-time feedback.

[00:16:44.798] Now we learned the Third Way the hard way. As we neared the end of the audit, we faced a really significant challenge when we reverted to our old ways of working for that final reporting stage. We reverted back to the auditor and the auditee structure when we compiled that final audit report and determined the overall rating. Instead of experiencing this part of the audit together like we had up until that point, reverting back to our old ways of working left Tod and his team on the opposite side of the table from me and my team. We were briefly adversaries.

[00:17:15.578] The partnership that we built and strengthened during the audit, as well as that commitment to the collective goal that we all had as one team, is really what brought us through this challenge stronger than before. Tod gave us grace as we navigated through this hurdle. He created a culture that fostered taking risks and learning from failure, which is consistent with the Third Way of DevOps, continuous learning and experimentation.

[00:17:42.838] We also implemented practices like making work visible, using the pull versus the push method of assigning work, and daily stand-ups. We used the planner functionality within Teams to make work visible. Each task for the audit was represented on a task card, and information like the accountable party and the status were always available for the audit team to see and act upon. I didn't have to go to the project manager like I would in other engagements to ask them for a status update. I could just quickly view the planner board and see where everything stood.

[00:18:15.978] In the traditional audit approach, we use the push method of assigning work, where the project manager assigns work out to the staff. Staff number one, you take these four controls. Staff number two, you test these five controls. In this audit, we used the pull method of assigning work, so the staff self-assigned their work, and this facilitated a much better matching between capacity and assignments.

[00:18:39.207] So I'm on vacation next week. I don't have a whole lot of capacity, so I'm going to assign myself work that doesn't require a lot of effort or a big time commitment. It also facilitated a better matching between those assignments and the team's interests and development goals.

[00:18:58.382] Another way we implemented these newer ways of working was to attend daily stand-ups as a collective team. This was Tod's team's idea, and it was one of the improvement opportunities that we identified in the sprint retrospectives. It was definitely met with some apprehension from the auditors at first, but it was absolutely an incredible success.

[00:19:20.982] The time that we spent sending a request and following up on it while Tod and his team fulfilled it was drastically reduced using this new way of working and using these stand-ups. In most instances, using our old way of working, we would send a question or a request for follow-up documentation, and we'd send it through email to our point of contact, who would try to figure out who actually was the right contact, try to understand what it was we were asking for, pick something, send it back to us, probably not the right thing. So it would take us days or weeks to get what we requested. Using these stand-ups, we often received whatever we were asking for during that stand-up or by the end of that same day. It was awesome.

Tod Bickley

[00:20:06.662] Although we were in these really great new processes, there were several challenges that we faced going through this. First, and I think kind of the biggest one was, although we used the sprint approach very effectively, and we were able to understand what the issues were per sprint based on the scope that we broke up, we sort of lost sight of how it all came together in totality. When we got the final audit document with everything we had discussed and accepted per sprint, it really looked a whole lot bigger than we thought it was going to be.

[00:20:40.902] One piece of advice I would say, or the thing that we're going to do different next time, is we're going to make sure that we're tracking and understanding and aligned on what those issues look like, not only per sprint, but as they continue to grow through the multiple sprints.

[00:20:59.081] The second one, which is honestly just a little bit of a circumstance of our environment, is we didn't do a whole lot of upfront planning. At the time, the team was really going with its agile journey. The auditors came in, and we said, "You're going to have to play in our processes." We didn't have a lot of time to do a lot of upfront planning to see how that would actually work. So next time, I think what we're going to do is we're going to take some time, a few hours, maybe a half a day, and really figure out and plan out how we're going to execute our sprints and the scope of these sprints, so we can be a little bit more organized delivering.

[00:21:34.562] The third piece was, the auditors didn't have a ton of knowledge of the agile processes. Obviously, with The DevOps Handbook and great books like The Phoenix Project, they understood concepts, but they had never worked inside those DevOps practices before. Even though the auditors are now more engaged and knowledgeable of agile, as well as the IAM teams, I think we are going to take a couple hours before we kick off the next audit, and we're just going to level set on terminology, on approaches, so we're all using the same language and sort of working off the same sheet of paper going forward.

Clarissa Lucas

[00:22:15.602] Tod and his team weren't the only ones who had some challenges as we tried this new way of working. The first was a fear that we would violate our professional auditing standards. We're switching to this new way of working, and can we still do this and comply with the standards that we're held to as a profession? And the answer is absolutely yes.

[00:22:37.422] One of the most common questions that the team had, and the biggest fear I think that we had specific to staying in compliance with our auditing standards, was the independence and objectivity that's so important to internal audit. How can we maintain that level of objectivity and maintain our independence when we're working so closely with our clients? The key to that is really just maintaining decision rights with the audit team.

[00:23:04.022] While we're collaborating with Tod and his team to identify which risks and controls do we want to include in scope and do we want to spend most of our time in, we're leveraging the knowledge that they're providing us, and we're making those key decisions ourselves. A lot of great input. I truly believe that we made better decisions, more well-informed decisions. It's just Tod didn't have the decision, and his team didn't have the decision on what we would audit and what we would exclude from scope. That really still stayed with my team and myself. That's how we really maintained that independence, which was probably the biggest hurdle when it came to that fear of violating the auditing standards.

[00:23:46.162] We also, as Tod mentioned, lacked experience with agile and DevOps practices, and we had significant cultural and procedural changes that we needed to support this new way of working. That fear on violating auditing standards, the lack of experience with agile and DevOps practices, really didn't help either.

[00:24:10.982] But as we talked through, we made it through these challenges. We learned a lot throughout the way, and we really had a lot of benefits. Tod and I walked through what it means to audit with agility and how you might do that. What are some concepts that you want to include when you're audited next?

[00:24:30.782] I think it's really important now to shift to the why. Why would you want to partner so closely with your auditors and invest all of that time? Here are some of the benefits that our team, so again, not just my team, not just Tod's team, but that collective team working on this engagement, enjoyed: greater collaboration and engagement. We focused on areas of the greatest value and highest priority to the organization. We successfully adapted to change. We had much greater buy-in, more timely communication of results, and my personal favorite is a reduction in the amount of time that was wasted during the audit. I think it's one of Tod's favorites too.

[00:25:14.158] Here you can see some measurable results that we achieved during the audit as compared to the last time we were in this space in 2019. The length of the engagement went down, was reduced by 10.5%. The amount of coverage that we got increased by 77%. The number of days from when we identified an issue to when we opened it and got it in Tod's hands so he could do something with it was reduced by 48%.

[00:25:41.518] The percentage of issues with progress made by report issuance had a significant increase. This is where in 2019 we followed that waterfall approach. We finished all of our testing before we delivered any results to Tod. The story there to our key stakeholders, so our audit committee and executive leadership, was: we did some work in this space and here are some gaps that Tod and his team need to start working on.

[00:26:07.318] Fast-forward to 2021, implementing agility into our work. The story there was because we delivered iteratively, Tod and his team got those findings and got those control gaps into their hands much earlier and made progress on those by the time we got to that final report.

[00:26:25.078] There were instances where they already had a plan in place. They knew how they were going to address these gaps, or they made progress on, "We're working on this plan, and it's not just a plan, it's things that we're doing to mitigate this gap." Or my favorite was, "Yes, it was a gap. We've already put a plan in place. We've already mitigated it. And then Clarissa and her team have validated that that indeed is no longer a gap." So completely different story to the audit committee and to executive leadership.

[00:26:57.498] From 2019, it was, "Here's a bunch of stuff that Tod and his team have to do." In 2021, it was, "Here are the results from the work that we did in this space, and look at all this progress that Tod and his team have made so far." Much better story.

[00:27:13.378] The last measurable benefit is the client survey results. I mentioned before that we send out a survey at the end of the audit to get feedback from Tod and his team, and part of that is an overall satisfaction rating. That improved by two rating levels, which is awesome.

[00:27:32.198] So what does this all mean? It means that we provided more assurance, and results were communicated and addressed sooner than in prior years, all while spending less calendar time auditing Tod's team. Based on those survey results, they were pretty happy about it.

[00:27:47.078] As a matter of fact, these are direct quotes from Tod and his team taken from our client surveys: things like exceptional, great time, very positive, worked hard, and enjoyable to work with. I want to put these on my fridge and read them every day because this is exactly what I strive for and what my team strives for in working with our clients.

[00:28:07.958] While we didn't have a real-time metric to measure every benefit that we experienced, like that greater team collaboration and engagement, we still experienced them. Our daily stand-ups were a lot of fun. We built a lot of rapport with each other, and when the audit was over, we missed working with each other. I missed working with Tod so much that I was just dying to work with him again on this presentation and was thrilled when he agreed to do this with me.

[00:28:34.618] The engagement and collaboration on the audit led to a number of lasting professional relationships that are still going strong today, about six months after the engagement was completed.

Tod Bickley

[00:28:46.818] So how can you get there? When your auditors come to you and you want to bring them into your DevOps and your agile practices, just offer to coach them through that learning curve. I think what you're going to find is that your auditors are going to want to learn how to work with you just as well as you want to be able to work with them to make these things as easy as possible.

[00:29:08.918] Second is demonstrate how to run effective stand-ups. We know how important an effective stand-up is to the agile and DevOps experience. Help them understand your Kanban boards, help them understand your Jira boards, help them understand how you're doing your flow of work and how work flows through your systems so they know how to better take the work that they're going to give to you as demand and be able to put it and utilize those systems.

[00:29:31.658] Then keep an open mind. Always think about how things may work now to how things may work better when you're bringing auditors into the practice. Encourage collaboration with your team members. Your auditors are really there to help protect you from the things that are going on. They're not there to find deficiencies in your systems to make you look bad. They're there to find issues in your systems to help you fix them so we all look good.

[00:29:58.658] Really encourage your teams to buy in. That's really the most important, because once you've got the team's buy-in, once you've got the people to be able to do it and work effectively, I think you're really going to see success.

Clarissa Lucas

[00:30:12.078] Wrapping up, it was really a great experience. I would even say we're even looking forward to the next one. A couple things that I would ask this group is, as you are going through agile and DevOps experiences, how are you seeing those evolve in your infrastructure areas? Do you see people adopting product practices? How are you bringing external teams like your auditors into your agile practices to make sure they're adopting those things, and are you seeing the same sort of advantages and sort of upticks in the flow of work that we've seen over the last year?

[00:30:47.838] I've also got a request of the group as well. For those of you who in the beginning you typed into the chat that you had great experiences with your auditors, I'd love to know what are some of those things that you've done or that your auditors have done to help make that a better relationship. For those of you who haven't typed in great, enjoyable experience, love hanging out with my auditor friends, I'd love to know what else is driving some of those challenging relationships.

[00:31:17.838] Because this is one way that my team and Tod's team have collaborated together to create a better experience, but I know there's more ways, and I would love to learn what are some of those other things that you're challenged with with your auditors. On behalf of both Tod and myself, thank you to Gene Kim and the selection committee for giving us a stage here. We love sharing our story and learning from each of you. Thank you.

Tod Bickley

[00:31:43.888] Thank you.