Host Intro (Gene Kim)

Yesterday we heard from Mike Carr, CTO of Vanguard, who described how excited I was to attend their three-day internal technology conference. There were so many great talks about amazing people doing amazing things on behalf of their fellow crew members and their clients.

There are three other Vanguard talks today and tomorrow, including the next two speakers. Devlin McConnell is Senior Manager of Emerging Technology, and I was so excited that his co-presenter is Matt Butler, a director in Internal Audit. And yes — an internal auditor. As many of you know, I was a dues-paying member of the Institute of Internal Auditors for eight years. They're going to be talking about some of the amazing distributed experimentation that they are doing across the Vanguard enterprise, exploring not just what's possible, but what is actually useful and valuable today. He'll also share one of the reasons why I was so excited to be at that conference earlier this year. So here's Devlin and Matt.

Devlin McConnell

Hello everybody. My name is Devlin McConnell. I'm joined on stage by Matt Butler. We work at Vanguard. I work on a team called the Emerging Technology Research team, and it's our job to help our colleagues around the enterprise capitalize on emerging technologies. My role specifically is generative AI. Before we get into AI, I'd like to introduce Vanguard really briefly.

Vanguard's core purpose is to take a stand for all investors, to treat them fairly, and to give them the best chance for investment success. We start with our mission because we're really a mission-first company. We are structured as a mutual, so there's no shareholders and investors — there's just investors. If you invest with Vanguard, you own Vanguard. So it gives all investors, in our judgment, the best chance for investment success, and we think it gives us a competitive advantage in the market.

We're currently the second largest asset manager with $9.5 trillion in assets under management. We've got 20,000 crew — we call employees "crew" — that serve 50 million investors all across the globe. Within IT specifically, we've got 10,000 folks trying to make Vanguard the most up-to-date and advanced technology firm we can. And generative AI is a big part of that.

For many of us, GenAI has really entered our lives maybe around the beginning of 2023, right as ChatGPT got released. Ever since, our most trusted advisors, confidants, consultants have told us it's going to change the world. Trillions of dollars of additional assets might come to the economy. It's more than just OpenAI and Microsoft and Google capitalizing on this — our firm specifically can see huge efficiency gains. And if we get even more granular, our employees want to use this tool. They see value in it, they want to use it.

Well, you can imagine at Vanguard — large asset manager, some top financial minds, top economists in the world — hearing about a new technology that could add trillions to the economy, they might want to do a little digging themselves. So Dr. Joe Davis and his team really dug into this idea and published our Megatrends research, which I highly encourage everyone to check out, that analyzes AI and other advanced technologies and its impact on the economy. Here's just a little excerpt from Joe and his research.

[Video — Dr. Joe Davis voiceover: "Could AI be as transformative as electricity? Can it change how we live, health, happiness? And if we use technology and if we use data, innovation can be predicted. You just have to know where to look. This is a future that very few are actually talking about. Not all technologies change the world. Think of electricity and social media. Both are ubiquitous, they're all around the world, but only one changed lives and propelled the global economy forward. Why is that? It turns out big technological advances — what economists call general purpose technologies — they have two distinct dimensions. They're innovative and they're transformative. Innovative technologies change how we work. You can think of the assembly line, or you can think of the computer. Assembly line, having some automation for us to run faster; computers to generate new insights. What I call power tools — they're compliments to the work we do. We almost take for granted how important computers are in today's society. But not all innovative technology is transformative. Transformation is different because it changes how we live. Electricity was transformative because it improved basic human needs. More importantly, electricity was transformative because it enabled knock-on effects and other inventions that without electricity simply would not have been possible. So which one will AI be? Innovative, transformative — and could AI be both?"]

So we have our top economists, our top consultants, everybody telling us generative AI is going to change the world. But today, August 2024, realizing business value with this technology can be challenging. For one, we face a myriad of existential risks. This technology has some existential risks that have yet to be solved — from hallucinations, biases, where your data goes, or what data has been used to train the model. And if you're in a highly regulated industry such as Vanguard, what sort of audit trails does a black-box model truly give you? A lot of questions that we need to answer if we're going to adopt generative AI today.

It's hard. So for us at the Emerging Technology Research team, we partner with often skeptical partners around the business, trying to convince folks: "Hey, we believe this technology is going to change the world. We don't want to be left behind. We want to go along for the ride. But we face all this uncertainty — a rapid environment, a ton of choices, and a whole lot of hype."

So the way that we approach these conversations is by pursuing this technology — these experiments — through this idea of small, fast bets. Similar to a poker player: you're okay throwing a few chips here and there for the whole purpose of gathering knowledge about all the unknowns that you face on the table. And when you're ready to push your chips into the table and transform the business, we feel a little bit more confident based on all those small bets that we've done throughout.

So where do we start with generative AI? Well, we've got these existential risks. Are there areas that we can put appropriate technological and strategic guardrails in place to mitigate those risks or flip risks into features? We currently bucket our use cases in these four categories: content creation, knowledge management — we'll go over today — and then virtual assistants (having ChatGPT or Claude on your computer). And then lastly, code generation use cases.

So for content generation or content creation, we'll often hear about use cases in a variety of different ways. The whole idea is these models can create novel content from scratch — where can we apply that? Oftentimes the easiest way to do that is to go to experts in these areas, such as marketing experts, and ask them a simple question: what's the most annoying thing about your job? What is the one thing where if we can apply this model — that maybe has some risk of lying to you or saying something that's completely wrong — can we flip that and turn that risk from a hallucination into maybe something more creative?

One of the tools that we really, really value and really enjoy is called Writer. It helps these marketers take, for example, a podcast or a webinar — something that subject-matter experts have put a lot of time and energy into — and helps them transform that content into maybe an email marketing campaign. Now, an email marketing campaign is where a marketer will shine, and that's really important work. But maybe the transformation of all that content into something more manageable for them to work with — maybe that's the most annoying part about their job. Can we automate that? Even if it comes out with some bias such as focusing on the wrong thing, or hallucinations saying something that might be not expected, we can mitigate that risk because we have this expert marketer who has also done the work — just help them along.

The other use case, or I should say umbrella of use cases, that we look at is called knowledge management. Oftentimes when we say this, the first idea that people have is, "Oh, a chatbot." But there's a lot more to that. There's a lot of ways to be a lot more targeted and add more business value. I'll pass it over to Matt Butler to give an example of a proprietary tool we've built in internal audit.

Matt Butler

All right, so I'm Matt Butler. I lead our analytics and automation function within Internal Audit SOX at Vanguard. I'm going to talk about our first GenAI use case for audit, which is AuditGPT. It aims to create efficiency and time savings in the audit research and planning process. But first I thought it might be helpful to walk a little bit through audit and what we do.

In your travels, you might have heard of the Three Lines of Defense model, which is a framework that was put out by the Institute of Internal Auditors a few years ago for structuring governance and risk management in organizations. The first line — it's business operations. It's probably a lot of you — you guys own and manage the risk in the day-to-day. The second line is our risk management and our compliance functions. They provide oversight and support and effective challenge to the business. And then the third line of defense is internal audit. We provide independent and objective assurance to the organization to ensure that we can meet the goals of the organization. So we focus on the things that could go wrong, and we determine if we have the controls in place to limit those things from materializing.

I like to think of audit as kind of like having doctors within an organization, diagnosing where things are running smoothly and providing guidance, and at times prescribing treatment on areas that need to improve.

So what do we audit? I'm sure a lot of you have wondered that when audit comes and shows up at your door. We have what we call an Audit Universe. We take an organization and we break it into component parts — a catalog of areas where there is financial or operational or compliance risk. That gives us our auditable entities. And under those entities are a variety of functions and systems and processes that inherently have risk.

So out of all of these possibilities, how do we determine exactly what to audit, and whose door to knock on? First thing — we use human brains, subject-matter expertise. Our auditors are experts in the domains that they support. They're continuously having conversations, not just with the business but also with our lines-of-defense partners, understanding the landscape and where things are changing.

But probably the most important part of the process is information. Data is key to determining what to audit. A lot of our information is text-based. It's unstructured. Being able to see internally identified issues, gaps that have been documented in our control environment — not just by audit functions but by the business itself or by our risk partners — seeing risk events, things that have actually gone wrong already — what can we learn from that? Then external information is critical to this process too. As the regulatory environment changes, the things that we need to audit are going to change as well. We also keep an eye on external events. If something happens at another financial services firm, it's really important for Vanguard to know about that so that we can ensure we have the controls in place so that can't happen to us.

So we take all of that information, and every year we do a risk assessment of our entire audit universe, and we build out our audit plan for the year. Then for those individual audits, we scope out what they're going to cover — areas with the most risk, the highest likelihood that something could go wrong, the biggest impact if something were to go wrong, maybe an organization where the structure has changed recently and we want to make sure that things are still running as intended, maybe an area where new technology and new risks have appeared, maybe places that are starting to use GenAI for some of their processes.

So this process of determining what we audit is not easy, and it takes a lot of time — finding the signals within the noise, collecting the necessary information so that we can ensure that we're auditing the right things and providing that independent and objective assurance to the organization so that we can make sure that we're going to meet our goals.

And this is where our use case comes in. What if we could bring together the right information faster, bring the relevant information to the fingertips of our auditors, connect some of the unconnected? That's what AuditGPT aims to do. So what will this look like?

On the screen you'll see a mockup — real data here — of the interactive information shopping experience that we're creating with AuditGPT. That's going to help auditors bring together the relevant information to assess risk and to scope individual audits. So for any Seinfeld fans out there — imagine we're going to audit Vandelay Industries, focusing on their importing/exporting business specifically.

Auditors are going to be able to come in, and leveraging semantic search technology, they're going to be able to see if there are any recent events or issues that have been documented that might be relevant for that. When they find those in the tool, they're going to add it to their shopping cart. They're going to look at historical audits — so you can see on the screen, in 2022 we went and did an audit in this space. You're going to see GenAI summaries of the risks and the issues that were identified in that audit. This would be helpful for the next audit they're going to do — they're going to go ahead and add that to the shopping cart.

Once they add all of the relevant information to their shopping cart experience, it's going to generate the audit research report, which is going to give you GenAI summaries of all the key information that you need for this audit, aggregated together. This is going to be a massive efficiency play — so much faster than going into every individual system and trying to find those signals within the noise.

All the data's going to be sourced, so our humans in the loop can validate and dive deeper where they need to. I know it's really small on the screen, but you can see that we have "event 1234" down there at the bottom that details significant delays in our latex shipments for Vandelay Industries. So an auditor sees that short summary and they say, "I need to know more about that for the audit that I'm going to conduct now." They go to that source system — they know event 1234 is what they need to search — and they dive deeper into that, helping them target the right research.

So we're in an early piloting phase right now. We've rolled this out to a few auditors a couple of weeks ago, and initial feedback has been very positive. People are seeing the possibilities to save time here.

You can imagine, we're getting feedback about the possibilities, but also people are skeptical — skeptical about what these tools can provide. I think part of the battle here is converting hearts and minds of people that typically play defense in this space, to think about the offensive possibilities that we can do with generative AI. So that's the goal of what we're looking to do.

From a help standpoint — I imagine we're going to hear this a lot over the course of the day — we want to hear about other people's experiences. We're figuring things out. We're learning in this space. We want to hear about the best practices that you've run into. We want to hear about some of the pitfalls and the things that haven't gone well. We want to continue this conversation — not just over the next couple of days, but let's build relationships and continue it over time so that we can continue to shape the future together.

So thank you. That'll wrap it up.