Host Intro (Gene Kim)

Hey, good afternoon. How were the morning sessions for the Ultimate Gen AI Learning Day? Awesome.

Okay, so before we broke for lunch, Stephen Magill gave us a pretty sobering talk on the potential risks of software and data supply chain poisoning attacks. The next speaker — this is also another breaker talk, so you got the builders and the breakers — George Proorocu. He's the IT OPS Chapter Lead for Cybersecurity & Fraud from ING Bank.

So George will talk about how adversaries are using deepfakes and other techniques to impersonate executives and circumvent critical controls in enterprises. He will demo these attacks and describe what ING is looking into, including training new controls. I suspect you'll be showing this video to your colleagues after he sees this talk.

And I keep thinking about something he told me during the speaker reception a couple of days ago. He said: we will probably be making more changes to our approval processes than we have in the last 50 years. So with that, here's George.

George Proorocu

Thank you. It's always a pleasure to be here on stage. Hello everyone, and welcome to the world of deepfakes.

My name is George, and in the past several years I've worked mostly within fraud and cybersecurity. Today I'm here to give you a bit of a different talk. So far we saw how AI is used for the greater good, but I'm going to show you now how AI is used by bad people to do evil things.

I prepared three examples for you, because I want you to take something back home with you from this talk. The first two examples will be focused on audio deepfakes, and we're going to see also a demo there. And after, we're going to see some video deepfakes and the C-level impersonation with a live demo to see how easy it is nowadays to create high quality deepfakes.

01Scenario 1: The accident scam, 2024 edition (audio deepfake of a son calling his mother)

This takes us to the first example, which is an audio deepfake. Maybe some of you heard, some years ago, about the accident scam, how it worked. Back in the days, someone would call your parents saying that John was in an accident — quickly, we need $1,000 from you to pay for a blood transfusion so that we save his life. Fortunately we're close to your home, we're coming to your house, please give us this money, and then we will try to save his life by paying it.

Fast forward to 2024: the scammers are doing this, but then they will call the parents with the voice of John.

Let's see a bit how a scam like this would look like. First of all, they're always taking advantage of a relationship between two people. This can be either family members, friends, colleagues, and they will try to exploit it. In our case, it's a son–mother relationship.

First they check on the son side to get the phone number, then initiate a few phone calls with the person to build the audio model. This only requires like 30–60 seconds. So it can be just some random calls just to get the person talking. And then they'll initialize a call to the mother saying something like:

> Mom, I'm — I'm — quickly, I'm calling you from this number. I hit someone with the car and they want to call the police, mother, but they told me that if I pay quickly $1,000 then they will just leave it like this and I will not go to prison. Mother, I've sent you a link, a quick link for Revolut. Please quickly pay it, and then I will get away with it.

Imagine that your parents will receive a call like that with your voice — what are they going to do? And nowadays we have all these fast payment solutions that will enable them to pay just in a few clicks. And maybe after you paid, you already realized that, yeah, it was maybe a scam.

02Scenario 2: Manager-to-engineer audio deepfake (give me your password, approve the MFA)

This takes us to the second example, where we see this scam moving towards enterprise attacks, to be used in a more complex attack. We're going to see an example where they try to exploit the relationship between manager and engineer, and trying to get the engineer to give the password and MFA access to the scammers in order to get inside and then do whatever they had planned for the attack.

In this case, they might not even need to get the number of the manager or any phone calls — because many enterprises already have a lot of videos with people online. Maybe the manager attended some podcasts, maybe even talking at some conferences, or any kind of public audio material that they can find so that they can build their model.

Now we're going to hear how a deepfake like this would sound. First you're going to hear my voice. I'm going to make some pauses here and there so that when you're going to hear the deepfake voice of Olivia, you're going to recognize the patterns from my original voice.

> Mihai, this is super urgent. We just been hacked, so this is super serious. My account has been compromised. I'm calling you from my wife's number because my phone has been compromised as well. The hackers are transferring money as we speak. Quickly, give me your password, and when I will tell you, please approve in Microsoft Authenticator, because I really need to quickly go and disable the servers that I have access to. So like that we can stop the attack.

[Original voice plays at ~5:13.]

And now we applied the model of Olivia's voice, and you're going to hear it. [Cloned voice plays at ~5:46 — same script, in the manager's voice.]

And this can be easily integrated using a jailbreak device into a live call, and then further calling Mihai, the sys admin, to try to get this kind of access.

03Real-world precedent: the Ferrari attempt

We see a lot of new cases on the corporate side, because of course the hackers know that the companies have money and they're going to pay if needed. There is a recent case — maybe some of you saw it in the news — where they contacted an executive from Ferrari trying to impersonate the CEO of Ferrari. They contacted via WhatsApp, sent some messages, then sent also some audio messages. Apparently the messages were that good that they were even simulating the accent that the CEO has.

Fortunately for Ferrari, the executive found it a bit suspicious, let's say. So he asked the scammer a question that only the CEO would know the answer to. Of course, the scammer didn't know how to answer, and the attack stopped there. So they don't really know what they really wanted. But because the executive was quite smart, they managed to close it — to end it before it started.

But probably in the future, we're going to see more and more attacks like this, focusing on corporations.

04Scenario 3: Video deepfake of the CEO — a $2 million wire transfer

This takes us to scenario number three. This one is a bit more complex attack. It's a case where the hackers would like to convince a senior accountant to transfer $2 million into their account. A senior accountant from a branch of a big corporation. And they targeted this specific person because, first, she has access to make this kind of transfers. And second, they saw on LinkedIn that she posted a picture with the CEO from an event some weeks ago. So she knows the CEO, she knows how the CEO is talking, how he's looking. And they'll try to exploit this by initiating a video call.

Imagine that you are Julia, and you receive a video call from the CEO saying something like: 'Julia, can you please go to a meeting room? This is very urgent. Please go to a meeting room to discuss this, because it's confidential.' And then let's say we go to the meeting room, and there the fake CEO starts like:

> Julia, you know, we met a few weeks ago at this event. I saw that you are a trustworthy person, and I really see that you have a big future in our corporation, but now we really need your help. You know, we are now under investigation. The police is here. They're in my office with the auditors. Apparently our boss, John, stole a lot of millions from us. And we need to quickly take action now in order to try to fix this and limit the damage that is done to the company. And can you please just go to your desk, bring your laptop — don't talk with anyone. Because we don't know who's involved in this. Since we know that John your boss was, but we don't know the other people, so don't say anything to anyone. Just bring your laptop here.

Let's say Julia comes back with the laptop. And the CEO continues:

> Okay, now I'm going to send you an email with all these details, so that you have the proof that I told you this. But please can you check in the spam, because the police is looking into our email servers. So we saw that some emails are going into junk or spam.

And they'll try to send a spoofed email to Julia, tell her the details, and then try to convince her to actually make the transfer into their account that they control.

Here it's an example that you can see on the screen, with a video that is generated just by using one image of myself, and my colleague Mihai — he's playing the role of the fraudster. And this can be integrated easily into a live feed and initiate a video call through a jailbreak device.

Imagine that you are in this situation. You receive a call like that. You are Julia, you're not aware that these kind of scams are lurking around. What are you going to do? And I think there is no right or wrong answer to this.

05Live demo on a MacBook

But what I would like to show you is how easy it is actually to generate a high quality deepfake. I have here an open source software. I'm going to select the picture of the CEO, then a video of the scammer — I'm playing the role of the scammer here. And then let's see how fast it is on my MacBook on battery, how fast it is to generate a decent level of deepfake.

As you can see, this is the initial video. Normally it takes a few seconds and it's done. And this is the output that we have. It's quite a decent — of course the quality can be increased — and this can be integrated into a live feed, but then we require more hardware.

06How it used to be: the head-rotation tell

Going back to our presentation, I want to show you also how this kind of deepfakes looked a while ago. Because if I would have this talk a few years ago, I might tell you that if you're in a call like this, just make them rotate their head — because if it glitches in any way, like how you see on the screen, then it's for sure a deepfake, and you are basically sure that it is a deepfake.

But then fast forward to what we have today, as you saw in the demo that we had — well, this would not really apply. The technology is advancing quite fast, and I think it's quite important for us to understand also what can be done with the level of technology that we have today.

07What can we do today? Awareness, skepticism, keywords, policies

This takes us to the step where we discuss a bit, what can we do against deepfakes.

First one is awareness. So I'm telling you, I'm showing you some examples of the scams and attacks that are currently happening. Then you go and you discuss this with your network and so on. And hopefully if this would happen to someone that you discussed with, they'll have in the back of their head, 'but he told me about this — maybe it's a scam, so I will be a bit skeptical.'

Be a little bit more skeptical. If someone is calling you asking for any kind of information — personal finance, financial information, anything related to transfers, to make a money transfer to pay something — always double check. And if you're really not sure, just double check with another person from your circle.

The third one is keywords. This might help if you set up some keywords, for example, with your family. However, if you don't repeat this frequently, then the scammers — since they're very good at social engineering — they can turn it around.

So let's go back to the accident example. Let's say the mother will say: 'Okay, we have the keyword, what is the keyword?' And then the fake John can simply reply: 'Mom, this is what you're thinking about now? You don't understand that I hit someone. You don't understand how serious this is. Just pay the money and then we'll discuss. I don't really remember this keyword that we put. Whatever — just pay the money, and I don't want to go to prison.' So they can easily just turn it around, and then you'll be a bit like, 'Okay, maybe John doesn't really remember the keyword.' So you really need to practice it from time to time just to make sure that all the family members know it.

Last but not least, policies. We see a lot of corporations putting in place procedures and policies in order to avoid cases like the one that we had with Julia, the accountant. I think in the upcoming months and years we're going to see a more standardized format for these policies that are going to be put in place.

08The future: platform-level detection and on-device hints

This takes us to a question that I often get: how do I see the future when it comes to deepfakes? I split this in two parts.

The first one is at platform level. I'm talking here about the social media, because there you find a lot of deepfakes, and there are already things in place to detect some kind of deepfakes to a certain level — the popular ones. But I think in the future we're going to have maybe one provider that is actually going to be very accurate and doesn't give a lot of error rates for deepfakes, and that is probably going to be used by all the platforms.

And then at local level — so on your smartphone — I think we're going to have in the upcoming months and years either integrated in the operating system or separate applications that are going to give you, of course with a high error rate (because you don't have the processing power of what the platform would have), but at least give you a hint if an audio call or a video call might be deepfake. And like that you are at least warned: 'Okay, be careful. This might be an audio deepfake and they might try to steal some money from you.'

09Help I'm looking for

What help I'm looking for: I would really be happy if you go back home and you share this with the people that you know, so that they are aware of what kind of deepfakes they might encounter in their personal life and professional life.

With this, I would like to thank you. You have my social media — X and Twitter — on the screen. I'll be here till the end of the conference. If you have any questions or if you want to discuss anything, please feel free. Thank you.