Daniel Pries

All right, everybody. Thank you for taking the time to meet with me today. To be honest, this is a unique privilege for me. When I got into speaking, I just got into this to meet my heroes, and I am knocking a number of those people off my bucket list today. I am humbled that you would take the time, with all the great talks, to speak with me today.

We are going to go over this talk, which is called "How You Can Trick Your Company Into Hiring ChatGPT." I have given this talk about eight times. Normally it is about 40 minutes, and they told me I have 20, so we are going to go a little fast. If you have ever looked at one of your coworkers and said, "How did you even get a job here?" you are definitely in the right place.

The organization I work for is M&T Bank, a super-regional bank based out of Buffalo, New York. You probably have never heard of us unless you are watching pro football, in which case you have probably heard of the stadium where the Baltimore Ravens play. We have about 2,000 engineers. I oversee DevX and tooling inside engineering excellence. Our goal is to make engineering easy to succeed in and difficult to fail. We work in a heavily regulated environment, so this is difficult. There are a lot of constraints; it is our North Star, even if we will never eliminate every constraint.

The space I oversee includes strategic portfolio management, work-item tracking, our systems development lifecycle, DevOps, and tooling. I am a full-stack engineer with about 20-plus years, closer to 25, working across backend, frontend, and digital. I am a manager of managers who loves to lead from the front. I believe the best of us help engineers succeed by providing a masterclass in engineering when they need it. I love writing code with our teams and watching the light bulb turn on in people's heads.

By the end of the talk, I want you to have a working understanding of how AI is changing the hiring landscape, particularly in technology, and some techniques you can use to battle-harden people-centric processes such as talent acquisition using AI. Talent acquisition, like any people-centric process, is non-deterministic, and AI is the perfect use case for these activities, not just for technology but for organizational processes and procedures.

Where I started was seeing generative AI as a potential threat vector when it was first coming out. Generative AI is a great tool, but tools only amplify behaviors. We can use generative AI to create the best products, but people who want to use the products poorly for self-interest are going to. I wanted to protect my team and my company. We have about 2,000 engineers in the space, and you spend about 1,800 hours a year with these folks. You do not want to bring an anchor into the team. I wanted to address the problem before it became bigger.

To understand the backdrop, I was interviewing a gentleman in the winter of 2023. He had a very bushy mustache, an "I speak for the servers" kind of mustache, fully over his mouth. We were going through a technical screen, where we cover dimensions and the questions get harder. On video, it looked like he was mumbling, but his enunciation was super clear. Something was off.

We were talking about fundamental object-oriented concepts, like SOLID: single responsibility, open-closed principle, substitution principle, interface segregation, and dependency inversion. A lot of people answer dependency injection instead of dependency inversion because frameworks have solved for this today. I asked the difference between dependency injection and dependency inversion, and the audio cut out.

I could see his mouth moving, so I said, "Your mouth is moving, but I do not hear anything coming through the speaker." He started typing that he would be back in a couple minutes and might have audio issues. But as he typed, I could hear the key presses in his microphone. The person I was talking to in the video was not the same person providing the audio. Somebody else was providing the audio.

I talked to our talent sourcers and said I thought I was being manipulated and asked whether to continue. They said yes, continue and see how they answer. We ultimately did not hire this candidate because they were committing fraud. It was not the first time I experienced it, and it was not the last.

There are a number of kinds of fraud. Gartner assessments in 2024 said 29% of candidates are using generative AI to answer assessments, 6% are committing interview fraud with somebody else posing as them or them posing as somebody else, and Gartner projects that by 2028 one in four candidate profiles worldwide will be fake. ResumeLab reported that seven in ten workers confessed they lied on resumes, with an additional 37% admitting they lied frequently. It got worse for people with higher degrees: bachelor's and master's degrees were in the 60% range, and PhDs were stretching their skill sets by about 70%. Most people are claiming mastery in skills where they have basic knowledge, starting as maybe a junior engineer and saying on the resume that they are an architect.

The other side is the threat vector. The Department of Justice reported that more than 300 U.S. firms inadvertently hired imposters with ties to foreign threat actors in IT, including Fortune 500 companies. They are using KVMs and laptop farms. The KVMs are IP-controlled and allow traffic from other countries so people can act like they are actually on the corporate device. The reality is that if you are not cheating, it seems you are not trying in the modern environment.

Application fraud happens at the gate, when people apply for a first role and misrepresent qualifications or identities. Common tactics include falsifying credentials and personation, misrepresenting experience and skills, and exploiting technological loopholes. Interview fraud happens after the application is in place and the person secures an interview. It includes falsified qualifications and experiences, impersonation and proxy interviews, manipulation of interview responses, cheating on assessments, and deepfake technologies. There are companies that provide real-time AI manipulation of the hiring process; I will not promote them because it is like fraud as a service, though the irony is that they have an ethics page.

My experience started at M&T in summer 2020, when the world shut down. From February 2019 to February 2020 there was a 47% reduction in job applications across all industries, before COVID shutdowns in March 2020. The market was shrinking. We also had the Great Resignation, social unrest, and Big Tech on a hiring spree, doubling down on AI and remote-work technology. They were drinking our milkshake. We had people trying to fraud the system, a shrinking talent market, Big Tech competing hard, and we thought COVID would be short.

I was told we had to bring people to where I live. I live near Western New York, where it is not abnormal to get seven feet of snow. Western New York does not get a lot of snow compared with the rest of the country, but we get it all in about a one-mile band. We dig ourselves out, but it makes hiring a real challenge.

Then we had automation in talent acquisition, reducing toil. We wanted to automate data-centric, toil-based activities such as resume screening and skills assessment, freeing people from pull activities like filling out applications, entering data into systems, and CRM work. We wanted more time in conversation and a faster hiring process, because we had a limited talent pool and did not want to hold candidates longer than needed.

Since this exercise began, AI has improved talent work: more enhanced and specific job descriptions, better alignment of talent to fit-for-purpose roles, better talent pipeline efficiency, and better branding and strategy for specific audiences such as fresh graduates or senior people. But there might be a problem: people are frauding the system, the global talent market is more difficult, and automation applied to the process could be exploitable.

Our hiring process includes talent acquisition and recruitment, a technical screen, automated testing, a manager meet-and-greet for red flags and selling Buffalo and seven feet of snow, and a competency-based interview. For time, I am not talking about the technical coding part, because AI can write code for us.

I had a challenge with the automated testing component. In spring 2022, I voiced concerns about how we were using LeetCode-style tooling. We used it as a way to ask whether someone could solve a puzzle in any language using for loops. I did not think that was the right way to solve hiring practices, because it did not talk about clean code, maintainability, or language features.

In summer 2022, I joined a panel to review a new set of questions and make sure they were fit for purpose. At the end of the call, I asked whether anyone had tried running them through Copilot. At that time, Copilot was just Codex, not the broader set of copilots today. It was crickets. I started performing the experiment on my own. No one asked me to; it was a little chaotic good. I wanted to see if the process was exploitable.

In November 2022, ChatGPT 3.5 was released to the general public. I decided not only to test programming puzzles, but also to run the technical screen through the process and see what ChatGPT produced. Nobody was listening; people thought the tools and processes were great and not exploitable.

At a conference in Sandusky, Ohio, in winter, my buddy Tom convinced me over a beer to create a talk about how I tricked my company into hiring ChatGPT, which trends poorly on LinkedIn and is why the title changed. The plan was to create a persona, use AI to render bio pictures, run coding problems, go through technical-screen prompts, perform the competency-based interview, and present it to talent acquisition. We first thought about putting fake AI pictures on LinkedIn, then realized fake profiles were a bad idea. We also realized taking it straight to HR as an attack on the process was a bad idea, so we brought someone from talent acquisition along with us.

The persona was a play on ChatGPT: Chet Gupta. We generated candidate pictures and picked the one on the left with a little mustard on his shirt, because he looked like a typical engineer. The middle one looked like he probably drinks IPAs and listens to indie music, and the other was too inspirational to be an engineer. Later, I removed the word "Gupta" from the prompt and got only Caucasian men with glasses. That was my first introduction to unconscious bias as part of AI: one word changed how the persona was represented. I am not sure what to do about it other than create awareness that AI-generated images can contain unconscious bias.

One rubric question involved concurrent hash maps, a Java capability. I am not really familiar with Java, so I rely heavily on our rubric. ChatGPT gave a well-reasoned response that we would have considered acceptable. I compared it with our answer and looked for multiple threads, segmentation or bucketing, and a locking mechanism. In a concurrent hash map, parts are locked for read and write to provide efficiency so a read operation does not get access while being written to. Our rubric answer did not seem accurate, so I went to the internet to see which was right. We at M&T were wrong. We had to change our answer. ChatGPT provided a better answer than our rubric in about two and a half seconds, meaning anyone could prompt and provide a well-reasoned answer better than our own in the hiring process.

Then we looked at the competency-based interview: situation, task, action, and result. The idea is to understand how a candidate responds given a problem space. One prompt asked for a time when they were on a team that faced challenges. The context involved APIs, third-party documentation, team members struggling to keep up with changes, confusion, misunderstandings, an actively changing API, and some team members lacking experience.

The candidate's answer was to organize daily standups, introduce a task management tool such as Jira or Trello, and pair team members with varying levels of expertise. The candidate had invented agile, which is funny, but it was not an unreasonable answer. Many candidates we interview are not exposed to agile; they come from waterfall, especially enterprise organizations, and only get a brief peek into agile rather than mature agility. This is a common answer and good enough. If you are not me, and you are generally a middle manager who does not understand the technical domain or the candidate's backdrop, you would accept it as reasonable.

We packaged this up and sat down with talent acquisition. At first they were apprehensive because it felt like we were attacking the process, and they were personally tied to it. We made clear that we were not attacking them as people; we were addressing process concerns.

Out of that, we had better awareness and education on cheating during the technical screen. New training and emails helped managers learn red flags for cheating and fraud in the hiring process. We moved from pure ignorance to being informed.

We also added scrutiny of vendors and tooling. We asked one vendor supporting our LeetCode-style tools whether anyone had talked about test quality and AI impact. They said there were no problems because the people who write the tests said everything was fine. When I asked who wrote the tests, the answer was the vendor. Of course the people paid to write the tests are not going to say everything is not okay, because it would impact revenue.

We realized we need better vendor partnerships, and the tools alone, used the way we were using them on the front end, were not an effective stopgap. We had to rethink the rollout. We are redefining how to roll out these capabilities, not to HR but to managers closest to the problem, so they make better decisions for the company and the candidate.

We also got involved with the NCFTA, the National Cyber-Forensics and Training Alliance. It is a 501(c)(3) group of companies working with the Department of Homeland Security and CISA to understand threat actors, mitigate their activities, and provide resources and training. In laptop-farm cases, a particular threat actor may work at multiple banks and insurance companies using one candidate and one national ID across multiple companies. They are effectively overemployed, but they are not the candidate we hired; they are a threat actor. We also took an additional role with NCFTA to help HR professionals through a specific branch created from this, equipping them to deal with crime coming into an organization.

The ecosystem has improved since I started this. This is not just a Dan Pries thing; it is the ecosystem understanding fraud is wildly out there. We are seeing better tools for deepfake detection, voice biometrics, and interview proctoring. Tools can replay candidate behavior to see copied-and-pasted code, IP addresses, tab switching, and, outside New York, video that helps understand what the candidate was doing.

But problems remain at every stage of the interview process. It is just a prompt away. The speed and rate at which people can commit fraud is ridiculous, and we need better processes. AI workflow automation, n8n, Hugging Face models, and other accessible tools create a cat-and-mouse situation: as we provide new tools in the ecosystem, people create better tools to defeat them.

There are also legal and ethical hurdles. From a government or regulatory perspective, we need clear guidance. We do not provide pictures of people to hiring managers because we do not want discrimination. Not discriminating and focusing on inclusion is good. But it is challenging to know whether the person who showed up to the interview, or to the job, is the same person who took the interview. From a regulatory perspective, we need clearer guidance on how to understand candidates.

If you want to try this, assume positive intent. In a bank, procurement cycles can be six months to a year. By the time you onboard a new capability, you are a calendar year away and the original problems may not remain. The original premise may have been good; our LeetCode-style tooling came in before generative AI was generally adoptable, and then we had a new problem.

Lead with empathy. If you are going to manage a non-deterministic process using AI and someone crafted that process, attack the process, not the person. Bring process owners with you. Promote continuous improvement not just in technology and leadership, but in the way the organization does processes.

The goal is win-win outcomes for the organization and candidates. We want to accelerate onboarding and interviews, bring the right candidates into the organization, and make sure they are not threat actors. If you want the best talent, you cannot make it frustrating. Reduce friction in hiring, but be critical about the kinds of automation you bring in, because it could create a threat vector.

With that, I think that is the end. If you have questions, connect with me on LinkedIn or post in the Slack channel. I will stick around for a few minutes. Thank you.