Host Intro (Gene Kim)

Okay. Our next speaker is Dr. Daniel Rock, currently assistant professor at the Wharton School and CTO of Workhelix. I have so much admiration for his groundbreaking contribution to the famous 2023 OpenAI jobs report. It was such an incredible study that looked at AI's potential impact on the labor market, and I've had the privilege of working with him over the last year, which has also been so fun and rewarding.

The punchline: they found that mathematicians, tax preparers, financial quantitative analysts, writers and authors, and web and digital interface designers are the most exposed; least exposed are dishwashers, floor sanders, and food service workers.

I was actually at the Vanguard Internal Technology Conference, and I was floored to see their chief economist give a presentation on AI and consistently present Dr. Rock's studies. I kept taking pictures and texting them to Daniel, saying, "Look, look who's citing you?"

So he's probably one of the most cited people in the Vibe Coding book, not just because of the research I mentioned, but also because of these incredible stories he's been telling us about how he is using agent coding techniques. That is how the movie Pacific Rim, about giant robots, made it into the book. So here is Daniel Rock.

Dr. Daniel Rock

All right. Thank you. Hey, what a pleasure to be here with you all today, and thank you so much to Gene and the team for inviting me.

Today I'm going to talk to you about a bunch of different work we're doing, some of it academic, some of it with the company I co-founded to help AI transformation go a little bit faster and be a little bit more productive. That's called Workhelix.

This is all from an economist's perspective. If you needed help getting a little bit more of a nap this morning, I'm happy to oblige, but some of us are excited about this stuff.

I think this is a little bit preaching to the choir, but generative AI is certainly going to be transformative, and it also requires practice. We don't get the benefits immediately. You have to get good at this.

Economists would call generative AI a general purpose technology. I'll get into what that means a little bit later. But it's of the transformative ilk that the personal computer, the internet, and long ago electricity and even steam power might have been. It reconfigures quite a lot in the economy, and that takes a lot of time.

Nevertheless, now is the time to get moving. You want to be relentless about measuring and making sure things are going well, but experimentation is going to be key to making sure that you get the most out of the assets that you have. In particular, the complementary assets to AI are data, talent, and compute. Those things will take you a lot further now than they have in the past.

Let me get into what a general purpose technology is. Economists define a general purpose technology as one that has pervasive impact. It changes lots of different things in the economy. It improves over time, which I think we can pretty much all take for granted is happening with these tools. And importantly, it spawns and necessitates complementary innovation.

Think about steering a really big ship and wanting to change the direction of that ship. It costs energy; it costs fuel to do that. We should be going in a new direction, even if it takes a little while to change that direction.

With some co-authors at OpenAI and GovAI, we set out a couple years ago to answer the question: is AI a general purpose technology? If we can answer in the affirmative, then this practice of saying all the jobs are going to go away, or all the jobs are certainly not going away, those binary extremes, are not enough.

The economist answer is a little bit like a Dickensian novel. You'll be visited by a pessimist, an optimist, and a realist. The pessimistic side is that if it is a general purpose technology, it changes so much all at once that we can't really be sure where wages and employment are going to go. It depends on too much.

The optimistic side is that we did find pervasive impact potential. We're now seeing that show up in studies from Anthropic, OpenAI, and Microsoft: our measures two years ago were predictive of where AI would start to show up first. We're seeing lots of different places. Mathematicians, for example: my co-author Pamela got a lot of angry emails from mathematicians saying, "No, AI can't do my job." She wasn't saying that it could, but even I can be a mathematician now, just one that makes a lot more mistakes.

When you get that pervasive impact, you have a lot of potential. That's an optimistic view of the world, where lots of things can get better and more efficient. The realistic view is that we all have a ton of work to do to make the release of drudgery and expansion into things that we really value more common.

The punchline is that around 80% of workers in the economy have maybe 10% of their tasks exposed. Think of a job as a bundle of tasks, things that you can mix and match. There are reasons those tasks exist together in the same bundle, usually coordination between those tasks. Writing an email to your boss is not separated from whatever work you're writing about; you wouldn't have a separate worker to do that.

You can think of things at the task level: where do LLMs help us do our work? Or you can think of things at the systems level: how do we configure different roles together to produce some kind of output? But that intermediate level of the role that matters to a lot of us individually is not necessarily where the impact is going to be driven.

In our research, in a paper called "GPTs are GPTs" -- that is, generative pre-trained transformers are general purpose technologies -- we look at 20,000 tasks listed out by the U.S. government in a database called O*NET. We now can build these taxonomies at Workhelix for every company, the 10,000 tasks that live inside your company as opposed to the 20,000 that the government says exist.

We ask a really simple question for each of those 20,000 tasks of both humans who understand LLMs and AI systems, in particular a version of GPT-4: could I cut the time it takes to do this task in half without a drop in quality?

There are only three answers to that. There is no, which we call E0, no exposure. That's things like washing dishes, as Gene mentioned. There is yes, like writing an email. We call that E1 exposure, where an out-of-the-box LLM right now or in the near future can basically help you do that a lot better.

Then there is E2, the yes-and category. This is the important one: yes, and you have to build a whole bunch more in terms of systems, software, other trainings, and organizational process redesign. It is the hard work of making a general purpose technology more general. If we can say there is a big difference between the yes and yes-and category, then we know we have a GPT on our hands, and this is going to be decades of work to incorporate it.

What do we find? On the pervasiveness question, the answer is yes, it's pervasive. Every industry has some reasonably high level of exposure, in the sense that there are big opportunities. We find that a lot of the higher-wage roles, knowledge workers, are the ones with the most exposure.

When we say exposure, we don't mean your job is going away. We mean something will change about how you do your job. That could be great. It could also be bad. We don't necessarily know. Anyone who tells you that they know doesn't have the facts or is trying to sell you a bill of goods.

It depends on a lot. We can't get away from labor markets. Is the demand elastic? If I drop the price or increase the productivity of workers in that category, would you hire more of them or would you hire fewer of them? Do you only need a few FP&A analysts to do the same work? Or, on the other hand, as we build three orders of magnitude more lines of code per engineer, why would we need fewer devs in that world? That seems confusing to me, but you can see where the anxiety comes from. There is a lot of change that happens for people who are getting paid very well, people who have made investments in their human capital and are worried that might go away.

The x-axis here is the wage. The y-axis is the rating, whether human or GPT; they mostly agree, so we see similar trends. We have pervasiveness. I'm going to punt on improvement over time because I think it's obvious, so we're going straight to the most important category: do we see that it's going to require and spawn complementary innovation?

We can think about exposure this way. Take the graph on the right. If we ask what percentage of employees in the economy have at least 40% of their tasks exposed at the yes-right-now-LLMs-can-help-you level, it is actually a small number. You go to 40 and then up to the first pair of dots; it's like 3% to 5% of employees in the economy have 40% of their tasks exposed.

But if we could unlock, if we could snap our fingers and all of that complementary innovation -- the right business model, structures, training, all of that -- existed already, what percentage of roles have a large portion of their work exposed? At 40%, you get to about half the economy having a lot of potential to use LLMs.

Thinking about this, it's not that much of a stretch. We can think about what the internet did. How many of us use the internet in our jobs? It's basically everybody who does knowledge work. LLMs are one more node in the software stack, a new kind of node, an important kind of node, but rhyming with historical general purpose technologies. Or, as my friend James Cham at Bloomberg Beta likes to say, LLMs enable fuzzy CRON jobs, another word for agents, I guess, if you want to link those together.

I said the realistic part is that we have a lot of building to do. The optimistic part is that we broke this down, looked at all the different jobs, and connected them by activities they share, things they do in common. You see clusters start to show up.

The really exciting thing about LLMs and AI is that the parts of the economy that are most exposed, where the most change will happen, are scientists, engineers, technologists, teachers -- groups of people responsible for building up capital and new ways of doing things in our economy. If you're really excited about economic growth at the macro level, this is exactly what you want to see. This is exactly what will lead to much better outcomes. You see things that are hard to price from an economist's perspective, like inventing new types of antibiotics using machine learning systems. That's tremendously exciting. That's what leads me to a lot of optimism.

But on the pessimistic side, or maybe the realistic side, I think a lot of large organizations -- and this is something we see a lot with partners at Workhelix -- if you have a money-making machine, if your firm works really well right now, what you do first with AI, both to build credibility and because it's a good idea, is you look at how you do things now. Think of this production pathway with a lot of different nodes, like a directed acyclic graph of production. You look at what you're doing right now and say, how can I put AI into each one of those steps? You may get large productivity gains in each step.

Then people ask, where's the ROI? The ROI can only happen in a couple ways if you do it that way. One is at the end of the pipe. If you make salespeople 25% more productive, that goes straight to your top line. That's great news.

What if you make a researcher 60 or 70 steps back 50%, 100% more productive? That's great. They might like their work a lot more, which is something. But then it gets bottlenecked or pushed through 60 or 70 more steps, and by the time it hits the end of the pipe, it's maybe a 0.2% gain. Everyone says, there's no ROI.

We have to wait. We have to reconfigure. You have to build new paths. You have to expose that productivity gain to the final output. Otherwise, you get folks saying we should just fire half of these people and take the cost out. You get money that way, but that's also a way to kill your business, maybe penny-wise, pound-foolish, because those people know a lot and they're highly valuable.

This process of trying to take your best and brightest people, many of you in the room, and build that new production path to do things more efficiently, but also to expand the capabilities your organization has, takes time. For a while it looks like we put a lot in, forego real opportunities, and don't get as much out. We get less for more, but you're building up an intangible asset. That intangible asset is a type of capital. It's just not well measured by your balance sheet.

Over time, that intangible asset starts to pay off, and it looks like we get free money and productivity booms. This is a very old story in the economy, but it's one we often forget: investment takes a while to pay off. Sometimes hidden investment is devalued relative to the obvious stuff. My co-authors Erik Brynjolfsson, Chad Syverson, and I call this the productivity J-curve.

At Workhelix, our job is to help shorten the time to ROI. Even if the J-curve is very deep, we want to shorten the time to "it's starting to generate value." We see value in this incremental approach. I'm not saying don't do the incremental thing where you look at what you're doing right now and build AI into those processes. That's a great way to get an organization to learn to trust AI and to learn where not to.

This is actual customer data, looking at devs with and without using AI, where the econometrics toolkit becomes very handy: pull request count, merge count overall, and average PR-to-merge hours. What you'll notice is that the dark blue line, the people who reached for the AI very quickly, tend to perform better generally. There's selection bias in the data.

Selection bias sounds scary and bad, but it turns out selection bias is actually useful if you are starting to deploy AI in your organization, or you already have, and you know who's reaching for it. Those people are often your best performers. They're going to be able to tell you quite a lot.

But there's another kind of best performer you shouldn't cast out. I call them craftspeople. If these straight-into-my-veins frontier people are racing ahead, that's awesome. But thinking about the DORA report, where stability and throughput velocity are historically positively correlated, with AI there is a potential for there to be a lot of velocity and a little bit of loss of stability.

Your craftspeople, the ones who are saying, "Hey, I just want to do a really good job at my work and I don't want the risk that AI dumps in," are the ones you can use to constrain the folks racing ahead with AI and learn best practices so you can still build stably as well.

Then there's a middle group that says, "I need you to set clear guidance for me. Show me what to do, show me how to build properly." Those people are valuable too. There is a third or fourth category where they're like, "I hate AI and I'll never use it, and I'll obstruct all of your efforts to deploy it." Then there's a different thing you should do with those people.

We take care of the selection bias, and what do we see? Matt was talking about causal inference. I say, yeah, that's great; this is the way. It's not an accident Matt and I agree on very many things.

Where's the ROI for this early stuff? Taking out the selection bias, the initial difference between users and non-users is 44.2%. When you control for that and look at the causal effect of deploying AI, it drops to about 11.2% on pull requests, which is something, but it's not transformative. Clearly there's a need in this organization to do something transformative if you want to get more juice out of AI. Granted, this is using an older version of the tools, so it's probably the worst AI they'll ever deploy, and the tools are getting much better.

Beyond developers, developers certainly have amazing opportunities and everyone in this room is thinking about that. The tech organization can be a leader for the rest of the organization to adopt AI as well.

This is synthetic data, but it looks a lot like one of our customers. Think about four different groups: sales in gray, research in dark blue, manufacturing in teal, and admin in yellow. On the top right, we've got some power users. On the x-axis, we've got the potential AI acceleration. We measure that first. On the y-axis, it's how many active days of AI use you have in your internal LLM system. The top right are people with high potential and high use.

There is arbitrage in all of your organizations. I've yet to see an organization where the use of AI isn't power-law distributed. Some people are really doing amazing things, and the rest of the organization has no idea how to do those things. In the top right, that's where you find those folks. On the bottom left, you find people that you should expect: they don't have a lot of opportunity to use AI and they're not using it. If you find someone in that quadrant who is using it a lot, that's an opportunity to reconfigure what you're doing.

In the bottom right, these are the folks where you've got arbitrage opportunities: lots of potential and not as much use. In the middle, something I brought up with sales earlier: one of the companies we were looking at, we found sales was their biggest opportunity. They said, well, our salespeople don't really use it, but we did find a few who were, and boy are they racing ahead. They're doing a great job. What you've got to do is talk to them, or list out the tasks they do with AI, and then teach their colleagues how to behave like that.

That's very micro level. What does this transformation look like when you start getting to the org level? Historically, we think of the technology part of the organization as a hub-and-spoke model. You build software for everybody else in the organization. Now we don't have to do that.

It's like the movement from steam power to electric power, where they took a giant steam-powered engine in the middle of the room and replaced it with a giant electric one, only to realize 30 years later: let's make those electric engines smaller and propagate them. Do that with your dev teams.

AI can annihilate the coordination costs between devs and people in the business. You build together. You have the agents write code, and within a few minutes you're seeing results that you can tweak or change, and the communication goes a lot faster.

There are lots of different types of nodes here. You can have humans do work on their own. You can have machines do work on their own. You can have humans manage teams and machines. You can have machines possibly manage teams of humans. You have combinations of these in the organization of the future that is potentially much more modular and decentralized.

Gene alluded to this. I call this the Drift a little bit. For those of you who would like to be piloting giant robots to fight evil alien sea monsters, this is going to be really disappointing, because that's not what this is. What we get instead is we get to work with coding agents and small teams to build software. It's kind of cool to pilot a giant robot, but how about going to work and getting to work with AI that will help us all build these faster paths?

When you've got these decentralized workflows, the risk, of course, is that you break things. You have lots of people making software that's poorly maintained. It doesn't have the best standards because the rest of your organization is going to be building software, and that's a bit of a nightmare. But if you can corral that, harness it, and have professional developer supervision on that process, then you can break that potentially negative correlation between velocity and stability.

Takeaways to think about here. We found this is not the robot apocalypse. Work will change, but it's not going to all go away. The things we do now are going to have returns later. How many ERP deployments were profitable within six months? I'm going to say zero. Some of them weren't profitable after five years. We have to have patience.

If you're examining this, if you want to track this, see what's going well, and make sure you detect those pockets of value so that you can deploy them and scale them, that's where the econometric toolkit is going to come in handy. That's what we're doing at Workhelix.

If you'd like to talk about that, please get in touch. We're also hiring, looking for a software engineering lead and forward-deployed engineers. If you'd like to work with us, that's on the right. Thanks very much. A pleasure to be here, and please enjoy the rest of the conference.

Thank you.