Host Intro (Gene Kim)

The next speaker is Annie Vella. She is a distinguished engineer at Westpac in New Zealand, one of the four largest banks in the country.

She recently got her master's degree in engineering, studying the impact of AI coding assistance on software engineering.

And I actually saw her speak at — I guess it was an online DevCon. You were right, Annie. And I loved it because it was pointing out that so many organizations have defined software jobs in such a way that it just doesn't lead to good outcomes. And when it comes to the increasing automation of the software development process, could be dismal or disastrous.

So here to present her own personal views is Annie Vella.

Annie Vella

Thank you. Thank you, Gene, and hello, everyone. Hello, San Jose. It's so great to be here. I've come a long way, but so worth it. It's my first time speaking at an IT Revolution event, so both exciting and a little nervous, but I'll get through it.

Okay, so today I'm going to talk about the specialization trap. But before I do, I might just quickly introduce myself.

Like Gene said, I'm Annie Vella. I'm a distinguished engineer at Westpac New Zealand — that's one of New Zealand's four biggest banks. And over the last year or so, I have been focused on AI governance, essentially rolling out AI safely and responsibly to the whole organization.

I am a software engineer, though — I have been for a long time. I have worked across four countries in 11 different domains. I've worked at startups and scale-ups and enterprise. I've been a manager and I've been an IC, so one could say I've had pretty broad experience, or that I've got itchy feet and I can't stay still for long.

And I've also just completed a Masters of Engineering at the University of Auckland, on the impact of AI coding assistance on software engineering, and that's what I'm going to be talking about a bit today.

---

But where it all started for me personally, a very long time ago. When I was six years old, my parents decided to get me a Commodore 64, and I taught myself to program using that manual right there. And it set me off on a lifelong passion for this field. I literally cannot imagine having done anything else with my life.

And then ChatGPT came along in November of 2022, and I thought, "Oh boy, everything is going to change, especially how we build software." So I thought I might just take myself back to university and see if I might be able to understand better what the impact of this technology might be on our field.

So I designed a mixed-methods longitudinal study over six months — really, end of 2024, beginning of 2025. I used two questionnaires, sent them out. I got about 200 responses; 158 of them were eligible, across 28 countries.

---

We've talked about adoption quite a lot. In fact, everyone talks about adoption, right? It's really high. In general, you hear 90% is a pretty common number to hear. And yet, many organizations are really struggling to see the clear productivity gains from it.

There was a recent article by HBR that talked about a study done by KPMG together with the University of Texas, Austin, and they were interested to understand why that is as well. So they did the study by analyzing 1.3 million prompts generated by 2,500 employees. They were finding 90% adoption, but only about 5% of that was sophisticated use. Now, what they meant by that was people were using different strategies — they were using different models depending on what they were using it for.

So that's quite interesting: 90% adoption, but only 5% using it in a way that might actually achieve those higher productivity gains.

And in talking with Gene a couple of weeks ago, I thought, actually, we're onto something here. Maybe my research — although it didn't set out to prove this — maybe it might help us answer why that is. Why do some people thrive more than others?

---

That's one of the questions I wanted to answer for myself through the research I did: whether AI coding assistants help engineers feel more capable. I don't know about you guys, but I have felt a wee bit of imposter syndrome throughout most of my career. So if these tools can help engineers feel a bit more confident, that's got to be a good thing, right?

So I turned to this concept called self-efficacy. That was proposed by Albert Bandura in 1977 at Stanford University. He's got a nice definition, but basically, in plain terms, it's: I've figured out hard things before, so I can probably figure out hard things again.

To measure this, I used the validated eight-item scale called the New General Self-Efficacy Scale. And I used that in the context of whether software engineers were feeling more or less confident when using AI coding assistants to do their work.

And what I found was that, yes — I guess it makes sense — most software engineers feel more confident when using AI coding assistants. So that's the results from the first questionnaire. It was moderately above neutral at both questionnaires, and it was remarkably stable, actually.

But that stability kind of hides some changes on the individual items. In particular, overcoming challenges — that improved over the six-month period. But comparative ability actually reduced a little between the two six-month questionnaires. And I wonder if that's because over the six months, more and more people were getting access to these tools, and it's harder to feel differentiated when everyone's got access.

---

So I started wondering — what else might self-efficacy in and of itself predict?

When I stepped back and had a look at that, I saw that self-efficacy actually predicted every other experiential outcome that I had been researching. So that's the three dimensions of developer experience — feedback loops, cognitive load, and flow state — as well as impact to skill development and perceived productivity.

And in fact, self-efficacy predicted productivity by a huge amount. It's a really big number. Those who were feeling more capable were over 10 times more likely to report improved productivity. Now, some of you might be wondering — correctly — aren't these things sort of measuring the same thing, though? I wondered the same. So I checked, and they are related, but over 60% of the variance between the two is not explained by the other. And self-efficacy and productivity actually predicted different outcomes that the other didn't. So I was satisfied that they were indeed different enough.

None of the other demographics — seniority, tool choices — predicted productivity at all, or not significantly. Usage frequency did, but not as strongly as self-efficacy. So I thought that was rather interesting.

---

So you might start asking yourself, what is it that predicts self-efficacy?

Well, I looked at all the usual candidates — demographics — and guess what? None of those were significant predictors of self-efficacy. There were two, though.

Usage context: I didn't mind if the respondents to my surveys — which were all professional software engineers who were using AI — were using it at home or at work. Those who were using it at both locations were more likely to report higher self-efficacy.

And juniors were also more likely to report higher self-efficacy. But that one, there's some caution there, because the qualitative analysis that I did on the open-ended questions implied that there might have been a sense of assisted capability — so through a reduction of friction rather than mastery skill-building.

So I want you to hold on to that broader context one.

---

Where does self-efficacy come from? To answer that question, we go back to Bandura's own research, where he identified four different sources of self-efficacy.

Vicarious experiences — that's when you see someone similar to you succeed at something, and then you think, "Well, if they can, then maybe I can, too."

Verbal persuasion — that's when someone you trust tells you, "I think you can do it." "Okay, I'll give it a go."

And emotional states — whenever you're presented with a challenge, how do you actually feel about it? Do you feel fear and flight, or is it a challenge you want to take on?

Then there's mastery experiences, and Bandura felt that that was the strongest source of self-efficacy — experiencing it for yourself. But particularly when you experience something that you weren't sure you could do, and yet you succeeded at it — that builds the greatest self-efficacy.

---

So we should be asking ourselves, what sort of experiences are our engineers having, or what kind of experiences have they had in the past?

And to illustrate what I mean by that, I thought I might just tell you a few little personal career stories of my own.

One whole year out of university, I was working for a software consultancy in Wellington, New Zealand, and we had the opportunity to do some work for the Civil Aviation Authority. And they had the great idea to send me in to collect all the requirements. Now, I was a whole year out of university. I'm a software engineer, and I was absolutely terrified — because if I screwed that up, literally planes might fall out of the sky, and I couldn't bear the thought of that. But I did it anyway, because that's what they told me I needed to go do. I brought back the requirements, worked with my team, we put together a great proposal. I don't think it went anywhere, but I learned a lot from that experience.

And then later on, I was working for a startup in London, and they needed to be able to deploy their software on-demand. And this was in the late 2000s when DevOps was just starting to gain traction. There weren't that many CI/CD tools around yet, so I couldn't download and install something. I didn't have many things to sort of inspire me. I had to work it out for myself. And I did. And I built a really awesome system that, to this day, I'm still really proud of.

And then later on, I was working for a US pension custodian just up the road from here in San Carlos — again, for a small software consultancy — and we were trying to digitize some very paper-based systems that they had. And one of the tasks was to automate the reporting of contributions and distributions from retirement accounts to the IRS. I'm pretty sure that the IRS does not have a sense of humor about this, and if you get it wrong, there are some serious monetary consequences. So again, I was pretty terrified. So I read the IRS manual — as riveting as that sounds, I read it pretty much back to back, because that's where all the rules were, and I could use that to translate it into a design, build the system, and again, it was something that I'm now still really proud of having done. It worked really well.

So I guess what I'm trying to illustrate here is that I was given pretty chunky projects and problems to solve, and I was given the trust to go and just figure it out. I didn't do it on my own — I always had others to work with — but I was given a lot of latitude, even to do things that I was scared of. And I think that that's where self-efficacy really comes from.

---

So what are the sorts of experiences that your engineers have had? Because we know that AI is an amplifier, right? We tend to say that a lot because it's true. It amplifies the good, the bad, and the ugly, though. But what it can't do is amplify what's not there. If the foundations that you start off with are very narrow, then there's simply not that much to amplify.

So software engineering is a very interesting role, because like I've described, when I started, I was taught at university to do the full life cycle: collect the requirements, do the design, build the software, test it, deploy it, maintain it, fix it — the whole thing.

But over the decades, I've noticed that our industry has kind of whittled it down into much smaller, more specialized roles — potentially because there's always been this huge demand for software, and the supply just hasn't been there to meet it. So we've handed off responsibilities to other roles, more specialized.

For example, product owners now are the ones who talk to customers, generally. They might work with a BA and put together the requirements. Some companies have architects who do all of the solution design. Quality engineers take care of all the testing, and platform engineers run the platform and make sure that everything's super stable.

So what's left for the software engineer? Writing code, primarily. That's the one thing that the software engineer — and only the software engineer — has been able to do until recently. And in many cases, we've even specialized on that. You've got your front-end software engineer, your back-end, and you see where I'm going.

---

What's really interesting is that as soon as GenAI came about, we started seeing very provocative statements like, "This is the end of software engineering" — very sad to hear for those of us who have loved this career. And the response from our community has been, "Well, no, that can't be the case, because software engineering has always been about far more than just writing code." And I completely agree — yes, and I want that to be true too. But unfortunately, in many cases, like I've just described, that isn't the case. The reality is that for many software engineers, all they really get to do in their work is write code. And now GenAI can do it for us.

In fact, it's very good at it, as we know. Andrej Karpathy recently said that AI is very good at automating tasks, not entire roles. But when the whole role has become about one very automatable task, well, you can start to see why so many software engineers are a bit hesitant about this technology, a bit resistant. They might see it as a bit of an existential crisis.

---

So trying to pull all of this together: on the right-hand side, that's what my research findings were, backed by the data that I collected and the analysis that I did. Self-efficacy and productivity actually have a bidirectional relationship according to my data.

And on the left-hand side, you've got this theory that I'm starting to conjure. What if the breadth of the role that you've had throughout your career has led to more mastery experiences — opportunities to experience more — which then in and of itself has led to self-efficacy, which now with AI is further amplified? You see, this chain either works for you or against you. If you've had broader experience, maybe it spins up, and you end up with a bigger amplification and better productivity gains. And if you've had a narrower role, well, the opposite can happen.

---

So we need to start asking ourselves: what is it that your system allows? If you're interested in understanding where your productivity gains are, then let's think about that.

But furthermore, what if the leaders who designed a system of work — what if they too have been somewhat atomized? What if they've never even seen or experienced what those broader engineering roles can do? Can they design a new system that allows engineers a larger slice of the problem pie?

---

So pulling it all together: we had the high adoption, the uneven returns, and perhaps we now have yet another explanation as to why that might be the case. Because if your job has been quite broad and you've been able to have broader experiences and build up more of those mastery experiences, there's more for AI to amplify. And for you, AI really feels like a superpower.

But on the other hand, if your role has always been about a singular task — perhaps you've had fewer experiences at end-to-end work — then there's less for AI to amplify, and it might actually feel more like a threat to you. What's super interesting about this is that it's the same technology, just applied in a different context or different system of work.

And as W. Edwards Deming would say, every system is perfectly designed to get the results that it gets.

So the good news is that systems can be redesigned.

---

So where do we start?

I reckon we start by trying to understand what our systems actually permit today. And I don't mean go and read the job descriptions and what the career ladder promises — actually go and talk to your engineers and find out what is their scope, what are they allowed to touch. And be honest with yourselves, because if this theory holds true, then this might actually be what explains the sort of productivity gains you might be seeing.

So before you add any more tools, consider widening the roles if you can.

If you've got software engineers who have never spoken to a customer, then bring them along to a customer meeting, or get them into the customer call center and get them answering calls — or at least listening in on them. Maybe even put them on call if they've never been on call. They're not necessarily going to like that, because that might not have been the deal, but it might be for a better outcome for everyone.

---

To continue to build the conditions for mastery experiences: the experiences we've had in the past, with today's AI, it's amplifying yesterday's experiences. But it doesn't end here. We still have to continue building those experiences so that tomorrow's AI can continue to amplify something.

But don't mistake assisted capability for the real thing. I'm not convinced that the sort of experiences that we might be gaining today — when we might be delegating a little more than we are willing to admit to the AI — I'm not sure that's building the foundations for AI to amplify tomorrow. It might be. Hopefully, we'll see.

---

And so if this theory holds true, then maybe start looking out for self-efficacy. It's not something we tend to talk about an awful lot.

In my research, 84% of my participants reported improved productivity, but 27% also reported worsening developer experience. Now, typically those two things travel together. But my research has shown that actually they're kind of decoupling now. So productivity gains alone is definitely not enough, but maybe it's time to even start considering what role self-efficacy plays in all of this.

---

What I need help with: I would love to know if anyone in this room or further afield has had any luck in actually de-atomizing those engineering roles. It's not an easy task, because the areas that you would expand into — they're not vacant. There are people in those roles. So how do you start to blur the boundaries of these different roles and make sure that there's space for everyone? If you've attempted this, what resistance have you faced, and how have you overcome it? I would love to hear.

And that's all I have for you today, folks. Thank you so much for listening.