Host Intro (Gene Kim)

So the next speaker is Dr. Tapabrata Pal. And I met him in 2014, and holy cow, did he make an impression.

He was a first distinguished engineer at Capital One, and many point to him as a leader of their technology modernization movement. This was in the era where 80% of all their engineers were outsourced, and this is something that he wanted to change. And rumor has it that he was one of the first enterprise architects back then that still had an IDE installed. Because real architects don't have IDEs — which I guess, if you listened to Steve Yegge yesterday, maybe they shouldn't anymore.

At any rate, in 2021, Topo joined Fidelity Investments as their VP of Architecture, and they are now with over $60 trillion of assets under management, over 75,000 employees. He gave this amazing talk yesterday that I had alluded to. He had talked about, among his many responsibilities, he owns the application that you go to when you ask, "Which of our tens of thousands of applications have Log4j?" He had these dreams and aspirations of what it should be, to which, when he asked his teams to do it, they would say, "We need eight months, and we have to hire two front-end engineers." And after vibe coding for five weeks, it then went into production.

And my favorite part of the story, as you shared in the monthly AI forum, Topo, was that no one on his team wanted to manage the application. Everyone said, "Not me." And then in the monthly AI forum, we met Swathi, the most junior engineer on the team, who said, "I will do this with you." And the observation is that she is a switched-on engineer who's probably having more fun and learning more than maybe not just his team, but maybe across the entire organization.

So a couple of weeks ago, Topo texted me and said, due to some reasons, in this session he's not authorized to talk on behalf of Fidelity Investments. So my immediate next question was, "Can you do a fireside chat where I could interview you as an expert who has spent over a decade in highly regulated financial services?" And he said, "No problem." So with that, let's welcome to stage Topo Pal.

Dr. Tapabrata "Topo" Pal

Thank you so much, Gene, for inviting me.

Gene Kim: Oh, my goodness. Yes. And so, let's first state that you are speaking as Topo Pal —

Topo Pal: Mm-hmm.

Gene Kim: — not as an executive at Fidelity Investments.

Topo Pal: That is correct. And whatever I say — don't assign it to Fidelity's point of view. It will be my individual point of view all along throughout this session.

Gene Kim: Fantastic. And so last time we were hanging out, we were talking about how weird it was that you're listening to all these talks and people are saying, "I don't know. I don't know anything." They were talking about some claims that it might be this, it might be that.

We were talking about how when you were at Capital One back in 2014, you probably thought you knew something. You probably didn't think that you knew everything, but you probably thought that you knew more than most, and you had a very startling reaction — you reacted very funny to that.

Topo Pal: Yes. Not more than most on everything — in certain things. Before I joined Capital One, I had seen continuous integration in my actual work, starting with CruiseControl, Hudson, and all that stuff. So in that domain, I thought I knew many things. And being in a financial organization, I did have that feedback too from our auditors and risk control. So I had a point of view.

Gene Kim: And you had a conviction that it could be done there.

Topo Pal: It could be done.

Gene Kim: And now —

Topo Pal: — on that topic, I'm like, "I don't know." I'm just looking for answers. I'm thinking about answers, and I'm just talking to people to understand what's going on.

Gene Kim: And how does it feel? You're a leader, and you're not sure you have any basis to have stronger claims than someone else. How does it feel?

Topo Pal: It makes you feel vulnerable to some extent. Since I own the strategy and the direction and roadmap for a certain important platform, I do feel that I should owe an answer to people who are asking questions. But in this regard, I really don't. I do know the things that are not going to work, but I don't know how to make them work again.

Gene Kim: Yeah. But before you go into what won't work, you actually described a feeling you had earlier in your life where that made you feel the same way. Can you share that?

Topo Pal: Yes. I had a PhD in semiconductor physics — material science in that area. So when I signed up for doing the PhD and got admitted, I met my advisor, and I asked him, "So what am I going to work on?" And he said, "I don't know." And I said, "I thought you had a problem in mind." And he said, "I do, but I'm not going to tell you." He told me that part of the PhD work is actually to find out where the problem is, what I should focus on, what I want to focus on — sort of filtering out the noise and focusing on something. So I'm at that point again.

Gene Kim: Sort of. But then, PhD candidates are, in general, miserable people.

Topo Pal: Yes. Yeah.

Gene Kim: But you don't feel miserable now.

Topo Pal: I don't feel miserable now. Yes. That is true.

Gene Kim: In fact, what is the opposite of miserable?

Topo Pal: Hopeful.

Gene Kim: Optimistic. That's the way you feel now.

Topo Pal: Yeah.

Gene Kim: Okay. So you can't give direct answers to the people who are asking, "What are we supposed to do?" You said one of the helpful things is you know what's not going to work.

Topo Pal: Correct.

Gene Kim: What is that?

Topo Pal: So since I vibe coded an enterprise application — if you will, even if it was not customer-facing, it was important — and sending anything to production in a big enterprise is not just coding, but a whole bunch of other things. So I did experience a few things — actually, a lot of things — that I know are not going to work. And I'll start with Git.

With vibe coding, if you include more than two or three people in the same code repository just vibing away throughout the day, I'm not sure if merging is going to be that smooth anymore, and I don't know how to solve that. So that's number one.

We are talking about a 10x code increase from every person. I'm not sure if someday GitHub will say, "We'll charge you by number of lines of code" or whatever that is, because it's not going to scale that well. Maybe the scanners that some of you are using, paying by number of lines of code, will cost more. Maybe the CI/CD infrastructure that you built over the last 10 years or five years or whatever it may be may not scale well anymore. Maybe the release process that you have today cannot stand up to the pressure.

I think the same story happened when Agile started. When Agile started, it created a tremendous amount of pressure on ops, and that's why DevOps came about. And now it's like 100 times more pressure at the beginning, where everybody's coding, everybody's building, everybody's creating something.

Gene Kim: Yeah. We were hearing from somebody that there are certain enterprise license agreements billed by number of lines of code, and now they're trying to re-project — when that was signed a couple of years ago it was just fine, and now they don't even know how many lines of code there will be in five years.

Topo Pal: Right. Yes.

Gene Kim: So those are the things that are breaking.

Topo Pal: They are breaking.

Gene Kim: Those are the things that are not going to work.

Topo Pal: Yep.

Gene Kim: So can you talk a little bit... Oh, and by the way, we were talking with a friend at a foundation AI lab, and this person was saying that they're actually rewriting Git because the merge times are too long — you can't afford to spend minutes merging, that has to be driven down to seconds. And hopefully they can get Git there, and if not, maybe we're going to see what's after Git.

Topo Pal: I personally spent two full days merging, and then suddenly — let's rewrite the whole thing again. Because, you know — one hour to code, and then two people, a bunch of comments went in, and now a merge conflict that takes two days to resolve.

Gene Kim: The juice wouldn't squeeze. Stop trying to merge it, just do it again from scratch.

Topo Pal: Yes.

Gene Kim: Because the cost of production is going to zero, and it's nearing instantaneous.

Topo Pal: Yeah.

Gene Kim: So last year I saw you speak at the New York DevCon conference that Patrick Doyle held, and you said something that I just loved. You said, "Would you trust AI right now to write code that handled monetary transactions?" And you said, "Not yet." And that was actually a couple of months after your presentation in Las Vegas. So speaking for yourself, not on behalf of any organization, how would you answer that question now?

Topo Pal: Not yet. But very close. If you can kind of put a guardrail around where the changes are made, and have actual manual review by people, and have test cases to prove it, and you feel confident — then yes. Otherwise, no. Not yet.

Gene Kim: Not yet. And so I had mentioned that we got a chance to — you kept on talking about this mysterious person who volunteered to help you manage that application in production. We got to meet her back in February, and it was so interesting to hear her talk. She said, "I love solving problems" — in a way subtly suggesting that maybe her fellow engineers don't. So given that experience, how do stories like that change your thinking about what juniors should do, what seniors should do? How do you become a more valuable engineer?

Topo Pal: There are a few things that junior developers can do today. If they want to learn any new technology, any new programming languages, or any technology stack for that matter, just start coding and you will learn. I think it's the easiest way these days to learn — actually, through vibe coding. And I learned myself. I learned a bunch of things I had no idea about before. I learned TypeScript. I never coded in TypeScript. I learned Cypher queries. I never thought I would have to learn, but I love it.

So I think that's one way. The other way that junior developers can do is pair program with senior developers, to bring in years of knowledge and experience around architecture, non-functional requirements, scalability, and all that into your vibe coding experience.

Gene Kim: Yeah. I just love that. Kent Beck — as you probably saw on the plaque — he said, "XP is coming back." Pair programming is coming back, if it ever left. But I think there's more of it than ever.

Topo Pal: Yes. And there are a few things that are coming back, which — I'll say number one — writing good requirements, clear requirements; writing clear non-functional requirements; having an overall architecture of your product that you are planning to develop; having a good structure of your MVP0, MVP1, phase one, phase two, phase three. And it's nothing against Agile or DevOps or any of these — some of these practices are lost, and they are coming back. I'm so happy about that.

Gene Kim: Yeah. By the way, I'm not sure if we have time today. One thing I'd love to do is actually share more of the details of the making of the game last night, but it was an experience like I've never had. I've certainly pair programmed with people, but what I've never done before is sit next to a fellow developer who shared a microphone. And for basically two hours, we were — I've known Brian for years, but we've never done that before. And it was just a surreal experience where it was like, this is so fun and yet so novel, and it's something I want to do more of.

Topo Pal: Yeah. Absolutely.

Gene Kim: So given that — okay, where do you think things are going? You say you don't know much —

Topo Pal: As I said —

Gene Kim: — with certainty.

Topo Pal: Yes.

Gene Kim: What do you do?

Topo Pal: So I don't know where it is going, but I do know the things that are not going with it. All your infrastructure to build your software — your processes, your code review process, your scanning process, your open source vulnerability guarding, whatever mechanism you have, your release orchestration, your audit compliance. None of these — or most of these — are not going with it. And then again, as I said, the old practices of good architecture, documenting the architecture, good practices of having a good set of requirements, good planning — they're coming back again. And I'm kind of happy with that.

So there are a lot of things that will change. I think the half-life of our time is not one year or two years. It's six months or less.

Gene Kim: Half-life as professionals?

Topo Pal: Half-life of any approach — or software delivery engineering in general.

Gene Kim: I think the half-life of models is about what — three months now?

Topo Pal: Half-life of what?

Gene Kim: Half-life of a model. A coding model.

Topo Pal: Oh, yeah.

Gene Kim: Yeah. That's right. It's three to six months now. So that is going to create tremendous pressure on our processes.

Topo Pal: Yeah.

Gene Kim: And by the way, Mik Kersten was telling us about an interview he did with Scott, who's now at Cognition AI. And one of the things they really picked up on was just how throwing away code is really a part of their philosophy. So they have, at any point in time, 10 parallel harnesses that they're working on. Apparently every Sunday night, the CEO picks one and they throw the rest away. Every week, 90% of the coding efforts get thrown away. So that's a level of throwaway that I think is pretty new.

Topo Pal: Yes. It is throwaway — yes and no — but it's still sitting on somebody's laptop.

Gene Kim: Oh, right. It's not in production.

Topo Pal: Right.

Gene Kim: And it's created knowledge.

Topo Pal: Exactly.

Gene Kim: One of the things that we talked about before was it's just not known when the auditors show up and start asking, "How are you going to prove that this is secure, meets contractual obligations, meets all your regulatory obligations?" And I think one of the things that you said was we don't really know how to have that conversation yet — it's not your top most urgent issue right now. And it was interesting to hear Tyson Matthew say, hey, they have highly regulated customers and highly regulated industries, and he seems to be suggesting that's not really an issue, right — as long as we are proving to ourselves that we have controls and that they are effective. What's your response to that?

Topo Pal: I think from an auditor's perspective, the controls and the effectiveness of those controls are the most important thing. So let's take an example. We wrote in our IT Revolution papers, right, that you can replace segregation of duties — or separation of duties — and implement that in the code review process. So imagine everything is code: your application, your tests, your infrastructure — everything is code. And unless you check it into GitHub or any other source control system, it's not going to production or it's not changing production, including your rehydration process. Now, in that model, just having a pure code review by a second set of eyes provides that same set of control that your auditor would agree with — yes, there is a second set of eyes on every change.

Now imagine a pull request with 2,000 lines of code, and then somebody's peer reviewing that and being the accountable person who says, "Yes, I have reviewed everything, and it's good to go." How comfortable would that person feel — that they can be accountable for the changes? I'm not. So that's probably one challenge.

The other challenge is, yes, you can prove the controls, but as you know, as developers, you can work around many of these controls, and that's why you have layers of controls. It's like every layer is a Swiss cheese, but if you put one on top of another, you get a solid block. That's kind of the idea.

Many of these controls fail. For example, you have open source vulnerability checks before you go to production. And with the volume of changes that are going into the open source world, what are the chances that between when you push your code to pre-production and the time you're pushing code to production, there's no vulnerability popping up? They are going to be popping up. And what are we going to do about that? So all these things make me — yes, one thing is being certified, but the other thing is that on a transaction basis, are we actually meeting all the controls?

Gene Kim: Yeah. And so scale of one to ten — even though you might not have all the answers — one is like, "Oh my gosh, we're going to be spending the rest of the next two years in misery trying to deal with our security and compliance people," and a ten is like, "Oh no — even though I don't know exactly how, I have moral certainty that I have the tools to have those conversations and walk out of it happy."

Topo Pal: Maybe formal methods become another control.

Gene Kim: The what?

Topo Pal: Formal methods.

Gene Kim: Oh, right. Formal methods. Yeah. The mathematicians to the rescue.

Topo Pal: Yes.

Gene Kim: And by the way, I'm not sure if you caught this, but when Eric Meyer was talking — I think he was joking, but I think he was not actually joking — is that when he presents to audiences of technology leaders, in most situations he gets laughed out of the room because no one likes formal methods. But I think what's very interesting to me is that among this community, there's a surprising number of people who not just like formal methods, but actually found a good chunk of their career using formal methods.

Topo Pal: Yep.

Gene Kim: Okay. Question from Will: Have you had any luck or experience using LLMs to produce evidence that's relevant to compliance, control objectives, and so forth?

Topo Pal: LLMs producing evidence? I tried. For example, I have a good set of data around what was the intent of the change and what was the change. And then I had independent scanners or verifiers — in terms of tools and other things, automated test cases. And I put this together and created a pipeline of that verification. But I have mixed results so far, I would say. In some cases — well, the first time I ran the thing, my cloud agent actually changed the code. I'm like, "That's not the intent." You cannot change the code just because some verification failed. But overall, I think that can be mixed with that formal methods kind of structure to have LLMs verify your changes from start to end.

Gene Kim: Oh, love it. Very good. Hey, Topo, I'm so glad that you could make the room to have this conversation in front of all of us, and you're here for the rest of the day. If anyone has any questions, Topo has vast experiences — I'm so glad you're going to be here to have conversations.

Topo Pal: Thank you so much.

Gene Kim: Thank you.