Host Intro (Gene Kim)

[00:00:17.400] Next speaker is Dr. Tapabrata Pal. I met him at the IBM Innovate conference in 2014, and holy cow, did he make an impression on me. He was the first distinguished engineer at Capital One, and everyone pointed to him as the person who was leading the technology modernization movement there. This is shortly after the era where nearly 80% of their software engineers were outsourced, which he was trying to bring to an end.

[00:00:41.400] Topo was one of the first speakers at the DevOps Enterprise Summit in 2014, and it was one of the most shocking presentations because it was an actual bank talking about doing DevOps-y things, which was at the time unthinkable. Like Jason Cox, his presentation that year was not allowed to be recorded because of enterprise reasons. But as a sign of Topo's savviness, Topo's future talks were allowed to be recorded, and I started seeing Capital One engineers present everywhere.

[00:01:05.800] Capital One declared itself to be a technology company, with a strategy of cloud first, open source first. Over the years, he has spoken with the former auditor who headed up internal governance, the corporate counsel that worked on open source software, and so much more.

[00:01:20.100] In March 2021, Topo joined Fidelity Investments as a VP of architecture. Fidelity was founded in 1946 and recently celebrated its 75th birthday. It is one of the largest asset management firms, with over $9.9 trillion under management in 2021, $24 billion in revenue, and over 70,000 employees.

[00:01:40.900] Showing how history does not repeat itself, but it can rhyme, I was texting with Topo on Thursday right before he was meeting with his corporate comms folks, working to get approval to present and have this recorded. Again, congratulations to Topo and to Fidelity for pulling it off. Here's Topo.

Dr. Tapabrata 'Topo' Pal

[00:02:06.500] Wow. So many people in one single room. I haven't seen that for a long time. I'm so excited to be here, but I need to get through this first before I meet you all personally, one by one. It has been a nice experience so far. Let's go through this.

[00:02:31.200] I'm going to talk about DevOps at Fidelity: investing in inner source and engineering excellence. Before that, I'll give you two seconds to read this, and I'll be silent. All right, done.

[00:02:46.200] After this long, good intro that Gene gave about myself, I still have something to say about myself. I'm a VP, Enterprise Architecture. My focus is on DevOps domain architecture, and I also am responsible for the open source program office.

[00:03:04.500] I'm essentially a developer, open source contributor, DevOps enthusiast, and recently became an author too. Now, being an author with IT Revolution, I get to sign books. Tomorrow is the book signing, so I'll be seeing many of you there. But before that, I have to get to this 18 minutes and 54 seconds counting.

[00:03:31.900] Before I go through this Fidelity culture code, I'll give you a story. First thing is, why did I leave Capital One and join Fidelity? Well, one October morning in 2020, I got two messages from my LinkedIn feed. First: congratulations, you lived through 10 years at Capital One. People started congratulating me. Then the next one was: based on your profile, look at these jobs that we found for you. So I'm like, okay, that's a hint right there.

[00:04:09.200] I started interviewing. I did not know where I would land. I had two requirements in my mind. Number one, the culture aspect: the company culture has to be something that encourages me every day. Number two, there has to be a tech orientation.

[00:04:25.900] On the second interview at Fidelity Investments, I asked the recruiter, what is your culture? This is what she gave me in writing: cultural portraits, who we are and what we aspire to be. We know that happy people are most productive, creative, and produce better customer outcomes. That checked the first box right there for me. It is not talking about technology or anything. It's just about happy people delivering customer solutions every day.

[00:05:00.600] The next one was this: we believe in leadership at all levels, and that titles don't determine what you can or cannot do. It's amazing. I go to every meeting on Zoom, of course these days, and I cannot tell who is who. It can be a developer, or it can be a VP, or it can be an SVP, or it can be a CIO. Looking at their title in the Zoom meeting, I can't tell. That is awesome.

[00:05:24.200] The last thing that actually floored me was this: no roles, only role models. That checked me. I'm floored. I signed up.

[00:05:35.300] Before I go further, let's talk about Fidelity Investments. Fidelity Investments is a privately owned company established in 1946. This year, we are celebrating the 75th birthday, as Gene mentioned. As of today, we have about 25,000 big businesses and small businesses that trust us with their work benefits. About 40 million individual investors trust us with their investments. We have $9.9 trillion under administration, $3.7 trillion under discretionary assets, and our average transactions per day are about 2.8 million transactions.

[00:06:21.600] Fidelity Investments also is tech savvy. It is actually technology with a mission to help people. Using voice-assisted computer response systems to provide price and yield quotes 24 hours a day, back in 1979. If you look at 1995 or 1996, when the rest of the industry, or everybody, was posting cat pictures on the internet, Fidelity was actually helping customers to access their 401(k) and make transactions on the internet. We do have Fidelity Labs and Fidelity Digital Assets focusing on AI, ML, Bitcoin, and blockchain.

[00:07:04.400] With that, we have 18,000 tech associates and developers in nine countries, with an annual budget of $2.5 billion. That's pretty significant. Now, where do I belong in this? Fidelity is a complex organization. A lot of individual, independent business units have their own technology stack. They have their own technology group. They make their own technology decisions. But there are some enterprise units at that level, and architecture is one of those enterprise units. Enterprise architecture. Business units do have their own architecture group too. That architecture group is where I belong.

[00:07:52.400] Now there are about 50 business units and about five or six enterprise units. Looking at the Agile and DevOps journey, 2016 was the first time Fidelity deployed an application on public cloud. In 2017, Fidelity had the enterprise Agile kickoff. In 2022, 50% of apps are on cloud. That is today. Our goal is by 2024, 70% of apps will be on cloud. I think we can do more than that, but that's our goal.

[00:08:35.400] As of today, there are 18,000 developers across 1,600 squads. By looking at Fidelity's cloud DevOps journey, we measured from October 2019 to September 2022 and noticed all these positive outcomes: business-impacting incidents have gone down significantly, change-induced incidents went down significantly, more than double digits, and MTTR had huge improvement. At the same time, the number of changes has gone up significantly over this period of time.

[00:09:17.200] So the question is, how do you go to the next level? What are our challenges, and what are we doing about that? That's what I'm going to talk about next.

[00:09:34.200] When I joined Fidelity, the first three weeks I met with a lot of leaders across the whole enterprise and business units. They ranged from CIOs, chief architects, lead developers, development teams. I also met with auditors, risk partners, and even legal partners. I asked them all three questions: what is going good today? What is not going good today? What can be done to make it better? Then I documented all that. I put virtual sticky notes and grouped them.

[00:10:11.200] I came up with this. The first one: developer experience. Onboarding is a challenge. Not that there is no onboarding; actually, there are too many. Documentation: not that there is no documentation; actually, there is a lot more documentation than everyone would ever need. Context switching between this documentation, that onboarding, that tool, and all that takes time.

[00:10:37.600] Tools for all. I think this is not a Fidelity problem. This is an industry problem in this area. There are a lot more tools than one would need. Not only tools. With tools, people now create abstraction layers. On abstraction layers, they create frameworks. Between the tools, the abstraction layers, and frameworks, there are too many things to handle. Then you get stuck because now everybody is tied to this particular set of tools. You cannot upgrade your tools. You cannot move from tool one to tool two, because all these abstractions and frameworks are all connected together. You are in a mess.

[00:11:14.900] In one of my previous lives, I'm not going to name that company, I counted 57 different pipelines created on this particular set of abstractions and frameworks. Why there are 57, I have no idea. I even heard from another company, that I'm not going to name and is not my current company, that one leader in their engineering department actually wanted every developer to run their own CI tool. That's amazing.

[00:11:48.100] The next challenge is security, audit, compliance. We talk about this a whole lot on this stage. The thing that I want to point out is that the main difficulty is ownership. Can the developers actually own the security vulnerabilities themselves and decide what to do with them? Can they get feedback early in the development life cycle?

[00:12:09.700] The last one is metrics. There are a whole lot of metrics, a whole lot of techniques, and many, many maturity models. At the end of the day, we are comparing apples and oranges. Between apples and oranges, it is very hard to tell where the team is in their DevOps journey.

[00:12:34.100] I took all these, and then I met our CIOs and asked them, this is what I see; where do I go from there? What do you want to do? They suggested this: let's get everybody together. From there we formed what we call the DevOps Council at Fidelity. There are CIO sponsors, and I am one of the council co-chairs along with two other colleagues in Fidelity. All business unit representatives, all developer tools, product owners, architecture, software engineers: they all have representation in the DevOps Council.

[00:13:22.300] What we do is we look at all these problem sets and try to find solutions in small groups. Time bounded, we find a solution, recommendation, or next steps. We come back to council, report on that, discuss, and find the next steps. That's the whole structure.

[00:13:43.200] How did that help in our solution? Let's go back to the challenges first. As a group in the DevOps Council, we decided that we are going to focus on these things. We need to create a unified developer experience. We need to standardize on tooling. We need to get to continuous compliance. Then we need to get to contextual metrics.

[00:14:07.200] Unified developer experience by itself is huge. In fact, all of these are individual conference talks, so I'm going to keep it simple and not talk about unified developer experience. That itself is a huge amount of work that has already started at Fidelity since last year. I'm going to talk about these three things: tools standardization, continuous compliance, and contextual metrics.

[00:14:33.100] On tools standardization, there are too many in number and variations. First, we are going to standardize the tools. There is no reason why you can have multiple source control systems in your enterprise. There is no reason why you should have multiple CI tools in your organization. Maybe it is technology-specific, maybe it is not, but there has to be a reason. It cannot be just out of control. That doesn't mean we are not going to look at new tools and technologies. But what I'm saying is that it needs to be controlled.

[00:15:09.500] The standard pipelines: if you consider teams building Java Spring Boot applications and deploying to Kubernetes clusters, there should not be more than one CI/CD pipeline. What's the benefit of that? If you have variations, then people will do their own things, and that is not scalable. With that, we are going to have automated onboarding and setup process. That's for the standardization.

[00:15:39.300] What that is going to give us is this next thing. We need to shift security left, meaning all source code repositories in a single source control system will have security scanning and testing built in, embedded in there. It needs to be pre-configured. The standard pipelines need to have the mandated gates that we have a list of. In the standard pipeline, it is all baked in and nobody can change the standard pipeline.

[00:16:13.200] The third thing is about pipeline execution data. As the pipelines are executing, we are going to speed out, even stream out, all the execution data and collect it in an evidence store. What that is going to give us is the next step, which is automated governance with DevOps.

[00:16:35.200] Let me walk you through the steps. The steps are kind of simple. The steps are simple, not the actual work behind the steps. The first is no production access versus production access to anyone. Anyone is bolded: no production access to anyone, underlined, bold, all that. Then the question is, how does a change go to production? The changes go to production only via pipeline. There is no other way to get changes to production.

[00:17:09.200] How does every change go from source to production? It is only possible if you have everything as source code. Application is source code. Infrastructure is source code. Your configuration is source code. Your pipelines are also source code, and your test cases are also source code. Everything is source code.

[00:17:31.300] What does that do to us? Every change is now peer reviewed. Every repository is locked down so that you cannot merge to release branch, or your main branch, or whatever branch you are focusing on, without peer review. That is segregation of duties.

[00:17:59.100] Now you must be thinking that segregation of duties is different. It's about who sends code to production. Well, that's not what segregation of duties means. Segregation of duties actually means that no single developer can change something and send it to production without a second set of eyes. That's segregation of duties.

[00:18:19.700] Traditionally, our risk and audit folks are very comfortable with the idea that a person who has not developed anything in the source code can deploy to production, and they are all happy because segregation of duties is maintained there. It's just a different person other than the person who actually coded. Everybody is happy there except the person who is actually sending the code to production, because that person does not know what is being done to production with that particular change. We need to change that with real-time audit of every change via pipeline evidence.

[00:19:04.100] The next thing we are going to talk about is contextual metrics. We talked about this pipeline evidence data that we are going to store as pipeline evidence so that we can validate our audit compliance. We are also going to use that same data to measure things.

[00:19:29.400] We are not going to get stuck on a particular system of measurement, not a particular set of techniques. We are going to measure the things that matter to us, that matter to a particular set of developers, a particular set of products, or a particular set of business units, and so on. The few things that we need to answer before we measure anything are: why are we measuring? Who are you measuring, or who needs that measurement? When are you measuring? What are you going to measure, and how are we going to measure?

[00:20:00.200] With that, we will also be very cautious about this observer effect. We are not going to measure something that will actually affect the behaviors of the people or the things that we are going to measure. We are only going to measure to improve something.

[00:20:20.200] Every request to measure something should be stated as this: as a blank, I want to measure blank so that blank. I'll give you an example. As a product owner, I want to measure a team's production access frequency so that I understand the team's technical maturity and help improve it. That will, I think, fix all our measurement problems.

[00:20:47.600] Now we are going to do this all by standing up a shared platform. That's what we are focusing on right now. In inner sourcing within Fidelity, we now have a model to incubate, fund, govern, manage, promote, and offer inner sourcing projects within Fidelity.

[00:21:05.600] Starting last year, we started five total inner source projects that launched in the last 16 months. In one of those, we have 174 features delivered within the last 16 months, with about 3,500 commits, committed by all 12 business units and with 50 active contributors. That's a big success for our inner sourcing effort.

[00:21:35.400] With that, I still need help. There are two things that I need help with. If you are using DevOps metrics, I want to learn what metrics you are using and how they're helping you. Number two, if you are invested in inner sourcing, I want to learn about your model, your process, and any challenges that you have faced with that. Thank you very much.