Host Intro (Gene Kim)

Ladies and gentlemen, please welcome back Gene Kim.

Good afternoon, everyone. Welcome back. By a show of applause, how has the day been for you all?

Awesome.

All right. Let me introduce the next two speakers. Christina Yakomin is now Senior Architect for the Chief Technology Office at Vanguard, who created the world's first index fund in 1975. They currently have over 7.5 trillion dollars in assets under management, as now the world's largest provider of mutual funds and second largest provider of electronic trading funds.

Technology has always been critical to their business, evidenced by the fact that they have over 7,000 developers. She presented two years ago during our virtual conference, and I've mentioned many times that I thought it was one of the best talks on next-generation infrastructure and operations that I have ever seen. It described so wonderfully how one modernizes a technology landscape that's been built up over 50 years.

So I was so delighted that she later presented with Robbie Daitzman, who's now Senior Manager for Intermediary Technology, who described how his group benefited from the work that Christina and her team did. So I am so excited. I finally got to meet them in person last night. And so up next are Christina and Robbie, presenting on what they've been up to since then.

Robbie Daitzman

Awesome. So good to see everybody here today. The morning's been great. We've heard a lot about different patterns we've all been adopting, the role of platforms and shared services, how all of us are trying to focus on enabling our teams to be able to deliver business value in the most efficient way.

What we're about to share with you is exactly along the same vein. We titled this talk "The Joys and Pains of Enterprise Abstraction Layers," and really what we want to be able to share are some examples of how we've approached being able to do that, and some of the benefits that we've seen from that as well.

So a little bit about us. My name is Robbie Daitzman. As Gene said, I am a Senior Manager at Vanguard. My background primarily through my career has been through developer infrastructure and cloud operations, before moving over into this role, taking a focus on how do we build out a cloud platform enabling more of our business-to-business interactions.

Christina Yakomin

And my name is Christina Yakomin. I am a Senior Architect in Vanguard's Chief Technology Office. Over the past several years, I have primarily specialized in site reliability engineering and building out Vanguard's SRE practice, which you may have heard me speak about at some recent conferences, including SREcon and QCon. In my current role, my scope is a little bit more broad now. I'm focusing on the holistic developer experience and working with teams that enable engineers on product teams all across our IT organization in every discipline.

Robbie Daitzman

A little bit about Vanguard. You may be familiar with our company from your own personal investing, perhaps from a retirement account through a 401(k) or an IRA. But Vanguard, as a company, has been around since 1975, and from our very beginning we did not have a brick-and-mortar presence. And so we were a different kind of virtual company at that point. A lot of the focus was on the operations around our phones, mail operations, as well as fax, to be able to enable investors.

We carry that forward to today, when we are primarily focused on digital moving forward, and 90% of our clients interact through our digital channels. So 90% of our 30 million clients interact with us through web, through mobile, to be able to enable themselves for their best chance at investment success.

Now in order to be able to make this happen, it takes over 750 development teams. To be able to coordinate and orchestrate across them is a feat in and of itself, especially considering this is among four different lines of business across three continents as well.

Christina Yakomin

So now let's talk about the role that abstraction layers play in all of this. We can find examples of abstraction layers at the very lowest level, right in the code that we're writing every day. Some common design patterns we're probably all familiar with are examples of abstraction. There's, of course, the adapter pattern, where you're taking an incompatible interface, wrapping it, and making it something that's either more compatible or more convenient for you to use. Or the decorator pattern: take something that already exists, wrap it so that you can add some more functionality that's related to the work that you're doing. Or there's the facade pattern: this one, you're taking disparate classes or endpoints or interfaces and combining them into a more simplified, streamlined, singular interaction point.

Now if you're an engineer and you're working with design patterns in code, it's fairly easy for just an individual to make the decision of whether or not it's time to introduce one of these abstraction layers for the health of the code base. But we take these patterns up to a higher level, and now we talk about introducing abstraction in the form of process automation or underlying platforms that our product teams can run applications on. Now we're talking about a higher level of effort and of complexity and cost. And so you're going to find that when you're talking about introducing abstraction at those higher levels, you're going to face some challenges when you're trying to introduce those.

Robbie Daitzman

Now it's not to say that doing this isn't without challenges, and some of the common things that we can think through are, first and foremost, hey, isn't everything a bit of an abstraction layer already? When we think about the different libraries, frameworks, shared services that we all take advantage of.

When we want to introduce another abstraction or new platform, you really need to be very intentional around what is that value that you're able to provide there. Is it going to help accelerate speed to market or writing that first line of code? Does it help to automate areas of compliance or governance for your organization, taking that off the top of mind to developers and helping to create that paved road for them going forward?

This really puts the focus on where do you want to spend your time on what may be a context investment. It may not be core to the heart of your business, but enables you to be able to focus on that core by reducing work and effort in those areas.

One of the things that we've seen be a problem, and I think all of us have seen this example, whether we're talking about internal shared services or even external service providers, is the creation of a bottleneck. This bottleneck could be from a speed perspective, or even just say, hey, there is something there, but it's not meeting my needs. I know that you've expended effort there, but I'm going to go around and do my own thing. These are all things that we need to be considering and taking into account when we ultimately want to answer that question: is this somewhere that we do want to invest in, or do we want to take a look at another opportunity?

And at the end of the day, what we want to be focused on is all of these different pressures that our development product teams are facing. They have all of these opposing forces coming at them. How do you try and tackle things like production support, creating new features, being able to manage technical debt, and give our engineers also time to be able to upskill and be able to mentor and grow in their own careers as well?

Being able to balance all of these while at the end of the day focusing in on client happiness, whether that happiness comes from increasing reliability or availability of an application or shipping that net new feature, being able to balance across all of that and think through how do we start taking some of these arrows off of that screen by introducing these paved roads, these abstraction layers, or platforms, to be able to get them to where we want them to be producing value.

To be able to answer that a little bit more, Christina's actually done a little bit of math.

Christina Yakomin

Well, this first slide I didn't have to do too much, because I borrow here from one of my favorite XKCDs, which talks about how long you can spend automating something before you've spent more time than you save across five years. And this chart works really well if you're thinking about it from the perspective of an individual. If I am going to go spend some time automating something that's then going to save me time down the road, I can use this chart to figure out how long I can spend on that automation.

But let's take that into the implementation of abstraction layers for the entire enterprise. And if I'm not just saving time for myself, what if I'm saving that time for every single development team? And what if you have a hundred of those teams? Well, then I did some math, and now we have the numbers looking a little bit different. Something to keep in mind about this chart here is this is not calculated in terms of business days, weeks, and months. So when you see here that you could save every development team one hour a week, and you can spend up to three years doing that before you spent more time than you saved, about three years is not business years. That's 24/7, 365, three years of effort. And suddenly it becomes very apparent that saving that amount of time is justifiable. The ROI is there, and we're going to take that opportunity just about every single time that we can.

So it makes sense that Vanguard has tons of abstraction layers, and I'll let Robbie highlight a few of our favorite examples.

Robbie Daitzman

Yeah, and what you can see here is just an example of some of our favorites as we think through, have we started doing this at Vanguard? And it's been a journey. Not everything went well right off the bat. We'll get into a little bit of that as well. But being able to focus in on where do we see this philosophy coming into play. I will give a shout-out to one of my teams right now focusing on the authentication authorization abstraction for the intermediary space. But what we really want to be able to share with you all today is a little bit of a dive into two specific examples and really getting to the heart of what we're here talking about today, which is how are we enabling developers to more effectively and efficiently produce that level of value and be able to focus on what is it from a business outcome perspective they're trying to achieve.

So with our first example, we're talking a little bit about how we've been able to mature in the serverless computing space. I'm sure many of you, as you've gone through and adopted different cloud providers, have begun on the space as well. And we can definitely do a whole separate talk on the benefits of serverless. But here what we're talking about is what we call functions as a service platform, or FaaS. And really what it is at the end of the day is it's an abstraction layer on top of AWS CloudFormation. What it's able to do is accelerate the time from actually the idea or creation to when a developer can start writing code. What FaaS enables developers to do is to create cloud blocks, or modular components of serverless resources, that they are able to configure and create their own serverless application with.

What's interesting is you can see the chart on the slide, and on the orange line you see what we call custom Lambda development, or actually using AWS CloudFormation, of which Christina and I were pretty strong contributors from the start. But what you can also see is a really astonishing growth as the FaaS platform gained maturity and really accelerated adoption. Developers are voting with their feet, saying where they want to be spending their time and energy.

Christina Yakomin

So here you see an example of what it looks like to be a developer leveraging the functions as a service platform to write a cloud block through this YAML template. I know the text is a little small, but it's really about the length of the template here. We have 35 lines of YAML to define a DynamoDB cloud block. This is really similar to something I wrote just a month ago for a project that I was working on, came out to exactly 35 lines. And then when I ran that through the CI build, the output was CloudFormation that was generated that would later be deployed, and the generated CloudFormation template was over 2,000 lines of JSON. I could not believe what I was seeing and how much effort the functions as a service team had put into building out their platform and these templates, and how much was baked in by default.

Two thousand lines definitely sounds like a lot, and some of that is the complexity of parameterization and customizability. But a lot of it was best practices by default, security controls built in by default, that I didn't even have to think about as the engineer writing the code.

But initially, I really didn't like this platform. Robbie mentioned that we started out writing custom Lambdas before this platform even existed. We were some of the earlier adopters of serverless application development at Vanguard, and I thought that this was way too much abstraction.

When I was trying to write my first Lambda function using this functions as a service platform, I went and asked one of my friends on the platform team, hey, where in the template am I supposed to be defining the IAM role and the permissions that my Lambda function is going to need? The response I got back is: you don't. That's all figured out for you. We auto-generate the CloudFormation with the role, and we assign policies to that role for your Lambda to use based on the least-privilege principle.

That's actually a really good thing. But in the moment, I felt the lack of control. I felt like I wanted the customizability so I could name my IAM role and then go find it in the console and all of those things. But ultimately ways we mitigate that risk of too much abstraction here is the generated CloudFormation is there for me if I'd like to see it. So it's not like I don't have the option to know what's being defined in my IAM role.

But a lot of the time, I really want to think about it, and it's best if I don't have to think about it. If I don't have to be an expert on all of these controls and the requirements as they change over time. So by the time I had to go through this again, just last month, I was so relieved that I didn't need to reskill on what had changed in the past few years, and instead was able to leverage the template and get all of the latest and greatest best practices right away.

Robbie Daitzman

So as we move forward to our second example, again thinking still from a developer's point of view. For us at Vanguard, I'm sure many of your companies are the same, there are processes that you need to follow as you create a new project, create a new repo, whether it is doing things like getting the necessary approvals, going through the correct governance processes, or getting something added to the CMDB. There are some disconnected processes in some cases and also expertise that a developer would need to build that, again, isn't a part of the core value that we're looking at them to be able to deliver.

And so as a way to counteract this, we actually saw two different offers start from really a grassroots perspective around how do you get the best of the governance aspect while not having that be an additional tax on every new project at the same time? First Time Setup was the first one that was created, and there what it really is, it was a UI-based form with workflow in the back end to help automate a lot of this. In contrast, Dexter is a very developer-centric CLI tool that was created. You can see these are two tools going after the same problem at the end of the day, but the success that you see from it is that in some cases the time to that first line of code written was able to go from weeks even down to days or hours. The value for the enterprise, we think through that, is immense.

Christina Yakomin

And we can quantify some of that value. I pulled here some metrics from the First Time Setup and Dexter teams. On the Dexter side, they calculate that over 35,000 developer hours have been saved since the inception of the tool and its automation, which equates to nearly five and a half million dollars of developer salaries. Now, obviously, we're not saving that money. We're still paying the developers their salaries. But instead of all of that time and money being spent on something as contextual as application setup, that time is put back into feature development or improvements to application availability and performance.

You can see some numbers at the bottom of the screen and the right-hand side of the screen that demonstrate just how much this tool has proliferated Vanguard's development product teams. Over 2,500: that's the unique users of the Dexter CLI just in the past month. So that gives you an idea, one, for the scale of Vanguard's IT organization, but also just how many developers have flocked to this tool. And those 2,500 unique users account for nearly 12,000 unique invocations of the CLI, again, just in the past month alone. So that's how we're accumulating all of those developer hours saved so quickly.

Now on the First Time Setup side, that's that UI workflow. We can also see an example of how are we saving so many hours, where are we recouping that from? Well, to call what we had before an application setup process is probably overstating it. As Robbie mentioned, it was very disparate and there were a lot of different processes that you needed to follow, and to do that very regularly took up to a month before you would be productive and you were able to write code and push it to production. Now for the First Time Setup process, the fastest requests take just 45 minutes, with the median request for a new application taking just over one hour.

There are still some risks here though. It's not the same ones that we talked about with the functions as a service platform. No one has yet complained that there's too much abstraction here and that developers really, really need to know how to put all of the metadata about their application into the CMDB.

Instead what we see here is the risk of duplication of effort and rework. And the reason that I say this is we have these two disparate solutions because we didn't solve this problem up front through a centrally funded shared service, through something standardized. So when you don't do that, what you end up seeing is highly motivated engineers in pockets creating solutions that work for themselves. The Dexter CLI tool came out of a team that's working on a product for our retail business line.

Initially these two offers really were competing more than they were coexisting. But we saw how quickly they were taking hold, and so rather than let them continue to operate in this odd competitive landscape, we coalesced the two products into something that's now actually a little bit more codependent with a shared feature set. The UI workflow invokes the CLI tool behind the scenes, and the CLI tool is using Lambdas that were originally written for the UI workflow. Communication between the two teams is incredibly frequent, and they share management of their backlogs to ensure that the feature set stays equal between the two.

And ultimately we continue to support the two offerings, and that's because the engineering teams really appreciate them. For the UI workflow, folks really like that their product owners or project managers can go take care of the task of application setup for them without needing to figure out how to get access to a repository or an IDE. And the engineers, if they have to take on this task, appreciate that they're able to do it right there in their IDE with some commands in the terminal without having to context-switch and open up a browser.

So to summarize what we've talked about here: too much abstraction can have negative consequences. We know that. We know that there are risks associated with the introduction of abstraction, but we don't want to be afraid of the added complexity that comes with abstraction just on principle and refuse to introduce homegrown abstraction just for that reason. We want to make sure we provide a standard path for teams to leverage, but still leave some room for deviation so that we avoid becoming a bottleneck. In our experience, when the standard path is the easy path, teams will flock to it.

And finally, the major takeaway we want you to leave with is that the ROI of reducing rework is extraordinarily high when you're working at enterprise scale. Those numbers were for a hundred development teams; we have over 750. So if we can even save a little bit of time for our developers, it is incredibly easy for us to justify.

Robbie Daitzman

And so where do we need help still? First, feel free to disagree with us, challenge us, share your stories with us around how you're taking advantage of different abstraction layers, different platforms within your companies. Keep finding ways to experiment with this. Contribute back to the community. Maybe you will find something new and novel in terms of a pattern or a way to be able to accelerate developers. And then finally, Vanguard is hiring. And so if our story is something that you would want to be a part of, feel free to talk to us as well. So with that, thank you very much. Thank you.