Host Intro (Gene Kim)

The next speaker is Randy, who I have admired for over a decade. I first met him shortly after he was the engineering director for Google App Engine. He's been at eBay twice, most recently as the chief architect, and he's now SVP of Engineering at Thrive Market.

It has been amazing to see Randy present this amazing recollection and telling of the eBay story. Here is Randy. Oh, by the way, Randy's also the most quoted person in the DevOps Handbook, and I just cannot tell you how much I admire his achievements. Randy.

Randy Shoup

All right. So happy to be back.

I want to tell you today about how we doubled engineering productivity at eBay, but somehow still didn't save the company.

Who has ever heard of eBay? Keep your hand up if you have bought or sold something in the last year. Okay, that's the problem.

This is what it shows up in, in monetary mechanisms. It's a household name, as I like to say to people I was trying to recruit when I worked there. It's a household name with a flat business.

This is eBay's gross merchandise volume over 15 years, and it is the same all the way the next three years to today. By contrast to this mostly flat gross merchandise volume, that's the sum total of goods and services transacted through eBay in the year. This is what U.S. e-commerce growth looks like. The scales are not the same. U.S. e-commerce grew 8x over this period. eBay's GMV grew 1.5x, not adjusted for inflation. Okay? So that's not great.

When I was asked back in 2020 by the CTO, who I'd worked with before, he gave me this prompt. He said, "Randy, we really need you to come back, shake things up, and bring eBay into the modern world."

So what did I have to work with? Here, eBay in 2020 looked like this. Our problem statement was essentially we were slower than market competitors. We had 400 teams, about 3,000 engineers that do customer-facing and back-office stuff, another 2,000 engineers that did platform-infrastructure-related stuff, 4,500 active applications and services. Multi-quarter initiatives across 50-plus teams were really common. Our deployment frequency for each individual application or service was about once or twice a month, and our lead time for change was 10 days.

This begat what we call the Velocity initiative, where we were trying to figure out how we could get faster than we were.

I will start from the outcomes. Thank you, Mark. Then I'll tell you about how we got there. The outcome was that we ended up doubling engineering productivity for the teams that we worked with. In the two years that I was there, and then the subsequent three years since until now, we have touched every team at eBay. So all of them have had the benefit of this work.

How do we measure engineering productivity, Randy? Thank you for asking. We used Dr. Mik Kersten's Flow Framework and Flow Velocity, which measures, for a team, how many features and bug fixes can they do in unit time, essentially.

On the DORA metrics, we were able to 10x the deployment frequency. We were able to improve lead time 5x. Even though we were only focused on the speed metrics and not on the quality metrics, exactly as Nicole Forsgren would have predicted in the Accelerate book and all the research that went into it, we also, as a side benefit, improved change failure rate 3x, and we also improved MTTR 3x.

So what did we do? Nothing magic. Absolutely nothing magic. I had no original ideas during this period at all. I simply executed the DevOps playbook that we are all learning together about.

We focused primarily on removing bottlenecks. What I would do is I would go to a team and I would say, "Hi, I'm your friendly neighborhood chief architect. I see that you're deploying once or twice a month. If I told you you have to deploy every day, tell me all the reasons you can't." And they said, "Would you like the whole list, Randy, or just the top 150?"

What was wonderful is I also owned the platform engineering organization. So when they gave me that list of 150, I said, "Great. You just gave us our backlog."

We focused on reducing build startup and PR validation times. We invested heavily, heavily in a staging environment. We automated upgrades. We streamlined processes. Most interestingly, for mobile, in the mobile releases, both iOS and Android, we moved from monthly to weekly, and now we can do daily.

How did we work? We collaborated very effectively. I had a co-leader on the product engineering side. I was the lead from the platform and infrastructure side. We had an embedding model where we had really sharp individual contributors that I would, in my phrase, send out into the provinces to evangelize, learn the problems out in the provinces, help them by actually embedding and pairing with them in their teams, and also bringing back their needs to the centralized platform group. We worked very closely together between the platform teams and the teams that we were serving out in the customer world.

From a communication perspective, we did daily leadership standups between my partner and myself, weekly team-of-teams meetings, weekly deep dives with teams, and so on.

In terms of measurement, we used DORA as our outcome metrics. We'll talk about what those meant in a moment, or how we did. But we used input metrics of various aspects of developer friction through the pipeline.

We iterated constantly. We absolutely used the Deming cycle, plan, do, check, act, all the time. If we found that something we were doing wasn't making a difference, we stopped it.

How did we scale the initiative? We started out with a bunch of pilot teams, and we worked with them for a year, which was about 10% of the total teams at eBay. Think 40 teams as opposed to 400, 300 engineers as opposed to 3,000.

Those pilot cohorts were actually already oversubscribed. We started sussing around to see if we could find a small number of teams. They told their friends about how cool it seemed to be when they were getting started, and we actually ended up having a lot more in the beginning than we thought we wanted.

The team has continued since I was there, and they've successively, quarter over quarter, added more and more cohorts. Now all 400 teams, all 3,000 engineers, all 4,500 applications and services are part of the Velocity umbrella.

We did a ton of automation: regular deployments for every application and service, even if they're not actively maintained. We automated what we called a patch pipeline, where we could consistently do infrastructure upgrades, security patches, upgrades to dependencies, et cetera.

One of the things I'm really proud of my team doing when I was not there, so I had nothing to do with this work, was integrating AI all across the SDLC. Not just the obvious things like code generation, test generation, test data generation, legacy code migrations as well, but also PR summarization, code reviews, and all aspects of the CI pipeline. They use AI to improve the CI pipeline, see what issues are occurring, make suggestions to improvements, and then implement them. Similarly, in terms of the production end of that pipeline, deployment monitoring and ML-associated automated rollbacks.

The last thing I was super proud about for my team was modernizing the mobile applications. eBay was pretty early to mobile, like 2010, I want to say. As a consequence, the iOS app and the Android app had accumulated a ton of technical debt and cruft. They were very much not modularized. They were very much balls of mud.

What my mobile architecture team did was go through successively, with each of the individual domains, and modularized the app so they could be built in individual modules and then assembled and released to the App Store. They went from a monthly capability. We were doing monthly releases. We got them down to bi-weekly releases, then we got them down to weekly releases. Now we still do it weekly, but the team has the capability to do it within 24 hours, so it could be daily at any moment.

We developed a lot of process around how to manage that with, again, many, many hundreds of engineers being involved in those apps. Then when problems occurred, because problems would often occur, they did blameless retrospectives, tried to learn from them and tried to be better next time.

Here's how we did on the DORA metrics. When I arrived, we were a very solid medium performer. By the way, that's 35th percentile. The improvements that we made brought eBay to a 75th percentile. So from a solid medium performer across the board to a solid high performer across the board. I feel pretty proud about that.

I particularly, though, feel proud about the influence that it had on the team and the culture. This was the mobile release manager, who I will say was the most skeptical person that we could improve the mobile release cadence at all. He didn't fight me, but he just didn't think it was possible. This is what he said when I recently reconnected with him: "You were kind enough, Randy, to help me adapt and see the light through air cover and rational small tests of the process."

The other thing that everybody I've reconnected with recently has said to me, and it makes me so happy, is they constantly, even three years later, ask, "What would Randy do?"

Okay, that's great. We're all done. Thank you very much.

Oh, this is still true. Why is this still true? Even though we had such successes with the engineering teams improving their ability to deliver software to our customers, why did Velocity not save the company?

I can think of three main reasons, and that's what we're going to talk about for the rest of this talk. First is strategy and planning. Next is execution and delivery. Finally, it's organizational culture.

In terms of strategy and planning, eBay is a classic case of Clay Christensen's innovator's dilemma. eBay hit on a wonderfully magical business model back in 1995 with Pierre Omidyar, and it basically hasn't changed that business model at its core in the intervening 30 years. It's tried to do lots of different things, but ultimately it's always sunk back to its original idea.

Everybody on the planet knows what eBay's about, so absolutely a household name, but unfortunately that particular class of market has not grown with the rest of the market. There aren't that many more Beanie Baby collectors or Cabbage Patch Kid collectors today. As a consequence, it's surprisingly easy for a place that should have such a first-mover advantage that competitors can come and disrupt or arbitrage eBay's business model.

If you want to disrupt eBay, the playbook is very simple. Figure out a small niche like auto parts, or musical instruments, or electronics, and be absolutely the best eBay of that little area and ignore everything else. Turns out I just named three competitors that are exactly those things.

Learned helplessness. If you have worked at eBay for this entire period of time, as most people there have, you have only lived in the world of a flat business. It's the exact opposite of living through zero interest rates, if that makes any sense, which, by the way, was this period. So what is adaptive? What behaviors as a human are adaptive in that environment? It's risk aversion, right? You're flat. What you want to do is be risk averse, and that is exactly what happens.

The other reason why the helplessness is correctly learned is that almost every user-facing feature that they've made that's been massive over the period has been met with near revolt from the seller community.

I'll share one example. When I was at eBay the first time, I worked in the search engine area. One of the improvements that we made was spelling corrections. If you spelled F-I-F-O-N-E for iPhone, we would correct it to I-P-H-O-N-E. Cool. That's super helpful. Why would anybody be angry about that? Well, it turns out that we disrupted arbitrage models from a bunch of people, where they were looking for these misspellings, buying them, and then reselling them with the correct spelling. Every single time we improved the search engine at eBay, we disrupted some seller's arbitrage model.

Okie doke. That's rough. Okie doke.

The other aspect of strategy and planning is the planning itself. eBay still, 30 years later, does very centralized waterfall planning. There is an annual, multi-month, company-wide planning cycle that involves everybody, directors and above. That's thousands of people. Everybody's writing documents and reviewing documents and making plans. The output of which is this plan from which you must not deviate.

Work can only happen if it is approved by the executive team. Work can only be approved by the executive team if it is a large enough piece of work that makes it to the executive team. If, for example, you have a smaller project, that only can survive in this environment by being tacked on as a rider to one of these larger initiatives. Does it make sense what I'm saying? Yeah. So that's odd.

Okay, execution and delivery. eBay still, after 30 years, massive coordinated releases. The cycle time of these massive releases, I mean individual things along the way, but the massive coordinated releases, time measured in quarters, measured in years. Commonly they involve 50 or more teams.

One example is eBay managed payments. People may remember that eBay and PayPal were once the same company from 2002, when eBay acquired PayPal, all the way through to 2015. They were one company. Then there was an agreement when they split that PayPal would continue to be the payments provider for eBay for the subsequent five years. That expired in 2020. It just so happens that that's when I arrived at eBay.

Previous to my arrival, there had been a three-year, multi-thousand-person effort to basically learn how to take credit cards. When you say three years times 2,000 people, what that means is $1.5 billion U.S. dollars in personnel costs alone. The end result of this was actually slower payments to sellers, because when somebody transfers money through PayPal, it's instant, because PayPal does that as its own sort of bank or transfer or whatever. But eBay was using classic banks, and so you would actually have to wait several days as a seller to get the money from somebody who paid you. You can imagine the sellers didn't feel very happy about that.

The other aspect of execution and delivery is, unfortunately, eBay has a reputation, and it's well earned, for being a feature factory. Dr. Mik Kersten maybe should go and have a chat with people there. We actually did try to bring in Mik when I was there, but that's another story.

Outputs versus outcomes. In this model, it's all about outputs, because the outcome, I mean, it's a flat business. So what outcomes, right? What are you moving?

Also, people get bonuses. What do they get bonuses for? They get rewards for hitting milestones, making effort, but not for moving the metrics or for customer value. That's sad, but true.

One example, which was shocking, shocking, shocking to me the first time that I joined eBay in 2004, was our VP of Engineering. We had a quarterly all-hands of the entire engineering team, which was maybe, let's call it a thousand engineers. We're all in one big room that was kind of like this. She gets up on stage and she says, "Everybody should be proud because in whatever this was, Q3 of 2004, we delivered 5,000 train seats to the business." Well, what in the world is a train seat? It's two weeks of an engineer's work.

What she said was, "We delivered 5,000 train seats," which means we delivered 10,000 person-weeks of work. Work, not outcome. Work. The way I reframed it in my head was, "Our organization just spent $60 million of eBay's money." That is exactly what she said.

My boss at the time, Karen Casella, who just retired from Netflix recently, we elbow each other and we're like, "Oh my God, this is nuts." Cool. Okay. Onward.

Last thing I want to talk about here is organizational culture. I would love to say that this one's not true, but it is. I don't know if Dr. Westrum ever talked to eBay. I think he probably did not. But it is the classic pathological organization in the Westrum typology.

It's very, unfortunately, deeply experienced by the people that have worked there that there's a culture of fear. Highly political, highly risk-averse organizational culture. Even acknowledging failure is seen as a little rude, at the minimum it's considered rude, but at the maximum it's considered personally threatening. Like, "Randy, if you point out this thing that's right there in the numbers, that's super rude and super threatening to me as a person." This is the kind of culture, particularly at the executive level, that it has been built as a consequence.

What do you do as an executive to grow? Build your empire. So it's a zero-sum scarcity mindset. My goal as an executive, it's not my personal goal. I did something else. I was trying to make the team better. But if I were playing the game correctly, I would have tried to maximize the control of my team, and I would have tried to maximize the number of people on my team.

The other aspect of the pathology is eBay exceptionalism. eBay's been successful, particularly in the early phase. There's developed, and has not gone away, this insular culture of not invented here. Industry-standard approaches take a long time to get adopted into eBay, if that ever happens. They're really strongly resisted and rejected by default.

One thing that contributes to this in a major way is really long employee tenures. In my team of 150, we had five people that had passed their 20-year anniversary. Wonderful people, but they had literally never seen anything else other than eBay. I'm sure some of you all have similar experiences at where you work.

Top-down waterfall organization. Again, given that centralized planning, there's really no culture of real-time autonomy for teams and individuals. If we learn something through the year of executing that plan, there's not really a way of changing what we do. There's a great experimentation platform, but it's mostly used to confirm the initial biases that went into the plan in the first place, and also to make sure that we didn't break anything.

Okay. I won't spend a lot of time on this. The woman that got me fired is this VPX. Culture of terror in this team. Of the fear at eBay, this person was really, really feared. The third rail that I touched was pointing out, in the rude way that I have, that when you talk about an agile project, it does not mean planning sprints and then design sprints and then development sprints and then QA sprints. A little bit karma, after I was fired, six months later, so was this person.

What did I learn? I learned that change needs to be top down and bottom up. I knew that, but it also needs to be middle out.

What do I mean by middle out in this situation? It means that I need to engage my peer executives as allies from the beginning. These people have the survival instinct, and I really need to engage them from the very beginning.

Also, when there's resistance, don't confront it, but route around it. Think like the internet. Better to bypass than to appear to threaten, because again, the people that have been execs at the company for 15 years, they're pretty good at surviving at eBay.

Also, see the whole board. I focused primarily, for I think good reason, on software development and on software delivery. We knew upstream in planning it was really problematic, but I think in retrospect, I would have gone back and tried to tackle that planning dragon a lot earlier in my time there.

Finally, this is a lesson that I've taken to my much smaller organization at Thrive Market: it's a lot easier to change a malleable organization.

Thank you all very much.