Host Intro (Gene Kim)

So earlier this year we hosted an event where we had about 50 of some of the technology leaders I admire most. And what was so surprising to me was somewhere between a quarter and a third of them were responsible for some aspects of managing their enterprise AI rollout, just as you've heard earlier today.

But one of the stories that blew me away the most was from Brian Scott, currently Principal Architect at Adobe within the Cloud Ops organization. So his remit is vast, which includes assisting teams with engineering and technology onboarding to help them move fast. But what is incredible is that he now co-leads the effort, and is collectively responsible for creating the generative AI governance strategy for Adobe, which I believe is now the third largest software company in the world. He's working with this breathtakingly broad stake of stakeholders across the Adobe enterprise.

And I'd love this because he's gonna talk about how he's dealing with the infinite demand of things gen AI related for the thousands of creators within Adobe, and how they're trying to both maximize liberty but also maximize responsibility. Here's Brian.

Brian Scott

Hey everyone. How you doing? Looking all good there. Everyone having fun? How was your first day? Woo. First day was good. Alright, now from here on out is Gen AI all the way down. So sit back, relax, and enjoy the ride.

So I'm Brian Scott. I have the pleasure of supporting all of our engineers over at Adobe. We are primarily a creative company supporting multiple different products, roughly at the moment, 50-plus. And with that comes a lot of responsibility.

Now, with that, Adobe has over 41 years of innovation starting from PostScript, when our founders created and founded Adobe, all the way through Macromedia Flash. The creation of the PDF — if you haven't heard of it already, it's this awesome portable format — as well as the acquisition of multiple different companies, including Omniture, which has now been rebranded as Adobe Analytics.

So whenever you're out in your day-to-day, interacting with the world, watching a movie, maybe buying a movie ticket, maybe drinking your favorite beverage or soda, you're interacting with some Adobe endpoint at some degree.

Now you've probably all seen the immersions of like Creative Cloud, right? And the coalition of multiple different products being offered up under a single umbrella. Well, Adobe is way more than that. We actually offer now not only creative products, but products that allow you to engage with your teams, allow your customer journeys, and understand the pathing of how your users integrate with your own products.

Now, we've also started to simplify some of our creative tooling with the emergence of Adobe Express and also Firefly. Firefly is our answer to text-image generation. And with this, Adobe has been leveraging AI for a pretty long time now, but where we started to struggle and started to rethink our strategy was the adoption of external tools and models.

Now with that comes a lot, again, of responsibility, but also asks of many different teams. Yes, it's great, and it's able to move fast when you're trying to adopt your own internal created AI, but when you start thinking about adopting external AI, you don't really understand all the factors involved. That presents itself with multiple different challenges and risks.

So I'm gonna kind of give Adobe — give you some understanding of Adobe by the numbers. So roughly 30,000-plus employees and growing, right? Roughly within, again, 41 years of just innovation, over $19 billion in revenue as of last year, and also roughly $95 million given back to the community.

Now with that said, out of that 30,000, we have a very large engineering organization. These are teams that are not only empowering and building up across all of our 50-plus products, but we're also offering up services internally to other teams making them successful, right? Things like identity service, maybe vault as a service, providing them with sequence management. How do we deal with cost management in the cloud, et cetera, et cetera.

So kind of give you some background on myself. Now, my colleague Dan Neff couldn't be here today, but I wanted to keep him on the slide because I think it's very important. Him, me, and him have been dealing with this strategy and defining what this means for Adobe over the last 12 months.

Now we are architects. We sit with an organization called Cloud Operations. Cloud Operations is chartered with maintaining the relationship between Adobe and our public cloud offerings, but also our private cloud and everything in between. This includes cloud cost optimization, but also Dan and I have the freedom of going across Adobe and solving enterprise-wide challenges. We're able to drop ourselves into any team, able to assist any executive with any challenges that they're trying to solve, and really trying to understand the entire landscape across not only vendor management, technology onboarding, but also working with our security and legal teams and finance teams to help them be their best selves and to exhilarate their success.

Now with this, Dan and I primarily help teams again onboard with technology and tech services. We bring decades of operational experience to Adobe — me coming from Disney, Dan coming from Friendster and Facebook — and we lead onboarding in three key areas.

The way that Adobe manages software internally is we break out different types of software into categories. So at the moment, Dan and I own a category called Engine Dev. Some example tools that fall into these categories could be GitHub Copilot, it could be your favorite web service, it could be Java, it could be Python — anything that falls in the dev and engine category. We typically try and foster and nurture that as a garden, maintaining that portfolio of software.

I also lead the open source office within Adobe. So maintaining that portfolio of software, ensuring that teams can actually contribute back out to open source in a very quick, fast, and secure manner.

And we also recently now have been given the generative AI category — so managing that large portfolio of tools and services across Adobe.

Now, a funny thing happens when you're working with SVPs, VPs, directors, and you're the only IC in the room — for some odd reason, you end up walking away with more work. I don't know how that works out, but it just does.

So Dan and I, with our collective experience of supporting all these different teams, including VMO, finance, and legal, we were tasked with: Hey, how do we develop a fast and easy strategy for Adobe to adopt, to allow teams to move fast? And so with that, and also supporting all of our existing responsibilities being architects, right? Had to come up with something that was going to allow Adobe to move fast.

Now obviously there's a lot of risks when it comes to taking on new large language models, especially in this vast landscape that's changing literally almost every day. I would probably say what, 10 new models probably just got launched while I've been on stage in the past 10 minutes. And so as quickly as these services are actually evolving, we want to make sure that the strategy is flexible enough to adapt to the needs of Adobe over time, but also to adapt to the changes in technology over the next 20, 30 years.

So with that, we created a process that required no dedicated staff over its first year, and we recently just onboarded the first FTE to support this process.

So there's a lot of tension. Early on, when we were wanting to adopt external services, tools, and models, we had a lot of tension between our teams. Legal teams wanted to basically allow us to balance maximum liberty with maximum responsibility. We had product teams that wanted to go out and solve customer needs. We had teams that wanted to onboard technology really quickly and really fast, because in today's world, if you can't adopt technology fast enough, you're gonna fall behind. They had POCs, prototypes, and hackathons — geez, the hackathons. Almost every week we had a new hackathon being launched where a team wanted to try out some new tech, or experiment with something, or even build something. We had vendors that were just slapping the label "AI" on it, but also trying to achieve integrations with their existing customers to allow them to even achieve better business value. So we had all these things from a developer perspective.

Now, on the responsibility side, we had legal, security, and finance that needed us to not only adhere to security standards, but also dealing with the ever-changing terms of use across every major version of these models. These TOUs are just flying in from left and right and they can change at any given moment. We had licensing issues, EULAs had to be accepted, and finally aligning with our government policies. And so with that, we needed to move as fast as a developer, but with the seniority of a lawyer.

Now, I added the "tech" right out here just to kind of show you that this problem is much bigger than just AI. It involves around any form of technology that you're trying to adopt.

So with that, our hero steps into this story. We developed this framework called the A-through-F framework. This framework alone drove the single biggest change across Adobe. Not only did it allow us to align all stakeholders in the entire process — from security, legal, privacy, and ethics — but it allowed our product teams to quickly understand what it meant for a use case to be defined within Adobe. So let's walk through it.

We have our team for the example here. I chose the customer support engineering team. This could be any team within your own organizations. This can be the Adobe Acrobat team, for example, with the objective to improve customer quality of service.

We then define the audience they want to target: internal customer service reps within Adobe. With the input data of published documentation and internal runbooks.

Now data is probably the core critical piece here because that defines everything. So when we ask someone to explain what their use case is — when it comes to the input data, we have four different data classifications. There's public and/or synthetic data, right? Easy. There's internal data. Then we have confidential data and restricted data.

So as teams go out filling out their use case, we then ask them: hey, which technology are you targeting? So in this example, I leverage Azure OpenAI. Now given just as context alone, security, privacy, and legal now quickly have a pretty good understanding of what the team is trying to achieve, the technology that they're trying to leverage, and the data classification that they're trying to choose for that experiment or for that implementation.

Then we have the output. What is your output? What is the classification of the output, and how's it going to be used?

This gave us a really insightful data trove of information that we can now present to teams.

Now, there was a point in Adobe's lifecycle when we introduced this framework where the use cases just started to grow and grow and grow. I think in the first month alone, we got roughly over 400 use cases submitted by multiple different teams across Adobe — not just engineering teams, mind you. Finance teams, legal, even the data analysts had their own use cases to make themselves productive.

Now there's a very important piece here, which is the audience. We divide use cases into two different categories. The first category is internal use cases, and the second category is external use cases. These are use cases, for example, of AI that's being included into our own products. And I'll kind of tell you why that's important in just a moment.

Now, as the use cases were being generated, we had to provide early warning systems to all the stakeholders involved throughout the review process. If you ever watch Twister, right, Dorothy kind of gives you that early warning system when a twister is going to be approaching. We wanted to provide the same thing to our legal teams and to our security teams, so they can act fast and provide fast approval for these use cases.

But we had some problems. These use cases were coming in too fast for us to actually keep up. We're only human, right? We had to manually review these — and that took time, that took effort, especially when we're talking about internal data or confidential data.

So we started to provide these sliders to allow teams to calculate their risk score. If you're less than, let's say, 1.5, that's probably going to give you an approval within about a day. The higher it goes, closer to 3, well, that might actually take a few weeks. And that put teams on edge. They wanted to move fast. They wanted to deliver customer satisfaction quickly and fast.

Now we started to transition from the chaos. We leveraged some core principles from Wiring the Winning Organization. We stopped and we did retrospectives of our process. We started to treat the process as a product, because after all, if the process breaks down, guess what? Team communication's gonna break down. Your teams aren't gonna actually be able to take action and deliver their products on time or their features.

We wanted to put up-front quality checks so teams understood: hey, your use case — your risk level is pretty high, but guess what? If you just simply change your data classification from, let's say, confidential down to public and/or synthetic data, you can now get through your hackathon on time and present to your judges. Or you can get through that experiment that you're trying to actually like get done.

And then we started to provide a tech radar to allow teams to understand which technology was actually approved. This was actually huge because before, teams would submit their actual use case, they would then look at the radar and understand if a technology was already pre-approved, or if it was still in review. Now remember earlier when I talked about how we have different categories of software within Adobe — those particular different category teams approved the technology that comes into their portfolio. This not only helps us reduce tool sprawl, but also help us understand if certain tools within our portfolios are offering some of the same features.

We started to simplify. Now remember how I said multiple different legal teams had their own intake for how they would actually onboard AI? We actually sat down with all the stakeholders and we consolidated everything into a single endpoint. What this meant was, instead of having security, legal, and privacy start to email all these questions to these teams to better understand their use case, we instead brought everything in and made a single entry point that allowed all these questions to be answered up front.

Now I know I'm running out of time, so I'm going to quickly go through these. We started to think about how do we separate tech onboarding from use cases. And then we started to actually also figure out how do we amplify high-value use cases versus low-risk use cases. And again, that risk score really was pivotal in that decision making.

Now here's some winnings and learnings. We wanted to, again, create early warning systems. Leverage your wikis, leverage your documentation tools to get the early warning systems closest to the left as possible. Create shift-left traffic-light filters. Allow your teams to understand what is it gonna take for them to actually onboard that technology, and what it means to actually get past all those risks.

Automate where possible. Again, early on we leveraged Microsoft Forms to actually intake all these use cases. We then moved over to a more automated platform so that way all the questions around privacy and security could be asked up front. And then we provided that tech radar so teams understood which technology they just couldn't touch, and technology that was already approved.

Be transparent with your users, and also focus on MVPs, not perfection.

Now here's where I'm looking for help from all of you. If you're developing your own gen AI strategies and you're facing challenges, I want to hear about those. I want to be able to also understand how are you dealing with the model zoos that are being created around you, right? We have new models being generated almost every single day, and keeping teams in front of that is key.

Also, I want to hear about best practices on platform engineering and productivity tools. So if any of you have any interest on those, please let me know.

And with that, I want to thank you.