Disney Global SRE – Creating Digital Magic
Let us tell you a story about a century-old organization that has scaled its SRE practice to ignite digital magic across the globe. This team of SRE Jedi Knights is on a mission to foster curiosity, communities of practice and technology awesomeness while venturing where no SRE has gone before. In this talk, we will deliver epic stories of successes, setbacks and failures while pushing large-scale platforms to their limit and delivering the best in-seat, digital experiences, products and content to our guests and subscribers across the globe.Scaling SRE & Embedded Teams in the EnterpriseChampion best practices in Infrastructure, Automation and Delivering with reliability at the core.Building a high functioning Global SRE Organization that fosters a great culture, believes strongly in automation, removing toil, IaC and improving processes.
Chapters
Full transcript
The complete talk, organized by section.
Host Intro (Gene Kim)
All right. I had mentioned that one of the key themes for this year's conference is to be a huge reunion. So we want to assemble some of the people that I've admired most over my career, and I know many of you admire as well. These are people that before the pandemic I would routinely see multiple times a year.
Without a doubt, one of the people on the top of this list would be Jason Cox, currently Director of SRE at The Walt Disney Company.
I met him in 2013. I had been looking for this person because someone was sending all these Disney ops engineers to all the same conferences I was going to, like Velocity, ChefConf, and so many others. This first picture is when I first got to hang out with him in 2014 on the Disney Glendale campus, and I saw so many amazing things there. Among them was that I got to see him thanked more people more times in one day than I think most ops practitioners get thanked in their entire career.
Over the years Jason's presentations have inspired so many ops leaders to rethink how they are organized, including Fernando Coronado, now currently VP of Digital at adidas, who will be talking tomorrow.
Jason spoke at the first DevOps Enterprise Summit in 2014, and it was so memorable for so many reasons. One reason was that his presentation wasn't allowed to be recorded or live streamed, for enterprise reasons so familiar to so many of you. So I'm so delighted that due to his relentless lobbying over nearly a decade, Disney now allows his presentations to be recorded. Without further ado, Jason Cox.
Jason Cox
All right. Wonderful. It's so good to be back. So good to be in person. Isn't this wonderful?
You probably know about Disney. But if you don't, I thought I'd give you a long history of where we came from. Instead of doing it from my mouth, I'll do it from a video. That's what we do at Disney. So watch this.
01Disney Technology Video
Now your host, Walt Disney.
In our modern world, everywhere we look, we see the influence science has on our daily lives. Discoveries that were miracles a few short years ago are accepted as commonplace today. Many of the things that seem impossible now will become realities tomorrow.
Today at Disney, we have amazing teams creating magic with technology. Technology is part of the DNA at Disney. It's always been part of the DNA at Disney, but it's always been in service of telling the story. If the guests walk away talking about the technology, we haven't done our job. It's all about the experience.
I care about creating technology that really builds wonderful experiences, and I think there is no company where you can do that in the way that you can here. It's an awesome, unique opportunity to connect compelling stories that we have at Disney with audiences all over the world through delightful product experience and through amazing technology.
We believe that a close partnership between art and technology is essential to creating compelling stories with engaging characters and believable worlds. There's so much passion on the part of all of the teams behind what we do to make sure that what we're building lives up to that Disney expectation. With a big idea, there is very little friction in making it into something through the use of technology. With industry-leading technologies like augmented reality and machine learning behind the scenes, we are creating awesome experiences.
Even to take content that people really want to consume, to give them exactly what they want by leveraging the power of data, to me is really super exciting as a technologist. To continue to move at the very front edge of technology in so many different areas.
Tech at Disney is so boundless, to take us further, to disrupt entertainment for our guests. I believe technology will power the magic.
During the last few years we've ventured into a lot of different things, had the opportunity to meet and work with a lot of wonderful people. We hope that you will join us and that you'll find here a place of knowledge and happy.
Jason Cox
Hey, what do you think? I'm going to invite you in to be part of the magic today. I want to talk to you a little bit about my group, Global SRE. We're a central shared-services team providing reliability engineering to the whole company.
I'm going to give you three takeaways. I'm trying to deliver some of the learnings that we had that I think you could use. That's what we're going to be talking about today.
This will set a little of the frame. Here's the company. If you start there on the left, you can see this is our creative powerhouse of the company. These are artists, our storytellers, and they're creating those magical stories, those experiences. Then they're shipped over to the next part of our company, where we connect it to you, to our audiences. We stream it out. On the far right, that's where we turn those stories into concrete, making those physical experiences that you can enjoy, and we can connect our guests to those storylines that they can be part of.
Then this big green bar at the bottom, that's the group I'm in. That's our shared-services team, our corporate team, and we support this incredible kingdom of magic across Disney.
But here's the thing: there are problems with centralized services.
Let me ask you a question. How many of you are part of a shared-services team in your organization? Let me see that. Wow. There's a lot of family here. Here's a second question. How many of you have to consume shared services from your company? How's that going for you?
Here's why I say this. You've seen this before, but this is so true. As a shared-services team, we often say, we're here to help. And this is what our businesses see: yeah, sure. In fact, this is how they react: seriously? You're here to help? Sure.
That's a problem. Here's some of the reasons why. When you start to dive into what is their mission, by how they are reacting, it's something along these lines: building world-class, elegant, beautifully executed solutions to problems that you don't even have. If you've ever seen that, can I get a witness? Has anybody seen that from a shared service? This is a problem.
Don't worry. If you don't need the service, that's fine. We'll still charge you anyway. You see that too.
What's a good model? What's a better model? What's a better way for us to offer our shared services to our organizations?
We looked at that and said, what are we going to do with SRE? Several years ago we began this journey to say: what if we embedded engineers into the very product teams, into the businesses, into those technology groups across the company, and we help them elevate their practice wherever they were?
I know what you're thinking: yeah, Jason, that sure looks like a Death Star to me. You wouldn't be wrong. It was not the intent. The diagram is a Venn diagram. It's just the way it worked out. I've heard the same thing from even the Lucasfilm team. They said, hey Jason, by the way, that's looking like a Death Star.
But they also said this, and so did several others across the company, over and over again: you're not like the other shared-services teams. Your team actually understands our business.
That was a clue. We started hearing all the time, Gene talked about this: people would approach me, send me an email, see me in the hallway, and say thank you. Jason, your team is fantastic. Thank you. You saved the day. Thank you for helping elevate our innovation. We hear this all the time from across the company. We're valued. Thank you for what you're doing. Thank you. You're fantastic. It wouldn't have happened without your team. Your team is unbelievable, and frankly, it shouldn't be unbelievable, should it? As a shared-services team, it should be believable. Your team brings innovation and creativity. Thank you for being an awesome partner.
So what are those lessons that we have learned to get to that point?
Lesson number one: listen. This sounds so simple, but I have to tell you this. I go to all the different business units across Disney. I sit down with the technology leaders, and I ask them, what is it about these other shared services? They told me over and over again: they don't listen to us.
Would you please tell them: don't come and tell us what you're offering. Come and listen. Understand what we need. Understand our business, please.
Know what the business is all about. Understand how they ship the magic. What are the complications to get there? Know the mission. What's the North Star? What's the objective? Where are they trying to go? You have to talk to people. You have to listen to find out what that is. Become part of the Rebel Alliance. Join the businesses to help them get to where they want to go.
This is the other important thing: you have to know people. Listen to the people. They're not on the other side of a ticket window. These are real humans like us. Listen to them.
When we started talking to different teams, we found out some things. What are they looking for? They're looking for help. They want help to create and ship content, products, and experiences better, higher quality, faster, getting it to market, lower friction, safer, protecting our treasures of the company, our company assets, and protecting our guest data. And finally, happier.
We're supposed to be shipping happiness, but happiness only comes from people that are happy. If our cast members, our employees, are not happy, how are they going to ship happiness? How do we remove the friction from the system to make it better, to make their lives better, so they're happier?
Listen. That's the number one thing. Know the business. Know the mission. Know the team.
Here's number two: have empathy. If you want to connect to the businesses, you need to know their frame of reference. Put yourself into somebody else's shoes. Understand the problems and challenges that they face.
That's super important to be able to go from their imperial Star Destroyers coming in to help you, to this: a family. We're connecting, one team, one family, pulling it forward together. Empathy drives that.
By the way, it's not this. I know there's gold in any mountain of dirt, but listen to me: this does not create empathy. This does not help you connect with other people. It creates silos. It creates walls. We see it over and over again, and we don't want that. We want people to connect.
It's not this: you need help, submit a ticket, fill out the 47 different fields, and if you do one wrong we reject it. Send it in again. Fill out ten more task forms, and then we might actually help you. That's not help.
Instead it's this: don't wait for them to come to your ticket window. Go to them. Change the direction. Go find out what they need. Proximity.
We saw this over and over again from our model: embedded, being close to the businesses, understanding what they're building, helping them do that work together, understanding the pains they face, and helping relieve it. Provide the real help in real time because we're there.
Have empathy: shared mission, shared struggles, shared wins.
Here's the third one: actually help. I'm serious. So much of what gets delivered from a shared-service team is theater. We're looking for real help.
I love this quote. Taiichi Ohno says this: do something to help them. If you do, people will come to expect that you can help them and will look forward to seeing you again.
Let's reverse that for just a second. If you're a shared-services team and nobody looks forward to seeing you, in fact they try to avoid you, you may not be helping. I'm just saying. Could be true.
Actually help. Here's the idea. We looked at all those opportunities where the business had a need that we could help with, and we plugged in. Over and over again they said: help us connect to other technologies across Disney. Help us connect to new technology. Build a community. When we have a need, burst into our teams. Flex in when we need horsepower. Embed more of your team into our group. We've heard this over and over again. So that's what we did.
First, be a community partner, not a command tower. We're corporate, we're here to help and tell you what to do. Nobody wants that. Help build communities of practice around great technology. We do that through things like Jedi Engineering Training Academy. We help technologists connect to other technologists, businesses to businesses, learning from internal and external experts.
We go on location. Teams in all these different locations to help ship great products, content, and experiences.
We look at all of our cast members and what they're working on. How do we hand them back time? Because if we're going to unleash magic, we want to hand them time. Give time back to the artists so they can create new environments, new characters, new experiences, great experiences on the screen.
All of that amplifies creativity, and we've done just that: being able to unlock things that are gross, frustrating, slow, and handing back the time to the business. As a shared service, that's what we should be doing.
All that results in some really great award-winning content. There's amazing artists across Disney doing incredible things. By the way, Grogu does say yes. Thank you. I appreciate that.
Let me tell you one story as an example of how we execute on trying to help. We had Marvel Studios come to us. They were trying to do this charity event. They really wanted to get behind it and support it. As part of the release of Doctor Strange, they wanted to get this out to all of our audiences so they can connect in, upload some user content, get these great experiences, and promote the charity event.
But we needed it done in less than one week. We were the only team, kid you not, we were the only team who could say yes. Not because we're brilliant or heroic or because we were going to burn all the midnight oil. No. Because we knew the business. The business needs to pivot like this all the time. So we plumbed in platforms, automation, technology that allows us to pivot, get stuff out fast, and hyperscale to support these great media events over and over again. Because we knew the business. It makes all the difference in the world.
We've done the same thing with attractions, handing back Imagineers the opportunity to sit on a ride vehicle, see a problem, a defect, check out the code, pull open their laptop, edit, upload, see it build, and on the very next cycle see their code in the attraction in production. What does that allow for? Greater iteration to polish, improve quality, experiment, and deliver more magical experiences for our guests.
We sent that into orbit and back on Earth. We're reminded we've got to actually help. Build community, build trust, and build magic together. Go and see.
There you go. There's the three: listen, have empathy, and actually help. Those are some lessons we learned.
Now I've helped them; I'm asking you to help me. I know I'm missing something here. Maybe something really big. If you've got a model, a shared service that you said really works well, maybe it's different than mine, and it probably is, I want to hear about it. Tell me what I'm missing. Tell me what we should be looking at. Tell me about other SRE models that we should be evaluating inside Disney.
With that, thank you. I appreciate this opportunity to be back in person again. This is so great.