Shakeel Sorathia

First off, I want to ask a question of everybody, which will be: how many people have heard this sound before?

Yeah. So can you guys believe that in the mid-'90s we used to wait that long just to get connected to the internet? That's kind of amazing, right? When you really think about it now, your phone is always on, and everything is just instant gratification.

Ticketmaster kind of had the same problem many moons ago, where it took us a long time to deliver software. So this is the story of how Ticketmaster went from six weeks to six minutes, almost, because we're not quite there yet. We still have a ways to go, but that's our goal.

My name is Shakeel Sorathia, and as you heard from Gene, I do go by Shak. A couple of questions that I always get about my name: no, I was not named after Shaquille O'Neal. I was actually named after a poet that my mother loved. And yes, I know that I look taller, darker, and richer on TV.

So the only thing that Shaq and I share in common is the fact that we both can't hit free throws. Anyways. All right, moving on.

A little bit about Ticketmaster. You may have heard of us. We sell a few tickets. We're now a part of Live Nation Entertainment, which is a global entertainment company. We have artist management, we promote events, all of those things.

Two questions that I always get about Ticketmaster when I tell people that. The first question is, "Can you get me free tickets?" Or, "Can you get tickets for X, Y, Z event?"

A little bit about Ticketmaster is that we're basically like a middleman. If you think about eBay, they don't actually own the inventory, so neither does Ticketmaster. We actually don't own the inventory. It's all the inventory of our clients, the promoters, the artists, things like that.

So no, I can't get you free tickets to the sold-out event for One Direction, whatever it might be. But once in a while, because we're part of Live Nation, we do have a lot of our own venues, and so we get tickets for that. But yeah, we can't get tickets.

And the second question that I always get is, "What's up with those bleeping convenience fees?"

So a little secret about the convenience fees. First off, what I tell everybody is, let me tell you guys the secret of how to avoid paying convenience fees. Very simple. All you have to do is wake up early, drive yourself to the venue, the box office, wait in line, and buy those tickets, if you get lucky. So that's how you can avoid those convenience fees. Obviously, it's quite convenient then to get it from your phone or your desktop.

But the other thing about the convenience fees, which is kind of like the dirty little industry secret: when a promoter goes to an artist and says, "Hey, you rock. You're awesome. Let's do a show together. We'll get my people to talk to your people, and we'll do lunch."

So what happens is that the artist, of course, is on a high. This promoter wants to do it. He's talking about sell-out shows, and so the artist starts seeing money in their eyes. So the artist will come up and say, "Give me 105% of the face value of the ticket, and I'll do the show."

So you think about it. The ticket is, say, 100 bucks. The artist wants $105. Where's that money coming from? The convenience fees. So some of that money goes to them.

Now realize, nobody else has gotten paid yet. We haven't gotten paid, the promoter hasn't gotten paid, the venue hasn't gotten paid. There's a lot of people with their hands in the pot for this.

So at the end of the day, when you're paying 20 bucks for a convenience fee and you're like, "Oh, those damn Ticketmaster people," we're getting two bucks. That's it. That's the amount of money that we get. So it's not a lot, and that's kind of the backstory on the convenience fees.

A little bit about, on the technology side, our scale. As I mentioned, we are a global company. We do have seven data centers worldwide. We have about 20,000 OS images, so that's VMs and bare metal. Not everything is a VM for us, at least not yet.

And we do transactions in excess of about $16 billion annually. Again, that's globally. And when we have these on-sales, and I'm sure at least some of you guys know what an on-sale is: Friday morning, 10:00 a.m., something goes on sale. Last Friday was Garth Brooks. I don't know how many country music fans, but Garth Brooks goes on sale. That's called an on-sale.

And during those peaks, we do over a million dollars a minute at times. That's where we actually make most of our revenue, those peaks. And that's actually a struggle for us in our business because that defines a lot of what we do.

If we mess up during that one point, Twitter is blowing up, people are complaining, and really the fans are upset, because if there's something that people are passionate about, it's their music, it's their sports teams, it's their live event. It's something that is non-replicatable. So it's a big deal for us.

The other thing is also we do have 255 million-plus user accounts in our system. So it's not gigantic. It's not Facebook-type scale here, but it's not trivial. So when we talk about compliance issues and regulatory problems, these are some of the things that we're facing because these are people that we have their personal information on.

So I want to talk a little about the story. I'm going to talk to you a little bit about the challenges that we faced, some of the early technical solutions that we implemented, what was going on behind the stage. Talk a little bit about the solution, what we're kind of doing now, where we're kind of going, the direction that we're in, and then the impact in terms of what we've seen so far.

Let me talk a little bit about the early tactical solutions. A little flashback to 2000, a little history there, is Ticketmaster and what you guys probably know most, ticketmaster.com, were not the same company, actually.

Ticketmaster was very traditional in that we sold tickets through box offices, through outlets, Tower Records, things like that. And when this person came around and said, "Ooh, I think you guys should sell on the internet," basically Ticketmaster was kind of like, "Yeah. Just be an outlet. Go over there by yourself."

I actually worked for a company called CitySearch at the time, and we were called Ticketmaster Online-CitySearch, and we were a separate company from Ticketmaster.

Back in 2001, we were owned by Barry Diller, USA Networks. We bought out Ticketmaster because the belief from Barry Diller was that the internet is going to take over all of this retail. You look at Amazon at the time, you look at all these different companies, and the thought was that the internet's going to rule, so why do we not want to own that part of the business and move more sales over to the internet?

Back in 2001, we purchased the company, and we started doing a little re-platforming. Well, the first thing that we realized was, at the time, internet sales were 3% of our overall ticket sales.

So the first 2001 plan, what we called I5, was our company-wide initiative. And it stood for Internet 5%, meaning 5% of sales should be on the internet. We're talking 2001. Five percent of the sales. But to go from 3 to 5%, that's a pretty aggressive increase.

So the first thing that we did, because I came from a systems administration background, my other colleagues did too, was we said, "We can't build out these many systems. We can't build out these many servers."

Systems administrators know sendmail. We said, "Wow, this macro language. Let's start building our servers with M4. It's the coolest thing in the world. We're going to create these templates, and we're going to kickstart these boxes, and it's going to be awesome."

So in 2001, we first started configuring systems with M4. In 2003, as we started re-platforming the product, we started onboarding a lot of developers. And the first thing we saw was that developers were building their code on their systems, and that was proving to be problematic because we had a different configuration in production.

So we started giving them systems to be able to develop their code on. Again, we're hiring developers at a faster rate than we can buy systems, so we implemented virtualization and we started automating these builds. So we had this "build me a cluster," and it would just essentially build out a group of servers for a developer to be able to develop on a systems- or operations-managed infrastructure.

In 2004, we realized that M4 wasn't really all that great and that we needed a new tool. So we looked around, and at the time, Chef and Puppet, they weren't really around. I think the only thing that was around was CFEngine, and it really didn't meet the needs of what we were looking for. We wanted something object-oriented, hierarchical, things like that.

So we ended up writing our own CMS. At the time, we called it Rubix, but when we open sourced it, Rubix was already trademarked, so we call it Spine now. So that was 2004.

The end result of that, through 2001 and what I'm going to show you in 2005, was that all through these years, we started increasing our internet sales. So we went from what was 3% in early 2001 to 70% to 80% in 2005.

In 2005, our biggest success was, I don't know how many Yankees fans are in the room or anything like that, or baseball fans. Okay, there's a few, I guess.

So you will know that Yankees fans are obsessive about their team. And what happened was, I don't remember if the Yankees were doing well or not. I'm not a baseball guy. But we had a presale, and it basically brought down our entire infrastructure. It was crazy. We had thousands and thousands of people, hundreds of thousands of people connecting at the exact same time.

And a challenge for Ticketmaster is that we don't have nice bell curves of traffic. If you look at our traffic, it looks more like an EKG. We love 95th percentile. It's the best billing model in the world for us.

But we had a major problem because we saw the presale brought down the infrastructure. And a baseball game, what's 80 home games, Yankee Stadium, at least 50,000 tickets a game. So you're talking a lot of tickets that go on sale all at once. And you've got a lot of people in New York that really want these tickets.

So we had this problem where we had to scale out, and the on-sale was going to be two days later. Thankfully, we had just purchased a few hundred servers or something like that. We didn't have them yet racked or built out. We had to scale out the entire infrastructure by 150% within 48 hours.

From an infrastructure perspective, that was fantastic. That was something that we hadn't had to do before. So that was a pretty good success for us. It took us about eight hours to sell out that event, or those events. But it was a pretty good success. I think the company really appreciated what we had done.

Then comes 2008. So 2005, 2008, not much going on, just kind of building, kind of doing things.

In 2008, we had formed a new company called IAC Interactive. And then Barry Diller spun us out and kind of saddled us with about $750 million in debt. Also, if many people go back to their 401(k)s in 2008, you'll notice you took a big hit because of the economy.

Well, if you think about it, disposable income is the first thing people are going to stop spending. And live entertainment, as passionate as it is, it's still disposable income. So we were taking a pretty big hit at the time.

So the economy kind of forced us down a cost-driven mindset. This was when we first did our internal cloud, and it really got approved because of the ROI that was associated with the internal cloud. We were scaled out pretty large. We had lots and lots of bare metal servers, and again, as I mentioned, our traffic patterns, they don't go up like this. They're very spiky.

So we need a lot of infrastructure to support that spike, but then afterwards, it's kind of useless. So if you think about a lot of the reporting functionality, CRM-type things, those types of activities, well, they can live right next to the servers that are selling tickets.

So the cloud ROI for the company was extremely... It was a great idea for them, and they loved it. That was basically the only project that literally got approved for 2008 through 2010. There was nothing else going on during that time because the money just wasn't there. It was basically drying up.

Then in 2010, we merged with Live Nation. We call it a merger of equals, but it was 51% Live Nation, 49% Ticketmaster, and Live Nation took all the top spots.

And at that time, actually, another story here, which was in 2009, as we talked about 2008, the economy was getting tough. So in 2009, we had an on-sale for Bruce Springsteen. And the Bruce Springsteen on-sale, we tried to get into the resale market.

Everybody had heard of StubHub and the huge market, and how are we going to refill our coffers with cash? And that was one place that we hadn't been. Most people in this room probably know of Ticketmaster. Most people have probably purchased something through Ticketmaster. It's not like there's much market space left in North America for us. So we looked at doing resale.

Well, the unfortunate side effect of that was that the way that we implemented it, it wasn't very obvious to fans, and they had a lawsuit against us. So we had a million-dollar outlay of cash that we had to give back to consumers.

So this kind of forced a decision on what does the business want to do with Ticketmaster. Does the business want to just say, "Hey, look, let's just close it down, and we're going to ride the wave, ride it off into the sunset," or are we going to go for broke and try to grow the business?

So we went to the board, and we kind of built the burning platform and said, "Oh my God, the sky is falling. And if you guys don't do anything, you're going to lose all the money here, and you're not going to get your next yacht." And so they gave us a whole bunch of money.

So that kind of started the re-architecture of the platform, and that's where we really started to say, "Okay, look, we just built this kind of burning platform. We're either sinking or we're going to swim. So we need to change everything up. We've done some things in the past, but let's build upon some of those things."

And so what we did was that we started to really look at what a DevOps strategy might look like for us.

Again, our problem was, one of the ways that we sold the platform, or the re-architecture of the platform, was the fact that we could not build features fast enough. That was a huge issue for us.

As Ticketmaster, again, we are driven by fans and clients. And so if clients don't like selling through us, fans aren't going to come to buy through us. So we have this incredible desire to please our clients, to please our customers. And if we can't get features out the door fast enough, and especially as cloud and Amazon is starting to pick up, scale is not holding people back. That was definitely a factor for people in the past, but it's no longer that.

So we had to really come up with a way for how do we deliver software faster and faster and faster. And so 2010 was, we started this process, and we started to say, "Okay, what are we going to do now?"

Now, as we started thinking through what are we going to do, we had a lot of things. As many of you people know, in an enterprise, it's different than a startup. In a startup, you've got nothing. You can start from scratch. It's great.

In an enterprise, you've got customers. And for us, we had clients and fans. We already have a lot of employees around all the world, actually. So what about them? How are we going to do that? And then, of course, the big thing is, we're a business, so we have to make money. So if it doesn't make dollars, it doesn't make sense.

So we had to answer these things, and there were a ton of questions that we had to ask ourselves.

For the fan experience: will we be able to provide an equal or better product? If we can't, why are we doing it? Because that's what we're trying to tell the board. That's why we need the money here.

Will this actually result in faster time to market for our requested features? Again, this is part of that burning platform.

And then the final question was: will our fans have to deal with problems while we figure things out? We already have a business. We don't want to alienate all of our fans who are purchasing tickets. And when you're making this drastic change to your organization, there might be problems. And if these problems are big enough, how long will somebody wait while you figure things out?

The next thing that we had to deal with was employees. Huge challenge, right? You've got a 40-year-old company at this point, almost. I think at our last monthly presentation by our president, we had one person who just celebrated their 35th anniversary with the company. I mean, that's huge. Thirty-five years at a 40-year company. Crazy.

So how will the people adapt, right? We're changing everything up. How are we going to get them to adapt? We don't want to just fire everybody and start from scratch. We can't do that.

What type of training will we need? Again, we've got people that have been doing the same thing for 35 years. How are we going to invest in these people to train them up into the new way?

How do we overcome fear in the minds of our people? The moment you start talking about, especially for the operations side, the moment you start talking about support at the edge, developers will push to production. They'll be on call. They'll be dealing with this. You start to get people wondering, "Well, if they're going to do all that, what am I going to do?" Right?

And that's a challenge. You've got to work with people and make people understand. And then how do we share this vision for the future? You've got to share what will happen here. What is it going to look like? What could it look like in the future? And start teaching people or training people to the new way of doing things, and to explain to people that people are more valuable than just pushing a button, right?

At the end of the day, that's really what it's about when we talk about the fear in people. They're worried they're going to lose their jobs.

Then the final thing was the dollars and cents of it all, right? So of course, the business is asking, "Well, what's this going to cost if we're going to do this thing?" Right? How about the learning pains? What's that going to be?

And then we're doing this. Is our sales actually going to be any better? Because if they're not, then why the hell are we investing this time and money?

And the final thing was, how will we know this is working, right? So these are obviously questions from a business perspective. They need to understand, if we want to invest in these things, how will we track it and how will we know?

So what we came with is where we were. This is going to be a slide of where we started from when we went down this journey with these questions, right?

Ticketmaster, we have 17 different ticketing systems. Most everybody in this room probably knows ticketmaster.com. It is one of the biggest brands.

So 49ers, right down here somewhere, they are a client of ours, but they are not on the ticketmaster.com platform. They're on a separate platform. And that platform is designed for season ticketing, for sporting events, things like that.

If you've ever been to Vegas and you've been to a show at an MGM Casino, KA, Michael Jackson One, something like that, that's on another ticketing system as well. We support shows like that, too.

So we've got 17 different ticketing systems. Ticketmaster has grown through M&A activity. We're good at the A side. We kind of suck at the M side, though.

So we've got these 17 ticketing systems. We have 30-year-old technology, right? So if you look at ticketmaster.com, the core ticketing system is 30-plus years old. Just to give you an idea of how old it is, and I don't know how many people will know, a PDP-11 is what it actually runs on, right?

So yeah. To be able to sell tickets like that on a... It's now emulated, but it still runs on... You can't buy them anymore. They're not making them. So that's how old the technology is. So how are we going to turn this around?

And again, you can't just start from scratch. We had a hard line between development and operations. It was basically develop the software, throw it over the wall, ops guys are trying to install it, and all the mindset that goes along with it. No, developers can't have access to production. No, you can't do this, you can't do that, right? Very, very hard to break down those cycles.

And then our release cycles, we were bi-monthly to quarterly. We were doing five to six releases a month on a good month. And that's if we had emergency releases because of the bugs that had to come out, right? I mean, it's crazy. So that's where we were.

So what did we do? One thing I should mention is that I'm sure most people have heard of Conway's Law, right? People who design systems will build the systems based on the communication paths within the company, right?

So firm believer in that, and firm believer in that you have to break down the organizational structures to change the communication paths if you really want to change the underlying.

So some of the things that we did was we developed... This is kind of a picture of the overall organization, with the top side being the operational side.

The first thing we did was that we developed a tier one TOC. There are still issues. There's still legacy stuff. We needed a front line to be able to prevent lots of problems from getting too far back.

We also developed an SRE group, which can really help actually maintain the software and do things. So we're in the process of developing this right now, and they're actually watching. They're doing capacity, they're watching configuration changes, things like that.

We took the systems engineering teams and we aligned them to the delivery teams, to all the Scrum teams. So that way, we could bring some of the operational knowledge down into the delivery teams and help the delivery teams with operational problems.

And finally, what we did was that we empowered developers to push code to production. So at this point, as part of the re-platforming, the new services that we're building, they are all being pushed to production by developers.

So that was the solution. We started this about a year and a half ago, I would say, maybe two years ago, but really it probably got started in earnest maybe about a year ago.

So what has that achieved? First off, 51% now of all of our services and products are deployed by development teams. That's huge from 0% a year and a half ago. So pretty big number there.

Development teams now have an on-call. They don't all like it, but now they got to be on call.

So on Saturday or Friday mornings now, when we have on-sales, we have a tier one through four level of on-sales. So when we have the top-tier on-sales, they are in the office at 7:00 a.m., because we're West Coast, and East Coast starts at 10:00 a.m.

So they're in the office, they're watching their systems, they're looking at their monitoring. They are part of the operational side of actually running the business.

In September, and that number is actually low, was over 109 deployments in September. Compare that to about five to six maybe a year and a half ago. Huge difference.

I should say, some of the teams are actually deploying, obviously, multiple times. Some teams are doing story-level deployments, so every time a story is done, they deploy. Some teams are doing sprint-level. Some teams are doing multiples per sprint. But that's the numbers.

On the tier one side, we've had 490 alerts in September that were categorized as tier one alerts that are resolved by the tier one team. So if you just think about what that might mean for engineers, even if you look at five to 10 minutes for a context switch, you're talking one to two man-weeks of time every month. That's half an FTE, basically.

The other side, this is the business side, and this is pretty impressive. Ticketmaster Resale, if you haven't heard of this, or TM+, it's basically our new way to do resale in the market. On one page, you can see primary tickets as well as secondary tickets. They're all safe. You know exactly what the tickets are. It's a huge change in our industry.

We launched it one year ago, and in the last six months, it has grown over 30%, and it's all been through iterative releases and things like that. It is a tremendous growth in our organization.

And the real bottom line was the increase in customer service rating. We've been asking this question of people for, I don't know, 10 years plus. The question is, "In general, how would you rate the stability of Ticketmaster products and services?"

So if you take 2012 as the baseline, and I mentioned to you that we started this around a year and a half ago. 2012 was the baseline, so that's zero. In 2013, the excellent rating increased by 5%. We just started, so not that great.

And in 2014, I actually messed up these numbers. Gene was just talking about it. I actually thought that new dad was too tired and he messed up the numbers, but I did. It's actually not 24%, it's an increase of 30% over 2013. It's a tremendous increase for actually our stability and what our clients and our fans are actually seeing. So amazing results for us.

Things we don't know. Oh, well, things that we still got to do, sorry.

Obviously, it's the continuous refinement, right? We haven't made it down this journey yet. We still have a long ways to go. We're not at six minutes yet. We're only 109 deployments. We're looking for at least an order of magnitude higher than that.

We're trying to push more services to the edge, right? Customer service tools: get it out of our hands, get them over to customer service, let them deal with that stuff. They're closer to our end users. They know what our end users want. They can fix it fast.

Client service tools for event building, that's a huge thing. If you've heard of Eventbrite, they allow anybody to build an event. That's obviously a big competitor for us. So that's one of the other tools that we're trying to push off to the edge.

Training on people. Again, employees are a huge asset to the company, and if we don't train them and if we don't teach them these new methodologies, then it'll be a problem for us. So things like Lean PMO, Agile, Scrum, Kanban, we have all of that going at the same time.

Things we don't know. There's a lot of things, and I just summed it up into four things really quickly, right?

When we talk about the alignment from the operations to the development, we don't know how many people we should be aligning. Have zero clue. What should those ratios be? Don't know.

When we talk about some of the responsibilities, we have 17 different ticketing systems. We're spread throughout the world. We've got ticketing in North America, Europe, Asia, Australia. So what are the different roles for people?

If you notice, I never touched up on security. How does InfoSec fit into this? We want to move fast, and I don't know if I'll offend some security personnel in here, but security people can be a bit paranoid. So how do we leverage that?

And then the final thing is, again, goes back to the metrics. What other things do we need to be looking at? We've got some metrics, but what are the metrics that are really going to help us know, are we doing things right? Are we making a mistake? And how do we course correct?

So thank you for allowing me to take up your time and tell you a little bit about our story.