Gareth Sephton

Hello, and welcome, everybody. Hope you're all doing well, staying safe, and living your best lockdown life.

Thanks for joining me today. I'm going to tell you a little bit about William Hill, and how we're using DevOps principles to take the company forward from good to great.

My name's Gareth Sephton, and I'm the Infrastructure and Cloud Tooling Product Owner at William Hill.

Let me start by telling you a little bit about who William Hill are, a little bit about the history of the company, and some of the major milestones in the last 80 years.

William Hill is one of the world's leading betting and gaming companies, employing over 16,000 people worldwide. It all started back in 1934, where William Hill started the company by taking bets over phone and via mail. Right from the start, he was taking an innovative approach to reach and excite our customers.

In 1966, five years after legalization of betting shops in the UK, William Hill started to expand the company by acquiring existing outlets. That acquisition would become a major driver for the company's growth over the following years from then and continues to be part of our growth strategy to date.

In 1998 saw the launch of the online sportsbook, soon quickly followed in 2000 by the online casino. Originally established in the UK, this was moved to Gibraltar in 2009. That move included the migration of servers from our UK data centers to the data centers in Gibraltar, which was a really interesting project, really quick timescales, but I think that nobody at William Hill would want to repeat anytime soon.

In 2012, we established William Hill US with a focus on retail and mobile gambling in Nevada. This gave William Hill a foot in the door for when, in 2018, the Supreme Court of the United States declared the Professional and Amateur Sports Protection Act, PASPA, unconstitutional. This allowed states to start legalizing gambling and regulate sports betting. William Hill was one of the first companies to be able to capitalize on that opportunity, with access secured to 24 states already and expecting more to come. Each state has come with its own interpretation of the rules, which keeps us on our toes.

In 2016, we acquired the betting and gaming digital solutions company Grand Parade in Kraków. This allowed us to rapidly ramp up the scale of our development teams, bringing in that existing expertise that Grand Parade had, particularly around UX, UI design. It really enhanced our creative capabilities.

Then recently, in February 2019, we completed the acquisition of Mr Green, and with that an expanded European footprint in fast-growing online betting and gaming markets.

Whilst this just scratches the surface of William Hill's long-established history, what it does show is that even from day one, we've used innovation, acquisition, and making bold decisions to take the company forward. And also, we are not an organization that was born in the cloud. In fact, we were born in the backstreets of London. This history provides valuable lessons that we can apply today to take the company forward, but also creates a few interesting challenges that we need to address.

One of the things I didn't show on the previous slide, but possibly an important milestone for William Hill as well, was I joined the company in 1999. Fresh from university, driven by a desire and a motivation to remove that minus sign from the start of my bank balance.

Without really knowing it at the time, and probably not it being a conscious decision, as I reflect back on my roles I've had, I can map them against the areas that DevOps is looking to address. It shows that I've kind of had this front-row seat to some of the challenges that have grown up over the years and that we're trying to fix.

I started in tech, my tech journey, as an applications analyst, which was a fancy word for a developer. I think back in 2000, for some reason, every single one of our tech jobs seemed to have the word analyst in there somewhere.

Back in that role, I took an interest in our version control systems and the delivery life cycle we're using. For the version control systems, it was an act of kindness. These were not so much servers. It was a desktop hiding in the corner somewhere, gathering dust. The only maintenance that was ever performed on it was when it accidentally got turned off, either to plug the Hoover or someone to charge their phone.

But recognizing the importance of treating these tools with care was key. This led me on my journey through the other roles shown here into my current position of tooling product owner.

Along the way, most of the roles have involved working closely with development and with operations people. Mostly, we're nice and collaborative. We've got things done. But it'd be fair to say that along the way, there has been the odd bit of frustration and conflict. In fact, I have a lot of sympathy for anybody that does an environments manager role if it was anything like the one I did.

One of the observations that I've managed to make along this path, and I kind of hope this isn't too controversial, but whether you work in development or operations, we all have a lot in common. Everyone I know wants to do the best job they can, but they get frustrated when they have no control over completing a piece of work or that an issue they're working on is being caused by factors outside their control.

For many of my roles, bridging that gap between development and operations teams was key to my success. That came from just ensuring we're providing the right information, just acting as that interface between the two.

So when I started to hear about DevOps and started to read up on DevOps, I was hooked from day one. In fact, I remember reading in The DevOps Handbook, there's a paragraph early on that says, "Imagine a world where product owners, development, QA, IT operations, and infosec work together, not only to help each other, but also to ensure that their overall organization succeeds."

Those words express better than I ever could what I wanted to achieve in my career and what I still get a buzz out of trying to fix.

Now, I confess that during my career, I was probably never the most technical person in my team, but built up a reputation of being someone that we could go to to get things fixed. That was mainly trying to get the best out of others, but fixing things myself. So providing that right information, connecting those teams together.

For me, that's working together is how I see William Hill tackling the challenges we face today. In fact, one of the things, if I'm ever asked to give a quick summary of DevOps, I'd like to summarize it as: stop asking people to do something for you, but ask them to show you how to do it. And if that's too difficult to show someone how to do it, make it easier, simplify it, automate it.

Let me just delve a little bit further into some of the challenges we face at William Hill, which I'm pretty certain will be not unique to us.

As I touched on before, William Hill has been around for a while, and over that time has built up a fair bit of technical architectural debt. In fact, at times, understanding dependencies between some of our older services is incredibly time-consuming. In fact, some of the older products, you realize that the one guy that might have known about it has retired, left the company, moved to the coast, completely off the grid. So we pretty much hope and pray that that service will never break until it's replaced.

Previous slide, I mentioned the positive side of regulation changes with what happened in the US allowing the company to grow. Well, that's probably one of the exceptions. Most legal compliance changes tend to require significant work for us to remain compliant and to maintain our revenue income. For example, in the UK, following the outcome of the Triennial Review in April 2019, new regulations came into force limiting machine stakes to a maximum £2 bet. That significantly impacted our revenue. But we don't sit there and moan about it. We just look at how we can innovate, reshape the retail side of the business, and move on, and learn and adapt.

With a large tech department, which is based in the UK, Kraków, Gibraltar, Malta, and the US, we need to look out for where teams end up working in silos. In fact, it's probably not just the geography here. Teams in the same country, on the same office, on the same floor, even sat next to each other, can sometimes end up working in that silo.

Those silo teams may at times give the appearance of autonomy, but it may also lead to duplicate effort and a culture that builds up that almost tribal us-and-them mentality, and at times even unknowingly impacting another team's work.

With the tools we use in tech, the siloed nature started to lead to a few issues as well. We had a lack of clarity around ownership and responsibilities of the tools and services we used. We had a fragmented toolchain, which was impossible, if not very hard, to support. There were bits of goodness not chained together. We didn't have a strategic direction for the tooling or no feature roadmap. So when someone had a problem they didn't know where to go to, they would end up fixing that issue locally or within their team, and that diverted their attention from the product delivery side.

We had no consistent patterns for integration between these different teams either. And the tools provided centrally were probably not exactly what the user wanted. We'd built what we thought people needed, not what they asked for.

Just one of the other challenges that we've all faced recently: coronavirus. It's definitely impacted everyone, even this summit today being done virtually. For William Hill and the industry we're in, that relies heavily on betting events for us to be able to bet on, this obviously worldwide pandemic has caused a few headaches. But on a positive side, it's also had a unifying effect. Where teams have been unable to progress, we've redirected that effort into our gaming, into the areas of the business where we would still get that return on investment. Also, with everyone working remotely, we've had to get better at creating those virtual teams. So that geographical split becoming less of an issue.

One of the biggest challenges I've listed here, in my view, is keeping teams motivated. It's one of these that if we get this right and have motivated people, they'll take care of the rest. In my early days in line management roles, I was probably guilty of falling into the trap of assuming that financial reward was a key motivator, so looking to where I could get the guys in my team their next pay rise or a bonus or some other financial reward.

But over the years, it's become clear to me that that's not the biggest motivator. And Dan Pink eloquently highlights in his book Drive, autonomy, mastery, and purpose are much more effective motivators.

So how are William Hill trying to give that purpose and help us address some of these challenges?

Well, we have a clear company strategy. In tech, we've talked about DevOps a lot, and I mean a lot. It's a very opinionated topic. But one of the pushbacks or more, I guess, negative comments we'll get in one of those conversations, particularly early on, would've been, "Yeah, that sounds great. Really good stuff, but our business will never back us with it." Well, I think that you can see from the company strategy, it's clear that whilst it might not say in big, bold letters, "Do DevOps," the company is behind this. This is not just a tech thing. This is an organizational thing.

Without going into too much of the detail here, we can see from the strategy where it will support the DevOps journey. We talk about having collaborative and agile teams. There's operational efficiency through automation, scaling by selected core platform components, and continuous innovation at the heart of this, all focused on giving our customers the best competitive offering.

Now, a company strategy is great. You would still get the murmurings down at a tech level. It's like, "Yeah, but what does that mean for me?" And we listened to that within tech operations and wanted to break it down and make it very clear where our people sat within this. We wanted to make sure the purpose was really clear to them.

Shown here, there are six areas to the strategy. For the purpose of today, I'm going to mainly focus on public cloud, people, and toolchain. However, let me just give a quick nod to the service side of it. For an operations team at William Hill, keeping a 24/7, 365-days-a-year operation going, we consider ourselves to be pretty good at keeping the lights on for our revenue-generating systems, those that our customers use. However, as part of the strategy, we realized we needed to look internal and make sure we were providing the same level of service to our colleagues internally at William Hill.

And the public cloud: William Hill are on a migration path from our physical data centers into the cloud, and at the moment, specifically AWS. Now, that would take a whole other talk, and a lot of time to go through all the benefits the move to public cloud is intended to bring for us. But I'm just going to keep it short here. I think for us, one of the key things here is it's acting as a catalyst for change for the other areas we need to address. It's given us that fresh playing field, so away from our on-prem data centers and the challenges and the tech debt we build up there, including the tools and the products we use to manage those on-prem physical data centers. It means we can take that step back and look at solutions without constraints, but making sure that we still understand the lessons that we've learned from running those tools and from running those physical data centers. The technology might change, but the knowledge and experience we have remains.

Within the toolchain, we acknowledge that to win, to be market leader, we have to get our ideas into the hands of the customers before our competition. Here, this is tech ops looking at fit-for-purpose toolchain that lets our product teams focus on delighting our customers. Our thought process here took us down looking at Gartner's pace-layering model, and essentially looking at the toolchain as a system of record. Which is to say whilst it's incredibly important for William Hill, it's not what our customers want. It's not differentiating for our customers, and that's the bit we want our product teams to be able to focus on.

At the heart of our strategy are people. Not to be corny, not being cliché, it's seriously recognizing that by having talented, motivated, and engaged people, they'll contribute to a high-performing organization. But to move forward, we have to provide training and development plans so they can provide the mastery of their craft. Alongside that, there's no point training them if we're not giving the opportunity to put those skills to use. In fact, we do monthly surveys, and in our monthly surveys, we're seeing more and more feedback that people want meaningful work. In fact, if we're giving the training and no opportunity, they're just sat on the substitute bench forever.

We've talked a little bit about some of the challenges and our strategy. It's probably good to understand a little bit how we're structured within tech. William Hill is divided into divisions and subdivisions. This diagram's showing the subdivisions. In places, they can be split further into more focused product teams, and these are spread within those subdivisions across the UK, Gibraltar, Kraków, all the other geographical tech hubs we have.

Not all teams have the exact same makeup, but essentially they are a mix of developers, testers, engineers, project managers, scrum masters, product owners, everything you would expect in a delivery development team.

Operations provide support and service to all these areas, which to a certain extent puts us in a very privileged position. We can take lessons learned from supporting one team and provide that help and guidance to another, sitting in the middle.

Again, historically, we've probably been guilty of doing this based on what we thought was best for the teams, pushing these as standards out to everyone. In my experience, as soon as you start talking standards, you're just going to find yourself in argument, discussion after discussion, new standard after new standard. It's a tricky word, and I've never really seen someone get a successful standard pushed out. So with people at the heart of our strategy, we thought it best to go out and talk to the people.

We started the conversations with our heads of tech from the divisions, and we discussed what it was to be above and below the line. That wasn't a new betting opportunity, a new fancy game out on the sportsbook. It was about what it made sense for us to look after centrally, provide as that service, and what we wanted our delivery teams to be focused on. What was really going to be a differentiator for them to excite our customers?

The overwhelming response from everyone was pretty much to put anything other than application development below the line or, in more key cases, on the line. On the line highlighting where it was important that we were collaborative. It wasn't what we thought, it wasn't just what the dev teams thought, it was a collaborative effort.

Following on from that, we then took a leaf out of design thinking approach. Again, design thinking is probably a whole different presentation. But a key to that design thinking approach in the first stage is to gain an empathetic view and understanding of our colleagues and their problems. This was a crucial step for us to set aside our own assumptions and gain real insight into the users and their needs.

We had to get to know our colleagues. What did they like? What did they dislike? What did they want, and what did they feel would make their life better and easier at work?

This took over 30 hours of interviews, and the majority were held face-to-face. That wasn't just because we wanted to have a jolly out to Kraków. We saw it as important that we went and met the people.

We didn't question why from our side. We didn't have any agenda. We didn't go out there with, "We like this, therefore, we're going to push it." We focused on their individual likes and dislikes. In fact, before we went out, we even talked about avoiding the question why. We always got a much better response if we asked, "What is it you like about something? What is it you dislike about something?" Rather than asking why they liked it or why they disliked it. Asking that why almost puts it on defensive. It feels like you're asking them or challenging their point of view.

Those interviews allowed us to build up a list of common themes, common challenges that helped us look at where we needed to focus our initial efforts.

The common responses coming back is they wanted things to be centrally managed but stable and reliable tooling. We wanted a low barrier to entry with better training and documentation. The teams wanted control over their pipelines, but not necessarily the responsibility for the underlying infrastructure. They wanted fewer tools, to consolidate all the tools we had available. They wanted autonomy, but at a self-service level. So again, not having to understand everything that happens under the hood, but able to do their piece of the puzzle.

They wanted better visibility. They wanted to see what was happening with our releases, where things were. Also, we wanted to have that clear ownership. It wasn't to look for someone to blame. It was to look to someone we could help, who we can move things on with, where we can put our requests in, where we can work together to make the best tooling for William Hill.

So in the tech ops and the toolchain piece, it gives us our goal, which is to create a centralized platform tooling service that any development team can use to become more productive. From getting production-like environments, setting up deployment pipelines, bringing the monitoring in, or even the security tools that our award-winning InfoSec team have to offer, bringing it all together.

And by dev team, that's not just our development teams in the channels. That's everyone. Our network teams, monitoring, infrastructure. We're all pushing to that as-code approach. Therefore, this platform's got to be right for everyone. The key for our success here, we see, is building that as a platform, not individual products or tools. It's a platform where we bring the best that all of William Hill's tech teams have to offer.

Again, to touch on some of the benefits of that toolchain platform: we can provide capabilities that we'd agreed earlier made value being provided centrally. But we can also tap into the skills and experience across all of our tech teams. We don't need to have one team that knows how to do everything. We can integrate services via APIs or even getting people to contribute to the code base.

The best practice and safeguards, we can build that into the tooling. And where we need to enforce it, we can do so. That not only contributes to improving our delivery timescales, but it also gives teams confidence that we can trust the products we're delivering to our customers.

Through those interviews, we discovered that of all the services we were using, all the tooling we were using internally at William Hill that were being used around the CI/CD area particularly, GitLab was the one that everyone liked. I say liked. No one had any real negative opinions, which for our developers and engineers is high praise indeed.

We'd been using GitLab as our source control repository for over five, six years. The license was up for renewal in 2019. As part of that renewal, we needed to start looking at how we can maximize the return on investment we had with that product.

We started to look at, well, GitLab, building the CI/CD, starts to give us that single platform. We could use that to power everything we're doing and essentially avoid any unnecessary complex compatibility with other tooling.

GitLab has the extensive documentation along with a strong community, so we started to address that onboarding aspect or the low barrier to entry. It provided teams with ownership of their pipeline without the support overheads. Moving to the SaaS version, to GitLab.com, even takes away the operational overhead from the team or my team that was looking after the current self-hosted version.

As mentioned, it was widely liked within the organization, so having that buy-in from the start is incredibly valuable. It started to reduce our tooling footprint. Where we were using Jenkins, Concourse, or our own bespoke tooling, we could start to consolidate that into one. It comes with reporting and dashboarding, things that we've been having to piece together from our own external services.

One other key thing for us was it supports our varied use cases. We are moving to public cloud, but we're going to be in a hybrid world for quite a while. In fact, there's a fair chance we will never be able to be fully cloud. Regulations, where bets need to be struck, mean we will always have some on-premise physical data center services.

If you looked at that, continuous integration, delivery, and deployment are the holy grail of the DevOps toolchain. Being able to safely take that code commit through the various quality gates and put it into the customer's hands with no human interaction is the dream.

We're looking to do this by taking the input capabilities of many areas and chaining this together. We recognize we're not going to be able to deliver this on day one. It's going to be an incremental approach. We're going to really have to work with the expertise within William Hill. Again, this is where the platform approach, API mentality, microservice, contract-based testing, starts to pull all this together. It gives us that flexibility and control we need to be able to adapt.

In looking to get the most of our relationship with GitLab, it's probably why we've decided and agreed across the whole of tech that GitLab Runners or GitLab itself will provide the platform or the engine behind the platform.

One of the areas we have been looking at: we've realized that we've got a big infrastructure operations team with lots of experience. We've got development teams who know how to deliver code. So we want to deliver the infrastructure for them into their channel accounts from the center and creating a pipeline to do so, ensuring that it's kept following all the goodness we get out of CI/CD.

It includes a layer that enables deployment of varying configuration, so we can still give the development teams control they need. But following that CI approach, right from the start, we're starting to run tests, code checks, security checks, and deploying that into our own testing, central TechOps test environment first, where we can do functional tests based on the contract of what we've agreed we are delivering to the channels.

This is using GitLab Runners built in AWS. The code is all stored in GitLab.com. So we're really embracing that cloud-first mentality.

Through this, we're working closely with GitLab and AWS ProServe team to not only look into the internal experience, but where we can maximize relationships with our vendors and third parties.

Part of the aspiration here is to provide a library or a shopping list of proven infrastructure builds, taking that open-source approach so anyone can contribute to it, and encouraging that reuse of best practice rather than dictating standards.

Let's summarize a few things here. There are different paths along the DevOps journey. Thankfully for me, William Hill are taking the approach of providing central tooling to enable DevOps. So tooling products aren't it. It was probably a good choice they made for me.

It's not to say it's the right way. It's not to say it's the way we'd even recommend to others. But for us, it's an approach that capitalizes on the strengths we have internally. But we do accept that change is the new norm. If it doesn't work, we'll adapt, we'll change, we'll move on. But we'll incrementally cut those changes now, and we'll listen to the feedback.

Autonomy, mastery, and purpose clearly are excellent motivators. But our experience says we need to use this with caution. Too much autonomy contributed to silos. The mastery is great, but expecting everyone in tech to be masters of everything is either going to be an impossible challenge or very expensive.

Again, check that that purpose is understood. If it's not, it's not going to have the desired effect. You may have all these fancy graphics and fancy slides, but if the people don't understand it, if the people don't see where they fit into it, it doesn't give that purpose.

And the thing here is use your lifelines. Talk to people. Don't feel you have to go it alone. Accepting that we don't have all the answers and leveraging internal or external expertise will take us from good to great.

As mentioned, this isn't just internal. This is with the third parties we work with, looking to establish working relationships, not just buying products off the shelf.

I think as a final word, if we are to build the world's most trusted digital gambling brand, then we need the people that work at William Hill to believe in it. And to do that, we need to nurture a culture of trust within our day-to-day work. For me, that is going to be key to our successful DevOps adoption.

Thank you very much for listening.