Markus Rautert

[Opening music]

Thank you for this. It's a great pleasure to be here, and thank you, Gene, for inviting us.

After seeing a short video from our company, I hope you all know it. Maybe first, a short introduction to ourselves. That's us. I'm Markus, and my colleague Fernando will introduce himself.

You see a little bit where we are in the organization. I'm heading Platform Engineering and Architecture, which is a kind of new and fresh group since January this year, actually.

Maybe a little bit about myself. I'm 39 years old. I'm a father of two kids, four and eight, and that's probably what keeps me busy all the time. The rest of the day, I'm working in this great company and driving the change ahead around DevOps Over Coffee, the story we're going to tell you today.

Fernando Cornago

I am Fernando. I am also a father of two kids, but I'm 20 years old.

I've been devoting my entire life to creating high-performing teams from Spain for software delivery companies. Since three years, I have, as I always say, the best job in the world, because I can create the new: first leading the software development organization, and now platform engineering, for a global company with global impact, a company like Adidas, and working distance from home, from where I was born. So that's the basics.

Markus Rautert

That's enough for us. Maybe shortly, a little bit about the company. This is the belief we have in the company: that through sports, we have the power to change lives. That's very important for us. It's the essence of all what we do. It's rooted in our heritage and the product.

Of course, this company is a large corporation. We also do have a strategy, and sometimes you need to link a lot of the investments and things we're doing back to the ambition. Looking at the company strategy, we call it Create the New. For us, being a marketing company to the heart, it's about the product, the brand, the desire, and the people who are wearing our products, doing sports, changing lives.

Of course, being a DAX corporation, we are also listed on the stock market, and there are financial targets, which you can see more on the right-hand side. The bottom and top line, efficiency targets. We also have that and bear that in mind.

Most importantly, how do we tackle this? This you can see in the three pillars of the strategy, which is speed. It's not just the speed of delivery of software product and/or supply chain. It's really about making fast and good decisions.

Another one is cities, which is more, for us, a synonym of getting our products and solutions closer to the consumers where they are: the big cities, the small cities, urban and rural areas.

But also about open source. As a non-IT person, open source sometimes translates a little differently. For us, it's open innovation, how we work with partners and colleagues. But we also translated it into IT, also jumping more on the classic open source term.

Last but not least, and that's the most important, and that's why we're here today telling you a story of DevOps Over Coffee, is the culture, the people. People working for and with us. This is the storyline we'd like to share with you.

Of course, some hard facts. I'll just jump over them very quickly. Twenty-one billion revenue, one billion contribution, 1.1. We have more than 58,000 employees all over the world, 170 subsidiaries, and of course, some major locations.

You probably don't find it on a map: Herzogenaurach. Spells very differently. But also larger locations like Amsterdam, Portland, of course, Shanghai, and so forth.

Fernando Cornago

Zaragoza.

Markus Rautert

Oh, yeah. Zaragoza. You also don't find it on the map that easily. It's like Herzogenaurach.

Maybe we start with a little bit of history.

Fernando Cornago

Yeah. You see, we do a handshake kind of presentation.

Let's just start with a little bit of history, because we are talking about that transformational journey, and in order to understand it better, I would like to start with the beginnings.

I like this picture because this is how our internal software engineering capability looked like five years ago, four or five years ago. Not because we didn't have anyone who knew how to develop. It's because, as a company, I think that we didn't value the capability enough. There was not a team behind.

We had a couple of guys that knew how to code, but they felt, I think, lonely and miserable working in different products. At the end, they ended up leaving the company or turning to our project managers, because we had an IT that was mainly demand-manager- or project-manager-based IT, where we had a single software provider, our software vendor, the development vendor, and our demand manager was putting some work into their demand manager. They did the job, and six months later, you know the story: "I deliver whatever you like."

So we decided to start growing and try out. Well, this is our engineering today. We have more than 150, and we continuously are growing, by demonstrating that cost has been extremely reduced, quality has been extremely improved. But that's not important.

The important thing is that we have enabled the speed in the business, and it's what we keep doing.

That was one day where we celebrated our all-engineering day with 150 people. Gene Kim. We had the pleasure of having Gene giving us a keynote remotely from a Starbucks in Portland to Germany. I think that it was an amazing day. One thing that we could not think, like two or three years ago. Impossible.

How did we start? For the very basics. We didn't have these things, so we had to get our code under control, with our ticketing system, CI/CD, quality, our code people. So we created a perfect and traceable software delivery life cycle. It was the first thing to do.

All our external and internal people started using our tooling. We had to categorize all the applications that we had. We had more than 1,000 IT systems, and every day we have more than 1,000 people developing code and putting code in our systems. Mainly external by that time, and now little by little, we are adding our internal capacity there.

We had a target of quality at the top. Being vendor independent, we started bringing different software providers and partners and internal skills. We targeted that for every one of our critical systems, we needed to have some internal guy knowing how to build and code on top of the platform.

And we are obsessed. We will get into details later. We are obsessed with data and metrics.

So we created a very round metric called first-time right. It was targeted to be the perfect metric. Lessons learned: don't do that. That's not working, because we put together the agile, or in scope: how many story points we deliver, the different teams. That's tricky. Velocity is not always the best thing to measure.

The defects and the quality. We tried to do a one-size-fits-all strategy. That was good by that time, but I think it was super complex, because we had a huge existing code base. Measuring this was difficult, but it was the seed of what we have right now with metrics, and I will explain to you later.

And how was the way we focused on our people? It was challenging, because when you start deploying new engineers, all of them, it was a must for hiring that they were awesome people wanting to do really cool stuff. But you deploy one engineer in a team of 15, 20 people from different software providers, it was super difficult.

At the end, the guy was doing, of course, coding sometime. It was what he was hired for, or he or she. But he was doing sometimes technical architecture, because depending on the level of the architecture, we have some software architects doing boxes and arrows. You know that.

We pushed a lot already, three or four years ago, for automation everywhere and code quality. So imagine one guy receiving pull requests of 15 people from different vendors. At the end, you have completely lost track of the delivery.

And at the end, who knows better? Because we wanted this traceability from contracts to delivery, even with our software providers. At the end, the guy was sometimes taking care of the contracts, of the vendor management, of the, I don't know, doing the scrum master stuff.

So it was super challenging. But at the same time, it was super rewarding for our people, because we felt like the Spartans, like the conquerors of a company that didn't believe in software development. And we stopped hearing little by little that, "But we are Adidas. Why should we build software if we sell shoes?" Now you don't hear this anymore. That's important.

We demonstrate by savings, by productivity, and of course, we demonstrate we create the global impact, and now it's the model. Now we are hiring like crazy, and we are creating these capabilities.

But as I said, the fun level of the guys was okay. They were more on the policeman side than on the technical geek side, which is what they were hired for.

But what happened in the meantime? They didn't feel so lonely anymore, because communities started popping up. It's one thing that, from the management, we boost a lot, connecting the guys.

We changed the model, so we brought different vendors. Different vendors, big ones. And also we brought what we call partners, not vendors. They are software companies or service companies that are very specialized, so you can learn with them. Sometimes they learn from you. Most of the times, you learn from them. Now you have one vendor for every kind of need.

And of course, there was a huge change on the architecture. We started playing around with containers already four or five years ago. But I will say that the major change didn't come until we implemented our own Kubernetes cluster on-prem and in AWS at the beginning of last year, of 2017. That was the game changer.

No excuse anymore to not deploy every day. Kubernetes was giving us and is giving us an unlimited set of resources. So directly, by default, we already build software, continuously integrated, ramping up mock data, enabling green-blue deployments. Everything is there.

No excuse about, "No, SAT is busy because this key user is testing, and UAT is busy because they are testing the release of three months ago." We had tools that were not deployed to production in two years, two years ago. So that's a no-go.

Now we do things completely differently. Now we have other game changers, like Kafka, that is changing for us the way our applications interact with each other based on streaming, or Terraform. Now all the things that we build, even infrastructure, are completely automated.

So that was. I think that now is when the fun level, when things have started getting fun and up to speed.

But I will say, and now Markus will tell why still, what keeps us awake at night, because we are pushing a lot. It's our obsession that all the teams keep providing the maximum speed possible and with the right freedom. All the possible freedom, but without killing the advantage of working in a big company sometimes: this community building by default, resilience and secure application, and standard. Not by a standard. A standard is because we don't want everyone reinventing the wheel all the time.

Also, we are Adidas, so at the same time, how do we make it fun? Because it's at the end what the guys that we hire want.

Markus Rautert

You probably heard it before from the speaker: if you want to go fast, and you don't have proper brakes, you might crash. So the faster you want to go, the better your brakes need to be.

That is exactly where we discussed: how do we actually set this thing up?

We were working in a kind of project mode with eight, nine, ten people maybe, as the people who were willing to drive the change. Then at one point in time, we said, "Okay, the way how we build the software and the way we set up the teams, it has got to change."

Why is it called DevOps Over Coffee? This is a small storyline I'm going to give you, and I also had it when we talked to Gene before.

It actually started like this. That's the Red Ox. It's a restaurant in Herzogenaurach. You will, even in Herzogenaurach, have difficulties finding it. This is actually where we sat down the first time with our CIO. I call it the CIO plus eight, because it was eight technical people trying to convince him to do things differently.

It felt a little bit like the Fellowship of the Ring, only we didn't have a ring to throw into some burning volcano. But this was really fighting against a lot of anticipation from the outside and the inside: how do we actually change the things, not in a small pilot?

And why it's over coffee: because at this point in time, there was a little coffee brand. We don't make a commercial here on that. We had the Spanish colleagues there soon. He was just spot on that and said, "Okay, how do we call this little fellowship now we are going to do to change the way how we do development in the company?" He said, "Okay, there's the intention," so café intención. I hope I pronounce it correctly, Fernando.

Fernando Cornago

Yeah, intención.

Markus Rautert

So we had the intention to change that. We met together, and it was around April 2017, and said, "Okay, let us come back with a plan how we can do that." Not in a way that we create a small digital lab or 10 people on the side, but how we're really doing it on top of what the company's doing inside of the operating model of the complete IT.

We sat there, and we did all the planning. As in the movie, The Lord of the Rings, when they lose Gandalf, we somehow lost the CIO on the way. He was not available. We couldn't reach out to him. It's like, "What happened? We have this great idea."

We looked into DevOps topologies and all of these things, and then we said, "Okay, we need to have some other kind of coffee." This is where we called it, "Okay, there needs to be some more [inaudible] to it." So we had another coffee with the same group of eight people, and we tried to convey back. How do we do this now?

We made a really stringent plan in two months on top of the complete IT organization on how we're going to change the way we're working together.

There was this final call. As in the movies, they find Gandalf somewhere later, and he wears a white robe. I was with Michael, our CIO, in a small town in Austria, Linz, where there's a brand we acquired called Runtastic, and they have an office there with 100 engineers. They're working in the way we wanted to do, but more like a startup on the side.

We sat there outside at, I remember it was 2:00 a.m. in the morning, in the small garden of the hotel, and we had a conversation. We put all the plans down. I said, "Okay, Michael, can we now do this?" Then we walked back and said, "Okay, let's do it." This was basically marking the cornerstone of the decision of getting it changed.

So what did we do? First, we looked at how we currently organized. You probably have some organization chart of yourself. I don't go into that very terribly.

What does it actually mean? There are certain units mirroring the business. We call them differently: digital IT, corporate marketing, you name it. And of course, we had started, the years past, some kind of horizontal delivery model. We had development, testing, integration, operations. But also we had the architecture side on the right-hand side and said, "Okay, we need to combine these teams together, not as a single unit, but as something working together on top of the current model."

How shall that actually look like? The way we designed the organization, also to our cultural change, was towards we did some kind of a reverse Conway here. So we defined the products we would like these teams to create. Remember, this is the better brakes we all need to go faster.

We set up, okay, what are the principles we apply here to have the teams, so that everybody in the team is empowered to lead, that we have a cross-functional collaboration on all of these silos you've seen before, and that we don't have a stringent set of hierarchy in the team, which is very important for such teams.

At the end of the day, it didn't look terribly different. When we set first the function, we said, okay, how do we work together? The first one was, okay, we give it a name. As usually, everything needs to have a name.

Then we decided, okay, how do we enable and co-create with our colleagues in the function for the time being? And also, as our other main horizontal unit, which was there for years, IT infrastructure, was down, we said, okay, how do we connect back to them that we are making sure we are not cutting the line on top of them when we have everything software-defined using the cloud or commercial platforms?

It was super important first to tell our team how we want them to work, and this is the culture thing. Because the culture will be applied.

In the company, we have what we call the three Cs: creativity, confidence, and collaboration. We said, okay, what is it that we would like to do? Ideas to action. Commit, then deliver, but also you're allowed to say no.

The other thing, most importantly: team success over individual success. Of course, we would like our people to be empowered, super strong individuals.

The platforms we're using are value-adding, and Fernando will show you afterwards a little bit how that looks like. But also, it was very important to tell others how it will be to work with us.

This is the most important point we started with: the transparency. Tell people what you're doing, what you're not. Because if you're trying to deliver the brakes to the bump, you can go really fast. You have to tell people what you're doing and what you're not doing.

The other thing was focus on value, to have shared common targets based on value. In the past, Fernando mentioned it, we had some metrics. We had a lot of metrics, like how much commit we do a day. But velocity means nothing if what you deliver is of no value.

Also, most importantly, remove constraint. You probably know this "ticket please" thing. You want something? Please raise a ticket. So what's the first thing we said? Hashtag no tickets.

That doesn't mean, though, we create them. We create them, but at the end of a conversation. The ways how we've embarked the teams, we are using also modern tools and ways to communicate and work with each other, but we are also doing, of course, our compliance check. And yes, of course, we are working with tickets, but the boundary is not the ticket. It's a means to document something for compliance. That's it.

Of course, the collaboration was one of the biggest, and experiment and learn, which led us then to a way, what we are calling for ourselves, the fluid organization, which Fernando is giving you a little bit of an insight into: how that actually looks like when you change teams differently and you disembark hierarchies.

Fernando Cornago

Exactly. We arrange ourselves also in products. But for us, also a scrum that is the default, de facto for the organization, was not working, because we have really, really short-term assignments, sometimes hand in hand with this co-creation concept that we have with the different software delivery teams.

So for us, a Kanban mode is working much, much better. And also this fluid. Why? Because of course, you need these kings or these captains, that are the guys standing in for the different platforms I will show you later, that we have detected.

But the rest of the guys, the rest of our techie teams, we have a total of around 60, 65 in platform engineering. They can be working in different products, different days, or even in the same day.

We have a common backlog, so we can manage our portfolio of platform engineering, but it's really fluid. And as we have an OKR process for setting up our targets, and it's done every quarter, it's really easy for us to change the direction.

Our platforms. Only to name it. I'm not going to enter into the detail. I can discuss with you later what we have.

As I told you, the mother of the beast is our container orchestration, based in Kubernetes. We have now around 18 clusters, I think, around the globe in different AWS regions, plus our on-prem for critical data that we cannot move to the cloud yet.

Continuous integration delivery, where we provide some seed projects in our default languages: Java, JavaScript for the moment, and Go, we are getting there. By default, you can fork it, and by default, you have security, auto-scaling, et cetera.

Of course, we moved also from our Jenkins to Jenkins as a service. We have built our own decentralized tooling for continuous integration based on GitOps plus ephemeral agents, because we realized that you cannot treat two projects the same way. Even ECOM was eating up 60%, our ECOM, 60% of our resources in our central Jenkins, and you need to provide the teams speed and freedom to create, to put their own plugins, install everything without affecting the rest.

Monitoring: not only application, APM, logging. Also we are monitoring business, with a phased database on phased data platform based on Kafka and Elastic.

Test automation. We have also a common infrastructure for test automation, also with some artifacts that you can download and fork, and by default, you have an end-to-end testing cycle.

API integration and streaming. For the new world, we apply streaming by default. And the old world, in the core integration, we are also working heavily with the responsible to make it also self-service, because it's one thing that we want to do: all our platforms being self-service for the different teams, so they can go as deep as possible.

Also, our last platform, also moving to self-service. That is our ambition, is our big data platform. Heavily based, this one, on AWS.

And our ERP also, where we also apply continuous delivery practices on ERP. Now we are pushing, because I was discussing the other day with our colleague in ERP, and we need to be faster than the business. So let's not wait for business to ask for speed. We know that we will provide the speed as IT. So we need to go one step further than business, also in ERP.

And what services do we do? Of course, all our platforms are supported by our internal engineers, plus extended as standard form with our trusted providers and colleagues from different companies. Continuously evolving them until they become commodity. We know that some of the platforms that we have now, they will not be there at the end of the year because one cloud provider will provide this by default, and it will fit into our request.

We consult. We co-create. We have different teams in the verticals: teams that need more babysitting because they are not prepared, and teams that, in fact, are teaching us things. So we are only the glue department that is putting things together.

Training and onboarding. We are onboarding 15 to 20 engineers every month that we are hiring, so it's heavily time-consuming.

Communities, meetups, internal, external, we do a lot.

What about the standards? How do we manage the standards? I know that much of you don't like this word. But we are clear with this. Don't expect that as platform, as the central engineering team, we are setting up the standards for you there. We are only centralizing.

We have created our technical reference model. Basically, we have different groups or areas where they have a responsible. And it's the only requirement that you need to do if you are working in a vertical.

One example is, I don't know, coding languages. We have there our defaults, our exceptions for Kotlin. Are you allowed to go Kotlin by default? If you are in the mobile space and you are an Android developer, yes. If not, not.

And the only requirement from a guy that is working in a product is, if what is there in the technical reference model is not fitting your needs and you have something better, it's only raise your hand. You don't need to create any paper or so. It's a phone call, one minute. And as we are in the platform engineering team, we are as nerd as they are. We want to try new stuff. We will go with you hand by hand. If really it's bringing a benefit for the organization, it's our next standard tool.

So it's basically about this.

And last thing: metrics. You remember the opening video? What did you see in the opening video? It was people, well, mainly women, doing sports. So this is what we see in IT in Adidas.

[Music]

Exactly. Who is playing now sports without the watch to improve or to gamify against one of his colleagues? No one. Who manages to watch a basketball game without the scoreboard there? It's impossible.

So the same we need to apply for software.

This is one sentence that I picture myself. It's my most-liked tweet ever, and it's not mine, so this is the pity. It's from the great Kelsey Hightower, the guru of Kubernetes. It states that data is what we collect, becomes information when you analyze, and knowledge once you understand it. That's the key.

Why are we talking about data? It's because it's our obsession. As a sports brand, performance brand, we are heavily based on data: data for our growth, trends, also sales, NPS, consumer NPS, and also employee NPS. We are measuring everything. Everything as a company.

And I was here. I had the pleasure to be here last year, also attending the workshop of metrics with Jez Humble and Nicole. It was also some eye-opener for me because I was super aligned with this.

Nowadays, no one believes that IT doesn't matter, like in the past. Even you see that piece, right? All the unicorns are, in fact, software companies, not business companies.

It's true, and if you read the book, I think most of you have read Accelerate, it's clear that software delivery performance drives organization performance. We are here because we are not unicorns, but also we can do a lot of stuff around that.

That's why we have our own tooling for metrics. It's a system that we built already two years ago. Now it's getting traction because all the teams want to onboard them. It's basically taking data from the APIs of our different systems: our code repository, our ticketing system, our quality system, our CI integration platform, and putting data together.

This data is being used by the teams to improve themselves in the dailies in front of the detailed data. Now we want to roll it out to put some global metrics in the company, and how IT is enabling business. That's our next thing with the platform.

As a new team, what keeps us busy every day: the onboarding, as I said. We have created a concept to onboard all the engineers together and work with each other, improving our platform. We are releasing our open source portal next month.

We are only focusing on things that matter, mainly our e-com, our mobile app, and so on.

And what's next? As I always tell, we don't know. We know that we are in a transformational team.

They are rushing me a little bit.

This is what I always tell our people: make yourself redundant, and you will always have the next thing to do. It's also our belief. If we don't exist next year as a team, we will have succeeded as a company because we will not be needed, and the different teams with business, they will work in a proper way.

And that's it, basically.