Log in to watch

Log in or create a free account to watch this video.

Log in
Las Vegas 2020
Share
Download slides

Powering Better Journeys for Our Developers

Product organizations are highly demanding for their internal support teams. Their focus is always high quality deliverables and faster time to market. Internal enabling services have to provide top notch tools, products and experience to their developers so that they can code better, release faster and sleep better (monitor better)!


This talk is about how Amadeus has been helping improve the Developers (& Ops journey) in transforming the way they design, code, test, build, deploy, monitor and support day to day life at work. Every company focuses on customer delight - this talk focuses on Developers delight!

Chapters

Full transcript

The complete talk, organized by section.

Arun Narayanaswamy

Good morning, good afternoon, or good evening, depending on where you are. It is a pleasure for us to be here. I am going to present the story of Amadeus and the transformation that we have seen into making our journey of a developer being better.

This transformation is really important for us, and it has really helped in how our developers spend their time on a day-to-day basis in the organization and how they build the best products for our customers.

So let us look at what we were in the '90s. Most of us were developers in those times, and our job used to be to know everything in the software world: know the infrastructure, know the networks, be a good coder, and eventually deliver the best to our customers.

But what really changed in these years while we transformed from being a small company to a large enterprise? We basically moved into silos, and I think this is not just Amadeus. Most of us did in our own organizations. We started having developers who were specialists: a C++ developer, a front-end developer, a back-end developer, et cetera. We also created roles which were siloed in terms of infrastructure specialists. We also had, on the other extreme, a front-end specialist, a front-end expert. Then we also created developers or database experts, et cetera.

All of these became larger and larger. There was nothing wrong in it. It was more about adapting or adopting to the situation in those times.

What has really changed again? Now we are in a mode where we expect the developer to know everything. They are expected to be a full-stack developer. What it also means is they have to understand the networks, they have to understand the infrastructure. Moving to the cloud, obviously, he or she has access to do whatever they want to do from an infrastructure angle and put things into production, or put things into action, fairly quickly.

So this is a journey that we wanted to cover at a very high level and also give you some case studies on how Amadeus has transformed. But let us look at what Amadeus is so that you get a feel of what our foundation is and where we are coming from.

Amadeus Video

Technology is constantly transforming travel. For more than 30 years, Amadeus has met that change. Always asking: how will global travel look tomorrow?

With our open source projects, we are bringing the world's best minds together to accelerate our industry's progress. We are building new tools that turn big data into better decisions and smarter actions. We are pioneering the way with cloud-based architectures that deliver unparalleled flexibility and agility. We are creating APIs that open the way for new partnerships, new ideas, and new growth.

We aim to develop rich platforms that deliver reliable and secure performance. And we are innovating at scale, delivering new value to the industry, and developing the new technologies that will shape the future of travel.

So if you want to know what travel technology will look like tomorrow, just ask Amadeus.

Arun Narayanaswamy

Now you saw what Amadeus brings to our customers. So I also call upon Dr. Udo, who has been with Amadeus for a long time, and we would like to hear the history of Amadeus through him. Then we will see how our developers are transitioning into this journey within Amadeus. Welcome, Udo.

Dr. Udo Seidel

Thanks, Arun. And also welcome from my side. As said, my name is Udo Seidel, and as Arun said, I am with the company for quite some time.

I have several roles in this presentation. First of all, I can give you some insights where we are coming from, since we are not a company who started business yesterday. But also my background is more operational background. So Arun represents more the developer side of DevOps. I am more on the ops side of DevOps, and we work together.

To complement a little bit what was shown in the video: what is Amadeus doing? We are a world-leading service provider in the travel industry, and we do have thousands of developers who do develop our own applications.

You can see some business figures here which give you an idea of how big we are and how many transactions, how many bookings we are processing. We normally count our business in passenger support and bookings. But of course, the services and the software cannot run on air. They need to have a solid foundation, which is the platform. And we do have quite a few technical challenges there, which comes on top of the transformations we have gone through in the past.

You see that we do process a lot of data. We do store a lot of data. We talk about 55-plus petabytes of storage to use. Twenty bookings per second is what we are processing. And as said, we have quite some challenges to master on the technology side. That is why we have to have the right skilled people. We have to have the right tools, the right approaches to collaborate, to work together, and to provide those services.

On the previous slide, it was mentioned that we are present in 190 countries with 17,000 employees, and two of them are talking to you, which you see on the next slide. Arun quickly introduced himself. He is a director of DevOps platform engineering, working out of our site in India, and myself working as an enterprise architect in the platform unit as well, but based in Germany. So you can already see that this is a virtual conference and we are working in virtual teams. Thanks to tools and the DevOps journey which we are going through, this is like working together sitting in the same office or sitting on the same floor.

Next slide.

Amadeus is in the business for quite some years, 30-plus years actually. And we have gone through several transformations technology-wise, but also skill-set-wise or working-together-wise in the past.

When I joined Amadeus more than 15 years ago, a pretty big part of the company was using green-screen terminals connected to mainframes, and this was the foundation we had to provide the services. But we had already started our journey going out of those legacy or traditional central approaches to distributed open systems, and we have replaced what is inside our data centers big time, going from those mainframes over to racks of servers, pizza boxes, storage arrays, replacing tape robots and things like that.

Of course, along with changes on the hardware side, there were also changes on the software side: new coding languages, new approaches how to develop stuff, new paradigms to provide services, to do code reviews, and things like that. We have also explored those paths already. And as I said, we went through several transformations.

Then internet came along, became available for everybody, became a reliable infrastructure even for us to transfer messages from A to B in a secure manner. And more recently, of course, things like new ways of application packaging, deploying using containers, and adapting the cloud and the different service you can get from the cloud.

So in a nutshell, with those technology transformations within Amadeus, we did of course as well transformations on how we work together, how we separate and distribute duties, how we make a good service for our customers, and also a good service for our developers ourselves. Because what we do externally, we also have to do internally.

And the technology journey has just started. Cloud and containers are not the end of the story. It is just the beginning. You see a lot more things coming up, emerging technologies with lots of opportunities, with lots of promises. And we want to be able to explore, to find out, to decide what is best for us internally, maybe and also externally or vice versa.

A DevOps journey is definitely the right approach to do so and has its own demands and desires and needs. Despite the fact that we have already mastered quite a few transformations, the DevOps journey is something not new, but it is something with its own challenges and desires and needs. Arun will give you more details on what we have done here in the past already and how this has enabled us to master this path. Back to you, Arun.

Arun Narayanaswamy

Thanks, Udo, for giving the history of Amadeus and where the developers have evolved from. I will take it over and basically give a view of what has really changed from a technical or a process angle.

For sure, this is something that you would have all seen. An enterprise journey is complex, and once you get there, it brings in a lot of challenges. When the environments get big, when the number of people in your organizations becomes big, there are things like change control, approval gates, large environments, environment shifting from pre-prod to prod, or dev to QA to prod, et cetera. What really enables developers to move in this journey to be fast?

Basically, one of the core components is the quality of tools or the new-age tools that we bring in. I will give you some case studies in terms of what tools we use, but we also look at what the transformations have been. With more people in, there are more tools. We also enable the platform to be available for developers to develop things fast, which means we need to provide them the best quality tools.

What has been the disruption for every developer within the organization? We have seen changes from waterfall to agile. We have seen open source to inner source. I cover most of these topics in individual slides, but if you notice, these are large-scale transformations. It is not something small which a developer just adopts in a day or so. Many of them go on for months. Some of them, like a legacy to a microservices model, probably go on for years in some cases.

These developers who are developing products for the customer have to be given tools and services which enable them to go fast and not really bog them down with technology challenges.

Let us look at the first one, which is waterfall to agile. This has been going on for 15-odd years now. We have also transitioned from agile to, say, scrum of scrums or SAFe. Then the developer lifecycle has radically changed.

What really needs to be enabled from a tooling standpoint is that we have to build in better collaboration tools. Automation is a mantra. You would probably see it on every single slide. The more the automation, the better or more agile we are. And flexibility and transparency are really important in the platform and tools that we provide.

I have mentioned platform and tools quite often already. What it really means is a platform which a developer can hop onto and start developing on. It is more like a cloud service or a tooling platform which people can consume. They have their own systems to build on and have flexibility to make changes to it, but governed by something central, which enables them to move fast but also gives them the best of the tools available within the industry for them to go faster.

What this all means is to be provided with a set of integrated tools which work seamlessly between each other. There are cloud services, let us say Azure DevOps, we have Google Cloud tools, et cetera, but how are they integrated with each other? The platform provides them these in a way where they can develop wherein they do not have to bother about what is sitting behind the scenes. This fosters DevOps because they transition between one environment to the other, or one system to the other, or one person to the other, in a more seamless fashion.

What is happening on the DevOps to the NoOps cycle? We have almost transitioned from Ops to a DevOps model. NoOps or ChatOps kind of a thing is something that we are transitioning to, and what it really means behind the scenes for a developer is they have to focus on automation. They have to focus on cloud first because the sort of NoOps, serverless, auto-healing, all those kinds of things which are provided by cloud, both public and private, is something that we are providing to our developers.

Microservices transition is also happening during this journey, and it is very important because the more connected tools or connected products we have, but even more disconnected from a point of view of how they interact with each other, whether they have contract-based approaches, et cetera. It is very important when we have to go to a NoOps model, because then we can build in auto-healing, recovery, ChatOps models over and above that.

But what happens on a tooling ecosystem around it is the process of building federated tools. What federated tools mean is very similar to a public cloud provider model, where there is a central set of tools or systems, which is governed, has certain practices and policies or security aspects put in place. But then we give control to a developer where they can make changes, give administrative access, remove access for people in their team, make changes, build plug-ins, remove plug-ins, pull out a tool itself, plug in a tool that they prefer, et cetera.

This enables them to be innovative. This enables them to move fast. This also enables them to have a feeling that they have control on whatever aspect of development they need to be. It is not just the code part, but also the tools that are provided for it.

Moving on to an open source to an inner source model. This is not a transition, but this is a scale-up or an increase in maturity, I would put it as. Most of our products are built on open source languages, but what is the next phase of it? We have to enable people to contribute within each of our own products.

The move away from silos is certainly enabled by this. We do multiple things here. We enable tools which are open. We enable access to all tools and products, which is globally accessed by every developer across the globe. The code could be reused because most of our Git, or rather the Git repositories, are public. Most of our Jira ticketing systems are public. Most of our Confluence pages are public. When I say public, it is public within the organization, where people can see what others are developing and contribute accordingly.

The core component on this is providing tools or platforms which are open in nature, which enables our developer to move fast and improve their experience on a day-to-day basis.

The bigger thing, which is with all, is the move from a data center to cloud or to serverless. Data center and cloud, I would still say, are closer to each other than on a serverless kind of an application. Here, we are looking at transforming a developer to become full-stack developers. We enable trainings. We enable them to understand other aspects of working together. So we first move them into a virtual kind of a team, where they understand operations and development well, and then eventually transition them to a full-stack developer, where they focus on automation, both on the infrastructure side, the cloud side, and eventually the application side.

What it means is we have to bring in new-age tools. The thing that Udo mentioned in the past is we have been a software-first company. What it really means is any tools that we built were initially built in-house. Be it our ITIL system, incident response system, everything were homegrown tools that were built for buyers and for our developers.

So what we have transitioned is transition to new-age tools, which are commercial in nature. We try to customize it less. We use the full functionality and adopt our processes and policies to work with the tools which are available publicly to other customers. But I would also give you another angle to this, which I mention at the end: bringing in new-age tools helps large-scale adoption within the organization. We are 16,000-plus developers, so we need to ensure most of these guys adopt single ways of working, but flexible ways of working.

The other aspect of it is the tooling team providing the tools as a platform. Because our products are scalable, our products are reliable, what it really means is the developer needs to get the same kind of experience when he or she is consuming tools and products from the tooling ecosystem. When I mention tooling ecosystem, these are DevOps tools. The bit here that is very important is the tooling ecosystem, or the platform or platform as a service that we provide, has to be scalable and reliable. These are connected tools where people or developers work with each other and they contribute.

A simple example: if I have to upgrade the Bitbucket instance from version X to version Y, I do not expect the developer to wait silently and wait for this upgrade to happen. Our platform has to be resilient. It has to be sort of zero downtime like our products are. That is a bit that I would want to pick up from the history: most of our products that we build have been zero downtime for our customers, because any person who gets onto a plane, gets to use our platform, be it the airlines, be it the airports, cannot really see downtime because the world keeps moving and they need to see products which are really up most of the time. The same we need to provide for our developers.

Looking at some of the case studies, which is important, I am basically going to tell about what tools we have in place and how these are enabling these aspects of being open, being connected, being integrated.

Let us look at Jira. This is not a new evolution. This is something that we transitioned a few years ago. The Jira ecosystem is helping us adopt agile in a better way. It is helping us transition from homegrown ITIL systems into Jira-based systems. Obviously, we create thousands and thousands of tickets. We have one of the largest Jira instances within the organization, and it helps build integrations and connectivity. If you look at a pull request model, how do we connect our tickets into the code that we are building? How do we automatically close tickets? Et cetera are all enabled through Jira.

Similarly, on a low-ops or a no-ops model, or the things that we are building from our bots within Teams and within the prior collaboration tools, we have 50,000-plus jobs on our Jenkins instance. We do 25,000 builds. Most of it, I would say 95% to 96% of them, are 100% automated. We still have about 4% to 5% which has to stay manual because of some processes we have in place, but we are moving faster and faster to 100% automation on this.

But notice, this is not small scale. All this needs to be enabled. We have to move fast on it. Collaboration on all these tools is very important. A lot of incidents or things that happen on Jenkins are automatically published to Teams and tools, and teams within Teams, so that they can react fast.

Bots are something that we have evolved. We do tons of hackathons within the organization, which enables developers to build bots and move faster and provide things to our customers faster as well.

Bitbucket is one of the core components for our platform. We have put in federated systems around it where people can give access, grant access, revoke access within their own systems. One of the core functions that it enables is being inner source, or enabling people to be, or products to be, inner source.

We do about 2,000 pull requests a day. We have 7,000 developers on it. What it enables is people contributing to my product or one product need not be within the same team, but coming or originating from some other teams. It is still not massive. I would put it as 10% of it comes from external, 90% comes from within, but it is still significant when we look at it from a point of view of how much we get from other sources within the organization.

Collaboration: most important aspect. We have about 12,000 users. We generate about 8,000 pages a day, which basically means most of our communications happen through Confluence and everything is public. I did mention this to you earlier. People can access this. We remove dependency on a particular team or a developer. We enable it to be public within the organization so that everybody can learn from what others have done and the mistakes that others have performed, probably.

One core aspect that I want to touch upon is that we are unique. Every company, even your organization, is unique. We adopt majorly tools and products from the market, but we also build tools which cater to our own systems. There is always that thin line saying: for us to be ahead of the curve, ahead of our customers, do we provide the experience to our developers so that we do not restrict to what the product alone has, but also build in some kind of tools and integrations which help them move faster? This is very important.

The only thing that we put above that is centralized governance and processes, which is a principle, not control, to say how you can be better, how you learn from others and build the products which the customers love.

Last but not the least, to wind it up, I would say we are providing new-age tools to become more DevOps-y. We enable developers to be happy, chatty. We contribute through collaboration tools like Teams, Yammer. We have products like Bitbucket, which enables people to contribute or chat geekily in some ways, and also have Confluence where they can share for long-running documentation.

All this is very important. The most important aspect is being transparent, keeping the whole environment safe from a point of view where they can make mistakes, learn from it, evolve. The core aspect of it is tools can be plugged out and plugged into the platform, which enables them to adopt faster. But not just that: also be innovative in a way where if they feel they have a brilliant tool coming into the market and they want to plug it into the platform, they are enabled to do it. All this helps them in this whole journey and keeps the whole environment fun.

Thank you so much. As I say, the future is now. It is not something that we are looking at in the future. We have already built in tools and practices which enable this currently for our developers. But again, the core mantra of DevOps is continuous evolution. We continue to evolve and continue to improve.

Thank you so much. It has been a pleasure to present to you. Thank you, Udo, as well, who contributed to this from the history and from the learnings that you have at Amadeus. Thank you so much.

Dr. Udo Seidel

Thank you.