Gebrian uit de Bulten

My name is Gebrian uit de Bulten. I'm from Accenture.

Who has ever heard about Ingenico? Well, not a lot of people, so I think, Vincent, there's a good job for you to explain what Ingenico is.

Vincent van Kooten

Can you guys all hear me? Yeah? Cool. I found the clicker. This is the clicker, right? I just found it over there.

So, Ingenico. How many hands did go up? Five or six? I'm not sure.

During the break, actually, I told Gebrian I went out to quickly get a small Paddington Bear for my kid at home, so I have something to bring tonight. And I went to pay, and what did I see? The terminal was an Ingenico one. It's one of those companies that you've never heard of, but once you see it, you'll never unsee it anymore.

So tonight, if you go to the hotel, or you go for dinner or whatever, and you have to pull your card and do a payment, take a look on the terminal, and I bet at least one time tonight it will say Ingenico. So that's who we are.

Before I get to that, Ingenico is a French company, was founded somewhere in the '80s, and the core business is payment terminals and all the backend systems. I think we have about 32 million terminals running right now globally. So that's a lot of payments every day.

But today, I'm not for that. I'm from Ingenico ePayments, and that's all the electronic payments. So no cards, no terminals, everything online, which is a very different business, I would say. And so we call ourselves a payment service provider.

I don't know if anybody joined the session earlier with Worldpay? No? So that was our competitor. It's good that nobody was there. Yeah, so nobody was there. I paid attention.

What we do is, for our merchants, for our customers, we make online payments very, very easy. We provide you one single API to be able to do online payments in 150 countries, in 150 currencies, with PayPal, or with credit card, or local bank transfers. And we take all these interfaces and group them together in one API, which you can imagine is very challenging. And we also need to be available 24/7.

We do billions of payments every year, so we cannot have one single second of downtime, actually.

Gebrian uit de Bulten

No. I think we had a discussion earlier. If this system goes down, I won't say the internet goes down, but you will be aware that some of the biggest sites in the world will not work.

Vincent van Kooten

Yeah.

Gebrian uit de Bulten

It is not a homegrown small application, let's call it that way. It's the real deal. The biggest sites in the world are using this system.

Vincent van Kooten

Yeah.

Gebrian uit de Bulten

I think that's why it makes this quite exciting.

Vincent van Kooten

It's quite exciting, to be honest.

And if you saw the title of the session, it says to go from code commit to production within a day, right? Well, to give you a spoiler, we're not there yet, but that's where we're heading towards. And you can imagine that on a system where we have payments real-time going on the fly, it's a very delicate way to do.

We come from a time where we would do five or six releases a year, and we'll show you where we are right now and where we're heading towards.

I won't bore you anymore with other numbers. You can look it up on the website, so I'll skip this marketing slide.

What we want to tell today is actually our journey. And our journey started as a startup. In the mid-'90s, ePayments was not a big business yet. It was a niche. The company started to develop some solutions, and as the market for online payments was growing, we were growing as well.

You could say maybe we were a unicorn back then. But everything was going very, very smooth. Over time, we had more competitors entering the market. We started to, I would say, make our processes more complicated.

It wasn't a startup anymore. We had all kinds of different roles, all kinds of controls. And what we noticed is that, I would say around 2013, we saw, "Hey, our business, it's stabilizing. We're maybe even declining a bit, and while the market is still growing, that's not good. We need to change something. We need to do something."

So that's when actually we started to scratch our heads and see what do we need to do next.

Gebrian uit de Bulten

We use it as a hook throughout the day. So we covered the past in a bit. We skipped the startup part.

Vincent van Kooten

And what we decided back then is to, on organization level, architecture level, on delivery pipelines, we need to do things differently. We'll get into that later.

Last year, we didn't have a lot of people there that saw the presentation, so maybe if you like the talk today, look it up, because we show you some of the techniques we used on making some technical improvements and changing organizations, and we can see that it's already paying back.

So what we're going to do is show you where are we now and what's the outlook, where we're going for.

I think this picture shows a bit the journey we're going through, and we're now at the point that things are getting much better, and we have a really good momentum now to, I would say, to start to fly. And I'm really proud about that, so that's why we're here today to share that.

Inspired by the best. Like last year, we have three major inspirations, which on the one end is Spotify. I don't know if people are familiar with the Spotify model. I see some nodding. Yeah.

So Spotify is really organized around features and products, not around technologies. I'll show you how we embed this in the organization.

Architecture: microservices. Last year, we went quite in-depth on how we've been approaching this to go from the monolith to small microservices. And this is also something which we made some great progress, but we're not there yet. Lots of things need to be done. But it is good enough already to actually enable feature teams and product teams. We'll show you in a bit.

CI/CD: we come from a time where we did a couple of releases a year. Well, that's not where we are anymore, and we've been investing a lot in automating the pipelines using all kinds of tools, from Docker, Jenkins, I don't know what. We've been trying everything, and we made some good steps there.

So these are our great examples: microservices from Netflix, CI/CD from Facebook, the organization, we take Spotify as example.

What I can say is that last time when we were presenting, my team, we had some squads, they were actually ahead of these things. So we were already having some feature teams in there. We automated the pipelines, and that really showed the organization on how it should be done.

So from just the teams that I was managing, that we were working with, we're now actually implementing this thing fully in Ingenico ePayments, because we set an example, we showed the results, and I'm really proud of that, that we were able to do that together.

I'm going to leave this to Gebrian. Ingenico is a French company. You see Napoleon sitting there, so I’d rather not do this part.

Gebrian uit de Bulten

No, else you'd probably get fired.

Vincent van Kooten

Yeah.

Gebrian uit de Bulten

So, I'm from Accenture, so who gives a shit?

This is where we came from, and I'm also not the creator of this slide, but I think overall, the goal was that it was really hierarchy-driven. It was really a hierarchy, top-down.

So if you can move to the next one.

This was a little bit where we come from, from the other items. So the organization, there were more people just looking at one developer that does some stuff, but the rest really had no idea, just talking and just drinking. Probably, we were also a little bit around this area.

Vincent van Kooten

Yeah, for sure.

Gebrian uit de Bulten

For sure. And architecture, it was really one big, overall monolith. It was really a big monolith, and it had a huge engine, but it couldn't drive. That was a little bit what was all going on.

And the delivery overall, as you can see, it was like a train, but a train left, let's say, a few times a year. And if you miss that train, well, a lot of stuff will happen.

Eventually, what is happening, everyone wants to get on that release train. Everyone wants to get to production, because it's a big monolith. Releasing, less than five years ago today, took really months from a code commit to go to production, and I think sometimes maybe even years.

We had a high defect ratio of P1 incidents. And already we started to talk with, hey, the system cannot be allowed to go down, but we still had a lot of P1 incidents.

So overall, if you then look at it, if we want to go with this one big release train to production, even the CTO of the company was involved. So really the whole company would look into how we would bring something to production. So there was a lot of pressure, et cetera.

So this is a little bit the world we come from.

Vincent van Kooten

I'm sure this is quite familiar to many of you, right?

Gebrian uit de Bulten

Yeah, probably. Hopefully. Or hopefully not.

Vincent van Kooten

Hopefully not. But since you're here today, I guess it is.

Yeah. We're one version behind on the slides. Never mind. Let's skip this one. You can do the Spotify, because this is good.

Last time what we talked about is that we had some great successes with some feature teams. So a feature team is really organized around a specific product and not a specific technology. Take, for example, credit card payments.

What we did in example of Spotify, we've been identifying tribes. And a tribe is a group of teams, Scrum teams. And a tribe has a goal, has a purpose. So for acquiring credit card, it is to give our merchants the best credit card payment experience they would ever have.

So what we do is we give the teams a very specific product-related purpose, so they can actually get better and better on the topic.

And also what we do is we get rid of the old way of organizing projects. You remember with waterfall, we had the design stages, we needed to get budgets, and we got rid of that. So what we do is we actually pre-finance for the entire year the tribes and squads.

We have a budget for development, and we just say, "Okay, for acquiring, we're going to spend X amount this year to have X teams that are going to do this." And then it's up to the business, together with R&D, to start prioritizing within that capacity.

So we have capacity as a given. We're not going to fight about budgets. This is the capacity we have. The next thing we need to do is set priorities within that specific product or feature that we want to work on. So that's what we have in the tribes.

Gebrian uit de Bulten

I think overall, what makes you strong is that it's, I try always to repeat this constantly, really about the product. It's not about anymore we have a backend system, we have a middleware, et cetera. We have a frontend system.

Vincent van Kooten

Yeah. It's really about the product itself.

For example, I used to be responsible for the Java development in ePayments. Right now, I'm not. I'm responsible for two tribes as a servant leader. I'm not telling them what to do. I'm just looking after they have all the means to work, that they can actually deliver. And it's a very different concept of how we worked.

We have different technologies inside the tribes to make them as autonomous as possible, because we don't want them to go to other teams to develop this or to go to another team to develop that. And sure, we still have dependencies, but we try to cut them down as much as we can.

This is actually something recent, because we redid the entire ePayments R&D organization. We made a new blueprint of what tribes we should have, what squads we should have. And actually, we let the people choose themselves what teams they wanted to join.

So we just put the organization on the wall, and the people said, "I want to be in this team. I want to be in that team." And it was really powerful, and as management, a bit of a risky thing. It felt like that, at least. But it worked out really well, and this is something we did just in the last two months.

What we want is autonomy in the teams. I'm not telling them what to do. Together with product, sitting together with the developers and the ops engineers, they have their own product. They own it.

They need to make sure they automate things, we get to production as fast as possible. They set the roadmaps, and I'm just there to make sure they get the stuff they need to get going.

And something that Gene actually asked us in the review of the slides is to go a little bit more in-depth on the collaboration between us and Ingenico, because when I joined the company, there was lots of fighting going on.

We were really collaborating in a traditional way, I would say, where most of the things were outsourced. We had contracts. Everything was pinned down very specifically. This needs to be done. We had fights about the things that were not done, and this didn't work out.

We were more wasting our time on actually fighting each other than adding value to the company. So on both sides, we had some efforts to get rid of that.

What we have right now is, within these squads, there are Accenture people in there, but it's tribes and squads. We don't say, "This is Accenture. This is Ingenico." No. The people are inside those, and we hold them responsible as a team.

And so we share the responsibility for the delivery, and we focus on the behavior of the teams, which is very important. We have hybrid teams, so mixed vendors, in-house people, all together responsible to do what the purpose of the team is.

And what the added value in this case of Accenture is, is that they actually inject knowledge in our teams. Some knowledge on maybe some DevOps transformations we don't have. Well, the guys join the teams, are part of the team, and they make sure the level of maturity goes up, and we can adopt and learn from that and apply it also in the other teams.

And it's all about trust, I would say, and autonomy, and in the end, that results in more value added to our merchants. And I think the collaboration works pretty well.

Gebrian uit de Bulten

Yeah, I think a few things to mention here is especially shared responsibility.

In the past, I was responsible for the part, and you just have, like you have in your own organization as well, there is stuff that you have trouble with, et cetera, but Ingenico didn't always see that.

And now we went to a shared responsibility, both Accenture and Ingenico, and so they're really fighting for the team itself. So it's not about the vendor or whatever. It's really about the teams themselves.

And I think that's as well, we have the called hybrid teams, where we have a team. One team consists of one in the Netherlands, one in Pune, one member in Mumbai. So one team consists all over the world, and we've got it to work.

Vincent van Kooten

Yeah.

Gebrian uit de Bulten

So overall, that's a little bit from the organization part.

If you look at the architecture, I'm not going into this too deep. If you really want to see a little bit more or have a lot of discussion, because this is quite a big topic overall, just talk to me after this talk or look at the one from last year.

What we do is what we call the strangler pattern. We had this old, big legacy system, and we slowly created a new, let's say, little bit greenfield architecture next to it and moved items from the legacy system to this new state-of-the-art system.

And from that, instead of just using only the strangler pattern to kill the existing tree, like the strangler tree, we also create new services next to it.

And that way, and to be honest, without that, we were not able to switch our teams in the way we did. Because this way, we have a service particularly, for example, for PayPal payments, so we can have a PayPal team as well. And that's where a lot of benefits come from.

Let's go to the next one, because I don't...

Vincent van Kooten

Yeah. To give them autonomy, you need to give them their own part of the application, right? So the only way to do it is cut it down in pieces.

Gebrian uit de Bulten

Yeah. Conway's Law.

Overall, I just wanted to add also a little bit to our pipeline. Probably you can't see it, but this is the whole pipeline. This goes from checkout to setting up a complete development environment, to integration environment, to creating a staging environment.

But what we also do in the pipeline is giving automation on approvals. So if you want to approve something to production, this is currently, to be honest, still in the works.

I got an email two weeks ago from one of our DevOps engineers, and he said, "This is the wet dream of every DevOps engineer," because it goes all the way to production. It's fully to production.

And be aware, we are a payment system, and officially, we're also a bank. So that means there's a huge amount of regulation on this. So we have tried to automate almost everything to also handle all the regulation.

And you can see the results. We have more than 30,000 full test cases. We have some environments in the cloud, but we're releasing to production at least every week. I think we had... Was it two months ago that we did 18 times in one month?

Vincent van Kooten

Yeah. There is a next slide. I'll show you.

Gebrian uit de Bulten

Okay. Next slide.

Decoupled releases and deployment. What we mean with that is that instead of just putting something to production and releasing it to the whole wide world, right? You deploy it, and you release it to the whole wide world. We are not doing that.

We are putting something on production, but it has almost the same functionality as is. So we're putting something on production, and then later on, like with feature toggles, we set new functionality to production. And that makes it a lot easier to have going to production.

So, yeah, 99% uptime, but I think this is really...

Vincent van Kooten

Yeah.

Gebrian uit de Bulten

...where it's all about, right? What is the result overall?

Vincent van Kooten

I'm really, really proud about this. This is for one of the teams, not all the teams, but this is the team where we started with and with the discussions last year.

You can see we had some struggles in 2015. We didn't get a lot of changes to production. And you can see every quarter, it goes up and up and up, and the speed is unbelievable right now.

And it actually results in bottlenecks elsewhere in the organization, because we need to be able to think quickly, prioritize what do we actually want to do. How do we keep our merchants up to date about new features? How are we going to do those things?

So I'm really proud of it. I looked into the numbers of Q2. I didn't put them in yet, but we're going to beat the Q1 again. It's going to go even more up than this. And this is why we do it, right?

Gebrian uit de Bulten

Yeah.

Vincent van Kooten

And this allows us to give features to our merchants and beat the competition, basically.

Gebrian uit de Bulten

And just to give you a little bit of the numbers. So this is not, for me, just a 10% or 20% increase of what we bring to production. This is 40 times more elements that we bring to production.

It's defect fixes. It's new functionality. It could be even bigger projects that we bring eventually to production. Oh, it's not projects anymore. Sorry for that.

Vincent van Kooten

Yeah, no worries.

Gebrian uit de Bulten

And overall, I think this is the core. We saw, I'm not sure if you were in the talk before as well, where you see that everywhere, those big numbers.

And this is a company, this is a system that is so important, has so much impact, and we have an increase of 40 times. And to be honest, if we can do it, you can do it as well.

Because we said there was fighting, but it was really almost... I don't want to say physical fighting, but there was a huge amount of challenges within the organization. And this eventually we have overcome. It's a lot of sweat, a lot of tears as well, probably.

Vincent van Kooten

I guess you need a good fight sometimes to improve.

Gebrian uit de Bulten

Yeah, I need to have sometimes a good fight.

And that's why we're here. And I think you heard my story. I'm also talking a lot about we. And while I'm still a vendor, I feel this really is something we did together. And I think that's really where we should go.

Vincent van Kooten

I think we only got three minutes left, so let's do this quick.

Gebrian uit de Bulten

I'm almost there, I think.

Vincent van Kooten

Yeah.

So what's going to be next? What are we going to do?

Three topics back. Organization: we need to stabilize what we created right now, because people still need to get mature. We need to improve there. So we do a lot of coaching in the teams.

Embed the culture. It's not just changing the organization structure, but it's also a mindset. So we're really investing in that.

From architecture point of view, we have microservices. We split things up, but we're not there yet. There's much, much more to divide, to really give autonomy to the teams, to develop their own bits and pieces without being impacted by the other teams.

And delivery: one day from code commit to production. Well, we're now about a week. It used to be months, and then maybe next year, we're able to reach this. I think I'm pretty confident about that.

Gebrian uit de Bulten

Yeah.

Vincent van Kooten

So that's really where we're going.

Gebrian uit de Bulten

Yeah. And I think overall, if you look from an architecture perspective, I like the Lego blocks as well, because there is standardization. There is exactly the size of one of those elements, and that's where we want to go with the architecture as well: plug and play.

I think from an organization level as well, there's a lot that we changed over the years here, and people need to get used to it, especially the culture is not something you will change overnight.

And I think the delivery, we have weeks that we have pushed three or four times to production. But we want to get into a more regular way. We have, like I say, the big release that goes at least every week out. And we call that a big release. In the past, it was extremely small release. And we do smaller ones per day. But we want to get into even a faster mode as well.

And I think overall, that's it.

Vincent van Kooten

Yeah. I guess that was the last slide.

Gebrian uit de Bulten

Yeah.

Vincent van Kooten

So I don't think we have any time for questions, actually. So if you want to reach out or want to know something, just ping us in the app or whatever, or visit us.

Gebrian uit de Bulten

Yeah, I will be here. You need to leave.

Vincent van Kooten

Yeah.

Gebrian uit de Bulten

I will start running.

Vincent van Kooten

I'm going to run off right after this meeting, but just reach out to us if you want to know something.

Gebrian uit de Bulten

Yeah.

Vincent van Kooten

So thanks for the attention.