Evelijn Van Leeuwen

Hi. Happy to be here, I have to say that. Really proud to tell our story.

This is me. I'm a manager at ING, and I have a background in banking operations and then software development, and today I'm working in infrastructure. So I worked on a lot of the parts of the value chain of ING. And I love sneakers, so that's why they're there.

Kris Buytaert

And I'm Kris. I'm Kris Buytaert. I started my career as a developer about 20 years ago, and then I became an operations guy because I was the only one knowing how to rack machines.

I don't work for ING. I ended up there because I was doing what I do, which is evangelizing DevOps, talking about how you do this stuff. And I do that because somewhere around six years ago, I accidentally started a conference in Ghent. You might know it. It's called DevOpsDays. And that's what brings you all here.

So I ended up at ING because I was speaking at an event about how to adopt culture, and then somebody who was sitting in the middle of the room here came up to me. One of my seventh slides was, "Do not call it a DevOps team," and she came up like, "We have 150 of those."

So I ended up at ING helping them out with doing what they're trying to do right now.

Evelijn Van Leeuwen

Something about ING. ING is a bank. We're working in Europe, and that's retail banking, and we work worldwide, and that's commercial banking. And this is what we are: 9,000 people working in IT. I think that's important. And as you can see, a lot of data centers because we're distributed at this moment.

Five years ago, that's I think where we want the story to start, and that was not DevOps when it started. We were in a situation like this. We had a lot of managers. We had a few engineers, because we thought outsourcing would be the way to make our IT better, so we outsourced a lot at that moment. And we had a lot of processes. So I think that was the most important part, and it ended up to be like a bureaucratic organization.

No focus on the primary process of making software. So people telling me, when I said, "So show me what kind of software do we make?" And they said, "No, we're not making software. We do projects." Okay. But that project, it was in the internet site. You can't go back. Yeah.

But what I really want to tell about this is it was really hard for people to take responsibility. We had so many processes, so many steps, so many different people doing different stuff, that you weren't able to take responsibility for something if you want. Only small parts, and that didn't help.

And great engineers didn't like us anymore, and I think that was a really important one. They didn't want to work with us, and that was our problem. So I didn't hear people address it, but that was our problem. They didn't want to help us out anymore.

Kris Buytaert

Part of the reason was, if you remember Damon Edwards' picture where there's a wall of confusion between development and operations. After about three months at ING, I was like, well, you have development, you have people who think they do operations, but what they actually do is managing some applications, and real operations work, they never even touched about it. And then it's a different team that's doing operations, and I think that's even not a clear picture.

I think most large organizations are looking like this. That's their silos. That's their walls of confusion. There's a zillion silos, people not talking to each other, people not knowing what's going on on the other side.

Evelijn Van Leeuwen

And so five years ago, a few people came together, and we had the same opinions and values about how you can change an organization, how you can go back to the primary process. And me and another colleague, we just made a presentation, and we went to the CEO and just told him what we want to do, and he loved it so much.

He was our supporter from day one, and I think that helped really a big deal because he helped us all the way.

And we just started like almost all organizations start here, with three teams, and just let it work. And I think what was really nice is that we had a real passion for change, because it was so hard to change in a bureaucratic organization and telling a different story than everybody else was telling. So I think it was really good that we really held on to the passion we had and just tried to change it.

And there was a lot of failure involved in the beginning, of course, because a lot of people didn't understand us. We talked about Scrum because this was agile, what we were doing, and Scrum, and we talked about Scrum. A lot of people said to me, "Scrip? You mean Scrip?" Because we're used to Rational Unified Process, so they were used to other things, so they didn't know, and they were really not used to looking to other companies.

They were really used to, we are a big company, and the things we do, nobody does. So we had to open the windows and show everybody there were other worlds that were the same like us.

Kris Buytaert

So one of the big changes, and that's where I came in. My background is I'm an open source guy. If it's proprietary, I don't touch it. I've been doing open source for over 20 years now, and that's pretty much the only thing I believe in.

And that was, for me, coming into an organization, I had been in banking 15 years before. I didn't like it. And it was about, how do you change this? How do you go from meeting culture to code? How do you go from sitting in a meeting for two hours discussing maturity levels and figuring out whether you are already on a maturity level two on continuous integration or not, versus actually going and fixing the pipeline?

It's about changing the mentality from, "We're going to buy this software, and then we're going to call the vendor," to, "Look, we're going to create new engineers and people who actually understand how the infrastructure works so that they don't need to call the supplier because, well, they know how it works."

So that was really difficult for an organization to understand, and bringing in somebody who actually had lived that, that was a good move.

Evelijn Van Leeuwen

So we just started, I think that's five years ago or a little less than that, and we just started with a few development teams and let it grow. And we failed a lot because in the beginning, we didn't know how to get it working. And I heard other people telling that the build processes weren't okay, the version control wasn't in control. We all had that kind of problem, so we had to solve them one by one.

And I think what was really nice is that people came to us and were really happy that we're trying to do this and came with really great suggestions. When I told my team that they can use Git, they were really, "Yay." And so, you really understand what we need. So that was really nice, but it was also sad to find out that they held them back for change. And that was a good experience for management, so it was really good to see.

And first we did agile and Scrum and test-driven development, version control. First version control was just a room with stickies everywhere, and then we saw we had a problem because we saw that there are the same versions coming back. So we didn't know that, so we really knew that we need to have a build process, and we had it in place really quick.

And then continuous delivery, we just bring it further and further, but we forgot something a little bit. Because we really forgot to bring infrastructure up to speed with us. So we just thought that they were doing this also, but they didn't. So while they're helping us speeding up, their own technical debt was getting deeper and deeper, and I think we learned from that one.

But this is something, for me, this is the most important one, that we were a bureaucracy, and I think a lot of times you are a bureaucracy, so we still are probably. But what we learned is this step from "from" and "to," so where we came from and where we went to. We just want to do that fast because we thought, "Okay, we want to know where we're going, so we just do it."

But the real pain was in the middle. People who couldn't code anymore or didn't code really well, people who could code really good but didn't want to work with us because our processes weren't aligned with the stuff they want to do. So in the middle, there is a lot of pain, confusion, insecurity. Can I cope with this situation, and how does it work? Tell me what I should do.

And sometimes we didn't. We said, "You have to find out with us what we should do in this part." A lot of frustration, but we built there a lot of pride and passion and ownership. So I think for us, we found out that the most meaningful part of our whole transformation is the middle. The middle, where it's the hardest part, and we keep going to the middle.

So the "from" and "to," we're going further and further, and we have every time a new "from" and "to" where we want to go to. But every time we find that there is frustration and we have to learn, and it's getting easier, but not much. It's not getting much easier because it's speeding up also really fast. So take care of the middle part because it's important and it's also one of the best.

Kris Buytaert

That was the transition to agile, and then they figured out we need to do something more. They figured out that actually doing DevOps, including everybody, it's reorganization.

And when I was around there beginning of 2013, they had just published a huge internal document where they pretty much announced a full reorganization by May 1st. I was laughing with that because part of that reorganization was everybody had to reapply for his own job, and he wasn't sure he was actually going to get it.

So they created a setup where everybody was insecure about their future and did not know if on May 1st they would still even be there. And for me and for a bunch of other people, that was complete crazy. This was not how you introduce change into an organization. This was not how you were creating a comfort zone for people so they could start experimenting, and you did not create an environment there where people were allowed to fail.

But if we look at that now, two years later, it wasn't just luck. What they did, they actually enforced an old-style management way to introduce change. And in a way, that solved a bunch of people. There were a bunch of people who were never able to work in a more agile environment, and they left.

Like every change management process, there are the 20% early adopters who say, "Yeah, we're going to do this." Then there's like 60% in between who say, "Yeah, whatever. We'll do whatever everybody says, and we'll just follow." And then there's like the last 20% who, well, they'll say, "No, no." And when we did workshops, there were actually people in the back of the room sitting there saying, "No, we're never going to do this." You recognize them.

Well, that part they did in two phases. The first phase was to actually combine development and an operations team. Well, those people called operations. And they still kept two managers. Now, six months later, I guess, they moved that to only one manager. So it is really a reorganization. It didn't look anymore like it looked before.

But that's a huge risk, because if you're trying to move a large organization like that, it could fail. People will actually leave. When you tell them that the job is maybe not secure, your good people might even just move. So that's one of the problems, keeping talent inside.

So what we did to get people on board, to get people engaged, is we started with a couple of proof of concepts. We started with finding ambassadors who really liked the idea.

I took those ambassadors to DevOpsDays Paris. And Ingrid already noticed, but that's a picture, and there's no laser pointer here, but in the complete right, there's two people from ING. They're not really involved in the open space discussion. This is where, in Paris, we started putting the dot voting and putting the Post-its on the notes, and they were pretty much in the back watching how stuff happened.

We started with the internal IT academy. There was already a presentation about continuous delivery. There was one about version control. Together with a team that was slowly getting ready to do continuous delivery, we actually created a tutorial on how they did it, and it was a document which was continuously being updated.

The first times we gave that workshop, they had a small pipeline which was completely red, and there was an actual screenshot in the presentation, was the broken pipeline. And six months down the road, when we did that presentation for a new group, the screenshot of the pipeline was a fully green pipeline, which actually was deployed to production. So while we were educating people, while we were talking to them how to do stuff, we were actually also improving the teams.

Another thing we did is organize an internal DevOpsDays ING. I'll come back to that because there's more and it's really important. But even further, they started organizing 24-hour hackathons to get people to work together to build really awesome things.

They now have an internal open source day already twice, where people talk about what they built, how they share code, and they talk about real open source. They actually included legal. First year, legal was there listening. The second year, legal was there explaining rules and regulations on how to deal with which license of code.

But the internal DevOpsDays ING, and that's, for me, a really important one, is after a couple of months walking around the organization, I figured out, hey, there's so many people here who never talk to each other. I came to a point where at a different event, I was actually introducing people from the same company to each other. And that was when I realized, hey, you guys should know each other. You're trying to solve the same problem within your organization, but you don't even know each other.

So what we did is we took the exact format of DevOpsDays, so the morning is formal sessions, the afternoon is open spaces, and we brought that to ING. DevOpsDays Amsterdam was late June. Damon Edwards and John Willis were coming over. We asked Kief Morris also to join us in the morning. And they actually gave pretty much similar talks to what they would do the day after on the actual DevOpsDays.

The difference there, however, was that during the open spaces, people could discuss the actual topics which they were dealing with, the actual tools, talk to the people who were really having those issues, and really coming up with solutions, rather than the days after at DevOpsDays, the public one, they could maybe not even mention the name of the tools or the kind of topics they were dealing with, because it was much more concrete for them to talk about their actual problems inside the organization, within their own walls.

So a lot of discussion happened there. A lot of people started changing the way they worked. They started adopting different tools. They started adopting different processes. And they also started realizing that they could go out to other conferences and share their experiences. So I'm sorry for the Target guys and the others. Pretty much our mistake that they now also have to organize internal DevOpsDays, but I guess it's kind of worth it.

During those DevOpsDays, we realized that a lot of the people there were having troubles understanding each other. Even defining what operations is was not trivial. To me, operations was something completely different than to them.

If we were talking configuration management, I was talking Puppet, Chef, they were talking ITIL. Version control had the same thing. So even though they were talking in the same meeting with the same terminology, they were actually meaning different things.

And that was the first things we started to change, really explaining, "This is what I mean with configuration management. This is how you build a pipeline." Another thing we had is, well, when they started reorganizing, we had a scrum master, and a scrum master was basically the old team lead. It was a new name for the same thing. So we needed to figure out different ways of dealing that. A DevOps team is basically a feature team.

Evelijn Van Leeuwen

That's because maybe until today, we know that we're confused, so we know that we have to ask what the other means, because that's also an important one. We're still lost and confused in a lot of the times. And in the beginning, we thought it was not true, but it is. So we really need to understand what the other person is talking about, show stuff, and show what you're doing.

Kris Buytaert

We learned that this isn't over yet. Absolutely.

Different teams within the organization move at different speeds. There's the mobile team, the internet team, who obviously have much more visibility, were moving at a different pace than different other teams. They have different levels of maturity.

And for some teams, that's good. Some teams really can go fast. For some teams, if they go too fast, they lose people because the people cannot adapt fast enough. For other teams, if they go too slow, they lose people, because trying to get new talent attracted, they started getting new people on board who were interested in doing stuff the DevOps way. So there were projects where they were going too slow, and the people who were there were like, "Yeah, but we want to do this faster. We want to automate this." And they lost people there.

Evelijn Van Leeuwen

It's also about the pain in the middle. I think that we really have to find out, and we have to understand that some people are really the people we want to attract as an organization, and we need to support them, too.

So we can't wait till other people are just adjusted. We can't do that all the time. We have to bring it up to speed together, and I think that's new for the organization because in the beginning, we just thought we take time, and then we wait, but there is no time in a financial market, so we have to speed up much faster, and we all know.

Kris Buytaert

One of the examples there was the agile team started realizing that they need to have some form of metrics. They need to have some more insight into the application. So as you can see, they started using tools like Graphite and Grafana to actually build metrics, and they started putting dashboards all over the building.

So one afternoon, we were both walking around in the office, and we come up to a team, go to the guy sitting next to the screen. It's like, "So yeah, it's a nice dashboard here. What's on there?"

"Uh, I don't know."

"Well, what do you mean you don't know? So you've got a nice graph here, so what's this spike?"

"I don't know. I haven't seen that yet."

"Do you even know what's on the X-axis and the Y?"

"No."

So the organization was trying to move forward, putting metrics in place, making stuff visual, sharing their metrics, but a bunch of the engineers were just not catching up. They were overwhelmed with the amount of information and just not realizing what was going on, and it took them a while to actually adapt that and put those dashboards so that they were becoming relevant for the teams that were sitting there, and they could actually use them.

Evelijn Van Leeuwen

But as management, we needed to learn also because what we did in the beginning, just told people you need to visualize, and you need to monitor, but we didn't explain really well why. And why mean time to resolve was so important for us and that you need to understand what's in your dashboards and what are you measuring.

So I think we learned to ask different questions because for me, it was an insight that we asked the wrong things from people, so we needed to learn from that one.

Kris Buytaert

And this for me is the biggest problem: operations and operations. Every meeting I had, I asked, where's the ops guy? And then there was somebody doing some administration on an application. There was somebody who maybe knew something, and it took me, I think the better part of two years, to actually find the operations people. Actually find the engineer who knew how to deploy an application and manage it.

IT infrastructure was not part of the initial reorg, and that was difficult. Not as difficult as typically five years ago where IT operations was the department of no. They were really trying to help, and they were really catching up, but it really is difficult to, if you see developers going faster, building pipelines, and then needing to adopt the same thing.

We spent I don't know how much time to say, "Look, guys, we need to start doing infrastructure as code. We need to start using, I don't care which tool, but start using one."

But the people that were doing that were pretty much the brains of the organization. Those were the guys fixing operations. Those were the guys keeping the platform stable, and those were also the talented people that were interested in doing Puppet and doing continuous delivery of their pipelines.

So what we eventually needed to do was take those people away from their day job, put them in a different city even. We moved them from the Amsterdam office to the Rotterdam office, and we said, "This is a focus team. You guys are going to work for a couple of weeks in sprints, and we're going to build a platform which is reproducible, which is automated," so they actually could get more focus.

Evelijn Van Leeuwen

I think what's also important is that infrastructure was not, for a lot of management, a place to go. It's maybe hard to say, but it is so, and I think it was really important that people went there and showed what in the software development departments, in development, was working now, and we have to translate to the engineers there because the practices weren't adjusted as well because they saw infrastructure as hardware, I think.

Kris Buytaert

Yep.

Evelijn Van Leeuwen

Probably. So I'm really happy that we are doing this now, but it was better if we'd done this before, earlier. We could speed up earlier.

Kris Buytaert

Like this guy. To me, that was one of the awesome learning experiences. This was when I first met him, really. The Unix graybeard, the IBM guy who said, "No, we're never going to change. What do you mean I'm going to automate everything I do? I've got this nice framework, the nice IBM tooling. It all works."

And he was one of the guys we put in that focus team. And after three or four sprints, he was actually giving the sprint demo and explaining to management what they built. And he was so happy about what they finally managed to build, that the way he was explaining it was almost putting people to tears. We're like, "Wait, is this the same guy from six weeks ago who was like, 'No, we're never going to do this. I'm going to do my old stuff again'?"

And the way he understood now that by collaboration, by working together, they could actually move much faster, and they could build a much more stable environment. For him, that was a really great learning experience, and now he's actually helping out the teams much more.

So putting those people together, helping them, for him, that was a really good experience, something he'd never experienced before in the organization.

Evelijn Van Leeuwen

Yeah, the takeaway. So I think DevOps is about reducing... Oh, it's yours. Because you're always telling me that.

Kris Buytaert

I'm telling you that?

DevOps to me is about reducing risk. When you talk to change managers, when you talk to people who manage the infrastructure, what they learn now is that when you automate stuff, stuff goes faster. That's one of the takeaways we have. People now understand that when you show it to them.

The second takeaway for me is that you need to include everybody upfront. Get the infra people in there, get the business people in there, get finance in there. Definitely something we learned along the line.

Evelijn Van Leeuwen

Yeah, and I think what we learned, because business is really involved from day one at ING, and I think that's sometimes different from other companies, and we involved other departments also, but I think we really didn't involve infrastructure, so that was not so good thinking about that.

And if people like it, it's going to grow. I have to be more specific because I always tell, if people say they like agile so much, I say, "I don't think you do it." Because in a large enterprise where it's hard to take responsibility, all impediments you are going to find are really hard to find solutions for, because a lot of people are involved. So it's not really easy.

But if people like it, it's like we talked about Rudy. If people find that they can be a part of the change and they getting better of it, that's what we learned, and most people find that in this change, and then it's going to grow if you want it or not.

And a change transformation process repeats. When you think you're done, you're not, and it's over and over again. So when you think, "Okay, now it's over and we did really the hard part and now we can relax," there is a new hard part. It's even harder than the first one.

I think that the nicest thing about this is that you get used, that you believe that you come through it, and you can learn from it, and that you have enough experience to go in it and get frustrated and find solutions.

Kris Buytaert

If we look at the two and a half years, almost three years that I've been working with them, I've seen a lot of things happen. I've seen a lot of change within the organization.

Yesterday, somebody was joking with, "Don't tell him it's a long road." I'm sorry, but I do need to tell people it's a long road. You do not transform a large organization in three months. It takes effort. It takes time. It takes people who want to support the change. But it is possible to change larger organizations into a new way of thinking.

Evelijn Van Leeuwen

We have one and a half minute left. We have one and a half minute left for questions. Is there a question, or just finished is also okay.

Kris Buytaert

There could be hands in the back, but we can't see anything.

Q&A

Q: Any questions? Did you include security in that middle layer that you're referring to?

A: Yeah, we did from the beginning. But we're still struggling because we're a bank, obviously, so we have a lot of security issues. But we include security from day one, by putting stuff they want in the definition of done or just working with them together and putting automation stuff for them in our processes. So that's what we did, and we're still working together with them.

And now one of our great security guys is going back to security to help them to look to all the rules and regulations and if you can change them more to an agile and DevOps way. So we're working together there.

Q: Which part of the bank did you start with? And any advice on where to start?

A: So the question is which part of the bank we start with. With agile, we started mobile and internet because that was the easiest one for us.

A: And the team I ended up working with was actually doing...

A: Infrastructure or mainframe. Yeah, I'm sorry. Yeah.

A: Ingrid, what was your team?

A: Mortgages.

A: Yeah. That was the team I started working with.

A: Yeah. But now the whole organization is doing it, also the commercial part is doing it, so we're all working together. But obvious mobile and internet is the most easy to do it there, to start.

A: So we definitely did not take the easy road by just doing mobile only and doing the internet. I mean, mortgages was using mainframes. I say was.

Okay. I think we're finished.

A: Any other questions?

A: Time is gone now, Kris.

A: Yeah, it's wrapped. Thank you.