Building Technical and Organizational Confidence Through Automated Deployments
UWV is a Dutch government agency responsible for the collection and payment of social security for all employees and for helping unemployed people (back) to work. As it is spending taxpayer money on critical social benefits, UWV is subject to intense public scrutiny.
UWV needs to comply with emerging regulations and customers' demands within a complex IT landscape that has evolved over decades and has a traditional Prince2 – waterfall based development culture. Delivering mission-critical software without failure is one of the core themes, especially since every unavailability of their applications generates lots of negative publicity.
Our core challenge was to build trust to change faster and more often in a change aversive culture. This presentation is about our journey and how we pave the way for DevOps in a government agency.
Mieke Deenen, Project Manager Deployment Automation, UWV, Netherlands
Chapters
Full transcript
The complete talk, organized by section.
Mieke Deenen
Hello everybody. Welcome to my presentation. I will tell you something about a Dutch government organization and its journey to DevOps.
First, let me introduce myself a bit. My name is Mieke. I'm from the Netherlands. I like to hike and climb in the mountains. I sometimes do it with my family. That's my girlfriend, Yvonne, and that's my son, Olaf. I sometimes do that on my own. I climbed Kilimanjaro last year. I'm now preparing for Mont Blanc.
I'm an independent consultant. I help companies with innovative projects, and right now I'm helping UWV with deployment automation.
We are a government agency. It's our vision that people are at their best when they can participate in society by working, and we also believe that society functions best when as many people as possible can participate in it by working. So we offer people new prospects, and if work is not possible, then we ensure that income is available quickly. So that's also our logo: working on prospects.
We are situated in Amsterdam. I have 18,000 colleagues. We execute the Dutch social and labor policy. So actually, almost every Dutch adult is insured with us, and we have at the moment 1.4 million benefit recipients, people that rely on us.
So our clients are a very vulnerable group. Usually, they are people that don't have savings. They live from paycheck to paycheck. And they're not on our website because they want to. They're on our website because they have to. And when we fail, when we are one day late with our payments, we get these people in serious trouble. They might not be able to buy their groceries. They might not be able to pay their bills.
So every time we have a little mistake or we fail, then it's big national news and a serious problem for people.
From an IT perspective, we have a large enterprise architecture, about 300 applications. The front side is quite modern, but the back side is a bit older. The oldest application we are currently running is from 1960.
We are a siloed organization. We have our development outsourced. On one hand, we have all the work on acceptance and production outsourced with another hosting party. We are a process-oriented organization. We are risk-averse, and our three main KPIs are stability, availability, and performance. We do everything to ensure that people have enough money to buy their food, and it's not our fault when that doesn't work.
So let me tell you something about our history. In 2012, there came new legislation, and we had to digitalize all our services. So we had a static website with some basic information, and we had to create a dynamic website with self-service portals for employers, employees, for our business partners. And we created a lot of new functionality, and we only had a couple of months to do this.
So we managed to do that, but then we suffered some small crashes, and we asked an external company to audit our website. And they wrote a report which made it very, very clear. We already had some fear for that, but they said, "Your website is not future-proof."
So the next year, we put all our effort into increasing the stability of our website. And nonetheless, in the summer of 2013, we suffered a major crash. It lasted for several days. It was big national news. It was on television every night. It was in all the newspapers, and it's probably the worst thing in our history.
So the result of this was that they tightened the process. This picture is actually our change process. Change management is the guard of this process. They're in the watchtower. Our three main KPIs: stability, availability, performance.
If you want to change something on production, you have to register your request for change 42 days in advance. Then there are several quality gates. There is a freeze period at the end where there's some testing, and you have to pass the whole process to get something on production.
So the result of tightening the process was that it was even harder to release anything. It was very difficult to pass the process, so we almost had no releases anymore. It was like once a year or twice a year, maybe if you were really good at it, three times a year, but that was it.
And that also created more technical debt, more problems, an increase of costs, and people had lost their faith in innovation and in change completely.
So that was the situation then. We figured, well, this is not what we wanted. We have to find something different. How can we increase the quality? How can we increase the availability of our applications? How can we provide new functionality to our customers and help them? Because they rely on us. We have to.
And then, of course, we came with this question: can we do DevOps? Because we are not a typical DevOps organization, but can we do it?
Well, that's actually when I started to work there. And we started very small with a proof of concept. We built a sandbox in a test environment with a small application with low impact, and our focus was mainly technical. We just wanted to answer the question, can we do this?
And, well, you might not be surprised: we did. We doubled the time that it usually took to deploy that application, but we had technical proof that we did it. And the extra time was because we had some extra tests, some extra checks, so that was okay.
So then we went on and we started a real project. We started for one division, customer service. Customer service has 30 applications. I wrote a project initiation document and a business case with the costs and the benefits, and then asked for permission to start, and we started it.
When you start to implement a DevOps project, I think there are some regular challenges that you have to face that every company will face, and I think that there are also some challenges that we faced and are typical for a government organization or an organization that is siloed. And these challenges are the ones that I would like to share with you, and also how we managed to get through them.
Well, the first challenge is that we are a siloed organization. We have our developers on one side, we ourselves are in the middle, and our hosting partner at the end. And people are not supposed to talk to each other. Developers are not supposed to talk to people who are working on operations and production. So that's very strictly separated.
And I figured, well, I cannot remove the walls. I cannot change the contracts. But I can show them what is behind the wall. And I invited them all to come at our site and work together, and I showed them who the next person in the chain would be and showed them what his work was.
So we started to communicate. It wasn't easy at first. It sounds very easy. At first, they just felt like they were forced to work at our place instead of with their own colleagues. But it was a start.
Of course, it wasn't a team yet. And in my vision, I wanted a team, a DevOps team with agile values. So I started to make a Scrum board, and I started with daily stand-ups, and we started to share things. And that was also very unusual to them because they were used not to sharing any details, especially not when it was difficult or something went wrong.
And I wanted transparency. I wanted them to share everything. And I created a place where it was safe to talk and to share, and it created more focus, more commitment. We realized that we were all working on the same thing, and it wouldn't help if we would blame each other. We just wanted to help each other. So it created commitment and focus.
And we became sort of an island, working in a different way than the rest of the company. But it became fun.
The next challenge was the process. This is actually the same as the airport before, but now it's drawn as a process. And I figured, well, I cannot register a request for change 42 days in advance. I don't know all the details yet, but also, I don't want to. I felt like this is the complete opposite of what we are trying to achieve. It has nothing to do with making things easier.
So I invited the change manager for a cup of coffee, and I told him that I was going to implement a new way of working and deployment automation, and I would like to talk with him to adjust the change process. And he just smiled and he said, "Well, lady, if your way of working is not fitting with this process, you have to adjust your way of working."
So we had another coffee, and I figured, well, we're both working for the same company. We're even both working for IT. We have the same CIO, and this is my assignment and that is your assignment. But somehow, our CIO seems to think that we are able to work together and that we are both able to fulfill our job. So we have to work this out.
So we agreed to disagree for the meanwhile and started to work together. And he showed me everything about the process, and he explained the purpose of the quality gates and what was really important to him. And I showed him more about our way of working.
And I could also show him that we would actually meet all the acceptance criteria of his quality gates. It was just done in another way and not so much upfront. So I gained his trust, and he became an ambassador, and he helped me through the process and through the bureaucracy.
And the most important thing: after a while, he even gave us some permissions that I could register a change only five days upfront. And he made it a lot easier.
He also helped me with the fourth challenge. We have a hosting partner, and we pay them on a time-and-materials basis. So every hour they spend on deployment means revenue for them. And I had to ask them to help me automate their work. And they, of course, perceived it as that I would ask them to help them cut their profit.
And that was only one part. The other part was that they perceived me as sort of a cowboy, someone who was not willing to follow the process.
So at first it was very difficult, and I had many impediments and challenges and things we had to talk about, and it was very black and white. And then, after a while, we thought that, well, there are different perspectives, but we actually have the same goal. We both want quality. We both want availability, stability.
So I started to ask them more for their advice, how they would do it, and they started to give more advice. So actually, we behaved more like partners, and that made it a lot easier. And they became part of the success. They were helping us to get to DevOps.
The fifth challenge: there was a big gap between the old world and the new world I was heading to. And it sometimes felt for me that I was with one leg in the old world and one leg in the new world.
In the old world, I was a PRINCE2 practitioner, senior ICT project manager. I report to my steering committee every four weeks. I have to ask the executive board for decision-making. And in the new world, I was a Scrum Master enabling a cross-functional team who was doing most of the work by themselves, and we have agile values and are autonomous.
And I figured, well, we have to build a bridge. And sometimes it feels like the bridge is not ready yet, and people are already walking on it, and we have to help them cross the bridge. And I have to translate a lot between those two worlds still.
The sixth challenge was that there were so many impediments that it wasn't always clear what our goal was because we sometimes lost it. So I figured you have to visualize your goal and just go impediment by impediment. Tackle every impediment. And an impediment is not like a showstopper. It's just like when you've tackled that, it's another step closer to your goal.
So you have to be very patient and very persistent heading towards your goal.
But after a while, after six months, we were finally allowed to deploy on acceptance, and that went well. So then I wanted to go on to production, and I figured that would be a new moment for some new challenges. So I had to work out a strategy.
And usually, when you want to deploy on production, you announce that months before. You do it in a weekend, or when it's a small application, in the evening hours. And I didn't do that.
I talked to my change manager, and I said, "Is it all right if we just prepare it and do it, and my responsibility?" And he agreed.
So we prepared it well. We planned it on a very busy office day. I waited till it was about time to go to lunch. Then I wrote an email to my stakeholders. I said, "Hi everybody. We are going to deploy on production. If you do have any questions, please let me know."
I pressed send. We deployed on production. We tested it. Everything was all right. So I wrote another email: "We did it. Everything worked. Thank you. Bye."
So by the time most of the people realized what we were doing, we'd already done it. And although not everybody was very happy with the way of announcing, they were also a bit proud. We gained some support, some respect, some confidence. This actually was a new way of working.
So from then on, it went fast. And during the summer months, we did all the other applications from customer service, and we onboarded them one by one. And by the time everybody came back from their vacation, deployment automation was business as usual for customer service.
So these were our results. We improved the quality and the stability. They now have biweekly releases. We even overachieved the financial benefits, although that was never our main goal. Our main goal was the quality. And we gained a lot of trust. We built confidence in innovation. We built confidence to change.
And I had to present these results in my end of project to the...
Oh, first I want to show this graph I made. We started with zero confidence, maybe even below. And along with our technical automation, we created organizational confidence.
I had to present my results from my end of project to the executive board, and they were very happy, and they made an important decision. They said that it will become policy that all applications should use deployment automation. And they also wanted to have this for all the applications, not only for customer service.
So this is where I am right now. I'm building a factory, because we have to scale. We have to go to 300 applications now. We have a whole bunch of new technologies, which are actually the old technologies, and we just want to make a pipeline where we can onboard every application, where we can adjust, where we can experiment.
So this is where we are now. And I started it, again, by visualizing our goal. We're going to finish the bridge. We're going to finish the bridge to the future so that everyone, every old application, every modern application, everyone can onboard and everyone can cross the bridge. So we have to pave the road to DevOps.
To summarize, these are my takeaways. Maybe you're in an organization who is considering DevOps. My advice would be: just do it. Start very small. Start with a small team with motivated and skilled individuals.
Dare to be different. Dare to work in a different way than the rest of your organization. Try DevOps even if the rest isn't ready for it yet.
Build trust, create bridges. Bridges to people, but also to the future. Make the goal visible and be very patient. There is no innovation without resistance, so you have to be very patient working to your goal.
Keep learning and improving. Look for opportunities. There is no moment waiting for you. You just have to grab the moment. And celebrate your success and do it again.
So this was my talk. If you do have any questions, I'll be happy to answer them. And these are my contact details for if you want to ask them later.
Thank you all for listening.