Log in to watch

Log in or create a free account to watch this video.

Log in
London 2016
Share
Download slides

DevOps for Enterprises; "Respect the Monolith!”

Margo Cronin from Zurich Insurance presents at DevOps Enterprise Summit in London 2016 about DevOps for Enterprises.


Margo spoke on the challenges large regulated global enterprises face when implementing DevOps and the impact of behavior change on transformational change.


There is a lot of information about DevOps, the technology, the culture, the behaviour. There is not, to be honest, a lot of information about tackling DevOps in large enterprises and there is certainly very little about tackling DevOps in large financial organizations.


This presentation will talk about lessons learnt rolling out DevOps in a large insurance organization. What are the characteristics of these organizations? They are very different to Facebook, Twitter, IT product companies and start ups. They are large, they typically have an "old brand" and maybe resistant to change, they outsource sometimes both Dev and Ops and sometimes to many different providers, they are global and built on a matrix structure of many business units and segments, they often are running application consolidation programs, they frequently are resistant to cloud based technologies, they are project rather than product driven with large project portfolios, they have stringent but maybe ineffective governance models. When approaching DevOps in this style of organisation Margo like to use the mantra "Respect the Monolith" - where the Monolith is not just the legacy IT systems but also some of the above challenges. Underestimating these challenges will be the downfall of a DevOps initiative.


Margo covers a "successful" DevOps initiative that she set up for the organization and show how it "failed". She covers how security, suppliers and regulation impacted them. This is a great presentation for anyone about to embark on a DevOps journey in a large disparate organization.


Monolith: a large, impersonal, political, corporate or social structure regarded as indivisible and slow to change.


Key Lessons:

Avoid creating DevOps in a vacuum.

Identify the monoliths of your organization to enable behavior change

Transformational change requires behavior change.

Chapters

Full transcript

The complete talk, organized by section.

Margo Cronin

Hey, everyone.

So, despite being called "Respect the Monolith," this session is not about microservices. So if you were looking for the microservices session and you want to slip out now, that's okay. I won't take offense.

But this session is about DevOps for enterprises.

At Zurich Insurance, we found, when we were embarking on our journey around DevOps, that there was a lot of information out there about technology, a lot of information out there about culture, but not too much information out there about the challenges large, distributed, regulated, global organizations face trying to embark and trying to adopt DevOps.

So, let me introduce Zurich Insurance.

Zurich Insurance is a great place to work. It's located in Zurich city. The head office is by the lake. You can go swimming at lunchtime and see the Alps in the distance. It's lovely. We have continuously created a positive working environment for our employees, but there are definitely characteristics in the organization that impact the culture as well.

And so let's look at these. We have three customer segments, retail, corporate, and commercial, that impact our business strategy from the outside in. And then we've three segments internally, general insurance, global life, and group functions, that influence our strategy from the inside out. So there's a balance in the business strategy.

We have a presence in 30-plus countries around the world, and across these, we've 50 different business units. A business unit is essentially a company in itself. It's got its own C-level board, its own dedicated departments. It will frequently follow regulation and auditing and local markets independent of the group. It's one of the things that can sometimes create a them-and-us kind of environment.

We have 4,500-plus applications built on 1,000-plus, officially 1,000-plus, unofficially a lot more, different technologies. And these are deployed across 90 different data centers. There is a consolidation program happening there, but that number is unfortunately roughly right.

We have 55,000-plus employees, and only a very small percentage of that are IT employees, only about 5%. And they make up architects, subject matter experts, so these are developers and operations guys, and business analysts.

And one of the most important characteristics and features of my organization is that we have a major relationship with a third-party supplier, and they do application service provisioning and infrastructure service provisioning for us.

So, I'm currently the head of technology architecture at Zurich. I have always worked in architecture and delivery. I'm very fond of the two. I have a tendency to oscillate between them. Prior to that, I worked for a product company called IONA Technologies. We did middleware. I'm Irish, and I live in Switzerland.

But let me introduce my hero, the monolith.

A monolith is a great big rock. That's definition one. Definition two I really like. It's a large, impersonal, political, social, or corporate structure regarded as indivisible and slow to change.

I really like that definition, and I think, somewhat maybe controversially, the definition in itself could nearly be applied to some of our organizations.

I really like the use of the word political. My organization is really political. I'm not going to talk too much about politics and DevOps, but it's definitely something to consider when you're embarking on a DevOps journey in a large organization.

But what I particularly like is the phrase slow to change. I was struck it doesn't say unwilling to change or unable to change. That was really good news for me. And it made me think, well, what are the factors in my organization that make us slow to change, that build up that viscosity and resistance that make us slow to change?

Have you seen on social media the story of the wolves in Yellowstone Park? Bear with me.

In 1920, wolves went extinct in Yellowstone Park. In 1995, a group of biologists decided to re-release 14 wild wolves into the park. Their goal: to regenerate the wolf population. The outcome was beyond anybody's belief. The ecosystem of the park, the geography of the park, and the topology of the park changed.

So what happened? In 1920, when the wolves went extinct, deer and elk lost their natural predator, and they overgrazed in the park, and they eroded the park. Humans had tried culling programs to control this and had no impact.

When the wolves came back in 1995, the deer and the elk started to graze in different parts of the park to avoid the wolves. So humans killing the deer and the elk had no impact. Wolves killing the deer and the elk, the number they killed didn't have much of an impact. But the deer and the elk changed their behavior.

And the scientist, about a minute into this video, really excitedly exclaims, "They changed their behavior," and that brought about transformational change.

I know in my organization, in Zurich Insurance, we are looking for and we need transformational change, and I believe that comes from behavior change.

But when we look at behavior change in our organization, the first thing we need to look at is what causes our behavior? What causes our current behavior? So what causes my behavior in Zurich Insurance, and what causes the behavior of my peers at Zurich Insurance?

These are common characteristics of an enterprise. They all exist in my company. I'm sure many of you will recognize some of them from your companies.

We outsource a lot of our IT, a lot of our application service provisioning and infrastructure service provisioning. That impacts how I design an application. That impacts how I deliver an application.

We use offshore and nearshore teams. Seven, eight years ago, offshore teams were really popular. Now it's nearshore. Doesn't make too much of a difference. It impacts communication.

We have global IT standards. This means for a particular capability, group IT mandate the technology you use to deliver it to the customer. So, for example, for claims handling, we would say use Guidewire ClaimCenter. For finance, we would say use the SAP billing system.

We have distributed business units that act like independent companies. And because they need to respond to market, and sometimes respond to independent regulation and auditing, they frequently have to go off on their own and build their own IT applications independent of group. This brings about a shadow IT culture, which large organizations suffer from.

And we have different departments that have the mandate to perform different functions. The obvious one here is development departments and operations departments.

All of these affect my behavior. All of these affect the behavior of my peers. And also none of them lend themselves particularly well to bringing about a global DevOps strategy at Zurich Insurance.

So, it's fair to say that we're quite young in our DevOps journey at Zurich.

Here I have a quote from KPMG in 2014 on the impact digital will have on the insurance industry. In 2014, group marketing and communications at Zurich Insurance, along with group IT, went about to build the Zurich global web platform to realize Zurich's digital vision.

This platform would give the business, group marketing and communications, the ability to manage the brand, the look, the feel, and global features globally, centrally, while at the same time giving the local marketeer autonomy over content for the local market.

We had very clear business requirements coming into the initiative: a global platform to realize Zurich's digital vision for branding, sales, service, recruitment, and communication channels.

A digital platform from which we could create a master site in zurich.com, and then create flagship sites for all of those countries I mentioned on the first slide.

It would have all those lovely features that IT marketing people love: personalization, fully responsive, customer experience management, data-driven insights.

And as well as that, a higher return on investment per marketing dollar while reducing IT spend. To that point in Zurich, whenever there was a change on a flagship site, it was a very heavy IT-centric project where the business had to bid for IT. There was a waterfall team put in place, an application team doing changes to the website, and a rollout through stages.

This solution would give the business autonomy that they could do all of this on their own without IT.

It's fair to say we didn't begin off trying to do DevOps. We began off trying to do Scrum because of another common characteristic in my organization, and this is the business versus IT.

And I have the business as the angel there. I'm very careful doing that because I'm IT. Just in case there's anybody watching.

This is a common characteristic in my organization. It happens a lot, and we know it. And for this particular initiative, for the Zurich global web platform, the business wanted to use a different technology than that that was mandated by the global IT standards, those Post-its I had a couple of slides ago.

And so we decided to use Scrum, where we would set as the definition of done a technical verification that would realize the business requirements to identify what we should use.

Global IT standards mandated the use of SharePoint, and the business wanted to use something like Sitecore or Adobe.

And this was definitely the right approach. And it was around about this time that I met my first monolith.

We were all ready for the business versus IT, but actually, my first monolith was us versus us.

The insurance product, it's not like a house or a car or a handbag or a pair of shoes, something you use and enjoy. Banking has the same problem. Your bank account, you don't show it off to your friends. And if you do, you probably don't have many friends.

Also, these products, banking and insurance, they're not like Facebook and Twitter. You don't check them every day. You don't engage with your friends or upload photographs or publish emails or statuses on them.

Insurance is something you buy, and then you put it in a digital drawer or a physical drawer, and you forget about it until you need to renew it or until maybe you have to use it because something has happened to your business or your car or yourself or your family.

And therefore, Zurich Insurance is about identifying and mitigating our clients' risk. We're about protecting our clients.

And when you're a company about protection, this generates many people and groups and processes that pivot around protection and that pivot around protecting the customer. And these people, they're all about keeping the customer safe, so they're all about being right. They're always right.

Have you ever noticed how difficult it is to negotiate with somebody who is always right? And you're probably thinking right now about your partner or your spouse.

And a bit like your partner and your spouse, these are the teams where you need to show the love.

Before you put developers in situ, before you investigate technology and tooling, reach out to the groups that are about what I call the culture of righteousness in your organization, that are about protection.

These were mine. I'm going to give a special shout-out to my buddies in group information security.

If you're in banking and insurance, and I'm sure in many industries, you'll have a dedicated group information security group. I found with them, surprisingly, I went to them, I had a story all around DevSecOps. They were specifically concerned around testing, and how do we maintain the testing milestones and the testing governance that you have in a waterfall initiative when you're continuously evolving a product in production?

So this was what I had to address with them to make them feel safe. And at the end of the initiative, actually, we were able to show them that our product was even better tested than a waterfall project because we were doing continuous code reviews continuously on the product.

I'm also going to specifically mention the operations group. If you're introducing DevOps into a large enterprise, a large organization with many departments, and you have an operations group, you need to sit down with them and you need to explain what you're doing and understand their concerns and make them feel safe. I'm going to come back to these guys a bit later.

But let's get back to my story. We're in the Scrum room. We're trying to do Scrum. We're trying to identify the correct technology we should be using.

I mentioned earlier the global IT standards. The global IT standards don't just dictate how we deliver a capability to our customer, what technology we use, it also dictates what tools we use to deliver that capability to the customer.

And as a result, we began off with a very Microsoft-centric stack. We had chosen Sitecore as the technology to build the Zurich global web platform on, and our developers had many, many, many manual steps to perform in every sprint.

They had to build packages for every stage. We had five stages: dev, SIT, UAT, prod, and failover. Then they had to run manual scripts to convert them to an intermediary state to pass to a run team, who ran another set of scripts to create MSI files to pass them to the infrastructure service provisioning team, this operations team, who needed specific documentation that was supplier-specific to do the deployments on every stage.

The deployments were manual, the backups were manual, and our packages had three states: raw, intermediate, and MSI. Our cycle time was two days, which is a long time out of a 10-day sprint, especially when you have one day for planning and half a day for retrospectives and reviews. So we knew something had to change for us to be successful.

This was when we started to look at DevOps in this early stage, and we realized that DevOps gave us the ability not only to continuously evolve a product for our client, the Zurich global web platform, but also the tool sets to enable us to do our jobs correctly. And so we moved from a Microsoft-centric stack to a more hybrid stack.

I'm not going to go through each of the tools, but just to mention, we moved off Team Foundation version control. Microsoft themselves actually say you should use Git for your source code repository. You should only really use Team Foundation version control as a centralized repository. But moving on to Git, we were able to have a distributed source code repository.

The developers could do branching as part of the daily flow. They really liked the pull requests, which enabled the code reviews, the rapid code reviews, and excellent collaboration, the like, the dislike, the one-click commit.

We had TeamCity for build, Octopus Deploy for automated deployments, and we also used Selenium for the business testers to give them autonomy from the developers. So even though they didn't have a development background, they could easily create test cases themselves.

We also noticed another advantage we weren't expecting, and that was because the DevOps team had both an ops mindset as well as a development mindset, they were able to support the business product owner to create those, what I call, the non-sexy stories.

The business product owner was very comfortable with IT marketing stories, but the non-functional requirement-style stories, the scaling of the platform, the uptime of the platform, the DMZs, the firewalls, these kind of stories. The DevOps team were able to guide the product owner to write these stories and to prioritize these stories at the right time in the backlog.

And so DevOps helped us to Scrum properly. I actually don't think we could have done Scrum properly without doing DevOps. Our cycle time went from two days to three hours. Everybody was happy.

It was around about this time that I started to see monolith two looming in my horizon.

I became aware that in my organization, everything is about project. We have project managers who are managed by program managers, who are managed by portfolio managers. We have project management offices, project governance offices, project strategic offices, and all of our governance is on project milestones.

We have an application team sit down and develop a project, go live, lot of governance here, and then it goes into BAU, business-as-usual state, managed by another team. Even if it's the supplier that does all of this, it's two different teams.

But with DevOps, we have a team continuously developing a product in a room. And I realized that I didn't have the governance in place to enable us to go live. And interestingly, I didn't have the governance in place to enable funding to continue on.

And people moan about governance, but in my organization, we need governance to get funding to continue. So it's very important.

So if you're embarking on a DevOps journey in your organization, look at how your governance is set up. Is it set up to support a project model, or is it set up so that it can support a product model?

Interestingly, I had a governance officer say to me, "When are you going to turn the lights out in the room?" They really had this idea that we had to switch the thing off, and the governance milestone was reliant on switching the lights out in the room. I was very tempted just to hit the lights and ask the guys to use their iPhones or something, but really you need to look at the project versus product model.

But we went live. And zurich.com up in the left-hand corner, and some of the country sites there around. On the platform right now, 18 of the 30 countries I mentioned in the first slide are live. The DevOps team is still working away. The product backlog is very healthy. Some of the stories now are changing. I'm going to come back to that again in a little bit. And some of us moved on to different initiatives, myself included.

But I've kept in touch with the DevOps team, and recently I met one of them for lunch, and he said to me, "You know, Margo, our cycle time has changed again."

And I was like, "Oh, cool. That's awesome. Is it faster?"

And he was like, "No, it's terrible. It's right the way back up at two days."

And I said, "How did that happen?"

And he said to me, "Well, everything is still there, everything is still automated, but the supplier has come in," and essentially explained that the supplier had put in place a series of manual steps and manual documentation that the supplier is reliant on around our automated processes.

And our cycle time went from two days to three hours, essentially right the way back up to two days again, but this time for a different reason.

So what was the reason? Why did it happen? Was it the DevOps team? Did they drop the ball? Was it the supplier being jerks? No, it wasn't that.

What happened was I didn't respect the monolith.

And what I mean by that is, I overlooked the fact that in my organization there are very restrictive contracts that explain and dictate how my supplier behaves. Back to that behavior point.

In our case, our supplier is penalized when there are incidents in production. Our incidents in production and in pre-production are given ratings: severity one, severity two, severity three. Depending on the rating and depending on the stage, you have a different type of penalty.

We talk a lot now in our industry about this idea of moving from mean time between failure to mean time to recover. There's no point making this movement if you have a group, or a person, or a supplier in your organization being penalized on incidents in production and pre-production.

So the supplier, understandably, was all over any server where they could get penalized on, and once they were given any kind of breathing room, they were all over our automated processes. It doesn't matter that there's an autonomous DevOps team in place to fix these issues very quickly. They still get fined.

I also overlooked the fact that there are many managers in my organization whose full-time job it is to manage this supplier. They were a nightmare. To keep these guys, rather than working with my customer, or focusing on the product, or supporting my DevOps team, I was being pulled into meetings, governance meetings by these guys, and being asked to explain what "Dark Ops" means.

And finally, I overlooked the fact that the supplier wants to be part of Zurich Insurance. And once our contract is over with them, they would hope that we could renew. They want to be innovative. They want to be proactive.

It's interesting introducing DevOps in a Scrum environment. In organizations like ours, in Zurich Insurance, we would ideally like something like DevOps to be adopted through the organization. For DevOps to live outside of the Scrum room and to be adopted wider through the organization, you need to be careful not to build DevOps in a vacuum.

By working around that particular monolith, the supplier, and not working with the monolith, I built DevOps in a vacuum.

So we're now addressing this monolith so that we can have DevOps adopted wider through our organization.

Which brings me to monolith four. This is a thing that we would like help with. Ironically, it's the classic monolith, what we all think of when we talk about monoliths in IT: the large enterprise application that's very difficult to break down.

So back to the Zurich web platform. As the countries are onboarding, some of the stories are becoming transactional in nature. The scope is changing a little bit. Earlier, microsites, transactional sites, and e-sites were out of scope. Now some of the country sites want to perform a transaction online. They want to enable an agent to close a policy and purchase a policy online.

This means that our DevOps product needs to integrate with back-end monolithic SAP applications that do finance. Now, I know SAP can do DevOps. It's great, actually.

What's important to mention about this is this particular group, the enterprise application, the SAP application, don't want to adopt DevOps. They're happy. They're doing one release every two years. The business units are doing well. I am, quite frankly, just a big irritation to them.

And so I've kind of noticed this. As we successfully adopt DevOps through our organization, the product needs to integrate more and more with back-end monolithic systems, some of which won't change in nature, not because they can't, but because they won't.

How do we correctly integrate those?

What we are noticing right now with our DevOps product is this SAP system rapidly becomes an impediment in our testing, and that sometimes the test data isn't current, and it impacts the quality of the DevOps product.

Right now, what we are doing, we're currently working on this monolith. Right now, what we are doing is looking at automation with SAP, where we would automate some of the aspects of the SAP product to enable that DevOps product to move at a faster rate.

And we are wondering, for DevOps to be successfully adopted through our large organizations, does automation have to come hand-in-hand? Is automation the other side of DevOps? I don't know. We don't have the answer, and this is what I would love to learn more about from other people at this conference. Come find me for coffee.

But in conclusion, these are the monkeys scrambling around the monolith from 2001: A Space Odyssey.

In your organization, as you adopt DevOps, avoid creating DevOps in a vacuum. To avoid creating DevOps in a vacuum, you need to identify the monoliths in your organization that impact your behavior and the behavior of your peers.

And by identifying those monoliths and understanding what creates them and making them feel safe, you can slowly create behavior change. And then from behavior change, you can bring about that transformational change we are all looking for.

Thank you.