Log in to watch

Log in or create a free account to watch this video.

Log in
San Francisco 2017
Share
Download slides

A Story of DevOps Transformations

As they say, nothing is more dangerous than using yesterday’s logic for today’s problems, yet we are still working in our organisations with mental models that were inspired by manufacturing. You can see artifacts of it even in the language we use: people are resources and they work in development factories. If we are surprised why our transformations are not progressing as fast as we hoped when Agile took the stage, then looking to these old mental models provides part of the answer.


In this talk I will explain from practical experience in my work, how the old models still influence us every day and how we can break away from them and learn new models. I will give positive and negative examples from real projects to show that it is normal to experience failures and how to course correct from the lessons such failures teach us.


I will also provide pragmatic steps that everyone can take in their own organisations that don’t rely on buying new tools or following specific methods. Charting your own course starts with understanding where the problem is and understanding where our mental models let us down is part of that journey.

Chapters

Full transcript

The complete talk, organized by section.

Mirco Hering

Thank you guys for coming to my presentation.

I do understand that I stand between you guys and drinks. And at the risk that you all are going to get up and leave now, I saw that in the exhibition hall, they already serve beer. So I'm actually not physically between drinks. You could actually leave now and have beers. So thanks for staying.

So I'm going to talk about... Well, the talk is called "What Got You Here Won't Get You There: A Story of Transformation." And I really want to share a bit of my experiences and what I've learned along the way.

You have to have a slide in on which company you work for. I do work for Accenture. I assume that many of you guys would've heard about us. I'm looking after our Agile and DevOps practice in Asia Pacific, or as I like to call it, just good delivery. Now, unfortunately, good delivery doesn't sell, so Agile and DevOps it is.

What do I do at work? I have a team of pretty passionate change agents that are working with a bit of our clients, and we're really trying to solve our clients' problems.

The other day I was talking to one of our clients, and he asked me why I'm still in the job, though I've been with Accenture for about 14 years or so. And it's really: we are able to solve problems now. In Agile, we are working directly with customers. It used to be you get the requirements document, which is 500 pages long, and all you're doing is implementing that thing that you might or might not know what it is about. And with Agile and DevOps, I think we're really starting to break the barriers.

I also blog at Not a Factory Anymore. If that title sounds weird, you will realize as part of this talk why I called it that. And then the coolest thing is, I saw today, first time, the physical copy of my book, which I'm also handing out afterwards.

So thank you. It is this really kind of nervous, scary thing where you're really worried that people will call your baby ugly. So be kind.

All right, so I do have to admit that I stole the title, "What Got You Here Won't Get You There." I'm not sure whether you guys have read this book. It is absolutely brilliant. It describes, kind of for business, why in your career you might get stuck when things that used to work for you don't work for you anymore. So I stole it. I wanted to acknowledge that.

But what I'm here for is really, this is the core problem that I'm struggling with, right? I came into the workforce 15 years ago, and I believe that we are still solving the same problems. So when I was working for DaimlerChrysler ages ago, we had built Control-M schedules to deploy software into production. We had some kind of monitoring that we had built into the applications to log some stuff out. And here I am 15 years later, and we are still talking about deployment automation. And we still seem to be in a situation where, like one of the speakers this morning said, source code is being emailed around.

So what happened? I guess I wrote a blog about it, but let me give you the CliffsNotes.

So here's where we were. This is a pretty generic DevOps maturity model, but it's just for illustration purposes. We were actually kind of in this place where, 15 years ago, you had to do automation because it was the only way for you to become more efficient. You didn't have any kind of shortcuts to take cost out of your delivery.

So what happened then is that we started to see more packaged software, which meant we don't have to worry about this anymore, right? It's just customizations. The product will take care of everything. We had the opportunity to take work and just move it to a different location to become cheaper. We had these kind of shortcuts that we could leverage. And then we started to get into kind of locked-in environments where you don't even see the code anymore.

And I think that's partly where we kind of regressed over time, and we are now starting to realize, well, now for the last few years, that we need to bring some of that stuff back, that we've run out of shortcuts, that the packaged software requires a hell of a lot of customization for what you need to really be different in the market. And we all know that in the mobile and digital market, you will need to have a differentiated experience for your customers.

So we needed to start to do something different, and that's where we ended up with DevOps practices and tools that are now coming back, and we are revisiting a lot of the things that we used to do.

To me, the problem of why we got there, and why things are changing now, is that it's a mindset problem. It's not that we consciously made these decisions. It's not that we said, "Oh yeah, forget about automation." We just had, or we convinced ourselves, that there are faster ways of getting the answer.

So that mindset shift, and a lot of the things that we've learned, came from manufacturing or from factory models. And I think that's what I wanted to explore a bit with you on why that is not helpful.

But before we get into that, let's look at the kind of meta model for all this. Perhaps this looks familiar to you. This is a transformation life cycle. It basically goes from, we have a problem, so we're running a two- to five-year program to get to the end state. When we get to the end state, surprise, surprise, everyone says, "We are done here, so let's start cutting some costs because we don't need these big teams anymore."

So we're going to start to lean it out. We start accruing some of the technical debt because we don't really want to pay it off at this point, and we start to deteriorate. Until, three years down the track, a new CIO comes in and we say, "Let's do a transformation. Let's implement this next big thing." And we are going through the cycle again and again.

Now, companies like mine make a lot of money out of this. And it's a pain for you guys. So really what we need to get to is this model.

And it's quite challenging because these two to five years were okay in the past. Well, not great, but they were okay. If you now have two to five years for this cycle, you're going to be out of business because any kind of digital disruptor can be much quicker than that. So you won't get a chance to do the next SAP transformation over five years. You really need to get into this model where, A, we appreciate that there's no end state. If you think there's an end state, you're dreaming.

It's going to continue to evolve, and you need to think about what that does to your architecture, to your tools, and to everything else that you do. And then you need to find a model where you can actually have this continuous, ongoing stimulus from somewhere.

And that's obviously a lot of my job, by bringing the ideas from across different clients to my clients so that we can all share, or going to this conference and bringing your executives here so that you can learn and figure out what's happening and what the next stimulus is that you need to give to your organization.

So this kind of anti-transformation transformation is really the transformation you should do, as weird as that sounds.

So let's explore a little bit mental models. I will ask you what you see in this picture, and you don't have to speak out loud, but I know what you guys are seeing. All of you are seeing nine dolphins, correct?

Well done.

And you won't believe it, but if you show this to elementary school children, they do see nine dolphins first. And that's because they have a very different mental model to you. I don't know what it says about you guys that you saw something different. But I've heard that it shows something different.

But that is really the risk of mental models. Why? Because you're seeing something and you are doing an interpretation of it, and that's going to inform your worldview and the actions that you take. Not in this case, please don't, but in general.

And here, I really like this quote because it really describes where we are at. We have an amazing power at our hands with the technologies and tools that we have and the automation, and we are still working with management processes that are coming out of the mid-'20s and based on management principles from the 1900s. And really, that creates a lot of friction.

So let's look at this in a little bit more detail. And I want to start with, because I sometimes get hate mail: why do I talk about not a factory anymore? "Don't you know about Toyota and all the good stuff they're doing?" I do know about them. And there's a lot of good stuff that we've taken from them.

What I really struggle with is that we are treating it as if it's a sweatshop, like a 1900 Charlie Chaplin Modern Times type factory. And that's the thinking that I don't like.

So let's have a look at what that meant. The first thing is that in manufacturing, we have a predictable process. What that means is when we change the process, we can, to some level, guarantee that we get a good outcome. And we can measure the outcome.

Who here has tried to measure productivity in IT and has succeeded?

Exactly. What? You can't. We're not doing the same thing twice. Most of the time, at least all the projects aren't. There's some services where you can do that, but in general, you just can't.

And if you are fixing the process, and we had a... I'm not sure whether you saw Damon’s presentation. The idea that we could put more process in place to fix a problem is ludicrous. You just add more and more Band-Aids, more and more checks, more and more process steps into it, and it doesn't guarantee you the outcome. So that obviously is not actually applicable in IT anymore.

Let's look at the next one. There's a functional specialization of labor. So the idea that we can define something down to the smallest step that someone can execute, like an assembly line.

And I was an engineer when I started my career, and I hated nothing more than getting a tech design, which was pseudocode, that I translated into code to hand it over to someone else who would then tell me that I made a mistake. That doesn't feel right. But that's kind of where we got to.

In manufacturing, that makes sense, that you have this guy putting the same screw in because we're doing the same thing again and again. In IT, we don't. So that doesn't apply either.

Importance of upfront planning. If you want to build a new factory, you do have to plan upfront because you have to set up the location, you have to build the factory, you need to bring your machinery, and then you can start building something. If you want to change what you are producing, then you need to change the configuration of it. So it was necessary to plan ahead.

In IT, a few years back, or certainly when I started, we had the same situation still. If I wanted to run and get a new solution, I had to order some servers. It takes three months to be delivered. They get put into a data center, and it takes another two months to just set all the cabling right. We install middleware. You get it. You had to plan ahead.

Nowadays, you go into AWS and you stand up an instance, and you have a new company very, very quickly set up. And that is different. And that's why we don't necessarily require that much upfront planning.

And if you set up your company in the cloud, don't do what someone did a couple of years ago, where they then deleted their S3 buckets and went out of business. That's also not something that used to happen much. You had to kind of burn down the factory to make that problem.

Automation is improving productivity. That's the one thing that actually applies. So all the good automation ideas, and that's where some of the Lean thinking this has inspired, that's absolutely applicable.

And then economies of scale. I want to expand my production, so I'm going to take one factory and build another factory that looks exactly the same. That works.

In IT, I don't know whether you have had that Agile proof of concept, the first guys who do Agile, and they are super successful. So how hard can it be to scale? You all know it actually doesn't work like that, right? Because the complexities of people and working with each other, the social interactions, the ambiguity that exists, no matter how well you write your designs, there's always ambiguity.

And because you're not talking to the same team, the same dynamics, you will get a different outcome. I don't know whether any organization has spent two times $1 million to see what comes out of two different teams, but I'm sure it would be different. And if you have two times $1 million, let me know. I'm happy to run that experiment.

So should we burn down the factory? And that's really where my blog comes from, and in the book that I talked about, there's a whole appendix that goes deeper into this. But to me, really, that mental model needs to stop because as long as we have this mental model, it pushes us into adding more process, relying on the wrong principles to achieve the outcome.

And any transformation you will do, and as you have heard in the beginning, an Agile transformation or DevOps transformation is not something you can do. You have to be on this ongoing journey, but you have to start breaking those principles.

All right. So here is the model that I use, and it's really three dimensions that you need to look at. You need to look at, of course, the technologies and the technology architecture that you're using, the tools, the principles, the practices. That makes sense.

I think most people have understood now that you need to do something on the people side, the culture side, and that that's really, really important as well. But then you need to look at your ecosystem because you're not alone.

One of the biggest frustrations that I have, and that might be idiosyncratic, no one ever talks about the culture shift or the culture challenge that you have because you're working with other organizations. People tend to talk about Agile: "It's really important that we have the right culture, and we have this cultural transformation." And then they talk about the bit that they're doing in-house, and that's great.

But how do you make sure that your vendors, your SIs, are actually adopting to the same culture or working in the same way with you? Because that's a real challenge. I don't know how many organizations are out there that don't have any partners, but there would be very few. I can certainly tell you that we do business with lots of organizations, and that means those ones definitely have someone in there.

I had this moment a few years ago when I started doing CIO workshops to talk about these kind of transformational ideas, where at the end of the talk, the CIO came over and said, "Mirco, that was a brilliant talk. Thank you so much. I need to introduce you to someone because they really need to hear that." And then he said, "Let me introduce you to your colleague over there."

And I didn't really have a good answer to that because it's kind of embarrassing at that moment, like, "Why are we not doing all this for our clients?" Until I realized that it's actually not set up that way. The contracts that are in place, the incentives that are in place prevent people from doing something.

And that's true not just for us, but for every other vendor as well. They're doing what you incentivize them to do. So we need to change that ecosystem and that culture if we really want to make a change, or you need to in-house everything, which is not terribly practical.

So these are the three dimensions. I will double-click into them a little bit, just so you get a flavor of why the factory mentality and mental model is not helping us.

And the first thing I do is actually talk about production processes. So if you have done an MBA or something and you look into production theory, there is a formula that tells you about batch sizes. And it's holding costs, transaction costs, and then there's kind of the sweet spot where the right batch size is. That makes sense, right?

Now, holding cost is something we can't do a lot with in IT. So that's basically you have all this functionality that you haven't deployed to production. That's all those value that you don't get. It's the cost of delay. That's ultimately your holding cost.

The transaction cost is everything we do to release something. So our governance processes, our deployment processes, our environment standups, our development, our testing.

Now, we all know we want to have smaller batch sizes. And we know that the new technologies, the cloud and all the automation, enables us to reduce the transaction cost. There's a corollary here, which is if you don't change anything, you can't have smaller batch sizes.

So if you're going on an Agile transformation, a pure Agile transformation, and you're not doing anything else, and you have pretty inefficient processes for releasing, you have manual testing, it's going to be really painful, and it's going to hurt you, and you're going to start increasing your batch sizes again. And I've seen a few organizations who have gone down that path.

You need to change those things. You need to look at your governance process that was great when you had nine-month releases. They need to be different when you are doing monthly releases. And you need to radically change that rather than saying, "Okay, how can I take a day out of it?" You really need to rethink your overall processes. Think about everything that goes into the transaction cost so that you can reduce it over time.

The second thing is governance. Now, again, I'm not going to ask for hands, but just think about it. Who here is running status reports based on PowerPoint or Excel sheets?

All right. So what does it mean? If you have a three-layer organization, you probably have more layers than that. That means someone is asking a developer on what the status is. So that's a team lead, and then that person will provide that status to someone else who condenses it into an executive status that then at some stage gets delivered to the governance board. Right? And if you have more layers, you have more than that.

So that means any information you look at is at least three days old, and everyone has changed it a little bit more green along the way. So you get this kind of green tint on it that makes it look good.

That is nonsense in our world. We've been talking about big data and analytics for so long, and you're creating so much data in your process. All those tickets, all the work items, the user stories, the Jenkins lines, the whatever, right? So much data, and it's nicely isolated in those tools, and no one looks at it.

The one thing that we've done in my team is we created a dashboard, and this is Splunk, but any technology will do, where we take all that data and aggregate it and have real-time live data about our systems and our projects.

One thing that you see there on the left, which is a little bit too small, so let's blow that up a bit. If you're working in an organization that does a lot of Agile, this is my Agile program view.

Yes, all Agile teams have different amount of iterations or sprints, and they're different sizes, and they apply different value. But if you normalize the time and the scope, you can plot them on this line, and then you can see when projects are falling behind.

If you are through 60% of your sprints and you have delivered 10% of the scope that's working, tested in an environment, not just code, then you know you have a problem. And you can quite easily identify the projects here that we need to look into. And the size of the bubble means the overall amount of money we're investing in that program.

Now, that is live data. You can then double-click, and you get into Jira or VersionOne or whatever. And again, there's all these different tools out there. As an executive, it's pretty hard to look at them because you have to look at five different ones. So here we have aggregated all that into one view.

And we should absolutely be able to do this. We're doing it for the business. Why don't we do it for IT? And again, it's that mindset shift and the speed that we need to look at these things that help us making a difference.

They have another pretty cool one, and you can't really see that well. But at the bottom, the bluish lines, they actually show you a release score, which is a number of factors that go into there from zero to 100 based on late changes, known defects, how many builds have failed, and then what the impact on production was when we went live.

So you can actually now look at it and say, "Okay, if we had a huge impact in production, what happened before and what was our release score?" And then you can tweak that. And there's no magic bullet. I can't give you the formula that will work for your organization, but you're meant to be learning. You're meant to be looking at this and tweaking it.

So then we look at the people dimension, and this is probably a pretty standard view of how you go through this. And the idea here is really you need to provide the right context for your people.

As I said before, as a developer, when all I see is a technical design, all I can do is translate that thing. I don't know what I'm doing. I'm just implementing it.

One of the phrases that I hate with passion is "build as designed," as if that's an out for people to say, "Okay, we did what we were asked. We didn't have to think. We're done here." And contractually, that's sometimes good for us, I will admit that, but that's not what we're here for.

So we need to provide the context, and that means you need to fly people in, or you need to bring people together upfront to talk about the business challenges you're trying to solve in IT. Don't give them the design or, "Here's a long document to talk about." That doesn't create context.

Context is talking to each other, planning things together, and breaking the scope up into what you need to do. And it's really important that we get away from project managers who assign tasks and chase people up. That's not your job. Your job is to manage what happens across the different teams, the dependencies, the outside.

So let's look at technology. Oh, sorry, let's look at the ecosystem. This is probably the thing that I don't see enough of in the blogosphere, in the community.

So the very first one is evaluating software vendors. I've done many projects where we look at software products and which tools should we use, and we're pretty good at looking at the functionality. But we tend to undervalue the engineering.

So you have functionality as one function. Then you have the architecture. So is the product that I'm using, the SAP, the Oracle, the Salesforce, the Pega, whatever you use, what are their capabilities for auto-scaling, for self-healing, for all the DevOps practices that you're talking about? Because they don't necessarily come out of the box, and you need to ask these questions, and then you need to understand how important is that for me. How would I deal with a failover for those?

And then what are the engineering capabilities? So can you actually get the source code? Can you enable parallel development? Can you do branching and merging on those things? Do they have APIs that allow you to drive the automation?

I had a tool earlier in my career where I needed to use a UI screen scraper to press buttons to do deployments. Why? Because there was no API for this to be enabled. That's nonsense.

So we need to look at those things. We need to look at the modularity and cloud enablement to make sure that you can actually break these things up. And I said before, there's no end state. So that means each of these products will be replaced. So you need to know how you would do that because in three, five, 10 years' time, you will do it. And you don't necessarily want to do that at the Big Bang.

And then we look at this from another lens, and you say, "What actual capabilities do I have?" Because if you don't have a lot of IT capabilities, perhaps the other things don't matter as much. But if you have strong engineering capabilities, then you want these things to be done.

And I think as a community, as an industry, we haven't done enough in this space because we have not asked these questions. If I'm a vendor and I can build more functionality, which you're asking for, or I can build better engineering, which you never ask me for, what do I do? I build more functionality, other than believing in the good of the world and doing the right thing.

But I think as an industry, we need to ask for those things a lot more than we do. And my frustration is that some of the SaaS products, for example, are actually pretty bad in this. You'd be surprised. And I'm not going to name names. I'm on video.

So the next one, and this is my favorite exercise that I run. So let's see whether this works here and whether the animation is correct.

Evaluating people is something that everyone cares about, right? You want to know whether you're getting good value for your money. So the traditional way of doing this is we're looking at a vendor, and we look at what their day rate is. So how much does an engineering day cost me? There's vendor A, $100, vendor B, $80. So I'm going to go with vendor B in absence of any other criteria to look at.

Okay. Now, this is how work looks like in every project of some size. There's more junior people to more experienced people, and the junior people are cheaper, and the more experienced people are more expensive.

Now, the more junior guys are probably going to do things that are a bit more repetitive, things that need less creativity. They might be doing the password resets, they might be doing the deployments that are scripted, so they can run through specific processes. When you automate that, you automate at the bottom.

Now, what does this do? It actually drives up the average day rate.

And I had this conversation with the CIO, and he said, "Yeah, Mirco, I just had one of the vendors that works for me come back to me with a 30% ADR reduction." So average daily rate is ADR, if you don't know that. "And now after this, I'm worried because clearly they're not automating things. They're looking for more junior guys that they can deal with."

And I want to do exactly the opposite. I want to have some higher skilled people that do more automation to drive the overall cost and the total cost of ownership down. And this is really one of those really counterintuitive things where the whole industry is looking at it the wrong way, and we haven't really figured out better ways of doing this.

Now, the next one, you're evaluating your own team. Your operations team. And one of the things that always comes up, again, I'm working on the SI side, so SLAs. How will you know that I'm doing a good job? SLAs are something that I find inherently dangerous.

Now, here is our two SLAs. A password reset, that's a small task, shouldn't take longer than two hours. A production SEV 2, which is annoying, but not super critical, will take me a day.

Now, if you have the password reset and the SEV 2 coming into your team, what will you do when you have these SLAs? You're going to do the password reset because that's on the clock. For the other thing, you have a full day.

And that creates a dynamic where this is just two, right? And normally there's a whole set of these SLAs, and they create a dynamic where you're actually not getting the outcome that you're after.

What you're really after is that over time, each of these things take less time and are more efficiently solved. And SLAs are really, they're kind of putting a number in there. And good organizations are not taking the end-state number. They're looking at learning and getting better.

Here's another example. First-time resolution rate and resolution time. You absolutely want that to be right. So we want to have a high resolution rate, and it should be really quick when it hits your service desk.

Now, you heard perhaps Damon talk about how there's a lot that we automate in operations. And you get this kind of tree here where there's actually very few things and only exceptions that hit your team.

Now again, what does it do? If all the simple stuff is automated, I get the really complex stuff. Now, that drives up the time it will take to solve, and it will drive down the first-time resolution rate. Unless I start measuring all the things that have been automated, which you probably won't create tickets for.

So again, you have this completely counterintuitive dynamic that plays out in organizations that we have created for ourselves. And it basically makes it really hard to happen in these situations where, if I do the right thing, I will start hurting these metrics, and I needed to go back to my client and say, "Hey, we need to change this. The contract you put in place, it doesn't, or basically disincentivizes me to do these things."

So really need to think about this differently.

All right, and then you will have these slides, but then I think there's the other bit where everyone talks about we are now partners, we are now vendors. But really we still treat them as vendors, right? Because when we talk about failing and learning, well, we really don't want that. We pay money for you. You can't fail.

Now, if I can't fail, that means I can't innovate. I can't experiment with something. So how do you solve these problems? And this is kind of my self-assessment for clients to say, are we really working together to actually become a better organization as a partnership?

So I can't give you the silver bullet. I can sell you snake oil in all flavors. But at the end of the day, unfortunately, it looks like this. So we're going through this process where we're going to get better, and then you get to this peak where you believe you've done it. And then from there on, we're going to fall a bit back, but hopefully not as far as we showed in the transformation slide.

And really, it actually doesn't look like this. Even that is not correct. It really looks like this. So we're going to go through this again and again and again and again, and we need to be resilient to the drawbacks and the setbacks that we have.

And we can't just declare success too early or punish people when we're moving backwards. You will have deployment failures. You will have environment outages. It's a matter of learning from it and getting better.

So I am... Oh, you have 20 seconds left. So there are still some problems. I haven't solved these, but I'm a lot better telling you all the things that don't work. But these are the things that I really think we should think about.

So the whole partnership model. How do we find out that this relationship is working for you? It's a bit like in a marriage, right? How do you know that it's working for me and for you? Anyway, I leave that as a thought.

And then last but not least, how do we get into the anti-transformation transformation? And if I can bundle this up and sell it as snake oil, I will be on my island enjoying the hammock.

Thank you, guys. I hope that was something that you enjoy.