Log in to watch

Log in or create a free account to watch this video.

Log in
London 2017
Share
Download slides

DevOps Transformation, Driving Culture Change

DevOps Transformation, Driving Culture Change

Chapters

Full transcript

The complete talk, organized by section.

Ruly Weisbach

Hi, everyone. My name is Ruly Weisbach, and I am the Director of R&D for ALM/Quality Center, PPM, and ALM Octane.

I'm mainly going to talk about the challenges in driving a culture change within an organization that used to do things successfully, by the way, for so long, and when it needs to start doing things differently because the market has changed, the technologies changed. It's many times hard for people to accept that there might be new ways of doing things. So I'm going to tell this story.

A little bit about myself. I have been an R&D manager for almost 15 years, and I've been around. I managed teams in small startups and large enterprise organizations. So I kind of had the chance to build my philosophy in the way that I think that things should get done, especially around Agile.

About two years ago, I got the role of managing the ALM R&D. When I joined the organization, this is kind of how I saw the organization. We were kind of the heavy-lifting world champion of the world because we kind of hold on our shoulders this huge product, ALM/Quality Center, that serves more than 8,000 customers. Most of them are large enterprises. I hope many of them are sitting here in this room. And in each, they have thousands of users working with the product for their day-to-day, mission-critical tasks.

ALM/QC is a product that was built for more than 20 years, and every year we continuously grow the product, adding more capabilities, more functionality. So it's a huge product serving huge organizations.

The average release time in ALM is about a year and a half. In each release, we used to introduce big features that required changes across most of the layers and components in the product, and also required long regression test cycles to make sure that we are bringing the best product possible.

We have about 300 engineers working on ALM. Most of them started their career with ALM, started to work with us and continued, and many of them are with us for more than 10 years. So talking about people who love the product, this is what they know how to do best, and they have been doing it for quite some time.

And as such a product, the ALM architecture is a monolith one. This is a big chunk of code that has technical layers. The foundation for the product and the org structure of the group is based on technical layers. We have the infrastructure layer, and on top of that, the platform, business logic, UI. And throughout our year-and-a-half release, we usually tend to do things in, let's first build the platform, then add the business logic, then build the UI, right? Waterfall. This is how we used to do things.

Right after I joined the organization, we took a business decision to introduce ALM Octane. ALM Octane is an addition or extension to our ALM brand, ALM offering, that is targeting to provide a solution for organizations who are moving to DevOps, moving to Agile, and need a solution more suitable to manage this type of project.

We had a very strict timeline because it was a new concept, and the business required us to provide an MVP, to provide an alpha within six months. Okay? So that's a third of the time that we used to do a release to bring something new to the table. And since it was a new concept, we decided that the best way, the best approach to introduce it to the market, is as a SaaS offering first, to provide it as a service.

Of course, we decided that we are going to use new architecture, new technologies, modern technology when building ALM Octane. And we also agreed that once we introduce the MVP, we want to show incremental improvement for early adopters, so we'll have push to production every two weeks. And the only way to approach this type of project, unlike the way we used to do things in the past, is to move to Agile, doing things the Agile way.

I know there are a lot of buzzwords. A lot of buzzwords about Agile, Lean, DevOps, many different methodologies and ways of doing things. But for me, when you talk about Agile, this is the most important thing: the ability of an organization to deliver the most important thing for our customer first.

First means let's build what's most important before starting to work on additional functionality, additional stuff, to be focused on the most important things. This is Agile. For me, this is priorities.

And we had a very clear goal for the MVP. We knew exactly what we wanted to deliver, what we wanted to build in those six months. And from my experience, the best way to build an organization that will do things agile and will be productive is to go with vertical teams, vertical groups, each responsible for a specific module or business domain that we want to bring. Those are kind of the names. It's not important.

So I had a meeting with my staff, with my management team, and I explained to them that I want to change the organization from being like this, to stand it up and be like this. And these are the benefits that I tried to explain to them.

I told them that, especially for them, but also for the team, it will drive much more sense of ownership and accountability, because they will not just be responsible for doing a technical component, but they will be actually responsible for building business functionality, business value.

I also thought that it would help us move away from the monolith approach to more of a service-oriented architecture, because each team, each group, will need to build top to bottom the thing that they need to achieve their goals.

And I also told them that it will help us release many of the barriers and the processes that we have when you build something from bottom to top, layer by layer, because the individual, the developer, will actually be able to pick something from the backlog and build it end to end.

And most importantly for me, I explained to them that this structure will drive a culture change. First of all them, but also their people, their team, will be more focused on what is the business value that they need to bring at the end of the sprint and the end of the release, rather than focusing, "Okay, I need to build that access layer, or I need to build this grid or that component." And having a vertical approach will help us to achieve exactly that.

Actually, honestly, to my surprise, my managers refused. They were unwilling to accept this change, and they simply refused.

And to be honest, they had very good reasons. Okay? They had very good reasons. I remember the guy managing the UI told me, "Ruly, you are crazy. This is totally not efficient. You are asking me to take my team, people who used to write just UI for many years, and now you expect them to think about the functional spec, and you also expect them to write the back end and the platform and the business logic to achieve functionality. It's not efficient. If I have someone who knows just how to write UI or a team that specializes in UI, it will not be efficient to ask them to do other things. We will never meet the MVP. We will never meet our goal."

And they also told me, this came from the platform guy, "Look, I have people that are doing their entire career just one thing." I remember he said they have someone who all he does is write upgrade. For 10 years, the guy only writes components for upgrade. And now he loves to do that so much, I need to ask him to do something different. That doesn't make sense.

And of course, they told me that they are working with each other, my management team, working with each other forever. They know each other. They are good friends. And they told me, "Ruly, you don't need to worry. It doesn't matter if we are like this or like that or like this. We, as managers, will make it seamless for you. Trust us. Give us a chance. We'll make it seamless for you."

And I tried and I tried to convince them, and eventually they told me, "Look, this is too much of a change for our organization. If you are going to do that, you are taking a big risk that people will not like it and will decide to leave. Our engineers don't want to work this way. Give us a chance. We'll make it work."

So they are great friends of mine, and they are great managers, and they resisted so much. And something I learned: you cannot push people to the limit. So I decided to give it a chance and to try to do things their way.

But not surprisingly, very soon they discovered that trying to practice and to do Agile but maintaining the waterfall org structure is not such big fun. It's not such a pleasure as they thought it would be.

The first thing that happened, I remember I talked with my product manager, who asked me, "Ruly, I'm a little bit confused. Are we building a new product called Octane, or are we building a generic, enormous library of generic components of every type and kind?" Because this is how the backlog looked like. We had a feature called Grid and a feature called Filter and a feature called Data Access Layer. And there was no visibility of what is the functional value that we are trying to bring, not in the backlog and not in the minds of the teams.

And of course, what started to happen when we started to develop, so the platform guy built one thing. The business logic or the right guy built the equivalent component, and we tried to move to Agile, so they tried to do it in one sprint. But at the end of the sprint, it didn't connect. Not the same parameters, not the same understanding.

Then we needed to spend another sprint just to make the different people from the different groups make the code work with one another. And once we managed to connect the UI guy with the business logic guy, then the platform guy said, "Oh, guys, I thought you were doing something totally different." And another sprint just for him.

And of course, another problem: for each, even the tiniest functionality, I remember about filtering on a grid, it was like a two-hour meeting with about 12 people because we had the infrastructure guy and the business logic guy and the UI manager, and each of them came with their functional architect and system architect, and they argued for about two hours. And it's such a tiny feature.

And then when we realized all of those problems, we said, "Okay, we need to find a way to fix it." And we reached a situation that we are spending most of our time not to discuss how to build the best software, how to build better software, what is the right architecture or code, but how we are going to manage ourselves in doing Agile while having an org structure that is based on horizontal layer.

So we wasted most of our time discussing the process rather than the software itself.

But still, they told me, "Ruly, we are just getting started. It's a new project. Give us some more time. Give us a chance. We'll make it work." And really, they are great guys. I said, "Okay. Let's give it some more time."

About two months after we started this project, I went to visit our site in Shanghai. We have a wonderful site in Shanghai. And I met, I did a roundtable with a couple of the teams that were part of the Octane project, and I was sure it's going to be the best roundtable ever. They're going to thank me. "Thank you, Ruly, for bringing us to this new cutting-edge project. No customer. We don't need to worry about defects or escalation. We can write code like crazy. We are learning new stuff. We are doing things Agile." I'm sure it's going to be that kind of a talk.

But the first thing they told me when I entered the room is, "Ruly, dear Ruly, please, can you please take us off this project? We don't care. We will fix defects all day. We'll handle escalation. We're even willing to talk directly with customers. We just don't want to be working on ALM Octane."

And I told them, "Guys, what happened? I was sure this is a dream come true for every engineer." And then they started to lay out how they see things.

First they told me, "We are feeling like we are part of an assembly line. We don't understand what we are trying to achieve. We are getting requests: build that component, build this piece, do something here or something there. And we don't understand why. What is the purpose? What is the value that you are trying to bring?"

They also told me that, "No, it's not such fun. We are not coding new technologies all day. We are sitting in meetings all day with people from other sites trying to understand what the hell is going on, and we have no clue why we need to talk so much about everything that we are trying to build."

Some of them were very disappointed, because I talked with one of the guys in the platform group, and he told me, "I spent so much time and effort building what I was sure is the right component. And then comes the guy from the UI and told me, 'No, you know what? You got it all wrong. I don't need it. I'm not going to use that. I'm going to write something else.'"

So they were very disappointed. They invested so much effort and nobody used their code. And they were very honest. They told me, "Ruly, I don't know from where you came, but you are probably the worst manager we ever had. We have never been in this type of project. Such a chaos. We don't understand what's going on. We have to change things."

And now it's me. It is happening for real now. They are really going to change me. No, the slides are back on.

It was shocking news for me, okay? And after I met with those teams, I went and visited our other site and met with all of the engineers, did roundtables with all of the teams, because I wanted to know, maybe it's just those guys. They are not happy. I don't know what's wrong with them.

But no, I got the same response from all the engineers, all the developers, testers that are part of this project.

And then I had an all meeting with my management team, and I told them, "Guys, this is enough. It's bad enough that we are having so many problems in advancing this project, and two months have passed. We have just four months to reach an MVP. We are about to lose our entire group because people are not happy. And the main reason you gave me is because of the engineers, that the engineers will not accept the change. But guess what? I talked with your engineers, and they're telling me exactly the opposite. All they're asking for is to give them an end-to-end thing that they will be able to build successfully and be responsible for, rather than building a part of an assembly line, feeling that what they are doing doesn't matter."

And I told them, "Guys, we are doing the change." And we did the change.

We moved to a vertical org structure, and we started to see improvement. Things started to go as they should, and we were back on track. But still, and it's always a surprise, even the simplest metric, like delivering user stories, delivering features, started to improve. Still, some of my managers were sure that this is the wrong way to go. Okay? They still were 100% confident that I'm making a huge mistake.

And now I saw that, in a way, they didn't feel comfortable really telling their team that we are doing this change. So instead of now building an end-to-end functionality as part of their domain as they were supposed to, they continued to build those generic components that no one used, just to keep face with their team.

They constantly complained, in the hallways, in meetings, that nothing good will come out of this change. And they didn't really try to drive the culture change. They didn't really give it a chance.

And it felt that they are going to their team and telling them, "We need to do this change, but it's not because I believe in it. It's because Ruly, he's the boss, he's the bad guy, and he wants to do this change. But dear developer, I'm with you. I have no choice. We have to do it because Ruly is saying so, not because we believe that this is a better way."

And every time, and we are doing software, things go bad, like every day for 50 times. Every time something didn't work as it's supposed to, they said, "Oh, Ruly, what do you expect? We cannot work in this vertical way. It's simply not a way of doing things."

And eventually, I realized that you cannot do such a major change, taking this guy to this guy, from heavy lifting to sprinter, without doing a deep change in the organization.

Eventually, I needed to do a real org change, departure from some of my leadership team and replace with people who are more willing to accept this approach. I also moved from a structure where I had four groups, based on the layers that we had before, to two groups. Each of them is much larger, but more important, the teams within those groups are bigger teams that actually can pick and do things end to end.

So we changed structure and the volume of the team to be much better suited to do things the Agile way, in an end-to-end way.

And today we have more architects, and the architects are the ones responsible to make sure that those shared components that still exist, we are doing things like this, but there are still components like data access layer or UI infrastructure that serve different teams. So the architects are the ones who are responsible to make sure that we will not make a mess out of those shared components.

Every structure, every organization, has its problems, has its issues. But eventually, when I look at the result of the change and what happened, and as I said, productivity, delivering the most important thing first is the most important. I think the numbers speak for themselves. Okay?

This is kind of the before, what we were able to accomplish in a release before, and this is after. And you can see that the overall workload is pretty much the same. We kind of did a little bit more before. But the number of the features that we managed to reach to done, to actually deliver functionality to our customers in each release, is almost double. Okay? And that's productivity.

And afterward, when I thought, because my managers were really there, great managers, and I thought, what did they see that it was so hard for them to accept the change? I think that for many years, they were grown up to believe that the most important thing, and it's true when you practice waterfall, the most important thing is efficiency. How many lines of code does my team or group write? How many story points? Okay? How much test? How much test coverage? How can I be more efficient, and in this year and a half or two years that I have, I can do more?

And I believe that when you are moving to Agile and when you're moving to DevOps, it's all about productivity. It's not about the volume. It's about how much new functionality, important critical functionality, I can manage and I can deliver within a release, whether this release is two months, one month, two weeks, or I'm pushing like every day. Okay?

And I think it was hard for them to see that productivity is the key, and not necessarily the efficiency of the team.

Of course, as part of this change, we didn't just boost productivity. Team motivation is at its peak. And today we have engineers that fully understand what they are building, and they can go and talk to customers and be accountable for what they are doing.

And from all the reasons, this change did us only good. And this is my story.

And now I'm proud to introduce Tomer, who is going to tell a similar story, but in the aspect of making a culture change when introducing security.

Thank you very much.

Unknown Speaker

Thank you very much, Ruly.

Ruly Weisbach

Thank you.