How BMC IT Learned to Reinvent vs Just Run the Business
Hear about our digital transformation journey, the challenges, lessons learned and successes. Agile, DevOps and SecOps enabled us to transform an IT culture that valued control & governance over agility, contracts over value, standards over flexibility and safety over failure. By prioritizing efficiency, business requirements and user experience, we now build in automation with business logic and ensure the entire deliverable travels through a holistic build and test process.
Vinod measures and models the current application development, deployment and support processes to determine if the processes meet the organization?s mission and strategies, and if and how well the current IT systems enable and support the processes. As a result, spends a great deal of time working with different PMO, BSA, application teams, infrastructure teams, and operations on how best to adopt DevOps and derive value from current and future technology investments.
Vinod believes that, an agile enterprise is one that reflects agile and lean values and principles. Agile development practices have been so successful at connecting the once isolated developer to their business user. DevOps extends the gains made with meeting business users' needs to operations' needs. The ultimate goal in this role is to provide perfect value to the customer through a perfect value creation process (Lean) that has zero waste.
David Riggan is an accomplished IT Executive with 20+ years of global experience leveraging information technology to rapidly deliver business value. He has experience with a broad range of disciplines including Business Architecture, Portfolio Management, Analytic and Data Strategies, general strategy development, investment strategy, Lean/Agile, offshore development, vendor management and process re-engineering.
David is always asking how to improve "Time to Value" and then continuously executing, measuring and adjusting. David asks, how can we make the whole greater than the sum of the parts? Where is the non-value add work and how can we remove it?
Chapters
Full transcript
The complete talk, organized by section.
Vinod Cheriyan
So today we'll talk about our digital transformation journey at BMC IT. So if you know BMC, we are also a software company, but at the same time, I work in the IT department. So we'll be talking more about how did we transform our IT organization, and how did we get eight times faster time to value? How did we save about 50 million US dollars in savings, and while keeping the headcount fairly flat, right?
Get this project up. So six years ago, we were an organization that was stuck in the 1990s. You know what that really means? We had business requirement document that formed part of a contract. How many of you are still lawyers? Let me ask this question very specifically. How many of you are still lawyers writing up your business requirements document?
I don't see any hands. Actually, only one, which means this section of it will go fairly fast. So let me fast-forward. So we were in a waterfall development model. We had very little automation. So if at all any automation, it goes more along the lines of this. We have a DBA. He does wonders in his automation. We can refresh databases, we can refresh application servers. Guess what? The applications team still need to submit a ticket to the
DBA to use the automation, right? That's the real world. So there was automation, but in effect, there was very little automation. We had tools for everything. So we had sales application ADA team using Excel spreadsheets. We then got JIRA, Rally, a whole bunch of tools just to manage the requirement pipeline. So we had tools for everything. But in effect, really not any useful.
It is also quality was someone else's job. So here is how it goes. We develop an application. QA completes his QA effort, goes into production, finds a bug. Now who is at fault? The dev will say it's the QA because you didn't test enough. The QA will say it is a dev because you introduced the bug in the first place. So that's how it went. We also had this silo mentality.
Remember when I talk about the DBA teams and the apps teams, what do they actually do? It's all my thing. I'm not going to share that with anybody else in the organization. So that was the situation that we were dealing with. Now what happened as a result? We had very long delivery cycles. At best, we could deliver one in every three months. In some cases, it took about a year. We have missed our commitments. What do I really mean by commitment?
So it's what the business is really expecting. So differentiate the commitment. In the waterfall model, our commitment is to deliver what we wrote up in a document. Here, we are talking about did the business really get what he really wanted to get? We also had a hero culture. So now, if I can do things on my own, and we are
rewarded based on the projects that you deliver, then we kind of incorporate the hero culture into our organization. So the hero culture actually led to poor morale within the organization. Somebody who delivers a project successfully, he's just a hero. But if you look at our overall transformation, is that really the culture that we want to adopt? At the end of the day, we had frustrated customers.
So this was just a disaster. So we'll talk about our IT portfolio. So we are a typical IT organization. If you look at it, this is all the tools that we use internally. So we got SaaS applications, actually a whole lot of them with the cloud that you're seeing. We have applications, digitally native containers, Docker, so on and so forth. We also have applications on-prem.
And we also have application that's been built 10 years ago. So what really happens in an organization is for you to deliver a product. We'll talk about what do I mean by product later. For you to deliver a product, you really need to have all the systems talking to each other. You cannot just say, "Hey, look, I'm all containers and on microservices." But if your back office system, or maybe your BI system is
running on an old technology, then your environment cannot move fast enough. Your business value cannot be delivered fast enough. So that was the situation that we were in. Now, we dug one layer down and really looked at what was the real problem that we were seeing. The crux of the problem was we
were spending most of our time trying to run. We spend very little time in reinventing. So what did we do about it? So when do I talk about run and reinvent? Run and reinvent is about maximizing the efficiency of existing operations without impacting the customers, but at the same time you are finding some value somewhere else, time somewhere else, to reinvent your team.
Remember when I talked about flat headcount? That's what I really mean. So how did we really approach it? We developed a multifaceted strategy, and we said-- Remember somebody talked about today, morning, I think it was Jim talking about it. When you get up in the morning, do you have one thing that you're going to do today? So that one thing is what matters. So we established a core mission statement that is all about how do I
maximize the time to value? Then we put a set of strategic themes around that to say, "Here are the things you do to strategically execute your time to value goal. So keep in mind always that simple message of one thing that you are going to do for the day really matters, right? So that's what sinks into people's minds. Now, let's look at each of these. Automation.
If you really look at everything that we do, so you automate, you find more time that you can spend somewhere else. We all know that. Now, what is really different about it? So if you really look at Agile, DevOps, all of these things together, what are we trying to do? We are going to go faster, we are going to improve the quality, which by the way includes security.
Then we want to make sure that the cost is optimized. So we cannot increase the cost. We want to move faster, but at the same time, we cannot compromise quality. Is that even possible? That's what we really need to look at. How do we automate things to increase that speed? Look at self-service capabilities. Remember the example that I talked about? The DBAs and the other teams have their own automation, but it's not really useful, so bring the capabilities to the people
who actually needs it. So in our example, our customer support organization. They get ticket, it needs to be routed some place. So these routing rules changes very often. So what did we actually do with it? So we said, look, in a project or in an initiative, we're going to take about three weeks out. We're going to build an engine that is going to help you build your own rules.
That's how we look at self-service. So now they no longer need to come to IT. It's all going back to the business data. Now you got the flexibility to do what you need to do. One of the other things are, if any of you dealing with your shadow IT situations where the business is developing their own stuff, how do we change that paradigm around? If you're going to tell, you're not going to, hey, buy new software, it's
just not going to work. You enable them a platform where they can develop things on their own. Now, it needs to be low code. It needs to be high-performing. And it should support rapid application development. Only then they are really going to buy onto it. Now, that's what we did with our citizen development. We trained people, we told them how to use the rapid application development platform so they can do the development themselves. So we enabled all bunch of innovation models.
Hackathon, ideation time, exploration time. Make sure this is part of your organization DNA. So enable people and provide people the framework to do innovation. You also look at to see how do we develop people. What is the best way for you to learn? So innovation, all of those things are good, but at the same time, you need to provide people a platform to learn.
So what we did is, we used a training tool that they can go consume on their own. So whether it be Udacity, Udemy, Coursera, doesn't really matter. Give them a tool to learn. And some of these things that you'll see, hey, they are going to learn, but then nothing really comes out of it. And that's okay. But they also provide them an opportunity to come and showcase
what they actually did. So by doing that, you are moving the people up, their skill set up. At the same time, you're moving the work down. How can they learn? If they don't automate their repeated things that they do day to day, they'll not be able to advance further in their career. Also remember technical debt is one of the key things. You have to identify where your problems are in your organization. Whether it be the bad code. Maybe you're not doing security scans part of your CI/CD pipeline.
That was our situation before. We didn't have any security. We will do at the tail end. Guess what? There was no reaction time, so we incorporated security into our CI/CD pipeline. So things like that. Look at to see where is your technical debt really. Look at the broader umbrella of technical debt. Now, how did we really address this? Problems are great, but what major steps did we take? We reorganized all of our IT teams into practice
areas. So when we talk about product, we are talking about a product that offers a sales capability, customer support capability. If you keep going down further, you will be too much into the weeds. So you need to look at what is your product that you are going to deliver. Remember the project versus product conversations. I'll not be able to speak to that within the limited time. But look at how do you organize your teams into the product-focused areas. Create enablement function. There are going to be some things that
the people are not going to go do on their own sometimes. So you want to have some enablement functions in your organization that will help you to do the always looking for the best in the organization to further advance it. So what we did is for Agile, DevOps, information management, and data science, we create a horizontal. They always stay ahead of time, looking at what is next.
Then they go out to the teams, whether you call it Agile coaches, scrum masters, embedded into the team, doesn't matter, but your fundamental is provide that horizontal platform. Our earlier horizontal platforms are used to be what? A database, a server. Change that nomenclature. Change that things around and say, "What matters to me in the organization?" Also, look at how are you going to do change and release management. Look, if you're going to provide all the tools for people to learn, do you think they are just going to learn by themselves, especially your senior resources?
No, maybe not. So tie some of the incentive backs to their learning. So what we did is we provided them a platform and said you can go learn on your own. But at the same time, you need to meet at least 80 hours a year in terms of your learning objectives. So think about some of these things. That's what we did. That doesn't necessarily mean that's an approach that's applicable to your organization. But at the end of the day, providing them a platform to learn and make sure that they are learning through that platform.
We align incentive to our desired outcome. We had just talked about what our desired outcomes are, right? The strategic things. So earlier, our incentives were given to the people who are delivering projects. We changed that around. We said, "No longer you are getting any incentive if you are not doing one of those strategic initiatives." That was one of the game changers.
You also look at nurturing champions. How do you nurture champions within the organization? So you want to have people with skill, but they also need to be training other people. So giving some incentives to them. One of the lessons learned for us, we thought, you know what? To make some of these changes, our best partners are going to be the most experienced people, the architects, the managers. But guess what? It turned out to be that some of the newer, junior folks
actually got onto the bandwagon, and they are driving the change. But provide them an opportunity. Now, what do I really mean by opportunity? One of our interns were presenting at our CEO SLT last week. That's an opportunity. So it's not only that culture that you create. Provide an opportunity, whether it be wall of fame, whether it be presenting at a conference. Pay for them to go present at a conference.
Let them submit call for papers. Somebody talked about the stereotypes and the opportunities today morning. But how do you really do that? Look at what are they delivering, and then based on the value that they deliver, end of the day, to the business, not the requirement that they deliver anymore. The value that they deliver to the business, give them an opportunity to go talk about it. Small wins and market your successes. It's very key. Anybody here in IT who is not doing
marketing? Not doing marketing. That's good. All of you are doing marketing on your own. That's great. So make sure you deck your cards. What are we really doing as an organization? Market those values to your rest of the team. Now, also make failure is cool. That doesn't mean we are asking people to just go fail. We are saying, look, if you try to innovate, you fail, that's cool.
And that's very hard for a company that's been in the industry for 30 years with all the historical things in place. Now, let's look at what do I really mean by run and reinvent. Your security team comes to you and say, "Look, I got a critical security issue identified that's going to be impacting 50 of your applications between sales and customer support and marketing." Now, what are you going to say?
Your business comes to you and say, "Look, I need this product function," that is, again, across multiple products, "that needs to be delivered right away." What are you going to say? We are building a digitally native application, and let's wait for a year, and it'll all be cool. It's not going to work. So you need to look at how do we really do that. So here is one of the examples that we did. Remember today morning, somebody talked about a data center with
open, I think it was Computer, with the place cards? That's exactly-- When I was looking at the numbers, I was going, this is exactly the numbers that we go to. 2014, we had 36 data centers, we had 64,000 square feet space, and we were spending about $7 million. Today, we have four data centers, 7,000 square feet, and we are only spending about $2 million in our expenses. So these are the ways to look at how do we reinvent.
Now, it's not only that. We had all of that space and the data center, but when the dev asked for, "Hey, can you give me an environment refresh like in production?" The infrastructure guys will go, "All right. Talk to me in a month." That didn't work. So what did we do? Today, they have a self-service capability. They push a button, they get an environment in about 15 to 30 minutes.
The next thing is how do you-- Now the infrastructure is all done. How do you really go about getting your pipelines streamlined? Once your dev is done, you should be going through the pipeline fairly fast. If you're not doing continuous deployment, you can still go faster to at least get to a state where a UAT is being performed. At least get to that stage of continuous delivery.
Now, part of the continuous delivery, incorporate things that is really needed in your organization. You'll find that some of you will need a security scan or maybe unit test or regression test. Whether it be your inverted pyramid in regression test or you're starting UI going up or the vice versa, you can still do this irrespective of which level that you are in. So as soon as one of these things fails, make sure that you proceed, you
get that feedback immediately back to the customer-- immediately back to the developer. We also look at optimizing our business flow. So let me talk about an example. To get our BI environment refreshed, we got a 10-step process to go. Pull from here, pull from here, get to the next step. Now, guess what? This process usually takes about 15 minutes, but sometimes takes two hours. So my next process is only going to start after two hours because this sometimes take
two hours. So why do you do that? Optimize your flow. You say, as soon as this step finishes, I know that step successfully completed, proceed to your next step. Whether it be in your CI/CD, whether it be optimizing your business flow, you want to get your Tableau report out to the business users, even there. So look at how do you optimize your business flow. Shift things left: security, your quality. And optimize your business flow.
Now, we did this all on the foundations of innovation, automation, and self-service. Now let's look at one of the key things that we learned in our experiences. We need tools. Tools that can do things that's more enterprise-wide. So we looked at the capability in terms of what do I really need? Whether it be the CI/CD or a business flow, I need the
ability to look at a predecessor and a successor. What is going to be next? We look at how do you optimize that flow. In some cases, you deal with your legacy application, you may want to just kill and restart a service. The tool should have the ability to do that. Sometimes you need to orchestrate your microservices on containers. Sometimes you have to do API calling into different applications. Sometimes you have to make HDFS
commands to execute on a command line. Sometimes you have to just go self-service. And mobile-enabled. So what if your application developer forgot to push the code and say, "Hey, I want to get feedback when I come in tomorrow." He's at a restaurant with his family. He can still push the code in through the self-service. So these are capabilities that you need to enable. And also make sure that you provide the team an ability to innovate
faster, and specifically around the automation. So when they develop the code, construct your automation at the same time. We call it job as code. So that allows you to build your automation at the same time as developing instead of lets all the development gets completed. Okay, now let me automate. Let's go back and automate at the same time that they start developing. And make sure that you have tools that's reliable.
We have used tools, we'll schedule the job, guess what? It didn't run. Now the next process is already kicked off, and we just lost million dollar. That doesn't make sense. Make sure that you select something that is sustainable and that you can run. Let's look at an example because we got only about 30 minutes, so I'll not be able to talk through hundreds of examples that we have. What did we do with our Salesforce platform?
On our Salesforce platform, guess what? You need developers and environment to develop from. Anybody using Salesforce platform today? You got a couple. So if you have to look for a full service sandbox, depending on your environment and complexity, sometimes it's $200,000, $300,000. Now I need 20 of them. You do the math. Now, if you get a sandbox with only the things that you need, then you are much more agile in terms of things that you
can deliver. Instead of sticking to three or four full service boxes, look for a sandbox and then push things in that's relevant to you. It's not only that, when really things happen in production on the right-hand side. You want to know what actually happened all the way. Your SOQL query is exceeding 100 per transaction. Your Visualforce page limit exceeding 170 MB. How do you really know? I am in the accelerator mode of development. So when it really gets to production, that's when I get to know that's what
happened. Now, what happens without something in place? Everybody's scrambling trying to figure out where is the real problem. We don't know. So what we did is we used Salesforce API to extract the log, use Control-M to extract the log from Salesforce. We load it into our TrueSight tools, which is doing analytics, which will then look for patterns and trends. For example, the 170 MB page limit. It will look for the patterns and trends.
When it identifies something, it will create an incident and get assigned to the developer actually worked on it. So that makes it a full loop. So when these things happen, the blame game actually goes away a whole lot. So it's no longer the business reporting, no longer the QA reporting, no longer the manager is on the issue. The person actually knows what to actually work on, even if it happens in production. Now that doesn't mean that we slip things into
production. Let's look at another example. See all these different boxes here? Do you think an army of people coming in is really going to help you? Not really. So what is the solution around this? All these boxes here, all these boxes with people. Look at how do we deliver value in these type of situations.
Take one issue at a time. Lot of times that you will see that you have automations already in place, but it is not connected together. So use some type of a workflow orchestration tool to connect all these different pieces together. For example, we had a service to start the application. We had a service to start the database. We have a service to push the data in. Now, the issue was these were all disconnected pieces. All we did, we put all these things together and orchestrated and
ran it reliably so that when we no longer need these people in between, is the people actually who really need to do their job through a self-service, they're able to do that. That's how you increase your agility in the organization. Avoid those white spaces. The tickets going back and forth, or emails, or hundreds of emails. Avoid those white spaces. So in effect, what did we actually do? So these are the
key elements that led to our faster time to value. So if you look at it, the faster time to value, we improved security, we improved quality and increased knowledge. We use some type of a measurement dashboard to tell people what your team is actually doing. Remember when I talked about how many of you are marketers? If all of you are marketers, this is what you'll be doing, is create a dashboard. Tell people what type of efficiencies that you are getting, and
how are you improving over a period of time. So we didn't get from 2014 to today with eight times faster. We started with one time faster. We went to two times faster. So if you look at some of these things in terms of how did we actually get time to work on it? If you look at this specific example, 80% of our repetitive tickets are resolved automatically. So you know what? Some of our repetitive tickets sometimes our CyberSource, which is
our payment gateway, goes unresponsive. You look at your typical monitoring stuff, you will see that the gateway is still up and running, but it is not processing transactions. So what did we do? We used Control-M to inject transactions into it to see if it's actually working. If it's not, then create an incident. That incident is then worked by somebody else, and that was our first step. The next step that we took is now we know fairly confidently this
is working in terms of the process. We said, "You know what? Let's automate that process." Why do we need this person to be in the middle to be looking for it, right? So we automated those processes. Now, let's look at a couple of things in terms of the lessons learned. What was our journey and the lessons learned in our journey? Culture is crucial, but it is not a prescription. You all know that. Now, also keep the system thinking in mind.
Part of your training, you want to let people learn from the other systems. So for example, if a change is happening in our Salesforce environment, and there is something happening in the back office in the Oracle environment, and then something happening on the front office in the Java application, you need to know how all these things are connected together. So you train people through your different training methods and make sure that the people really know how the systems are really working.
Now, that's one way to address this. So there is technical ways to address this, too. API management, for example. So you don't have to be knowing all the dependencies. All you need to know is where is my gateway, where is my API? I pass this, what am I going to get back? So there is different ways to approach it, but in any case, you want to develop that understanding of the system thinking. Automation is a design principle, and it is not really a
tool. So remember the example that I talked about when customer support agent was trying to route? We could have just automated that, but what did we just do? We just automated a process that was not really useful. That's not the real problem solver. So incorporate your automation and into your design patterns. Don't automate copy processes. So in the same example, if you just automate 100 different ways to create your routing rules, then you are basically not
really effectively impacting the user community. Communicate the intent and don't dictate how. One of the key things. Remember our mission statement? It said, "Here is what you are coming to work to do today." That's the level that you want to communicate, actually. They know what they need to do after that. So don't dictate how. Don't get into the weeds of, hey, now do the SQL statement and do this Java application. Keep your managers away from it.
Let the team do what is right to get the value delivered. What is in it for me? It's very important. So remember the waterfall model where people would write document? Do you know why did they do that? Because they could avoid the blame game. And they could say, "Not me." But what are we changing it to now to say, "Look, here is your incentive. If I don't see the value, I'm not going to do it." So how do you market your successes and make sure that there is something
in it for each individual that is working on it? Get your executive-level commitment. In fact, David, who was our VP, was planning to be here today, but he could not make it. But he's also SAFe-certified, and he also teaches SAFe classes. Now, whether you like SAFe or not, different ballgame, but that's the type of commitment you want from an executive.
An executive teaching people how to do these things. That type of commitment is what you need in your organization. Be prepared to fail and accept it. We kind of talked about it. So keep in mind, it is not a sprint, it is a marathon. For us to get here, it took six years. From here to the next step, that'll probably going to take several years also.
Reinvent is going to get finished by a certain timeframe. Make sure that your executives buy into that. The first thing that when I see these values, I was going back and talking to our CEO. CEO didn't tell me, "Okay, now where are you giving me my money back? Can I cut your headcount?" That's not how you want to approach it. That's why executive support is important. The question should be, where are you reinvesting now?
How are you getting to the next level? So change that paradigm around in your organization. With that, thank you. I know we didn't get a chance to do the Q&A. Hopefully, this was helpful. I'll be at the BMC booth if you need to find me, and I'll also be at the speaker sessions if you have detailed questions. And also, one other thing, I know if you have a card with you and scratch off that card, there is a whole bunch of gifts waiting for you.
So thanks to our marketing team getting that arranged. And thank you all for coming today. Really appreciate it.