Ramona Jackson

Thank you for joining us. Aji and I are very excited to share, for a few minutes, the transformation that Cisco IT has been going through for the past 15 months. We've had the privilege of working with a great group of people to lead this transformation at Cisco.

We're going to share a few things with you today. I'm going to spend a few minutes just talking about what continuous delivery means for Cisco IT and Cisco. Aji's going to then take you through what our approach was and how we implemented this transformation. I'll then take you through some metrics, show you what our metrics framework was about, how we measured our success, and what those metrics look like today, which we feel is a very powerful story. And then Aji will close with some takeaways and a couple of things that we could certainly use some help on from all of you.

Continuous delivery at Cisco, for us, was about doing three things. It was about speeding up the way we were delivering, or how we were delivering, business capabilities to our clients. Secondly, it was about improving our quality. I think that's something that all of us want to do. And finally, about optimizing our cost. Anytime you have a major program in IT, it's all about driving down cost.

From a time-to-capability perspective, we looked at a few best practices in the agile industry to come up with our approach. It was really about creating smaller working teams, making sure they were embedded, dedicated to efforts, and making sure our stakeholders were actively engaged. That was a big deal for us. We were a typical organization where you bring your business in up front, they give you a lot of information, they go away, and they come back to validate you've delivered what they asked you to do.

From a quality perspective, this was really about automation and automating the way we were doing our testing, and you'll see some of that when Aji takes you through the toolchain that we have adopted at Cisco, as well as embedding our testing SMEs back into our development teams. At Cisco, we were organized with application development, QA, and release as separate organizations, and we've done some major changes to make sure we have that embedded situation work for us.

And finally, cost of delivery. Again, this has been driven by dedicated teams and providing tools to our development delivery teams that would automate and, in many cases, standardize and optimize the way they were delivering software.

From a change-in-the-way-we-work perspective... I just noticed our slides didn't format well when we sent them over from the Mac to the PC, so apologize for that. They looked a little bit better than this.

From a change-the-way-we-do-work perspective, this was the biggest effort for us. The technology was simple to implement, simple to drive adoption, because you can see real value up front. You can see value day one. But changing the way we work together as an organization, changing the process of how we deliver software to our clients, that was really about a mindset change.

Obviously, it's not easy. We wouldn't be having conferences like this if it was, right? And for the past 15 months, this program we've been going through hasn't been easy. There have been a lot of challenges. But this is the agile methodology that we chose to use to deliver the software and to change the way we do business.

So now I'm going to hand it over to Aji. He's going to talk more on the approach that we took.

Aji Rajappan

Thank you, Ramona. This is not working, in fact, the timer. So we still have 30 minutes?

Ramona Jackson

Yes.

Aji Rajappan

All right.

Continuous delivery is a way of life, or becoming a way of life at Cisco. We try to adopt agile and continuous delivery principles in everything that we try to do, irrespective of whether it is resulting in application software or not. It has become our operating model. Now, we used the same approach for running this program as well.

Let's take a look at some of the core components of the program execution and our approach.

In terms of the core team, we constructed the core team in such a way that we have a program director, chief architect, some of the components, PMO, and the process team, and the tools team, and adoption team, and sustainability and operations team. Each has its own charter, responsibilities, and accountabilities.

Then we looked at what is our target population. There were 80 services, 600-plus applications. It is impossible to do everything at once. So we adopted an approach where we have a pioneer team and wave one, wave two, wave three. Wave one: some of the teams were experimenting at a department level, some of the agile concepts, some of the tools and technologies. So they were very excited to come on board and start showing some of the results. And they became ambassadors and champions and reference points for other teams to follow. Wave two was the business-critical teams, and wave three, all others.

Next is end-to-end tools landscape. I know we get very excited when we say that technology has nothing to do with continuous delivery, or technology has nothing to do with DevOps. To me, it's like saying automobiles have nothing to do with transportation. If you don't have the right tools, your dreams will remain as dreams. You need them to go faster. You need them to achieve certain things, right?

So from a portfolio and business outcome perspective, we have the investment tools, priorities, and three tools that I want to call out here are Jama, iRise, and Rally. Jama is used for complex requirements management processes, iRise is used for user experience design, and we have Rally, which is connecting this to the execution model.

With me? Okay.

So when it comes to the execution part, it's very easy to say that everybody follow the same toolset. But that's going to be a constraint unless you adapt and provide the right tool for the right type of application. Six hundred applications. Some of them are web development, some of them are database-driven.

From a web development perspective, we have a set of tools, the common standard tools that everyone uses: Jenkins, Selenium, Git, Stash, et cetera. We have IBM's UDeploy for code deployment. We have Delphix for virtualization.

When it comes to the ERP area or the database side, we have some of the tools provided by Oracle. That is the Oracle Application Testing Suite, Flow Builder. We have Delphix again there for virtualization activities. Delphix is a core component of our continuous delivery activity.

We also have a locally developed tool called AppDB for Oracle database code versioning and deployment. We couldn't find a tool in the market that would satisfy our needs, so we created our own. Something that took two hours to deploy, with this tool, we can deploy it in less than 10 minutes. There you go. The technology makes a difference. Even if we have the mindset, everyone is together, passionate about going fast, if you don't have the technology to do it, it's not going to happen.

Okay, next.

Let's look at the conceptual model of release and environment. What is the path to production? Again, we would love to have everybody follow the same path, but that's not practical. So we devised three lanes.

One is a medium-frequency lane, where some of the projects would take that much time to go to production. Even if they want to go faster, the nature of the project is such that it takes that much time. So that once in six months, if you want to deploy, you have a medium-frequency lane. Large-scope items.

High-frequency lane is if you want to deploy every day, every week, every month, that's the lane that you take. You also have an emergency lane. Bug fixes, patches. You have a security vulnerability detected today, you want to apply something, that is the path to take.

Now, it's very easy to say that one application takes only one lane, but that is not reality. We have the same application taking all three lanes. But where do they converge? They converge at the stage before going into production. Conceptually, it looks very easy, very simple, right? But we implemented it, it ended up like this. So it is much more complex. That's the reality between the concepts and the reality, how it is implemented.

With that, I invite Ramona to talk about what results did we see.

Ramona Jackson

Great. Before I get into the metrics, I want to talk a little bit about why we're measuring what we're measuring.

This program started, well, the formal program started, I mentioned, about 15 months ago. The six months prior to that was the time that we spent on the technology. So we made sure the technology foundation was in place. Then we formed a formal program structure that Aji referenced earlier, in July of last year. And for the past 15 months, we've been driving adoption of that technology and driving adoption of our process. That's what the core team has been focused on.

The program has now wrapped up just three weeks ago, and we have put it in its operational end state. And with that, we've made some major restructuring changes in our IT organization. Our IT organization has about 3,500 full-time employees and probably, we don't like to talk about the numbers, but probably twice that many contractors. So it was a big shift to move that many people in this new methodology, and that's why it's taken us about 15 months, but we are very happy with our progress.

From a metrics framework perspective, you can see the things that were important to us: speed, cost, and quality, which, remember at the beginning, were our three goals. For speed, we're looking at our user stories that we're delivering. For cost, we're looking at the cost to deliver those stories. And for quality, we're measuring our incidents and our downtime, which is something that we've been consistently doing at Cisco for a while.

From an adoption perspective, this was a metric that really was to track our program progress. How quickly were teams adopting the technology? How fast were they adopting the process? How quickly were the people getting educated? It was important for us for employees to feel good about what they were doing, right?

We're a very experienced organization. People have been doing the same things, the same job roles, for many years. We needed to give them some motivation and incent them in a way that they wanted to make this shift, this change. And so we created a certification program around that, around the adoption, to give teams a competitive means of seeing who could get there first. And it's been really effective for us and really driven our adoption metrics up.

And a final metric, which to be very honest with you, we're not measuring yet, is business value. Because having just started doing our regular continuous delivery releases over the summer, we don't have a way yet to see what dollar value that's bringing to our company as a whole. We're working with our business on the algorithm on how we will measure that, from how we used to operate to how we operate today, and we're hoping to start reporting that out on a quarterly basis at the end of the upcoming quarter.

So where we are from a metrics perspective, when the program closed three months ago, this is where we looked. This is how we looked. From a platform or technology adoption, we were over 90%. From a process perspective, almost 80%, and our people adoption, again, is over 80%. So we're very pleased with the progress in that 15-month period that we drove the program. We obviously still have some work to do. And I'm happy that the new leader of the CD team is here watching us present because she has big shoes to fill. I'm joking with her. She knows that.

This is the metric slide that I think is the kicker for us. Waterfall will never go away at Cisco, and we all know that. But the idea here is that the pendulum shifts more toward agile. And you can see where we started a year ago, where we were 60% with our releases, over 60% waterfall, in the 30s for agile. And just two quarters ago, that shifted. This is where that meeting point is, and you can see where we have gotten as our fiscal quarter end is actually this weekend.

So our Q1 ends this weekend, and with our latest releases, we're at 70% agile and 27% waterfall. We expect that to continue to grow. I expect we'll still probably have 10% to 15% waterfall releases on a regular basis at Cisco going forward.

Aji, you want to close us out on some key takeaways?

Aji Rajappan

Sure. Yep.

Ramona Jackson

Awesome. Oh, I'm sorry.

Aji Rajappan

No, I'm going to stop. Yep.

Ramona Jackson

I have one more slide. Apologize for that.

Productivity savings. This is something that was important for us as well to drive. When we looked at the metric, what was that really saying about our organization's productivity and efficiency? To ensure we weren't biased, we actually brought in a third party to go out and talk to our development teams, and to understand how they were delivering with traditional waterfall methods and how they were delivering now with agile methods.

And we looked at each phase of the delivery life cycle to see where we were seeing productivity gains, and these are real numbers. These are real numbers that teams have validated with us, our delivery teams have validated with us, that they're seeing. And with a total savings of, on average, 32% in productivity gains. So the estimate was that teams would get there within two years of adopting the technology and the process end to end, with a starting date of 15 months ago. So we're seeing real savings today, and we're very excited to see that 32% number continue to grow.

Aji.

Aji Rajappan

Thank you.

All right. So we were asked to include a slide where we need help. We have a number of things still unresolved, a number of strategic activities, a number of process areas. But we have a feeling that we can get there. It's only a matter of time. Or even if we have a solution, our group is not ready to implement it.

But today, I want to specifically talk about two operational tactical-level issues that we are dealing with, in the hope that some of you have encountered this, or many of the product exhibitors may have a solution to this.

The first one is about refreshes, right? You saw the release environment model. So every six months, we have to refresh the entire stack, which includes a dev, dev int, and a QA environment. Each one takes around six days to do it. That is an interruption for almost a month. Once in six months, it's interrupted for a month. If there is a solution where we can do it in less than two days, that's going to give us productivity in the exponential rate.

I know it's very trivial, but these things are important. That will determine your speed. I'm not talking about standalone independent small databases. This is a whole stack of 30-plus databases, ranging from 30 terabytes to 40 terabytes, with Oracle applications on that.

Next item that I want to talk about is coexistence of multiple active versions of code, front end and back end, in the same environment. We talked about a high-frequency lane where people can deploy things into production every day, every week, every month. But if you have to do that, you have to time-box them. I hate time-boxing, where you're going to say that you can't come in now because someone else is already there in that, who is going live today or tomorrow.

So how do we break the time box where multiple releases can go in parallel without conflicting the other team? That is something that we are looking at. If you have a solution, we would like to know. This is not just the front end. This includes the back-end database applications. This is not just the web development portion of it. This is not content development. This is transactional processing systems.

Next item.

So this is our takeaway. We want to give you our secret recipe for a successful CD transformational project. Useful if you're planning to start something, useful if you're halfway through it. And the delivery is through a typical agile fashion, something that you're already familiar with: sticky notes. Okay. Stickies is an Apple application, used to be there.

All right, the first one is the platform. Elastic infrastructure. That's the key. Virtualization is the key. Can you provision a full-stack environment for a developer, or is a developer going to ask for bits and pieces and then you try to piece it together?

Next one: tools. Tools are important. Without the right tools, your dreams will always be dreams. You need to have the right tools: ERP, web development, back end. They need to be automated and interconnected. A developer clicks on something, everything should be triggered automatically. It should do the security check, automated test. It should be reaching the destination, whether it is stage or production. You need to have the collaboration tools, WebEx and other chat.

It's not enough to provide platform and tools. You need to support them. So we have P1 support in our non-prod environment. Once you build everything and everything is interconnected, something breaks, the entire assembly line comes to a grinding halt. That's not allowed.

Next is training. People need to be trained in the agile area. We also implement SAFe for the scaled agile practices. We have training programs, and I'm a big fan of applied agile training. A lot of people go into the theoretical training. User story looks so simple. They go back to work, they can't write a user story. So you have to actually work with them, pick up their situations and scenarios to understand what it is, and have that hand-holding approach in training that.

Next is some of the generic things. This will set the stage, whether it is transactional processing systems or standalone, whether it's content-based, you have cross-functional systems. So don't apply the same rule and tools for everything. You have to have that custom approach in order to achieve the maximum velocity.

Minimum viable product and the component WBS. We heard about that example about car, somebody asking for a car. Don't build a wheel or door. It's not usable. Because naturally, our tendency is to break it down into smaller components and start building them. But think in a different way of arriving at a minimum viable product.

Distributed and co-located. Definition and measurable speed. And agile team workspaces. Do you have your workspaces designed for agile teams to work?

Vendors. Most of the places, 50% of them are vendors. Do they have the training? Do you have the provision in the SOW to make sure that they are able to deliver, and we can hold them accountable for those deliveries?

Development part, agile hybrid. Sometimes it's a hybrid, not true agile. Estimation is always tricky. I talked about the theoretical part and applied part of agile. The theory of agile estimation looks very simple, easy. You go back and try to do that, then you realize how difficult it is, unless you know some of the techniques. Okay? And code merge.

Built-in compliance. Compliance is not an afterthought. You have to design for compliance. You have to build compliance as part of your development process itself. Compliance includes security, ISO, and SOX.

Testing, test automation. Without test automation, you're not going anywhere. It's plain simple fact.

Risk-based testing. Can you test everything? No, you have to selectively test. Identify the high-risk areas, test them.

Test data management: setups, orders. If you don't do that, it becomes very difficult to move from one environment to another environment. It's a nightmare.

Performance test, it's again not an afterthought. You have to design for performance, develop with performance in mind. It's not at the end, days before you go live, you think about performance. That's not going to work.

Release: your go-live windows, freeze windows. We were over-freezing. Every month we have one week of freeze. Every quarter, we have two weeks of freeze. Year-end freeze, don't even go anywhere near the data center. Right?

And then process, important. How do we put all these things together? You have to have the development process, test process, all these areas.

Production support. If you're deploying things so fast into production, your production system is always going to be in flux mode. There is nothing called normalization. Before you normalize one, the new change is coming in, so you have to have a process for that. DevOps. Okay?

Then compliance, we talked about this. A lot of people ask us, "I want to go really fast. Can I get a pass on compliance?" It doesn't work that way. You have to design and build with compliance in mind.

"I'm going to be agile. I may not have all the documentation. I am entitled for more P1s in production." I've heard that comment. It doesn't work that way. In fact, you should have less P1s in production because of agile. Right?

The other one is a program team that is holding everything together to make sure that people are in the right direction, and we have the process and tools, as we spoke about.

We have an adoption team to make sure that we are directly working with the application development teams and businesses to make sure that the program is running at the right pace, and it is reaching where we want it to reach.

And the ambassadors. You have to have champions and ambassadors to talk about success stories. So it's important to identify those people well in advance. Let them speak about those success stories. Let them sell your ideas and cases.

That's all we have. Thank you.

Q&A

Thank you guys very much. We have one minute. Do you want to try to do one question?

Aji Rajappan

Sure.

Q&A

Who's the lucky winner? Anyone have a question? Back here. Oh, good.

Q: Thank you. So have you thought about how you would, we're a relatively large Cisco shop, how you would roll this philosophy out to the products that you sell, in terms of the software that runs, for example, in your call center application? In terms of the firmware that constantly needs to be updated and implemented. Have you given that any thought in terms of how you would approach that?

A: Yes, I'll take that. Actually, IT was a little bit behind. Our product development organization started their agile transformation probably six to nine months before us, so they're still going through it as well. But we should start seeing positive impacts from that very quickly, if we aren't already. They started a program called Escape Velocity, where all of our product development's shifted into this agile manner.

Thank you for the question.

All right. Thank you very much. Thank you guys very much. It was great.