David Rizzo

My name's David Rizzo. I'm Vice President of Product Development for Compuware.

Compuware designs and develops tools for the mainframe, and that is to make developers who are developing applications on the mainframe more productive. People at Fortune 1000 companies around the world use our products to help make their jobs easier.

About three years ago, we went on a transformation to take the company to agile DevOps, and we have achieved that to some extent today and are still along the journey. I'm going to share with you what we've done.

The first thing: as I said, about three years ago, we started on this journey, and we recognized that there was a problem. The problem was that we were a 40-year-old company that was rather stagnant in a technology that was rather stagnant, that being mainframe technology. We wanted to make a difference, and we wanted to be innovative and make some changes in the ecosystem. So we decided that we had to make some changes within the company.

Forty years of waterfall development. We've been around for over 40 years, and we had a nice system of waterfall development that allowed us to deliver products. Every 12 to 18 months, we would come out with something very systematic. But we were slow-moving. We weren't able to meet all the needs of our customers and demands that were there, so we moved at a slow pace, and we recognized that that was a problem.

Especially in the modern digital economy today, fast beats slow. So you have to move at a more rapid pace, and we needed to do something to make that happen.

We had been a company around for 40 years. We develop software products. We hadn't come out with a new product in over 10 years. We needed to inspire innovation and start becoming an innovative company and deliver new innovation to our end users and to our customers. That was one of the other problems.

The other thing, if we do deliver to our end users, our customers around the world use our products, we had to maintain a level of quality that was second to none, and not only maintain it, but also improve it. And of course, to maintain and improve anything, you have to be able to measure it. So we had to come up with metrics to be able to measure what we were doing and how we were going to continue to do that.

The first step we took was to define our desired state, where we wanted to be. As a company, we made a commitment that we were going to now deliver new software, new feature functionality, new products to our customers every 90 days. The first business day of every quarter, we are going to do that. I am extremely proud to tell you that in January, we will do it for the unlucky 13th time in a row, but it will be the 13th time in a row that we will do that. That was our commitment to the market that we made publicly.

How we were going to do that: we had to put some agility into our organization. We had to be able to deliver code changes that would fulfill business needs for our end users, and we had to do that in the right-size pieces and be able to do it frequently.

The other thing we had to do is be able to have confidence in what we were doing. When we make changes, we're moving at that rapid pace. We had to make sure that developers had tools and could make changes that need to be done and be confident that what they would put out would not have unintended consequences and would not negatively impact the end users that were using it.

Ultimately, we also had to have efficiency. We had to make sure that we were maximizing all of our time and the skills that we had across development, testing, and on to operations and ultimate deployment out to production. We had a focus to make sure that we kept our quality and we were very efficient while we did that.

The other challenge that we had: we're a mainframe development company, and some of you may or may not know some challenges for mainframe companies. There's an interesting workforce dynamic where the statistic today is somewhere around 65% of the people that work on mainframes won't be working on mainframes in five years because they'll be going off to do something more exciting, like retirement and sitting on the beach.

So there's a lot of inexperienced people, whether they've developed on other platforms or whether they're new, fresh out of college. There's a lot of inexperienced developers working, and we had to create and make it easy for those people to work in the mainframe environment, give them tools that they could use so they could make those frequent right-size changes with good quality and good confidence in an easy manner.

The other big challenge we had was integrations. There is no such thing as just a mainframe application that just runs on a mainframe and doesn't touch anything else. Today, you think about mobile banking. It starts at the beginning from your mobile phone. It goes back, and on the back end, there's a mainframe sitting there.

We've heard from some companies this week. They talked about some of the things they did on the front end, and a lot of the back ends are a mainframe sitting there. So we had to develop integrations with a tool chain to be able to work across those platforms so that the mainframe could be used with all those modern platforms.

Ultimately, that means that we had to have the right tools to make all of this happen, and that, what we feel, is the key to our success. Of course, we develop tools, but also giving tools to the people to do the job the way they need to do it becomes essential.

This is the Compuware DevOps tool chain that we have implemented and that we use at Compuware every day to do our work. What's interesting about this tool chain is you'll see a lot of different names on here. A lot of them are related to Compuware because we do use our own tools.

But ultimately, everything starts out with trying something new that isn't working. So we'll give that up.

We start out at the upper right-hand side of the slide here where we have an idea. I talked about innovation, coming up with new ideas. We start with an idea, whether it comes from our product management or whether it comes from our customers, and ultimately, that goes into Confluence, which is the tool that we use to spread ideas throughout the organization. Across the world, every person that works for Compuware and some of our customers can collaborate using the Confluence tool and form ideas that we will be using and doing.

Ultimately, that all moves into Jira, where we do all of our tracking. We do all of our agile boards there, Scrum and Kanban. We use Jira to do all of that.

We move down to our interactive development environment, our IDE. We use and created an Eclipse IDE that allows us to be able to develop code all the way from, again, if you go from the right-hand side, you have making data available, creating test data to move into your application, writing, editing your code, validating the code. We use Sonar to be able to do some validation on the code as it's created, right when it's being written.

To debug the tool, we use Xpediter, which is part of our Topaz suite of products, and we also have some analysis tools that we've created as part of our initiatives over the past few years: to create graphical representation of applications as they're running. So those are the tools that we made available to our developers to be able to use, to be able to have that ease of use to be able to develop.

Once they've written the code, then you have to move along, and the first thing you have to do is keep it into a version control system. We at Compuware also have a version control system that we developed, and you'll notice another one up on the screen is Git.

Our mainframe source code sits in our version control system, which is ISPW for mainframe source code, and our distributed source code, non-mainframe source code, which is for our web tools and our Eclipse tools, that sits in Git. Our belief at Compuware is that the source code should be maintained and housed on the platform where it's going to be built and deployed. So mainframe on mainframe, and the other platforms on the platforms where they go. That's how we do it.

In the very middle of the slide, we have Jenkins that we use basically to put everything together for us. Once we've done our code, once it gets in, it goes into Jenkins, and Jenkins is able to initiate automated tests to be run both on the mainframe and off the mainframe.

So we can get automated tests run efficiently, and those results then get fed into Sonar. We can see how code quality is being impacted by the changes that were made, and we do that on a daily basis. We want to run extensive regression testing and automated tests every night so that we can see how we have impacted the quality of our code.

Then ultimately, as part of the testing, we have some automated test tools that we use. We have code coverage tools to make sure we're testing all the pieces of the code. Again, those are parts of not only our Topaz product, but we do integrations with different things, with Jira to put our test results in using a Zephyr plugin and some of the other tools that we have.

Once our code is all ready to go and tested, then we have to deploy it. Our ISPW tool, we do some source change management as part of it. The other part of the tool is deployment feature. So we're able to deploy not only our mainframe source code, but from other platforms. The deployment can all be managed and also has rollback capability, and for some cases, we use XL Release from XebiaLabs. So integrating with some of our partners to ultimately roll that out into and across all the environments from dev, test, and production.

Then once we get it into production, to be able to monitor it, so operations on the bottom right-hand side there. Operations has the tools to monitor performance, monitor any issues that come up, and across the whole application. When you look, see AppDynamics and Dynatrace, you can see performance outside of the mainframe and how the whole application is working.

One of the biggest things that I didn't talk about, because I think it's the most exciting thing, is in the upper left-hand corner. About 35 days ago, we announced a partnership with Amazon AWS, and our Topaz product, which is an Eclipse IDE, can be deployed on AWS so that it does not have to be deployed on individual machines in the facility, which makes for rapid upgrades. It allows new features to be experienced by developers immediately, and it only has to be done once. It's a great accomplishment to be able to do that, so we make that available and take advantage of some of the cloud services working with modern mainframe technology.

Ultimately, what we have to do: we have to have metrics that matter to us, and we have to measure what we're doing so that we can improve and keep going. Some of the metrics, three things that we measure that we've measured over the last three years and where we've seen great improvements, is our time to deployment. We've reduced it by two-thirds using deployment tools and putting that modern tool chain in place.

We deliver two times the amount of code per developer per year, and we don't base that on lines of code that are done. We base it on the number of enhancements, features, and functionality that we put out. So that's how we get to the two times more code per developer.

The thing I'm most proud of is our external defects that our customers experience in our products have gone down 7% year over year, and that's by adding in automated testing and continuing to improve the process, and testing more regularly and doing those small drops allows us to be focused and deliver better quality code.

Ultimately, what our goal was and what we've achieved is mainstreaming the mainframe. So we've enabled DevOps across the enterprise, and we've done that by putting the proper tool chain in place.

You can see in this slide at the bottom is what the Compuware tools do, and the developer productivity tools basically are what we use for Compuware, our Compuware tools at the bottom. But we also use all of those things that are at the top. We've created over the last three years partnerships, over 20 partnerships. Many of them are listed there, and those partnerships that we have are all used by us on a daily basis to do our work.

So we didn't create those just for marketing, so to speak, but that's how we do our job every day and how we've been able to take modern tools that aren't mainframe to implement them, to have a DevOps strategy and have a DevOps tool chain across the enterprise.

So that is the high level of all the things that we've done. We were so proud of what we've done, and we looked back in the beginning of this year, we said, "How did we get here and what did we do?" We created 10 steps to mainframe agile development, and we created a white paper. It's available on our website, Compuware.com.

But when I tell people the 10 steps, these are actually the 10 potholes or the 10 cliffs that we walked off of, so to speak, as we were going through our transformation. These are learning experiences for us, and we put this out for everyone to read and see because we don't want other people to have to experience the same kind of things that we did.

A lot of them I talked about: creating that modernized interface, having a good interface to use, automated testing, having graphical, intuitive visibility into your existing code, being able to understand it and know when you make changes, what changes you're making and how.

Enable detection of application quality, so we use the same things in development that we do in production, so we can measure performance and those types of things. We know how and can reasonably expect how it's going to perform in production based on how we have done it in development.

Making sure you train people, and some people think that's an odd spot. You train in the middle because you have to have an environment set up. One thing I didn't talk about that I could go on for hours about is the culture, which is a big thing, and we've heard a lot about it this week. But once you have the tools in place, getting people trained, having that data run through, having a good source code management system, then you'll be able to get automated, intelligent deployment, which is, again, what we have so we can roll out our releases every 90 days and have great confidence that what we are delivering and what we are putting out to our end users has good quality, it will work as they expect, and we're meeting the needs of the market on a daily basis.

Again, that's what we put together because we realized the journey we had been down and the things that we struggled through and worked through, we want to share with everybody because if you get these things in place, it'll make your journey much easier to do.

With that, I thank you, and I'm going to turn it over to Rosalind. Thank you much.

Rosalind Radcliffe

So it's always great to hear multiple mainframe stories. That's why I'm happy to be up here speaking, because it's the stories, it's the fact that different people are doing this that helps bring the value to your organizations.

What I want to talk about is actually our internal transformation story. We have a couple, so I'm going to do a couple of them.

One is related to our development tools. We also build a set of development tools for the mainframe, and we provide a full end-to-end development life cycle. But it's not just for the mainframe. We provide a set of tools that provide a collaborative life cycle management solution end to end, whether or not you're distributed or Z.

We want to focus on making things not different. So if you're doing Z development or you're doing distributed development, you should be able to do them exactly the same way: the same processes, the same procedures, and the same tools.

In this particular case, we wanted to take the DevOps practices and apply them to the tools we were building to help people build mainframe development, distributed development, et cetera, and get them together so that it was easier for you.

What we had in this particular case was a set of tools that people were using that allowed them to do the actual development, the configuration of the CICS environment, the working with the MQ environment, all of these things that everyone, system programmers, developers need to deal with, as well as our specific development tools.

These all were developed by different teams. They all had their own pipelines. They all had their own testing. But you, as customers, had to use them together.

So what should we do? We don't want to change the teams that are doing it. The CICS team should build the CICS Explorer. That makes perfect sense. But what we needed to do was bring it into a single pipeline so that they could all work together.

We needed a set of automated tests, and this is where everybody has a problem. I need automated testing, so we stop development. Think about that. I ship a product, and I stop development.

Well, that's not totally true. I still satisfied APARs or PMRs or anything that came in from the field, but I didn't do any feature enhancements for this three months because I wanted a fully automated test pipeline across all of these products. And so we did it. We did this back in 2016, and each year we provide or add more products to this pipeline based on customer requirements, customer demands, and other feedback.

If you look at this, you'll see we're releasing monthly. So every month, we ship or release external to customers, so you have the new capability. We have a continuous delivery pipeline underneath this so that we can ship the function when we need to, and we provide a mainframe developer site so that you can get this, all the pieces that you need, the specific pieces that you need at any given time.

It's built based on z/OS Explorer. z/OS Explorer is an Eclipse-based interface that you all have. It's provided as part of z/OS, so you all have it. It's there, and all you're doing is adding into that the capability for these other parts.

So if you happen to be a developer, you get the developer pieces. If you happen to be a system programmer, you get the system programmer pieces.

The way we did this, we learned from past experience, and I'm going to talk more about the CICS team, but they started in 2005 with their transformation. We had to make sure the teams could work together. We had to work them so that they understood that even if they were in Hursley or they were in Raleigh or they were in wherever, they were working together in this common delivery pipeline.

We are a very geographically dispersed company. That's probably an understatement. IBM is relatively large. We have people all over the world, so we have to work together. We have to have the right retrospectives. We have to figure out how to do team collaboration. And so we used our product stack to be able to do this. It's very easy to collaborate when you have an integrated tool set that allows the teams to work together to provide feedback at any time during their day.

The CICS team is a story we like to tell because, well, think about it. CICS is at the core. It is traditional z/OS development. If you can do DevOps with CICS, you can do DevOps with any z/OS application. This is core application code. This is core system code. It's assembler, it's PL/X.

They started this process with understanding their backlog, managing their backlog better, having a combined set of processes. The CICS team also builds Java work, so they do Java, they do CICS Explorer, so they do distributed development as well as Z development, and they brought all those teams together.

They brought them together into a single tool chain, and we found significant value in having a single SCM across all of the code and all of the work. Whether it be assembler or Java, it's one place. They can work together. They can share the information. They can collaborate.

You also see on this that we're applying this from the very beginning of the business process all the way through. So we have the CICS Design Partnership Forum. These are you clients that are participating in this journey with us. So we have the discussion, we do retrospectives, we do playbacks with the clients involved in this, so the end users, so they can participate in the capabilities.

We have a pipeline behind it. The CICS team uses our CLM stack, so they're using Rational Team Concert to do their development. They're using UrbanCode Deploy to do their deployment capability. They have a set of automated testing tools. We call it JAT. It's built on the Java automated testing framework, so they can do it, but they're testing assembler code with it. They're dynamically provisioning systems, including CICS, in order to do their development.

If you look at the picture down here, you'll notice something significant: beta-quality builds created daily. Every day, they're building a beta build to provide to you customers with new functions, with new features. Daily interaction. That's what you call a pipeline.

This is traditional Z development done in a modern process way. You can also see here we have Jenkins doing polls to see what's going on in the environment. So we also want to understand that there are open source tools that you may, or we may want to use as well as part of the pipeline. And so we do that. We include those tools where it would be appropriate to see how the open source tools play in the pipeline as well.

The CICS team has done so much that they had been sharing their stories, and we needed a better way to share the stories across the teams. So we built a community around this transformation for the Z Systems organization and anyone in IBM who wishes to share. There's a community to share best practices, to share stories.

Every month, we have phone calls, video conferences to help share information. We have a Slack channel to allow people to ask questions, to get feedback, to understand.

We defined a golden topology for IBM internal to use for their development process. As one of the quotes from the team, the golden topology is actually a key piece of this because by using our tool chain, we make it easier for teams to do their job. By having the integrated tool chain available to them, they can do their job in a much faster way.

There are also some fun quotes on here. We have the same problems you do deploying systems. How do I deploy that Z system and run those tests and then throw it away?

Well, nice, simple environment. They went from three days down to 30 minutes to deploy an environment. Now, think about this. This is z/OS. So I'm deploying a new z/OS environment using zD&T in my internal cloud environment. I'm deploying my application to it, and I can run my test now. I can do updates to all my environments that are in the environment. I can do whatever I need to do in the timeframe that's specified.

Now, we've extended this for a number of different things. We don't just build code. We build documentation. We do translation. We do a few other things. When you think about your stories and you think about your work, you may not have some of those requirements as part of those stories, but we do. So we include them in our stories to say, for this feature to be done, it has to include all those pieces. So all those pieces have to be part of our DevOps pipeline. And so we've added them.

We also take the insights from the systems and feed that into our information. So we use our tooling to look at the code coverage, for example, for our tests to understand whether or not we are testing everything we should be testing.

Now, I'm always looking for help. I spend my time with clients. I spend my time on this DevOps journey. The biggest problem I have when I walk into many of your shops is hearing, "It already works. It just works."

The mainframe guys say, "It works." So take stories-- oh, he's left. Take the stories that we've told you. Think about this. If CICS development, if z/OS development, if the core development teams can do this and are shipping more function that you're asking for faster with higher quality, why can't your teams?

More importantly, I understand that it works. I've been in IBM for 30 years. I started in ISPF development. I know a little bit about Z, maybe. Maybe a little bit.

The development practices that I see when I walk into many shops look exactly the same as 30 years ago. What else on the planet is the same as 30 years ago? Nothing else. So I know it just works because it worked then, but do you still use a phone that's tied to the wall? That would work.

Think about it. I know that it works, sort of, and the people that have been doing this job like me for the last 30 years want to retire in five years or six years or whatever it is. I get that. But we need the next generation to be able to do development on this platform as well.

If we think about it, the mainframe really is just a big server. It just happens to have better qualities of service. It happens to have better reliability characteristics, but it's just a big server. Why are we doing things different on that just a big server?

The other thing I need help with is this idea of two-speed IT. There is absolutely no reason that your system of record cannot move as fast as any other system in the environment. In fact, if you talk to many of your Z guys, they'll probably tell you, "I make changes weekly, maybe even daily," because they're delivering small bits of function into that environment on a regular basis.

There is no reason to say the system of record needs to be slower. There is variable-speed IT or multiple-speed IT or whatever.

I'm sorry, if you build a good API and it provides your customer record, if you're having to change that 50 times a day, then you wrote an awful lot of buggy code. I'm sorry, a service is a service. If I need a customer, it's probably not changing 50 times a day.

So let's think about the business value we're delivering through these changes and make sure that we're doing the right thing for all of our applications. And for the system of record, it needs to go at the speed business requires.

So thank you, and I appreciate your time.