David Rizzo

All right. Thank you, and good morning.

I'm David Rizzo. I'm Vice President of Product Development at Compuware. Compuware is the mainframe software partner for the next 50 years. We develop mainframe software development tools. We have customers around the world, and I'm proud to lead our organization through our transformation to an Agile and DevOps company.

I'm going to share a little bit of information with you about that transformation. The first step that we took was recognizing the problem that we had. Forty-year-old company, been doing waterfall for 40 years, slow-moving development organization, trying to compete in the digital economy, which requires you to be fast. Fast beats slow. And you had to have innovation, new ideas. Ideation is the key to success in the new economy.

These were the problems that we identified and things that we needed to look at. We also realized that quality, being a software vendor, we provide software to our customers, quality is number one. So we had to make sure that we were able to measure, maintain, and ultimately improve our quality as we moved through a transformation to a full DevOps organization.

So the solution is define the desired state that we wanted to get to. To be fully Agile so that we could have continuous code drops and fulfill business needs in a timely basis. Confidence that when we made changes, what we were delivering was going to work and was good. So we had to know that we could move at a fast pace to be able to deliver what we wanted.

We had to be efficient, move across the entire enterprise. It had to be easy. Our tools had to be easy to empower our developers to be able to move at that pace, and to make updates and enhancements that would be valuable, provide business needs, and meet the business objectives.

We also had to integrate. Integration was key. Being an enterprise company, the mainframe integrates across the entire enterprise through the cloud and all the way back to your mainframe system of record. So we had to make sure that those integrations would be available.

So how do we do that? The right tools to make it all happen. That was the big thing that we had to look at. How do we put in place something that would make it possible for us to move at that pace and do the things that we had our objectives?

So what we did is put in a toolchain, and this is the Compuware DevOps toolchain that we use to develop our software. We start in the upper left-hand side. We start with Salesforce, which is how we interact with our customers, so they can provide ideas to us and we can communicate using that portal.

We then use Confluence internally to collaborate. So that is our internal collaboration tool to share ideas. When we have a new idea that comes in, we put it out on Confluence, and we get a lot of collaboration from our people around the world who are interacting with our customers and using our products.

And then we use Jira for all of our project management, for all of our task management, for our Agile boards. All of that is done through Jira so we can keep track of everything that we're doing.

And as we start working on a new issue, we move through the phases for code editing, application understanding, to validate our code, debugger code, edit and manage our code. So we use the Compuware Topaz base, which is an Eclipse-based IDE, which interacts with our mainframe tools. But that's the IDE that we're using to do our development. It works with our interactive debuggers, and it can integrate with all the different tools.

So as we move through doing our basic code editing and debugging, making changes, the next thing that we have to do is make sure, of course, that we have version control. And we use two version control systems. We use ISPW, which is our mainframe source code management system, and we also use Git for our distributed code.

So we believe that the source code should be managed on the platform where it is going to be built and executed. So mainframe code stays on the mainframe, distributed code stays on distributed platform. So those are the two that we use.

And then we use Jenkins for our continuous integration. We have integrations with Jenkins into ISPW and Git to identify when code changes are made. When those code changes are made, that can automatically initiate testing, automated testing, both mainframe and distributed platforms, as well as builds.

And the results from our testing go into Sonar, so we can go into Sonar and use that to identify code quality, identify how changes that are made to our products and our code on a daily basis is impacting the quality of what we are delivering.

And for that test automation, we use not only our Topaz products, we use Zephyr and HyperStation and Strobe for performance. So we're monitoring the continuous performance and continuous viability of our code changes. We have a code coverage product that we use to make sure that our testing is getting proper code coverage along the way. We like to ensure that we have at least 50% code coverage as we move along.

And then ultimately, we get to where we have to deploy. Again, we use ISPW and XebiaLabs XL Release to do our deployments across different platforms. And that allows us to move, and confidently move, code changes from development to QA to production.

And then once we're in production, we ultimately have to monitor what's going on through faults, if there's any faults or errors, and any performance issues. So we monitor and continuously monitor performance using Strobe, and we have an integration back into Jira.

So within the environment, if an issue is found for performance or a fault is found, Jira issues can be automatically opened and go right back to developers so that you can keep a continuous integration in the true spirit of being fully DevOps enabled and have that continuous feedback.

So metrics that matter. Those are the tools that we use, and by using those tools, moving through our transformation to DevOps, what have we accomplished? Through this toolchain, we're able to cut our deployment time by two-thirds. So we're able to more rapidly deploy the software that we have.

We estimate that we do about two times more code delivery per year per developer, and that's not measured by lines of code. We actually measure that by stories and tasks and enhancements that are delivered. So we're measuring it on the value that we are providing to our end users and applications that we're delivering, not just simply lines of code.

And one of our biggest accomplishments that I'm most proud of is our code quality. So being a software vendor, we provide our software to customers, and the worst thing that can happen is an external defect be reported to us. And so through automated testing, through the use of continuous integration, year over year, we have reduced our defects reported by our customers and end users by 7%. So we're very proud that we're able to take what we've done and use that to be able to ultimately create a more positive experience for our end users.

So in summary, what we've done, we've mainstreamed the mainframe, which was our goal. Being able to include mainframe development into the mainstream of an enterprise, do cross-enterprise integration. Applications don't run simply on the mainframe. They run both in different platforms other than the mainframe, and there has to be integration.

The toolchain that we use is used on both sides, whether it's mainframe or non-mainframe, and we have that integration. And as our DevOps infinity loop shows, as we go through the process, we show on the bottom many of the mainframe components that are needed to do that, and at the top are a lot of the integrations which are not typically considered mainframe, but they do integrate with the mainframe and be able to allow us to achieve the complete DevOps experience on the mainframe and across the mainframe.

And ultimately, the question is, we did this, we've been doing it for about three years. How did we achieve this, really? What was the process that we have?

And what we've done is created our 10 steps to Agile development and DevOps on the mainframe. So the first and last step, number one, which is to define your desired state. As I said in the beginning, we defined where we wanted to be. And number 10 is to get to that cross-platform continuous delivery, true DevOps point.

Number one and number 10 fit in place, and they stay there. Number two through nine can go in any order based on the maturity level of the organization and current processes, policies, procedures that are in place in your given organization. You can do any one of those in any order based on what you have.

But the things that we found to be able to be successful in getting the mainframe mainstreamed with the rest of the enterprise and getting that cross-enterprise integration, the first step is to modernize the development environment. We use Topaz, as I said, which is an Eclipse-based IDE. In all of these arenas, there's other tools that are available. That's what we chose, and it's worked for us to give that modern interface. So when you're looking and talking to a mainframe developer versus a non-mainframe developer, they're working in the same environment. It's essentially an Eclipse environment, so they can interact and understand what's happening.

Adopt automated unit testing. Unit testing, we use a tool that we created, Topaz for Total Test, which allows us to do unit testing on the mainframe, automated unit testing, which is essential. There's things on the distributed side, such as JUnit. It's similar. It gives you the ability to do that. But having automated testing was essential to being successful. And again, the success we achieved is because we had that automated testing.

Four, which is to gain graphical intuitive visibility into applications. Many mainframe applications people are intimidated by because they're large. They've been around for a long time. We don't know how they work. Being able to graphically look at those applications and see how they fit together was critical.

And then step five, which is enable early detection of quality issues. Again, that's using things such as Sonar and that continuous integration and continuous testing to be able to see your quality.

As you move into step six, that one may seem out of place as initial training on how to do Agile, but training on Agile and DevOps is essential within an organization to get everybody on board.

And then as you move into step seven, you get into use operational data. That is on the ops side to be able to identify your performance and look at your faults as they may happen in your production environment. So having those tools available to do that.

Step eight is core source management for an Agile environment. We went to the tool ISPW to be able to have a truly Agile source management on the mainframe, fully integrated with Git to be able to do Agile source change management and have that DevOps integration across the enterprise for source code management.

And then automated intelligent deployment. Ultimately, all your development practices, all your testing, the way they get realized by all the end users is through the deployment of them. And that has to be automated so that it can be done continuously.

So using those tools and that toolchain, we've been able to accomplish those things, and these are the 10 steps. And there's more details about the 10 steps, of course, available at our website, Compuware.com. And that's how we transform and what we do with DevOps at Compuware.

So thank you.

Rosalind Radcliffe

My name is Rosalind Radcliffe. For those of you who don't know me, I am an IBM Distinguished Engineer. I'm responsible for DevOps for enterprise systems, so I help our products transform as well as help clients in their transformation.

And what I want to do is describe a similar story. Since the Compuware story was based about their tooling, I decided to do the same side. And so we're going to start with IBM's tooling in the same space and give our story, and then we'll talk a little bit about CICS because I can't help but do that, to show how a traditional z/OS product that's been around for the last, I don't know, forever, also can transform.

But in this particular case, the goal was to bring all of our z/OS development tools together in a single delivery pipeline in order to deliver better value to you clients, to make sure that you had an integrated stack, to make sure it was easier for you to adopt, and to allow us to deliver function faster and more efficiently for you.

So we were shooting for monthly deliveries. What we had was this wonderful set of products at the bottom. We still have those products, but we now worked to deliver them together instead of separately. And we started with the bottom of z/OS Explorer because that's what's provided with z/OS. So you all have it. It's available. We wanted to get a full DevOps pipeline for that product and all the products that sat on top of it.

When we started this process, we had release cycles that varied from relatively short to 18-plus months. And instead, what we wanted was a single pipeline that could give us the value and give you the value and make it easier for all of us.

What we did, we have a single development pipeline. Now, this was a challenge. If you think about it, it's 17 separate products. They actually are still separate products, and in some cases, they're totally separate products. But we wanted the IDEs and the environments to be built together.

So we literally stopped development for a while and said, "No, I don't care. Other than customer critical situations, we're going to deliver those. But other than that, we're not going to do anything else. We're going to build a delivery pipeline." And so we stopped and built the delivery pipeline. It took about four months. We built up automated testing. We built up this delivery pipeline that all the products could be part of so that we could deliver this toolchain and provide the value.

We decided to do this in pieces. So we started with the base set of products, so z/OS Explorer plus the CICS tools to give us a start and to get that base there. And then we added products throughout the year, and we continue to add products.

So any product that has an Eclipse-based IDE that is related in any way, shape, or form to mainframe development, which could be almost anything, as in any tool, will end up on this pipeline because you don't just do mainframe development. There's cross-platform development.

The important thing to recognize about this product set is, yes, it's Eclipse-based IDEs, but it's also the back-end pieces of the development for this. So it's not just Java development, it's traditional PLX development that we use internally. It's assembler development. And we have the DevOps pipeline for all the parts of this.

We don't use separate tools. We have one toolchain, whether or not yours is building the Z side or the distributed side. We have one SCM. No separation. All the code is together. Everybody can see what everybody's working on, so we have it together.

And if you look at the pipeline, you'll notice open source tools in it in some cases. One of the things that we're doing from my standpoint, this team uses Rational Team Concert because that's our product and because they actually find that most valuable. I'll tell another story where they also do that.

But we also have worked to get Git ported to z/OS. So if you happen to be a customer or an organization that wants to use Git as your SCM, go ahead and use it for your z/OS source as well. There's no reason not to. If that's your choice of SCM, use it.

We want to make sure that the pipeline is consistent, that when you're doing DevOps, it doesn't matter if you're doing DevOps for Z or distributed. It's one story, one set of processes, one set of capability.

There's some fun facts up here: 94 releases, 17 products. We've gone to one-month releases for fixes, and when we say fix pack, that doesn't necessarily mean it's just fixes. It's fixes and small function every month for the product set. And it is a single repository, so they're all sitting in this environment. They have a set of automated tests with them. They run every night so that we can deploy the function, and we have what we need to deliver customer value.

What did we learn? Well, we had an advantage. We had the CICS team who had already started. They had started in 2005, so we had a lot of good lessons learned on how to do things and how to do this transformation.

You have to give the teams time to work. You have to give them the education, the training, the understanding. They have to know what they're going to be doing in order to do this, and they will be less productive in some cases as they start.

Now, in our case, they were already using modern development tools, so that wasn't a problem. In the CICS team's case, in 2005, they weren't. They weren't using modern development tools. So it does take somebody who's been using a green screen a little while to learn how to use an Eclipse-based tool. Their productivity is not the same day one. So give them time to do this. Give them time to learn.

We took the lessons learned from each team that adopted into the pipeline to help improve the next product team's adoption. So take those lessons learned and bring them together. And we had to have management support, obviously, for this. Imagine telling a development team you're not shipping any function for the next four months. See how well that goes over with the sales team saying, "I want new function." You've got to stop. You have to get support. You have to acknowledge that this transformation is then going to give you value, and now we can provide much more value, much faster.

But the CICS team is a fun team to talk about as well. Now, if we think about the story I just told, it's about a set of products coming together. In the case of CICS, does anyone in the room not know what CICS is?

Really? Okay. CICS is one of the two traditional transaction processors that run on z/OS. The IMS guys would give me a hard time if I say it's the most popular, but between the two of them, they're both used by most of our shops. Many of them actually have CICS and IMS. But it is a transaction processing system. It's been around for 49 years, I guess, as CICS. I might have the date wrong. It could be longer. I wasn't around then, at least not in IBM.

So what have they done? This is the CICS end-to-end DevOps life cycle. If you look at it, you'll notice it's talking about design partnerships and working with clients. It's talking about what are we going to build in the first place. This isn't just a continuous integration cycle. This is the entire business process around how we're developing CICS and what we should be building. Complete transformation of the teams into modern tools and continuous delivery of the product so that we can get instantaneous feedback from the customers who are using it.

When you look at the continuous delivery cycle, this is assembler PLX code, but this is a complete automated pipeline, every 30 minutes pulling the code and running it, running a set of tests against it so that every build I have a version that could be deployed. So imagine this is the same way distributed teams work, right? It's no different.

So they're doing z/OS development in the exact same way. If you notice here, we have nice dashboards, nice reporting, so everybody can see what's going on in the environment at any given time.

Now, there's at least one person from CICS here who'll be happy to tell you stories afterwards as well from the development team. But what else did we do? Now, the CICS team started a long time ago, and we took their information and their capabilities and the rest of the organization, we created a CI/CD community so that the organization has the support across all of IBM in transformation.

We have Slack channels so that we can discuss what's going on, so people can ask questions, so people can get information about how to do this. And we have a golden topology for z/OS development. If you're internal in IBM, we have one topology that specifies the set of products that you're going to use, and I'll share that externally as well so you can all see it. But it has things like Jenkins on it. It has Rational Team Concert on it. It has ZUnit to do unit testing. So it's the standard set of tools and products you would need to do development.

And here are some fun quotes that are in the deck. Things like three days to 30 minutes for deployments. Things like doing testing of collection on platforms from two days to 20 minutes. This is the value you can get out of this transformation.

And what do I really need help with? This one's really simple. There is no reason that the mainframe should not be included in your DevOps transformation, so please remove the line "mainframe excluded." That's what I want. That's what I need help with.

I need everyone to help the mainframe developers who are as old as I am understand that, yes, the mainframe can do DevOps, and we need to kill this concept of two-speed IT because it doesn't work. I said that last week at the Gartner conference, too. I'm happy to stand up and say this because it doesn't work. Clients explain that it doesn't work. We need to help everyone understand multi-speed IT is really what we need.

So I think I'm done. Thank you very much. Hopefully this was helpful, and I'll talk to you the rest of the week.