Rosalind Radcliffe

Hello. My name is Rosalind Radcliffe, and I'm so glad to be back here with you again at DevOps Enterprise Summit, London virtual event.

Today I get to talk about automated testing for the mainframe. Yes, it's possible, and we get to have a fun conversation about how you really can do automated testing. So let's get started.

Looking at the test pyramid from Martin Fowler, we see what areas we should be testing and sort of how much testing we do in each one of those areas. When we think about this testing pyramid, it's big at the bottom in order to support that unit testing, that fast feedback. The point of this is we want to make sure that we get that fast feedback back to the developer when they're doing their testing, and getting that back to the developer as quickly as possible.

So what can I do? Unit testing: the smallest piece of code I can test. Let's do that unit testing, make sure I get that perspective. And then what's the next stage of testing? What's that next level of testing that I can do in order to do the testing? In most worlds, when they look at Java development, you look at things like JUnit. When you look at other environments, you look at the fact that normally you do unit testing and then you do automated testing. But when we look at the mainframe space, traditionally there hasn't been a lot of that automated testing.

Why not? Why not? The biggest reason why not is the environments. If I'm sharing a dev environment and sharing a test environment, my data isn't static. It's hard to build those automated tests in order to run this frequently, in order to do this kind of repetitive testing. I can't reset the data because everybody's using the environment.

And so we've done a lot of work to help make this better and make this possible, starting with z/Unit. Right? You've got JUnit for Java, so why not z/Unit? Let's think about z/Unit and a COBOL program. I can build a z/Unit for that COBOL program, and it will stub out all of the program calls. In a way, just like I build a JUnit, I can do my testing of my individual program without that environment. The way it stubs it out, it actually removes the need for the middleware, so I don't have to deploy it. I just compile it on z/OS, and I can run my test. I can run my unit test on that load module that I could run in production, but I'm doing my early testing first.

Then we want to get to the next stage of testing, and what's next? Well, we have this capability called Wazi Virtual Test Platform. With Wazi Virtual Test Platform, I can do that next stage of testing. I'm still in the build process. I'm still in that build phase, actually where I do all my compiles and link-edits, but I can then run this Wazi Virtual Test Platform in order to run that transaction test, again without the middleware. I don't have to deploy it into an environment yet. I can run my testing.

And then when we get to the next stage of testing, we have all sorts of different tools that may fall into this space. We have tools like Galasa. That is an open source capability that allows you to do end-to-end testing. It has test managers that know all about z/OS but also know about things like Selenium, so I can integrate my test. It is a test framework similar to JUnit, except on a larger scale, to allow you to build out those tests and really allow you to get that environment. We have test products like Rational Test Workbench that allow you to do those test cases that you're building for functional test or performance test.

A whole bunch of different tools fit in this space. And then, of course, the end-to-end testing: I'm looking at UI testing, things like Selenium to do that testing. But if I can focus in that bottom part and get that fast feedback, just like I do in other spaces, in the Z space I can get this continuous integration process. I can get this fast feedback.

Here, you see what we're trying to do is get this test pipeline. I'm trying to get this ability to do this code and build, and get my testing in that process to allow that first loop to actually happen. Then I move on to later stages, and I've got this provision, deploy, and test. I can provision a z/OS environment. I can provision it in OpenShift using Wazi Sandbox. I can provision it in any cloud environment using Z Development and Test, running real z/OS in the Intel environment. I can run my automated tests, and all of this helps remove this challenge of those static environments that we once had in the development environment.

One of the things I want to do is actually show you a little bit about this. It's fun to talk about it and it's fun to show charts, but how about this time, let's actually show what we're doing?

Here, let's start out with a modern development environment. Yeah, this is VS Code. This is my VS Code environment. In my VS Code environment, I'm running the Open Editor to allow me to do my COBOL programming. Here you can see I'm using the IBM Z Open Editor. It's available in the marketplace. Go download, and you can edit COBOL, PL/I, assembler programs, JCL. There you can see there are a few other pieces around here. I've got the user build capability. I've got debug capability in here. I also have the Galasa capability so I can do my Galasa testing.

I also have one other thing in here. I have added the Zowe environment. I've added the Zowe Explorer into this environment to allow me to interact with my z/OS system from my familiar interface. So when I'm building my source code, when I'm building my test, I can easily interact with that system without having to go to the system.

I know some of you might be more familiar when you think about z/OS. You might think this is what z/OS is, this 3270 green screen. Yes, that's there still, and if you are familiar and you like ISPF, that's fine. That's where I started in IBM development. But I can use a modern editor if I want to. I can use a modern environment so I can get that experience that I want to and allow me to use the IDE I want to.

There's an Eclipse editor. There's this. There's also an Eclipse Che environment. If you want to, you can run this edit experience in the Red Hat CodeReady Workspaces with IBM Wazi for Red Hat CodeReady Workspaces. You can run this in a cloud-native development environment sitting alongside the Wazi Sandbox. You have that full z/OS environment to do this testing, to allow you to write your code and test it before you ever share it to anyone else.

If we look at this Zowe Explorer, what I'm doing here is connecting to my z/OS environment to deal with the files that I have, to deal with this file system. I can look at jobs. I can look at the environment and do what I need to do with the system. What I'm using here is the Remote System Explorer access to z/OS. It gives me that full-function development environment that I need to allow me to connect, do the kinds of things I need to do, allow me to do my user build, allow me to do my different steps using the Zowe Explorer.

You'll see in this environment, I'm integrated with Git. My source code is actually in Git. In this case, it happens to be in GitHub, and it's sitting here managed in GitHub. You can see I've got all this COBOL code sitting in the GitHub environment. Okay, that might seem a little unusual, but how else am I going to do this? If I'm going to do my automated testing, if I'm going to do my processing, why shouldn't I have everything together in one SCM, building with my source code, with my tests, with the Java code that I'm writing that's part of the application, and with the front-end application? All of the pieces can be together in the single SCM, and so I can work together across my teams.

Now back to the code. I love doing live demos, but I'm not totally crazy. When I do a live demo, I make my changes in comments, so I can't make a code change. And so I make a change, and I save that change. When I save that change, I know I've made a change. Git tells me I've got this change. I would logically do a user build and then make sure my code works before I move on. But in this particular case, since, well, we don't have forever, we want to see this working and I want to kick off this pipeline. I'm going to go ahead and check this in because I know it's going to work. I did it right. It's a comment, so it's a pretty good chance that it's going to work.

And you don't want to watch me typing, really. But here we go: new change for the demo. We're going to commit that, and then we're going to push it. And so it's going to flow and be checked into the Git environment. What happens when I make a change and push it? My branch, I'm pushing it up. Well, it should kick off a pipeline like it would in any other environment.

And so let's see what it does. It kicks off a pipeline. It's kicking off a pipeline to run my process.

Now while that's running, we're going to take another look at one other program here. When I said this is my COBOL program that I'm editing, and you can see this is a full COBOL program, and if you're not familiar with COBOL and you don't know what you're supposed to do, it's a full editor. So you have the full experience to understand how to write whatever you need to write. If you take a close look at COBOL, it's really just English. Move something to something. It's not that hard to understand.

My full modern editor. But I also want to have a test case. I want a z/Unit associated with this program, and this is what I've done here. This is my z/Unit that was created using IDz and generated for me based on record and playback. Using the IDz environment, I can create this unit test. Once I have this unit test, I can edit it anywhere. It's COBOL. If I do a COBOL program, I generate a COBOL unit test. It's what makes sense. You can see this looks like a normal unit test. I'm checking data to make sure that my program worked as expected. I can have a series of tests within this one program that get run whenever I change my program. And so I can get that fast feedback and get that feedback right away as soon as I make that change.

That's what has happened here. I've done my build. I've done my dependency-based build, made sure it built the right pieces, and then I ran my unit tests. I got that fast feedback to say it was successful. Then I ran this next step. I ran this Wazi Virtual Test Platform, or Wazi VTP, test that allowed me to test not just my program but my program within the entire flow of that program, the transaction that I'm running. This happens to be a CICS DB2 transaction, and so I get to test that whole transaction. If the program change that I made caused a problem in the flow, I'd find out right away. The system would tell me that what I changed broke something else in a related program. I can get that really fast feedback before I've ever deployed it into an environment.

Then I take my change, and I put it in Artifactory, just like I would for any other system. Alongside the Java code, alongside everything else I've got in my environment, I put it in Artifactory so it can then be deployed. And so I then do my deploy. The deploy process puts it into a CICS DB2 environment so that I can run my integration tests. In this case, I'm using Galasa to run my integration tests, and it's going to run a set of tests.

But I've also put something else in here. I don't always have to run this. I can selectively run this. But I'm doing a recording of the data before turning on the recording, running my integration tests, and then turning off the recording so that I can create a new data file. So if I change my program significantly, I make new environment calls, I can get new data to use in the build phase.

The idea with these tools and this capability is to allow you to do things faster and sooner. By being able to stub out the middleware entirely, I can run that program as a batch program. I can run it standalone. But I haven't modified the program at all. It's the same load module that you can run in production; you can run in this way.

It's a set of new capability to allow you to do that using the characteristics of z/OS. It allows us to do this. And so we can have this early testing and early feedback. Running in this environment, even without the middleware, I can run it in debug. I can step through the program. I can get that fast feedback without having to have that full environment. Yes, I'm running on z/OS still, but that z/OS can be running in my OpenShift environment to allow me to do this. So I can do it all in a cloud-native way and get that fast feedback and get that understanding of what I've done and how I've built it.

It's really important that we're facilitating this whole lifecycle when we're doing z/OS development, and in the same way that we're doing it in the rest of the world. We're trying to bring z/OS into the rest of the processes. You have existing processes around Selenium, around other testing tools, bringing those into this environment. In this picture, what we're trying to represent is there are a number of tools that we provide and a number of open source tools that play a part in this pipeline.

In fact, virtually, it's your pipeline. Whatever pipeline you're using, you plug in the z/OS environment into it. If you're currently using Cucumber, I know people who have set up Cucumber to talk to z/OS to allow you to front-end that capability. That'd be another place we could look at building a manager for Galasa to extend it into that space. It's an open source tool. You can extend it. Anyone can extend and add additional capability. But it's your open source tools that you're using in this environment alongside the specialty capability for COBOL, for example, to allow you to go through this process and allow you to get that feedback.

But now, if we think about what I've talked about, I write my code using a modern IDE, modern experience. I build my z/Unit. I can modify my z/Unit. I'm doing it at the exact same time I'm doing my development. And when I do my build, I get that instantaneous feedback. I get that fast information back about whether or not what I did worked and whether or not it broke something else. Realistically, when you've got a big program, what's your biggest worry? That you wrote the part of the code that you changed correctly, or is it that you broke the stuff around it? It's the stuff around it that usually comes into this problem.

The other piece is those include files that you have in your programs. In COBOL, it's copybooks. It's the data structure that's shared across programs. You've got to change a copybook. You're always afraid about recompiling everything because what else is that going to break? Well, with this automated testing, I don't have to worry about making those changes. I can get that fast feedback in the system.

Building up my early integration test using dynamic environments allows me to do this experimentation, allows me to build up my capability and test it in isolation before I affect anyone else. The existing environments that are running on existing Z hardware can be dedicated to those late-stage performance integration testing or late stage in the pipeline. Doesn't mean late stage as in late in process; it means late in the pipeline. I can use those static environments for that later-stage testing. In these dynamic environments, I can provision and provision the data to it that I want so I can run that automated test, and I can get that fast feedback. And I keep going around the circle.

Now, there's no perfect picture, and there never will be a perfect picture. Realistically, that monitor needs to be happening as I go along the whole process, and so I should be monitoring the system all the way. But we never have a good picture. There's always a question, and there's no way to draw this perfectly. So I'm monitoring these systems. Since it's still running on z/OS in every one of these cases, I can use the existing monitoring tools to get that feedback on how it's performing. How is it really doing? How do I get that? Am I performing the same way I did before? You can do those like-to-like comparisons.

So yes, when you're running your tests on ZD&T or on the sandbox, it's not going to perform like it's going to perform on real Z hardware, but I can do a like-to-like comparison. I can see that it performs similarly, and I can get that understanding of how it's doing, and then do the performance test on the real Z hardware in the environment.

You'll notice here I have GitHub, I have GitLab. I've put Git in the picture because Git is so common as an SCM, and it's the place you store all of the front-end artifacts. So I might as well use it for all of the artifacts. I can store my testing artifacts. The new z/Unit capability, I can store it in Git alongside my source code. I have all of that information all in one source code manager, and it can flow through and it can process all in the same way.

Another really important thing about this pipeline and testing: I've been talking primarily about development testing, testing to make sure the code is correct. But you'll notice something else in this environment. I've got AppScan in here. I've got security scanning. When we're thinking about this process, security scanning, security by design, needs to be included from the very beginning. And z/OS development is no different than any other platform when it comes to those requirements. I still need to think about how I write my code. I still need to make sure I'm following good coding practices, I'm following good coding standards. Am I doing the right data validation? Am I running the scanning to make sure that I'm meeting those requirements, I'm satisfying what I need to, and I'm checking and validating?

The developer can get this feedback. It can be part of the pipeline so I can scan and watch. You'll notice here I don't have any errors: zero AppScan errors found. AppScan has a set of rules. They may not be the best, but they're a set of rules that look at COBOL. Things like SQL injection, data validation, the way you're dealing with data in general. All of those kinds of things that you should look at in any environment, you should also look at when you're paying attention to COBOL or any other language on z/OS. Same rules, same process.

It's just as important to remember those security scanning as part of your process and security by design as part of your process. That's all part of the testing that you need to be doing from the very beginning to make sure you're not introducing problems.

I know it might seem a little strange. You don't hear z/OS and testing for security very often because we talk about z/OS as a secure platform, because it is designed and built to be secure. But you're still writing code. Doesn't matter where that code's going to run; you're writing the code. So the code practices need to be the same no matter what platform or language that you're writing for. It's all fitting together in this process.

Hopefully, I've given you a good picture of automated testing on the mainframe and how it's possible as well. It requires a change. It requires doing it. It requires implementing change. But it's there and it's possible. You can shift that testing all the way left. You can get that fast feedback. You can get this testing in the build process just like you can for any other environment, platform, or language.

Now, one thing I always like to do as part of this is ask you. I'd love to hear your stories about automated testing and how you're doing it, and what do you find most useful? What tools are you using? What do you find most useful? So you can give me that feedback. If it happens to be open source, let me know if you want it to run on z/OS as well, and we can look into it. We've taken a lot of things and worked to get them running on z/OS or just tried them, and they work on z/OS.

So it's important. Let's look at the appropriate tools and help make sure that z/OS is just another platform, and you can see it as a big server that just happens to be reliable, scalable, and provide those ilities that we all talk about. But from a development perspective, a testing perspective, I can do things just like I do in any other development, and I can work in a cloud-native development way, even for traditional mainframe development. Thank you.