Keith Puzey

Okay, and welcome to this session, where we're going to talk about modern test automation from cloud to mainframe.

We're going to be two presenters today. There's Sujay Solomon from our DevOps Broadcom Mainframe team, and myself, Keith Puzey, who's a DevOps architect.

Through this session, we're going to cover, to start with, the testing challenges, so those challenges that you're all aware of. We're going to drill into things like what are the current trends around test automation, and then how we have these solutions that can enable you to improve your test automation story. We'll then move on to some of the more complexities when it comes to automating in an enterprise and how that will benefit you. And we'll end on some getting started: how you can get involved in our platform.

So when we look at digital transformations, testing is a very large impediment these days to these transformation programs. When you're testing, a lot of the time is spent on gathering those requirements, understanding what's being developed, and then also looking at how we can then test that efficiently. Obviously, testing takes time, and this affects the time it takes to actually deliver applications.

What we've found is, still today, 70% of testing is still manual. And enterprises still have COEs, and those COEs tend to be quite risk-averse. Their goal is to ensure that good quality software is released. But also, we have this impediment of needing to speed up. So we're seeing enterprises moving towards a shift left, moving some of this testing to the development teams.

This causes a balancing act. We have our agile teams. Their goal is to deploy quickly. Generally, because they're developers, they tend to use their own tools, which tend to favor things that they've used in the past. You tend to find that open source frameworks are used a lot in agile teams. This is very efficient for the teams to actually develop and test quickly.

But what we find is the tools aren't developed for enterprise-scale, full-scale testing. They're designed to be used by developers on their environments when they're actually developing on their machines. But when we look at the COEs, they tend to have the more enterprise-scale tools for their testing, and they tend to be very risk-averse. Their goal is quality. They want to make sure that whatever goes out the door is the best quality that we delivered.

We have this balancing act here: we need to be delivering quickly, but also we need to always bear in mind the risk. Velocity doesn't always mean good things. You can be very quick deploying bad software. We need to balance this velocity and risk.

There are also other test automation impediments that we need to look at as well. If we look at some of the other challenges we have these days for test automation, it's the testing bottlenecks. Traditional testing used to be this time-boxed event. We would go through the waterfall, we'd get to the testing phase, and we'd then do our testing. We spent a lot of time in that area, testing. We need to improve how we can do testing, so speed up that process, but keep the quality.

What we also see is delays in getting access to environments. Enterprise tools are complex. Generally, they're made up of lots of different systems, and those systems are different endpoints, APIs. You need data to test them. For a team to be effective, they need to get this on demand. They need to be able to quickly access environments, quickly access test data.

This kind of moves to the engineering shifting left. How can we enable engineering teams? And when we say enable, give them the tools that they need to actually do this efficiently. Those tools need to be things that, one, the engineers are comfortable using, but also are things that the COE can also enable them with.

Our last point is we need to get away from just doing performance testing, and we need to move on to performance engineering: designing products early on, considering how they will perform in production, and also test early. Very early in the cycle, we start doing performance testing, not just functional testing. All of these impediments affect teams today.

Now, if we look at some of the trends, these are some of the trends we're seeing in test automation. The first one is this blending of agile and DevOps. What we mean by this is agile teams take the requirements, convert those into stories, and then drive that to deliver product. But they need to also consider the ops side, so the implementation, the maintenance, the ongoing support. We're now seeing this blending where the DevOps teams are now more focused on not just delivering the functionality the agile teams are working on, but also how that will be maintained and supported in the whole framework, from engineering all the way through to production.

That kind of leads into the provisioning of these test environments and the provisioning of this test data. We're seeing more organizations that are looking at how they can speed up the delivery of these environments and this data to teams on demand, so agile teams aren't waiting for environments to become available and slowing down their delivery cycles.

The third bullet point here is we need to be testing the APIs and the microservices. Modern architectures are made up of many services within an application. Instead of just testing the UIs, we need to be testing those APIs to ensure, one, they're performant, not just that they're working, but they're performing in a way that as this scales, they're going to work correctly, as well as the fact that they functionally are going to work. So we need to automate that.

The final point we mentioned in the previous slide is this shift towards performance engineering. Looking at the product from the ground up, understanding where bottlenecks may be in performance, and making sure that we're catering for those performance needs as we go through the development process.

We've got here a typical enterprise architecture that we should all recognize. Enterprises are complex. You've got the network infrastructure, you've got the internal infrastructure. Most organizations have some form of cloud infrastructure. And then you have the application layer. Basically, as part of the testing, you may need access to any one of these components, in fact, maybe all these components, to do testing. As you can see now, there's a complexity here that is difficult to manage and difficult to test against.

So how do we unify these agile teams and the COEs? How do we make sure that our agile teams are not waiting for environments, waiting for test data, but in a way that our COEs can also work with those developers? So a platform that our COEs can work with and then enable our testers and our developers. Everything we do has to be integrated within our development pipelines. This is how we achieve quality with speed.

What we're talking about today is our BlazeMeter product. When we talked about developers liking open source tools, but our COEs obviously wanting tools that are enterprise scale, BlazeMeter does this. We support all these open source tools, things that developers are comfortable developing and using, building their tests, but on a platform that gives them those enterprise-grade functions: the security, the reporting, the support, all the integration points.

On the right-hand side, you'll see this thing called Taurus. Taurus is our open source initiative sponsored by Broadcom, and it's used for all these different executors. JMeter, Selenium, Gatling, JUnit, they all work with Taurus, and it allows you to do your load generation locally on your machines and in the cloud. Again, this is a way that we can enable developers to use tools they're used to and they like, but in a way that better fits into the enterprise.

Now we've talked a lot about functional testing and performance testing. What BlazeMeter also does, as well as our functional performance testing, is we have built-in test data. We can generate data, we can use your existing data, we can build that data and include it in the platform so that the COE and the development teams can leverage it.

We also have built-in mock services, which allows us to provision environments that may not be readily available, like third-party or internal, and a full monitoring and testing platform for APIs, allowing us to test APIs within your complex environments, all those different APIs that you may have inside your environments.

With that, I'd like to hand over to Sujay to talk about some of the cross-platform challenges.

Sujay Solomon

Hey, thank you, Keith. As Keith said, applications today are really multilayered, and they have a very large tech stack, as shown on the screen here. There are front ends like mobile browser UIs, Open APIs, and oftentimes these are powered by APIs that are routed through gateways, service meshes, and so on. Keith kind of talked about how to test that.

But when you dig deeper into the APIs, they're typically backed by either cloud or containerized services or some legacy services that are behind an enterprise service bus and still very productive doing their work. When we look at the cloud or containerized services, they were born in a DevOps-ready world. When it comes to testing, they're in a relatively good shape.

But when we look at legacy services like CICS or batch services running on mainframe, midrange, and others, there's really not that much DevOps or automation practices employed, especially when testing. This has to change because these legacy services are so critical to most enterprises, and really, they're unlikely to go away given their current throughput.

Without implementing DevOps and automating the testing of these legacy services, your full-stack application change cycle, those teams are really going to suffer, and they're not going to be able to get code from dev to prod as quickly as they could. One of the barriers to this happening is really culture. So let's take a look at the culture at present.

If we look at the SDLC here, most often when we take a look at the stack, the API and the consumer layers are typically system tested through a solid set of automated tests, which include functional, load, stress tests. So not too big of a problem. There are definitely improvements to be made, but they're in decent shape.

But if the app change actually needed a legacy code change or a database schema change, these teams are very siloed from the API or the consumer layer teams, and they just haven't really had the chance to adopt the DevOps culture of automation that we see that's so prevalent today.

We see high levels of manual testing from early-stage dev unit testing all the way to integration testing that's done typically by manual testers at the QA level of the SDLC with legacy services on the mainframe and such. Especially the QA-level integration testing, that eats up multiple sprints of your team's cycle time. Or even worse, it can cause some of these teams to actually just be stuck in a waterfall state where they finish their coding and they kind of just throw that code over the wall to testers, which isn't necessarily good from either a quality perspective or a cycle time velocity perspective.

If we move to the next slide, where we're trying to move some of our customers to is to change this. The simple answer is really to empower your development teams to actually own the responsibility of creating tests for all levels of the SDLC, and ideally doing that in an automated fashion. If you think about agile and agile teams, anyone with the right skills should be able to contribute to the team's cause. Whether you're a developer writing legacy COBOL code, or you're a tester who's maybe following regression test plans and doing manual testing today, together, you can put your hand up and say you're willing to take ownership of quality within the team.

With that responsibility comes ownership. Once you've got ownership, you're starting to build inherent quality into the code that's produced by these teams.

What are some tools that we can take a look at to actually make this happen? We're going to talk a little bit here about Test4z, which is an open-first mainframe test automation approach that Broadcom is bringing to this. When teams start shifting left, we don't really need to reinvent the wheel when it comes to automating the testing of some of these legacy applications.

Broadcom contributes heavily to the Zowe framework. It's hosted by the Open Mainframe Project and the Linux Foundation, and ultimately, that's a modern bridge between the mainframe and open tooling and frameworks. What we're doing with Test4z is we're applying these same open principles that were established by Zowe, namely REST APIs and SDKs, to really make it easy for developers to use existing test frameworks and languages like JMeter, Mocha, Python to automate mainframe testing.

In terms of how the experience actually looks for a developer or tester who's writing these tests, with Test4z and this open approach in place, test developers have their choice of framework and language to first create those scripts that turn all of those manual steps in a test plan into functional test scenarios, using this combination of Zowe to submit jobs perhaps on the mainframe, and then to use Test4z APIs to perform deep data assertions with a lot of services, like being able to handle large volumes of data, doing pattern-based search and compare, as well as snapshotting test data before you might do any modifications to it.

Being able to do all of this on-platform, such as on the mainframe, without the need to actually move any of that data off, matters because there's always sensitivity that's involved in moving such data off-platform.

Once you've got those scripts, it's quite straightforward to manage those test cases using test platforms like BlazeMeter or Jira Xray, or even execute them from CI/CD tools like Jenkins or GitHub Actions. This really makes mainframe and legacy testing, from a user experience perspective, really not that different from testing applications on other platforms.

Specifically, these Test4z services that we're talking about are great for programmatic testing of interfaces on the mainframe. But today there's still quite a bit of legacy terminal-based UIs that folks still have to test. So I'll hand it over to Keith to talk us through how to test some of those terminal-based UIs that we still have in these enterprise application stacks.

Keith Puzey

Thanks, Sujay. We've looked at how we can now test a mainframe, but we can also include that in our BlazeMeter testing. We talked about how BlazeMeter integrates with open source tools, and we created a plugin for JMeter that allows us to do an RTE test. We can do a functional or performance test. This top screen shows us a functional test where we're walking through testing an application on the screen. We can use that same test to actually then run a performance test. We can test tens, hundreds, thousands of users running this test against your platform for your performance testing.

Let's go and look at how this actually looks. If I go to my BlazeMeter console, we're now looking at BlazeMeter, and you can see it on the top here: we've got our different tabs. We've got functional, performance, mock services, and API monitoring. We're going to start with a functional test, and you can see we can start by creating a functional test by uploading a script. We take existing scripts and upload those.

We're going to switch and create what we call a scriptless test, so build a test using our scriptless palette. We can use a recording, so we have a built-in Chrome recorder to record these types of tests, but we're going to do this manually. If you imagine we want to test something like the Google Search screen, the first thing we need is to go to that URL. We're going to drag over go, and then type in the address where we need to go to, so google.com.

When we get there, we want to type in our search box to search for some text. We use the type action to say we want to type, and we need to know where on the screen to type. We have something called an action library that records the location of objects on a webpage, and we've already captured these objects here. We can use a picker to go manually add these, but we want to click on this entry panel. We need to type in some text, so let's just type in there, DevOps.

Then we need to click on the search button. That will do our search. The last thing for our test we need to do is obviously an assert to check the test worked. Let's do an assert on the title. I know that the title of the search window should be the search value we typed in, Google Search.

I can run a test here. I can put this in debug, run this locally on my machine. The test will change color as we're stepping through the steps. You can see we've gone to google.com. We've typed in DevOps. We're going to click the search button, and if you look at the title on the top, you'll see it goes to the Google Search. That test passed.

So we now have a test. We can now run this test on the platform. We can choose a location, so we can say we want to run this test either inside our firewall or outside on our public agents, and then choose which browser we want to use: Firefox, Chrome, Edge, and the version. We'd then stand up the relevant infrastructure at this location and run these browsers and run this test.

The test we've actually done here has static data. I could also say, instead of putting here DevOps, let's create some test data. We go to our test data panel, and we can import data from CSVs, we can take data from your on-premise systems. We can create our own synthetic data. Typical examples are things like addresses, UK data, a driving license, date examples such as five minutes in the future, five minutes in the past, next week, next day, and a registration form example.

Now we've got dates of birth, names, and if I just say my iterations, I can say I want 10 rows of data. We've now got 10 rows of data, their first names, last names, addresses, social security numbers, email addresses that are consistent with the names. We now have something that looks like valid data we can use in our test.

To actually test this, all we do is take our parameter. Perhaps we take the address. We copy the address, and we change our test. Instead of that hard-coded value, we're going to say we're going to take that data from our test data. That's going to be what we type in there, and obviously we need to change our assert to say we also need to put that in our assert. We now put that in debug, and that test will now run using that test data. If I mouse over it, we'll see what value we're using. That's the address we're using.

We now run that test, and it will run exactly the same as before, but instead of looking for DevOps, we're going to enter the address that we've generated. Google will do a search. We'll make sure that search comes back with the assert. And we now have a test.

If we look at some previous examples of these tests that ran before, this is an example of a test that's run. What we get is all the steps of the test. This is a test we did for actually checking out a banking application. We also get a video that shows us the test, the logging in and everything that happened, with a waterfall to show you the performance of all the components of that test.

If we were to choose a test with multiple browsers, each of these lines shows us the results. A tick means it passed. The iteration shows the data we used, so that was the test data we used.

We looked before at the mainframe testing, and if I go and look at an example here of a mainframe test, it's the same thing. If I expand out this mainframe test, you can see where we did a connect, like we did with the go on the web browser. We do the same thing: connect to the server. This is the screen that comes back. We then have the steps for enter our username, enter our password, click on the start monitor, and this is the screen that's on the mainframe. We can run the same functional test on the mainframe as we do with our web browsers.

If I now switch to our performance test, we can look at some other examples. When we're talking about a performance test, in this case, this again is a mainframe test. What we have is now number of users within the load. If I look at the timeline report, we can see for each of those steps we just looked at, so we had the connect, the disconnect, the enter password, enter username. We have all the response times for those. We can say what's the average response time for the start monitor? What was the average response time for our connect? We have all that data. We have much more here. We have latency time. We can drill into these transactions and run a full load test against the mainframe.

If we look at how we actually create a performance test, it's just like we saw the functional test. We'd come into the UI, we would upload our test. We can point to our JMeter file. We will load that JMeter file natively into the platform, and at this point, we can run this test.

This load test actually has some test data, and we can choose to upload the CSV file that comes with it, or we can associate what we call a data model. A data model is something within the platform where an example of the data we need. Like we saw in the functional, we now have our test data, and all we do is specify the amount of load, how many users, how long the test is to run for.

We can then choose the locations. Perhaps we want to run this in different parts of the world, different platforms: Google, AWS, Azure. Again, we can run this inside your firewall. We can have a mixture of load coming from inside the firewall and on public locations.

We can also link this to what we call our mock services. We mentioned how environments are important. We can specify which mock service is needed, and not just the endpoint, but also the transactions in that mock service. We can say we want a poor-performing Visa payment system. We want to test how that API responds when we've got this load running.

We then define our failure criteria. We can integrate with lots of APM tools to gather backend performance metrics as well as those frontend performance metrics. We can then run this test.

If we briefly look at an output from this test, what we'll see in the results screen is these are all the various tests we've run. We can look at the summary of the performance of those tests. We can baseline these tests and compare them against a baseline. We can also look at trends, so trend for this test over time.

Now we can also, like we discussed in the session, build mock services to provide those endpoints that may not be readily available. We can mix these endpoints with that data, so we can supply an endpoint with the data we need. We can also do API monitoring, allowing us to monitor if an endpoint is actually working, how it's responding, and whether it is performant and matching those criteria.

We've covered a lot in the last few minutes, and obviously, I hope you can see the value of BlazeMeter. I'll pass back to Sujay to talk about how you can get started with our platform.

Sujay Solomon

We kind of talked about that dichotomy between agile teams looking to shift left and the COEs being risk-averse, and Keith highlighted the keys to achieving that right balance there.

We also talked about the full enterprise stack where you've got legacy teams that are looking to shift left, and from a culture perspective, we want them to take ownership of testing at all levels of the SDLC, including automating it.

If you're looking to get started with addressing some of these in a structured and proven way, we have a few resources here that I'll highlight. One is BlazeMeter University, which has a rich set of self-paced courses that can help you level up on your knowledge with testing practices, especially at an enterprise scale. The variety of courses available there means you can identify exactly where your gaps are, and you can cherry-pick courses that are of interest to you.

We've also got our blog site on blazemeter.com. When it comes to the mainframe side of test automation, we've actually got a GitHub repo: github.com/broadcom/mfd-test-for-z. It contains a few use cases that we've come up with working with our customers. Most of these samples are JavaScript test scripts. You're welcome to clone those, and the license will allow you to use it and get started.

Another resource I'll point out is our modern mainframe blog site on Medium. You can check this out to learn about mainframe DevOps. Specifically, if you're looking for test automation blogs on there, I would urge you to search for Test4z on that blog site.

Thank you so much, folks, for attending our session on modern test automation from cloud to mainframe. For all of your enterprise-grade testing needs, I welcome you to visit blazemeter.com.