Tim Girouard

[~00:00] Hello everyone, and welcome to this session titled "Intelligent Test Selection." I'm Tim Girouard, Product Manager at Tricentis for LiveCompare. So today's agenda is all about the two main components that make up intelligent test selection.

[~00:00] So the first item is development, all those changes that are happening in your development environment. And for this session, I'll be primarily talking about changes as they relate to an SAP environment.

[~00:00] So these would include SAP-authored changes like support packs and enhancement packs, SNotes, and also local development changes. So this is change related to your own local system, your own customizing efforts, configuration, as well as any custom code that you create or maintain in your own local SAP development environment.

[~00:01] Then we're going to take a look at the operations side, and when I refer to operations here, I'm primarily talking about the activity in an SAP production environment, mostly usage.

[~00:01] So what objects or capabilities are used? First of all, what's available in SAP in terms of what you have your transactions, Fiori web apps, executable programs, APIs. And then the third part is all about the intelligent test selection, connecting the dots.

[~00:01] So taking and examining the changes that are happening in an SAP development environment and see how they relate to what we actually use in production. So we have the best test candidates to test in our QA environment when we try to invoke those changing objects in development.

[~00:02] So let's get started with the development piece of the puzzle here. So there's change going on, and the first question about change is, well, what do we test? Now, like I said before, changes can come by SAP or our own local developers and configurators.

[~00:02] So when there is this change event, we all have, I guess, an inkling of what unit tests would be acceptable to invoke that change. But all these objects in SAP are really connected.

[~00:02] So what else is impacted, and are there other objects on our test radar that should be considered? So whatever the change event is, that's the key question. What do we test? The first step in intelligent test selection is to examine the change.

[~00:03] And for today's demonstration, I'll be using the Tricentis LiveCompare tool to not only analyze changes in development, but also usage in production. So this intelligent test selection tool will take a look in development, analyze and identify all of those changes.

[~00:03] And on the right-hand side, that usage data, so those used SAP capabilities. But this is what a good intelligent test selection will do. It will analyze what's changing in dev, analyze what you use in production at the operations side, and find a list of core capabilities that are most at risk that you should test.

[~00:03] In this demonstration, not only will we find those capabilities, but we'll relate them to your test assets, so your test cases in your testing tools. On the left-hand side, I took a look at one of our systems in the labs, one of our SAP systems, and we applied a bunch of support packs in 2019.

[~00:04] Those support packs contained over a million changed objects. So it was a huge stack that we applied in 2019, and that is not uncommon. Oftentimes when I'm at customer sites, some SAP accounts are a full year behind on support packs. So when they apply that upgrade stack, it really is hundreds of thousands or millions of changed objects.

[~00:04] So what I did when examining these changes, I said, "Okay, what are the top 10 by type?" And here we see, well, the first one is SMIM.

[~00:04] Not sure if that's really invokable. That's more of an image icon type thing. But we have our methods, our report programs, web apps, table definitions that is, classes, functions, and data elements.

[~00:05] So the key thing, the key categorization of these changes is they are ABAP. They are mostly ABAP components and table definitions. So that's what's going on when you apply these support stacks.

[~00:05] SAP generally aren't changing your master or configuration data. They will from time to time change data related to their standard roles. But when we look at the bulk of the change, it's very ABAP heavy.

[~00:05] And when I say ABAP, that's SAP code. So it's very code heavy. Now, I did a similar analysis for non-SAP-authored changes, and that's on the right-hand side.

[~00:05] So I took a look at some of our most common customers that we were working with about a year ago, and I said, "Well, let's take a look at what your own local developers and SIs are doing in development in your SAP development environment. So we took a year's worth of local transports, and what we found was that most of the change is related to data.

[~00:06] So where you see TABU, that's data from tables, VDAT. And when you see AGR, those are the roles. Those are security roles.

[~00:06] So there's two really separate streams going on here when we examine the core changes. And I like to use this graphic by James Bach in one of his blogs, and I posted the blog at the bottom there, the link.

[~00:06] It's very interesting. In this heuristic, it shows this round Earth, right? And you have this inverted pie shape.

[~00:07] So I took the SAP support packs and the customer transports analysis, and I lined it up to this heuristic to just go right under that development perspective dotted line there.

[~00:07] So this is all that activity that's happening underneath the front end. Right? These are changes related to, in the graphic, your subsystems and your units. So at this point, we don't have the visibility from a user perspective, what should we test that are ultimately going to invoke these changes coming from SAP and from your own custom development?

[~00:07] Now, as we go further in the presentation, I'll talk about risk, right? And most at risk and intelligently selecting the best capabilities that interact with the change. Now, all of those capabilities that I will refer to are all from the user perspective.

[~00:08] So on this graphic, it's the front end. It's that lake or ocean where there's the user input and the mountains and the field. And then you can see the production data kind of flowing through both the front end and the subsystem layers. And we'll relate some real-life examples to those scenarios there.

[~00:08] But I just like this graphic to first, when you talk about the development piece, see from this heuristic where it all relates, especially as we discuss the next topic, which is more on the operations side. So for SAP production, we're going to look at all of the used objects that sit on that front end.

[~00:08] So whether it's data dictionary, standard, or custom code, all changes relate or kind of bubble up to that testable surface area. So I'll use that term throughout this presentation, testable surface area.

[~00:09] And when I mention that, what I'm referring to are the transactions, SAP transactions, the executable programs, web apps like Fiori, and APIs.

[~00:09] And bringing that heuristic back into the picture here, these all sit on top, right? That user perspective. So there's a lot of transactions in SAP. There are a lot of programs in Fiori web apps and tiles that are available.

[~00:09] What's important from an intelligent test selection perspective is to figure out the ones that you actually use that support business-critical functionality, and ultimately that interact with the changing components below.

[~00:09] And that brings us to this next topic. And before we get into that, let's take a break from the slides here, and I'll show you a quick demo.

[~00:10] So here's LiveCompare. I'm actually on the Tricentis LiveCompare tool, and to give you some real-life examples of what we're discussing here, I built this really small workflow, and these workflows sit on the back end.

[~00:10] Most customers don't really see them. Everything is automated when we intelligently select tests and things that you should look for at any given change event.

[~00:10] But it's good from time to time to peek behind the scenes to see what these intelligent test selection tools are actually doing. So I'm going to run this workflow. This action here is collecting all of the transports or, in this case, support packs from 2017. So these are SAP-authored changes.

[~00:11] So in 2017, in the environment that I selected, I applied eight support packs. Well, let's select a more recent year. Let's do last year.

[~00:11] And here I can select any of these SAP systems, development systems that I'm interested in, and I'm going to stay in the system called SAP43, one of my development systems. And the goal here is to find how many SAP changes or support packs did we apply to the system.

[~00:11] So now there's a much bigger application of support packs here. So let's get an idea of how many objects are contained in this 142 support stack. So let's bring out another action to do that.

[~00:11] Okay, so the results show that there are over 1.1 million changed objects from those 142 support packs.Now taking a look at these changes, here's where we'll find changes related to security, as well as all of those ABAP components, like function groups and so on.

[~00:12] Okay, so what about our local development? Let's take from that same timeframe, 2019. Let's take a look at all the non-SAP-authored changes.

[~00:12] So in our development box, we had 21 transports. These changes will mostly be related to table data. Okay, so we established what's going on on the development side. Now let's take a look at operations.

[~00:12] And in this workflow, we can read the performance history to find all of those capabilities we're using in our production system. So in this system, ECP100, we're using about 3,100 capabilities. These include executable programs, transactions, and Fiori web apps.

[~00:13] We can also find APIs, IDocs. And you might be saying, "Well, we have more than one production system. We have dozens."

[~00:13] What's great about this particular tool is no matter how many you have, you can retrieve multiple data sets from however many SAP production systems you have.

[~00:13] So if you have one or two, you just run two. If you have dozens, you just keep on collecting more application stats. So let's grab these stats, and we'll paste them here.

[~00:14] And next, let's grab all those changing objects. Let's bring that used objects here and our changed objects here. Now it's all about connecting those dots. So we want to know what our applications are using. So uses-what, we'll connect there. And from our changing objects, we want to do that where-used analysis. Now, developers in SAP are really familiar with that

[~00:14] where-used analysis. Now it's all about understanding the relationship between the two data sets, and we do that by finding object links and then analyzing object links.

[~00:15] So let's see what we got so far. We have our used objects. We can find their dependencies here. We have our changing objects. We can find their dependencies by using this find object links action. And then we can analyze them to get to the core, what we call most at-risk objects that you should test. Now getting back to the presentation, let's take a higher look at what we just accomplished there with intelligent test selection.

[~00:15] So we went over the used objects at the testable surface area. And we talked about how changing objects can be authored by SAP or by your local developers. Now, sometimes objects that are changed can be way deep when you look at this heuristic, which means you're less likely to invoke it during your day-to-day business supporting scenarios in production.

[~00:16] So there has to be a way to intelligently select the best types of test that most directly interact with changes. Now, the average customer uses about 3,000 of these capabilities, and about a third are custom, so their own custom transactions or programs or web apps.

[~00:16] When we apply an SAP upgrade stack, like the one that contained 1.1 million changed objects, we're going to find that, yeah, there's going to be a lot of impacts to what we use just because of that massive amounts of change.

[~00:16] So we do have a bit of a test reduction. If we were to do a full regression test, we're talking about 3,000 executables, whereas now we got it down to more closer to 2,000.

[~00:16] It's a little bit of a decrease, but it's really not enough. It's not enough for our customers that want to become more agile. They want to adapt to change faster, apply these support packs and upgrades to their systems, to their production systems faster to get to the most updated SAP versions and take advantage of all those new capabilities.

[~00:17] So for our customers who just need that more agile adoption of change, they use our most at-risk capability algorithm. That AI-powered what are the core things I need to test.

[~00:17] Now at this point in the presentation, a lot of folks ask me, "Well, what's the difference between impacted and most at-risk? Because I see we significantly decreased from a full regression test perspective what we need to test.

[~00:17] It's over 85%, under 500 transactions or web apps. This is much more manageable, but how did you get to these numbers?" Well, going back to this graphic, it's all about looking at, from the front end, what at the testable surface area is most directly going to invoke those changes below?That's exactly what those most at-risk capabilities do.

[~00:18] And oftentimes, SAP will change a really popular object. It could be some kind of function module, include or BAPI, that's reused hundreds and hundreds of times by many, many different transactions.

[~00:18] So one change can impact hundreds of transactions. And when this happens, LiveCompare takes a look at the change and says, "Okay, well, what transaction at that surface area most directly interacts with this change?" Okay, I want to test it with the most intelligent scenario,

[~00:18] the transaction that directly interacts with the change, maybe we use it the most in production, and it supports business-critical functionality. And so that's what the most at-risk test selection does. So at the end of the day, whether you test all of the most at-risk capabilities or the impacted, you're invoking the same changes.

[~00:19] It's just that with the most at-risk capabilities, you're using the most optimal, most efficient transaction set possible to invoke those changes. So what do we do with those 479 most at-risk capabilities?

[~00:19] Well, we integrate it, right? We integrate it with our testing tools like Tosca and qTest to see, of these 479 most at-risk capabilities, what tests relate to them, what tests contain those most at-risk capabilities.

[~00:19] And because we don't have 100% test coverage yet, there are going to be some gaps. So we'll find those, too. These are most at-risk capabilities that when we scan all of the design steps and details of your test plans, we could find no reference of that most at-risk capability.

[~00:20] So what we do with those gaps is we'll place those in a folder under requirements for qTest or Tosca. So those can be your next test scenarios that you build and create, preferably using the ARA tool, the Tricentis ARA Recorder.

[~00:20] Now, those tests that do contain the most at-risk capabilities are hits, and those are automatically populated in your execution list. So they could be automated test or manual, but they're there in your execution list and ready to be executed.

[~00:20] So just to recap, so when we find most at-risk capabilities, and if they are related to test assets or test cases that you have currently, we're going to populate that execution list, and all those gaps will go into your requirements.

[~00:20] And what's great about this particular test selection tool is we capture all this information, and we capture it for smart insights going forward. So here's a snapshot of about seven months here.

[~00:21] Now, let's say we implemented LiveCompare back in August. And at that time, we didn't have a broad range of test cases.

[~00:21] Okay. We're still getting used to this technology. We're still bringing it on board. So at that time, there were 63 gaps and only 10 hits of the 73 most at-risk capabilities that LiveCompare identified.

[~00:21] But take a look at this trend. Those gaps are trending down. In fact, the last month that we ran the tool, there were only 24 gaps. Okay. So we're getting to that point where we're closing all those test asset gaps, and we're creating the test that not only are covering the most used transactions, most business-critical transactions, but they interact with the most common change events that are occurring from SAP and from our

[~00:22] own local development. In other words, we're creating the test assets that we need to create that support our most critical business functionality and that are interacting with change events both locally and from SAP. When changes come from SAP, like we mentioned earlier, the most common change event is ABAP related.

[~00:22] But what about your local changes, things like configuration, master, and security? So if you do a configuration change to a process, maybe it's in material master or logistics or sales and distribution, you can have dozens or hundreds of test scenarios related to a most at-risk transaction. But it's the data component that really helps us filter to the best possible

[~00:22] test selection. And when you look at the process, think of process as all those impacted capabilities, and think of the data as a further way to filter those capabilities to make sure that you're using the test that most directly interacts with the data change.

[~00:23] Now, this all equates to quality, a better quality of test. For example, a very popular table in sales and distribution is this TVAK table, and those are your sales document types.

[~00:23] And in this example, we're changing a sales document type called TA. And in development, we set the lead time to 99 days, where in production, it's still wide open.

[~00:23] So not only do I want to test my test cases that contain, say, a create sales order transaction, we may have dozens or hundreds of these scenarios, but we want to limit them to just the ones that interact with this sales document type TA.

[~00:24] Now, when we do something like this, it equates to fewer test runs, more accurate testing, and a maximum risk coverage because we're focusing our testing efforts not only on the impacted business process, but the data related, the exact scenario that needs to be tested.And here's a real-life scenario where, prior to looking at the data component, we may

[~00:24] have almost 100 test cases that interact with the process that was ultimately impacted. But when we bring in the data component, we can see we significantly reduce it to just five. We don't have to test 100 different test cases that relate to creating sales orders, just the five that relate to creating sales orders that use the sales document type TA, because that's the core of what we really need to get at. So data is a huge component, and this

[~00:25] particular intelligent test selection tool, Tricentis LiveCompare, not only looks at those impacted processes, but it also takes into consideration the data changes. And then before you release these changes into production, you'll want to do a test audit. So everything you see here in green, that means these are your most at-risk capabilities that the test selection tool identified that have been tested and have passed.

[~00:25] So when we look back at that execution list in Tosca, we want to make sure everything's been tested with a pass status. The red indicate, yeah, it's been tested, but it hasn't passed yet. There are still some tweaks that need to be made.

[~00:25] Now, you'll see some gray here, too, and those grays represent tests not run. So we actually get this data right directly from SAP QA. We look into your QA environment during the test week, or whatever the test cycle is. You get to decide.

[~00:26] Maybe it's a three-day window, maybe it's a five-day window. But we look and see in SAP QA, has this most at-risk capability even been executed? So when you see a gray or red, if you have a failed test or a most at-risk capability that hasn't been even invoked yet or executed yet in QA, as a release manager, you're not going to release this change into production,

[~00:26] until everything here has a pass status. Think about your external auditors. They come in every year. They do an audit of your change management control practices. What is your process for implementing change?

[~00:26] And when they see that you have this kind of control in place that mitigates the risk of untested change capabilities finding their way into production, they're going to give you what's called a better opinion on your change management control practices.

[~00:26] You have the control in place to make sure that untested changes don't make their way into production. So it all starts with an SAP application change.

[~00:27] Tricentis LiveCompare can examine the quality of that change, run the smart impacts, intelligently select the right tests for Tosca, then finally do that test audit and make sure there are zero defects in production.

[~00:27] So I just want to take a moment to talk about the latest technologies and the latest releases for this fall and winter for LiveCompare. We introduced a developer impact analysis, which automatically analyzes your development environment, picks up any new changes that are going on, and finds the best unit tests. So most developers, they may know something that they'd like as a unit test to invoke whatever change they made. So I think back, and one of my

[~00:28] first jobs in SAP was an ABAP query writer. So I worked in HRIS, and I would create these info sets. Think of info sets as the data components that you build your ABAP queries against.

[~00:28] And just think about if you're not familiar with ABAP queries, they're basically just report programs. So I would change these info sets, and I had all these reports that are dependent upon those info sets.

[~00:28] So I made sure I would do a unit test on my HR reports that interact with those info sets. But I had no idea I was messing with other teams' reports that also interact with my info set. There were reports in benefits and other groups that also were dependent upon my info set. So when I changed it, if I removed the field, and some other report was dependent upon that field, well, I just caused an issue with their reports.

[~00:28] So what this intelligent test selection does for unit test is finds things that may fall off your radar, even APIs. So when you do make a change, you can still focus on your favorite unit test, but this tool also provides the other best candidates that also interact with the change, and it prioritizes those candidates by the most at risk first.

[~00:29] So what does intelligent test selection support? Ten times faster testing with accelerated release cycles, 90% risk reduction, improved software quality, and 50% lower cost, reducing the cost with automation.

[~00:29] Use intelligent test selection to become more agile. Apply these support packs faster. Don't wait a full year and apply a full stack. Maybe apply them quarterly or as they roll out, because with intelligent test selection, we'll find those optimal test cases that are most at risk that you should focus on. So thank you for your time today.

[~00:29] Please let me know if you have any questions.