Jason Secola

So if we're ready to go ahead and get started, I first want to thank everybody for joining us here today on the first day of the conference. I know there's a lot of really interesting topics that are being presented, so thanks for joining me for this one. What I'm going to be discussing today is AI-driven testing — how we can start using AI to more effectively start removing some bottlenecks in the testing lifecycle and improve quality in a very complex and fast-moving IT landscape.

My name is Jason Secola. I'm the product marketing manager here at Tricentis for our Tosca solution. I've spent the majority of my career in software testing and software quality, going from test management to functional testing, performance testing, and performance engineering. But like I said, today we're going to be talking about that in the context of AI, and where we want to see that start taking shape.

I'm going to start off by talking about some of the perceptions and the realities of testing, because testing is certainly of critical importance — but there are also some challenges that testing can sometimes pose in an Agile and DevOps framework. And those challenges really need to be addressed as we start seeing more and more AI-augmented tools being used to further accelerate Agile and DevOps. We're going to discuss where AI can be leveraged to enhance testing today: where it's already being used, some areas where we think it can maybe start being used more effectively, and a little bit of future vision around that as well in the very near term.

Testing as the perceived bottleneck

So I want to start by talking about some of the testing challenges in DevOps. According to the 2023 State of DevOps Report, 52% of DevOps teams have cited balancing speed with quality as a significant challenge. And what I want to key in on here is that they're citing testing being a critical area where this tension is most felt — where there's the most friction. So as teams are pushing for faster releases, more frequent releases, maintaining high-quality systems and software becomes increasingly difficult.

And this isn't on the screen, but another stat that I wanted to mention that I think is relevant to everything we're going to be talking about: in the World Quality Report of 2022 and 2023, it found that 44% of respondents were identifying testing as the primary bottleneck in their software delivery process, mainly being attributable to the time that it takes to execute and analyze tests. I'm going to be digging into that a little bit more as we go through this presentation, and also focus on how the use of AI in development is further increasing the speed and volume at which testing needs to keep pace.

But before we just pile on testing too much, let's be fair to our friends in the testing world. It is a crucial aspect that must be done and must be done well. But because of where it sits in the lifecycle, it's an aspect that for many teams can sometimes conjure up maybe some unfair reputations, depending on which side of the DevOps world that you live in.

If you're on the development side of the house, this might be how you view testing: sometimes you're working hard to meet your deadlines, to keep your sprints on track, to manage your backlogs, to ship your projects off for release, and then maybe it hits testing. Things take a little bit longer than you're hoping, and sometimes it feels like you're running into a bit of a wall where things come to a grinding or screeching halt. But if you're on the production side of the house, and testing gets rushed — applications are getting pushed into production and testing has moved on to their next project already — maybe this is the result that you experience: you're having to deal with frustrated customers, end users, or internal users that are dealing with a buggy application that doesn't function as it's expected to, or doesn't perform as it should.

So let's have a little sympathy for our friends in testing. The IT landscape is very complex, and testing needs to make sure that they're managing quality across all of this. Yes, testing is more embedded these days across the entire lifecycle as a continuous function, but ultimately the burden is falling on testing to make sure that quality is being achieved — across multiple application types, multiple deployment models, API testing and integrations and customizations and configurations, early-phase testing, all the way through testing very robust end-to-end business processes that span multiple systems. Speed needs to be achieved in test, but so does quality throughout all of this.

What the data actually says

I want to throw some of those exaggerated perceptions aside for a minute and ground some things in reality with some stats that we've seen. This came from the Role of Testing in DevOps Environment study that we did with TechStrong Research. When we asked, "How often is testing slowing down your ability to release?" — 38% of folks were saying always or frequently. That number by itself is pretty big. 44% are saying sometimes. You combine that, that's 82% of folks saying that testing is slowing down their ability to release. That's a pretty staggering figure. And it's hard to ignore, and it's certainly one that needs to be addressed.

Often this can be attributable to teams that maybe are still leaning on a lot of manual processes, where maybe effective automation is not yet in place, or effective tooling is not in place for test management, functional, or performance testing. But even in instances where those tools are in place, it takes time to really start building an effective and scalable test framework to support all of those complexities. And further, skilled resources are required to get there, and those resources are limited. Their time is valuable.

As we start to see more AI-driven development come into play, this perception and experience that we talked about on the testing side may end up being felt even more. And that's something we want to make sure we're getting ahead of the curve on.

AI's current uptake in DevOps

To that point, I think there's some interesting information here that I want to cover. This was taken from a recent report we worked on with TechStrong Research about the use of AI in DevOps today. I want to give you some context, because I'm going to cite it a couple of times. We had over 500 respondents — globally, all DevOps professionals, ranging from practitioners through various levels of management, in a few different industries and across organizations of different sizes. The report was really aiming to try to understand: what are AI's positive impacts in our industry today, and what are some of the challenges that we're still facing?

What's kind of interesting, as I was doing a little bit of research on this, some of the results we saw from this report are very consistent with what other people are finding as well. What this is showing is that many organizations are already using AI in their development activities today. And this highlights the areas in which those teams are leveraging it most prominently. You can see that the majority of where it's being used is on the development side of the house. And perhaps just as important, we're expecting to see substantially more organizations begin to make use of AI-augmented DevOps tools in the very near term.

From that same study, we found that those teams that are using AI in their development activities are reporting that AI has boosted their productivity — 60% are showing this. If you contrast that with a similar study we did in 2022, folks anticipated that productivity and efficiency gains actually had a lower expectation than what ended up being realized just two years later. If you imagine 60% of your workforce being more productive and more efficient in their work, to some degree or capacity, that's not insignificant. And as AI continues to evolve, and as the capabilities of AI-augmented tools continue to advance, I think we can pretty safely assume that those productivity gains and efficiency gains are also going to continue to increase.

While Agile and DevOps have already done wonders for developer and release efficiency — even predating the AI stuff — it's also done wonders for cultivating a more integrated and continuous approach to quality, where quality is more of an embedded function, where more people and more teams are responsible for ensuring quality, allowing testing to be done earlier and allowing everything to move faster with higher degrees of quality being achieved. But that's in cases where testing is already being done efficiently with appropriate tooling in the hands of skilled resources. Even for those more advanced teams, there isn't more time in sprints. Testing teams' budgets aren't always getting bigger. The IT and application landscape is only becoming increasingly more complex. Demands on testing are higher than ever. And as development and releases speed up, testing teams can struggle to speed up as well. We're on the precipice of really even greater acceleration in this side of the house as AI is becoming more adopted, like we just saw.

This is really set to take Agile and DevOps to the next level, and testing teams need to be prepared to stay ahead of the curve to keep pace. Faster, better, and more testing is needed to ensure application quality. If we look at this graphic — and this is something we're all very familiar with — all these areas, this is dependent on all these things working in harmony. As we start to see AI used to enhance and accelerate development, for this to stay in harmony testing needs to stay in step to avoid that bottleneck — or that perceived bottleneck we talked about — becoming even more magnified or exacerbated.

So this is really about trying to remove that point where we're pointing at testing as being the point of most friction when we're trying to balance speed and quality. From that same study, 60% of DevOps professionals find testing to be the most valuable area of AI investment moving forward — because testing is such a critical activity, and if developer productivity is going up because of AI, it makes sense that testing needs to respond in tandem to keep pace.

A layered view of AI: machine learning, generative AI, autonomous AI

So now I want to start taking a look at the actual AI side of things and where AI is already starting to help testing and where we see it being able to help testing in the future. Right now, when people are thinking about AI, they're thinking about generative AI and things like ChatGPT, because that's what's been getting a lot of interest and attention recently. But it goes beyond that. There's a lot more to the AI story and different types of AI and machine learning that fit into the broader scope of AI capabilities. And it's not just about using one piece — it's about bringing all these pieces together and meshing their unique or specialized capabilities to really start helping drive efficiencies across the testing lifecycle. It's about bringing them together in a way that's easy to use, easy to interact with, and is getting us the results that we're looking for quickly. These pieces are working together to make sure that we're getting competent answers to the questions we're asking, and that we're getting results from the actions we need it to take.

The change that we're seeing more recently is that, from a front-line perspective, as we get into these LLMs and generative AI solutions, we don't have to go in and program the question into a specific piece of AI to get the output we need. We're able to ask the question, and it's then able to go out and use judgment to figure out where to route that question or instruction, to get you, as the user, what you need faster using all the power of these things that are sitting underneath. So the key to where we're at today, and where we're going from an AI perspective in testing, is combining all of these elements: combining machine learning, deep learning, and generative AI to be able to assist in managing, designing, creating, executing, maintaining, and analyzing tests.

Layer 1: Narrow AI / machine learning in testing

If we start with narrow AI or machine learning as a piece of the overall picture and in the context of testing, these are some areas we've already seen folks be able to start using very effectively to drive some efficiencies in their processes.

Vision-based recognition (vision AI). It's able to start detecting UX patterns, objects, or components, and take that and start making decisions based on the elements and controls just by looking at the actual UX experience, without needing to go into the back end. So this is the technology — not the user — doing this: using visual recognition and interacting with visual elements of a user interface rather than relying on code or text-based identifiers for testing. This helps capture things to build automation and scenarios where traditional automation methods might struggle, or people might struggle to manage.

Self-healing capabilities. This is where we get into when things move on the UI and web pages. It's able to understand and recognize those changes. So if, for example, the button on a webpage moves from the left to the right side of the page, it recognizes that the function is remaining the same, but just the location has changed. It's understanding that maybe things have changed with a slight change of code, but the important part is remaining the same. And this allows for the automated recovery and updating of your test cases in those instances to make sure that test doesn't break due to UI change when the function is remaining the same. We see a lot of gains in this area, in overall test maintenance.

Impact analysis / risk AI. This is another really powerful use case. It's able to recognize and map the impact analysis of code and data changes to your application — how much will a change in one place impact things over here? It's providing highly accurate insights on where you actually need to focus your testing efforts based on risk, based on impact of those changes. So rather than doing unnecessary amounts of testing just to be on the safe side when changes are introduced, you're able to cut down on test time and test effort substantially for a more intelligent approach to testing. A lot of these use cases have made their way over into the mobile space as well.

Layer 2: Generative AI in testing

Now as we get into the next layer — which is where we are today — this is the generative AI piece. Here, we're able to start to do even more things, and people are doing this today.

Test creation. Allowing users to leverage AI capabilities to create tests based on requirements or context that's being provided. The output can be very detailed test cases that are generated very quickly with high degrees of reliability.

Test portfolio optimization. Using generative AI to review test case libraries, identify redundancies, identify where tests and test cases can really start to be streamlined.

Intelligent assistance. We've all interacted with this at this point in some capacity or another. But what we want to start having folks do now is, rather than going to outside sources, we want them to start doing this within the tools that our teams are already using, within the processes we're already using. Here the value is we can start to query based on the information being fed into the AI from these tools and from the records associated with these tools, allowing users to get very accurate and complete answers more quickly — scouring documentation and data in the system and knowledge-base articles to immediately put answers in front of users. The main thing is it's really speeding up productivity. It's reducing the amount of time that your subject-matter experts have to spend helping or assisting maybe less technically proficient users with certain activities. It's reducing the amount of time you have to go out and log support tickets and wait to go through formal support ticketing processes. It's even taking things that may have been very complex and hard to understand or difficult to track down without certain levels of technical proficiency, and making them available and understandable very quickly to users at all levels of expertise. So that not only increases productivity, but really increases the pool of resources that can start contributing to testing and quality in much more meaningful ways than ever before.

App exploration. Capabilities that are validating what exists between things like requirements and test cases — generating those test scenarios and making sure that the testing being designed and the testing that's occurring actually makes sense. Not only helping to build and generate more efficient tests, but to help ensure that the right things are being tested and tested reliably.

Layer 3: Autonomous AI — the near-future holy grail

We get up to the last layer here, and this is where we're getting up to maybe what was the future not that long ago, but really is the very near present — autonomous AI, and kind of the holy grail of testing: autonomous testing. So with everything we just discussed, you combine machine learning and generative AI, and that takes us into the realm of autonomous.

Here, we're not relying solely on specific data that's being provided as a direct input — for example, like what I referenced when we're talking about providing requirements that are already drafted. What's different is the AI is leveraging all these different inputs and sources to make intelligent context-based decisions based on existing data that it's looking through, or documentation or historical information on applications under test. Not just take what's being directly fed in, but where it's starting to be able to more intuitively translate things like business intent into actual test cases. So we're removing some effort and it's able to start taking on more and more itself.

We then get into the realm of being able to get into autonomous testing — where we're taking these rapidly generated and rapidly developed tests, and creating and executing automated regression and progressive testing from a given intent.

Three pillars: Discovery & Design → Generation → Execution

To expand on everything I just talked about: autonomous testing and autonomous AI is where we're not only looking into auto-generating the test cases and auto-generating the data, but actually driving the automation using AI itself across all the phases of the software testing process.

We've put three pillars together that are part of that process:

Pillar 1: Discovery and design. This comes down to understanding what needs to be tested. What's the system under test? What are the business requirements? What are the user requirements? Different scenarios that need to be tested. And how do we ensure that we're covering all cases and variations? To automate that process, we can start leveraging machine learning along with LLMs to be able to analyze and extract relevant information from unstructured and structured documentation — Word or PDF docs, user stories, even getting into things like freeform text or images or videos.

Pillar 2: Generation. Auto-generating test cases and automating them. Generating synthetic or realistic data. Exploring new paths through the systems. Automating test instructions from discovered processes and requirements.

Pillar 3: Execution at scale. How do you execute tests at scale? This is a challenge for a lot of folks. How do you run only the tests that matter? Where have the changes been made? How can you get intelligent insights and analytics to help you debug applications? Where do we start getting into the self-healing side of things — not just for the testing, but also autonomously healing the code itself, diagnosing errors swiftly so we can start shipping applications more quickly? Because that's what it's all about: marrying speed with quality.

These are real, actual technical challenges that are difficult to solve, even with tooling in place. But for this, what we want to start doing is integrate machine learning and LLMs and generative AI — all the things we discussed in that general AI overview — and we can start to look at these problems in new ways. We can start to address these challenges in new ways that are much faster and much more efficient, where even less technically proficient users are able to handle tasks and activities that would have been much more complex not that long ago.

Brain, eyes, hands

A way to conceptualize this that I kind of like, that was explained to me once: you can think of the LLM as the brains of the automation, and some of those vision capabilities and impact-analysis capabilities and healing capabilities as the eyes of the system. And then your testing tools powered by AI become the hands that start moving things around and making things happen. That's the mechanism for your executions. These are the main things we need. We need the testing tools and the automation in place, and then we can start building on that. We can embed these various aspects of AI-based technology into the tools we're using, into our testing lifecycle, to enhance the way we test — to be able to deliver smarter testing much faster.

All of these are areas of AI we've discussed. These are all things people are able to do today. But this next step of AI-based autonomous testing is exciting to start thinking about and planning for and building strategies around, because it really has the potential to transform software testing from what has historically sometimes been a labor-intensive and time-consuming process into something that's really highly efficient and intelligent and scalable and reliable.

Why this is urgent

All this is to say that AI is here today and it's already being leveraged. This is not a future vision anymore. AI and AI-augmented tools are being used today already, and it needs to be incorporated into testing and quality in order for testing to keep pace. That can be done and is being done with a lot of the features and functionalities and capabilities we've already shown. This is how people are mitigating some of those testing bottlenecks. If you're not doing it now — if you're not bringing this stuff into how you're doing testing right now — you can be sure that your competitors already are, or they're already looking to evolve their approaches and strategies around this.

If we harken back to one of the earlier slides where I highlighted the productivity gains on the developer side of the house, that was kind of only half the story. We see similar results being reported on the testing and QA side as well. 42% of folks are saying that AI is improving their productivity on the testing and quality side of the house. We see the same sort of results: people anticipating less just two years ago than what they're actually achieving today. One thing I do want to note: we do see higher productivity gains being reported on the development side, and that's something we want to keep our eye on as we look to close that gap on those potential testing bottlenecks.

With the continued adoption and advancement of innovation, organizations that are strategically starting to adopt these tools and technology into their testing processes today — I think this is kind of a no-brainer — they're going to be in a much better position much faster than those who do not. We're at a very important place right now where we need to start looking at staying ahead of the curve. For folks that tend to be more on the conservative side of technology adoption, the rate at which the delta between some of those more conservative organizations and some of the more tech-forward organizations grows is likely going to be increasing very rapidly as all this innovation starts happening more rapidly. And there's a lot of really great results that have already been realized and that are going to be able to be realized in the months and years to come in this space.

Tricentis copilots

If I can just do a little shameless self-promotion before I wrap things up: at Tricentis, innovation is nothing new to us. AI is nothing new to us. We've been delivering AI and machine learning capabilities and features into our testing tools portfolio for a number of years at this point. That continues today as we're making significant investments into the generative AI space, delivering copilots for some of our key offerings.

That started with Testim, which is our low-code test automation tool for custom, web, mobile, and Salesforce applications. We have a copilot there to help auto-generate customized coded steps, explain selected code, in addition to helping to fix and debug test issues faster.

The next one we rolled it out to was for Tosca, our model-based end-to-end automation tool, which also has a copilot to help optimize and drive more efficiency in our functional test automation for end-to-end testing.

In beta right now, we're starting to cover our test management capabilities and bringing generative AI into that part of the picture — with a beta for qTest that's going to be very soon available to the public for use in our test management and analytics and centralizing of testing efforts side of things.

Next on the roadmap is where we start to get into our performance testing side of the house and delivering a copilot for NeoLoad to drive more optimized and more efficient performance test design, but also the ability to unravel what can be very complex performance test result analysis as well.

I know we're basically out of time here. So what I will encourage you to do — we have a lot of folks on site with us, a lot of folks who are a lot smarter than me, over at the booth right now. If you have questions about what our AI strategies are or how you can start to build and advance your own AI strategies, I'd recommend you come by booth 403, talk to me, talk to some of those resources. We're also doing a demo on Thursday at 1:10 where we're going to be showing Testim Copilot in action. So I'd recommend you stop by and see that as well. I know that's all that's stopping you from lunch — so I'll just say thanks for joining, and enjoy the rest of the conference. Thank you.