Ryan Burlage

So you're probably wondering, "Who are these weirdos up here?" Myself, I'm Ryan. I'm a solutions architect with Tricentis. I've been there for about four years. Prior to this, I was a global QA manager for an asset management company. So then I joined the ranks here, and my colleague is with me today.

Scott Erlanger

Hi, I'm Scott Erlanger. I'm also with Tricentis, where I'm a director of product marketing, specifically for our DevOps products. Prior to working here, I worked in a bunch of different places. I worked in the DevOps space for a while, previously in microprocessors, all kinds of different industries as well, and I'm looking forward to talking to everybody here today.

The world's changing quite rapidly, and it's just amazing what's happening in today's world. Applications are growing in a number of complexities, from mobile apps. There are over nine million apps in the App Store today. We have 24,000 different models of Android phones out there in the world today too. Combine that with web technology and then other third-party applications as well. The interactions between those different technologies and being able to work seamlessly is really important in today's world.

Yeah, absolutely. Just putting some numbers to this as well: even though there are this many types of interactions out there, there's still an expectation of a really positive experience from the users. Looking at some of these statistics right here, you'll see that if a user doesn't have a positive experience, it's too buggy, or it takes up too much space from your application, 50% of them are going to actually delete it. And 90% of users also consider what the performance of an application is going to be before they decide to purchase it. And as well, 77% consider user return to be due to user experience as well.

That graph on the right there is actually particularly interesting as well. It shows, as the delay from application increases from three to five to six to 10, the bounce rate, so people starting to leave, also will increase. When it's just three seconds, you have a 32% increase in the number of users that decide to bounce. When it goes up to 10 seconds, you have a 123% increase. So you can see directly that when users don't have a positive experience, they don't stick around very long. They're going to try to find an application that serves their needs better.

And the thing to mention too is it's not just you in one place. Applications have to work everywhere. You take your phone to a conference like we're at today, you take it to a baseball stadium, you take it on the subway. These are different types of experiences and different types of needs, and it has to work in all these different places there. So it really does create a wide space that you have to test with any kind of application, either web, mobile, or others out there, for it to work correctly.

And as I mentioned before, there's real-world impact. We talked about user churn, but there's also loss of trust. If people can't rely on your product and your applications, they're not going to think very highly of your brand, and they're not going to think about your company when it comes time to purchase their next product as well.

And the other thing that's really important too is that when you have a lot of bugs in your application, it's more time for your team to go and diagnose those, to fix the bugs, to test out new fixes, to try them out with your users, and hopefully you get it right the first time as well. So if your applications are released without high quality, it really impacts your business in lots of different ways.

Ryan Burlage

Yeah, each of those stages are progressing as you're developing those applications. And if you don't catch those bugs early enough and it reaches production, the more it's going to cost the company, the more it's going to cost you.

Scott Erlanger

So this situation is probably familiar for people in this audience here. Look at the stack of papers there on the left. What ends up happening is that people test out their applications and their browsers and other types of software manually. Many testing cases effort, entire teams and developers, QA, everybody basically go and pick up the application and try it out. It's time, and the more things you have to try out, the more time it actually takes.

Well, eventually, you find that you may not have enough people to test out everything manually. So what are the most important things to test out? Everything might be just enough. What's happening is that work piles up release after release after release, and sprint after sprint after sprint, and it becomes hard to really catch up and make sure that you have high quality on everything here.

Does this sound familiar to people here in the audience?

But just a little more graphically here, because that's really an interesting one as well. Debugging, maintenance, team productivity. If you look from the left there to the right there, this is showing that subsequent releases or subsequent sprints, you start off with maintenance being sort of just a subsection of the different pieces, from development to planning, that all contribute towards an application.

Well, this little window gets bigger in the next one, and bigger and bigger and bigger. We're on the right side there, and most of the time is being spent in any given item on maintenance there. So it ends up taking up more and more. And of course, the number of hours in the day doesn't increase over time either. So the height of these stays the same, and you have less time to do the development, the planning, deliver the great features that your users are looking for in their applications. And this obviously isn't sustainable, since the main goal is to deliver applications and deliver great experiences to customers.

Ryan Burlage

And I like to call this a slow progression. It's like putting a frog in a cold pot of water and slowly turning that heat up. The frog's not going to jump out until it's too late. And then you just experience that maintenance nightmare that you have to kind of go back and look at how you can resolve that from that perspective.

Scott Erlanger

Absolutely.

Ryan Burlage

So yeah, a revolution has taken place with that. We started out with manual testing.

Twenty years ago, I entered the QA world of testing. You wouldn't realize it, but I'm actually kind of old up here. So back then when I was testing, it felt like caveman testing. I worked for a software and hardware company, so we were literally throwing stuff off the building to break it. We were pounding keys and buttons and stuff. It just was really kind of like the start of the evolution of testing back then.

So then what we realized when we were doing all that testing, we just weren't fast enough. The quality wasn't there because we're prone to human error. We just didn't have it there. It required a lot of effort too as well. Sixty to 80% of the testing is still done manually today.

And just to poll all of you out here, is everybody still kind of doing some manual testing out there, or mostly manual testing? Right. It's kind of painful a little bit, right?

So with that, the evolution turned up. We started to scale up. So that's when we started doing offshored testing. Back then, my colleagues were all worried about losing their jobs because we're shipping everything offshore to testing. But that wasn't the reality. The reality was that we're doing more releases, more tests. We needed more people to do it.

So it was easy for us to start doing things offshore, and I went over to India for a month to help train and onboard people and do all that. So when I got back, it actually helped kind of relieve us from some of that mundane testing that we were doing on the newer stuff and kind of look at automation at that time.

Scott Erlanger

The manual previous graph showed it just increased. It's increased until it reaches a point, with the frog example there, and you can see it just isn't enough. The same old way won't work.

And the thing to mention here too is that there's also market pressure to move faster. Competition is getting products out the door faster. You need to have the same features that are competitive, to be able to compete in the marketplace there.

But as you talked about before, this pressure can lead to the ability to choose which quality is important to get out the door, which test was important. And there's an impact when you have to make decisions like that. One thing to note here, and consistent, is 54% of companies have a hard time keeping up with customer demands. So they have to pick and choose which ones are really the most important there, or choose those things they're not going to include.

And the other thing to mention too is that when people try to get things out the door quickly, quality can suffer. Bugs in production, which are obviously more expensive than finding them prior to actually going out into production, lead to a worse user experience and a worse product being sprayed out the door.

And finally, one thing is that 49% of companies have lost revenue due to customer experience. That really impacts the bottom line there as well. Without a doubt, being able to test and provide user experience has a measurable value for the customer.

And there's an interesting thing just to talk a bit about here as well. We'll obviously be talking a lot about Agile and DevOps here at this conference. Both of them sort of have the goals of increasing efficiency and productivity amongst teams. Agile, with the development team being able to do releases and sprints more quickly, more features out the door, teams doing agile practices. And DevOps, being able to get these releases into production more quickly, through a better methodology and higher-quality releases there too.

But the thing that people may not talk a lot about is that testing has to keep up with both of them as well. As you develop software faster, and as you release it more quickly and more consistently with better processes, you also need to be able to test those at the same time. And testing hasn't always kept up. So you have your fast-moving development, and you have your fast-moving business processes, but you have the same old testing in the middle there, and it just simply falls behind as these ones begin to move faster and faster. Something else is really clearly needed here to be able to keep up with these new Agile and DevOps processes.

Ryan Burlage

And one thing that I think is important is the visibility as well. Back in my caveman days, when we were sitting there pounding on the machines and pounding on the keyboards, we really didn't have that visibility that we do have today: agile best practices and stuff, and really seeing what we're testing and how we're testing it. So that's a really great advancement as well too.

Scott Erlanger

Absolutely. One of the things that you think about, this is about the survey that we had actually gathered some research on as well, is how often does the need to test to achieve user experience quality slow down the release of an application, slower time to reach the market as well?

If you just look at combined, the always, frequently, and the sometimes, it's up to 82% of companies have had some degree of impact of testing impacting their overall release time. Only 18% have said never.

Let me ask the audience here: has testing never impacted the ability to get software out the door? I think that seems fairly consistent with what the slide shows.

And the question is, why is it slowing us down? The thing I mentioned as well is that testing can't keep up with features, as we've already talked about. It can't keep up with changes from other teams. And ultimately, there's a growing list of testing expectations. There's just more and more that has to be tested, and the teams just aren't able to scale to meet that in many cases there.

Ryan Burlage

Yeah. And so we realized that we're running into a bottleneck with things. Organizations are looking at how we can resolve this testing bottleneck. And testing bottleneck has been around for years. Every time I look at a Gartner report or something like that, the number-one thing is testing. It's why slow.

Understanding the pain points and really being honest about what those pain points are, and acknowledging those, and then really attacking them from an org structure. Somebody made a note that I call it the three Ps: people, process, and programs, or technology. So you have to bring all that on board when you're doing this big change and you're trying to address this testing bottleneck. We all want to deliver better products, absolutely.

So the evolution of testing is changing. Right now we're going into automation. We're doing test automation that's helping us kind of remove that human error. This is where manual testing still didn't cut it. So we're looking at automating, removing those human error points.

So the automation cycle still looks the same. We're still doing test authoring, we're still doing test execution, but now it's all done by machine. So we can focus on the things that we need to focus on while we let the execution run, and get that faster feedback from a CI/CD pipeline: increased application quality, improved productivity as well, and continuous improvement. That's a big one that I'm always a fan of.

Just get that faster feedback, working with the development team, making sure that they're developing correctly as well too, to help with the test automation. Because it's not just we can't work siloed. Development needs to work in a way where they're automating things, or developing things in an automation way, so we can automate around that development as well.

Scott Erlanger

Absolutely. Even as you try test automation, people often run into some key challenges that hold it back. The intentions are good, but they run into some issues there.

One of the things we talk a bit about is integrating with a DevOps core team. Obviously testing as part of your continuous integration, as part of your releases, is important, but there's obviously work to be done to integrate, to make sure that it works automatically as part of all flows there. And this is something that takes time, that takes knowledge in that space. So it's not always just testing. It's how it's part of the overall infrastructure as well.

Coordinating manual and automated flows. It's not like a light. You go from being manual to being automated. You pick your flows to automate, and having to coordinate those with the manual ones that are still in process, making sure that the right gates are in place, that the right oversight is there with the automation, the combination of the manual, is something that takes some time and effort as well.

And similarly, what are the right places to start off with automation? What's going to provide the biggest value to your team? You need to know sort of what are the bottlenecks, like what's holding back the process there, so you know what's going to give you the best benefit across your overall process.

Finally too, one team typically might start off with one team trying out test automation. You'll build it across the entire organization, but then building it across the entire organization, making sure that each of the teams who have different people on the teams and different problems will be able to use these best practices effectively.

And then finally, this process is continuously improving. You have to take feedback down across the organization so everybody can learn these best practices and continue to improve the overall cycle of test automation and efficiency.

Ryan Burlage

So this leads us into our next slide: Don't let DevOps become devs. What does that mean? There's a lot of people that have different terms of what DevOps is. There's DevOps, what it is to them, or it's the buzzword. So the company wants to do it because it's the big buzzword, and we need to be DevOps. Or they just want to do speed. It's the thing to do, so let's just try to do it.

Those are all going to be ingredients for failure. So you really have to understand what you're looking at. You have to look at your ecosystem. What are your pain points? You want to make sure it's also automatable across. You want to reduce that manual intervention and kind of analyze things. So if you've got a lot of manual tasks, make sure those will get automated. Take that workload off of the people so they can focus on and get back to doing development or testing the things that they need to do, and look at that from a whole ecosystem standpoint.

Scott Erlanger

Looking at this picture here, it looks complicated. There's a lot of different tools that are on here across the different stages of your development lifecycle, but it's not really that complicated. I'm sure many of the people here have at least as many tools and then some tools across as many buckets as there. It takes this complicated understanding of how the different pieces can come together and how to build something that delivers the efficiency is really important.

Ryan Burlage

So here's some great real-world results that we've had with our customers. Cutting their testing time down from 50 days down to 13 days. That's pretty impressive from a delivery standpoint.

And Vodafone, one of their achievements actually was delivering one of the five largest SAP S/4HANA migrations in just 10 months. I'm sure people have been through migration, and it can take a lot longer than 10 months. So being able to achieve this was a remarkable achievement by those ones.

And then McKesson: biweekly down to daily releases. That is really cool, being able to release that quick, that fast, on a daily response.

So now we're at AI. We're seeing ChatGPT coming into the picture now, and now what can we do with AI when it comes to testing?

Scott Erlanger

Absolutely. Interestingly enough, AI is impacting many different areas of testing, and we're going to walk you through some of them and talk about the ways that AI really not just improving, because everybody's starting, and if you're a, because your technological advantages. So there are different categories.

First, what we're going to call machine learning, or narrow AI in this particular case. One particular use of this is the vision. It can look at the different elements on a page or an application. It can tell what's changed, and it can use this information automatically to provide feedback to what went wrong with the application.

Deep learning is an interesting one too. Many people may not think about this, but as you know, applications change, tests often break. What deep learning is able to do is detect what exactly changed and really update the tests and make sure they continue to work even when the application changes. And it really saves maintenance time across the board. So production productivity across the board there for that.

The other thing too is risk. Being able to determine risk ahead of time, before it could cause a problem, is another really big one there as well, that AI can kick in and help provide a better user experience and less risk in field.

Ryan Burlage

Yeah. And one thing I really like about the risk AI is really reducing your testing efforts as well too. A lot of people will see, "Hey, we have this big release. We need to test everything." Well, that's not necessarily the case with the risk AI. It can analyze all the areas that are impacted in a way where you just need to focus on that subset instead of the complete picture. And what we're seeing is it's reducing 80% of your testing, allowing you to release much faster, test faster, and release faster as well.

Scott Erlanger

Absolutely. But as I said before, this is just sort of the foundation. AI is generative AI. Many of you have used ChatGPT for all sorts of purposes here, but test creation is one potential area there where you could put in a description using English language and it spits out your test cases for you automatically. This is real, and this is actually happening as well.

Intelligent assistance: you don't necessarily need to go to customer support if you have that that can answer basic questions for you and improve productivity there as well.

And one of the biggest categories, the biggest areas for improvement, is what we're calling autonomous AI. Autonomous, picture the dream here. We're able to actually load an application. It's able to figure out about it, able to test content based on loading that application there. And this is really the general direction that autonomous testing is going. Businesses' higher-level sort of thoughts there, and load that in and create test content based on that, and automate the regression, have everything sort of test autonomously.

It's a productivity improvement. It's also going to be able to make sure that the test coverage that's needed is generated by the testing process.

Ryan Burlage

Yeah, that's pretty exciting, what the future holds for AI in the testing field.

Scott Erlanger

Oh, for certain.

Ryan Burlage

So the revolution's never going to stop. Technology's going to separate the leaders who deliver great user experience from the laggards who struggle to keep up. And really what companies are faced with today are two options: do something or do nothing. Do nothing's going to come at a cost.

Scott Erlanger

Absolutely. And the costs that we've already talked about before, some of these you see here: brand reputation, user satisfaction, and ultimately your bottom line.

Ryan Burlage

Yeah. If you're not part of the revolution, your competition will be.

Scott Erlanger

And a little bit about us, just so you know who we're here. We're Tricentis, and what we do is we create software for testing applications. The thing is that there's so many different areas to test in terms of the application development and delivery process there, and you really need lots of different types of testing solutions to cover this entire space. There's not really one solution that covers everything.

You can tell sort of the number that are on this particular chart here, that there are many different areas and many different types of problems that people face, ranging from test management to how you track your test cases, on to performance testing. Even functional is not just enough. Again, you need to make sure that you don't have that lag that causes your users to churn, and solutions are needed to that as well, as well as automation and authoring as well. Again, there are lots of different solutions that can manage all your different testing needs.

Ryan Burlage

And from an enterprise QA solution, we want to encompass all that to allow you to report, have that visibility, and report everything that you're doing, because that's really important. You have to be able to make decisions based off of what you're seeing and happening.

Scott Erlanger

Absolutely. So we'd like to thank you today for attending this talk and for learning a little bit about test automation and AI. We're going to be on the floor as well at our booth. We'd love to chat with you about any solutions, about your testing needs as well. We're getting a live demo over there also.

And the other thing too is that at 12:35 today, Ryan and actually one of our other representatives is going to be doing a demo as part of the Solutions Hub. Particularly with the demos that cover is how do you actually create performance tests, make sure that your application is performing from your existing functional tests, not having to generate two sets of tests.