Marc Hornbeek

Hello, everybody. I'm aware that I'm between you and lunch, so I'll try to go quickly, but try to get some points across nevertheless.

My name is Marc Hornbeek. I work at Spirent Communications. Spirent is a test and measurement company, and we build test and measurement systems, tools, and solutions for networks primarily, or situations that involve complex test situations involving multiple nodes and interconnected things.

Many of the principles of these complex tests are taken to the limit in a DevOps environment where the testing cycles have to be very fast. And in fact, I think many of the principles and practices can help with less complex test environments also. So I'll be talking about that.

In my role as a senior solutions architect, I get involved in a lot of customer engagements and helping customers trying to understand what they need to do in order to improve their testing, especially automated testing and automated lab applications, and more and more in the DevOps world.

The question that I'm often confronted with at the beginning of these discussions is, "Why am I not getting the wonderful results that everybody talks about in DevOps? My DevOps doesn't work."

And the root cause frequently, maybe it's because they're talking to us, I'm not sure, but the root cause is frequently that they haven't really got the testing part of DevOps right.

Testing seems to be the undersung hero of DevOps in some ways, in my opinion anyway. There are a lot of nuances associated with testing. It's not just a matter of running a test after you make a build and then deploy it. As many of you, I'm sure, know, there's testing aspects throughout the pipeline from design all the way through to deployment. And even after deployment, there's interesting information that can be gleaned that can support the testing practices in a DevOps pipeline.

My career has taken me over into many organizations, all the way from startups. In fact, I started a couple of firms myself, and also large multinational companies, mostly in the domain of networks, but also in other domains as well.

So I'm going to show you what I have learned to be characteristics or practices that most relate to what I see to be successful testing, continuous testing practices for DevOps environments, as well as underscore some of the issues to be aware of and avoid.

So without further ado, next slide.

Oh, by the way, I use Aston Martin as a theme in the presentation. My company, we're a sponsor of Aston Martin Racing, but also it happens to be a good metaphor for speed and acceleration. So hopefully it'll make it a little more interesting.

The point is there's really not any continuous delivery if you don't have your testing right. There's no way you're going to have a development complete and integration either if you don't have a great way to make sure that the development is ready to be entered into an integration pipeline and a DevOps pipeline.

And on the other end of the scale, I'm sure some of you have been involved in a situation where there was not enough testing before a deployment and bad things happened. So again, testing is also important at the other end, but also all the way through the pipeline. And that's a theme you'll hear throughout this presentation, is that for goodness' sake, think about testing as part of a continuous pipeline. It's not just a phase of DevOps. It's really an integral part of DevOps from the beginning to the end.

Okay, so DevOps, it's all about speed, making processes faster, delivering faster, and all the benefits that are accomplished from that. But how do you do that?

In some ways, testing fast is almost an oxymoron, right? You want to have comprehensive testing, thorough testing, high-coverage testing, testing that makes sure it works in all high-performance capacity situations, reliability testing, and all of those tests that go along with a good, solid product.

So it's not as simple as it sounds. "Well, just test faster."

Right. How?

If you have a complex product, certainly many of the products I'm dealing with, they literally have millions of test possibilities. They're not all important. But how do you sort that out, and how do you know which test to run at what point in time in the DevOps pipeline to be optimized so that the whole pipeline is optimized and that you ultimately end up with a quality result despite the millions of possibilities?

So I'll be covering some of that.

And of course, running fast, if you run your car fast or anything else fast for a long time, as we want our continuous processes to run continuously, eventually they break. As we all know, you can't keep running them fast without something breaking.

So the point is that the environment itself, the test environment, needs to be fault-tolerant and self-healing and auto-recovering and all of those cool things. It's not good enough or else you're going to have a lot of rollbacks and wasted time in the whole DevOps process. So make sure that your test environment itself, all the servers and all the processes in that, have sufficient infrastructure to really be a reliable infrastructure.

I'm sure many of you had the situation where you analyze problems that are resulting from testing, and after so many hours find out, "Well, wait a minute, it had nothing to do with the purpose of the test or the product. It was some infrastructure problem." It was a false negative. What a waste.

So again, the test environment needs to take care of that so that you can distinguish the infrastructure issues from the actual purpose of the test. Similarly, on the false positive side, you don't want test escapes getting through to the customer. So it needs to be a tight mesh, but not too tight, in other words. So that's part of the nuances of best practices of continuous testing.

Okay, getting it wrong has catastrophic results. Certainly, we've heard from some of the other speakers lots of situations where when DevOps went wrong, oh my goodness, or at least when the development process went wrong, there were major failures.

That's true not only once you deploy the product, but even throughout the pipeline, if you have, as I mentioned, situations where you're reporting a lot of false positives or negatives back to development and wasting their time. That's kind of a catastrophe, too. Or if the test infrastructure fails and you're not able to complete a cycle, then you're interrupting a lot of developers and people involved in the pipeline, and that's also a catastrophe.

At the root cause of many of these problems, there was an interesting report I tend to quote from IDC last year, but it basically said that a lot of folks are trying to hang on to their old tool chains or their old tools, and they really aren't necessarily the right ones for DevOps.

So the key is to really be open to looking at new tools that are truly DevOps-ready, or at least looking at ways to make them more DevOps-ready if you really need to hang on to them. Because that was a startling statistic. Eighty percent of the DevOps deployments report, at least in that report, said that using current tools is often the source of a problem in their DevOps environment.

And I have seen that myself, where an organization is just hanging on to some particular test tool that they just have a religion about, or Mary created it 10 years ago and we're going to keep it, that sort of thing. That may not be the right solution. So something to be aware of.

Okay, so I'm about to talk about these seven magical best practices, but before I do, there are two principles underlying all of these practices.

It's one thing to test fast and have a powerful test environment that can spin up many tests in parallel and get to results quickly. I was involved in one project where we were able to get 25,000 test cases overnight, and that was great. Lots of parties about that, because previously it took them weeks to do that.

Unfortunately, they got 25,000 test results the next morning, and it took two weeks to analyze the results. So the party only lasted so long.

Anyway, the thing is that the other principle is relevancy. You want to make sure that your test results are relevant, that your tests are relevant, that you're reporting the relevant results to the relevant people at the relevant time, because otherwise you're probably wasting a lot of time. So that's a key principle. It's not just a matter about running fast.

We hear that continuous testing is all about shifting left testing or fail early. These are words that the DevOps community uses. Yes, but again, testing is throughout the pipeline. What you're trying to do is to get the most relevant results as early as possible in the pipeline, but testing is everywhere.

I would argue that testing is a key element in any successful DevOps deployment. And every phase in the process, whether it's pre-flight before integration, needs thorough testing, and obviously throughout the DevOps process from commit, build, code testing, regression, system, and delivery. And whatever your process is, testing is there, and you really need to have a strategy that covers explicitly every one of those phases. It's not just a matter of doing a test after a build is done. That's for sure, right?

And also the infrastructure of the pipeline. There's different layers, but if you look at it this way: monitoring and management as a top layer. Again, are you really making full use of all of the test results that are extracted from the entire set of stages? And looking at things like trend analysis, and what can one stage learn from the next stage so that the next stage can work more optimally from a test point of view, and things like that.

Yeah, testing has got to be automated as much as possible. And the orchestration tools associated with automating DevOps and the orchestration tools associated with testing, they might be the same or they might be different, but basically, they do need to be connected and efficiently connected so that at every one of those phases, you have a really smooth integration and none of them are requiring manual triggers. Or if they require triggers, even the thresholds are automated so that you know which test to trigger next, depending on the results of the prior stage. As much as possible, automated.

We talk about elastic infrastructures, obviously with virtualized or non-virtualized in many environments. Certainly in the world I'm in, in the network world, there are a lot of times when you really need physical resources and lots of them to test the performance of network elements. So all of that has to be elastic, and it's not just elastic for a system test, but elastic for every one of those stages. So you've got multiple combinations of test environments happening throughout the chain, and probably multiple of those in parallel, depending on which release you're talking about.

So what are the practices? There's seven. There's no magic number in seven, it's just a convenient one to talk about. There's plenty of others if you really want to know, but these are the ones that I think are important.

Certainly culture is part of DevOps, but in particular, culture for testing I'll be talking about because there are a lot of concerns in the QA communities about what happens to QA when you accelerate testing. Does QA go away? Does QA become more important? How does that all work? And how does that affect my job as a tester?

And also the flip side is how does it affect my job as a developer? Do I have to do more testing? Oh, my goodness. And so on.

Then structure. How do you make sure that the underlying structure is continuous testing best-practices ready? Tools, of course. But there are certain attributes for tools. What are the attributes that make them best for continuous testing? And integration, what attributes for integrating testing and test tools.

If you have a complex test situation like a network or something that has networked applications or complex topologies, then for goodness' sake, your tools ought to be topology-aware all from the get-go. From creation through to execution and analysis, all of that needs to be topology-aware, and I'll talk about that.

Of course, test execution and the practices for that, and analysis. So as I say, there are others, but those are, I believe, the key ones that I often run into when we do engagements with customers.

So let's start with quality.

Obviously quality, as Mr. Deming said, is everybody's business, which is not just a platitude, it's real. I've seen many cases where developers are saying, "Hey, well, QA will take care of that. I'm just writing the code." And that's pretty extreme, but I've actually seen it even lately.

So having a culture that really has quality embedded as a requirement, objective, and a mode of thinking throughout the process from design through to deployment is critical so that everybody shares quality and it's not just a matter of finding defects.

I've even seen organizations that rank testers on the number of defects they find without adding the word relevant to customer defects, which is important.

I'm not sure what's happening with that slide, but I'll go to the next one.

Okay, so continuing on the... Hmm. Something happened to the presentation. Hold on.

So continuing on the culture aspect, design for continuous testing. There are plenty of cases where I've seen products that really are difficult to test because they didn't think about testing as part of the design. They've got hidden attributes that are important to customers. So for goodness' sake, make sure that the attributes for anything that's relevant to customers are in fact visible and accessible and testable.

And that the developers have access to a test environment. I've seen organizations that really only have a production test environment somewhere close to the end of the delivery pipeline, and so the developers had no chance of testing the product prior to integration sufficiently. So not a surprise, problems occur near the end.

So your pre-flight environment prior to integration needs to be a good facsimile of the production environment and hopefully scalable enough that depending on what the developers change, it can actually replicate a pretty large, complex test environment if need be. So it needs to be scalable and flexible.

And of course, teamwork. Just because something's been integrated doesn't mean that the developer's job is done. It's all about trying to make sure that everyone participates in taking those products through all of the test phases that I mentioned and get a good result in the end. And they can all celebrate together and not complain that somebody else raised a bug that they didn't have time to think about yet.

Okay, the second best practice has to do with structure. Process design is not just a matter of saying, again, let's do a test after a build, but how do you interconnect all of the different test stages properly? What are your test stages? What is the goal for each stage in terms of timing and coverage?

You need a test strategist to think about that. So your entire test architect that thinks about the entire pipeline and designs the interlinked test processes as a symphony of automated test stages.

Fault-tolerant. The infrastructure itself, as I mentioned before, if it's not fault-tolerant, you're going to get a lot of false positives or false negatives, which is a big waste of time. So it's not just good enough to have an interconnected set of processes, but you need to have them very reliable and self-healing, especially in a complex environment.

The next one has nothing to do with infrastructure, but for some reason it comes up when I talk about this. I've seen so many times when people write scripts and they write the credentials of different parts of the systems into the test scripts. So don't do that.

As you're building out your test environment, make sure the infrastructure has, as part of it, the ability to have, let's say, a credential server or something separated so that you don't have to burn in the credentials into the scripts.

Intelligent dashboards. I'm not talking about the test results, I'm talking about the infrastructure of the tests. Make sure that your dashboards are monitoring all the different components of the continuous test infrastructure so that you've got some predictive ability to see, "Oh, wait a minute, we're running out of resources in this area or that area, and we should be resourcing that better."

The third practice is to have continuous test-ready tools. This could be a very long list. The main items really are to have test tools that are organized as microservices available through RESTful APIs. So not only test creation and execution, but analysis as well. All of those components need to be available from a RESTful API, so they can be integrated well with the rest of your tool chain very efficiently.

You need tools that allow you to pipeline or cache tests. Don't wait for the end of one test before you start the next one. That's one of the ways you can get things a little faster. So that needs to be done efficiently.

If you have a product like a network product or some other product that really performs differently in a physical environment, then you need virtual as well as physical tool capability.

You don't know in advance necessarily what programs in the future might be added to your product, so it's best if your test environment can be developed. You can develop test cases with something that's a little bit program-agnostic or at least compatible with whatever programs you might ultimately want to use.

Scaling, both vertically in terms of being able to get a lot of resources when you need them for a test, but also horizontally so you can have many different kinds of tests going on in parallel. So the tests themselves need to be organized so they can run independent of each other.

The integration of the tools, again, all these different phases are required and it's not good enough to just integrate it with the build. How do you integrate all of the tools in the test environment for pre-flight, for CI, all of the different test phases through to deployment?

The topologies, again, if you have a complex product that would benefit from testing tools that are topology-aware, such as a network product, then again, you want to be able to orchestrate topologies that are comprised of physical, virtual, or a hybrid of physical and virtual resources. You want to be able to manage and keep track of all of those many resources and utilize them as needed, bring them up, save them, and recall the topologies and bind them together with the actual test cases at the runtime and integrate that with the test automation environment.

Accelerating testing, the most obvious thing to do is make sure you're using powerful servers. I've literally seen cases where in some big companies, they have an aging policy for their servers where they take their last-generation servers and give them to the test organization so the development team can get the next generation.

That's not necessarily the best approach if you want to have accelerated continuous testing. In fact, in some cases, you want to have even more powerful servers so that you can exercise the product or system under test, more than overpower it in order to determine how it reacts in situations that it might run into in a real-life deployment.

The tests themselves need to be designed for acceleration. So the timers of a test might be set differently if you're in a CI cycle that is only minutes or an hour versus a regression cycle that is perhaps many hours. So the tests themselves need to be designed according to the different phases in the pipeline.

Pipeline the tests so that you've got a way to keep all of those tests independent of each other so they can be scaled horizontally. Use as many resources as you have available for the test to get them done quickly.

Set thresholds. There's no point in completing some tests if you've already hit your failure thresholds. You might as well stop, get on with the next results, and report the results. So that's important. A lot of times I've seen test suites that are fixed. They run to completion no matter what the failures were.

A really important one that I don't see that many folks doing but is extremely valuable is if you can somehow adjust what tests are selected and run depending on what changed in the build. And there are different techniques for that.

The one I personally prefer is keeping track of test trends, the failure trends, and correlate that to code changes, and then tend to select the tests that are the ones that most fail when those particular modules are built. But there's a lot of other methods for that.

Relevant analytics is the last best practice. So again, it's not good enough to just report tons of test results. You really need to make sure that the right results are going to the right people at the right time. And you do that through somewhat smart dashboards and alerts that are keeping track of who really is responsible for what test results.

The analytics can help you with that. So the logs need to be kept and forwarded along with any results, as well as make sure that the thresholds... There's no point in sending results to folks if a threshold has not been hit. So that can also help cut down on the chatter of test results.

Aggregating across a large number of simultaneous parallel tests is important so that you can still organize the results as they occur. Don't wait until you have all the tests completed. Aggregate the results as you go, report them as you go in a dashboard in real time as much as possible.

Take snapshots as well as trending views. I often see snapshot views but not trending views of the test results. So how well did that particular test or module run in the past or over the last so many builds?

And of course, have the ability to telescope into the results to help with diagnostics once they're reported so that the developer does not have to reproduce the test environment to figure out what the problem might be.

So those are just some of the key items I often find folks are not really attending to and best practice environments do.

This is a case study, an actual study of an actual DevOps environment I was involved in developing. 360 developers, six sites, 36 million lines of code in the product, 100 features per release, 86 build targets. A lot of complex interdependencies. It was a network product. Over 200 topologies, 12,000 test cases per day. Well, actually per run. And basically, all together, it was over a trillion test possibilities that we figured out.

And the situation was that this particular customer had two attempts at implementing their DevOps environment with their own engineers, and they were not really succeeding. They were still struggling, and it was primarily the testing part that they were not getting right. They were not able to get automated testing to where they needed it to be or integrated well with their processes.

So after the project, at least the first phase of the project, because it was a multi-phase project, multi-year, but this was the six-month view. Basically, pretty dramatic results after only six months of the project.

Their release cadence was reduced by half. So they were able to get major releases and their minor releases both reduced to half. And now it's actually better, but they used to take six months to make a major release, and at the time of this writing, it was three, and I think at the moment it's one. So they continued to improve after this.

Yet at the same time, number of features increased per release, and the really dramatic thing was the number of defects that were cut down from literally 1,260 in that statistic down to 10 per deployment. And the underlying, I think, reason was adding continuous testing, automated testing, and the monitoring that's relevant to go with that testing. They were able to get their automated test up to 85% of their total tests.

Okay. This is a takeaway tool that I wanted to offer everybody here. We have this approach. We use a best practices tool to help with customer engagements. There's no such thing as standard best practices, but it's helpful to have some practices that people have found to be successful, and this tool enumerates what Spirent has found to be successful.

This is just the first page of the tool. It basically generates a gap assessment. After you fill in what is your situation, then it will calculate a gap between what your situation is and what is considered, at least by Spirent, to be the best practices. And that can help you kind of organize where you really want to focus in on improvements to your DevOps environment.

And it covers seven different areas of practice across DevOps. Continuous testing is actually only one of those areas.

So in summary, continuous testing best practices emphasize speed and relevancy. We have some tools that can help you. If you want to benchmark, we have a blueprint on our website and the best practices tool.

For goodness' sake, if you're not sure how to do it, hire somebody to help. You can save yourself a lot of time and failures, potentially.

Some further takeaways, you can go to our website. We have an e-book there, the tool. And if you want to know more about Spirent's activities in sponsoring Aston Martin races, there's a link.

How can Spirent help? Again, we have a continuous test and a DevOps solution with tools, CT orchestration, automated lab management capability, and professional services. If you do need help, we're one of the resources available to help you.

There was a question, what would we like help with as well? I'm constantly on the lookout for more and more analytics that are really intelligent as far as analyzing test results. So if anybody has some really good analytics, especially predictive analytics associated with testing, that would be pretty interesting.

Okay. That's the end of the talk. Any questions?

Anyone got a question?

It is lunchtime, I guess. So thanks, everyone. Okay. Thank you.