Tracy Bannon

Hey there. My name is Trace Bannon. I'm a software architect, engineer, DevOps strategist, and a senior principal with the MITRE Corporation. Let's bring up some slides.

MITRE is a federally funded research and development corporation chartered by U.S. Congress, and much of my focus is working with the U.S. government, but also with coalitions and partner nations, to solve complex problems with software.

Today, I want to share with you what I'm observing as a missing link for DevOps, and that's data.

With the help of conferences like DOES, we are all talking about the challenges of adopting DevOps. Together, we're wading through approaches and learning from one another. We're moving toward a more common understanding of what DevOps is, and we're creating that common lexicon. We're working together to figure out how to reduce resistance to change and how to remove that dev-versus-ops mentality. We're even learning to rein in our over-focus on tools. Even so, research shows that our quality is not improving at the rates that we expect.

I first heard Joe Wessel use this phrase, and it resonated with me: "Quality is going down the drain with the pipeline." If we are automatically testing, how can quality go down? We're all unit testing, right? We're all focused on code quality and coverage, right? We're adding in some Selenium for good measure. If we are automating testing, why are there still quality escapes?

We are all familiar with the infinity loop for DevOps by now. Looking at this, can you spot the problem? Notice that test is an isolated step. Simply shifting testing to the left or right into production is not the answer. Quality and testing are part of every aspect of DevOps. We need to rethink the wording of the infinity loop. Testing and quality are the very fabric of the entire repeating life cycle.

We need to be more specific about each and every phase, step, and transition, whatever you want to call those colored chevrons, to call out specifically how to amplify testing and quality continuously. I have a love-hate with that word. It makes so much sense: constantly and chronically evaluating and improving. I had an opportunity to meet with Janet Gregory, co-author of Agile Testing and founder of the Agile Testing Fellowship. She shared a version of the infinity loop that I can really get behind.

The term continuous testing has been hijacked. It was intended to mean focusing on testing and quality across the entire effort. Let's use the term holistic testing to encompass all quality focus. During discovery, plan, and understand, identify risks, test assumptions, and build prototypes. During build and deploy, instrument the code, automate tests, and include stories and features, infrastructure testing, pipeline, and quality attributes for the whole system. Moving forward to release, observe, and learn, test in production and observe how customers are using the product and how we should adapt.

Now that we have a focus on quality throughout, let's talk about other gaps that can reduce the effectiveness of testing.

We know about some of the more obvious gaps: lack of cohesive test planning, skills gaps and a need for training, and overly fragmented tests. We know that software is still tightly coupled and that there was a flawed belief that architectural design just emerges. Spoiler alert: it doesn't.

Given these known knowns, and given that industry is starting to address them, we're still seeing failing tests. According to a 2020 report by Undo Corporation and the Cambridge Judge Business School, failing tests cost the enterprise software market $61 billion per year, with nearly 620 million developer hours wasted on debugging. Six hundred and twenty million hours. This amount is staggering.

Why is debugging so difficult? Because we can't reproduce the defects. I sometimes call these phantom defects.

We have so much available to us, literally decades of test engineering theory on white box testing. If you're not familiar with white box testing, it means you have access to the code base and you construct your test based on what you see in the code base. We have direct access to stakeholders, and we have tools, and we have tools, and we have tools.

I'd like to share a story. For a very large organization, I was an enterprise architect with a team of architects, infrastructure, application, and security all on my team. We worked across many lines of business as a shared service capability. The testing group also lived in the same IT shared services organization, ITSS. There was a heavy push for automated testing. Budget clipped the wings of the testing group, and they fell back to muscle memory: manual testing, late in the cycle.

I began to hear reports of inconsistent test results and errors that couldn't be reproduced. Phantom errors. What we uncovered was that some applications had heavy state management and needed refactoring. We also found out that the data being used by QA was stale. They didn't reset it after every test cycle, and the quality of the test data was suspect too.

This highlights that exercising the code isn't enough. For a test to be valid, there has to be repeatability, and the data has to be reset. Manual exploratory testing shouldn't be part of the pipeline. It has to be asynchronous and autonomous.

Essentially, lack of adequate test data is eating into the productivity gains and into DevOps investments. We need to know that there's a change on the horizon, so let's work at changing the narrative. The mad dash to automate testing has delivery teams asking, "How will we test this?" If you're going to stay true to holistic testing, you need to ask a broader set of questions: What should the test accomplish? Why am I going to test this feature or capability? How will I test it? What data is needed?

The determination of what to test and how to test it is a decision of the architects, engineers, and the business. The key to effective, fruitful testing is having an intentional focus on defining, acquiring, and managing data for testing.

What types of data do we need for testing? If we were together in an auditorium, I would have you yell out the types: unit, security, performance, integration, end-to-end, usability, and so on. Beyond the types of data, there are myriad considerations overlooked when gathering test data. What's your domain? Are there unique requirements, for example banking, insurance, healthcare? Are there regulatory constraints: PII, PCI, PHI? If you work with the government, are there data classifications to be concerned about? Is the entire team aware of the architecture, requirements, and quality attributes? Quality attributes are the cross-cutting capabilities, those -ilities: portability, scalability, and so on, sort of non-functionals.

Once we have a better idea of the profiles of the data necessary to support holistic testing, who's going to make the data? Often test data is created as an afterthought, and the responsibility is loosely defined. It might even not be defined. Generally, test data will be made by some combination of developers, QA testers, security testers, and database administrators, DBAs, and each approaches it from a very different vantage point.

Developers focus on white box testing, based on knowing what the code looks like, and their data represents best case. Their focus is writing new features quickly. QA testers often drive their data efforts using spreadsheets and may inherit developer data. When they can, they will swarm to using a set of masked production data. Security testers create their data almost exclusively manually right now, including an ad hoc approach to it. With a focus on DevSecOps, there is a pivot toward intentional generation of security testing data. DBAs feed the whole team by bringing production data into a lower environment. If we are lucky, they may even apply some masking techniques.

But I'm not a fan. I'm not an advocate of depending on production data. Production data represents an accurate image of data that works well. However, you're only covering 70% to 75% of the scenarios, and you're missing those edge cases that cause headline defects. The most important reason I frown on using production data is that we're not masking it, and we're not protecting it in non-production from hackers. According to the State of Database DevOps from 2019, the report by Redgate, 65% of companies are using production data for testing. Only 35% of them are masking. The result is a vast cyber risk footprint.

Where do we get the data from? We should take a blended approach. We need to build, borrow, and buy. Building data involves using manual techniques, interacting with the front end, and applying business rules to synthetic data generation tools, and that forms a strong basis for your testing suite. Borrowing data means selective production data borrowing and masking it. It also includes referencing prior systems and applying an extract, load, and transform process, an ELT. Buying data from other corporations or data brokers, using publicly available data, or even web scraping can round out your approach to creating a robust suite of test data.

Let's say that we have a good basis and followed the principles for test data acquisition. There are still some stumbling blocks: preserving test data and loading or reloading it. To repeat an earlier lesson, if you don't have a valid, repeatable initial state, a test is not valid.

Tying this back to DevOps, delivering value means delivering quality. Delivering quality demands holistic testing. That can only happen with robust test data management, test data management tools, and integration into the value stream and the DevOps pipeline. Like other DevOps hurdles, applying technology is not the challenge, and it's not the solution.

You need to be aware of likely obstacles with delivering value using modern software practices. First, have a test strategy. This does not need to be long and complex. Create a living strategy and point folks to it. I touched on data acquisition earlier. Another hurdle teams face is massive use of databases instead of a focused subset. This is unwieldy, costly, and without a strategy for managing it, prone to stagnation and leaks.

I've been surprised at the number of teams that try to create their own masking tools. There's an explosive growth in data profiling and masking tools, and those can fill your needs. There are budget constraints, but they're being applied to test data creation. That needs to be rolled back. It isn't free, so it needs to be included in planning. I mentioned data sprawl. Logs are a type of operational data, and they're needed for testing as well. Access to logs is often a sticking point for teams.

I want to share another experience story. I was working as a software architect with a massive government agency, one that has very complex test data requirements, the most I've ever seen and experienced. Every public release of software had to have a full production data archive aligned to policy and law. The agency had put a priority on automating all tests and therefore all data. A centralized team managing the test data was understaffed, and they attempted to either use production data sets or create them from scratch. Many setbacks came from this approach.

The delays and failure taught us the value of making risk decisions on what to automate and the value of profiling data and using the information to feed synthetic generation. The QA data team also became strong supporters of self-service data provisioning.

Once data is created, it's got to be managed. Managing test data carries its own considerations: deliberate and controlled data access, maintaining test data accuracy, securing the data in non-production environments, storing filter criteria, data sets, and tagging against the user stories they're relevant to, and the test cases or code branch. Dataset versioning is important, and data refresh on demand. Remember, you should delete test data, including database logs and other files.

There are test data management tools and platforms that can help. Selecting a TDM means you need to do a trade-off analysis. Each category has subtopics to think through when designing your test data management platform: data source diversity, data cloning and versioning, sensitive data masking and encryption, security and access-controlled data generation, test data warehouse, open APIs for integration, and self-service features. I've included what a colleague of mine calls an eye chart. There's a lot of detail here, and when you download today's presentation, you can check it out in more detail.

Let's get back to the DevOps or DevSecOps pipeline. There can be data friction moving toward quality-first value delivery. Version your datasets alongside the code and configurations. Reduce unnecessary copies, but enable self-service. Leverage and integrate APIs. Think about virtualization. Allow saving of data after a test is complete; sometimes you need to come back to it and evaluate that exact state. Ensure cleanup happens. Remember, logs are data too. Consider applying and adding performance and load testing into your pipeline.

Over the past few years, I've learned that if something is really common sense, I should probably state it out loud as well. I'm going to give you a couple of things that you shouldn't automate. Don't try to automate a test for an anti-automation feature like CAPTCHA. Don't focus on infrequent or low-risk tests, much like the user experience story I gave earlier. Allow exploratory testing to be manual.

There are many players in the test data management space. Here are a few examples. I encourage you to check out these links, talk to these vendors, and understand what they are using and doing. Ask them about their lessons learned and what industries they're focused on.

Looking ahead, we are going to see an emphasis on testing and test data management. We're also going to see an emphasis on test data management environments. IaC and automated dataset provisioning allow us to spin up new test environments at parity to production. Digital twins are gaining importance, especially with some of my sponsors in the U.S. Department of Defense. The ability to have full simulation of hardware virtualized gives the ability to react earlier. Service virtualization has been around for a number of years. The tools to support mocking can improve fine-grained unit testing and help decouple brittle integrated testing environments. Data virtualization and database virtualization are growing as well, with technology now at enterprise grade.

We've covered a lot today. I'd like to provide you with my contact information so that you can reach out to me. Let's talk more about DevSecOps, let's talk about software architecture, and let's talk about the data. That's the missing link that we need. Thank you.