Tatiana Salazar

I'm going to present to you about how we are accelerating testing in a high-risk environment.

Before we get into what's Experian, I want to give you a little bit of an overview of myself. I am the manager of the Testing Center of Excellence, and I am based out of the Costa Rica office in Experian. This office opened 10 years ago, and I've been working with Experian for these 10 years, so basically I'm one of the founder employees.

Fun fact: when we started at Experian Costa Rica, we were barely 80 employees. Today, we just got an email that was telling us that we should celebrate the 1,000th employee. So it has been quite a journey.

Now let's talk a little bit about Experian. If you're familiar with the company that I work for, probably this format is what you are likely to understand, and it is a credit report. It looks like a credit report, although the information is not a credit report.

Experian is one of the three main credit bureaus in the US. We have headquarters in the UK, but my point with all of this is that although you all relate it to credit reports, we are much more than that.

What else are we? We are an army of data scientists that work with a lot of data coming through to make your lives easier in terms of making decisions. For example, if you want to buy a used car, our automotive business unit will help you make that wisely decision because we will have all the information of what that car has gone through since the very beginning.

We also help insurance companies to provide you health. We also have a marketing services business unit, which helps companies advertise targeted to your needs. And also when you go and you Google something and you see a small advert at the right of your screen, and probably it's something that you bought a few weeks ago and you say, why in this world would this know? Yeah, it is probably Experian.

We operate in over 35 countries around the world. There's a lot of countries where maybe there's not big operations, but there are small sales offices.

Now, a little bit about the background of the business unit that I work for. I'm part of the internal IT services. I am not part of all of those business units that I showed you, so I am not customer facing. But we work with all of those business units.

Here in that picture is Barry Libenson. He is my CIO. A few years ago, he announced that he had a very defined vision and mission for us, and it was to align ourselves into the DevOps journey, and that we were starting a DevOps transformation. He was planning to turn all the Waterfall into Agile.

A little bit of an overview of my team. We are mostly all of us based out of San Jose, Costa Rica, which by the way, if you haven't been to Costa Rica, you should. It's paradise. And we have two team members in the Kuala Lumpur office in Malaysia. This is my team. We are a team of 10 in Costa Rica.

We are a merge of two quality assurance teams, which gave services to completely different products. One of the teams tested corporate type of very fun products, such as Oracle and Salesforce. The other team tested more internal type of web-based tools and web services. So we did a merge of both, and we created the Center of Excellence from scratch.

This transformation began in, let's say, August last year. But it was a rough start, because when you get two teams that work totally different and then you tell them, we have to be agile, we have to move into DevOps, and probably half of the team doesn't even understand what we're talking about, it's a struggle, and they all want to keep working only in their silos. So first thing was to make them feel as a whole team, and then we started the transformation.

What do we do? What was the first thing that we did once we were already settled as a team? We looked for the better tool for testing automation that would support our DevOps transformation, and that tool was Tricentis Tosca.

We've been very committed into this transformation and to build a very solid Center of Excellence. We've created different roles. We started our Tosca journey in February this year, and as of today, I have the two first Tricentis-certified testing architects in my team. I also have, and I'm very proud to say, two Tricentis Testing Heroes winners in two different categories. So that's how committed we are to this transformation.

Now I'm going to tell you about the team that's based out of Bogota, Colombia. This team is not exactly my team, but it is the first team that partnered with me from a Testing Center of Excellence perspective. These guys came to me and told me, I need your help because we're working on a huge project, and we need to do fast testing.

A little bit of the team, but important: Experian bought the credit bureau in Colombia and is now in the middle of transforming it into Experian standards. So it is a big project. It's a huge project.

These guys work in SAFe, so they do project increments every three months. They get together in the office in Bogota, Colombia, and all the different small teams, scout teams, how we call them, get together. They talk about what they did, what they're going to be planning for the next three months, and then everybody goes back home and works on their normal Agile teams within their scout teams.

Based out of this experience and this partnering that I did with this team, I have a few points that are kind of lessons learned about a DevOps journey. This has been very interesting. It's our first experience, and we've learned a lot from them as they've learned a lot from us, too.

First point: define success. What defines success in a DevOps transformation? I would say that the fact that our CIO is so committed to bringing this to reality, that we are all as committed as he is, and we are all walking hand-in-hand with him. We live it by heart. We truly believe that this is what is best for us, and we all work every single day to get into DevOps.

It takes an enterprise. What do I mean by this? Right now, we are doing an approach that is from the inside to the outside. As I mentioned before, I work for the IT business unit that gives all the services to the rest of the business units within Experian. So this transformation that I'm talking to you about is from the inside. We are the core, and we want to give the example to the rest of the Experian business units, the customer-facing ones. But we need to do it with the best practices and the best standards. So once we're ready, we will start preaching out. We do already, but... so, okay, building up.

I would say here the most important thing is that we all need to understand what is exactly our place in the DevOps journey. In my case, I am in charge of all the testing, automation, and quality assurance. So I know exactly where I have to sit and who I need to collaborate with.

How did we start this journey, specifically within my testing team? We were stuck with a lot of Waterfall projects, 100%. So what did we do? We just grabbed everything that we had pending and all our BAU activities, and we created a backlog, which then we weighted. Once it was weighted, we started tackling the less weight type of activities. We planned to use 80% of the capacity of the team so that we can leave some time left for getting architect training or other sort of emergencies. We also have an on-call rotation because we still do a little bit of operations, so we had to leave time for that.

That's how we started tackling, and we started transforming our Waterfall projects into Agile, which I can tell you is a good success story because right now we have our daily scrums that nobody misses and everybody enjoys, though it was a difficult transition.

Second step that we did: our definition of done. How can we begin to partner with different teams and tell them, you have to follow our best practices, if we don't even know what our definition of done is? So before we started giving advice to any team, we decided that this is our definition of done.

The automated test is defined. The code has been reviewed by other peers. We've done at least three successful runs with distributed execution. It's documented in Confluence so that if any other team member from the development team wants to use it for daily builds, they know what type of tests we have in our inventory. It's decommissioned in UFT, which is our old tool, and it's also integrated with Jenkins because we want continuous testing.

The next key, I would say a success factor in the DevOps journey, is having the right roles and having the right tools. First of all, I'll give you the big picture of how our DevOps infrastructure looks like. That's what we have, that's what we use. I think it's pretty standard. It's what mostly all the companies use.

Like I said before, in terms of testing automation, the chosen tool was Tricentis Tosca. You can ask me all the questions that you want from a testing perspective. If you have other questions about the rest, I can refer you to my colleagues, though.

Bank of experts. What do I mean by bank of experts? I don't want to confuse you because in a DevOps transformation, you have to think about the teams not being siloed and that they have to be collaborative, and they should not be differentiated. But what we've learned is that although you may have your scout teams, as we call them, that are the mixed teams with the different roles within the same team, you need experts outside those teams that will consult and help them with their expertise.

I want to quote one of the dev managers who is always telling us: it is impossible, and it's not sustainable, to think that in a scout team you will have all the expertise and all the knowledge. You need experts that will help you.

So that is exactly why the Center of Excellence was built. We have the Testing Center of Excellence, we have a DevOps Center of Excellence, which are all the experts with the tools that I showed you. And we also have Release Center of Excellence, and this Center of Excellence will keep building.

It is very important because from the feedback that I've gotten from scout teams is that normally when the Testing Center of Excellence didn't exist, then you would have to fight to get a tool which wasn't standard. This group used this, the other group used this other tool, and now we have a recommendation, and we have a set of experts that will guide you through the process of onboarding with the tools and onboarding with the best practices.

Also very important is that when somebody needs support, it's faster rather than what we had before, that it was nobody's land.

Finally, what do we give from a Center of Excellence perspective? It is a double impact. Because we build standards, we have a tool, we have all the infrastructure in order to support this transformation, and what people are getting is they are going through a smooth transformation, and they are not feeling it as painful as they were feeling it at the very beginning.

Very important to say: since this was announced, it hasn't been as great a story as I'm telling. We have had a lot of stones that we've walked through. I think that's it, and now we have questions.

Q&A

Question: You have some regulatory challenges at Experian. Could you talk about that?

Tatiana Salazar: For example, we went through the European regulation. I just forget. The GDPR, yeah. So we had to accommodate, in three seconds, how we were going to handle test data, for example. Thankfully, Tosca is a great tool for that. You have different approaches that you can use. You can build your set of synthetic data. Synthetic data will take your real data and mask it up, or you can start building your own set in the database to try to emulate the database and create data that's dummy, but it works with the tool that you're trying to test.

It depends. There are several different tools that we test. In terms of a production environment, we had to mask that out completely. For example, for our EVS tools, that's completely masked out. Nobody has access to production data but us from a testing perspective, and, well, we have to go through signing contracts and things like that.

No, sorry, in production, no. Just in our test environment. Production data is totally covered. Nobody can see it. No, at least from an EVS perspective, no. There are other tools that don't have this type of difficult data, so we can do more tests within the production environment. Sorry.

Question: You mentioned that you had to come up with your definition of done before you could go out and teach this transformation among other areas of Experian. Do the teams that you're teaching your methods to get to define their own definition of done, or does that become the definition of done?

Tatiana Salazar: My definition of done is what we call the standard. My speech would be they have to follow. Now, that doesn't mean that it has to fit everybody or that, because we're barely starting, there's something that we cannot improve. So it's welcome to discussion, but it's pretty closed. Those terms, we really thought them through before we released it and presented it in a place like this. So yeah, those are the definitions of standard. If you want to onboard with the testing team and use our tools and follow our standards, and you want to get our full support, you have to be on board with what we tell you is the best thing to do. If we permitted non-standardization, then support would be a headache.

Question: Two questions. How many tests do you currently have automated within Tosca? And what were the key reasons why you decided to move away from UFT and go to Tosca?

Tatiana Salazar: I'm going to answer them the other way around. The first main reason why, and I actually have a slide for that. I knew this question was coming.

First of all, UFT is definitely not a good tool for supporting a DevOps transformation. It's not very agile, and it is a pain to integrate it with Jenkins, for example. Whereas Tosca is super fast, and with the new release, it's going to be even faster.

Then people required extensive programming knowledge. I'm not saying that Tosca is like you can bring your little sister and she will be able to automate with Tosca, but I can tell you it's way less time-consuming. Scripting is way too time-consuming in terms of developing from scratch and maintaining. That is good.

That said, we still need the heads that will be super technical. We still need them, but we need less of those. And they are focused on infrastructure, architecture, customizations, higher-profile things.

Also, UFT tended to be very slow. It would bring the machines down, so you need a VM in order to be able to do stuff while you are running tests, and that's more expensive. Again, a little bit with the programming knowledge is the learning curve. The learning curve is higher. You have to know Visual Basic scripting in order to use UFT.

What was your other question? Right now in our Tosca repository, we have two repositories, one for the Colombia team, one for my project. In my repository, we have about 250 tests. In the Colombia team, we have about 170 because we are moving them out of UFT into Tosca, and also we are building new tests. That's what we have right now.

Question: Could you touch on non-functional requirement testing like performance and security? Have you guys integrated that to the pipeline yet?

Tatiana Salazar: Excellent question, and I forgot to mention that during my presentation. Right now we are just using Tosca. But in our roadmap, we will include Flood, which is performance testing. We're still working on that with them. Security testing is something that still they have their own people, and we don't mess with them. So in our pipeline, we have Flood for performance, BI for BI testing for big data, and we also are going to move out of HP ALM into qTest.

Question: What kind of infrastructure is your automation running on? Are you using on-premises or cloud services?

Tatiana Salazar: No, on-premises. Right now, we are in the middle of moving everything to cloud, but since we handle so sensitive data, we cannot just say, today we're going to use the cloud. We have to go through a very big security process. Right now, they're on-premises, but we should be moving into cloud eventually.

Question: What percentage of your testers are actually still doing manual testing versus using Tosca for automation?

Tatiana Salazar: That is the best part of my story. I have zero manual testing. Believe it or not.

Question: How many, in terms of resources?

Tatiana Salazar: All the team is doing automation. Maybe I would say that the two folks in KL would be the ones that are still not automating as much because the Oracle EBS team consumes a lot of their time on gathering requirements and things like that, that are not testing per se.

Question: They're not doing in-sprint automation, right? It's a lag, right?

Tatiana Salazar: You just touched my pain point. There is a big time zone difference, so I have different Scrum meetings with the Costa Rica team and then with the KL team. I've been trying to onboard them, but it is quite a pain point. The other problem is that the experts are in Costa Rica, and it's difficult to support them, although we do breakout sessions very late at night, but it's still a pain point.

Question: Do you factor in or handle data quality, data integrity in your testing? And if so, how do you deal with that?

Tatiana Salazar: Specifically when we are transitioning from one big data handler to another, for example, from mainframe to Hadoop, we do a lot of data integrity.

Question: Anything database specifically related testing, keeping database changes going together with application changes? Do you test directly database changes?

Tatiana Salazar: No. We don't test at a database level. But if there are database changes, we would test at an application level.

Question: When you said in your current roadmap you have ALM and you're trying to move to qTest, which tool does the Agile team, the project team, use to track stories and other things?

Tatiana Salazar: For managing test cases?

Question: No, the stories.

Tatiana Salazar: Oh, no, the stories is Jira.

Question: So why not use Jira for test cases also?

Tatiana Salazar: Because Jira is not meant to do... If you have manual tests, my team does not have a lot of manual tests, but other teams that you partner with will have, and Jira is not exactly made to handle manual testing because it will not keep the registry of how many times it has passed per round, things like that. There's a lot of statistics that you actually need that will be in a test management tool.

Thank you.