Brian Timmeny

I'm here today to talk a little bit about BBVA, our transformation, and scaling at extreme measure. What is our DevOps journey?

I'm going to talk through a little bit of our journey, some of our challenges, and the tact that we've taken in terms of executing our transformation.

BBVA is a multinational bank across Europe, across Latin America, about 140,000 people strong, and within IT, about 12,000 people strong in terms of the transformation that we look at. And so today, I'm going to be talking a little bit about the scale of our transformation and the way in which we're attacking the scale of that transformation, and what that means to DevOps in terms of not only a transformation of each of our geographies individually, but what it means to lead a transformation across a truly multinational company, BBVA.

There we go.

So a little bit of background. My name's Brian Timmeny. Currently, I'm the global head of DevOps and our engineering processes. One important note: BBVA has decided to put all of our processes, all of our transformation, within one group, which is the organization for which I have responsibility in helping us scale across all of our geographies in BBVA.

BBVA, side note, already, I would say, a fairly dynamic bank, especially in the southern geographies of Europe and certainly within the geographies of Latin America. Currently, we've got number one mobile app, so already some great technology advances. But we've got a long way. We've got a lot to go.

Our objective is to be one of the global leaders, both in terms of banking, but most importantly, in terms of customer experience within all of the geographies where we operate. This comes through having top technology, top experience, and ensuring that we can execute in a way for our clients that is extremely fast and that is an experience that is both what they're looking for, what they need, but what is going to help us adopt clients and ensure that we can continue to advance that forward.

Some of the challenges that BBVA has today and some of our keys that we're currently looking at, and I don't think any of these are secret challenges that are only to BBVA, but we have some very inconsistent processes. We have extremely manual processes. We have low levels of automation across our system. But most importantly, I think one of the biggest challenges that BBVA faces today is the fact that we're extremely siloed.

We're not only siloed within each of our geographies, we're siloed by the fact that we have each of our geographies. And the transformation that we have is not only to transform each of our individual teams, but to bring us together as an entire organization on the DevOps journey.

Within the early stages of our transformation, and I'll call this within 2017, early into 2018, that we're looking at, there's tons of metrics we could bring to bear for what DevOps means, what we could do. We have two that we've set up: velocity and quality.

Reason why we started with these two: one, they're easy to say, and they're extremely easy to transmit over a large geography. If we were pulling in 10, 15 people to explain what our transformation looks like and what we're trying to achieve, we can go much deeper. We can go much deeper in the complexity of what that means. But when we're suddenly having to bring all of our teams to try to report on common metrics and execute in a common way, what's really important for us is the fact that we want to make sure it's a simple message, and it's a simple message that we can dive into and that we can measure practically. And that's one of the keys of our transformation: ensuring that we have the ability to truly measure this practically.

So there's two sides to what we look at when we're driving this across all of BBVA. One is the fact that we have one common framework across all of the global organization. Today, we have segmented framework that exists very federated in every local geography. Some have no framework, and some have ad hoc framework.

What's important in this is that does this mean we'll drive one global framework that'll be exactly the same rigid within the context of global? Absolutely not. But what is really important in this is that we have the concept that we're all rowing in the same direction and that we all deliver generally the same.

And I think what's important about this is, is it important that we define the processes consistently? Absolutely. But is it as important or even more important that we're defining that we are trying to adapt to one common culture that delivers in one common manner?

The second is that we have one automated ecosystem. Same rule applies. What's important on this is that does this mean we will have one single CI/CD ecosystem for the entire organization? No.

Will we eventually get there? Is the hope that we're eventually driving toward that? Absolutely. But the objective that we're really looking at is how do we get everyone to drive consistently toward this level of automation, and how do we get everyone to drive consistently toward this level of one common view of the ecosystem?

What's important about that is as we start to think about the individual components in an ecosystem, that we do have a common platform, that we do have a common CI/CD ecosystem. We do have a common deployment ecosystem. That becomes extremely important in our journey, not only from a technology perspective, but as importantly from a cultural perspective that begins to align everyone around this.

So within this, now we get into a little bit of the strategy of how we think about execution, how we think about transformation. There's a top-down component and there's a bottom-up component that has to go into every transformation. Ours is no different.

And we have six stages that we look at, or six components that we look at, as we drive this execution for BBVA. And now we're going to dive into each of these six components now. Maybe in just a minute.

So the first component is around our framework. And what's really important about the fact that we're looking at all of our teams in a consistent way, and this brings in the cultural aspect as much as it brings in the framework aspect.

And things when we talk about, we want teams that are dedicated teams. We want teams that are internal teams. They're not consulting teams that come for a while and then leave. These are long-term persistent teams that exist within our framework. And the whole idea is that we want to optimize that end-to-end framework and ensure that we have a process that optimizes that end-to-end framework.

What's important about that is when we think about the technology and the pipeline from end to end and the components that have to exist in that end-to-end framework, we have to have people that also support it. And when we think about those people that support it, they have to begin with dev, and they have to end with eventually being responsible for both monitoring, and they have to also be responsible for taking care of the application long-term.

And if we don't have a process that supports that, and we have technology that supports that, we have an incongruent organization. And so even though we have a large engineering transformation ahead of us, we also have a huge cultural transformation ahead of us, and this is foundational. And foundational to the reason of why we built the group as we did, having both engineering process and cultural transformation and the engineering of DevOps, the CI/CD ecosystem as well.

Second component that we look at is around metrics, and this is around ensuring that we have extremely tight visibility in terms of what we're doing. Early in our transformation, we're going to be extremely focused on visibility and quality.

When we talk about visibility, do we have visibility from the time a component enters the global repository to the time we can deploy it inside of production? Our definition early of velocity.

Are we good at it? I'm almost going to put it in the category of doesn't matter to start. If we can see it, we can now start to improve it.

Second, on quality, do we have the ability to understand the state of our pipeline and the state of all of our tests on those pipelines that are automated? Are we good at it? Again, I'll put it in the category of almost doesn't matter, because if we can see all of those tests, we can now begin to improve it, and that helps us focus on the journey more than the outcome.

Because at the end of the day, it's not about how fast did you get to production, because tomorrow you have to go faster. And in the end, it's not about do you have high quality and did you pass all of your tests, because tomorrow you're going to fail your tests. The idea is, are you on that journey to continually improve? And if you can't see it, you can't improve it.

So it starts, but it also starts extremely simple because it's important that we start just like everything else. We start MVP like any other execution that we drive. We'll get better over time, and we'll get more complex over time, but we've got to be great at the basics.

We have a strategy of automate everything. Something that's really important inside of this, and this is as much a cultural event as anything else. Does this mean automate everything? All of our bad processes, all the things we don't want to do? No.

But is it really important culturally that we say automate everything? Absolutely. And what's important about that is because we want to align the thought that I don't want a human in the middle of this process. I want a machine to do what a human doesn't have to do. And I always want to take it to the nth degree to ensure that whatever process we can automate, whatever process we can make easier and we can lift off of someone personally having to do, that's what we want to drive in this process.

Always means we have to re-optimize our processes constantly, but we're always looking at our process, and our process is always concurrent with automation. There's no process we think of that we don't concurrently think about how are we eventually going to automate that same process.

The next side of this that we look at is around ensuring that we have global collaboration. And what this means, and I'm going to jump a little bit into detail, is we have execution across many of our various geographies. In doing that, what's really important is that we want to begin to drive consistency across those same geographies.

Does that mean they have to be the same? Absolutely not. But does it mean that we want to get some sense of common contribution? We absolutely do.

And this means both contribution from what are the processes, what's the culture, how do we deliver, and the way in which we form teams, the way in which we insource teams, the way in which we deliver on a day-to-day basis, that there are communities that constantly report into that, as well as what we'll call more traditional open source communities, where we're going to open up things in our repo and there's going to be contributions globally to those from a technology.

The technology contributions, in some cases, is an easier paradigm for us to conquer than the cultural. But for us, the idea that saying you can contribute to a repo is something we can do. Saying that we want you to improve your process and share that process with a different country, that's a bigger challenge. And so forming these communities for us is key.

So when we dive into this now and we think about each of our individual geographies, what's important is that we have a lead of what we call our mirror organizations that exist in each of the individual countries. What's important about that is now we don't try to impose from the top what the culture is from global. Instead, we try to build those organizations inside each of those countries.

So now it's the country talking to the country instead of global talking down to the individual countries. And what's really important about that is now it makes it a group of peers instead of a group of corporate talking to each of our individual units. And it lets everyone begin to let down their defenses a bit, but by establishing that on an extremely local level becomes really important to us.

The second side that we look at is we have a DevOps center of excellence exists across three various geographies, but also exists inside each of our individual countries where we have representatives as well. This is important because it allows us now to, on a much more regional level, give that service, and it also distributes how we think about dev.

And the paradigm that we wanted to break out of was that we will build one ecosystem and then we'll give it to the globe. This forces us out of that paradigm because it means naturally that part of our dev will occur in Madrid, part of our dev will occur in the US, part of our dev will occur in Mexico, and what's so important about that is that we're now forced to distribute.

We're forced just by the nature and composition of the team to think globally about how that ecosystem is going to deliver globally. And it lets us out of the idea that we'll simply build it and give it to the countries.

The other piece that's important about these hubs is they are made up of individuals in those countries. So we're actually working with the dev teams, with the development and deployment teams that are in those countries that make it up, so that now someone from each of those geographies are the ones that are creating the ecosystem itself.

And then we also have the mirror organization related to, I'll say, our cultural transformation and process transformation. This is the same idea that we have the individuals who are building, what are the local processes? What's the local culture? What does the transformation look like?

And what's so important about that is that now it's not single geography dependent anymore. It's not simply one country determining what the process will be globally, but rather globally determining what the process is going to be regionally.

And when we determine those process, we have to deal immediately with the trade-offs, because if we have someone from two different geographies, three different geographies, four different geographies, they have that debate very upfront about what will work in the process because they're already from the country. They're already from their local geography, and those debates surface immediately. Versus if it's in a room of a single geography, those debates might not surface until it actually breaks at the time of implementation.

The other component that is extremely important in driving this framework forward, and when we think about this paired with the automate everything strategy, we also have a strategy that allows us to ensure that we enforce all behaviors in a manner consistent with how we execute.

What's so critical here is that when we simply say, "This is a new process. This is the way we want to execute." For example, don't put any consumer data in the public cloud that's not encrypted at a field level, if we're technical. Or if we're saying, "I want to ensure that no one has access to production. I want to make sure that our networks are divided so that things like WannaCry can't happen."

What's so important about that is that we then automate all of that at an event level, and we ensure that any of those behaviors that we expect to be true, that we have a way to automate and ensure that it's true.

If we don't have a way to automate it, if we don't have a way to interrogate it, and if we don't have a way to enforce it, it's not a real behavior, right?

Saying, "I want people to work more collaboratively," it's not really kind of complex to do that. But if we say, "I want to ensure that there's a minimum of two people from two different countries that have to sign any individual component before it's able to move into production," we can automate that.

So every process, it forces us to think like an engineering company because we have to build every behavior into our engineering principles of how we intend to execute this. And that's extremely important.

Because when we think about the transformation of what BBVA is going through, when we think of the transformation of moving us to a truly digital company from what was a banking company, we need to think like a digital company. And when we think like a digital company, the idea that we can automate those components in that way becomes critical.

And then one of the last components that we look at is around how we actually execute. There are two sides to everything that we do. One side is the framework, and the other side is the engineering and the ecosystem. The key on this is those two always have to move in parallel.

If we're advancing the culture, if we're advancing the process, we have to, at the same time, advance the ecosystem of our delivery. And when we advance that ecosystem of our delivery, the two always have to work in parallel.

If we're expecting that there are teams that are responsible for that end to end, the ecosystem has to allow them to, from the time of placing their component in the repository to the time in which they actually execute and deploy that into production, that they have full responsibility and the ability to monitor, the ability to move those applications into production.

Every team we have and every change we have, either on the framework side or on the ecosystem side, both form what we call a package. And the idea is there'll be various cultural components in any change, there'll be various technology components in any change. They form what we call our change, and every one of them is formed into a pilot. All of those pilots is the way in which we execute our transformation.

And what is an underpinning of how we think about our culture is that this is simply the way we transform, and this is a forever, for now. This is not a we'll pilot five times, and then we will achieve our victory of transformation. This is simply that we will transform wave one, and then we'll transform wave two, and then three and four and five and six.

And what's really important about that is that when we execute a pilot in Mexico and we have a learning of what goes well, what goes poorly, we then take that pilot and we bring it to the US. When that pilot goes well, we learn, we take that third version, and we take it to Argentina. The fourth version, and it goes to Spain.

What becomes important about that is that we continue to then package all of these elements more and more into the framework, and it does become more complex, but it's a continually evolving framework because it's always in the field. And what's so critical about the idea of it being in the field is that now we're always ensuring that whatever change we have is real.

And so if we step back a bit and we think about what it truly means to be a DevOps culture, the idea is that we want to ensure that these things are realized as quickly as possible to our end clients, right? To our customers. And in our case, for transformation, this is to our developers so that they can then realize those changes onto our end customers.

And for us, then we want to ensure that every pilot is always an MVP working version, and then we'll have the next working version, and we'll evolve the next working version and the next. But there's never a sense of we're going to build an entire end-to-end process, or we're going to build an entire end-to-end ecosystem and then deploy it.

We'll always deploy at the component level, and we'll actually live what it means to deploy iteratively.

Extremely complex, because it's very hard to sell that change overall because that means that the change does not come as fast as leadership wants. The irony is it's extremely easy to sell at a team level.

And so when we think about truly the catalyst that sits behind this, it's that the changes that we execute are really, they come from the team level. And at the end of the day, if you're going to transform an organization, we can execute a strategy, and we can understand it at the board level, but the real transformation happens at that team level, and everything baked into the strategy lives by that thought, that if we change the teams and more teams, and more teams, and more teams, that is what is going to transform the organization.

So a little bit on our results. So we've had, I would say, some pretty amazing results from an implementation perspective. We've had several major ecosystem components, primarily our repo and our pipeline that is currently used within several of our countries.

Have we executed our entire ecosystem in all the countries? Absolutely not. But what's really important about this is that now as we get those early foothold components of the ecosystem into the wider organization, they use those components, and then the next components become built into that ecosystem. And then the next.

And what's so critical about this, both in terms of the process and the culture, as well as the ecosystem and the technology that we implement, is that we roll it out just like a product.

When you roll out onto Facebook or Gmail, et cetera, they don't tell you when a new update is coming, and then you go in and check out the latest change. It's just there the next time you use it, and that's equally the way we think about deployment. That we have to be seamless, we have to be invisible to all of our teams.

And one of the biggest changes, though, that I'm really proud of in terms of our team and our culture is that we've open-sourced one of our first components out of BBVA, which is a visibility framework called Mirrorgate, that is currently publicly available.

What's so important about that culturally for us is that's a huge paradigm shift as we start to think about what is it truly to be an open source culture, and what is truly to live an open source culture, and to be an evidence-based culture, right? Not to say that we're going to be open source, we're going to be collaborative, but that we truly are.

So we're not done. We're not even close to done, right? I would say we're very early in our journey for sure. And places where we still have challenges and certainly welcome collaboration at every level is that the ecosystem itself, we're continuing to evolve, we're continuing to grow.

Second aspect to me that's extremely important, we're continuing to evolve our visibility framework. We've open sourced certain components of that, but we have a long, long way to go in terms of our ability to truly be able to see in the way in which we want to envision that transformation.

And then the last is truly ensuring that we have an invisible deployment framework, and what I mean by that is ensuring that as we change the ecosystem, as we roll out these new changes that automate our processes, that those become invisible to our developers.

They're not new changes where it's largely impactful to them, but hopefully it's new capabilities that are then available to them as new components arrive in the ecosystem. And ensuring that we can continue to drive that as the ecosystem becomes more and more complex is one of the challenges I know we're going to have as we continue to scale.

So with that, please feel free to connect with me, keep in touch with our journey. And with that, thank you very much.

And did I have time for questions or no? We done? Quick question. Yeah.

Q&A

Q: How is open source going through the legal challenges of open sourcing code, right? I know with a lot of large enterprise, maybe you can share some examples of that.

A: Yeah, absolutely. I'll actually give you an interesting note of why we ended up open sourcing.

As a global organization, and the way that capitalization works against open source and inner source, inner source is actually a far harder legal challenge in a global organization because you have to determine what they call operative use of the code base, which means how do you get taxed if you didn't make it, but you use it in a different country?

And so actually, we ended up open sourcing because it was easier than inner sourcing. So our strategy was actually to start with inner source, but it actually proved to be more legally challenging that we ended up with an open source method.

That said, there are several open challenges related to what open source does, who can use it, et cetera, and what the policies associated with that are. And right now we're working with legal tightly in terms of what the processes are because the major effect of open source is when you allow someone to contribute a change is when the legal kicks in.

And since we're still pretty new, a lot of our contributions are still internal even though we haven't faced that yet.

Q: Thanks.

A: Absolutely. Okay. All right, and I'm told I'm done. Thank you, everybody.