DevSecOps: Beyond Theory, a Strategy with Real Customer Impact
Our success story is based on the joint work carried out between BCP (Peru's largest bank with more than 9.7 million customers which belongs to the Credito Group and one of the most important banks in South America, BCP’s Memory) and NTT Data (IT services company part of NTT Group that is in the top 6 at the level world, number 1 in Peru and with a presence in more than 80 countries https://www.nttdata.com) in the implementation of DevSecOps as a cultural and technological paradigm in BCP, demonstrating how BCP evolved from a merely technical approach to an approach of strategy that served him to leverage and improve the speed of his business.
Chapters
Full transcript
The complete talk, organized by section.
BCP Brand Video
At BCP, we are convinced that digitalization is the best way to continue offering our clients wow experiences.
Through our mobile application, Yape, we have transformed payment processes, allowing users to transfer through their phone number. Yape users include more than four million individuals and small businesses.
We are also looking for new user-friendly ways to serve our clients and answer their questions. Through Clara, our social network client service bot, clients can reach us without the need to call our contact center. The different functions we offer through mobile banking are constantly evolving.
We have also brought our products and services to where our clients are, eliminating the need to visit a branch.
For companies, we were the first bank to launch in the market the service of digital invoices in installments and digital bill of exchange, adapting to the situation and solving our clients' pain points. This all reflects a client-centric strategy focused on improving user experience and user interaction.
We are constantly learning more about clients' needs by leveraging data and analytics while optimizing and streamlining our IT processes through open architecture.
In parallel, we protect our clients by developing new capacities to control digital risk as we adopt the highest standards for cybersecurity.
These developments are driven by our quest for constant innovation, which places our clients at the center of our decision-making by offering solutions that help them transform their plans into reality.
---
NTT Data Brand Video
A song can be a series of notes in harmony or become a true anthem for a generation.
A math formula can be a sketch on a blackboard or be the spark for the next revolutionary idea. An initiative can be left alone, unnoticed, or be discovered to transform the future.
When you put your heart into it, everything changes.
That's what we do, and now we can do it even better. With new capabilities and the same passion as the very first day. With a new reaching power and the real closeness that you know us for. Envisioning the challenges of the future and facing the needs of today. Together, we are more than ready. More technology, more innovation, more diversity, more talent, more commitment, one single heartbeat.
Everis is now called NTT Data. Future at heart.
---
Maria Luisa Polo
Hi, everyone. My name is Maria Luisa Polo, and I'm the head of software engineering at BCP, the largest bank in Peru. And I'm very happy to have the opportunity to tell you about our experience.
In 2019, I remember that I came to the DevOps Summit event in Las Vegas to learn from all of you how to start DevSecOps in BCP. Today, a few years later, we want to share with you our path to DevSecOps and the achievements we have made.
A little more about BCP.
At BCP, we have a very clear purpose: to be allies of our clients, our employees, and of course, our country, transforming their plans into reality.
In 2019, we began banking new segments using new data and analytics models with more than 9.7 million customers. Today, we are more than 16,000 employees organized in the agile way in tribes and centers of excellence.
At BCP, we want to combine the best of the banking world with the best practices of the tech world to build a digital bank. Today, 60% of our transactions are digital.
Our transformation aims to create distinctive experiences for our customers, supported by technology and agility. We have more than 70 technologies and digital products, such as Yape, our digital wallet, Cocos and Lucas mobile banking, and TiVA, our personal investment app.
BCP is always looking for excellence, and we have made this route to DevOps together with an ally. Luis, tell us a little bit more.
---
Luis Alberto Guevara Sandoval
Thank you, Maria Luisa. Hello, I'm Luis Guevara, senior technology manager at NTT Data, the sixth company in IT services worldwide and number one in Peru and Latin America.
NTT Data has more than 140,000 employees, 15 high-performance centers in distributed technologies, and is present in more than 50 countries.
We are part of NTT Group, a Japanese holding company with operations all over the world, and we are proud to say that 80% of the top 100 Fortune companies choose NTT Data as a strategic partner to work on their digital transformation.
As Maria Luisa said, BCP and NTT Data are strategic partners who have walked this path together. Now, Erika can tell us how this amazing journey began.
---
Erika León-Ravinez
Hi, everyone. I am Erika León-Ravinez, and I am a DevSecOps and Resilience Tribe Leader in BCP.
Digital transformation has brought challenges worldwide, such as speed and adaptability. Together, BCP and NTT Data have joined forces to improve the delivery of value to the customer and the speed of response to market demands.
Today, more than 4,000 employees use the pipeline that we have available. This is due to the commitment of the adoption promoters, who help us by incorporating and disseminating their use in daily practices.
The creation of the DevSecOps Academy managed to train more than 300 people so far.
The implementation of a set of tools allowed storing more than 60 million lines of code in more than 70 technologies, generating with all these initiatives a reduction in the time of passing off to production from 38 to 5.5 days.
Different approaches bring amazing results, and as you can see, this was not a minor challenge to tackle.
The main achievements I would like to mention are: multiply by three the number of deployments, reduce by 77% the delivery time, and increase our frequency indicator to high.
These are great achievements from where we started. But getting into the result was not easy. In fact, we're about to tell you how we managed this in three main moments.
The origin of the approach, and how it worked at BCP in a more traditional way at the process level and with technology more oriented toward application and not so much toward the development and deployment process.
The focus on technology and how we started to accelerate implementation of DevSecOps, building a new toolchain to automate the application development and operation cycle, and creating technological capabilities and templates for the automation of deployment in several technologies.
Finally, the change of the strategy with scalable visions and results, mainly looking forward on the involvement of processes and people as agents of change.
Our journey started in 2016, and the entire software development process was manual from end to end. We had several documents to fill out, approvals at each stage, manual dependency that could take between three to six months to complete features.
In 2017, we started with a DevSecOps approach, but out of the three pillars, we only focused on the technology one.
In fact, that's what we did. We managed to accelerate the delivery of the product and the value to the market.
We started gaining many financial and technological resources to be able to implement a base platform that allowed us to meet the demand of the different projects and their different technological characteristics.
Precisely because of the different technological characteristics and automation needs, our work initially focused on the implementation of a framework for the automation of continuous integration and deployment processes. With this, we managed to generate many efficiencies and reduce a large part of the integration and continuous deployment process.
But as we kept the same structure and process and culture, the results were not as we expected.
That is why in 2019, we changed our strategy. We took the current documentation, the experience of other institutions, and what we have learned in events. We reformulated them in three pillars: process, technology, and culture.
In process, we thought about automating all the steps, eliminating dependency, and generating autonomy within the same team.
In technology, we made use cases available to encourage the adoption of the tools deployed.
In culture, we invested a lot in training and communication. We also obtained the sponsorship of the C-level, which helped us speed up our adoption strategy.
To measure our progress, we established three OKRs: reduce implementation time, increase the frequency of production passes, and improve software quality.
With all the products we had, we began to categorize them into those that could do DevSecOps and those that could show results quickly, which we call drivers of change.
For the part of the process, we defined our exploration. Next, we prioritized small but highly disruptive change.
On the technology side, we promoted the available tools and pipelines to start the adoption process with our promoters and define the DevSecOps roadmap.
---
Luis Alberto Guevara Sandoval
With the new strategy, as Erika said, we needed to redefine the aspirational process. And to achieve that, we had to detect and eliminate the bottlenecks that were not generating value, lack of agile practices on DevSecOps, and the lack of autonomy in teams.
To achieve this, we focused on two mechanisms to do it. Starting from two opposite points would help us to achieve the goal.
First, the process improvement, challenging each of the activities and points of potential improvement in the existing process to make it more efficient.
Second, the redesign of the process. We started by thinking how we would carry out the process internally without considering the aspects currently existing in the organization.
With both approaches, we were able to establish a first improved version of the software development process, which we began to use through an MVP for some business teams to validate its feasibility. This showed a significant improvement in the indicator results.
And well, this new approach allowed us to continue growing capabilities, such as extending to new technologies. We began to grow in capabilities to handle the automation of new programming languages, new extensions and capabilities in the DevSecOps platform, and the inclusion of security capabilities within the automation flows, reaching more than 100 applications in the bank.
It also allowed us to further mature capabilities in existing technologies and achieve it with the stability that the company demands. All of this through the implementation of monitoring in the platform, ensuring availability levels of more than 99%.
And finally, to extend the practice not only on continuous integration and deployment process, but also on continuous delivery, monitoring, and feedback. These pillars are a great point of extension to support and achieve the operation in search of the highest level of automation.
---
Erika León-Ravinez
As Luis told us, by making available our technical capabilities, we were able to adopt DevSecOps practices. We would like to share with you some of our achievements.
In continuous integration practices, we have enabled break the build of the application code quality analysis for PaaS. We have automated the code integration task, reducing the risk while merging it in the code, and we have automated some of the notification and the steps of the software development process to facilitate the developer deployment.
In the security practices in the pipeline, we enabled break-the-build static analysis of application code and automated the use of keys through the Vault applications.
In continuous testing, we enabled a framework, giving us a clear guideline for automation, prioritization, and more. We achieved between 60% to 80% in the automation of testing cases in many applications, reducing testing execution time between 40% to 80%, and increasing 100% the testing coverage that helped us to reduce issues in production environment.
In continuous deployment, we have enabled technologies to reduce the time it takes us to bring products from development to production, delivering value to our customers across multiple platforms.
And finally, we have enabled capabilities based on SRE, site reliability engineering practices, to ensure the reliability of the bank's key platforms. These capabilities are distributed transversally, so the practices can be adopted by the business team in order to improve the availability of applications.
We still need to make more technological capabilities available in our DevOps pipeline. So our next challenges are visual testing, ensuring the documentation of the application is complete and updated, applying shift left when provisioning infrastructure resources, including monitoring configuration in the code pipelines, and implementing and deploying DataOps.
---
Luis Alberto Guevara Sandoval
But not everything was about processes and tech as a part of the strategy. We also focused on people and the culture, promoting different actions for a cultural change in a sustainable way over time.
This model based on culture and people was relevant to bring the DevSecOps practice to the expected level. Really focusing on the needs of the people who will use the process and technology is just what allowed this change to take place in the organization. Erika, please tell us about that.
---
Erika León-Ravinez
Of course, Luis. As the theory says, culture is the main pillar to work on in every company that wants to adopt DevSecOps. That is why we created four main axes that we use in any new practice, tool, or process. They are comprehension, leadership, recognition, and talent.
Something that worked very well for us was giving visibility on what has been achieved and the value the company gained. We also meant we created promoters, who are people from the same business team who promote adoption of DevSecOps, and we share the results for our OKRs.
In leadership actions, the commitment and sponsorship of the C-level is key. Recognizing each application and team that adopts DevSecOps and giving visibility to this achievement to everyone helped to motivate the rest of the team, showing the benefits that they could also achieve and reinforcing the behaviors that we expected.
Finally, we also had an important talent challenge. And to face it, we invested a lot in training because it is not easy to find professionals with these skills in Peru. We created the training spaces, the virtual academy, and the follow-up to ensure that we had the necessary talent.
Today, understanding the challenge and having made great progress in this, we are able to share our experience and knowledge with other companies, facilitating their progress on the path of DevSecOps. That is why we created the DevSecOps Day, that brought together many people from Latin America for free to share our challenges and achievements.
As you can see, we were more than 1,600 people sharing knowledge. We found many people who want to collaborate to improve the technological level in Peru, and not only for DevSecOps. This is one of the main reasons that led us to launch our technology community. Maria Luisa will tell you shortly.
---
Maria Luisa Polo
Thanks, Erika.
In September 2021, BCP launched a tech community called Punto Impacto BCP. Its mission is to promote the development of tech solutions to solve social necessities. Punto Impacto seeks to give a positive contribution by capitalizing the talent and knowledge of digital professionals. Our goal is to be a space for ideation and innovation.
From the beginning, we set three main objectives aligned to such mission.
First, build a self-organized and dynamic community. We want to enable the members of our community to collaborate in the development of ideas and projects driven by technology. To do so, we have started by creating content that ignites discussions and attracts people and organizations driven by our mission. So far, we have achieved the following: 2,300 community members.
Second, build tech solutions that improve financial education and financial inclusion.
And third, use the resources and capabilities of BCP to promote technology as an enabler of change.
Our community has been growing steadily, and we expect to see innovative ideas emerging from it very soon.
We want you to be part of it. Just scan the QR code or find us on LinkedIn as bcp.impacto. You will find access to specialized content on financial inclusion and education, suggestions and tech resources, events in favor of the development of our community, and much more. We are counting on you.
Thank you very much for this space to share knowledge. And for closing, I leave you with Erika and Luis to tell you about our learnings.
---
Erika León-Ravinez and Luis Alberto Guevara Sandoval
Well, everything we have achieved in this process has brought us lessons and conclusions that we would like to share with you.
First, do not just focus on technology. Also focus on processes and people. Obviously, technology is the engine of digital change, but there is no change without people and changing the way they work.
The commitment of the C-level, sponsorships, and allies are important on every strategy process.
Starting with one or two applications is fine. This is not a big bang.
And finally, it's very important to measure all the process and identify the impact of every change you make. This will make the whole difference.
We hope you enjoyed the presentation as much as we did. This is all for now, and thank you.
I hope our journey will be useful for those who are starting on this road. And we look forward to talking to you soon.
We leave you our LinkedIn so we can share with you more knowledge and experience. See you soon.