Don Almeida

[00:00:12.690] Hello, everyone. Hope you are having a great time here at this summit, and thanks to Gene for weaving in a great group of presenters at this year's DevOps Summit. Some really high-quality information and people, people who are changing minds and hearts and the very fabric of how we work. We are thrilled to share our journey, our DevOps journey.

[00:00:36.460] My name is Don, Don Almeida, and I'm the manager at the Service Benefit Plan Administration Services Corporation, which is an affiliate of CareFirst, the largest insurer in the state of Maryland. I'll be tag-teaming with Rajat Sud, our principal engineer and chief evangelist at the DevOps Center of Excellence.

[00:01:01.220] To help set the context of our journey, I thought I would let you know something about our business. We are a healthcare technology organization providing centralized operations and technology solutions for a national health plan with over 5.5 million members globally. We are headquartered in the Washington, D.C. area, and our team includes 1,300 individuals. And we are all dedicated to transforming healthcare technology in service of our customers' goals for improved member experience and health outcomes.

[00:01:33.000] SBP ASC ensures seamless delivery of claims, enrollment, customer service capabilities, and we deliver data and analytics that enable critical business decisions.

[00:01:45.180] Now, we do live in a heavily compliance-centric industry where there have been specialist roles carved out to support various business functions, even within the business of software production and delivery, from engineers to analysts to configuration management and environment management specialists and infosec management specialists, all working in concert to orchestrate the release all the way to production.

[00:02:12.240] So very early on in our journey, we understood and charted out what has now become a value stream analysis where we captured multiple handoffs and wait states. And as you can see, there were quite a few, each burdened with the creation or management of auditable artifacts in various ticketing systems. And what's even more noticeable are the amounts of wait states, where individuals are waiting on others in other job functions to complete their tasks while they embarked on their own.

[00:02:47.920] To streamline this was a challenge, and that proved to be the genesis of our DevOps Center of Excellence, essentially to reduce wait times and handoffs along with realizing goals through focus areas, essentially go from what you see right now to this, where we could leverage automation to reduce handoffs and wait states and, more importantly, promote this then-novel concept of delivery team instead of siloed roles and job functions.

[00:03:24.880] So as we just mentioned, our Center of Excellence was established to streamline the software delivery life cycle, thereby helping the organization scale effectively. Scale effectively by delivering features to our customers with improved quality, baking in automation in order to reduce wait times and break down silos. And we did this by focusing on these areas on the right side.

[00:03:52.240] Consistent and repeatable practices: for instance, leveraging automated unit testing, CI as a practice to create builds once and deploy anywhere, providing the ability to have effective branching strategies for parallel development, and fostering the treatment of everything as code, which means applying consistency and versioning, controlling all artifacts in addition to source code, that compose an application.

[00:04:21.940] And then take these artifacts and subject them to optimized and predictable flow, streamlining processes and automating handoffs with the goal to eliminate them. And this is facilitated via improved feedback loops. We want to make it safe for teams to make changes, and that can be done by providing feedback faster. For instance, baking in code quality gates into the CI build process itself to halt the process if the thresholds are not met. Once deployed, functional test suites automated and executed on a cadence. And most importantly, features are made visible to teams so that they can react.

[00:05:03.080] And the culture part: we continue to evangelize safe zones where we encourage continuous learning and experimentation via program-level retrospectives, blameless postmortems, and leadership actions. And then tying this into our discussion today, how we have empowered teams to make good on these initiatives by in effect shining a spotlight on their practices, showing them introspect, adapt, and improve.

[00:05:35.000] As we share an excerpt of one such insight, first, a few details about our Center of Excellence empowerment engagement model. First things first, we are obsessive about our platform, which is composed of industry-standard tools that provide CI/CD, code quality scans, testing, binary management, planning, and source code management. We provide a guaranteed uptime to our customers, the various lines of businesses, and we have well-structured training guides and an enterprise DevOps playbook that provide easy how-tos for everyone interested in learning about adopting and even contributing to the betterment of our platform.

[00:06:20.556] Given the scale of work that we perform and the new initiatives that the business undertakes, the other arm of our Center of Excellence is arguably the busiest. From rapidly onboarding new teams that are deploying using our existing tech stack, to working alongside others that are adopting new novel technologies such as low-code, no-code platforms, we rigorously apply DevOps and systems thinking to planning, development, deployment, testing, and monitoring practices.

[00:06:53.356] Whether we have to understand and configure the technology-native DevOps capabilities or whether we have to customize our platform, the goal is to foster consistent and repeatable practices while educating and empowering teams to improve their DevOps maturity. And as we deploy these capabilities, we turn around and ensure metrics insights are being effectively used, and also if there are any gaps and opportunities for improvement. Finally, these opportunities are wrapped into an enterprise-wide initiative.

[00:07:28.816] Initiatives such as these surrounding the improvement of the stability of our pipelines and plugging in any gaps that may exist in them, all with the intent to make for consistent and repeatable practices while reducing manual, time-consuming wait times. Promoting improved development practices such as pull request declarations and increasing the scope and occurrences of peer reviews. Increasing code quality awareness to remediate failed quality gates and improve our unit test and code coverage.

[00:08:03.016] These are just a sampling of our initiatives that we have embarked upon in the past and present, and to a large extent have demonstrably moved the needle, which brings us to a keynote of this conversation, which I will hand over to Rajat.

Rajat Sud

[00:08:22.296] Thanks, Don. So as we embarked upon maturing our practices and were tasked with improving the adoption rate of the various capabilities that our DevOps platform offered, we wanted to approach it in a very deliberate, scientific manner. And to do this, we really started analyzing the various scan reports offered by our disparate tool chain and realized that this required a fair bit of manual processing to really convert them into actionable insights that our teams, essentially our customers, could avail of.

[00:09:01.356] So fortunately, given that all our tools expose all of their data via APIs, we were able to probe and ingest them into a BI tool that allowed us some degree of creativity on how best to construct the insights, how best to visualize these insights into opportunities, and most importantly, in true DevOps fashion, switch from once-quarterly or once-monthly insights that require two to three person-days to produce to once-daily and, in some cases, even real-time sets of insights.

[00:09:42.796] We'll talk about one such set of insights and how we've been able to wrap them into company-wide initiatives, all the while having some fun while doing it in terms of drives, events, and DevOps days. But before doing that, we wanted to share some of the initial challenges.

[00:10:04.216] Publishing insights are fine and dandy, but if there's no one looking at the other end, then guess what? You're just publishing yet another pretty report. And just like many of you, having been in the business for many decades, we don't consider ourselves unicorns. But what we are are thoroughbreds of well-seasoned professionals who have gotten us this far, and with the right levers, can propel us even to newer heights.

[00:10:36.156] And as a rather large organization, we do have an influx of well-experienced talent from various parts of the country. So it was imperative that we conceive and publicize forums where our team members, new and old, are in a position to share their experiences. So we leveraged levers from the bottom up in terms of adoption drives, building a DevOps community of practice in a speakeasy format for folks to contribute and discuss in a no-judgment zone, all the while making it fun, all facilitated by our DevOps Center of Excellence.

[00:11:16.936] On the other side, getting the alignment of our peers in leadership positions was also critical. So we also conceived of a gentle top-down lever to really up the ante in adoption, in baking into individual performance reviews up and down the organization chain the responsibility to improve the DevOps adoption rate.

[00:11:42.276] And once these levers were in place, so began a cultural shift led by the production and distribution of actionable, just-in-time, just-enough insights, providing clear-cut direction on where the current maturity level lay, where it should be, and what steps needed to be accomplished, and more importantly, made habits out of the need to improve. And all this in conjunction with providing forums for expressing feedback, opportunities, challenges, and general thought leadership.

[00:12:23.352] So enough talk. Here's how we make the rubber hit the road. On to our headline. One such initiative that got the ball rolling and put us on a fast track to maturity: Got Score? Essentially, what we built is a score, a single score associated with every application based on how that application implemented the various DevOps capabilities. Capabilities such as producing automated builds, deployments, tests, code quality scans, et cetera, and just as importantly, how stable those capabilities were. And then wrapped those insights into a single aggregated numeric score.

[00:13:15.591] This score, at a glance, provided an objectified indicator on the DevOps disposition. Call it the DevOps health of an application, which in turn was reflective of the team that contributed code towards it.

[00:13:34.312] The instant we published this, we noticed a, some may say, visceral reaction to those with scores on the extremities. The ones with the highest score, of course, walked with an increased swagger, while the low scorers instantly wanted to ask why. And in doing so, rapidly explored our DevOps offerings and began onboarding them. And about the leadership, let's just say who doesn't love a good old-fashioned healthy competition among teams and lines of businesses to get the highest score, right?

[00:14:14.292] And here's an excerpt of the methodology on how we derived the scores. Like we just mentioned, we automatically award points to every application based on their adoption of our platform's DevOps capabilities. So for instance, if they have implemented automated tests to execute as a part of their pipelines, they get 10 points. And from a practice perspective, implementing the capability is not enough, but they have to effectively use it, and that is measured and reported in terms of stability.

[00:14:57.012] If their builds and deployments and tests stability exceed 80%, they get an additional five points. If between 50% and 80%, they just get three points. An unstable build or deployment will not net them any points and will represent an opportunity for improvement. And as we continually add new capabilities, we refactor this methodology to provide that insight to our team to ensure they've implemented it and, more importantly, how effectively they've implemented it.

[00:15:39.112] And this is where all the action is. As you can see, we first draw up the composition of our various lines of businesses, how many applications each support. And then on the top right are the aggregated scores. Remember, we spoke about promoting a healthy competition between lines of businesses? So I would imagine this is where our senior leadership spends their time.

[00:16:08.312] In the lower left are the details for every application. We do apologize for the obfuscation of company-specific proprietary information, but hopefully you can imagine that every line represents an application and what its score is. And clicking through the line refreshes the widgets on the lower right to reflect the application score and the health of the various DevOps measures. In this dashboard, specifically, the health of the CI/CD measures, along with the code quality measures at the bottom, complete with actionable letter grades.

[00:16:55.892] And this translates into an actionable, quantifiable plan with easy-to-digest insights that have paved the way to create action plans and facilitate conversations about how best to improve the applications, thereby the teams, percolating down to the people composing them and their DevOps capabilities. And also to bake in that oft-updated feedback loop where improvement initiatives can be recognized in various organization-wide forums, while degradations can be course-corrected.

[00:17:35.092] And worth mentioning again that behind the scenes, as we onboard new DevOps capabilities, most recently especially in the area of automated security, we are constantly refactoring the methodology to account for all of those. Not sure if you picked it up in the earlier screenshot, but our maximum attainable DevOps score across all applications currently stands at a maximum score of 90.

[00:18:11.456] So hopefully this illustration has given you some food for thought on how to effectively implement and scale existing and new DevOps capabilities across your portfolio of applications, while also having some fun along the way by appealing to the competitive streak in your colleagues. And more importantly, educating them and bringing them along for the ride.

[00:18:37.456] And now, this provided somewhat of a blueprint on how other business functions have used a similar model to further their maturity efforts and enable a somewhat apples-to-apples comparison of teams with the intent to identify opportunities or hail and share successes and share them with the broader organization. Essentially start with the least common denominator, the deployable unit, to compute the effectiveness of adoption. Then aggregate it up to the team to gauge efficiencies, the product to gauge predictability and quality, the portfolio or program to determine effective resourcing, and then finally all the way up to the enterprise for quantifiably demonstrating maturity. And in doing so periodically, be it yearly, monthly, or again, in true DevOps fashion, in real time, demonstrably moving the needle in the positive direction.

[00:19:48.096] So in conclusion, here's what we are seeking from this wonderful community. Even though a data-driven approach brings about a ton of insights, we do recognize the human aspect of DevOps and that a lot of opportunities are ripe for the taking in nuanced conversations. We'd love to hear about how you've elicited those insights about those conversations. What forums have worked in your experiences to facilitate those insights, especially centered around bringing your colleagues along for the ride, especially well-tenured employees that have been consistently performing in the same role over the past many years?

[00:20:33.516] We also have a keen interest in hearing about tackling the burden of technical debt, especially for older applications developed and in production the past five or six years. Every time we've introduced a new tool, with that comes new sets of insights. How have you effectively prioritized these insights? And finally, we'd especially love to hear, out of vested interest, from those of you in regulated industries, how do you move the needle for processes and policies, especially those inherited from the pre-DevOps world?

[00:21:15.796] And with that, we thank you for lending an ear and look forward to learning even more from your experiences.