Michael Bueche

Thank you, Gene.

All right, slides. Good morning.

It's a pleasure for me to be here. If you had any idea of the week that I had to go through prior to getting here, you'd understand what a small miracle it was for me to get here.

What I'd like to do is share with you today USAA's DevOps journey we're currently on, but really kind of give it to you from a little bit of the softer side. And so I'm going to share with you the way we got our DevOps journey started, kind of the way we got motivation and things rolling inside.

A little bit about myself. I've been in and around IT for about 20 years. I spent 12 years as a developer on all kinds of technology from mainframe, I know we were joking about that this morning, a little bit in the web and mobile, et cetera.

I spent about three years with USAA as a technical fellow, which is, I guess, a nice way of saying executive architect. And then the last five years I've spent with USAA in executive management, leading large initiatives internally. And so about two months ago, I took over for the DevOps journey here at USAA.

Before I get started into that, let me talk a little bit about USAA, if you're not familiar. USAA was actually founded in 1922 by 25 military officers who were tired of not being able to get auto insurance because they moved frequently. And so they started their own company in San Antonio, Texas, and they have left for us the legacy that is USAA today.

Our mission really is to serve the military and their families in a host of integrated financial services. I'll talk a little bit more about that and some of the challenges that brings for us in IT.

Just our claims, our insurance group: we've done over 100,000 disaster claims. We've done over five million auto and property claims for our members, as well as we have one of the highest claim satisfaction ratings in the industry. We're one of the few companies left in the United States that's actually AAA rated, so very proud of that, and we're a Fortune 150 company. And so if you haven't heard of USAA, I like to joke that we're one of the few big companies that you've never heard of.

This slide's a little outdated, but all I could get approved by legal. But we have close to 27,000 employees now. We're approaching 11 million members, and our net worth is actually closer to about $30 billion today.

All right. So let's talk about what we're here to talk about. For those of you who are Princess Bride fans, you might like this one, right?

So I took over this DevOps journey about two months ago, and the first thing I started to think was, "Well, what do people think DevOps is? And let me go talk to people about DevOps."

And so I went around. We have about 3,400 employees in our IT department at USAA, and at least that many third-party contractors that supplement. So, big organization. As well as we support, we'll talk a little bit more about this, four major lines of business in four highly regulated industries.

And as I went around and talked to folks, I found out that not everybody had the common understanding of what exactly DevOps was internally. Why is that? Well, gosh, it's really simple, right?

So everybody understood the high-level principle, the fact that we're bringing development and operations together, but then it started to get really complicated and confusing.

So when we started this, we said, "Okay, what problem are we really trying to solve?" And so in 2008, we were a primarily waterfall shop, and our average project took about 158 days to complete. And I draw this diagram for you to say that a lot of times we put a lot of pressure on the developers to develop faster, when really the time is spent in the upfront, in the requirements, the analysis, and the design, and it's also spent on the back end in the testing and the release cycle.

And so really, what was our ideal state? We started on our Agile journey, and we obviously wanted to get the quick iterations. We wanted our business co-located with our developers, but we wanted to get stuff quickly to production so that we could get that quick feedback loop.

And so through our efforts in Agile since 2008, we've been able to get our time to market down to about 90 days. And so a vast improvement over the 158 where we started. But what we realized was that while we were iterating quicker and while we were co-located with our business and able to understand the requirements better, we were still struggling to get stuff to production. It was just taking too long. So hence the DevOps journey.

Just to give you some idea, we have a full host of insurance products from auto, motorcycle, property, renters, valuable property. We have a complete suite of banking products, so we have checking, savings, credit cards, consumer loans, et cetera. We also have a whole suite of investment products, including our own mutual funds, and we offer life insurance.

So really big businesses, 11 million customers. We have over 40 years of legacy technology at USAA that we had to modernize. And so it's a really big challenge.

And so where did we start? Well, the first thing we had to do, and this is what I talk about, the softer side: we really had to put together our elevator pitch internally. And I still remember my professor from MIT told me that what's often perceived as resistance is really lack of direction. And so really spending time putting our strategy together and getting everybody on the same page.

And so I went and talked to the different departments. They all had different ideas. I had to get them coalesced around a single idea that we could all agree upon. It needed to be aspirational. It needed to be something we said, "Yes, if we could achieve that, we've arrived." And then lastly, we had to stop using the term DevOps internally because it meant so many different things to different people. So we started calling it rapid release.

Let me give you some idea. So this is just a quick layout. I won't go through all the details, but essentially from the time a developer has finished their code, it goes through a series of environments before it goes to production. And our current release process prevented over 10,000 defects last year, so we're quite proud of it. Unfortunately, it's still taking 28 days.

So code is complete. We're not making any changes. We've got to get it to production. It takes almost a month.

So what we all coalesced around was this idea that, what if we could do it in seven? Now, I know we all want to get to release when ready, but again, we've got 40 years of legacy code. We've got 40 years of practices, disciplines, et cetera, that we're trying. So seven days was huge. And if we could go from 90 days time to market to 70, it's a huge win for both IT and the business.

So how did we approach it? Really, we had to start in layers. And what I mean by layers is we had to start taking small approaches that each one could stand on its own from a benefits perspective.

So we decided to start with test data management. And I'm sure I don't have to preach to all of you how important it is to have good test data. It's that thing that's really not very sexy but provides a lot of great value, and I'll talk a little bit about what we did here in a minute.

Then we decided to go and create a common automation framework that all of our teams could use. So we've got mainframe, we've got distributed, we've got mobile, we've got portal applications, et cetera, internal and externally facing. And so we had to get everybody on a common automation framework for this to work.

Next, we had to virtualize. And so we don't do a lot of our banking services. We have third-party providers that do that. We don't do our trades, et cetera. And so we needed to virtualize a lot of those services from those vendors that we rely upon.

The next thing we had to do was get away from our practice of running regression one time per release, and I'll talk a little bit more about how we ran it every day.

And then this is where we are in our journey, really getting to the point where we can get to continuous integration. And lastly, I'm going to talk about something that we refer to as developer-driven testing.

All right. So test data. We actually leverage a tool called Optim internally, and we use that tool to either pull data from production and de-identify it or generate data for our test suite. But what we did was create a tool that allowed two points of entry. One was our self-service tool, where a developer or a tester or even a business user can go in and say, "I want a member with a credit card, with an auto insurance policy," et cetera, and it will generate that data and put it into their environment for them.

Or we had our API, which would allow our automation tests to pull that data. So no longer did we have to keep these stale data sets with our automated testing. And as you can see, we had to build it for all of our lines of business.

The second thing we did was our automation framework. We decided to standardize on Selenium. And so we had some regression in UFT. We had some regression in Robotium for our Android apps. We had regression in Perfecto for our iOS apps, et cetera. And so we had to create this common framework. We chose Selenium from which to do that and run all of our automated tests through that.

I mentioned virtualization. So we started with those third-party services, the ones that were external to USAA, and we virtualized those. That allowed us to do two things. One is it allowed us to shorten our testing cycle, and second, it allowed us to not have to rely on those vendors to be able to make sure that their test services were always up and running for us.

One of the things developers will always tell you is, "I'll run the test, but they have to run fast." And so we had to leverage virtualization for that.

The next thing we did was once we got that in place, we went to our service-oriented architecture governance board and we said, "Look, if anybody's going to modify or create a new service, then we want them going through you to ensure that they've created the appropriate virtualization for those services." And so that was the stopgap to ensure that no new services got created without also having their virtualization as well.

Now this is where we are in our journey. And so we've created something we call daily release regression. What we used to do is all the projects, we run about 400 projects simultaneously every year, they would all come together to, not all 400, but a number of them would come together and say, "We're joining this release." And then we would wait till everybody was integrated, and then we would run our regression tests. And the unfortunate thing was, that was about two weeks before it was supposed to go to production.

And so there were all kinds of heroics that would come in trying to save the release and things that were broken at the last minute. And so I offer to you that if you're having to rely on heroics, your process is probably broken.

So we decided to create something we call daily release regression. And essentially what it does is it runs our regression suites, about 540 tests, and it goes very broad but not very deep, and essentially tests major functionality across our four lines of business. So it'll make sure that you can open a checking account, do a credit card transaction, that you can issue an insurance policy, et cetera.

And then what it does is when it runs at night, if any of the tests fail, it looks at any of the teams that have merged code within the last time that the regression ran and notifies them to let them know that something they introduced into the environment broke the build, essentially.

And so this is a baby step for us to get to continuous integration. And we've created these dashboards, kind of a screenshot here you can see, that give the developers an idea of what tests ran and what broke.

This is something that we're working on, and I think there's actually about five USAAers here at this conference who probably could talk more to this than I can. But essentially now the next step is taking that regression and tying it to the build. And so we're leveraging UrbanCode and a lot of their tools to help do our release planning, as well as get to our continuous integration. And I'll talk a little bit about some of the successes we've had with that in a minute.

And then lastly, I'm going to talk a little bit about developer-driven testing. So you notice I didn't say TDD, or test-driven development, and I wanted to talk a little bit. Again, I told you I was going to tell you about the softer side here.

We go to developers and we say, "You need to write more automated tests." And they go, "Oh, okay. Great." But the issue at USAA was the fact that we had been putting a lot of pressure on them to meet these time-to-market goals, drive your time to market. And so as a result, the developer often said, "Well, I don't have time to develop those tests because I have too much schedule pressure."

So because of that, the tests didn't get written, and we continued to do things our manual way.

So what we decided to do then was we wanted to start small. We went and found an adventurous team in the bank who was working on some of our credit card products and said, "Look, we'd like to use you as the example." What we've realized internally at USAA is there aren't a lot of people who are willing to stick their neck out and be the first one, but once someone sees big success, everybody else wants to jump on and follow.

And so we leveraged kind of our internal "me too" culture by picking that one team, getting them on board, and really doing a bottoms-up approach on our unit and integration testing. And so this is going very well so far.

And so I leave for you, I really think that it's important for us to stop saying "push testing to the left," because who wants anything pushed on them, right? I certainly don't. But if you tell them that it's developer-driven testing, in other words, the developer is responsible for ensuring that their code is tested appropriately, they can buy into that.

The other thing we did was kind of talk to the developers and say, "You don't really push your code over to the testers without testing it, right?" And of course, they say, "Well, no, I test it. Of course I do." And we say, "All we're asking you to do is test it in a standardized way." And everybody could kind of get on board with that.

But it's a big cultural shift internally, thinking that quality is an activity rather than quality is built all throughout our process. And so I'll talk a little bit more about how we got that going.

Another thing one of my professors used to say was, we as humans are really bad at learning things when there is a long delay between the time we make a decision and we see the outcome.

And I offer for you this example. If you go and put your hand on a stove, what happens? It burns our hand, right? And what do we do? We immediately take our hand off, and we learn, don't touch that hot stove again.

Well, imagine if you put your hand on that stove and your hand didn't burn until a week later. How likely would we be to associate the fact that us putting our hand on our stove was the cause? And so this same dynamic plays out within our software development lifecycle.

And so we said, how can we leverage short feedback loops to our development teams so that they can see leading indicators to quality rather than lagging indicators? Things like defects per 10,000 function points or number of high production defects: those are all lagging indicators and not a lot of value to the team.

So we created something we called the All Day Dashboard. And this All Day Dashboard was actually built by a group of interns who worked at USAA over the summer, so it's quite impressive. And what they do is they collect all of these leading indicators to quality, and they put them up for the projects in this dashboard. So things like, what's your test coverage? Have you performed code reviews? What's the velocity of change on a particular module? Has the regression run? Was the regression run successfully? Et cetera.

All of that's available in this All Day Dashboard for everyone to see in real time. And so we've seen this immediate impact that it's had on the developers to say, "Wow, if I don't do my code reviews, I can see this thing goes red on my dashboard. My project manager doesn't like it, and ultimately, my business customer doesn't like it." And so it started to change their behavior.

So how's it going? Well, as I mentioned, we're still on this journey. But so far, we've been doing quite well. One of the nice things about our layered approach was we've seen benefits all along the way.

For example, in our test data management, we built that originally for developers and testers. But what we didn't realize was that our business is actually now leveraging our test data management tool to generate data so they can build reports, so that they can build training environments for our customer service reps, et cetera. And so we actually have more business users using our test data management than developers and testers. So it's a pretty neat outcome. So we've seen huge value there.

But another one is that about three weeks ago, we implemented something we called independent release, and this is kind of the first foray of getting us back to our original goal of the seven days. And with independent release, if you go through an assessment and if your code is not necessarily going to touch anything that's high risk, we can fast-track you to production as long as you make sure and leverage the automated testing and automated builds process.

It's also hooked into that All Day Dashboard that I told you about. So these teams now are seeing significant acceleration to be able to build something for their business and put it into production quickly so that they can see those results.

So now I'm actually going to turn it over to Dibbe. Dibbe and I met about a year ago or so, and she's been around the DevOps journey for quite some time. I've learned quite a bit from her, and so I'm actually privileged to be up on the stage with her.

Dibbe Edwards

Okay. Thank you. Thank you very much. Thank you, Michael.

So we can come back to it? Oh, yeah. One second. Yeah, it's okay.

Okay, so first of all, let me start by introducing myself. I'm Dibbe Edwards. I've been with IBM a little over 20 years. I started as an intern when I was going to NC State and getting my master's degree as a developer, and then I have been really a part of our software development organization all my life and led development projects.

And about 15 years ago, I became an IBM executive. And in my roles as an IBM executive, I've led all different types of development organizations from the CICS Transaction Server team in Hursley, where we transformed that team from being a very waterfall development organization to now being a premier sort of Agile development leader in the Z community and inside of IBM, and they're doing some really cool things.

I've also led the WebSphere Application Server organization, and we went through a similar focus on automation. And about five years ago, I joined the Rational organization, where not only have we continued to transform our development processes, but also I get the really cool chance to use my tools to build my tools, as we call it, drink our own champagne. So we get to practice our own preaching, as it would be.

And then I also get to work with great clients like Michael, USAA, and many clients that are on a journey of transforming their current development infrastructure and application structure to leverage the cloud and enable things like DevOps. So it's a really, really awesome position.

So why I'm here today is just to share with you guys the journey that we've been on. And I appreciated from Gene the opportunity to come back and give you guys an update this year.

So let's start a little bit by looking at my organization. I'll put it like this: it's big and global. We have over 650 people in my personal organization. Obviously, IBM, our software organization, is much, much larger. And then of those 650, we have about 600 technical people that are either architecting, developing, testing, or deploying code.

We are, as you can see, very diverse. We have a lot of cultural different organizations that have come together through a number of acquisitions and also partners that we work with to do our development.

We organize ourselves into what we call feature teams or squads, which are responsible for, in each one of our releases, delivering user stories into our deliverables. And we do our best to try to keep those teams as co-located as possible, but the reality is, given skills and resources, we do have a dispersed sort of feature team structure.

Gene also asked me to comment a little on our management structure in IBM. We have had a tremendous amount of focus on trying to flatten our organization. In my team with the 600 people, we have two layers of management up to me in many of the areas, and then three in the areas that are a little bit more complex, where there's difficult business and client issues, and so we need an executive focus over those particular areas.

One of my main pain points and challenges that I'm facing right now in my organization is transforming a set of products with very diverse architectures, very diverse code bases and teams, trying to move those to a software-as-a-service model. And it is a huge undertaking, helping those developers understand how to write code differently, helping them understand changing their culture, changing the business processes associated with that, and allowing us and enabling us to move forward quickly.

And also, one of the other key things I think that's so different about software-as-a-service is that client experience that you need to be able to have. And so in support of all this, we're going through a huge transformation around what we call design-led development.

So let me give you guys a little peek into what we have going on here. So we've been at this for a long time across this multiple large, different organization, and this is basically the table that I showed you last year. And what we did is we basically started on the journey.

We took some advice that many of those Gene and those were talking about early. We identified sort of what our biggest pain points were. We set some baseline metrics, and we got started. And we focused every single sprint on doing retrospectives and improving our measurements and removing the impedance or the friction points that we had in our development process.

So this is what we've been really focusing on: time, removing waste, eliminating any sort of manual work. And now this year, we really are sort of flipping this a little bit more to talk about the scale of it and the amount of volume of work that we can actually push through our process. Because as we go from larger products to more teams working in parallel, we really have to make sure we have an excellent, scalable environment within our development organization.

So what I want to focus on here is I'm just going to touch on a few metrics that I think have been absolutely essential to our success.

So really the top part of this slide talks about the relationship that we have in our business. As I talk to clients, and I've spoken to probably 50 clients this year on this particular topic, this is one of the toughest nuts to crack, right? How do you make business decisions? How do you interlock between your lines of business and what the priorities are for your development organization to work on?

And as you can see, when we started this, we had a real big challenge in that area. It was basically a goat rodeo anytime we wanted to make a decision about what we were going to work on because we didn't talk to each other, we didn't stay interlocked, and we weren't aligned around our portfolio strategy, and we did not have a good prioritized backlog for the teams to work on.

So over a period of time, we have made significant improvements in that area. We use a lot of the Scaled Agile Framework principles to support that, and we now have aligned ourselves around a groomed backlog, which we manage on an ongoing basis. So making decisions is very, very easy for us.

The other area that I wanted to focus on is how many times we're now deploying per month. We used to have very manual processes, and over the last several years, we've completely and totally automated everything about our deployment. We use that in our testing environments. We use that when we go into production, and it has tremendously improved our velocity and our ability to be efficient in the development organization.

As it relates to test, we were also a very manual shop. We did all of our testing both in the development time in our developers, and our unit and function test was not automated. We've had a tremendous focus on that. But also we've gotten a lot of benefit out of what we call our customer scenario testing being automated.

It used to take us in each one of our sprints up to 30 hours to actually get our environment set up before our testers could actually even start doing any testing. And what we did is we invested in creating patterns and the definition of our different applications and the different environments in which we need to test in as a set of patterns.

So now our testers can deploy those patterns into a virtualized environment and test much easier. So that 30 hours that they spent is now three hours getting the environment set up and 27 hours actually testing.

So the cool thing about that is that we used to develop in one sprint, test in the next sprint. Now we're able to do development and test in one sprint and exit with good production-level quality. And that is a huge accomplishment because that now gives us the opportunity to shift what we're working on depending on what the market demands are, and that's a very powerful thing to do.

One of the other key focus areas that we have within our business, and I hear this from other clients as well, is sort of this focus on innovation and maintenance. I had in the earlier slides one of our key challenges is overcoming what we call keep-the-lights-on type of investment. How do we make sure we minimize that or make it as efficient as possible so we can focus on innovation? And we have continued to do that. This is one of the key business metrics that I am measured on from the success of this project.

So you can see there that we've been able to make quite a shift from being very sort of maintenance-oriented to being much more innovation-oriented.

This is the, I think Gene called it the sausage maker, behind that slide that you see there. And this is really a picture of the delivery pipeline that we use within my organization. We've had this in place and it's continued to evolve over the last three years, and we've just matured it. We've populated it with tons more automated tests. We've got a really awesome virtualized infrastructure that runs underneath this. And it's just something that has now become the heartbeat of our organization.

What we really focused on initially was sort of getting it right for really one team. Getting it right. And now we have this scaled up where we have 14 different instances of this pipeline running concurrently for different code streams and teams that we have in our organization.

We have it completely automated from a quality gates perspective, and we have a whole path from the time that a developer commits code all the way to deployment into a production-like environment, automated for us to do all of our smoke testing and early testing.

You can also see that we don't only use our tools within IBM. We also use many open source tools that help us to facilitate our collaboration and really create an environment which is very much like the ones that our clients use.

And the little diagram on the bottom there in the right-hand corner is sort of what we use as like our daily dashboard type of thing, where this is really what the team uses to drive its focus on the build, the health of the build, where the problems are. And they can hover over it, and it'll actually show them the test cases that have failed, et cetera. So it becomes really the heartbeat of the organization.

So as we've gone through this transformation, we're always looking and striving for how we can get better. And so we've been trying to come up with new ideas to help identify areas for improvement.

So let me ask a question. How many of you have ever heard of Watson, IBM Watson?

So Watson was a computer that got onto Jeopardy! and they gave it a whole bunch of information, and it beat some of the best ever Jeopardy! contestants. And basically what Watson is, is a cognitive learning computer and a machine, and you feed it information, structured and unstructured content, and then you can ask it questions and it will share you, and it will provide data, and it will make connections and correlations between things that you had never done before.

So one of the guys on my team had this great idea. He said, "You know what, Dibbe, we have a whole bunch of data. Why don't we involve Watson in this and see what Watson can tell us?"

So we just did an experiment, and it actually was pretty interesting. We fed into it our sprint data from one sprint. You can see 562 test cases, 1,106 work items and defects, and it came back with some really sort of interesting information.

This is a very MVP type of thing, a skateboard for sure. But the idea here is it was able to very quickly, literally within a day, help us see some correlations between things that we had not seen before.

So first of all, it helped us really highlight our test coverage to make sure we were comfortable and that we actually understood the relative test case coverage on platform database, as well as operating system. It showed a high correlation between the fact that our database and operating systems and certain components had a higher propensity to have defects. So that gave us areas to go focus in.

And then anytime we have a defect that's open that doesn't result in an actual code change, we view that as an invalid defect. There's something wrong. It could be that we need to change documentation, improve the usability, but it is a problem for us.

And we were able to identify lots of invalid defects and actually be able to dig right in and focus right in on the things that we needed to do to eliminate those. And you can see here there's a link at the bottom to an experience report that the guy that led this on my team has written. So I encourage you to go out and take a look at that.

So Gene asked me to write up a slide about what I was most worried about, and I said, "Well, I can't really write a book, and I've only got about 10 minutes."

I'm just teasing. Actually, there are things I worry about, but they're mostly my daughter being away at college and my husband and all that kind of good stuff. But I am very excited about some things in the next year.

I think this thing with Watson has an opportunity to be cool. I've never really bought into predictive types of things. I'm much more of a show me based on experience, let me make decisions. But this thing was pretty cool.

And then the other thing that I am really excited about is what is going on at IBM across our entire company. We have more buy-in to change than I've ever seen. You may all know we kind of restructured ourself at the beginning of the year, and we have the highest level of support for transformation that I've ever seen. Everything from how we measure our executives, to how we promote our employees, to the leadership training that we have, the whole process, and this design thinking, and breaking a whole bunch of inertia, and then being really open about our technology, embracing open, and then also focusing a lot on experimentation.

So just quickly, wanted to say that one of the things we are focused on is making these practices and some of the things that I've been doing, as well as other parts of IBM have been doing around what's being called the IBM Garage Method, where we're sharing our best practices, our methodology, how we're doing this design thinking, how we're driving this transformation, and how we are scaling that up. So I invite you to come by our booth and talk to some of our folks.

I have top five takeaways here. I think the main thing I would focus on is just treat your delivery pipeline like it's the heartbeat of your team and treat it as your own. It's something that needs a lot of nurturing. It needs focus. It needs really good people working on it. Okay? And your team needs to view it as a real essential part of the health of the organization.

So Gene also asked us to focus on some areas where we need help. So Michael and I combined ours here. And so as you can see here, the automated impact analysis, getting away from kind of rules-based and doing more risk-based analysis and helping us, could really use some help if some folks have some good strategies there, as well as dependency.

We've grown up on a number of different systems from dev and test, et cetera, and so really kind of figuring out how we can get that to understand our dependencies all the way through. And then, of course, getting from the seven days to release when ready.

Michael Bueche

Excellent. And then for me, I'd love to talk to folks that are shifting to the cloud. We're going through this huge transformation, and other thoughts and advice from people that are going through some of that would be also very helpful.

Dibbe Edwards

And then I have another problem. I have some beer, and I have some music, and I have some food that I'd like to share with all of you from an IBM perspective. So I'd like to invite you all to come and join us at our local garage here that we have in San Francisco. This is where we're bringing a lot of our clients to show them some of our best practices, showing them how to do design thinking, and we're going to have a really good time tonight.

You can stop by the booth, or you can just meet us out front, and we're going to take some trolleys up to the garage location. So I thank you very much, and Michael and I will be around, and we'll be at the garage tonight. So thank you.

Michael Bueche

Thank you, Michael.

Thank you.

Thank you, Gene, from IBM.

Thank you, Gene. Appreciate it.