Michael Hrenko

We're not a glamorous company. Blue Shield of California isn't a Facebook. It's not a Google. We're not pioneering technology as we know it.

What we are is reality. We're how most companies, I think, behave on a day-to-day basis. And that's really my motivation to coming. One of my colleagues was sitting in a DevOps presentation and hearing all the wonderful use cases about how Netflix is doing all this amazing stuff, and Amazon, and all this great stuff. And he raised his hand, and he says, "Tell me about the health insurance companies. Tell me about the banks. Tell me about these huge institutions with more legacy technologies, and how are they transforming?"

So at Blue Shield, we've started a transformation of our own, and I wanted to have the opportunity to share it with everyone here.

A little bit about Blue Shield of California. We are not Blue Cross Blue Shield. We are Blue Shield of California. We are a not-for-profit entity that operates within California.

And that first bullet point I kind of like: we were established in 1939. Again, this gives you a sense of the nature of the company, the roots. There weren't computers in 1939. Everything was manual. Everything was paper. And so we've been around a long time, and consequently, we have a certain mindset, a certain approach to how we do things. And oftentimes, it's steeped in that 1939 behavior.

We serve three and a half million members across California, 65,000 physicians, over $9 billion in revenue, so we're a fairly sizable organization. I took this all from Wikipedia, so it's probably true.

The second-to-last bullet point there: over 300 deployable application components, and that's just around our Facets ecosystem. I don't know how familiar you all are with healthcare, but Facets is an off-the-shelf claims and adjudication system. It handles all of our workflow. It's made by a company called TriZetto. They developed it in the early '90s. It's not the most sophisticated application in terms of lending itself to automation.

But there are a lot of pieces and parts, a lot of dependencies, a lot of technologies, capabilities, and interrelated components. And of course, we have large databases with personally identifiable health information. And when you're dealing with health information, that comes with a lot of extra baggage, and weight, and care, and scrutiny. And so it doesn't always permit us to move as quickly as we would like.

And then, of course, I have to mention regulations, mandates, government. That's what healthcare is about, and we have to comply with that, and we have to move at a rate that's sort of consistent with those mandates. So we're operating on a lot of different fronts, and it does add a lot of complication. It doesn't really lend itself to agility.

So a little bit about me. I am a lead automation engineer. I work within our infrastructure group, but I've jumped around a little bit within Blue Shield. I've been with the company for about six years, and during that time, I was brought in on the SCRM QA team to help implement our defect tracking solution.

I moved over to our release management team, which was very interesting. That gave me sort of the insight to the complexities of our release process. As an example, we have three large integrated environments so that we can support three releases in parallel. And within a given release, we have a number of projects that are participating in that release.

Some of the more complex releases may have upwards of 15 or 20 projects participating, and those projects have several pieces and parts within them. So a project might touch our EDI platform, our Facets platform, our web and mobile platforms. And those same components may be getting touched by other projects to serve other pieces of functionality, so it does create this complicated web of dependencies. It was very interesting having that perspective on the release management team.

And then I moved back over to the infrastructure space, where I am today, and focusing on a lot of the stuff we'll discuss today.

And beyond that, I'm a geek. I've been working for about 15 years in the technology industry. Did a lot of software development. Always had a passion for automation, making things easier, making things go faster.

And I play around a lot at home. My friends kind of make fun of me because I have a server closet. I have a 48U rack environment. It's got Sun. It's got Cisco stuff in it. It's got a VMware infrastructure environment. So afterward, you can leave me tips up here or pay for my electric bill.

All right. So let's talk a little bit about some of the challenges that we face. These are actually quotes that I hear all the time.

"Healthcare applications don't lend themselves to automation." The technologies are old. I told you Facets was developed probably in the '90s. It's COM. It's DCOM. It's Windows. It's not Linux. It's not like they're releasing Docker containers for us to leverage. So we get a lot of resistance within the company just due to the nature of the applications that we run.

The second one: "Facets is too complicated. There's too many dependencies." Facets is always the elephant in the room when we talk about automation, because it is a big system. There are a lot of moving pieces and parts. There's a huge database there. There's batch. There's Tidal. There's EDI. Our entire application ecosystem revolves around Facets, and so you can't really address automation unless you address Facets, at least in some way.

I like this one: "DevOps is confusing, and we have more important things to do." You get this one a lot. One, what is DevOps? It's kind of a nebulous thing. People don't really understand. Is it a tool that HP sells us? Is it a technology? Is it a cultural thing? And so there's a lot of discussion around, what does DevOps mean to us? And at the end of the day, people want to keep the servers online. They don't want to talk about HipChat or Slack. So breaking down those barriers is a challenge.

And then finally: "It takes too long to get blah done." You would think that if people are complaining it takes a long time to do things, that they'd be in favor of automation, but we'll talk about, in a minute, really, I would say we're kind of in a position where we have functional silos. There aren't a lot of integrated teams.

So there's the DBAs, there's the middleware team, there's the ESB team, there's the application development team for mobile. There's the guys who maintain the virtual infrastructure. There's the guys that do the network. And so when we talk about creating an environment or something like that, or even refreshing an environment, you're talking about 10, 15 different teams. It's a struggle, and it really exemplifies the need to break down a lot of those silos.

And because we do have these entrenched functional silos, it can take a couple of weeks to get an environment built, even a simple one. It's not just as simple as, spin up a Red Hat Enterprise Linux template off of VMware and you're off and running. There's the networking, the DNS. It's got to be monitored. It's got to have the logging. Every team has to have their hands on it. It's very complicated.

And we have a headcount-driven coping mechanism. The easiest solution when we are confronted with a problem is, well, let's get funding to hire a couple extra people to help move that along faster. Or, hey, business, if the project is important, we are going to need three more resources to help us get it done.

And of course, we all know headcount-driven coping is not sustainable. Eventually you hit a ceiling, and I think we want to avoid doing that as much as possible.

And then, of course, when you're talking about hands-on and manual intervention, you're talking about inconsistencies, like environment inconsistencies. We've noticed that across our environments, they're supposed to look the same, they're supposed to be configured in the same way, but I was just talking to one of my colleagues before the meeting: for us, every time seems like the first time. There's a process and a procedure, but we always have to go and re-ask the same questions over and over again. So it's like, why can't we just capture this in a way that can be automated?

So those are just some of the challenges. I've got my little tug-of-war thing because it seems like when we talk about DevOps, there's always that push and pull between the application development teams and the operational teams.

The development teams are always saying, "We need more environments. We have demand for more QA environments, more development testing," and so on. And the infrastructure teams are like, "Well, we can't support 50 more environments. It's going to take us time to build them for you," all this stuff.

And so infrastructure is always very resistant to creating environments. They say, "Well, what about the 100 we created last month? Why don't you use those?"

"Oh, those aren't working for us. We're doing other things with... They're not configured right. Let's go back to square one."

So it's always that give and take.

So where did we start? How did we begin? We decided to sort of dispel the myths. And we talked about those quotes. We talked about the challenges.

So we said, let's do greenfield. Let's put aside all of the stuff that we do every day, and let's see if there's a way that we can take an application, a fairly visible application if possible, something that stands on its own and is functional. Maybe it's got a UI. It's not like we're going to develop greenfield some shell script or something like that.

And then we're not going to build a Ferrari. We're going to build a bicycle, but it's going to be end-to-end automated. And we're going to use DevOps practices. We're going to work with the infrastructure team. We're going to partner with the application development teams. We're really going to work together. We're going to use our HipChat environment. We're going to use our Confluence, the wikis, and we're going to do this right. This is an example of how we want to work going forward.

And then we had a plan for a roadshow. We were going to show it off. I always liked reading Slashdot back in the day. It was always like step one, do something cool, and step two, profit. It's just that easy. But that was sort of our philosophy as well. It was, let's put together something really cool that will wow people and dispel the myth that we can't do what they said we couldn't do.

So how did we do that? We picked one of our web portals, our provider portal specifically, and it had a technology stack that consisted of WebSphere, Oracle database, Red Hat Linux, and we knew that there were some changes coming up for an upcoming release. And we decided that we were going to work with the development team on that release to build a fully push-button environment.

Literally no hands-on. You go into the catalog, you request a provider portal, and 15 or 20 minutes later, out the other end comes a fully functioning, fully working, and ready-to-test, and even smoke-tested environment.

And so here's a list of some of the technologies that we used in the space.

Database as a service. This one's a really interesting one. When we talk about environments, a lot of times we just think about the operating system running the middleware. But, for us, the data was a really key piece. I mentioned how Facets is the elephant in the room when we talk about environments. Database is always sort of the elephant in the room. It's how do we capture the state of the database? How do we enable it to be rolled back and refreshed? When we go back to retest something, how do we reset our test data in a way that can be made consistent?

So we wanted to make sure that we solved for the entire environment, including the database piece, and we leverage a product called Delphix to help us do that. It's a lot easier than building something ourselves.

Configuration management. This is something that did not exist. Our idea of configuration management is, hey, Linux administrator, log into the box and run some scripts, and we wanted to do this the right way. We went with Puppet for our configuration management solution.

We wanted automated testing. We wanted to be able to not only deploy the code and deploy the environment, but we also wanted to be able to run a set of automated smoke tests against that, to essentially certify it before passing it off. So we use HP ALM, QTP, JTest, Veracode. We started dipping our toe in Selenium and some of the fancier tools, but a lot of our test scripts are already in ALM, so we wanted to leverage that.

We're using VMware as our infrastructure service platform. We've sort of productized it internally. We call it Blue Cloud, private cloud. And we're using Jenkins/Nexus for our continuous integration deployment pipeline, kind of the orchestration of deploying all the code.

And of course, we are going to follow DevOps.

So I think you guys are probably familiar with all these tools. They're a pretty standard mix. And we really wanted to do the whole stack.

So talking about solving for the database a little bit, this was kind of cool. The QA teams, given the nature of the application and given the nature of Facets, you can't just go into the database and run an update script to reset the data back to some point in time where you can then retest something.

So we needed the ability to essentially capture it at points in time, the testing. So essentially, a tester would start with, let's say, a known baseline with the data online just as they want, snapshot the database at a point in time, do some testing, and let's say at step five of their test, they run into some defect. They hit a showstopper. They can then snapshot that database again and share that snapshot with the development team, who can then try to easily reproduce the problem just by bringing up an environment attached with that snapshot.

Now, let's say the developer goes, they fix it, they redeploy the change. The tester wants to go retest it. Well, the tester doesn't want to go back all the way to the beginning of the test process. Maybe they just want to go back to step four, right before it failed.

So using a tool like Delphix, we're able to just bring up that database, move it around to different points in time, have the tester get right back to where right before that failure occurred, and see if it reproduces the issue.

And of course, speed is a huge thing. The grid on the top, our current process, I would say, takes about three days for a refresh. A lot of times they want fresh data. They need it re-baselined to some known good state. And so we call up the DBAs. We say, "Hey, guys, can you refresh environment XYZ?" And then there's this whole scheduling thing that occurs, and we have to schedule downtime, and this is a whole process, a multi-day. Three days is actually generous.

But with this solution, we're able to reduce that just down to a few minutes. And so it really enables a completely different mindset, both in terms of development and testing, because now we can literally shift that database around on a timescale on demand.

And so this is a capability that we rolled out for our greenfield solution, but I'll explain in a minute, we've also rolled out for our next steps.

So I think it's a real game changer for our testing organization and for the complexity of our environment. We can't just run a database script to populate a database.

So the impossible is possible. Not sure how easy it is to read, but what we're looking at is really the stack, the full stack that we've automated. On the left-hand side was the before.

So we're talking 16 hours to provision just the virtual server from a template. That includes calling up the team, the virtual team, saying, "Hey, spin me up a VM," that goes into some queue, blah, blah, blah.

Four hours for the middleware WebSphere install. We're just talking the base installation of IBM WebSphere. That's time-consuming to install. Someone has to do it.

The WAS configuration. There's a whole team that configures WebSphere, configures all the parameters, does all the works, the magic. And that's a very time-consuming effort.

And then, say, 32 hours for the whole application deployment piece, moving, building the binaries, getting them loaded onto the machine, and executing some testing.

We reduced that down to about four hours from 80 hours. And literally just a push button, just getting rid of those handoffs. So it was really an example of, hey, we actually were able to show some real value. We've sort of moved the entire chain out of the way, and we've discovered a lot in the process of building this out.

And the success of this greenfield approach is due in no small part to a lot of the people in attendance here today. So, certainly, thank you for all of that help.

And we discovered a lot along the way. We discovered that things weren't documented very well. We would say, "All right, how do we build a Red Hat server?"

And they'd say, "Well, I don't know. Here's a document. Follow this document."

We go follow the document, and then we show the end result to the powers that be that give the thumbs up on the environment, and they say, "No, this is all wrong."

And we say, "Wait a second. We followed your document to the T."

And they say, "No. That document's out of date. That was the standard from six months ago."

We're like, "Well, where's today's standard?"

It's probably locked away in someone's head. You've got to track that guy down, and they're probably offshore. So it becomes very challenging.

There was a case where we were configuring middleware, and it's like the application server's not coming up. We keep hitting the homepage. We're getting a server 500. Why? No one could explain it. We dig and dig. It's like this one missing parameter. It's like there's this one dude that knows this parameter has to be added, but it's not part of any script. It's not part of any document. He's just the guy that, when the environment comes to him, he goes in and adds the magic parameter.

And then we found a lot of areas where we could improve in terms of loading static content, and we just sort of made note as we're going through this process of areas where we want to go back and sort of readdress at a future date and kind of hack some stuff together just to get it working.

But I think we, as automation engineers, weren't even aware of just how discordant or disconnected the actual process was, or how many people... I think it took about six months to build this automated push button, and that was just a discovery period. It was going to every team, interviewing them, asking them what they do, how it works, how do you set it up, and it was like pulling teeth.

Like I said, some people are resistant. They say, "We don't have time." Some people are worried they lose their jobs when we automate. But I would say that the key for us is really identifying a group of people that share your mindset. Find the folks within the company that want to make a significant change and work with them. And that sort of energy is infectious.

So I think, back to the slide, the impossible is possible. It's not that we reinvented the wheel, or we did anything amazing. What was really important here, the big takeaway, was that we, Blue Shield, accomplished this thing. We see other companies doing it. We know that it's technically feasible, but I don't know that anyone acknowledged, at least maybe not our leadership, that we had the skills, we had the people, we had the capability of actually delivering something this cool.

And we did roadshow it. It was really awesome.

So what happened after we did the roadshow? Well, we went to a brownfield approach. Our hope after the roadshow was, "Oh my gosh, you guys are amazing. This is the coolest thing we've ever seen. Here's $20 million. Go run with it."

We were sort of hoping for a mandate down from our CIO saying, "From now on, everything will be automated, and the automation team is going to drive the direction for everything." And that was our fantasy. That isn't actually what happened.

So what actually happened was, "Okay, guys. This is really cool, but how do we operationalize this? How do we make it so that these environments are something that we can support? Something that we can expand on and develop? What about all the other people in the company, the 100 other folks that maintain our environments that don't know how all of these fancy tools work?"

And so we say, "Okay, those are good questions. Well, we can send them to training. We can send them to DevOps class. We could do all this cool stuff, Puppet, all this great thing."

But we quickly realized that we had an entire legacy... I don't like to say a legacy environment. We have an existing environment that exists, and it has problems, and the people that are there are there to mitigate those issues, and they've got their day jobs. It's not like we can just push pause and go off and make everything perfect, and then tell the business, "Okay, now we're ready to work again."

So we decided on this sort of brownfield approach. It was, all right, so we've got the tools, we've got the technology, we've got the people, we're starting to get a sense of the mindset. Let's see how we can start to integrate what we learned into our day-to-day process.

So we took this brownfield approach, and what we said was one of the things we wanted to do was uplift our skill sets. We wanted to take the folks that were doing sort of the firefighting on a day-to-day basis, and we wanted to sort of introduce them to Puppet. We want to introduce them to Delphix. We want to introduce them to our infrastructure as a service and our Blue Cloud and the inner workings.

We not only want them to understand. We want them to own it. We want them to support it. We want them to be partners in helping us develop that capability.

We want to move out of firefighting mode. People are so locked up in their day-to-day work, it's very hard for them to dedicate time, even just a few hours a week, to learning new skills.

So we said, is there something that we can do to help mitigate some of the day-to-day pressure, some of the day-to-day fires, and free up those hours so that they can actually focus on more important things? So we wanted to look closer on how we could sort of free people up from that day-to-day firefighting.

And of course, what are the immediate wins? What's sort of the low-hanging fruit that we could go after and show some real value in the first quarter or two quarters? And everywhere you look, there are things that we could fix. And part of the greenfield proof of concept, we did identify things, that low-hanging fruit that we could go back and fix. So we want to come up with that list.

And of course, I mentioned freeing up those resources for value-add work. We scale, again, with headcount. That's our coping mechanism. Those people are working on project activities. They are 100% dedicated to projects. They don't have time to go off and explore and do fancy things.

And then finally, automated testing. This is a big one for us. We have a lot of QA. We have a lot of testing. It's very manual. And so what can we do in terms of our automated functional testing?

Now, one of the things I didn't mention in our greenfield approach was that we actually did execute automated smoke testing after the environment got built. So we deployed the code. Puppet said, "Hey, all the code is deployed. I'm ready for the database." Puppet calls to the database, calls to the Delphix platform, "Spin me up a database of this particular version that matches the code that we're about to deploy."

Once that environment's online, again, Puppet is sort of this orchestrator of all these things happening, says, "All right, ALM, execute this sort of test suite." And that job gets created. We execute the test suite. And those results, the screenshots, all that good stuff that happens as a result of the testing, gets stored and brought back into Jenkins for review from whoever kicked off the build or the request.

So we wanted to make sure going forward in our brownfield approach that we had the ability to continue this automated testing approach as well.

And I think I want to mention again here with respect to the testing and brownfield is this whole database solution. The ability to roll back, the ability to snapshot, was really key.

I was talking to our director of QA just last week, and she was saying, for us, Delphix and our database as a service solution really is a game changer because what it means is we don't have to try to reset data all the time.

Now, if you imagine a claims system, you want to find a particular member in the system that has these attributes, maybe a woman between the ages of 40 and 45. Maybe she's pregnant, and she has this particular disease. That's the testing criteria.

Now what I want to do is, she's a patient. I want to submit a claim. I want to take the claim through the first four steps, and then I want to get to the point where I'm actually exercising this new functionality that we've deployed.

Well, how do you adjudicate a claim? You go into the UI, you click all these drop-downs, you toggle all these switches, all of this stuff. In the back end, there's hundreds or even thousands of tables that are being adjusted by the Facets application. There's batch running. There's imports. There's exports. There's data loads. All of this stuff.

And the test case fails. Well, okay, great. Now they go back. The tester says, "Okay, well, where's my woman between 40 and 45 with this particular disease but in this particular state that I want? The record I had is now moved past. The claim's been adjudicated or whatever. It's closed. We paid it. How do we get back?"

So it's like, well, do we just create 100 records of this person automatically so that after we use the first one and it fails, we try the second, the third, and the fourth? The answer is no. We don't want to run scripts that try to reset the database using direct SQL. We don't want to run automated tests using QTP to go and actually click and toggle and do all this stuff real time. We just want a way to take the database back.

And so that snapshotting is a total game changer for our QA.

And there's the whole, I want to say, the time-sharing aspect of it. I mentioned we have a lot of projects sort of participating in a release at a given time. Those projects compete for the database as well. So if we have the ability to say, "Okay, project A, you're up. You're ready to test. Great. We're going to load in the dataset that you want, bring it online, do your testing." Whatever the results are, then reset it back for the next project that's ready to begin their testing.

And so it really lets us time-share the environments in a more unique way.

So we're sort of in this mode now where we're doing the brownfield approach, and actually, I think it's working very well. The roadshow really helped. Seeing that we can automate this stuff, seeing the tools are present, seeing a lot of the success that we had around that portal has really changed the conversation.

People are now talking about automation. They're talking about, how can I leverage these tools in my space? I have infrastructure people coming to me and saying, "This is pretty cool. How would you deal with NFS configurations in Red Hat? How would we solution that?"

The middleware teams are coming to us and saying, "Hey, we've got a new project coming along, and we want to be able to leverage these tools. Help us understand how we can do that from the ground zero, the right way."

So I think that we're really starting to change that mindset, and it starts at the bottom. At Blue Shield, it is not a top-down type of organization. It's the people that make up the day-to-day work, and changing their hearts and minds is a matter of really improving their day-to-day lives.

And so if we can do that in small, incremental steps, we free up those people to really do some value-add work.

So what is the value-add? The business value, this is pretty standard stuff. We free those people up. They can actually focus on automation. They can really focus on taking us to the next level. They can get us out of the firefighting mode. They can reduce the number of incidents that we have due to environment misconfigurations.

The second bullet point there: reducing dev and QA downtime. This is a really big one. This is a hard problem, because due to the misconfiguration, our environments go down a lot. The environment's pointing to the wrong database. The environment doesn't have the right certs loaded. It's misconfigured. It doesn't have the right versions of code on it. We don't have the data that we need in the environment at the right time.

These are all things that plague us day to day. They stop QA. They stop development from occurring, and when you've got 300 testers and 300 developers sitting around waiting for an environment, that's a very high cost. So it's a very real problem for us to tackle with automation. It's not glamorous push button, build an environment. It's how do we get consistency around the environments that we have today?

And then the last one, service enablement. This is sort of one that really speaks to me because one of the things I love doing is integrating systems. I like to say if data could be shared amongst all the different systems in a very consistent way, then we make those systems more aware. We separate them from the silos, and we kind of free data.

So service enablement is a big one because having the automation means that we can consistently keep our CMDB updated. We don't have to rely on a person to go update CMDB. We can keep our monitoring tools up to date with respect to the environments that we're provisioning. And so all of this sort of back-office stuff that occurs as part of provisioning an environment, you become able to automate a lot of that stuff.

And so automation breeds automation, and I think the ability for other teams to leverage the tools and the capabilities is a huge piece.

And of course, service enablement also touches on, I guess you want to say, anything as a service. It's kind of that shift in mindset that I am not... Let's see, what do I want to say? I'm not doing this job because this is my job. What I'm doing is I'm making it easier for you to use my services.

So it's really instead of looking introspectively, it's really opening up externally and becoming more service-focused. Let's enable our development teams to be able to click the button to make a change to WebSphere, to add storage to their server. Why do we have to go through a three-month process to add storage to a system when the storage team could make a service that follows their rules and enables that to happen?

Since we started this whole process, we're seeing a lot more of those conversations happening. People are saying, "Okay, well, it doesn't mean I'm being replaced. It just means that instead of me focusing on a ticket queue, now I'm writing services and enabling you to do that stuff yourself."

So let's jump to top five takeaways for us. One of the big lessons learned was, okay, well, we can achieve DevOps in a large healthcare organization. We can take a large problem and cut it down into the specific piece, smaller bits that are consumable. We can tackle low-level problems, and we can do this in a very incremental and evolutionary way, maybe not necessarily revolutionary, but evolutionary, and solve them in chunks.

The other thing is, let's address the challenges holistically, and that means people and technology. And sometimes I'm sort of at fault being a tools and technology guy. That tends to be my go-to answer every time I want to solve a problem. I want to say, "Well, what's the technology? What's the capability?"

But what I've learned going through all this is that people play a very important part of the success of the organization, both in terms of adoption of the tools, but the implementation of it as well. These are folks who have that day-to-day innate understanding of how the application works and how it functions.

And to change their minds, you have to demonstrate to them that it's going to benefit them in some way. And that's a challenging thing to do.

Our greenfield experiment, you get excitement with a big win. I think that helped us tremendously. That got visibility at the highest levels within our organization. And seeing excitement at high levels of the organization gives excitement to people at all levels within the org.

And I think that seeing that we could accomplish that and that we have the tools, we have the people, we have the skill sets in-house to achieve it, was a huge sea change for our organization to start moving in a new direction.

And last, chip away at the biggest time sinks. Shift resources to more value-add work. Free them up from the day-to-day. That's the biggest problem we have is getting people's time, and that's always the mantra. It's always, "I'm sorry they don't have time, but if you want, you start a project, get it funded, do a business case, get an ROI. We'll hire a resource, and they can focus on it."

It's like, no, we don't want a resource to help us build something. We want your team to engage and change the way they work and understand this better.

And so making that shift in mindset has been a slow progress, but we've definitely been marching in the correct direction. It's been working slow, maybe too slow for some of us. But it is a big problem to solve. We've got a lot of people and a lot of tools and technologies. We've got a lot of processes.

So lastly, what am I looking for help with? Automated testing. It's hard to do automated testing around the Facets platform, Tidal, batch. I don't know how many folks in here are in the healthcare industry or use these tools, but they're fairly antiquated. You interface with them through UI. It's very hard to automate them in a way that it can be done quickly.

Test data management is another area where we have some confusion around, I guess. How do you load a database with the data that you want to use? You can take our production database, you can de-identify the information, you can subset it and mask it. But at the end of the day, that only contains the sampling of data that exists in production. If you're developing new functionality, that data doesn't exist, so it has to be created somehow. And you want to do this in an automated way.

So currently, we do this manually. Now, having the database snapshotting lets us get back to a known good baseline very quickly, but it still means getting up to the baseline is a manual process. We haven't quite cracked that nut yet.

And then finally, key performance indicators. We've been moving pretty quick. Depending on what director you talk to, they're interested in different indicator of success. Some say, "Well, a reduction in headcount is an indicator of success." Some say, "Doing something faster, getting an environment in 20 minutes instead of 4 hours is an indicator of success." Some say, "Reducing the number of incidents in our non-production environment is an indicator of success."

So we haven't quite found the right formula we want to use yet. I'd be very interested in talking with anyone here around what KPIs they're using, or you're using, that management seems to like to listen to. So that is a big one for us.

And then finally, questions. Any questions?

Takes me to the end. We're a little early, but just by a show of hands, how many folks here are in the healthcare industry? We've got one guy there.

Other large organizations that kind of are hard to transform? Or are you working for the cool people?

I think we can all relate to reality, though. And it's ongoing for us. We haven't slayed any dragons. Our whole environment isn't managed automatically. But really the reason, again, I wanted to come here and speak is because I wanted to demonstrate that, yeah, you can have success in changing even a large organization.

And I would expect as we move forward for the excitement to continue, and for us to move faster and faster, and we get those ducks in a row, I think we'll be able to start really competing.

So that's it. Thanks, guys.