This is a talk I gave at Agile 2013, and Gene came and saw that and said, "Hey, that is a great story. Could you come and tell it here?" I have half the time, so I know it's supposed to be the long ugly story. We're going to make it the short ugly story for today. But I'm going to try to go over the main points that I think were the most impactful for us.

So real quick, here's what we're going to do. Fairly simple agenda. I'm going to set up a context, the company that I was working for at the time. I'm going to tell you about the challenges we had.

Then I'm going to talk about the improvements in four main areas. We're going to talk about collaborative teams. We're going to talk about automated deployments and environments, automated testing, and some things of that nature. And we'll talk a little bit about consolidated tools.

And then I'll go over our results, both our anecdotal results and then the actual numbers. So just real quick, just to introduce you to myself, I work for a company called Helen Square Group out of Nashville, Tennessee. I run their Agile practice and .NET practices in there. I'm a Scrum guy, so Scrum certified by the Agile Alliance, scrum.org.

Also a Microsoft MVP in their ALM Platform TFS, and we'll talk more about that in just a second as well. Contributor to InformIT and Safari Books Online, do some Scrum videos on there. I run the Agile Nashville User Group. I have a blog at tommynorman.com, talk mostly about Agile stuff and some ALM stuff.

And then Twitter feed is @tommynorman, so if you wanted to tweet while I'm doing this, feel free. You can follow me where I talk about ALM stuff. You can see what my kids did today and what I ate for dinner and those type of things as well. So let's set the context first.

The company I was working for. I'm going to say Company X because I don't work for them anymore. It's going to have a happy ending, but it's got some, like I said, it's a ugly story in the middle. But Company X, number one, was a healthcare company.

And why is that important? As we just said, a lot of people say, we can't do Agile, we can't do DevOps, we can't do all these things that we're talking about at this conference because of regulations. And we were definitely regulated. We were meaningful use.

We were with the FDA. We were distributing drugs. So we had all the same regulations that you would have in that type of company. And we were still able to be a very, I think, a mature Agile shop.

We were also pretty good at all the DevOps stuff we're going to talk about in a little bit as well. So all those things that people talk about, oh, you can't do that with these. Yes, you can. The second part is we were software and hardware.

So these were cabinets that were distributed into the actual hospital systems holding controlled substances and those type of things. So a lot of people say, "Oh, you can't do that with hardware." Well, yes you can. We did it with hardware, and we'll talk about how we did that with hardware in just a little bit as well. So this company, close to 1,000 people, getting close to there.

Between $300 and $400 million a year. They were the number two. There was the big 70% number one, and we were the 25% number two, but nipping at their heels. Will we ever take over number one?

Maybe. But their R&D department's budget was our budget. But we're going to talk about how using some of these nimble techniques with DevOps and Agile, we actually started to increase the ability to take away their customers. Also, we were global.

When I started there, we were in India, we were in China. In the States, we were in Chicago, Nashville, right down the road in Mountain View, and in Houston. So distributed teams, and my first team had members in all of those areas. Eventually, we started consolidating, but at first it was really hard because we had such a distributed team doing all sorts of different things all at the same time.

Lots of moving parts. Also, diverse platforms. This company grew by acquisition, and each time they got a new company, usually they got a new technology, too. One of their benefits that they were selling to their customers is backwards compatibility for 10 versions, which means I also had to deal with a lot of older technology.

You see FoxPro in there. Yes. FoxPro very much in our ecosystem. C++, VB6.

Lots of different .NET stuff in there. And even though we had multiple .NET, each time we bought a company, they had their own take on how they did their .NET platform. So we had to merge all of those into one kind of seamless ecosystem. And then lastly, when I joined, it was about a year into their next gen product.

So like I said, they acquired a bunch of products and kind of put them together as one suite. And of course, if you've ever done that, you know that sometimes those suites kind of look different, and they're not always talking to each other, and sometimes you get complaints. So our next big thing was to merge all of those on to one unified platform. So not only were we building our next release, we were building the release that would hold the platform for everything from that point on, which is a big task unto itself.

So that's kind of where the company was at the time that I joined, in the middle of this project. And when I joined... And I also want to point out that we're not a unicorn. You hear the word unicorn used a lot when they're talking about the big boys and how they do things.

We were not a unicorn. I'm going to talk about how when I had gotten to a level of management, I didn't have any budget. I had to beg, borrow, and steal to get these things done. And I want to bring that up because a lot of times you hear the stuff that people are doing, you're like, wow, and you have all that money and all those resources, you can do it.

And I'm going to talk about how we did it with almost none of that. So if you're a little company thinking, I don't think I can do these things that these big companies are doing, yes you can. And we're going to talk about also, when I came on to that company, I was just a developer. I had gotten out of consulting.

I said, I'm kind of burned out, I'm going to go back and just be at a company, and I was just a developer on the team. And then when we started getting into it, because I had done a lot of Agile stuff, I'd done a lot of ALM stuff, they started asking me to step in a leadership position. We lost our QA manager. I became the interim QA manager for an interim of nine months.

It was only supposed to be a month, but it kept stretching out.And then eventually became their permanent agile coach. So I got to see all of these things in many different views. At first, I was just a developer on a team, then I was over a team. And when I was in that QA manager slot is when a lot of what we talk about got implemented.

So I'm going to talk about a lot of these initiatives that you wouldn't necessarily say, "Well, that's not a QA initiative," but we kind of brought it in under that umbrella. All right. So let's talk about the challenges that we faced. The first one I want to talk about is, we talked about all those different technologies we had.

When we brought in all those different technologies, we had teams built around them. So there was the FoxPro team, and there was the .NET teams, and multiple .NET teams because we'd bought multiple products with that. We had the C++ team. So with our teams split out like that, we had to build products across all of them, though.

To get one piece of functionality out, all of those teams had to touch it. The back end services had to be written in FoxPro. A lot of the clients were in .NET, and then a lot of these satellite clients were built in these other technologies like C++ and VB6. And the problem was, with our teams distributed like that, it would be three or four sprints before we got anything that we could actually test, that actually worked.

Because somebody wrote this piece and we wrote this piece and this piece, and then we had to kind of fit it together. So that was one of our first challenges when we got there. The other one was, when I first started there, they didn't have any quality assurance at all. They'd started the project, had been a couple of months into it, did not have any QA people.

When we did get QA people, we pretty consistently had about a seven to one ratio. And what would happen is they were always running behind us. So all the developers would write new features, hand those over to the QA people. As the QA people are struggling to keep up with that, they would write a whole bunch of more features and hand those over to the QA people.

And you see this time and time again. They start getting behind. And what happens when we start getting behind like that? Well, we started getting a lot of defects escaping because there was just no way to keep on top of that.

Also, everybody kept saying, "We need to automate. You need to automate." Well, we just didn't have the time. We didn't have time or the skill sets. And we'll talk a little bit later about how we did automate eventually.

But at that time, we were barely even just doing all our manual testing. Plus, with the hardware part, there was always going to be some form of manual testing as well. Also, we had this large amount of bugs that were growing and growing and growing. At one point, they became almost 50% of our backlog were bugs because of this technical debt and these things that we were moving on, and we didn't have a very good way of attacking those.

So there was this loosely prioritized bucket of bugs, and developers were encouraged to just go knock them out. And so they would do that. He might write a little piece of code and say, "All right, I fixed that bug." The problem is that was usually done outside of QA, outside of operations, outside of everyone. They just kind of did it on their own.

Because they were loosely prioritized, they usually cherry-picked. "This is the one. Ooh, I know what that one is," or, "That's an easy one. I'll take that one.

I can knock it out." And what happened is, without people being involved in that, all it did is usually spawn more bugs, and it just became kind of a vicious cycle that was going on for us. Next, so we did have some DevOps people, we did have some QA people, but the problem was, as I said, they were kind of burdened with all these defects that they were trying to manage and work through all those features that were being dumped on them. The DevOps people were even a little bit more abstracted from us, so they were trying hard to get us to where we could get releases out. And the developers, though, are all getting together all the time, and they're collaborating all the time, talking about new features and having all this information sharing, and they're sharing all this tacit knowledge.

And that was not conveyed very well to the rest of the team. So you have a bunch of the QA people going, "I don't know what we're talking about." DevOps are like, "What are you guys talking about?" All that information was captured only with the developers and poorly related to everyone else. Next, hardware. So I told you we had a pretty intensive hardware offering.

Every piece we put out had some kind of hardware element with it. We had a dev lab with a bunch of the junk stuff. The stuff was expensive, so we got all the second generation stuff, or stuff nobody else wanted. Then there was also a QA lab, and we had less stuff because, like I said, QA came in late, so we got the third and fourth generation stuff to the point of where even when I plugged in one of my first things that I got, it actually caught on fire and started smoking.

And the developers were like... I'd say, "Can I borrow that?" "No, this is mine. It came from my cost center and you can't have it." And there was this battle over these labs, and we were actually physically separated, and if I was caught snooping around, "What are you doing?" "Well, I am looking for stuff. I don't have anything over here." So that was hard for us because to get stuff out, I had to test it on production-like materials, on hardware.

But I didn't have production-like stuff, and to try to get it, I was constantly getting my hands slapped of, "That's not yours, that's not yours." And I had no budget. So the labs were in a huge disarray. And then estimation and planning. So management, we were using user stories, we were using backlogs.

We were a Scrum company, and I think we were doing an okay job. And management, bless their hearts, they said, "I don't want to bother all the engineering people. We'll just estimate for them." "I used to develop, so I know what that is." So they estimated for us. Not only did they estimate for us, but they didn't have any representation from QA or DevOps or operations.

So when they would give us these estimates, sometimes they didn't include any of that at all, any deployment time, any QA time, or if it did, they just said, "Oh, just tack on 10%. Right? That's good enough." And of course, then we saw it from the QA perspective and said, "That's insane." And then DevOps was even... The operations guys were getting it even later, so they're like, "What's insane?

What are you talking about? We don't get anything until the day before. By the way, we need to go to production tomorrow. Can you build a server?" "What?"So we had a problem with the original estimates, but that's where our plan came from.

Here's all the deadlines we've promised all our clients. All the marketing stuff's already gone out. And when we started getting into it, of course, what happens? Things grow.

It grew and grew and grew and grew. Did our timeline stretch out any? Nope. Not at all.

So we started really having to work at a pace that was not sustainable. It was the nights and weekends and those type of things. QA really got the short end of the stick of that. Operations too, but QA really were the ones who I just saw in there all the time.

So that's kind of where we were at. And what was happening is our release was in jeopardy. We weren't going to get the amount of features we had been promising out to the market. We had a lot of defects.

We got good at getting some of the critical defects, but we just had a buggy product. Nothing was going to reach out and kill your soul or anything like that. It wasn't that bad, but there was a lot of stuff just made the experience not good. It wasn't very reliable.

This is healthcare. They like reliable. Didn't look like we were going to make our date, and so management was just constantly on us all the time, "Work harder, work harder, work harder," trying to put in all sorts of different things. "Let's add more people," because that always works.

"Let's give them incentives." One of the incentives was, we'll give you a quarterly incentive if you increase your velocity a couple of points. So what do you think happened to all our estimates? Just went up. It's like the Dilbert where he says, "Every bug you write, I'll give you a bonus." I'm going to write myself a minivan.

Awesome. But it didn't matter because remember I told you all the times were set and given by management, so we could estimate all we wanted, didn't matter as long as it fit in that timeline. So we were always robbing Peter to pay Paul. So that's where we were.

So here are some of the improvements we put in. Not all of them, but here are some of the big ones that I personally was a part of and I personally felt really progressed this to the happy result we'll get to at the end. So first I want to say I'm an agile guy. I am an agile coach.

That's what I do. So all the things we did, I don't care if it was in development or if it was in QA or if it was in operations, I tried to base on the agile values. Main one, working software is my primary measure of progress. Tried to capitalize on all of them, but that was the big one.

We, for so long, did not have working software. We had a lot of code. Code was king at this company. We had engineering key people who went and checked in how many check-ins were going in TFS.

"I see you had a lot of good check-ins. You're doing awesome." Okay, but that's not working software. And sometimes that code that was being checked in was breaking other code, which is not good. From that, from this principle and some of the other principles, we started implementing a lot of agile practices.

So one is the scrum practice of potentially shippable at the end of each sprint. We threw down the gauntlet and said, "No more of this stuff that's halfway done." Oh, the UI's done, or the backend's done, or this is done. No, this should be potentially shippable at the end of each sprint, even if I'm not going to ship it. Has to be.

We really worked on our definition of done. At first we had definition of done, and at the bottom of it said, "And here's some stuff you should really think about." We're like, "No, no. We're either doing it or we're not." This is our definition of done. QA is part of our definition of done.

Rolling out to production and like environments and testing there is our definition of done. We also did things like limiting our work in progress, so more of lean and Kanban type of things. We were concentrating on too many things. Slow down.

The whole slow down to speed up idea. And also, making our work visible, another Kanban type thing. So we use TFS, and I'll talk more about it later, but one thing it gave us is all this data that we could then share, and we talk about all the time, putting that up on the big boards and everything. We even had it to where all of our video conferencing software I could tap into.

So when you weren't doing the video conferencing, all of our data was up there, constantly in your face. So these principles and these practices from agile is what I tried to apply as we move forward. So the first place we tried to do this is in cross-functional teams. So remember I talked about we had these teams that had all these different disciplines in each team.

We had to merge those. I need to have all these disciplines on one team. I need to have the correct disciplines for that team to be able to get a working piece of software all the way through. And you see I've kind of got the little icons for my QA and my DevOps person side by side.

We didn't have operations, a very official group of that, and my QA actually kind of stepped up. I had some very engineering-focused QA who actually did a lot of that, and we actually shared that responsibility with them. So we started, as I said, we were distributed. Even when we consolidated a couple offices down, we still had four or five offices that we were working out of actively.

So we did a lot of investment in a lot of tools to help us with that. We had the Smart Boards. I don't know if you've used the Smart Boards before, and you can connect them so that I can be drawing on it, and you can be drawing on it states or countries away. We can interact with it.

That's where we would use our Kanban boards and our task boards, things like that. So when I'd get up, I could be talking to someone in California, and I could be moving it around and they could see us in Nashville and those type of things. We invested in other tools. We were using Skype very heavily.

One thing we did that anybody can do was we always had this thing of like I owned the conference room number for my team as the scrum master, and if I wasn't there, I had to give it to someone else. Just simply giving every conference room its own number and some guy went in and did a script where you can come in and double-click on something. It automatically dialed into that phone, automatically started screen sharing. So it was one click to start immediately sharing with each other.

That actually helped tons to lower that barrier of, that threshold of entry to doing anything with collaboration. And then for those people that were located together, sit together.We had this big new office and we said, "Sit where you want." And all the QA people gathered in one place and all the devs went one place, and we said, "No, let's sit together as a team." And that was business owners, that was everybody. Now, remember, DevOps was separate still at this point in time, so I'm going to put the little tiny DevOps because they would come over every now and then. QA really kind of filled a lot of that role at that time for operations.

And then we were encouraging a lot more collaboration, where before, it's not like it was discouraged, it's just that it wasn't something that was actively thought of. And kind of unintentionally, people would put barriers up to that. So what we did is we would tag things in TFS on our backlog of this is going to require some operation stuff, so that I could see, hey, this is coming up. We're going to groom that story.

Let's make sure we get some of the ops guys in there. Here's a story that needs performance testing. Let's make sure we get those guys in there. So when we groomed them, they were in there.

When we did sprint planning, they came in and they had their own task. So we really integrated them. They weren't a dedicated part of our team. We didn't have enough people to do that, but they were very much integrated and very much visible of when they were going to be needed.

So next I want to talk about as an organization, we had to change as well. So initially, like I said, we had all the developers who were under their own management, QAs under their own management, operations under their own management, and we had this nebulous cloud of leadership. Just people who came and told you what to do every day. From all different places.

And there was this nebulous cloud of products that you worked on at any given time that you may know or know nothing about. And it just rained down on you all the time, just products and stuff to do just rained down on you constantly. And what happens? Well, we start getting a little drowning here.

Oh, we're getting under. And then we get these lightning flashes because I don't have time to get something done for you. So that was a problem in that just this no sense of direction of where we're going. You're getting pulled 20 different ways.

This guy pulls you off, then he pulls you off, and then he says, "Why didn't you get what I wanted done?" All the time. So we had to change our product focus. And they brought in a lot of product and executive consultants, and they did an inventory of all these apps we had, because we just kind of acquired these companies that kind of brought their own baggage. And one of the things they did is, "You need to get rid of some of this.

You need to consolidate some, and some of this is not core to your market. It's not core to your business. You got it as part of something else you bought." And we either got rid of it or merged it into other things. Narrowed down our product offerings.

And then when we did that, the next step was reorganizing ourselves into what we called core teams, a core team per product, and there was the core team leads. At the head of this was a core team lead, and he was kind of like the CEO of a small little company within a company. Uber product owner, if you want to call him that. And in that core team, he had all the people he needed.

He had sales, he had dedicated representation from sales, marketing, and ops. He also had dedicated people, a developer lead from his teams, a QA lead from his teams. He had a program manager that was kind of like his scrum master helping organize his feats. And then we had product owners that each had their own dedicated teams, their own dedicated scrum teams, and those scrum teams, of course, were filled with all the people they needed to get an actual product done.

And sometimes these were by product lines or sub-products. But it really narrowed down. Instead of this cloud of leadership, you had focused leadership. That core team kind of set the direction.

I want to go this way, and here are some of the big ideas I have. I want sharks with lasers on their heads. And then they give that to the product owners and they decide how deep we can go into each one of these. So if the product owners needed help with marketing things, like I want to know, does the market tolerate this?

Can we do this? Can we do that? They had the ability to go to the core team lead and had sales marketing people who could do that for them. So how many people are doing enterprise agile in a fairly big company?

Some hands out there. So this might look familiar to you. What does this look like maybe? SAFe.

SAFe. Scaled Agile Framework. Now this happened before SAFe came out. So when SAFe came out, I said, "Hey, we did some of that." We kind of organically got there.

So I'm not a big SAFe guy. I do see a lot of advantage of it, especially because a lot of things that are in there are things we kind of organically got to on our own. So that's a whole other topic. But that was one of our big things, focusing our teams and then focusing the management and the products around them.

The next thing I want to talk about is automated environments. We're a little short on time, so I'm going to speed up a little bit here. So remember we had all these disorganized labs? Well, I went out and said, "No, no more.

We have one lab, and I want to build a complete lab per team." And I went and begged, borrowed, and stole to get equipment. I found out that there was this huge warehouse of all this returned equipment. So this one had a busted door, this one had a busted CPU. If you put them together, I've got one unit.

So I got all the operations guys who fix them to come in and teach all the dev people and all the QA people how to repair them so we could make our own. And as long as I paid for shipping, this guy loved to get rid of the stuff because he was just having to pay to store it. So very cheaply, I was able to build these labs. One of the things we needed was these high-end medical-ready refrigerators.

But I didn't really need it. I just needed a refrigerator so I could actually test the sensors. So we just went out and bought a bunch of Walmart refrigerators. We built these labs, and I taught all the team how to build it, and I eventually told the team, "Now it's yours.

You own it." Every team had a complete lab, and then there were several satellite labs that may not... This one doesn't have the refrigeration stuff because not everything needs to be in there. And they owned it. That was the big part of it because it lived past me after we'd gotten it done.

And something broke, we could easily fix it. We were self-sufficient. So again, we didn't have a lot of money. We did that on the cheap, and it was just a matter of finding it and working through it.Now, we also invested a lot in automated deployments.

This is actually being done even when I got there. So I think we were pretty good at this and we just got better at it. They were already doing a good job. We had various clients, and those were always imaged, and it was very easy to script those images and deploy them out.

We really invested in virtual machines and templated virtual machines so I could spin up one of these back-end servers very quickly. We also had very good automated deployments. The dev team invested in that from day one, which is great. From day one, they had deployment scripts and update scripts.

It was part of their definition of done. So that was one thing I didn't have to worry about. That was already there. The big thing I did have to worry about is data.

We had to scrub data and get production data down. We always had a problem where we tested on 10 records and then we get it out there with 10,000 records or 10 million records and it fails. So we started getting production data, invested some time in some scripts that would scrub that for us and automatically deploy that with it. So we got it to where within less than an hour, I could deploy to that.

Now, I know some of these big unicorn guys are saying, "We deploy a million times a second." And that's awesome. We were not that company. We were not the unicorn, and we didn't need to. Our market did not meet that same pace.

So one hour was actually way better than the half a day to a day it took us before, and it was automated, so whatever I put out into that lab was the same thing I used to go to production as well to cut down on all that. Next, I want to talk about tests. So the first thing when I got in there, my boss said, "We need to automate all the things." And I said, "No, we don't." Because I went and talked to my team, and we had lots of problems that were bigger than testing, and I'm not going to test a bunch of crap because then I'll just have automated crap. So we went in, just real quick, I won't go into these in detail, but these are some of the things.

We just didn't have good testing principles. We had to get that standardized across the team. Once we did that, we had a standardized way of doing our principles and guidance for test case management and test plan management. We had to work on consolidated tools.

Everybody used something different. And then we had to work on getting testing as part of our definition done so we weren't always behind everybody else. I did that before I started automating because if we had started beforehand, it would've been a disaster. Then he came back and said, "Now that you've got all that in place, can we do it now?" And I said, "Okay, we'll do it." But this is two years into a project, and we had lots of stuff that we had to get around.

The first thing is we had to talk to hardware. The development team had already built-- and we don't have all this hardware, and we can't do it when we're doing automated, but the development team had already kind of settled this once. They had built a little bit of an emulator because not everybody could have these hardware cabinets in their desks. And then we would use that when we were doing manual testing as well.

But then also with a little bit of elbow grease, we made that emulator scriptable because not only did I need to be able to talk to the hardware, the hardware needed to tell me, "Hey, a human did this. You told me this to open the cabinet, and then this guy pressed this button." And we made it scriptable so that we could automate those tests. Still not 100%. We still had to do a lot of manual testing, but it got us a lot further along than we were before.

And then we could automate those type of tests, which we had not been able to do at all before. When we did this, like I said, we had about two years' worth of stuff already in there. So we got a dedicated team that we contracted with a company, some offshore elements, some of our own elements, and they spent about eight months behind us getting us caught up. Then while they were doing that, I spent time teaching all my developers all my testing tools and how to execute tests.

And then the developers taught all the QA people a little bit more advanced on how to create automated tests, how to write code. So we rows all those boats together so that once we got to the point where we didn't need that team, they had caught up to us, then what we had in place was during the sprint, I expected the team, they have all these test cases, they don't have to do all of them. Automate the most critical ones. Critical path, good build verification tests, things like that.

And then the other team would run behind them and do all of that. All right, so we're really running out of time. Real quick, consolidated tools. We had a ton of different tools.

Everybody's using them. Quality Center, Excel, we were using a little bit of TFS, but they were reporting out of with Project and Excel. Ops, I don't even know what they were using it sometimes. They had a homegrown kind of thing they were using.

So we had this fog of information. We went to TFS. There's plenty of other tools out there, but TFS was what we already had in place. And when we went with TFS, what that helped us do is we could map all the way through that life cycle.

Features to user stories, to tasks, to code, to test cases, to builds. Now let me quick do results because we're out of time, but I just want to quick. Anecdotally, just a shared vision. Everybody understood what we were doing.

We had distributed product knowledge, which gave us so much more confidence. That's what came away. We came up with this confidence as a team. Every time they put something out, they felt very good about that.

It wasn't the fingers crossed, I hope it works. And management confidence that we were going to meet our times and meet our dates and get good quality products out of there. As far as actual numbers, 50% less regression testing time. We still always had some regression because of the hardware.

We also had 30% immediate drop in defects. Like I said, half our backlog was defects, and we immediately got that down. Five to six of our last sprints released to beta. We had never done that before.

In fact, when I told other teams we did that, they thought I was lying. "There's no way you can do that." Yeah, we're going to our clients immediately. Client sees it at the sprint review. That afternoon, we're deploying it, which for us was huge.

25% decrease in our support overhead. A lot of that was because I had a lot more people who could do it, and we built a dedicated team for it. And then we had that successful 1.0 release. Got us into the market, and again, we saw a 21% jump in the market that year.

And just also, management came back and said this focus was the biggest thing we did that year. It was the biggest change in our company that had happened in the last 10 years. And lastly, just because Gene said we should talk about things we still need help with, I still work on muscle memory and management buy-in. This is great, and then when everything hits the fan, still remembering to do that.

So that's my goal when I'm here is to go to a lot of the different places and see how people are putting in that discipline and keeping it. And with that, I want to say thank you everybody. Sorry I went a little bit late. Thanks, and are we doing QA or are we done?

We cannot. All right. Sorry about that. Not to say, but this is Tommy Norman.

You can find him anywhere online. Thanks. I'm going to talk about it.