Start out by maybe congratulating all our sponsors like ElectricCloud and our gracious host, Jean Kim. Has it been a good conference? What do you guys think? It's wonderful.

Earlier in the week, I heard someone say, "These are the people like myself." So it's good to see DevOps practitioners actually getting together and truly collaborating. So what I want to do today is tell you a little bit about our story. My name is John Kosco. I'm an executive, over 26 years of experience.

I've played both roles. So I've actually been a delivery director in IT, and I've been an operations director. So I know both sides of the fence. If you'd asked me before DevOps if I liked releases, I would tell you, no, I didn't want a release to go into production.

I knew my staff would be up all night. We'd have long weekends trying to address the stability of production. But I've learned if it hurts, do it more, and really work on some of the techniques that support that. I do foster a culture of change, and I've heard that throughout this week, the theme of it is a cultural change when you bring DevOps to your organization.

So you need to be aware of that. And looking at transformations, it does not happen overnight. It is an investment that has to happen and continue to invest. And then I am recognized with continuous improvement, safe practices, which is called Scaled Agile, which we'll talk about today, Agile Scrum, DevOps, continuous delivery, service virtualization, and Lean IT.

So I've been involved with various things within the DevOps community. So today, I just want to share a story. I was told if you can share a story, you usually can make some good points and folks will like to hear it, hopefully. I'm going to tell you about a Fortune 100 company.

So this is one of the larger organizations. They're in the financial industry. They're over 100 years old. So when you have a company that's over 100 years old, you can imagine how much legacy technology they have.

And with this customer, the one thing that we always would say is a vendor would come in, they would ask us about our technology, and we would just say the word yes. Yes, we have it. Yes, we have it. Yes, we have it.

Name the database, we have it. Over the years, we've just collected more and more technology. But what we were able to do with DevOps, where we were able to increase productivity, the delivery of the team by 16 times. And that's 1,600% increase of speed year over year.

It was almost to the point where legal didn't want us to document that much amount of speed because we must have been very slow before we increased to that type of a speed. But it really did increase, and it's a concept of multiplicity, where it's building blocks that give you that speed. So I want to kind of take you through those building blocks. There's four key items that were in this case study for improving delivery.

The first one was leadership, and I think you've heard that throughout this week. Executive sponsorship is really key. How to get your executives to understand how DevOps can help their organization. If they have a vision, how you can align to that vision.

I think lean concepts are really about alignment, and if it's not connected to a vision or to a value stream, then it's waste. And if it's waste, then you should eliminate it. So leadership is one of the building blocks. The second is a proven framework.

When you look at Agile Scrum, Agile Scrum by definition was built for small teams, five to 10 teams. If there's any telecom folks in the audience, years ago, I went to Horsley, England, and did direct talk and call path, and they were small teams doing great things. All right? But once you start building those Agile Scrum teams, 10 teams, 20 teams, 50 teams, 100 teams, now all of a sudden you're going to have some issues around scaling.

How do I get all these Agile teams that are very good at what they do, and they can burn down their backlogs faster than pretty much the things you're putting on the backlog, but how do I scale and keep in align with the vision? So you really need a framework. And I'm going to talk today about Scaled Agile Framework. It's actually a framework based on three tiers.

So you're going to see two more levels added with SAFe. You have your team level with your Scrum. It's still there and doing well. We want to continue to leverage it.

But it also has a program level above that, and then it has a portfolio level. And I'll share with you why the importance of those levels really will help your organization. The other is clear definition of roles. Being a change agent, and I'd ask all of you to consider, are you the change agents of your organization?

People have a tendency to get emotional waves where they wonder, where do I fit in? Where am I at the table? Where is my seat? I always joke about Thanksgiving.

I don't know if it was you, but it was a big moment for me when I went from the kiddie table to the adult table. I felt like I have arrived. I'm finally at the adult table and listening to that conversation. Well, imagine not being at the table.

And although you represent your company today, think about your organization and who's not at that table. Who would possibly like to be at that table? Sometimes it surprises you who those individuals are that want to be at the table but not being asked. And then automation.

You cannot scale if you're going to do everything manual. You really need to leverage automation. So where do we start? Okay, John, I've heard what you said.

Sounds pretty good, but I don't know where to start. Where do I get to start? Well, where you start is a value stream. You need that executive that has some problem, some idea, some new service, some new product, some new sales channel that they want to exploit.

They want to be able to change and disrupt the organization. All right? In our case, for this case study, there was a senior executive that wanted to sell a certain product and services that the company, in its 100-year history, had never done.So you can imagine the disruption, all the traditionalists coming out of the legwork saying, "We don't do this. What you're asking us to do, we don't do this." And the executive said, "I don't really care.

For me and this organization, this is the direction we must go in." So themes were itemized with business objectives. He knew exactly what he wanted to do, he had a clear picture in his mind what needed to be done. He brought that business context to the decision-making process. I heard earlier this morning, IBM was saying they had a little challenge with making decisions.

And a lot of times with engineers, I love engineers, but a lot of times they swirl through decisions and revisit. What you can look at within business context is help make decisions for your portfolio so you can continue to move forward. And then I'd ask you this, in your organization, who truly can look at the entire delivery life cycle from end to end and make changes to that system? Is it the team members?

Is it the scrum master? Is it the product owners? Perhaps it's really the leadership. It's the senior leadership that truly has the view of the entire life cycle from end to end.

So having that senior executive is important to have that. And this is where we're at that top level, that portfolio level. So what we're going to do working with the executives, we're going to create business epics. Members of a portfolio management team, and ask yourselves if you have this, it might be called something different, but a portfolio management team established to work through a work in process and a Kanban system.

If you haven't learned about Kanban, there was a gentleman, I think it was from Capital One earlier in the week. He actually said he did a Kanban system with his wife on his honey-do list for the weekend. He put a Kanban board together. I don't know how it would work out in my home, but he tried it, and he kind of moved it to done.

And I will say this, when you finally explore Kanban, there is a human reaction to done. When you can move it from the states onto done, it literally is a human reaction to pride and a sense of accomplishment and a sense of worth when you have Kanban. So you review, you analyze, you estimate, you rank, and you seek approval for implementation. So you get that buy-in.

But here's the thing that we learn in this case study. There are limits of how much you can take in. Before this, I would've said, "Oh, we can take in as much as you send it. Bring it on.

We'll take it on." But in reality, you have to put WIP limits in place through that Kanban system, because if you don't, you're going to overload your system, and you're not going to give that team enough time to be able to work through that process. The second thing is lightweight business cases. I know we all hate them. We would rather not have to do business cases, but finance and financial people need that business case because they're writing the checks.

They're making the investment. And the one thing I'll ask you to consider is at the portfolio level, you have concepts that your executives have, and the finance people actually want to just follow the dollar. They want it to go from concept back to cash so they can reinvest. So having this portfolio level established within your Agile and your DevOps practice will align itself with business cases so that you'll get that investment from their financial backing and see how that will move forward.

And the key, what I'll share with you, because someone said, "I'd like some stuck in the mud stories. Don't give us all this rosy-colored glasses. There's got to be something. There's got to be something there were some problems." Dependencies.

Knowing the criticality of the dependencies between teams, between technology, between middleware, between your network teams, those dependencies, the earlier you identify them in that portfolio level, the better. Because you're inviting those dependencies or coming up with a plan on how to deal with those dependencies earlier in the process, which would really help you execute later on. And then finally, the business cases were then used with the appropriate parties to make a go, no-go decision. I've been in an operations position before, and we've had the big call.

Do we go forward and go into production, or don't we? And it's really important that when you get everyone saying, "Yes, we're going to go forward," well, get that at that portfolio level, and that can establish that buy-in. So let me continue on. Now I'm going to take you down from the portfolio level to the second level.

That's the program level. So now you have product management. These are members of a product management team, which are defining and ranking features. You have this vision that the executive has.

We need to do this. This is going to make us competitive. This is going to push out our new services. But what you really need are the features.

It's almost like a spec sheet. If you go to Best Buy and you want to look at what they have to offer, you look at the spec sheet, and you see what's there. What are these features? What are these things going to be doing?

And then you want to know the benefits. Right? Having a program level, you can now start identifying in a program backlog the features that will be delivered within your Agile teams. So as you're doing your delivery, you're mapping that delivery to the features that are going to be offered and the benefits of those features.

So the program backlog will ensure, and what we call a release train, and I'll explain that in a few minutes, is that the value delivery system you have, that concept to cash... thought process allows you to put all those features on the train, those that you rank at a high priority, and get those delivered for your organization. And then once again, the funding is requested. You've done your business cases, you've made your pitch as far as how much investment is needed, and you ask them to fund the entire release train, not just one project, not just one program.

Fund the entire release train to deliver on the value statement. And I heard another organization, I think it was Nordstrom's earlier in the week, mention about value stream mapping. If you've never done value stream mapping, it is a wonderful practice to go through, and it will really help you figure out how long does it take, what are all the various steps within our organization to take that vision and turn it into cash, seeing how to make it to be delivered. So now I'm going to take you down into the agile release trains.

Now it's an interesting term, release train. It's not actually a train, but it has some concepts of a train where it moves down the track. Once it leaves the station, it keeps moving forward. So a release train by definition are teams of agile teams.

These are for larger organizations, 50 to 125 individuals, all working together for a common vision. They have a delivery mechanism around the value stream. So this is where alignment comes in because a lot of times when you have agile scrum teams, and now you have 50 of them or 100 of them, you're going to start asking yourself, "Well, how am I aligned with my executive's vision? How do I know I'm lean?

How do I know I'm not working on something that may have nothing to do with the vision that's been put in front of us, or the features that we should be delivering?" So a release train aligns the teams to a common mission. It's about eight to 12 weeks of planning, deployment, retrospect, which is another big aspect of agile practices, right? And then continuous product delivery flow. The flow always needs to move forward.

And when you look at lean, I've been mentioning lean a lot, but you'll notice DevOps and lean have a lot of things that are in common. And looking at your flow, your delivery flow, and seeing, well, how many times does it just stop? I think Nationwide had mentioned, there's a point where the code's done, but I just put it over here in the corner. I'm just going to let it sit there.

And I know it's not getting any better, but it's going to just stay there for a little bit. So the flow has stopped. And in some organizations, they talked about the flow actually going in the opposite direction, and now I'm reworking the flow. So looking at your delivery life cycle and seeing how the flow is always moving forward should be key.

So one of the things about a release train, and you can ask yourself, do you have this in your organization, is a release train engineer. That's another term, a release train engineer, right? But basically what that person is, is they are the chief scrum master for the entire team. They are the person that all the scrum masters report up to.

So they are a point of, if the scrum master can't clear the impediment, they can bring that to the release train engineer that's tied to the release train, which is tied to the delivery. They facilitate program-level processes. They are the ones ensuring that everyone is working together in a predictable manner. And they're looking, as I mentioned, a point of escalation for the scrum masters to help facilitate continuous improvement.

That's key. When you're on a journey like this, you're not going to get it right the first time. You're going to have to keep trying and keep improving upon that process. So that's a role that we identified, and I know the gentleman's name.

I can see he actually does microbrewery, so he always would bring his beer in, the beer of the season. But he was very good at organizing and really listening to the scrum teams and seeing out how he could help towards the delivery on the release train. The next role we have in this story, and this is now going at the third level, which is the product owner, and you probably have this in your organization. The product owner defines and ranks stories within a team's backlog.

The product owner is empowered to accept new stories and indicate when the story is done. I can't stress enough the definition of done is critical when you're in these agile transformations. Expressing what done is, identifying acceptance criteria is key for the teams to know when they're done. And a product owner will always tell you when you're not done.

They're going to know when more has to be done for that team. And they're a key role between the teams, and they need to be a part of that team. And one of the things I'll share with you, just like the release train engineer is like the Uber scrum master, where all the scrum masters report up to them for checks and balances, the product owners report up to the product management team. So that once again, all your product owners are all working in concert with each other and sharing some of the challenges they're having and being able to escalate it up to the product management team.

So you're scaling down from the team level back up to the program level, which will help align those features that have to be delivered. And it is a critical role. If you don't have product owners and product management in your organization, really consider that because they are truly the folks that can really give you the definition of done. Now, scrum masters, I had some really good scrum masters that we worked on this account, and they were all over the world.

They always used to say to me, "You know, John, you're a director, and you tend to direct a lot of things. I'm a scrum master. I collaborate. I work with my teams every day to try to bring out the best that I can." So scrum masters' responsibilities help self-organize, self-manage, and achieve the goals.

And the other thing about a scrum master is they are educating and implementing scrum practices. They are the teachers of scrumSo once again, you may have this already in your organization, and you want to continue to leverage that in your organization because it's really probably working very well for you. And it was working very well for us. And then once again, now you have your developers and your testers.

And this is where developers and testers make up the majority of your agile team. They conduct the analysis, the design. In scaled agile, we have a concept called spikes, where if they're not really sure, they can go do a little R&D, a little research, and try to solve that problem. Maybe they haven't tackled it before.

So they do the prototyping, and then they write code for the stories. But working side by side, and I remember the concept with Agile Scrum of standing meetings. We actually had a Scrum master that made no chairs in that room, that everybody was standing. And he really wanted to see the developers and testers working together to do analysis, design prototyping, and then work on acceptance test criteria together and leverage automation.

So that's pretty much, when we talk about roles, those are some of the roles. I'll share with you the scaled agile framework, and you'll see more roles on that, but I kind of wanted to pull out for you today some of the cultural changes, some of the roles that you may have already, as I walk through those roles, you say, "Okay, I know who that person is. I know that. Oh, maybe that's a new role I haven't thought about." But the next thing is automation.

We talk about those four building blocks. Automation, Wikipedia says continuous delivery is the design practice for software to automate and improve software delivery. It looks at testing, automated testing, continuous integration, continuous deployment, at a high standard and easily packaged deployable test to go-to test environments. And to be able to be rapid, reliable, and pushing out enhancement with low risk.

That's what continuous delivery is by definition. So let me show you an organization prior to what we did with DevOps and some of the things that they were dealing with. One is delayed integration testing. Now tell me, you may have this in your organization.

Too many bugs are escaping downstream. One of my favorite things as an operations director was caused by change. We had it in our ticket system. Everything that went in that was caused by change would get flagged, and boy, did we not like caused by change.

So bugs are moving downstream and delayed integration testing. I think one of the speakers this week mentioned about idle time, where either developers are waiting for environments to be available or the testers are waiting. So the second one is lack of release or environmental automation. Manual processes are going to lead to poor release quality.

So once again, that idle time. We actually had some teams that actually put the idle time in their estimates. They said, "Based on manual release practices, we will wait hours per person for the environment. We're just going to put that in our estimate, so just expect us to pay for us to sit and wait for it to be moved in." Lack of automated testing.

Small changes can have big, major, unintended consequences. As a director of operations, after we did the code freezes and I would see about a code freeze getting broken and something last minute just has to get in, a lot of times you see big impacts from that little tiny change that went in that was supposed to be very insignificant. And the reality is because it doesn't go through full regression, it doesn't have the same coverage that the earlier changes had within the development. And then lack of visibility.

For operations, DevOps, that's our practice, DevOps. Ops needs to have monitoring. Ops needs to understand what have you just done to the production environments. Have you improved things?

Have you slowed things down? What have you done into the operations area? And our business operation area wants to know what impacts have you done to what they do, how they do their job. Did you make their job faster?

Did you make their job slower? Are they now juggling multiple screens to do their job where they used to only have to do it on one or two screens? So these are some of the challenges before automation was introduced. Now let me take you through the transformation to DevOps.

In the middle of the slide, you're going to see something called a deployable asset, and we started using virtualized assets. The reason why this is so significant is as developers were working in their unit test environments, in addition to the system under development, we were virtualizing the assets in the upper environment that they integrate with, and then we were making available on their desktop. I'll never forget the day we did this. It was a proof of concept, and we had a developer come in.

It was a retirement account. It had $80,000 in it. I said to the developer, "Go ahead and liquidate the account. Take all the money out." He says, "Okay." Ran my test, everything worked out fine.

I said, "Okay, well, do it again." He said, "John, I can't do that." I said, "What do you mean?" He goes, "Well, I liquidated the account. There's no money left in it. It's been wiped out." So I said, "Okay. Well, let's do with service virtualization.

Now we're going to shift you from the physical systems to the virtualized asset. Do your same test." He said, "Okay, I did it." I said, "Go ahead and do it again." "John, I've told you, I've liquidated the account. I can't do it." With a virtualized asset, you can do that thousands of times over and not actually impact the actual account. So it really gives that developer who's shifting left the ability to retest and retest and retest with whatever data scenarios you need.

I said to him, "You could change it from 80,000 to eight million. You could make it $80. You don't have to change any of the upper systems in the upper environments to make that happen. You could just change it in the virtualized asset." So once again, by using systems under development and virtualized assets, you have a true agile development.

You've shifted left and made that developer more integrated within their testing.Also with continuous delivery, you're moving your deployments, these assets, into the upper environments in a way that is getting more solid and more solid and more predictable and more predictable as you go. So the more you do it, if it hurts, the more you do it, the better it'll get from a predictability standpoint and a stability standpoint. And then continuous validation. You need the automation to run.

We used to call it follow the sun. When the coders checked in their code and the builds kicked off and we ran all our JUnit and our Sonar and our security vulnerabilities against their build, we were also running all our automated tests. So by the time they came back in the next day, all the results were already there about what did you just do to last night's build? What was that automation result?

What was the feedback we could give the development team? And then the other thing I'll share with you about service virtualization, a lot of organizations that do testing struggle with test data management. How do I get the coverage? How do I get enough data?

And I love this one as an operations director. I need to go to production to get data. Wow! Well, I'm just rolling out a new feature that no one's ever bought, and they have no experience with it.

So where are we going to get this data out of production? And it's always that challenge. Where do I get data out of production or data to be able to coverage my test data? So looking at virtualization, you really can start creating parameterized tables, which will allow you to create those scenarios that you need to do that coverage.

Release automation, once again, sped up the release cycles. Our first release was into dev int or into dev. Prior to this, it took us two days through ServiceNow tickets and manual processes to move code into that dev environment. When we were finished with automation, we were doing it in 10 seconds.

Imagine the amount of time you can now save and how many more attempts you can to migrate that code into those various environments within seconds as opposed to days. Reducing errors, higher quality, releases are simple, standard release processes because they're scripted. You can now become predictable because you've already worked out how the scripting's going to occur, and now you're just running that script again and again and again. And it enables frequent releases.

When we introduced this, we were happy when bugs were found earlier on because the cost was cheaper, and we knew we could address them very quickly and it's about feedback loops and closing out those loops. So the more frequent releases you can do, the faster you can get to address and create feedback loops to address any defects that are found. Reduce costs and improve visibility. Once again, we always want to be able to monitor and see how the delivery deployment chain is progressing and be able to share that information.

With virtualization, as I mentioned, I shared the story of the developer that shifted left. And he was known as the shift left developer. People would come up to him, "Oh, you're the shift left developer." He said, "Yes, I did. I shifted left." It was important because they made that change.

They literally took a defect process, and they put it up on their desktop. Integration testing with virtual assets. I've seen virtual assets where you do the testing, and then you switch from the virtual asset to the physical, and there's no difference. Imagine the amount of time when you're waiting for a vendor to deliver something to you three months out based on a spec, and you can't test it.

Well, what are you waiting for? Well, waiting for them to deliver it. I've given them the spec, but they haven't built it yet. If you create a virtualized asset, you could have that same spec built out in a virtual asset and start testing immediately to see how you integrate with it.

Increased infrastructure availability. You can do performance testing with service virtualization. You can make your virtual assets slower. So you could do high availability, standard response, and slow response.

And you can see how will the development community react as things are slowing down in a controlled environment, in their unit test environment. So now you're doing performance testing earlier in your process. It's not that last thing you do before you go into production. You've done performance testing earlier in that process, so you know will the system stay together as a degradation of speed occurs.

So you want to be predictable. So this is the last slide I have. This is the scaled agile framework, and what I'll ask the team, I think they asked us to say to the group, have an ask, right? Something to ask as a presenter.

What I would ask you to do is look at the scaled agile framework, and that's the website, scaledagileframework.com. It's out there right now. It's public. It's free.

You can have access to it. Scaled Agile Framework. This is a drawing. This is called the big picture.

Every one of these icons that are on these pictures, you can click on, drill down into it, and you will get complete explanation of every role, every process, taking you through all three levels. And my ask for you is, please spend some time, maybe not today because it's been a long week, but spend some time looking at this website and send feedback. Just give impressions, good or bad, about how potentially you might be able to see these additional levels being introduced in your organization and helping you scale. So with that, I want to say thank you for your time today.

It's been a pleasure, and have a wonderful week.