Q&A

A: So if your question is what's deployed where, today it's primarily handled through a BMC tool called BladeLogic, and that will show you what version is deployed to which environments. And the assets are in Artifactory, so that can get the binaries and stuff in there.

Most of this is handled directly by our DevOps team, so it's actually very small. The teams are best suited to know what's going where, when. So we've handed a lot of that over to them directly, and they handle the CRQs, the change records, and all those sorts of things to specify all that.

One of Erica's teams is called the ESM team, so they actually run the pipelines. They build the CI pipeline. But then the teams do all the work off that CI pipeline.

Q: And for our extended to the keynote, and how you feel about...

A: You can still use Diagonally.

Oh, no.

So yeah, we get audited, obviously, every year by Ernst & Young, and we have a lot of different traceability. They typically, when they come in to do the audits, they ask for tons of ad hoc queries. So you have to have that data ready to go and available to you.

We like XebiaLabs because it's simple. It's a very simple solution. A lot of the ones that are out there, like UrbanCode and Electric Cloud, they're very powerful, extremely powerful tools. But at the end of the day, we just wanted something that was very model-driven so that we could control it from a centralized release team and not let things go wild, wild west.

We used HP PPM Release before this, and there were a million spaghetti diagrams out there that people had no idea what the heck they actually did, and we did not want that moving forward, if that helps.

Moderator: Thank you. Who else has got a question? I saw some other hands up. Sorry, can you say your name and the question, and where you're from as well?

Q: Yeah. Alice Gray at Permitting. So I have a question for any of the Montana group. If you could roll back a year and a half, two years, what would you do differently?

Moderator: So that's from Kaiser, and the question is, what would you do if you could roll back two years and do it again? What differently?

A: You go first.

Yeah. So I think one thing I touched upon in our talk this morning was we didn't have enough of a focus on culture early enough in our process. And we really took the approach of, these methods speak for themselves. And for many people, they really do. But to get everyone in the organization on board with what's going on, I think we just should have had that targeted culture focus earlier in the process for us.

And we often get so busy in the day-to-day execution and making sure we're delivering new features, and that our operations environment is up and things are functioning, that it's very easy to forget that part without that targeted focus on it.

A: And I was going to add, too, I think when we gave our speech, we talked about D17 and First Niagara. We were re-platforming our application, rewriting the entire application, and then we had to accelerate all of that.

So if I could go back a year and a half and bring in more people to help us, I would have loved to have been able to embrace the microservices approach a little bit more in our new application. So that's a journey we're going to be going on now, but the application is already running. It's already in production. So the way we roadmap that out, there's going to be considerations that we'll have to work through now that we wouldn't have had to work through, maybe, if we had done it from the get-go.

A: Yeah. I was just going to say, I would have hired way sooner. Not all resources are going to be transferable to the new skill sets. And we should have hired two years ago. We should have hired three years ago. X amount of years. Before I started there.

A: Well, I do want to say, too, that was the hardest thing, because you have great developers on your team that have been with the bank or been with your organizations for years. There's a lot of tribal, built-up knowledge. And as you start to get, at least on the delivery side especially, maybe on infrastructure too, as you start to get into these newer tool sets, it is a completely different way of thinking, and you can't always retrain that. It takes a different mindset.

So we've been lucky. Even with First Niagara, we got some great engineers on our team, and it just made the light bulb go off. It's like, "Oh, wow. Yeah. This is really where we need to go and what we need to be asking ourselves." So I think realizing that early on to help you further your journey even more is definitely the way to go.

A: And I would just add that our story is a grassroots story. So we're creating a fair bit of waves within the organization. So equipping the engineers in my team with more of the softer skills, more orientation around empathy, and giving them a vocabulary to work with other folks.

Because, as I mentioned in my talk, going from that we're going to buy everything, there's a lot more things we're going to disagree about. And if we don't have a constructive way to work through those, it's just going to cause a lot of friction and tension that doesn't need to be there. Some creative tension is fine, but just kind of keeping an eye on one another and being good to each other.

Moderator: Okay. We've got a couple more questions. First, Steven from—

Q: iConnect.

Moderator: iConnect.

Q: I just wanted to follow up on the one question when you said you start with culture earlier. What does that mean?

Moderator: What does start with culture mean?

Q: Yeah. If you could just elaborate a little more.

A: Yeah. So we're still figuring that out, I would say. That's why we asked you guys to help us.

So we actually were spitballing some ideas on the plane of some different things that we want to do as far as next steps. One idea, if you're familiar with the Atlassian model, they have a monthly topic that each manager discusses with their direct report. So they've replaced their performance reviews with these discussions each month, where they're having a topic each month, and it's something in the leadership space. So it's not even always just DevOps, although DevOps is important, but just taking a step back and focusing on the people part. So that was one idea.

Scott, I haven't chatted with you about this yet, but another idea was having targeted time at staff meetings, where you're actually talking about that concept each week, and maybe somebody's responsible for that, and you reserve time at that meeting. Because again, it's very easy to spend a half hour or an hour, however long those meetings are, and talk about execution and operations and how are things going, and to neglect that leadership and that culture side of things.

So those are a few spitballing ideas, but I'd love to hear others, if you guys have things that you're doing that work well.

Q: Did Scott talk about the performance evaluation being a train wreck? Have you said that too, like change your performance evaluation and stuff?

A: Yes. So I think the question is, can I elaborate on what we did with the performance reviews to make it safe for cultural growth, right?

So I will mention that we didn't go right at culture directly. We tried to bring some new practices that revealed new cultural components indirectly. I'm a big believer that—I have a young child. No amount of me telling him the stove is hot is equal to him putting a hand on the stove and saying, "Wow, that's hot." So...

Hopefully there's no child services in the—

And I'm not calling our team children at work either. But I'm just saying that letting them feel it for themselves and having small vignettes that are in a safe place for them to experience those cultural changes has been important to us.

And just to elaborate on the year-end review process, we had all this stuff, all these competencies, and it was mind-numbing. And it was awesome because the product that we used, you could do the writing assistant, and you could actually pre-populate what your response was in terms of your competency. And so it was just a zombie exercise where the engineers would just click, click, click and not put any self-reflection into it.

So we stripped them of that, and we said, "We want you to do some self-reflection. We want you to think about what your own learning progress has been. We want you to be accountable to yourself and to your team in terms of that learning." And so we really structured the questionnaire to reinforce that. I don't know if that answered your question.

Q: No, she did. I didn't.

A: Yeah.

Q: I didn't want to take your time.

A: And we knew that people were going to be failing, so we went at that head-on and said, "Hey, tell us about your failures. And if you don't have any, maybe you're not pushing hard enough. And if you do, let's celebrate those, let's learn from them. Let's move on."

A: That's actually a great illustration of what Kotter talks about in Leading Change, right? That you can't change culture directly. It doesn't work that way.

A: Right.

A: In fact, he says culture often comes last. But what he says to get there is you start changing habits, and that's what you guys are really talking about. We start changing the language. We start changing what we do and how we do it. And over time, that difference will arc the culture in a different direction. But it's a lagging indicator. It's not a leading thing.

Moderator: Sorry, Barbara, did you still have a question?

Q: Yes, I do.

Moderator: Barbara from?

Q: Sun Life Financial. So my question centers around our organization's very large and old. We celebrated our sesquicentennial this year.

So navigating change and that kind—150. So navigating that type of change within a large historical organization can be somewhat challenging. But we have an incremental challenge in that each of our typical lines of business within the organization are almost like mini companies within the large company.

And I've been given the incredible mandate to implement global DevOps for the organization. So within that, there's a significant challenge traversing the global organization, never mind these little microcosms within Canada, let alone within the U.S., but trying to do that from a global perspective.

And I just wonder if any of you have had experience in terms of making that type of organizational shift, where it becomes somewhat spidered, if you will, to implement that type of shift.

A: I just want to ask, did you ask for that job?

Q: No, I did not. I was asked if I would like to take this job.

A: That might be the first problem.

Q: And it would be a great challenge. And I said yes.

A: That would be the second problem.

Q: But all kidding aside, it's a great challenge, great opportunity, but it does have its challenges.

A: So can you start with one team and show one success first?

Q: Yes. And that's exactly how we're trying to structure it, to take pockets. So we've broken down in the DevOps team, if you will, or the core team that we've now pulled together. We've broken it down into product lines of business that we were asked to load, client and profit centers in the organization, and we're starting to build.

So our team that supports our robot application is the first candidate to be deployed, right? And then we're going to start to deploy and roll out from there. That's the approach we'd like to take. There's detractors, there's resisters. As we heard about leading organizational change, there's—

A: I was going to interject real quick. I know that this is not the Chef conference, but there was a talk from about 2014, 2015, and Nordstrom was talking about how they were leading change. And there was a great talk about bootstrapping that, that we borrowed heavily upon in terms of formulating our own journey.

And Courtney Kissler back there in the back should be, I'm sure, very happy to talk to you about the early-stage formation because I think, as Scott mentioned, start with one team. Maybe those folks can actually go train the trainers after they're exhibiting the patterns and behaviors you want. You can start to then sprinkle them out and help other teams.

A: Exactly. So if you show success with them and build advocates to do it in other places, it's going to be a lot easier.

A: And I just want to add, I think you have to have the right team. So I wouldn't just pick whoever's available and, "Hey, let's put them on this project." I would definitely make sure that you get the right thought leaders if this is something that's going to start small and then grow, because there's so much opportunity that if we didn't have the right people at the table to call it out, there would have been things we would have missed.

So I think it's just being able to put the right team together to start your journey is also really important.

Q: Great. Thanks.

Moderator: We had a question over here from—

Q: Brian from Walmart.

Moderator: Brian from Walmart.

Q: Speaking of habits, have you found new habits or character habits for career? Have you found any effective ways to retrain managers and others to not go back to old habits and drive teams backwards?

Moderator: So that was a question to anyone. Have you found a way to retrain management to avoid them going backwards, back to the old habits?

A: I'm sure Charlotte has an answer about one.

So we kind of faced that at SRA. You saw from the numbers, we trained the top quarter of the company in changing their leadership mindsets and behaviors.

Q: Especially the lower levels is my thought.

A: Yeah. But as you saw, we worked at every level. And one of the things that we did is—so there's several things.

One, we brought that middle management into our leadership activity. So we would have these strategic offsites and so forth, and we were in the middle of this program, and we said, "We've got to demonstrate how important these folks are because they lead the people who really do the work." And so there's a practical and also a messaging thing there. So our meeting exploded in size. But wow, you talk about getting a lot of really positiveness from that.

The other thing we did is just this culture of mutual accountability, right? So it's one of the reasons why we formed people into cohorts. We did check-ins. It wasn't just go to a day of training, and then off you go, and there's no support system. The mentoring, the coaching, the follow-ups, the whole package was—and the people doing the coaching and the mentoring were very trained individuals, really good coaches.

So they were looking for the right things, they were hearing the right things, and providing that guidance. But what I loved is seeing the mutual—we'd be in a meeting completely unrelated to that, and I'd hear the conversations afterwards where people were calling each other, like, "Look, we're not going to lead that way anymore." Right?

And that's when you knew it was catching, right? It was becoming intrinsic in the thought process, in the mindset, and the behaviors. But that's not one meeting. That was a journey of constant reinforcement. Like David Marquet says in his video, you're going to go down this journey, you're going to fall back, you're going to get angry at yourself, and then you're going to pick yourself back up, and you're going to try it again, right? And you just got to do that, because it's not easy. This is a hard process, but you have to change and keep going.

A: Communicate, repeat.

A: Exactly.

A: I might have to tell the same person the same thing. I really did. I would hear myself telling the same person the same thing several times, and then finally, a light bulb goes off. But yeah, definitely over-communicate.

A: And I think the approach of go-see is really important here. I think a team standup is probably the best meeting if you want to spend 15 minutes and be down in the weeds once in a while. Obviously, we don't have time to attend all of our teams' standups all the time. But I think it's important, really, even at the higher levels, to once in a while go to those meetings.

You see how's the team interacting, what are they working on, how much WIP do we have, those sorts of things. That plan-do-check-act cycle, right? So we've told them to do a number of things. I've told them to do peer reviews. Are we talking about the peer reviews? Can I look at the work that's going on?

And just spot-checking those things and then following up. When you have your one-on-ones with that manager: "Hey, I saw these three things at standup that were a bit odd to me. Let's talk through why did this look that way." I've found that to be a pretty effective mechanism.

A: I'll add one more thing. So over-communicate and repeat. And I'm not great at this, and I'm working on it, but the tenth time that you tell someone needs to be just like the first time. Because if we're super frustrated, if we're sarcastic or disingenuous, that's going to undercut what we're trying to say, right? And it's definitely something I'm working on.

And the other thing I'll say is amnesty. So yes, you told the person a year ago about the thing, and they weren't ready. But if they're ready now, let the past be the past. Don't hold it over their head and say, "Well, you weren't ready to work when I was ready to work, so phooey on you." If they're showing up, you're like, "Okay, I'll take you in any form, and I'll start to work with you." So amnesty and patience, I guess.

Moderator: So there's a question at the back there. Sorry, it's you. Yeah.

Q: My name's Dave. I'm working with Jody, and I think this is a question for Keith. You showed us the charts on performance, I think, of the change that you found with incidents and features and stuff like that. But I was wondering what was very confusing was that—

A: Okay.

Q: It's been a long time. I was waiting in the back. But it looked like the conclusion that I draw was that throughout your transition, the blue was the change in the number of incidents over just about a year. I understand, even though there were lots of other metrics that were clearly improving over time.

So the theory as we go through that slide was to show that relationship with DevOps versus teams. And so I was wondering if you—

A: I was going to say, yeah, maybe. I'm sorry, Keith.

So you're asking about the metrics?

Q: Yeah. And then there's one other point, which is that the biggest drop for the number of incidents was this, I think, January 2018. But there was no explanation of whether that was a detection or a guess or what.

A: Yes. So there are a couple of things.

One, demonstrating API growth and subscriber growth. So the green metrics, the good metrics, are going up. You're making more money. TPS on the platform, consumption's going up, the other metrics around quality.

So the incident metrics were basically showing the monthly intake of incidents, basically, on the platform. And just last month, we dropped to around 630 incidents per month. And I just extrapolated, assuming at least we'll do that well towards the end. I just looked, actually, before this, and we hit last month 512. So we've dropped another 100 incidents per month. So we continue to go down.

That monthly run rate is attributable to a combination of, one, having build-run teams. Then they basically own the service quality of what they built, but also really having to focus on that kind of quality investment and looking and going and finding these root-cause issues that have been around for a while.

And those root-cause issues were probably originally fixed by either level one or level two, and fixed in air quotes, because they just put a Band-Aid on it. But then the issue comes back again because it might be a data cleanup, but the root cause was never really fixed of why it was there. Does that make sense? And does that explain what that graph was saying?

Q: Yeah.

A: Okay.

Moderator: Very good. Okay, thanks. One more. Sorry, what was your name again?

Q: Steve Thomas. Hi.

Moderator: Steve.

Q: So if you had to pick a driving metric that you're giving to Keith, other than a bunch of goals, what's the first goal in that 29 dominoes that will start driving change in test automation? Is it coverage? Is it time to commit to time to auto-test it? What's a good one driving metric goal?

Moderator: Okay. So what's the first metric when you're starting a transformation?

A: We were starting at a fundamentally different place, and a lot of times you'll hear a DevOps story that really starts dev into ops, and ours was really infrastructure engineering out.

And as I recall, we focused on one thing, and that was version control, because everything else needed to flow from that. And if we couldn't even version our infrastructure, that would be a problem. And while we're in the process of versioning infrastructure, we were teaching a whole bunch of other things. So it was that gateway drug to create opportunity for other conversations.

So we overloaded that metric over time, but it was still, year one, version control for us.

A: So I guess I'd follow the question up as, what are you trying to accomplish? Because I think for us, some things that we looked at metrics, really important: lead time. How long does anything take to get? Is it a server that takes eight weeks, and why?

Or the other one to look at, I always tell people, focus on lead time, and that's lead time around everything. And the other is time to recovery. So if incidents are your problem, looking at your time to recovery is something that's extremely important because it surfaces a whole bunch of other things.

One, visibility in what's going on in the system. It forces people to learn how to deal with it correctly. It forces good architectures, because if you're recovering quickly from a failure—you're always going to fail. So you can't prevent all failure. And I think Sidney Dekker has a talk about that.

But how do we recover from failure as fast as possible? And then that means you're creating really good architecture. So again, for us, it was some of those things around lead time and then recovery time. How do we recover faster, and how do we learn from that recovery and then make the system better afterwards and continue to do that over and over again? Because that enforces all kinds of great things in the environment and in the people.

Moderator: Okay, we have a question there.

Q: I'm Peter from Switzerland. I want to ask about DevOps and shared service. In particular, the CNC guide, you have this concept of service owners.

A: Yep.

Q: And you draw the example of InScope as a team, which is quite similar. But an autonomous team that is responsible for a service could also bypass, for example, the load data, the team leads. They're all load data, or there are others, like dev identity, I should mention security topics. I think that is a trust concern.

A: Yeah.

Q: How do you separate the teams from most autonomous connections?

A: Yeah. You want to answer, Erica?

A: Sure.

So I think there are probably a couple of different questions there. The question about how do you make sure that somebody doesn't go off and do their own load-balancing solution. We do have an enterprise architecture review board that you basically have to go justify why you would want to do that. So there are things where you do want to standardize. I'm sorry?

Q: Has that happened?

A: No. Oh, God, no.

Q: It sounds like it.

A: It does. Well, we're still looking at transforming that. And you're right, those can get dangerous very quickly, right?

But I would say that, for the most part, we don't have truly autonomous teams. They are dependent on a number of shared infrastructure services.

With the concept of a service owner, that service owner, for the most part, most of our service owners own products that are customer-facing products. And then you also have some infrastructure resource. So I've got some of both. I've got the load balancer. I've got some customer-facing products.

But often, if it ends up where you've got an issue with, say, the load balancer is affecting another product, those two service owners partner together. Scott talked about swarming, how both of those service owners are on an outage call if it's a major outage. After the fact, both of them are coordinating together with their teams to hold the post-incident review, to put the AARs together.

So there's still some crossover between that. But it's still those people. The idea is that they're looking across and we're meeting each other. Where we used to have silos, where it was handoffs between, now it's a partnership together. Does that answer your question?

A: So I think there's a couple of other things. Shared services is difficult to do when you start thinking about the traditional handoff mechanisms and ticket management that's done in IT, because you basically just end up with all these queues and people can't see, really, the end-to-end.

So one of the things that we're doing, what Erica has done with a lot of the tools, is this self-service automation. That basically now really opens up the standards and makes it easy for people to use. Similarly, with our Chef infrastructure, we basically make that available for all the teams to consume, train them, enable them. We don't want requests coming to those central teams to do what I would call traditional IT tasks.

We want those edge teams to do it. And if you make it easy for those teams to consume the standards, then they're much more likely to. And it becomes an evangelization problem and really an ease-of-service problem. If I have to wait six weeks to get a new endpoint for a load balancer, well, I think I'll just spin my own up because I don't want to wait six weeks. But if it takes two minutes to click on a button to create myself a new endpoint, that's very different.

So it is always a challenge because, I think this was a discussion at the last talk, how do you provide governance, right? Because you are required to do lots of stuff around security, but then how do you allow the teams the freedom to innovate? And balancing between those things is a constant challenge.

A: One other thing too, because I think for some of the shared services at Key, they're shared services, so they don't report in a hierarchy, but we have dedicated people in those teams that align just to the work that we're doing in digital. So they've got their own best practices and governance. There's experts in their particular field, but at the same time, they're part of our team. They're in my staff meetings. They're just like anyone else that would be non-shared. You know what I mean?

So I think that you can sometimes find a nice balance with that approach too.

Moderator: Thanks. So we had a question just there.

Q: Sure. Lance Taylor from Edward Jones.

Moderator: Lance Taylor from Edward Jones. Okay.

Q: I'm curious to hear from you guys, kind of related to the question that was asked here. How do you guys get that next step as you spread your organization at the team level?

We heard about pods, squads, how you're kind of using cross-functional teams, cross-skill-set teams, which is, we had to do that, put together and take across that further to say, "Hey, to build progress, we need to move people. We need to move them somewhere else," and have even 20 rules happening in your organization.

How did you guys kind of get through that, where you felt confident to say, "We can have these service teams in the organization, and it's a good thing"?

A: The short answer is, what we were doing wasn't working. And we had some leadership changes occur, and at that point, we decided to just blow it all up and basically move all the operations resources that were different groups doing different stuff. And we said, "Well, why don't we just put those on the teams, the folks that are building the software?"

So that happened in March of '16. And what we were doing previously, it wasn't getting better. It was people miserable. The performance of the system wasn't where we wanted it to be. So this was a big step for us, to basically do that. That's the short answer.

A: And I think you had a model to already kind of follow with Agile, right? Because once upon a time, you had dev teams and QA teams and maybe an analyst that was somewhere else, and we had seen how successful it was with Agile to bring all those people together.

So to us, it was just a natural extension of that concept that we were already operating with. Now, if you're not running Agile teams, then it's a much larger leap to go straight to that endpoint.

A: Yeah. And I'll just add, I think, so I actually have a team that we just moved to Agile earlier in the year, and it was just like, "We're doing it." It was in the middle of a project. We didn't start nice and clean, but in this particular situation, I think it was what we needed to do to get the—so there were gaps in, okay, what was being tested wasn't what we thought we were designing. There's just gaps all along the entire pipeline.

So we just said, "You know what? We're moving to Agile." We got the business together. So it wasn't like we were doing something to somebody else, but it was we were collaborating and saying, "How do we solve a problem?" And it really helped. I think this team specifically, we were starting and stopping so much because there were so many unanswered questions, and not everything was designed fully before the developers got ahold of it.

So in some cases, you just have to say, "You know what? We're just going to do it," and get everyone together in a room and just do it. And it's never perfect. It's never easy. We have gone through so many different variations of where we started to where we are today with that team, and I don't think any two teams are alike either. They all have their own kind of model.

It's all within an overall framework, but some have backlog grooming meetings, these durations. Others do it a little bit differently. They're still having the conversations and the collaboration, but I don't think you have to worry about carbon-copying either. You can pick a team, do it, and then another team could be doing it but differently.

A: At the end of the day, it takes the one common thing. So you can't force cultural change, right? So Steve, to your point. But doing the work differently, it starts to change people's perception of what's accomplishable.

But the thing it does require, regardless, is leadership courage. So if you don't have leaders at these different levels that are willing to actually take a risk and change, it's really, really hard. And I think that, to me, is some of the number one thing that a lot of these organizations struggle with. The changes that need to be made at the leadership level are hard because they risk people's jobs, and it's like they might not be the right people to do it.

And those things are extremely difficult for folks to make those decisions. And if they can't make those decisions, it's then hard to show how the teams are going to work differently to then get them performing better. So extremely hard, just to be clear, and it's extremely hard in really traditional IT shops.

We kind of have the benefit that we were a product-development shop that also had IT that we expanded, but we kind of have a bit of the DNA of building our own products and not being necessarily harnessed to kind of the way traditional IT thought processes were, where IT is off in the corner, and you send them business requirements, and they'll come back later. And so it's going to be even harder for a lot of more traditional IT companies to bridge that divide.

A: I think the technology is forcing the conversation more and more, right? Everything's becoming more software-defined. You have to be more cross-functional in nature in order just to support the new stuff that's coming out. So it's just going to be an evolution, and I echo the comments completely about the leadership side of it.

It's not going to fit everybody's job description and job role. Some people are going to want to stay Cisco-certified experts in specific engineering technologies. And guess what? That's just not going to fit the new world of where enterprises have to go.

A: Yeah. I'll just mention that I think Scott was kind of implying it was kind of a benevolent dictatorship, like, "Hey, this is the way it's going to be." And that's exactly what happened. I had all the disciplines in infrastructure engineering minus networking, and we just said, "This is what's going to happen. We're going to commingle storage and systems and databases and all that other stuff, and we're going to build these cross-functional teams."

And actually, a couple of the folks on the team suggested, they're like, "Hey, can we try this?" I'm like, "Let's do it." So it wasn't very clean, and we got it wrong the first—some personalities didn't quite click. So the next time, we're on our fifth cycle, we mixed it up a little bit.

A: And the example I gave in the speech this morning, my friend Steve in operations, he was like, "Well, this is ridiculous. Why don't we have these people sit?" It seems simple at the time, but you're like, "Yeah, that makes a lot of sense. Why don't they have one standup? Why do they have these separate meetings to talk about what we're developing and then what we're operating?" Right?

And so some of those things now, looking back, those are some things I wish we'd done earlier. But it took leadership courage to do those things, to be like, "Hey, why don't we try working differently? Let's try it on a team, right? And then see how it works."

Moderator: Okay, we've got a couple more questions. One from here.

A: So the answer is, we run everything. It's not as fun as we wish we did. That being said, we do have some kind of common standards, which we are kind of the bedrock to: .NET 2.2, operating systems, RHEL and Windows, and then some minimal viable standards around Chef automation and deployment.

Some standard databases like Oracle, SQL Server, and Postgres are really kind of those standards. Outside of those, then teams have a bit of freedom to—

A: Do what they want.

A: Also, we're trying to move more now to outcome-based. So for example, there's certain outcomes or technologies that are non-negotiable, like you have to use our ADFS and some of our security tools. Those aren't negotiable. There's other stuff like NoSQL databases where it's like, "Hey, we have two teams using Cassandra, but if we want to use, or whatever, Elastic, you could pick something else." We're not really rigid about those types of standards. They can verge off and innovate in those other areas.

A: And if you pick it, you've got to run it.

A: You pick it, you've got to run it, right? And you've got to pay for it.

And some of it, having to combine operations and development, right, when they look at the whole cost, they've got to be like, "All right. Well, I'm going to go to the product manager and ask for $500,000 for this new software development," or something. Well, they're looking at all that stuff now and being like, "Well, if we want to do that, why don't we use this open source or this other database that's cheaper, or one that we already have?"

Because it's going to take us six weeks to build out an HA MySQL environment or something like that. There's not expertise in what we've got to be able to accomplish, let's say, but we already have HA across all those others. So the outcome's the same. It's still going to provide high availability, BCV.

If you want to go do that yourself, you can, but it's going to take you six to eight weeks of product management. It's not adding any value to the business, and they're going to kind of shake their head and say, "No."

A: So Gene always throws this question out, actually, around standardization and how do you account for it with DevOps. And it's gray. It's not a black-and-white answer by any means.

I will say, though, because we actually went through something like this when we migrated HelloWallet over. So they were this startup, 32-employee firm that had all these different tools, running MongoDB, MySQL. And at Key, we have database teams that have really deep expertise, right, in the database platforms that our enterprise architecture team has said, "This is Key standard."

So right now, we're running on Mongo because we needed to get them over quickly. We had to be out of Capgemini by a certain period. But the intention is we're going to be moving them into our standard databases. And I can tell you, though, this team, they're mostly engineers, are very excited about it because they don't want to necessarily have to worry about databases.

And some of their jobs and queries aren't performing like they know they should be. They just don't have that full breadth of expertise in everything. They're generalists, which is great, but then there's also some areas where, in some ways, it is helpful to have these deep experts that can tell you exactly how to change your query to make your app run a lot more effectively.

So I think it's both ends of the coin. In some areas, I think it's great. In other areas, there's benefit having a single solution that you've got deep expertise in.

A: So I was thinking, this topic, this area, is just crazy debate right now of innovate, standard, innovate, standard. But I think, look at other industries, right? Let's look at the airlines and look at Southwest, right? They fly the same damn plane. And the reason they do that is you can train people, make it predictable, you can repair it, you can patch it, all those things.

If you run 55 different OS variants, and I know people that do that, then how do I maintain those and how do I make sure there's QR? And when there's an exploit, my exploit surface area is massive now, right? You have to patch all that other stuff and all the exploits.

So there are really good reasons to have some solid standards that are consumed, let alone how do you train people, right? If I've got all these different OS variants, or I've got all these different programming languages, at some point you need to hire, and you train, and you need to provide enterprise, really, stability for those types of things.

So I do think it is important. I think a little bit goes back to what I talked about in the earlier reference previously, is can you make those standards really easy for people to consume and use? And can I push a button and then can I get a VM, or can I push a button and get a database environment, like the public cloud? And if you can, then it makes it really easy for people to consume those because you're bringing them value quickly, especially if HA is dealt with and backed up.

And developers in DevOps don't worry about this stuff. My database is backed up, and as long as it's resilient, I'm good to go. But then if I've got to go do that, it's like, "Oh, man."

Q: Jim, do you believe that in the next five years all your standards will be legacy?

A: We will be. Definitely. Guaranteed.

Q: All the current standards will be legacy?

A: Yeah.

Moderator: Okay. We've got a couple more questions, I think. We've got one over here. We've got a quick one from Tobin.

Q: When you moved into the DevOps offices, how did you—

Moderator: So I think it is the operational cost. What happened when you moved to the DevOps environment? Did it go up or down, and what did you do with the people to keep that cost?

Q: All developers can do operational work.

A: So when you say that, I wouldn't consider that's the definition of DevOps—

Q: No, it's not.

A: —is that operations or developers can do operational work. That's not the definition of—

Q: Well, I'm not really sure what the definition of DevOps is. But the standard is better.

A: That's not the intent.

Q: Don't you still divide the cost into some operational cost, though?

A: Why do you care?

Q: Well, because everyone's looking at me.

A: Understood. But I have this discussion with my product management. And so I show the backlog with all the stuff on it that needs to be done, right?

So that cost, and you have an operations cost, and you have a development cost. Well, the cost is the same. Let's say it's 50/50. So you spend $10 million on that product, right? The real thing is how much business value can I deliver? Am I getting $1 million of business value? Am I getting zero, or am I getting investment?

And if I then improve the service operations, I can drive the cost. But I think the question for me that I look at now is really, you don't want to spend a lot on service operations, but the only way to drive that cost down is to then invest in it up front so that you make the services reduce.

And the question we ask is: did it go up or down? Right now it's the same. The cost is really the same.

A: There's a lot of different variables that go along with this, right? I think if we look at total cost of ownership to run and manage applications, for us it's night-and-day different from what we were doing.

We embrace open source wholeheartedly, and that itself is a cost-savings opportunity for most enterprises that really start looking in that direction. Our team sizes are smaller for applications that maybe run on our Kubernetes platform versus other platforms. But because most of it's all automated and scripted, you don't need an army to upkeep it. So I think that there's cost savings there.

I would say we've never done the full analysis. Some of the licensing stuff we have, and that's been substantial. But overall, I think there's a benefit there. But it ultimately, going back to what Scott said, it's about what you want to do with that number.

A: I was just going to say, so our presentation last year was called "When Ops Swallows Dev." And I think if you bring two organizations together and you've got a lot of tech debt in the operations space, it's very easy to see that happen. And I think we saw that happen at the beginning of ours. And I had developers in my office saying, "Oh my God, what did you do?"

But over time, you apply the development concepts to those operational problems, and they become development problems too. And no longer are they ops problems versus development. Now I've got infrastructure as code. So while I'm running infrastructure, I've got people who are checking out a source code. I've got automated tests to run it. I've got CI, all those same sort of concepts, and it becomes kind of blurred.

So I'll back up the comments about the cost being kind of merged together, but just kind of the evolution of what your work looks like over time does change.

Moderator: Okay, last couple of questions. Thomas from—

Q: Yeah. Thomas Omaros from Teleflex Medical Systems. I've got a question probably for Keith Lambert about regulatory compliance.

The main part of our solution is considered an FDA class medical device. So we have to comply with regulatory environments and all that. And we've been working on implementing DevOps mostly around the fringes of our medical device. But more and more, we're starting now to encroach on that core territory that's considered a medical device.

Anybody I've talked to, as far as our auditors are concerned and our QMS people and so on, they're not familiar with it, right? And wondering how you guys deal with this, because some audit probably went to the auditors that have never heard of DevOps and how you deal with the continuous deployment pipelines.

A: Yeah. The one thing our CIO is actually extremely passionate about is being upfront and forward with our auditors, with the OCC. So we have monthly and quarterly sessions that we do with the OCC to tell them about ideas that we're working on. And before we implemented anything, we had a conversation with them about it.

They didn't understand what we were talking about. There's no doubt, right? So I'm sure they had to go back and Google a lot of the things that we said we were going to do. I think that can work in your benefit, or it can also hurt you. In our instance, it helped us because they went and did some homework on it.

And I think we're making things actually much safer. We're not making anything worse. The separation of duties is still there. In fact, it's actually even better. And Topo Pal, he did a talk, I believe, at DevOpsDays that talked about the concept of the clean room, where really it's completely hands-off. There is no duty because it is 100% automated for what gets put into production. And so no one really has access at that point in time.

So I think it gets, for some reason, since it's so much change, you think it's scary from an auditing perspective. But you have to look at every single thing that you're talking about and say, "We're actually going to make it better. It's going to be more transparent, it's going to be more traceable, and it's going to be less human interaction."

If you run into a situation with auditors where they first react with some, "Hey, hope you've got to meet your compliance here," then you have to come back and say, "No, obviously, it's not working as normal." So that's the point about being proactive with your auditors.

A: Yeah. With an auditor approach, we involve them. The same discussions came up right when we merged the teams. They're like, "Oh no. Now I've got operations people sitting next to the development people." And it's much better, right? They can plan this stuff out, and we can automate.

But a lot of it was working with the auditors up front to get them to understand. But then you get the rest of the great benefits of infrastructure as code. So we use InSpec for compliance, and we had to explain that to the auditors. And once they get it, they're like, "Oh, I see that you're actually validating your compliance with it." And we actually produced a Word document from the InSpec so that it's easier for them to read.

But someday, my hope is that you've got auditors that can read that, and they'll go read it themselves, right? That's something that I'd actually like to work on and say, can we get an auditor trained in code so that they can read it, so that we don't have to translate it into the documents for them?

So yeah, a lot of it's really a collaboration, but you can get all the right controls with tools like Rundeck, Chef, that can then really control access to those environments so you don't have people getting into them.

A: Yeah. I was going to mention that Topo Pal from Capital One and Bill Shinn from AWS are doing some really good work, and you can take a look at some of those thoughts. And there's probably some things that are even auditor-ready. You could actually watch a talk with one of your auditors and actually help them to understand what you're trying to achieve.

A: Yeah. But putting on the other hat from the SAFe perspective, I came to the company after 15 years working with the federal government, so I've sort of lived that world and actually wrote our compliance guidance on the framework.

And what we found doing transformations in the government space is—in that article, we talk about things like, we've been talking about building quality in for years and years. Why can't we build compliance in, right?

So we actually look at the elements that drive the compliance, and we make it visible, and we build it into coding standards, definition of done, all these different things. The other thing we do is we invite them into the process sooner rather than later, because what any auditor, any person worth their salt, will tell you is they actually have a better result doing their job if they're looking at it in smaller pieces.

Now, that's not an easy step because they've got entire work models and staffing levels that are built the old way. But if you get them one-on-one and ask them, is it easier to assure quality and verify a small thing or a ginormous thing? If they're honest, what they'll say is, "Well, we're relying on a statistical sample if we do the big thing. If I see the small thing, I can actually look at it and understand it."

And then the last thing is, how valuable is it to have that feedback early on? We invite them in and say, "Look, if you see something and you tell us now, we can go fix it when the cost of change is small." And because we're continuously learning, we can do all those good things of, okay, how do we bake that in so we don't have that problem again in the future? And so the quality over time just gets better and better.

And we do work with medical equipment, some of the largest in the world, and we see they have FDA people in their back pocket every single day. So we understand the problem, and this is how they work. This is how they do it.

Moderator: Great. Thanks. We're out of time there, I'm afraid. So it just leaves us to thank our panel. Give a round of applause.