Gene Kim

One of the people who was at the DevOps Enterprise community from the very beginning is someone named Ross Clanton. He helped lead the DevOps transformation at Target, a large US retailer, since 2012. Two years ago, he joined Verizon as a technical fellow to help Verizon elevate their software engineering capabilities.

He spoke last year at the US conference, talking about what he's been doing at an enterprise level from Verizon headquarters, which is in New Jersey. He encouraged the Verizon Enterprise EMEA team to present, and when we found out what they're presenting on, we were very excited.

Sanjeev Jain is a CIO for Verizon Enterprise EMEA, and John Scott works for Sanjeev and is embedded also into Ross's team at headquarters. Presenting with them is Oliver Cantor, Director of Product Strategy, who will be bragging about how, as a result of Sanjeev and John, we're able to identify a market need, we're first to market, and will likely have market dominance for two years.

I'm excited about this story because it's another example of how technology is creating material business value. So come on out.

Sanjeev Jain

Thanks, Gene. Thank you very much for that introduction.

So let me start with a question. As we go through this DevOps journey and digital transformation, one of the most talked-about things is the role of a CIO going forward. The chief information officer, as we used to call them.

So raise your hands if you think the CIO going forward stands for Chief Important Officer. And please don't raise your hands if you think CIO stands for Career Is Over. So raise your hands if you think it's Chief Important Officer, and please don't raise your hand if you think it's Career Is Over.

Now, right now, I'm not feeling too good about my role. Thank you. I think there were quite a few hands. I'll sleep well tonight, might as well have a gin and tonic as well. So I think I'm safe for the next few years.

Now, let me tell you a story. About three years back, I walked into an ELT meeting, and ELT stands for Executive Leadership Team. And that meeting is made up of a CEO for the EMEA region, a CFO, a CMO, head of sales, head of operations, head of engineering, and me as the CIO.

For years, I was used to walking into that meeting with a tail between my legs because I just don't know which organization is not happy with the speed at which we're delivering software. I was the chief supplier in that meeting.

Till that day when I walked into the room and the head of sales says, "Oh, here comes my most important customer." And I'm thinking, who's he talking to? And that's when the penny dropped on me. He said, at Verizon, when we're developing networks, products, services, we're selling it to the CIOs and CTOs of the world.

Now, as a CIO for the international region for a Fortune 32 company globally that invests $17 billion every year, capital investment, with 150,000-plus employees, and pretty much offices at all parts of the world, if our products and services are not good enough for you to consume internally, there is something wrong. There's something either wrong with the speed at which we're delivering the products or the customer delight we're putting into our products.

Wow, that was a heart-shaking moment for me. From a chief supplier, as one part of my role, I had become the chief customer of the same products and services. Speed, the time to market, wasn't the only part of my responsibility. The quality and the efficiency and the customer delight in our products was an important ingredient.

I think for me, that was a perfect definition of DevOps: speed, efficiency, and quality.

That was the day where we embraced DevOps pretty much in every life cycle, in the life cycle of everything we did. We embraced DevOps into how we transform the products and services. We embraced DevOps into our customer engagement, measuring the NPS and the customer satisfaction score. We embraced DevOps into empowering our employees with digitizing their experience. And we embraced DevOps into optimizing our operations.

We had to go through a whole new technology stack and scale it to every product and every release, every software release we did. We had to change our organization. We had to change the kind of talent we were nurturing. We had to create a culture where IT, or the technology organization, was not in the back office supporting business. It was our business.

We had to change the organization culture where our product development team came and talked to us about their vision rather than the user stories. We had to change the culture of our organization where product development teams came to us and talked about the urgency, the speed to the market. And in return, we gave them a competitive advantage by delivering that software of high quality and very fast and giving them something that's going to last a little bit longer than the first few months of the product because we were first to the market.

So today, we're just going to talk about one of those use cases. And if I can just bring on stage Oliver Cantor, who's the Director of Product Marketing, and he can explain how he had one vision that he came and talked to us and with his blue-sky thinking about transforming network.

Oliver Cantor

Thank you. Thank you, Sanjeev.

Yeah, so this is the easy bit for me, really. Normally at a DevOps-type conference or an IT conference, there's a lot of talk, obviously, about DevOps, Agile, these kind of things. And if anything, showing off and talking about products and leadership is not really the done thing. The kind of salesy piece is put to one side.

But I get to come here today and say this has been very successful. This has been a real market driver for us.

What did we do? Well, Verizon traditionally builds global networks, sells and manages them, and multiple layers: IP networks, MPLS networks, optical networks. These are big, vast, static ecosystems globally that move traditionally pretty slowly. And they're very fixed. The engineering probably hasn't changed radically in about 10, 15 years. People buy big chunks of bandwidth. They think of us as expensive, slow-moving, takes a long time to deliver anything.

With all the pressures that are in enterprise today, enterprise going through a revolution with, you can list as long as your arm the changes out there: the fourth industrial revolution, climate change, millennials, social media, 5G, digital transformation, blockchain. You name it, businesses are under pressure.

And every type of business as well: pharmaceuticals, big banks, pet shops even. My brother-in-law runs a pet shop or pet industry that has to change. They all need better answers from IT, but they also need better answers from the network.

So about four years ago, we had a vision. We decided that we needed to make our networks agile. This is unaware of what was going on in IT at the time, unaware of Agile and this whole movement of producing software quicker.

We traditionally bought things from vendors, the traditional Ciscos of the world, very slowly implemented them, put equipment out in customer sites, CPE, customer premise equipment, data centers, big chunky bandwidth around the world, submarine cables. This was no longer going to be sustainable. We needed to change the network, transform them to be agile, to be at the speed of software.

So what did we build? Our vision was to have an entire ecosystem of software components, traditional routing and software-defined routing and security and optimization. All of these network functions we wanted in standard software blocks now.

We wanted to deploy them on traditional equipment, on new standards-based white boxes and gray boxes, server-based equipment, put them in our data centers, and ultimately put them in the cloud. And we also needed middleware or orchestration layers to manage all of this automatically, zero-touch provisioning. You name it, it went on and on.

So the ask was absolutely massive for IT.

Let me give you an example. A very big bank, this is just one of many. Normally, they wanted to run a new application in their branches. Banks' branch real estate is really dying as a concept. They don't know what to do to reinvigorate the branch experience. So they wanted to run an app that would stream video and do an ask-the-expert kind of sessions.

Normally, that would take us months to deliver a router, a firewall, an optimizer, lots of expensive, heavy bandwidth that we would implement, normally just MPLS. A lot of cost, a lot of truck roll, a lot of people going to site. It would take months and months and months.

Well, we could roll out one box, a white box. We could do it on hundreds of sites. When it was configured by whoever was on site, i.e., plugged in, it would automatically, over Wi-Fi, find home and build itself and self-configure through the orchestration systems. And then they'd be up and running in minutes rather than weeks and months and days.

And this bank is literally now giving their... We've saved them months and months and months of implementation time, which is all customer experience and business that they're getting.

So it's been a massive success. And all of this massive IT task was started in 2015. When we were originally asked in May could we deliver a minimum viable product in August, we presumed they meant the following year, but no, they meant in three months' time. And we'd never done anything at such scale and so fast. We wouldn't even write the requirements in that time. But we did.

Maybe the minimum viable product was slightly more minimal than viable, but we rolled it out, and we couldn't have done this without our IT partners. We went into partnership, and I think, importantly, we needed much more than just a technology answer. We needed a whole cultural change from IT, which is what we got.

Sanjeev Jain

Thanks, Oliver. He really didn't give us an easy challenge. For years, the way we consumed a network with routers and CPEs, he just wanted to put everything as software.

So I just wanted to show you, DevOps was at the heart of it, but we had a whole suite of ingredients that let us transform ourselves. We went into this mode of running hackathons. We did design thinking solutions, crowdsourcing. We started using the MLs. We started using the cloud culture, the APIs, the analytics, the Agile, everything.

But right at the heart of everything we did was the speed of the software, whichever ingredient we looked at. And right at the heart of everything was DevOps.

And I want to bring on stage our DevOps evangelist, John Scott, who really took Oliver's idea, his vision, and converted it into reality and started a DevOps journey.

John Scott

Thanks, Sanjeev.

Okay, so how are we going to build that muscle? We went for an in-your-face learning style called the Verizon Dojo, and this is the Dojo in Basking Ridge. This is how we're going to build quality software to keep pace with market demand.

So I don't know if this is what you're expecting, but this is the setup. It's light, it's modern, it's breezy, sort of place you want to feel inspired to come and work. As you can see, there's plenty of areas for breakout sessions to aid collaboration. There's plenty of technology there. There's Jamboards, there's ClickShares, there's iPads. It's the sort of place where we want you to come, get creative, think big, and go for those blue-sky solutions.

So let's delve a little deeper into the Dojo. What happens? This is where teams come with their business partners, and they spend six weeks learning about engineering with their Dojo coach. Sounds a little bit like training, but here's the key: it's training on the job, bringing their real work, real business problems for real business outcomes.

So how does it all begin? It starts with a charter. Gives teams a goal, a sense of purpose, a raison d'être. So what happens? They spend six weeks iterating very quickly towards their various MVPs, learning all the time about advanced technologies from their Dojo coach.

This is the important bit. It's all about gaining knowledge, doing stuff, failing fast, understand your context and constraints, and then iterate, all within a very short cycle.

So how many Dojos do we have? We have five. Three in the US and two in India. We've now gone upwards of 50 teams that have actually gone through the Dojo journey, and we have a big backlog of teams waiting to start their journey.

We've up-skilled over 500 engineers now, and what we're seeing is, on average, a 30% increase in speed of delivery for those teams that have gone through the Dojo.

But it's not just the flat metrics. We are very keen on metrics, but it's also the feedback. This guy here, he talks about being in a silo. He's lost, but now actually feels that he's actually part of a team. He's excited, he's energized, and that's what we're seeing.

If I can take these guys from being skeptics to believers, to being evangelists, that is how we're going to build the community, from the grassroots up.

A couple of patterns emerged coming out of the Dojo. One is, sometimes teams just cannot physically be co-located for six weeks together. So we're hoping later on this year, we're going to bring you Dojo in a box, which will be a mixture of that physical Dojo experience as well as virtual.

We also realized we needed to pivot the organization and the way it was structured. Let's move away from silos. Let's think about being mission-focused, stable teams.

So we had a squads and tribes model. So for our friend here, Oliver, we had created an SDN tribe, which created two squads underneath: a BSS squad, so these are the guys that are looking after the quoting through to the cash. Whereas the VNS guys, these are the more people that are interested in actually getting the product into the customers' hands.

We had to be thinking about product-based, thinking along the whole stack towards business outcomes.

So how did our engineers find this? This is the exciting bit. They actually feel full of energy. They're interested in the product. How's the sales doing? What else can we do next? Really exciting stuff.

I don't just want it to be about delivering software. We don't just want it to go, "Oh, it's come in my in tray and it's gone out my out tray and I'm going home now, and code, I don't really care." It's about the excitement around that product, and that's the key.

The other thing which we needed to provide was chapters, so we could have like-minded people all talking the same language, so our engineers could go across squads. Otherwise, we've just created more silos. So we had the concept of chapters.

So we started with development, testing, analysis, but latterly, we've gone more granular. So now we're talking about machine learning, artificial intelligence, monitoring, databases.

So what else could we do for our teams? We've given them an immersive learning experience. They're product-focused. They're excited. They're energized. So we need to give them a blueprint. We need to start thinking about practices over tools. Give them some guardrails with a supporting technology radar.

So I introduce to you the Verizon delivery pipeline. The key here, it's all to do with implementation and integration, and that will make sense in a moment.

So we start again with Oliver. He's got another aha moment. "Guys, we need to go and do this." Oh, God, here we go.

So what happens next? We actually have created a funnel process to get all those ideas into our issue tracking system, into Jira, and we somehow need to get it from there all the way over into here, into production, in a better, faster, safer way.

So I'm a developer now. I don't really want to go and spend time in Jira and context switching. I want to be adding value, which is in my IDE, so in Eclipse, for instance. So we need to implement and integrate Jira into the IDE, and so on.

So if you take a look at our commit stage up there, you then start moving into source code management, so then my IDE's integrated with my source code management. Then what's going to happen? We then string up maybe unit testing, maybe performance testing, maybe some level security scans, and QA. It's all to do with implementation and integration.

We have to remove those manual wait times and laborious manual processes.

So we've broken this down into a number of stages to keep it simple for folks. So we talk about enablers. Those are the smaller boxes and stages. So we have a commit stage. We have various testing stages, all the way through into release stage into production.

So again, I can't quite give you all the secrets today, but if there's one thing you take away, implementation, integration is key.

Sanjeev Jain

I tell you one thing, I think this was brilliant for solving what Oliver wanted with his blue-sky thinking for one product. Now, unfortunately, in our space, I've got 50 more Olivers thinking about new products, some of them much better looking as well.

But 50 more Olivers all coming out with different thinking, different products, and we had to scale this. We had to scale this very fast across our organization, across all geography locations. So John, do you want to touch on that one as well, please?

John Scott

And I think the key point here is, and I really wanted to bring this up, I came to last year's conference, and if you can make this pipeline better, faster, and safer, as Mr. Smart educated me, it doesn't half make everyone a whole lot happier.

So let's take a look at a couple of the ideas around how we can help you scale this.

We've got something called Verizon DevOps Days, which we've recently rebadged Verizon Tech Days because we're embracing a wide-ranging set of technologies across the whole of the organization.

These are internal conferences. They are driven by the community for the community. They're the guys that set the agenda. It all starts with a call for papers. Guys, what are we going to talk about? What's hot? What's not? What do you want to connect on? What do you want to reuse?

So let's take a look at some of the metrics on the screen. We love metrics. So 7,000 attendees across five global events. A great mix of internal as well as external speakers, and probably the most exciting metric for my daughter, 400,000 Twitter impressions.

It's not just about the metrics, though. I love the quotes. "This felt very un-corporate." No, this is the new Verizon. It's exciting. We've gone digital. This is the new Verizon.

If we just take a look at a couple of the photos on there. There's one of our DevOps leaders there with a British bulldog mascot, a certain Mr. Clanton, to put everyone at ease as part of the whole experience. Then we've got the chap here with his seven-foot dinosaur. Again, is he a bit mad? Maybe. But there's actually a serious note to that. We need to adapt, or we're going to die.

So let's bring on some other fun concepts that we're trying to do. Let's bring on the gamification. This is the DevOps Cup. This is something we're particularly proud of, and pretty unique, I hope, in Verizon.

This is all about rewarding teams for doing the right things, growing their capabilities, and contributing back to the community. Key point is contributing back to the community.

I'll talk a little bit about the history and about how we're going to scale this. And a bit of a spoiler alert: funnily enough, it's all to do with automation.

So it all started in VES by a few of us that were trying to make engineering a little bit more fun, a bit of excitement around the whole piece. So let's bring on the gamification.

So we started small, and then we slowly built it up and up. So for instance, in 2016, we had a security-first focus, again, exciting people around that. Then we started increasing more events. We then started doing flash builds. We then started thinking about best in classes.

So are you best in class for mainframes? Are you best in class for COTS products? Hell, this year we've even got stickers, so we're really on it.

The key here is you start small. Make sure you align to your corporate goals and build that core muscle. Please do celebrate successes. We really need to celebrate those because change is tough, and organizations just don't want to change. They are resistant.

So how do we sit in 2018? 350 applications, thousands of developers across the globe, and now eight lines of business, which is all lines of business in Verizon. So we're all-encompassing.

So the key question becomes: how the hell are we going to score this? We've got this game. We've now got everyone involved, excited, wanting to win. Well, clearly, I'm not going to be able to manually score this anymore. And Google Sheets, it's one of my friends, but it's not going to save me this time.

We need to ensure we've got telemetry across the whole of that pipeline at all those different enabler points. What we found is a lot of teams were measuring a lot of things, but not necessarily the right things. One of my key phrases is "no more vanity metrics."

So what did we do? We partnered with Capital One on their Hygieia product to standardize and streamline our data collection. This has allowed us to automate, baseline, and track over time our data collection and also our scoring for the cup.

What's great about Hygieia is it's very lightweight to onboard for teams. And secondly, it also interacts with tools that teams use on a daily basis, so like the Jenkins and the Jiras. So we've made it a no-brainer for teams to jump on board. You need to make it easy for people.

But what makes up the cup?

Engineering. This is underpinned by the pipeline.

Technology. Are you a modern app? Are you able to refactor your app? Do you have a well-architected framework?

Community. Are you building that community? What are you delivering back? Are you doing tech bites, which are our small elevator pitches? Are you doing megabytes, which are the longer talks? Are you running events? What are you doing back for the community? Are you sitting with your sales teams talking about how digital IT is, and they really understand how you can help them?

What's great about this model, and Sanjeev knows this, it's very simple: E, T, and C.

So what happens at the end of the cup? Well, clearly, the top-scoring teams go through to the final. And in the final, they are judged by the internal Verizon DevOps gurus, who will let through a chosen few. These will then be judged. They will demo and present to external thought leaders.

And so at this point, I'd like to thank certain people in this room, because they've been on our previous judging panels. So the Amazons, the Disneys, the Atlassians, and again, even Mr. Gene Kim, he's been on our previous judging panels. So again, I thank you very much because without that insight and feedback, it's really helped us on our journey.

So let's just take a little look at some of the things that they are being judged on. Again, a lot of these are around the whole DORA model. It's a really good, solid model.

Business resiliency, so stability. Are your guys just waiting for a production to go wrong? And then you're going to get phoned up, fire drill, middle of the night, "Oh my God, oh my God, we need to fix production, we need to fix production." As we heard yesterday, we're just sticky plastering it and moving on because everything's changing so quickly.

Or are we practicing proactive business resiliency? So are your guys trying to run chaos scenarios and test or even in production? If you are, we're going to reward you in the cup and you will do well.

Security. Are you just firing off scans and forgetting about the feedback? "Oh, I've run the scan. That's right, tick. It's compliance." Or is there a good feedback loop? Are you actually failing?

So if you do code commit, you run a security scan, you have some criticals, you have some highs, and it actually breaks the build. If you do, that's pretty sophisticated in my situation. So we will reward you for the cup.

So how's it made our developers feel? Well, the exciting thing is it's really given them a sense of purpose. It's made learning fun. They do feel excited about these things. And that manifests itself in the metrics. We're seeing a 50% increase in production deployments.

2018. Well, we're making it bigger and better, obviously. But we need to also be looking at the improvers, the guys that aren't at the top of the table. We need to be making sure we're not leaving anyone left behind. You're only as quick as the slowest part of your value chain, and that's key.

Let's talk about applications and their maturity level. Can you go from crawl to walk to run? Legacy should not be seen as a bad word.

To keep the momentum up, how are we doing that? Smaller rewards, more recognitions, rather than just a big bang finale at the end of the year. But hell, this is the year of the World Cup, and remember, there can be only one winner. So in Verizon, you get a great big cup when you win and bragging rights for the whole year.

Sanjeev Jain

That's excellent. And to be honest, this is the most excitement I've seen in the organization. When John and the team ran the first cup in 2016, teams competing against each other to score points and optimize their operations, that was the most excitement I've seen.

And so what is it all about? So we've talked about cup, we've talked about gamifying it.

John Scott

I like to think of it as this is the plumbing. We're promoting best practices across all our teams at scale, regardless of their maturity.

Sanjeev Jain

Now that's excellent. Now, if you hear about John's story, he started small with Oliver's project. We scaled it with the cup. Legacy is not a bad word. We brought a lot of our legacy organizations into the cup.

However, going back to the title of our presentation, I was very conscious that DevOps is not a hobby project for IT people. It had to result into something that makes money for us, something that makes revenue, a new avenue for revenue.

So I'm going to bring back Oliver onto the stage again and see, did we actually become successful with this product? Did we actually get some success in the market as a result of what John did?

This is a test for John. This is your scorecard coming out.

Oliver Cantor

Let me give you some marks then.

So, it's very simple. Gartner is the biggest technology analyst out there in our market. I believe if you put all the others together, they still don't add up to as big as Gartner. And every year, they measure us.

We spend a lot of effort in showing Gartner exactly what products we've got, what our vision is, and as well, also how we execute. So it really is the most in-depth measure of our product strategy and product capability. And more importantly, all of our customers, our big global customers, go to Gartner and take their advice all the time about who are the right people to go with for long-term strategy.

And last year, sorry, this year, importantly, we achieved the leader of leaders' classification in the Magic Quadrant. So for us, there can be no better stamp of approval from a product strategy.

This shows that both for the vision, also the breadth and the sheer large, sheer massive scope that we had for creating an entire app store ecosystem of network services as software, so the network becomes software, and for execution, we couldn't have been measured better.

So, I guess it's an A+ to these guys.

Sanjeev Jain

Excellent. Thanks, Oliver. Thanks.

Thank you. And I think, if I'm being honest, for an organization the size of Verizon with a huge legacy stack of applications as well, not each one of our applications are running at the same pace. Some will score probably one out of 10, some will score 10 out of 10.

And John, while we've got a room full of people, I don't want to leave this chance of not asking for help. So we really need some help, and John, if you'd...

John Scott

Sure.

So, one area you've probably heard a lot of: product focus. Anyone who can give us any help in that area, that would be fantastic.

Engineering. Engineering productivity and gamifying. You've seen we've done some pieces in there. We're still interested about the whole metric side of things. What else can we go after?

And then finally, Hygieia. Really join that community. It is a great product, so please do join us.

Sanjeev Jain

And again, thank you. It's been a great pleasure, and thank you very much, guys.

Thank you.