Bringing DevOps Success to Every App, Tool & Role in Your Org
Robert Stroud is a recognized industry thought leader in DevOps and Continuous Deployment. Before XebiaLabs, he was Principal Analyst for Forrester Research, Inc., where he helped large enterprises drive their DevOps transformations and guided them through organizational change. As VP Strategy and Innovation for IT Business Management for CA Technologies, Rob developed the strategy and product portfolio for products within multi-billion dollar markets. He also served as CA’s Vice President, Innovation and Strategy, where he developed and executed strategy for IT Business Management solutions.
Kurt Straube has 30 years of experience in IT in large and small companies, in a variety of technical and managerial roles in both Development and Operations. His current role is as the John Hancock DevOps Tools Pipeline Lead. In this role Kurt leads the effort to identify, select, and enable the tools in the DevOps stack. He also has a passion for developing technical talent and leads the enterprise wide Tech Lead Practice, along with the new university graduate Software Engineering hire program.
Chapters
Full transcript
The complete talk, organized by section.
Robert Stroud
So I want to start by just sharing a few words on what's going on in the industry.
One of the things I see in all the organizations that I talk to is that the metrics of long releases and sustainable releases are being replaced by speed and differentiation. We heard a little bit about that in the opening keynotes today.
Recently I was at a large insurance company, and one of the interesting things that was said there by the CIO to the entire staff that were gathered is, "From now on, we're going to be measured by how quick we can actually deploy business differentiation to the market. So instead of me measuring you on just availability and uptime," which is important, by the way, don't get me wrong, "we're going to be measuring you on the number of deployments and new functionality you can deliver to the business and drive differentiation."
So you're all in this room, and many of you have been on this DevOps journey for some time. One of the fundamental issues with the DevOps journey is that DevOps typically starts, as we all know, in small teams and small groups. Our product teams are formed. We get very good at delivering product. We get very good within our own insulated group. We go pick our own tools. We pick our own processes. We pick our own ways of doing things. And within our isolated team, we get very good, and we are going quickly.
But the reality of it is, and the value of it is: how do we transition that team initiative to scale? How do we transition it to the organization?
So we have what we call the DevOps adoption curve. Like any adoption curve, you get the first iteration. And then after you get to the first iteration, you want to actually then move on from that and scale it across teams. So you end up in what I like to call the trough of disillusionment, where you're going and working out how to scale and how to drive it well.
And then over time, what we hear is that organizations will start to develop constant, continual processes while still giving the development teams and the product teams the flexibility to differentiate and transition and change. And then you get into what I call a steady state, where you can actually deliver and deploy.
DevOps is not just being driven from the bottom up. It's not just about being driven up. One of the interesting things that we've seen is that organizations, and once again we've heard it today, and you're going to hear it in tomorrow's keynotes as well, you're going to hear that DevOps is now, for many organizations, being driven top-down.
And, once again, as I can tell you, we're actually seeing this velocity, this transformation happen, where the business, whether insurance, banking, finance, manufacturing, whatever you are, you realize that technology and software is actually driving your business. So what you have to do is push down.
And one of the things that I've been involved in this industry for a long time, we've been trying to do, is drive quality and consistency and repeatability. And I love it. I love this quote here. When talking to the CEO of a large Fortune 500 organization, they basically said, "We have to do DevOps. It's the only way we can get the quality and consistency in this organization."
This same organization has DevOps as a board-level reporting item for the CIO. Every month, the CIO goes to the board meeting, and we've also talked in the past about the CIO being relevant at the board level, and actually communicates with the board on their key DevOps metrics. This is a transformation like we've not seen before in IT.
And as we transform, we call it DevOps, but it really should be DevSec Audit Compliance Risk Ops, right? I was very fortunate to be involved with a bunch of industry leaders very recently, where we actually identified that audit, risk, security, and compliance need to be part of our going faster and going forward. So we wrote what we call our letter to the auditor. But that was really an invitation for the auditor to acknowledge the fact that in going fast, we need to bring those pieces in.
So fundamentally, the value proposition we have to identify is we need DevOps to be for everyone. And everybody needs to be involved, whether you're an auditor, you're a security officer, you're a risk, you're compliance, you're operations, you're development. We all need to transform and transition into this world.
So with that said, what I'd like to do is have Kurt just share with us how DevOps for everyone really looks. And I'm going to hand that over to you as well. Thank you.
You can have the domineering side.
Kurt Straube
All right. A little about myself and about my company.
I've been in IT for perhaps 30 years, various technical management roles, mostly in financial services, all pretty much in Massachusetts, mostly Boston.
The company I work for is John Hancock, John Hancock Manulife. We offer a wide variety of financial services: insurance, mutual funds, asset and wealth management, private commercial banking, commercial mortgages, real estate worldwide.
John Hancock was founded in 1862 as John Hancock Mutual Life in Boston, acquired by Manulife in 2004, also a very old company, 1887.
John Hancock is named after a famous U.S. founding father and signer of the Declaration of Independence. For those of you who may not be aware, every schoolchild in America grows up learning about the Declaration of Independence and John Hancock's signature on that document. It was three or four times the size of the other signatures, and at that time a very bold and strong move, considering the folks who signed that document would've gotten in a lot of trouble with the British Crown had the American Revolution not succeeded. And he made the boldest of all statements. The gentleman had a lot to lose.
So we're about 34,000 employees, 20-plus million customers. We're acknowledged as one of the best-known American-Canadian brands.
Here is part of the mission statement of my group. I'm in a shared services organization. We provide services across our enterprise. We provide tools and such that folks can use. We don't necessarily mandate, but we like to think that we do such a good job of providing the tools that they sell themselves, and folks adopt them because they see the value and the need immediately.
So our journey and mission: establish an environment where building, testing, and releasing software can be done rapidly, frequently, reliably, while maintaining predictability, efficiency, security, maintainability of all the applications in the enterprise portfolio. And you're hearing a lot about this faster, things like that. But I think the key here is all the applications in the portfolio. And you can imagine a 146-year-old company has some legacy apps floating around, many of which are going back decades to perhaps the time before I even started in my career.
So having said that, here are some of the breakdown. I'm not going to go through all these in great detail, but the first one is the biggest: enable true transformation to modern software development practices across a varied portfolio.
We're about four-plus years on our DevOps journey. We've had great sponsorship by our senior management. We've been doing Agile at Hancock in pockets for over 10 years now. And really in the last two or three years, it's really accelerated with scaled Agile, and it's being adopted across the entire organization at a very rapid pace. I've been on the scene, I've been there 10 years, as I said, and I've been involved in the DevOps day-to-day for two and a half years or so on this team.
Increasing and improving cadence of delivery, you're hearing this from everyone. It's top-down from our CEO. We've got to go faster. We've got to get the products out faster. Nothing new here.
Leveraging new and existing automation from build, test, deploy, more visible and accountable operations, integrated security and code quality, standardized management of regions and provisionings. We'll get into this in a bit more detail. Efficiency with scale, insights, visibility, and all this good stuff I'm sure you're hearing, or you're perhaps living yourself in your organizations.
Here's the pipeline team. This is my team. I'm just going to go briefly through this.
I want to talk about here's what we do in the middle here: identifying emerging needs for tools. We don't necessarily get requirements. We do a lot of research, read a lot of white papers, talk to a lot of people, go to conferences such as this, about what is the next thing? What do we need to be moving on as we build out and perfect the stack?
New tool R&D, new tool enablement. We don't just dump and run, right? We help bring it in, get it started, get users using it, demo it, hundreds of demos, new tool adoption, tool support, maintenance, and then drive new technical and service capabilities. That's the big deal.
Some of the more interesting stuff we're working on, the emerging capabilities: infrastructure as code. We're bringing in Chef, along with our global folks who have the keys to the kingdom and the infrastructure way up at the application level.
Accelerated environment provisioning. We've been working on this for well over a year, and it's still going to go on. We want to be able to give developers, and as you can imagine, we're in a highly, highly regulated industry, right? Insurance, mutual funds, highly regulated. So we hear a lot of, "You can't do that." And we are breaking down those barriers, though.
And what we want to do is give a developer, almost instantly, an environment that they can work in that's got the access, the data, the metadata, everything they need to start developing or fixing a problem within minutes, not within weeks. Okay? So that's what we've been working on for the better part of a year, and that story is coming to a close soon.
Database code checkout and governance, fast database cloning, that's part of that. And then, of course, self-service. Self-service for developers, self-service for just about everyone. And that's where we're going with DevOps for everyone.
We have teams in Boston, the biggest team. We are also offshore. Manila, they work a variety of second shift, third shift.
Here's our stack. Now, as I said, this isn't the only stack, but this is our stack in John Hancock that we offer out. We read it down and then left to right.
At the top, as you can see, is XebiaLabs. This is our latest acquisition in the last year or so. We spent a year studying application release automation and what we could do with it. And during that journey of just studying the tools, talking to people, we came up with the three sort of philosophies you'll see in a moment about our DevOps offering.
There's a few interesting things here to talk about as we go left to right up top. We're using Rally and HP PPM for our planning and demand management, of course user stories and such. We've had Rally in the company for over 10 years now.
Subversion, GitHub, GitLab. Going back four or so years, our architectural review board was not so welcoming, if you will, of open source. So as you can see, a lot of products up here are products you actually pay for. And I don't necessarily have a problem with that because you get support. There's someone I can call in the middle of the night for support if something breaks, as opposed to maybe something else.
So we are now able to bring in more open source, and in fact, we're going to start contributing. We've just gotten the approval to start contributing to the open source communities, which is something we're going to be doing real soon.
So as we move across to the right, our CI tool space: TeamCity, Copado, dbdeploy. These are just three of our build-deploy tools. We have other BUs and other divisions that are using a whole different set of build-deploy tools, and we'll get into that in a minute. It's part of one of the philosophies we have going.
So security code scanning, you've seen a lot of these tools: Fortify, Black Duck. We've been doing Fortify scans on our code for two years, right? And several development groups now have the requirement for a definition of done. They can't have the code be considered done until it's received both Fortify scan and a SonarQube code-quality scan. Right?
In addition to the Fortify scanning, we're implementing Contrast, as you can see down here, which is more of a security monitor. Right? But for now, we're doing the Fortify scans.
Some other couple of interesting things to mention: Parasoft Jtest, this is for automated, system-enhanced unit test generation. Not just automated unit testing, but unit test generation. Pretty cool company out of California called Parasoft. Fortunately, it's just for Java, so it's just the Java side of the house.
We have a couple of little code-quality tools. Androlu, a little company out of India, they do code-quality scanning for Informatica code, if you're familiar with the ETL tool Informatica. We're going to talk about some of the unusual pipelines we have in a minute that involve tools like Informatica, things like that.
We're getting into Chef, as I mentioned. A little company called Windocks that we're using, they do legacy SQL databases, 2008, on Docker containers. So you can run a 2008 database in Azure IaaS that's running any new version of the operating system, lift and shift. Okay? That's pretty cool. That's getting us to our self-service and our fast environment provisioning. These things spin up in seconds.
The typical JFrog Artifactory for our open source. We have very good automated testing with HP ALM and the New Relic for monitoring, not necessarily tied into this yet.
We're doing a lot of integration work with XebiaLabs and with a company called Tasktop here, which are here. We got into their product a year and a half ago or so. We're doing defect traceability between HP ALM, real-time defect traceability between HP ALM, Rally, Subversion, GitLab, and TFS for the Microsoft. So the defects are showing up in those systems.
We also have integrated the workstreams, the XebiaLabs XL Release workstreams with Rally, so you can open a demand management ticket to our release team via an automated process. It goes into Rally, opens up a ticket, and then can update and change the state of a ticket to complete, things like that.
So that's in a nutshell. I could go on for the rest of the talk about this, but we don't have time. I'm sure we have other things to do.
So DevOps for all apps. Very iconic picture from our nation, Statue of Liberty, New York City in the background. If you visit that, you'll see a plaque at the base written by an American poet named Emma Lazarus in the 19th century, and it says, "Bring me your tired, your poor, your huddled masses yearning to breathe free."
So that's obviously very inspiring, and I think in metaphors, too, and I try to visualize things. And so one of the philosophies of our DevOps is if I were to put a plaque in our offices, first of all, I would reference this, and this is Lady Liberty, and I would say, "Bring me your old, unsupported, third-party-based legacy apps and we will modernize them in short order for short money."
We're doing CI/CD with apps based on Microsoft 3.5, which hasn't been supported by Microsoft in, depending on what you read, five to eight years. We're doing DevOps with Pega version 6. We're doing DevOps with Informatica. We're on the verge of a full-blown workflow with Informatica ETL code version 9.2, which is very old, N minus two, automated deploys, build, and delivery into Informatica repositories. And the list goes on.
We're doing Salesforce with the Copado tool. Various other things. We have a list of these capabilities, and all of these are showing value in making the whole things faster, right? Delivering, and the business is seeing this.
The other concept, so we had DevOps for all apps. The other concept I came up with, and the team started before I came on the picture of the all apps doing the older things, right? But really, as I started to research application release automation, this concept hit me: bring your own build-deploy.
Within John Hancock, we have several BUs, and people were bringing in their own build-deploy tools. You heard about it in one of the keynotes. We had folks who were using Octopus Deploy very successfully for five years. We had our own TeamCity. We were being asked why we moved to Jenkins. Canadian division is using Jenkins, right? They brought in Pivotal Cloud Foundry, big investment there that uses Concourse. Another BU wanted to bring in UrbanCode Deploy. It fit their use case perfectly, the IBM shop.
So months of going back and forth: why don't we do this? Why don't we do that? Why don't we standardize? And it occurred to me, why bother? Why mandate something that doesn't work for a use case? These tools are relatively inexpensive. They work relatively well. They don't require a lot of maintenance. And why take something that's been working well for five years and overhaul it without any sort of value add? So that's where it came to me with the whole orchestration thing: bring your own build-deploy.
So my attempt at art based in engineering here getting together is tools integration plus orchestration equals product delivery value stream, and obviously that's value add, right?
So we now have pipelines built using XL Release, XebiaLabs XL Release, that are orchestrating these. We've got plugins working. We can orchestrate Copado with TeamCity, with not UrbanCode Deploy yet, but they're next, but with Octopus Deploy. We have the plugins working, and they're all showing up in workstreams in real time as we run the releases.
And then the next thing, though, that hit me was that, and this is where DevOps for everyone came from, is that successful product delivery with DevOps is many different engaged stakeholders, from highly technical to business-oriented. So I'm talking about the actual workflows in real time, the things that are happening, not necessarily just Kanban boards with project management stuff.
And so we brought it in and started demoing, and within 15 minutes or so, people are getting it, even executives. We have one highly technical executive, and within 15 minutes, he was asking such engaged questions and was so thrilled that he could see all his work, all his pipelines, and calendar them out, and things like that. Tech leads and folks like that get it within less than 20 minutes or so and start asking really engaged questions.
So, if you can log into a system, you can see the pipelines in real time. The devs, the QAs obviously, the release management folks, the ops, the business side, security, compliance, and management. We have all these folks logging in and looking at their things. Of course, they've got to be given the access to. But we have it on Active Directory, and it's really nothing to set up.
So I wanted to show a case study here we did. On the left is a DR test. Before we went live with our XebiaLabs XL Release implementation, we had to do a DR test, and that's kind of a requirement. And the DR folks are assigned, and they emailed us this spreadsheet, and this spreadsheet looks just like our release grids. And I said, "Well, why don't we load this up in XL Release?"
So we loaded it into the test system. In an hour and a half, we took that DR test, loaded it into our test system, and ran the DR test on a new production system from test. And the DR team logged in and was watching it in real time with a five-minute introduction to the system.
So they liked it so much, they now are really, I won't say mandating, but they're now advising every team that they do a DR test with to load up the DR test in here, so you can watch all the triggering, the workflow, and everything in real time. This saved a lot of phone calls, emails. You spend hours and hours on these tests. Did we finish line 52 yet? Did 47 complete? Done, gone.
So that's one real simple use case, hour and a half. We have other things going on as well. Really proud of some of the integration work we're doing, and maybe next time we'll demo that.
So that's pretty much it.
Robert Stroud
So, Kurt, one of the things that... Kurt and I had the value of a DevOps.com podcast this morning, so maybe I'll toss there. Tell us about your cloud and container initiatives and where you're going there, and how you're going to fit that into the DevOps for everyone plan.
Kurt Straube
We have a strong commitment with Microsoft Azure. We have a hybrid cloud, and one thing is to use Chef for quick environment provisioning. But the other part of it is the database stuff and containers.
We have groups doing development containers and such, but I'm focusing on third-party tools in containers. For example, I mentioned the SQL database stuff in Docker containers.
We're working with a company out of New York called Robin Systems that just re... And this is cutting edge. They released 1.0 of their product in March: DB2 databases on containers. So spinning up DB2 databases with everything a developer needs for DB2.
So consider the implications of if you had to move your databases from a mainframe to some cloud and lift and drag legacy DB2 databases and run them in Azure. It could be pretty interesting. And that's really focused on that self-serve environment provisioning fast.
Robert Stroud
So, the other interesting aspect of bringing the complete lifecycle in, so you actually bring your own pipeline. How have you found that in terms of DevOps acceleration and adoption, which is a big topic here?
Kurt Straube
Well, that speaks a lot to the human side. As I said, there's little value in getting somebody to move off something if there's no motivation. Integrate. We'll take it in, we'll integrate it, get it to work that way.
Robert Stroud
And how are you measuring success? This is the magic metrics question. Right. So at the end of the day, I want to know how successful and what the business impact is.
Kurt Straube
Well, certainly, with build-deploys, that's pretty linear to show success. For example, we implemented Copado for Salesforce. We were taking six to eight weeks to plan a Salesforce deploy, an entire weekend to do it. A tool like a Copado build-deploy reduced that to 10 or 15 minutes of planning and less than an hour to deploy it.
Take that up a notch. We're looking at releases that can involve 90-plus person-hours to plan a release that happens every six weeks. This is outrageous. And six hours or more to perform the release. Bringing that down, putting it into these pipelines that everyone can see, scheduling it out, the value there is pretty obvious.
One last piece of integration, let me just add, is what we're really working on, completing the pipeline, all that, but the integration is kind of interesting. We're doing things like tying... XL Release has a great calendaring system, and we're tying that into our configuration management database, so that if you schedule a release out, two or three weeks out, it will then look into the CMDB to see: is there any work happening on that release this weekend? Are your systems going to be brought down to patch in the middle of your release? And giving that information back, so sort of a proactive look at that. So that's going to be this year as well.
Robert Stroud
So you're really integrating the dev and the ops part there...
Kurt Straube
Sure.
Robert Stroud
...so you can show some visibility back in the ops side.
So I'm interested, being an ex-security person, which I am, how have you found the integration of the critical aspects of, and I know you touched on it, but the critical aspects of ensuring that a release is not transitioning or moving on to production unless it's protecting the John Hancock assets appropriately?
Kurt Straube
Right. Well, the mandate that all code that goes to production has to be scanned in one way or another. And we have a lot of teams that now require scanning as their definition of done in Rally, right? Cannot move with that. How else? Contrast security scanning, that's more of a monitoring thing.
Robert Stroud
Yeah. And the next question is what's next? You've got to this point, success is there, the world's going faster. What's next?
Kurt Straube
Well, we still have some work to do. Finish the pipeline with Chef, really get that environment provisioning fast. Today, you fill out forms to get a VM, forms, days to get databases. They work as hard as they can manually. I want that quick. I want a VM in a few minutes. I want a database loaded with everything a developer needs in a few minutes more, or bring that down to even a couple of minutes.
And then self-service, a couple of clicks. A developer, whoever needs it, couple of clicks, and they get all the stuff presented and usable within minutes.
Robert Stroud
And as part of the culture transition, what would you describe as the issue that caused you the most angst, if there was one?
Kurt Straube
Oh, well, there are several. Obviously, highly regulated industry. You get into processes that have existed for perhaps decades. You challenge them, and you hear back reasons why they have to be in place.
A lot of people have bought in from the top down, as I mentioned, but it's then kind of re-engineering these processes. The XL Release is less of a tech tool and more of a process helper, and that's a big part of the challenge. I wouldn't say it's the mindset or the attitude, but it's now, let's rethink these things within our regulated industry.
Robert Stroud
Ah. So you brought up the word regulatory, but I'll be careful here. So, in terms of the regulatory environment, one of the things we see in the industry is we're seeing a movement to deploy straight to production.
Kurt Straube
Right.
Robert Stroud
How are you handling that issue, and then how do you plan to handle it?
Kurt Straube
We have the capabilities. We absolutely have the capabilities. We're still working on the process to do that. If you were to tell me you're good to go, I could put something in within a week, and all of these things could go straight to production, frankly, from the capabilities. But process, we have gatekeepers on a lot of stuff.
And in some of the unusual things like Salesforce, you can't deploy directly there. That's got to be gatekeeper. Or Adobe Experience Manager, you can only deploy automatically to stage, and then they take it from there, or mobile, of course, things like that. So some things I don't think we can fix, those being three examples. But we're working on the process.
The auditing has come a long way in their understanding of this. We meet quite regularly with auditing, and they are coming around.
The first meeting I had kind of blew me away when I was talking about... And this subject is obviously very exciting to me, and I get animated, and I made this great pitch about how this is the best thing you've ever seen, auditor folks, and blah, blah, blah, and there's logging, there's archiving, there's everything you've wanted. And then they said, "But still, no one's checking it off before it goes into production." And I said, "Okay, we've got some work to do here." That is thawing, if you will. We're working on that, right?
Robert Stroud
Yeah. It's an interesting problem statement, which I left my Dear Auditor note up earlier, and it's one of the challenges we have at the moment in the industry. And I know in Topo and the others in the team, and I worked on the Dear Auditor letter, one of the things we started with was an admission: "Hey, we've been going really fast, and we've probably not kept you in audit, risk, security, and compliance totally informed of what we're doing."
And start with that statement, then a commitment, which is what we do in DevOps, which is, from now on, we're going to change that. We're going to involve you, we're going to insert you, we're going to bring you into the process.
Because one of the things we're talking about now as we take DevOps to the next level, and we go up the trough of disillusionment into business as usual, which is where we're going, is you really need to bring all those impacted groups in to actually understand what's going on.
So I think it's a real opportunity for us as an industry, by the way. One of the things, if you see that Dear Auditor letter, that is an open source opportunity for you and your teams to contribute, and I encourage you to put pull requests up there for us.
But the reality of this industry, and this is one of the key things, is we're going to see multiple environments with lots of tools that we're all going to use, open source, non-open source, tools changing. So what we need to do is really look at how we're going to bring that all together as an industry.
And as we finish up this session, one of the points I'll make is both Kurt and I will be available for Q&A, both now, and if you see us in the hallway, drop by. And if you've got another question, please do drop by the XebiaLabs booth, ask a question, get a free bus, which we have.
Alternatively, we'd like to see you have a look at our new DevOps pipeline generator tool, where you can actually merge all those tools together and actually print out your own pipeline to look at what your tools for your particular pipeline looks like.
So with that said, I want to thank you all for joining us and send you on your way to your next session. Thanks very much.