The Future of DevOps for the Enterprise Trends and Insights
DevOps is no longer fringe, and most enterprises today are either adopting DevOps or well into their DevOps journey. As DevOps practices and tools continue to evolve, enterprises have more opportunity than ever to gain competitive advantages.
In this talk you’ll learn what are the latest DevOps trends, where DevOps is headed, and understand key emerging practices and technologies that are driving success, including:
- How enterprises are overcoming challenges like legacy systems, regulatory demands, and security.
- Emerging technologies and tools like containers, microservices, pipeline-as-code.
- How businesses are leveraging AI/ML for predictive feedback and smarter Continuous Delivery.
Wesley Pullen is the Chief DevOps Strategist at Electric Cloud. In this capacity, he leads the strategy for DevOps, Release Automation and Continuous Delivery (CD) solutions for Electric Cloud. Wesley brings over 20 years of industry experience in enterprise products in Application Release Automation (ARA), Application Lifecycle Management (ALM), IT Service Management (ITSM) and Application Performance Monitoring (APM). Prior to joining Electric Cloud, he held leadership positions with CollabNet, Automic (now CA), and BMC.
Wesley holds a BS degree in Electrical Engineering from University of Pittsburgh.
Chapters
Full transcript
The complete talk, organized by section.
Wesley Pullen
I'm Wesley Pullen. I'm the chief strategy officer for Electric Cloud. That's all I got to say, and good day. Good night. No, I'm just kidding.
We're going to get started. Just like the song says, we're going to take care of business today in looking at the future of DevOps for the enterprise, looking at some trends, things that emerged in 2018 that we can expect to also see in 2019. I am going to be on my absolute best behavior today because my wife is in the audience, and this is her first time hearing me speak. So all my jokes, please laugh. Make sure that, too, so when she takes pictures, I look great, and then she can go back and tell the kids I did a phenomenal job. So that's what I need from you. I need your support. So with that, let's begin.
I don't spend a lot of time talking about marketing slides, but I do get passionate about technology. If I talk fast, feel free to slow me down. My wife is the only one allowed to throw things, but you can just raise your hand, and I'll slow down a little bit. But just in case I talk too fast, just let me know, okay? So we only have 30 minutes, so it's not a lot of time, but it's a good precursor to what we're going to talk about.
So Electric Cloud, you heard probably. How many people are just hearing about Electric Cloud for the first time? First time ever hearing about Electric Cloud? Okay, not bad. So the best way I would describe it, if I had one slide or one kind of statement, is like, we're the software behind the software. How many people have ever heard of 1-800 numbers and 1-877 numbers, toll-free numbers? Well, we're the software helping Somos behind that. How many people have children age five to 18? Okay, how many people have heard of their children playing a game called Fortnite? My wife doesn't count, okay? We're the software behind... Fortnite is built from Epic Games, the Unreal Engine. That's part of our software helping Epic Games to build the Unreal Engine, which is the software behind Fortnite. So we're the software behind the software is a good way to think about Electric Cloud.
All right. Another thing about Electric Cloud, I'll end with this, is our focus. I think that in software, you have to draw a line, and the late Steve Jobs said it best when he said what people think about focus. And what we kind of envision it is, is that we have to say no to a thousand things to be the best at one thing. And we finally achieved that. We got an award at the DevOps Industry Awards in London just about a week or so ago, and we did quite well in the Gartner Magic Quadrant for Application Release Orchestration. Our focus is to help you deliver software better, faster, and cheaper than your means today. We want to be like the UPS of software or the FedEx of software, if you will. That being the case, let's get started. Future trends. Willing to take pictures and everything.
How many people have heard of Marc Andreessen? Okay, Marc Andreessen. He made a quintessential statement during an interview with a Wall Street Journal individual where he was talking about why is he doubling down in his investments. The article, and what he was really trying to say is that Marc Andreessen was investing his own money, not just the company, but his own money into software technology companies when everyone else was betting into the more hardware, the ExxonMobils and individuals like that, Walmart. And so they said, "Don't you remember just 10 years ago, Marc? This was a technology bubble. All these companies kind of disappeared." And he said, "Listen, software's eating the world, and you need to be doubling down." Let's see if he was right.
Two years later, after he made that prediction, look at the red versus the blue. The technology companies with a market capitalization in the top five versus the blue were kind of your standard run-of-the-mill Exxons, Berkshire Hathaway, PetroChina, Walmart, were very familiar. 2018, now look at the picture of Marc Andreessen smiling. I got that real special picture of him smiling because he was actually right. All of these tech companies now are the largest. These are the top five companies by market cap in 2018.
How many people have heard of Tencent? Heard of Tencent? All the subsidiaries they own. They also made a minority stake investment in Epic Games, Fortnite, these popular games. So it's amazing what technology and digital transformation is doing. We're seeing technology companies now being the largest, the biggest companies publicly traded, by market cap, if you will. And one that may be happening, something that we see, we don't know, Uber right now is being valued at a possible IPO, $120 billion by two different banks, both Morgan Stanley and, what, Goldman Sachs. So that's large. As a tech company, that's major. How are they doing this?
What is it that Marc Andreessen was really trying to say? Companies are now trying to become more digital. With the consumerization of IT, companies are now looking at ways of adopting practices. This is why you're at a DevOps conference. They're adopting it more. We did a study, or there was a study that was done, a friend, Yariv, he's the CEO of DBmaestro, and he started to do some study. He did some trends in February 2018, and one of those trends was looking at how are companies beginning to adopt DevOps. And he said in the trend, it came up to 83% of the companies were adopting DevOps at a team level. 30% were beginning to adopt it company-wide, not just one or two teams, but company-wide adoption of DevOps. That's significant. The quote I took from his article, what Fawaz was saying was, is that he's starting to see C-level executives increasingly ask for strategic initiatives to transform their delivery pipelines to support digital strategy. Everyone now is no longer saying, "Hey, I don't want one or two teams implementing DevOps." We got to get to that point where the company, the culture company-wide, is adopting DevOps to get digital transformation and realization.
So that being said, three key trends we'll hit in the small amount of time I have with you. Cloud and container adoption that you see here, we'll talk about that just briefly. I'll show you some pictures of what it looks like. I'm kind of a practical guy, not just marketing, so I'll show you kind of what it looks like and how we've helped some companies there. Then we'll go to the rise of DevSecOps. I keep running and saying, "Security's coming. Security is coming." It's no longer just good enough to have quality, but we've got to be able to inject security and start shifting left, and security is more important. For instance, if I have a library that I'm downloading, I did great in my quality initiatives, my pipeline's great, but it's built on a framework that's vulnerable and has a malicious code in it so that it can be hacked and exploited. And I went all the way through my pipeline, it's in production, I said, "Yes, we passed all of our tests," and someone hacks it and steals personally identifiable information, in other words, your credit card out of your bank, sir. I don't think you would think that's a successful pipeline anymore. So security is going to be critical. We'll talk about DevSecOps in the conference that we have running after this one, or the workshop. And then finally, AI and machine learning. How can we apply artificial intelligence and machine learning, deep pattern recognition, to DevOps in the form of feedback? So we'll cover all this hopefully in the 20 minutes we got left.
So let's go. First trend, trend number one: migration to the cloud. How many people here in the room, your companies have some type of cloud migration strategy, looking at cloud migration, maybe some private stuff? Okay, that's not bad. Maybe 20%, somewhere around there. What we're seeing is that companies, enterprises, are taking kind of a hybrid approach. You can't just abandon your legacy investment. And we see companies saying, "Hey, look, we have some traditional applications over here. We do have some greenfield aspects or some microservices that we'd like to begin with." And ultimately speaking, we have hybrid applications that kind of, hey, we'll take the database tier and have that in the container, but keep the legacy stuff where it's at in blue, which is the legacy application, just traditional monolithic application. It runs through a pipeline, and ultimately it starts to distribute itself to some cloud or container-based platform in the end.
The key here is when you see some of these stats, you see multi-cloud strategy. We're seeing enterprises not necessarily say, "I'm only going to do Azure," or, "I'm only going to do Google Cloud Engine," or, "I'm only going to do AWS." Maybe in dev, we start playing around in AWS because we can spin something up very quickly. But the corporate strategy may be Pivotal, it may be something else. So we need the ability to run fast, and we don't want to hold back the teams as they begin to experiment. But at some point it has to converge and get a little bit more consistent. So you see adopting containers, 58%. This is a study that was done through several studies, 451 Research, RightScale, and you see traditional on-prem. All of these numbers are starting to bolster up, that companies are starting to say, "We need some aspect of not only microservices and containers where things are loosely coupled, loosely put together, but I want to start migrating them out to the cloud and start getting some adoption going with smaller teams."
And then ultimately speaking, and I didn't put this here, I could've added another block where I say mainframe, because I believe mainframe is never going away. Just period, I don't think it's ever going to go away, particularly in financial services. So that's what this can represent, too, traditional and legacy apps. Mainframe adoption is still there. These are large investments, but you want them a part of the pipeline. Whether I have mainframe, Kubernetes or microservices, legacy, .NET, PHP, Java, I need a discipline with which I will get it over to its designated area.
Now, I said this a little bit earlier, greenfield's different. We stole the term, I guess you could say, in the tech world from construction, right? Where I have no encumbrances, I can go build upon a piece of land, and I don't have to worry about the building problems or infrastructure problems. Everything's open for me. In the greenfield technology world, it's I have one team, I take this team on the front row. We got one team, one app, we're releasing into one environment. That's easy. The dots are real easy. But everyone in the room is looking at me like, "Wes, come on. This is our world. We don't have one dot, one team. We got multiple teams. They're geographically distributed." It's easy to get the first Kubernetes instance or some container instance up, but it's all the other dots, all the other pieces, all the other dependencies, you can say. I have to get all that working together. And if that doesn't work together, then the fact that I did one little instance for the dev team is not going to bode well for my CEO or the executive team saying, "Hey, look, we need all of this tied together. We need this to be in one release cadence or one release stream." That's like having one product that relies on other frameworks. You release one piece of your product, very great, very secure, everything's working great, and the other pieces don't work. They're not going to consider it a good job. They want it all together, okay? So managing that complexity is hard.
So what they're doing, what companies, enterprises we work with are doing, just to kind of shed some light, is building a single pipeline platform to integrate all the moving pieces. I shared this in an interview this morning: it's okay to have lots of technologies. I guarantee if we asked everybody to raise their own hand, how many people have two, three, four, five, 10, 15 DevOps tools, the tool shed is going up. But how many email tools do you have? How many payroll systems pay you as an employee? You don't have ADP on one and Paycom on another, and another one for this. You don't split that the development team will be paid on this system, and the engineering team will be paid on this system, and the QA team will be paid here, and executives will be paid here. It is one payment system. We need to start centralizing on one single pipeline platform, and then let the teams use whatever tool they want. So certain things have to start converging over time.
We also should see some support for agnostic app models. That if I have an application, it could be PHP, it could be Java, it could be .NET, that's okay. I need to be able to model it out in a way, shove it through that pipeline, and get it to its necessary location. I shouldn't have to have different technologies just because I chose that I have a legacy, that we do acquisitions, companies acquire technologies, some had .NET, others had Java. I shouldn't have to get a whole new tool stack in order to get them through the pipeline. I ultimately want to get them new capabilities, new features into production. I should have a single system to help me make that transition.
And then enabling pipeline dependencies. You see some graphs here of just being able to model this out. And just to show you that it's real, I like to... I'm a techie at heart, I guess you can say. I think that you have to have something that will ingest. Since they only give us something like 20 minutes, I can never really do demos, but since I helped build some of this, it's personal to me, it's passionate, right? So you should be able to have things like a self-service catalog where I can just click a button and say, "Make all this happen for me." We shouldn't have to always model it out, script it out, and do all the work. There has to be capabilities like self-service where my team, as an IT team, says, "Here's our best practices, give this capability out," and now when I want to build my Kubernetes pipeline, I want to do stuff, I just click the button and say, "Go get it for me. Go grab all the data for me and build my pipeline, build my model for me so my teams can operate a little faster."
And so what I did for you, since I knew I only had so much time to cover all this, is I said, "Hey, look, this is an example of that pipeline," starting off with a catalog. You can see me saying, "Hey, I'm going to click here," and it's going to build out all the data for me. And then I did this one to say, okay, once this guy is done, I want to click one more button, and I want my multi-stage CD pipeline to be built for me. I don't have to go do all the plumbing. We have to get to a point where templates and things, reusability is available for me. This is what it looks like when it works, where I can say, "Hey, look, here's my Kubernetes instance that I just built out. It has all this capability for me." I can then see, and so that you know that it's real, this is not me playing with Flash and doing all this stuff, or Wesley could have made it up. I can then say, I auto-discovered a Kubernetes instance. I can click on it to say, "Well, Wesley, do you know that this is real? Is this a real Kubernetes instance?" Yes, that's a live environment. I am actually auto-discovering it. I'm looking at all the dependencies.
We've got to get to a point where we make life easier for your teams that are working to get delivery of software out into production or to the end user. And so all I did here when I was building out these graphs is I took screenshots to show you this is me ingesting and grabbing all that Kubernetes stuff. Then this is me building out the pipeline for that next button, and when I run that pipeline, I have all of the things, all those little dependencies. You could see why there are so many here. This is just some of the list that I have. Then this is me running the pipeline.
And this is the most important picture for you guys, of taking pictures to take, is that it means nothing for me to do COBIT 5 compliance and do all this work here in my pipeline if I don't build evidence along the way. Integration is no longer the high bar. Everybody can integrate with technology. It is what do I get to feed data into all these other stages that I got from integrating with your technology that makes the difference. So again, integration is the... Everyone can integrate. We can write Perl in this front row, and we can integrate with any technology through REST API. It's not enough to integrate. You got to be able to grab the evidence and feed the data in through a stream so that you make sure that you can get the job done. Okay?
All right. That being said, trend number two. We got about 15 more minutes, so I'll speed up a little bit. DevSecOps. I cannot emphasize enough the need for security. Again, I'm going to show you some statistics. And I got this, I got to give credit to John Willis, because when I sat in on, we did the first conference, a DevSecOps workshop in London just in June. He put up some staggering numbers, and me as a developer, I'm like, "Oh, that can't be true." But this is what he was doing. He put up some numbers that showed, and all these are hyperlinks. If you want the slides, we can provide all that. These are hyperlinks to the report that we got from Sonatype here, that 1 million downloads of vulnerable libraries since the 2017 Equifax breach.
Let me give you a common example. That's like me and my wife asking one of the teenagers, the college-bound one, that we just got a car, get all this stuff, he's off to college. We say, "Please, clean your room before you go out to your party, whatever. Just make sure your room is clean." Cleans the room. Three days-- No, I'm sorry. Twenty-four hours later, it's like it's completely destroyed again. It's like, you knew to clean the room. Why are you still doing the same things and we have to go back over through it? This is exactly what's happening here. We knew that the patches and the frameworks that we were using caused this breach in Equifax. The fact that people are still downloading it and using it is just baffling to me. What that means is there's no security. Why would we do that when we know it can be hacked? You're literally inviting... That's like buying brand-new furniture and everything, and you leave your windows open, the door open, no alarm system, and say, "Hey, I'm going on vacation for three months. Have fun." Why would you do that? I don't understand why we do that. But we do it. And it's just habit. But this is a problem.
So we're seeing security having to come in. In the world of lack of security, Marc Andreessen's statement that software's eating the world becomes software's infecting the world if you don't secure your systems. It's like you're infecting it because now every line of code, every vulnerable library, you just literally gave an example to everyone. We have some videos that we shared on DevSecOps of a video that John Willis shared that shows a hacker literally showing you, I'm using Nmap here to find this port. I then move over to Skipfish to do this. I then use Metasploit, bam. Okay, now I'm into their private network. Oh, they are using this SSH technology. Great, I can use that as a tunnel. He opens up a bridge. Voila, credit card stolen. Beautiful. It's videotaped. I mean, it's on YouTube. They have YouTube channels showing people how to do this.
So we've got to get better. We're the technologists. We're at a conference that helps us use and get great ideas. We got to do better about securing the libraries and giving security personnel, our CSOs, a little bit more advantage of knowing what we're doing in our DevOps pipeline. So I can't emphasize enough the need for security.
This is what we run. It's going to happen on Thursday. This is just one example of the DevSecOps conference. We actually show you how to take advantage of a particular system in our partnership with Sonatype. We run the Sonatype technology to see how many policy violations, security violations, and others. And so what I took the liberty to do is, for those who can't make it, to show you an example of what it looks like. You take a pipeline. We're using Electric Cloud's technology, of course. We tie in technologies, and from the time a delivery team makes a change, I can be in GitHub, I make a commit, and it kicks off a series of technologies. And then ultimately speaking, as we go through the various stages, you can go from dev to QA to stage, but we have an SLA or a policy that is implemented before you get into prod. And so that way, if you're not secure, we're going to bounce you. You're not going to get a chance to go all the way to prod. And then we show you how you can remediate that.
This is what it would look like in real life. You take the pipeline. You can see here I highlighted kind of just the beginning, the policy violations. These are policy alerts, eight critical, three severe, zero moderate. You put the hyperlink in so you can go straight to the Sonatype report. You click the button. This is what it looks like. There's the eight and the three. And I could have displayed all of these. I just displayed in the initial instance, since I only had so much time before we got up here, so I showed just some of the policy alerts. And this is what it would look like when you get a little bit further. That there is a policy violation. So you did well. You got all the way through here, but we stopped you because you didn't clean up those policy violations and license infringements, and we don't want that going out into a production state. Right here, these are the uncontrolled environments. This is the controlled environment. We don't want you going into production with something like that, so we stop it, and then you can do auto remediation.
Now, some people don't like to look at it this way, so I took the liberty to show it this way too. So you can look at the pipeline. We are a big proponent of sometimes DevOps your way is kind of the best way to do it. Some people prefer the Kanban view of things because we've used Trello and we manage tasks. I think of a pipeline as more than a task. I think about it as the integrations or the seams in between. So I prefer the other view, but sometimes people like it this view. You can use either way within our technology.
So we're on our last trend. We've got a few more minutes, so this is great. We're on the last trend, and this is the one that I would say I've been the most forward-thinking in trying to evaluate how we can help companies in this area. And this is kind of the emergence or what I call the DevOps operating system, predictive analytics. It's how do we apply artificial intelligence and machine learning, not as just buzz terms, but where would it actually apply in a real-life pipeline? Why would we do this? And this is good because at Electric Cloud, I get a chance to say this. We kind of applied it to ourselves. I changed the names because I'm going to show you some live data. I did change their names. So if there's any developers from Electric Cloud in the room, I changed the names. Just letting you know, so we won't know who you are.
But this report, I did an interview this morning with Torsten. He's awesome. And he's been studying as an EMA research analyst, manager of research. He's been studying a lot of these trends, and one of them is this rise of companies making investment in 2018 in artificial intelligence and machine learning. And you can see right here that artificial intelligence was high. But look at the container management and DevOps adoption, DevOps pipeline adoption. Now, I didn't know he was writing this report. It's just we're converging on things. He's an analyst. We are a solution vendor. We provide solutions, and we go in, and we see some of the things that are kind of converging on the same light.
So, for those who are looking at applying machine learning and artificial intelligence to your platform, you have to draw a stake in the ground of where you're going to begin because this is what we're essentially doing. We're taking technologies that you're using. As a solution vendor, I'm going to tell you how we do it. We take technologies that we're using. We're not going to ask the people in the room, "How is things going?" We're going to go to the tools and study through log analysis and through the actual use of the technology by applying deep pattern recognition and statistical analytics. We then understand from the statistics that there's a grouping. There's a pattern emerging from those analytics. We then can predict some risk patterns for you, and then from there, make some recommendations. And if I apply it in real life, it might look something like this.
It's like you want to be able to move from a reactive mode to a more predictive type mode. I want to be able to see if I were to make a couple of changes, if my developers did this, or if my QA engineers did this, could I get this release out a month early? Could I be better? Could I get it out faster? Could I reduce the cycle time in my pipeline? Because at the end of the day, a pipeline is just the time I begin with an idea to the time I have it in production. Let's say it takes everyone in the room, we're great, we get it done in three months. What if some executive is like, "My competition's doing it in two?" How do we get it done in two? Do you have the analytics to say, "Okay, here's what we would need to do to get it out in two months"? So what we're trying to do is take the patterns and factors like developer code base, builds, and things like that, provide a risk score, so you get some provenance about root cause and specific contributing factors.
We applied it to ourselves. This is Electric Cloud on Electric Cloud using our product DevOps Foresight a year in. This is why we made the announcement in June, for those who weren't aware, is that we took some developers. I changed the names, or at least they told me that I changed the right names. So we look at what they're doing. Oh, went by that a little too fast. Code smell statements. That's from technologies like SonarQube that we all know. Functional complexity per function, lines of code per comment. We're looking at things that they're doing, code they're writing, commits that they're making, and saying, "I'm going to try to identify a pattern." That's the first thing. I'm going to study the developer persona.
Now, right now, with Electric Cloud, we focused in on the developer. But you can see where this is going. It'll go to developer, then QA, release and build, then operations, then everyone else. It's just an algorithm. It's applied to the developer right now because they do a lot of writing of code, they do a lot of commits, they have comments, they have things that we can study. Once we get that study, it contributes to a risk analysis for your release. I have high-risk releases and some low-risk releases, and when I drill down into one of those releases, I get the ability to say from each contributing factor, where am I problematic? The objective is I can take in the pattern. Now, just so you know, this is like big data at the next iteration. I know that people saw this. This is not data you can get in two months or three months. This is a year's worth of data of looking at developers within Electric Cloud and then studying that pattern so that it could understand, here's the leading contributing factors to what you need to do to improve for this particular release. So, this takes time. Anything with machine learning and artificial intelligence, you're not going to do in a month or two. You're going to apply some data over the course of a year, of collecting data or collecting logs. So, just to make you aware.
So, as we have to wrap up, there's a lot of stuff. We covered a lot of things from container adoption to DevSecOps and my concern about vulnerable patches and vulnerable frameworks that we're using, down and ending with machine learning. Here's some areas I'd like to highlight. I don't think we need to know everything. I just want you to highlight the key takeaways I want everyone to have.
Number one, DevOps is no longer a fringe movement. We got enough state-of-the-DevOps reports for everybody in this room to go back to your senior execs and say, "Look, this is real. We can really get competitive advantage. This is not a fringe movement. This is serious. This is a major competitive advantage for companies who start to adopt this not only at the team level, but corporate-wide." Number two, the multimodal approach, that cloud and container adoption, you're going to have people that do it from the greenfield aspect with, "Hey, I got a small team," to, "Hey, in order for me to tie this in truly, I got to get my legacy applications, my in-between or hybrid applications, as well as some of the greenfield things, too, and I want to tie them all to one pipeline."
DevSecOps is not an acronym. It is real. It's something serious that we need to take seriously. Starting with myself as well, we got to stop using vulnerable frameworks and patches and things that we know that we need to get a little bit better on, a little bit more discipline. We got to start adding security into our pipeline, not just quality. Quality is one thing, security is another. And we need to start shifting left and allowing the security teams to have some stake in checking and validating the code that we are about to push into production.
And then finally, getting feedback from artificial intelligence and machine learning. If you're not doing it with Electric Cloud, we'd love to help you. If not, we have some sessions that I'm going to show you in a second, but please do focus in on some of the platforms where you can gather that data. For those of you, if you just had to know, since I'm getting on the yellow time, there are some other sessions where we're going to drill down on these. Torsten's going to talk about the seven steps to move a DevOps team to machine learning and the AI world. Eric is going to talk about some common processes and communication of AGCO, they're one of our customers, and how they grew. And then the final one, the key metrics that matter, our CTO, Anders Wallgren, is going to talk about how to measure DevOps and looking at some of those measurement and feedback loops.
So again, thank you so much for your time. I appreciate no one threw anything. My wife gets to give me that great score now back to the kids. Appreciate your time. Enjoy the conference. Thank you so much.