Anthony Robinson

My name's Anthony Robinson. Until recently, I was the Billing Offering Manager at Allstate Insurance Company in Chicago. Just recently, I've transitioned into being the Mainframe Modernization Product Owner, again at Allstate.

I'm very glad to be presenting today. Thank you all for sticking around, particularly on the last day of the conference, and coming along to hear about mainframe.

First of all, a bit of introduction. I've been with Allstate 15 years. Before that, I worked with Lotus in Cambridge, Mass. I manage mainframe at Allstate. Allstate is a large U.S.-based insurance company. We sell out of Northbrook. We have operations around the globe in India, Northern Ireland, Chicago, Charlotte, and now Irving, down in Texas as well. We're the largest publicly traded insurance company in the U.S. We sell products across Canada and the United States.

We're very much, at Allstate, a large IBM mainframe shop, with a long, distinguished history running on the mainframe platform. Up until recently, we were very much traditional, using green screen, just industry standard. But two or three years ago, SLM, which was our source code management product, was coming to end of life, and we needed to get off it and get onto something newer, something better. Along the way, we discovered DevOps. Part of my message today is to show how you can do DevOps on mainframe.

Rosalind is in the audience today. Thanks for coming along. She spoke yesterday about infrastructure as code on Z. While you said that mainframe is different when it needs to be, we don't see mainframe as any different. Z is the same as everything else.

We're like many traditional Z shops. We're challenged in terms of our portfolio of developers, in terms of their profile. As they reach the end of their career, we're trying to bridge that skills gap: having college hires come in, having them working on Z. It's still a hard sell. And as we have our more experienced colleagues, we're trying to keep them around long enough that they can help share that domain knowledge with our new hires.

We see the mainframe moving more open source, moving towards DevOps, more common tooling. And Rosalind's point yesterday: it's just the same as any other platform.

I enjoy coming to events like this because, in our mainframe journey with DevOps, we sometimes just don't have the vocabulary to describe it. We come to an event like this, and we suddenly know how to describe it.

We were agile before we were doing agile. When the company looked around, we moved off our SLM source code system in an agile way before we knew what agile was. Along the way, we discovered DevOps, and our distributed and Cloud Foundry colleagues were well ahead of us, and we wanted to catch up and equalize that gap. We're just first amongst equals.

Many of our mainframe engineers are the most distinguished engineers at the company, so we want them to stick around and actually have new tooling. We still see the mainframe as a platform to develop new services and products at a speed, at a scale that maybe other platforms can't cope with. Far from it being a legacy platform, we see it as cutting edge. We see it as a place to build out services and products, and it's actually a revenue generator rather than consuming revenue.

Our journey from a very traditional standing start was that we have our own homegrown deployment system. We used SLM. We used Endevor, very much industry standard. We knew what we wanted to do with Team Concert and to move towards a more agile way of working. We just didn't know how to do it. Part of our journey is that we're still doing it.

We come to events like this to show you what we've done, to encourage other people to at least look at starting a DevOps journey on mainframe. It is possible. Is it hard work? Yes. Is it worthwhile? Definitely.

Along the way, we've pivoted in terms of what's valuable to the company. A lot of what I've seen this week has changed our journey. Instead of finishing off our DevOps transformation, we've taken a rather circuitous approach, but it's actually added a lot of value along the way.

In terms of DevOps, when we started our migration off SLM, we didn't really know what DevOps was. We read about it, and then we very quickly realized that most of the industry was already doing it, very few on Z. Really, the more interesting challenge was day two DevOps, and then we discovered that that phrase had been coined as well.

What we're going to try to show you today is some of the work that we've done in mainframe, where we've vaulted straight from not doing any DevOps to going straight to trying to address the DevOps day two problem, and some interesting things along the way.

A lot of this is well out there in terms of where DevOps came from. We were aware of all of those methodologies. Culturally, our mainframe shop, our developers were very much waterfall. We're looking to pivot to become more agile, to move towards a product mindset as we move towards a more digital transformation effort at Allstate. It's still a hard sell for some of our mainframers, but our journey to DevOps has certainly brought a lot of them along with us.

A lot of the motivators and drivers for doing DevOps are speed to market, agility, driving down cost, reducing administrative overhead, all the well-known drivers. We were reading about them and saying, yes, that's a good reason to do it.

Where we were agile before we were doing agile was that our developers embraced it. They wanted to learn new tools. They wanted a new way of working. Rather than management sending us the new tools, we advocated and went off and did it ourselves. Despite what everyone else says, we didn't really have a lot of management buy-in. We didn't have any budget. We just did it because we felt empowered.

We were agile before we were doing agile. We didn't have a scrum master or product owner. We had that agile mindset, almost operating like a lean startup. People told us after the event that we were doing it, and we didn't realize that we were. We stopped along the way. We prototyped it. We demoed it. All the good stuff that you would build with a minimum viable product, we were doing that before we realized. We just didn't have the vocabulary or the expertise to describe it that way.

It empowered us that we were doing the right thing without a lot of the management structures and maybe a lot of the baggage that goes along with it. People were coming along with us. We actually rolled out our migration in an agile way. We called them phases rather than sprints, and they were maybe months rather than weeks, but it was doing the right thing. We thought we were onto something.

What we did was set up a series of change champions. We called it like Fight Club, where we tapped up people in different teams and said, "Come and join us. We'll tell you what we're doing, and then it's your job to roll it out within your team. But you're not allowed to go and tell everyone, because if you did, we'd just be overwhelmed by everyone wanting to embrace these new tools and the new way of working." It was controlled, but uncontrolled at the same time. It was disruptive, but in a very planful way. We like to be conflicted. We have to do it in such a way that we can be successful.

Certainly Allstate's DevOps journey so far started very much on the distributed side with .NET. We're a big Pivotal Cloud Foundry shop as well. They got it, but they built it out really as day one. It was lots of freestyle jobs, a lot of really smart people doing really smart things. Maybe security and those things were an afterthought, and then you come to it later on.

The thing that stood out was that mainframe DevOps just wasn't even on the roadmap. It was an afterthought. Mainframe was going away. Why would you do it? It would be wasted work. We said, "No, certainly not." Most of our eminent engineers are on Z, on the mainframe. If distributed can do it, we can do it better. We took it as a challenge rather than a threat.

The current state is that we're doing pipeline as code on the mainframe, on Z. We baked in security from the get-go. We discovered secure DevOps, or DevSecOps. That phrase had already been coined. We didn't know what to call it, but we knew that if you're doing DevOps, it should be secured.

We're making the mainframe a lot more cloudy or cloud-like. We're moving towards things like Brightside or Zowe, those command-line interfaces. We're also trying to bridge the skills gap. As our profile of developers alters, and we have more experienced folks maybe moving out of the business or out of the firm, we're trying to distill a lot of their secret sauce, their expertise, and put it as code. Rather than have people train up with job aids or learning documentation, we distill all of that expertise and put it into code. The code is auditable. It's version controlled.

Our journey was that we went from zero to day two DevOps and tried to solve some of the more interesting problems, and really to do it at scale. On the distributed side, it was very much hundreds or thousands of Jenkins jobs. It's very hard to manage. We wanted to keep it simple, but at the same time, we knew that what we built on the Z side would probably be pretty clean code. If we did infrastructure or pipeline as code, it would be tightly written. If we find vulnerabilities in the code, the Z engineers would be all over it trying to get it fixed.

It also helped us that the security engineers' focus was more on Kotlin or some of those newer languages, maybe written by contractors. You think maybe the risk profile is all of the Z developers are still around, but it has to earn its right to exist at Allstate like everything else. We can't just trade on past glory and say that the code is secure because it is secure. We had to have evidence to do that. We hooked it up to the company's security scanning software.

At Allstate, particularly in Z, we like reusing what we've got. We've got a great mainframe, great hardware, great software on it. Rather than buying even more servers, we think we should reuse what we've got. We tried to reuse a lot of the pipeline technology that our distributed colleagues had done for mainframe. The more we understood it, the simpler it would be.

What we wrote was pipeline as code. If you didn't care as a developer what was in the pipeline, you didn't need to. But if you really wanted to pull back the covers and see the code, we also showed it as a roadmap: this is what the pipeline is going to look like. Even though we didn't have some of the steps necessarily done, or we still don't, they're in the pipeline. It's not just on a PowerPoint slide or up on a screen. It's in the code in Git, in Team Concert. The developers can come and look and see: if we want to turn on quality scanning, this is how you do it. The company can control that at their leisure.

We also feel as if we're getting out of that as well, that the technology is becoming mature. If we want it better governed or audited, we can turn the governance of the pipeline over to the governance team. We don't need to be running it on the Z side.

Again, day two issues. Peter Eales, who used to be the worldwide lead for DevOps adoption at IBM a couple of years ago, listed out all of the challenges, and those have been well documented. We thought it looked way too hard trying to do it as day one.

When we come to conferences like this, we see what everyone else has done. We come to it thinking we're maybe a bit off the pace or really behind, and I think we find sometimes we're not. Everyone's at a similar level of maturity, with some outliers really leading from the front. Rosalind and others have demonstrated DevOps on Z here. It sets the standard high in terms of what we aspire to and what we strive to do.

We like to come along with some real-world examples and say, "Yes, big shops like us are actually doing it." It's an encouragement. It's a shout-out to people who are looking at it. Is it possible? Definitely. Is it hard work? Not so much. It's actually becoming easier. We also find that a lot of our distributed colleagues are only too happy to help. They're interested to see what we're doing. We like to be provocative, thought-provoking, and say, if we can do it in Z, then our distributed colleagues should be out-pacing us.

With day two DevOps, maybe the thought is how to do it rather than what it is. What is DevOps? I think everyone here knows what DevOps is. They have a good grounding in it. It's more how to do it at scale, how to measure value streams, measure flow, all of those concepts. I think we're all reaching common ground, and it almost feels like a tipping point, certainly with mainframe, that mainframe DevOps is becoming mainstream. DevOps is becoming mainstream. Everyone's coalescing around core principles and best practices. I hope to show some of those today.

One of the things I've learned this week is around value streams: what's valuable to a company? Certainly for us at Allstate, it was around security. That's the thing that keeps us awake at night, trying to secure our applications. It was very much driven from the cloud world. They wanted to be able to do almost continuous deployments, pushes to production, but in a secured way as well. We thought, well, we have to earn the right to do that too. We need to be secure. Mainframe code should be secure. We just can't trade on past glories. It has to be evidence- or data-driven that the code is secure, and if not, we'll fix it, remediate it quickly.

That was the pipeline that the company built out for secure DevOps, and the thing that stood out to me was there was no mainframe on it. Lots of Jenkins, lots of Pivotal Tracker, lots of WebSphere. You see Agent Smith and Compliance Buddy. Those are internal security and auditing tools that Allstate built for cloud. We thought we should be able to use them for mainframe too.

We went to who wrote the slide and said, "You need to put mainframe onto that. We're just the same as everything else." And if you're struggling to understand why, with the Linux penguin on there, we're pretty Linuxy as well. If you want to put us on there, put us beside the Linux. We're just the same as everyone else. We're running Unix on the mainframe, USS. We're no different from anything else.

The idea with Veracode is that it's an external source code scanning system. We push our code out through Agent Smith, and then the results of that are sent back to Compliance Buddy. We like the idea at Allstate, and particularly on Z, of continuous compliance. We don't like the idea of stopping and then doing it after the event or in a last-minute rush before we push to production. We like the idea of being able to continuously monitor our compliance. Compliance Buddy looks at source code, SOX compliance issues, Jira stories, testing results. It's almost like our internal compliance tool. It's a natural fit for cloud, but as we make our mainframe more cloud-like, we think we should be able to do the same thing as well.

How you do DevOps: a lot of these things are well known. We decided to use that framework as well, derived by Jez. Where we're at with Allstate in terms of how we approached it: we moved off our SLM system, and we moved my former application, the billing application, which is the biggest, most complicated one. We did that first. It was a statement of intent. People looked at us and said we were crazy. Why would we do it that way? Really, we had a lot of empowered developers. They wanted to make it successful, and it was a challenge to the rest of the organization. If billing could do it first, then every other application was going to be easier. We took the risk on. We pump-primed it, put it at the beginning of the pipeline, and did the billing one first.

We're just in the final stages of moving all of our Z developers over to Team Concert. We like Team Concert. It works really well for us. We're interested in Git as well. As part of mainframe, we're agnostic about technology. We use what is suitable. To Rosalind's point yesterday, it's all about the code. We like infrastructure as code, pipeline as code, compliance as code. We think that paradigm is well set in the industry now, and it feels good. Our developers are lining up with their distributed colleagues. We see ourselves as peers. It's not us and them, the Z. Our mainframe is just the same as everything else.

We built it out for waterfall because, as part of our journey, we were always compromising or choosing our battles to fight and win. We weren't going to be able to do all of this necessarily and move the whole mainframe Z world over to agile quickly. With Team Concert, we built it out in a flexible way. We future-proofed it. We liked the idea of having succession planning and future-proofing, so that while you could start in waterfall, as we understood agile better, we could start transitioning over and people could move at their own speed.

We were mindful that it's more people embracing it rather than having it inflicted on them. We would let people experiment with agile, but at the same time, they still had the waterfall background to fall back on. They use whichever methodology is appropriate, whichever is most cost-effective. Some work is done agile, others is done as waterfall, but Team Concert and the pipelines we've been building out give us that flexibility.

The future state is probably moving towards more hybrid cloud, making the mainframe more cloud-like, making it easier to integrate with our other distributed and cloud platforms. We're integrating it more deeply with Jira and some of the other tools that the company uses as part of agile, so Confluence, all those good things.

Our developers are using Team Concert with RDZ. It's an Eclipse-based interface. We went all in. We didn't try to let our developers use green screen with the new source code system or with pipelines. At the same time, we didn't completely deprive them of green screen. If there is some technology or some patterns that they find useful for it, they use green screen. But for development on Team Concert, they all use RDZ, and increasingly IDZ as well. We're also getting interested in building out APIs on Z.

As I look back on it, a lot of it was more cultural. It was that we've moved the whole organization away from green screen towards a new way of working. But like agile, we don't get hung up on tooling. It's just a means to an end. It was us signaling to our Z developers that we're heavily invested in them. We see them as on a platform that's successful and is future-proofed. It was a statement of intent that the mainframe is in good health at Allstate. The developers have earned the right to those tools. They deserve it. They've labored long and hard with older technology, but it served us well. Now is the time to move on.

Similarly, using Z Unit, we also see it as future-proofing or investing in our future. With the modern tooling, IBM and other vendors are at more of a continuous delivery model. That suits us too. Some of our developers are bleeding edge. They like testing out new features. As soon as it arrives, it drops, it's installed. They want to be prototyping, trying it. Others less so. It's sure and steady. That model works well for us.

We like to run prototypes, experiments, MVPs with both features on Team Concert and on the client side as well. If it reaches mainstream and people like using it, then we roll it out. But again, we don't inflict it. We let people find it themselves. We run lunch and learns, community of practice. We do a lot of hands-on demoing, prototyping, showing it. We like to do real live demos.

For anyone who's not familiar, that's what RDZ, IDZ looks like. It's Eclipse-based, a modern paradigm. We also have some of our developers using Visual Studio Code as well. We're not fussy around the technology. We work with what people find appropriate. It's what they're happiest to use.

In terms of deployment tools, this is probably still a work in progress. Allstate's been served well for many years by our homegrown in-house deployment system, APTF. We've run it for about 25 years, 22 years, so it's getting to end of life. We're planning to move on to UrbanCode Deploy. We did a proof of concept of it a couple of years ago, and that turned out to be very successful. But we chose our battles. We thought that certainly in the billing application, we could pull further and further ahead and leave the rest of our Z developers behind. That was maybe the value stream that was more useful, more valuable for the company: for us to slow down to bring all of the developer community with us, rather than us becoming an outlier team pulling further and further away.

With that, we parked some of the UrbanCode Deploy work, and we're picking it up again. That's based on the community of developers now all using pipelines, all moving towards DevOps, and then UrbanCode is a good fit for us. It still is very much a work in progress. It's not completed, but UrbanCode is a useful addition to our toolkit.

Again, it's pipeline as code. We're experienced users of Jenkins now. What sold it for our developer community was when we told them they're going to have to be using pipeline technology, a lot of it was, "What do I need to know?" And it was, "You don't need to know anything." The security scanning was the bit that was most valuable for the company. If you wanted to scan your mainframe code, you just pushed it into the Jenkins pipeline. It seemed obvious after the event, but not at the time. People thought they would have to do it manually. Pushing it out of Team Concert and kicking off a Jenkins job scanned the code through Agent Smith and reported the results back. It was win-win.

Now we know that that was probably a large value stream. It was highly valuable to the company, so much so that we've probably outpaced our distributed colleagues. While they had done the day one approach of freestyle jobs, we went straight to day two with infrastructure as code, pipeline as code, and got to the right answer first. You imagine it didn't make us very popular. But it was a good thing for our leadership in terms of, even the Z guys are doing this and they're fully compliant, so everyone else get a move on. It was almost like a feedback loop. Then the developers on non-Z thought, "Well, we're being outperformed here." It was a good feedback loop, and it spurred some healthy and unhealthy competition. But we're glad we did it.

The pipeline is what we're trying to do: build out, either using open source tooling, Team Concert, IDz, dependency-based build. It's all the good stuff that Rosalind and others have spoken about. It's becoming mainstream. It is mainstream. But the challenge is, where do you start? However small at Allstate, once it's up and running, then it gets traction. It's almost unstoppable. Even if it's one team with a lot of clout behind it, other people sit up and take notice.

Very quickly, in terms of going through it: pipeline as code, just doing it the way probably a lot of you do it already. Team Concert, it's pulling every R, scanning COBOL, copybooks, all that good stuff. We just make it easy for the developers. They don't need to know. They only get interested if the results come back and they see vulnerabilities. Veracode is reporting CVE numbers or issues around dynamic SQL, all those good things. We found that all of our mainframe apps have been scanned. That was a big win for the company and for us too.

In terms of what it looks like, it just runs the same way as everything else. It's not very complicated for mainframe. Rosalind's shown us how to do it. We're doing it. Have we finished it? No. We're on our way, I would say. It just works the same as distributed.

Maybe the clever bit, or the insight that we had, was to go straight to pipeline as code. Rather than trying to build out lots of freestyle jobs and engineer a lot of technical debt out of the application into the Jenkins infrastructure and DevOps pipelines, make it very light. Our pipeline code really doesn't reference mainframe. Our plan is to have a single mainframe pipeline rather than lots of developers having to write their own pipeline code. It's almost like the common developer framework or platform that the guy from ITV spoke about yesterday: keep it simple, make it easy for the developers. If you're a mainframe developer, check the code in, and then it's scanned, built, compiled behind the scenes. Just keep it very simple.

In terms of running it as pipeline as code, most probably all of you are aware of what it looks like. We keep it like a declarative pipeline in Jenkins. We keep it very high level, and it's almost like the more we understand it, the simpler it becomes. We started off down and dirty, in the grass and the weeds, and then abstracted it.

In the pipeline, we put in all of the steps we'd like to do and comment out the ones that we haven't reached yet. In terms of the UrbanCode Deploy step, we haven't done that yet. We're still using APTF. The pipeline is not friction-free, but it shows our developers that the roadmap for the pipeline is the pipeline itself. When you look into the pipeline, you can actually see the steps that the company will evolve and reach.

We spin a lot of plates. We keep lots of things running. We push on what we can get some traction on, and other ones we hold back. When we go through procurement or legal or licensing for certain technologies, that's not going to stop the pipeline. We just comment out the bits that we can't run yet. But if developers care to look at the pipeline, they can see where we're going with it. It builds that adoption, and as people onboard to Team Concert and onboard the pipelines, they find it becomes easier. That's the big selling point or value add for the developers: we make their job easier to do.

It hasn't come easily. We've had to work with everyone from, "I can't wait to work on this. Can I be an early adopter?" to people wanting to walk out of the training room, and everything in between. But that was what we expected. We did a lot of coaching and mentoring. We discovered paired programming for Z, where developers work together.

When we printed off all of the manuals, we looked to see who was finishing using the manuals first of all, and what they were writing or annotating in the diagrams. When we asked people, "Did you get it?" and "Did you have any issues?" everyone always said, "No. It's fine. I completely understand it." But we looked very carefully to see who was still referring to the manual, and that was our counterintuitive, or anti-metric. The people who finished the manuals and put them away and locked them in the desk, we knew they were well on their way. For the people who wrote in the margins, we were interested to see what we needed to double down on in training. It was an anti-metric, something very counterintuitive that we heard this week, that our success was we almost gave our developers a waiver on being able to be measured with this.

Just to finish up in the last couple of minutes: we're getting really interested in using APIs on mainframe. We're following the same pattern, the same paradigm, where we build it out as infrastructure as code, pipeline as code, storing all the source code, all the build artifacts in Git or Team Concert, using Jenkins, Artifactory, UrbanCode. It's almost becoming industry standard. That's the way you do it. While some of the tools may be different, the takeaway is mainframe DevOps is mainstream. It's doable.

No matter where you start, you're going to progress with it. You'll be surprised in terms of the skills gap. We have interns or new hires coming in without any Z experience, and we see this as a good way to get them onboarded. It's a hard sell to have people learning ISPF and green screen. It's a much easier sell to say it's like a command-line interface. It's like AWS.

The current work is that we've moved on to the newest version of Team Concert. We're doing SonarQube scanning soon, moving towards a single pipeline, and also trying to be more mindful in terms of the work that we do as waterfall.

Probably like most big shops, we don't find it easy, particularly on the Z side. There's a lot of culture that we're still working through, but that's a good thing. The tooling helps, and it's a signal to our developers that we're interested. We think there's nothing worse than working on a platform that no one's talking about, no one's interested in. People are very interested in mainframe at Allstate. That's not to say there are other great platforms that people use, but certainly mainframe is back and visible in a way that maybe it wasn't a couple of years ago.

We're interested in Zowe, Visual Studio Code, and it's almost like a skunkworks or early adopter, where we're interested. Events like this, we encourage our developers to keep up to date with technology, to see what other folks are doing, what they're doing in the industry, to read the blog sites. When an opportunity comes along to do a proof of concept, an evaluation, or a POC/MVP, people are ready to go. They're tooled up. They have the expertise to be able to do it.

What worked for us at Allstate? Leadership support. I think we have lots of it now once we're successful. Particularly with APIs as well, it's gone from, "Why are we doing it?" to actually, "How soon can we build them?"

I'll probably finish there. That's me. I'll call the wrap. Thanks for your time today. Sorry I'm running slightly late, but thank you.