Panel Discussion: Extending DevOps to the Mainframe

George DeCandio: Good morning, everyone. My name is George DeCandio. I'm the CTO of the mainframe division in Broadcom, and it's a pleasure to be able to host for you today a panel discussion on extending DevOps to the mainframe. So we were pla

ing on having three panelists here today, but we've had an emergency come up, so we're only going to have two. So let me introduce to you these two panelists.

As I said, my name is George. We've got here Matt and Venkat. Would you like to introduce yourself, Matt?

Matt Hogstrom: Sure. Matt Hogstrom. I work for IBM. I'm based out of Raleigh, North Carolina, and my day job is working on how to manage the platform better: monitoring, system automation, incorporating things like Ansible into our runtimes. As well as I spend a good portion of my time working on Project Zowe, which is an open source project helping to modernize the mainframe.

Venkat Balabhadrapatruni: Hello, everyone. My name is Venkat Balabhadrapatruni. I'm the solution architect for the DevOps portfolio of products in Broadcom. I'm based out of Plano, Texas. Essentially, the goal, or my day job, is to make sure the topic that we're talking about today, extending DevOps to the mainframe and how to enable that.

George DeCandio: Okay, great. So the format we're going to use here is I'm going to pose questions to the panel, and you will answer, and just interrupt with any insights that you have any time we're going through.

The first question is just about mainframe and DevOps. What's the main difference, the difficulty in integrating DevOps with mainframe? Why is it so hard? Why is it the final frontier? Why don't we start off with you, Matt?

Matt Hogstrom: Okay. Well, one of the big challenges with the mainframe is it's got 50 years of investment in it, which means customers have been building business logic. They've built processes, they have culture, and they've got a substantial, I'll say, velocity of what they've been doing for quite some time.

And so, when you bring somebody to the platform, oftentimes they feel like they have to go through a whole set of different training. I'm going to use something like ISPF. How many of you folks are mainframe people? Okay. Ooh. So when I say like 3270, you'll know exactly what I mean, right? And so it's not exactly the best IDE for developing complex programs.

And so our big challenge is how do we take the investment that we have in the platform, the existing processes, et cetera, and then how do we enhance those and make them so they are just as familiar from a process standpoint to new developers coming from, say, cloud native.

I'll just add one little bit as an anecdotal data point. In the development that we have for systems, meaning the things we're building with our monitoring solutions like OMEGAMON or NetView and things of that nature, I have a number of new hires. These are our, I'd say kids, but that might be considered derogatory, some really bright folks coming out of college that never worked on a mainframe before. What they find really compelling is leveraging REST APIs, Angular, and new technology so that they can build the next generation of system functions, and we want them to be up to speed as quickly as possible. Making sure that we have the right platforms, the right REST API endpoints, et cetera, makes them productive on a platform and continues that investment.

Venkat Balabhadrapatruni: So, I think to build upon what Matt was saying, why is the mainframe considered as the final frontier? I think the key thing that Matt hit on is the history and the heritage. As you heard this morning during the general session, there is 50 or 60 years of processes, tooling that is already in place, and customizations that organizations have put in place.

When you take a step back and look at DevOps in general, one of the key words that you heard this morning repeatedly is the word culture, the word people, starting with that. The whole idea of DevOps is going through that cultural transformation. When it comes to the mainframe, that cultural transformation, the complexity of it is a notch higher compared to any other distributed systems that you would see. The reason is because of the history. Traditionally, mainframe has grown up to be change-averse, risk-averse, has traditionally been siloed and operating its own environment. That's not good or bad. That's just how things have evolved over the years.

Now, in this current digital age, when we're actually talking about REST APIs, mobile to mainframe, bringing mainframe into the fold of the enterprise and making mainframe be the same as any other platform and driving that cultural transformation is key. That's the primary aspect of it.

With culture, we talk about people. When we talk about people, the whole idea of skills actually come up. The willingness to adopt new technology is, I think, the key aspect of it as well. Matt referred to the kids coming to the platform, the fresh college grads who are actually coming to the platform, and that's inevitable. There's a lot of COBOL code out there. There are a lot of mainframe applications that are pretty much ru

ing the organizations and the whole world, and you need folks to actually manage, maintain these applications. So how do you actually bring those people in? Part of it is actually changing that culture and having more open-mindedness to embrace new technology, embrace new set of tools that are actually coming out. Keeping up with that, I think, is the key aspect of it.

George DeCandio: Okay. So it sounds like it's a combination of process, tools, culture. All of this plays into making mainframe a difficult environment for adopting DevOps.

Venkat Balabhadrapatruni: Yep.

George DeCandio: Okay, so we're going to switch gears here. Those are some of the problems, but what are some of the techniques that we can use to incorporate modern DevOps practices and tooling into mainframes? How can we overcome these problems? Specifically, we've got Muharrem here, has joined us. Would you like to introduce yourself real briefly?

Muharrem Gun: In Akbank, we also use mainframe. Mainframe is the main backend platform for us. So in order to be successful for customer satisfaction, we need to improve ourselves in distributed side and also on the mainframe side. The technologies now emerged in mainframe helps us in that domain.

So we are looking forward to modernize our application, our way of working now. It includes version control systems and also pipeline management. Like we do in the distributed side, we can do the same technology. We can use the same tools, technologies. Most importantly, our developers can also feel the same like in the distributed side. I feel that it is very important to use those technologies wisely. I think we will be more productive with using such tools.

George DeCandio: So Matt, you mentioned Zowe in your introduction. So I think Zowe is an important part of trying to modernize the mainframe. That's part of one of its goals. Can you describe a little bit about Zowe for this audience and how it will help?

Matt Hogstrom: Yeah, sure. So Zowe, if you're not familiar with it, was an interesting transformation that occurred about a year and a half ago. What happened is IBM were looking and were investigating how we can modernize the mainframe, including new REST APIs, et cetera. Rocket Software approached us and said, "Hey, we built a new desktop interface, so basically we can put a desktop motif on the mainframe because we think it would be more engaging," because people know where to go on a desktop. I can go, oh, lower left, Start. Oh, there's all the things that are there. So it was more of a guided activity than being put into, say, like TSO.

But we looked at that and said, "Well, it doesn't really make sense to do a proprietary solution and we need more than that." We ended up having this conversation with Broadcom, and Broadcom was in exactly the same place we were, investigating how do I improve access to the mainframe, et cetera. They had created a technology called Brightside, which we now refer to as the Zowe CLI.

And so if you're familiar with the story of stone soup, we went from boiling water with stones in it to some vegetables, chicken, and other things that made it actually very flavorful. What we did was we practiced the three of us coming together, who compete in the market, I think pretty aggressively, but we knew we had to cooperate with each other as well. So we wanted to open source the three technologies. By combining the three together, Broadcom brought an API mediation layer, which gave us a way of federating disparate REST services on the platform. We've introduced single sign-on.

Zowe has given us a way of cooperating and giving the platform, I think, a better position to be competing for the new work that's coming out from new applications and things of that nature as well. We just passed the year-and-a-half mark back in August. We a

ounced at SHARE, and we're right now in the process of coming up with version two. For mainframe guys, we're releasing just about once a month with new features, functions, capability. I'd say bug fixes, because that would imply we had bad code, but bug fixes as well. The point is we've got pretty good velocity.

In fact, we've also seen other vendors. Syncsort recently joined the Open Mainframe Project, which is where Zowe is hosted. We have other vendors now coming alongside as well. We've got a really great collaboration point where we can make the platform better, and as the phrase, "A rising tide lifts all boats," and that's what our strategy is with Zowe.

Venkat Balabhadrapatruni: So just a curiosity question: how many people in this room have heard Zowe? Fairly similar. And the other key word, I don't know if you guys caught what Matt said, it is an open source project hosted under Open Mainframe.

So again, going back to the earlier discussion that we were having, why is mainframe the final frontier? We're talking about cultural change. Now think about this: open source software on mainframe, and APIs. These are some of the things that were foreign words about six to eight, ten years ago, but now these are pretty much the standard. This is the new way of actually working with the mainframe.

The key thing there is, as part of that evolution, mainframe is becoming like any other platform, thanks to APIs and CLIs. The beauty of the CLI is scripting, which is the whole purpose of DevOps. The key thing that we talked about, the blocking inhibitor, or one of the things that is lacking on the mainframe, is automation. When you have CLI and you have the scripting abilities, that opens up the doors to more automation, and that is the key part that Zowe is actually bringing to the table and has become an enabler for DevOps adoption on the mainframe.

George DeCandio: Do you have any experience with Zowe?

Muharrem Gun: Yeah, actually, we plan to use the same tools, as I said, in the distributed side, like SonarQube. It's a good tool, for example. In order to use such a product, we need to have sources outside mainframe. For that purpose, we can use Zowe if we have code also synchronized, also in the Git repository. For example, we can easily use SonarLint and SonarQube. Another aspect is we can also use Eclipse IDE, another VS Code IDE, like we use also in the distributed side. So some kind of synchronization-

George DeCandio: Zowe enables you to use-

Muharrem Gun: Yes.

George DeCandio: ... IDE of your choice.

Muharrem Gun: Yeah. Standard operation, actually.

George DeCandio: Okay, great. So Zowe's one way. Yes.

Matt Hogstrom: I was just going to say, to the IDE point, everything doesn't run on the mainframe. The CLI doesn't run on the mainframe. We also have made VS Code extensions available that'll plug into VS Code, that'll integrate with the mainframe as well. So we have lots of distribution cha

els for improving the velocity for developing and interacting with the mainframe. Sorry.

George DeCandio: So clearly, Zowe's a key piece of opening up the mainframe to scripting, of opening up a web desktop that's easier for, let's say, younger developers to use, so kids. Really clear, and we're going to have a deep dive on Zowe. We're going to have another panel discussion next half hour. So if you're interested in deep diving in Zowe a bit, come join us for that.

Still talking about opening up DevOps for mainframe. Command line interface is always one area. Any other areas that people should be looking at to try to move into DevOps on the mainframe?

Venkat Balabhadrapatruni: So I think the key thing to keep in mind when we are talking about DevOps on the mainframe, and this goes back to what you guys heard during the general session today. Everyone who talked about DevOps, independent of whether it is mainframe involved or not, the key thing to remember is basically it's a journey. DevOps won't end. It's about continuous improvement, and you actually continually improve what you're doing today.

So in terms of talking about what to do, where to begin, which is the common set of questions that pretty much everybody has in their head - where do I start? The key thing is to figure out where you are today. From a business perspective, what are your key business challenges or business goals that you want to achieve? Then starting small, trying to figure out what is the current state of affairs, what are my key bottlenecks? Then having a priority of those bottlenecks and say, "Okay, here are the things that I'm going to address that is going to give me the biggest bang for the buck."

Then going back to that crawl, walk, run, and then come back to crawl, because the key aspect of DevOps is actually learning from what you've already done, because it's about continuous improvement. In terms of the practices and enabling that culture change, starting small is the most important thing, and remembering the fact that it's not a flip of a switch. Things are not going to change overnight. It's a journey, and it's about continuous improvement and having an understanding of that as an entire organization. That's part of the cultural change that you need to have.

And then there are enablers like Zowe, which are actually going to help you through that journey. Because if I look back, as part of my own experience, as part of this DevOps journey, four or five years back, we've been talking about DevOps on the mainframe for a very long time. It's like, okay, here are the things that you need to do. But now there are concrete set of tools, practices, examples that are open and available for everybody to enable that journey. There are these enablers like Zowe, which will help you go faster in that journey.

It's one thing to say Zowe is basically the enabler. The other part of it is walking the talk. What we are doing within Broadcom, there are several traditional mainframe teams who have adopted Zowe to just change the way they actually work. Be it automating their builds, or creation of automated test cases, ru

ing their existing test cases off the platform, improving their build time.

One classic example that we heard recently is there is a classic traditional mainframe product. The developers were spending 40% of their time coding and 60% of the time in actually doing the build, setting up the environment, writing the test cases, ru

ing the test cases, and things like that. So 40% actual coding, 60% on the real work - I mean, the setup work. And with adoption of Zowe and the automating of their tests and the provisioning of their environments, it went to about 85 to 90% of the time now their teams are actually able to spend on coding.

So again, what they accomplished is they eliminated the waste through the automation process, and elimination of waste is a key principle of DevOps. The key thing, again, just to recap, is start small, remember that it's a journey, and remember that there is a community and set of enablers that are there to actually help you go through that journey.

George DeCandio: So do you have any experience in how you got going in doing DevOps on mainframe?

Muharrem Gun: Actually, we are trying to improve the customer experiences. So it doesn't matter if you are on the distributed side or on the mainframe side. Your customer actually doesn't care about your technology. But what is important for them, what is important for customers should be actually improved. Agile and DevOps methodologies now complement each other in this work.

George DeCandio: You've integrated your mainframe and your distributed DevOps processes together?

Muharrem Gun: Yes. It shouldn't be any different. It should be end-to-end efficient. So your developers working in distributed side and on the mainframe side should actually live the same experience towards the customer satisfaction. So we think that we need such tools in order to improve their daily life and also the way they work towards customer satisfaction.

George DeCandio: That makes a lot of sense.

Venkat Balabhadrapatruni: The other thing that I'd like to add in here is, there was one point that Muharrem made about using SonarQube as an example. One of the things that organizations are actually looking at is standardization of tools across the board. You don't want to use a different set of tools for distributed application development versus a different set of tools for mainframe, because again, you don't want mainframe to be different than any other thing. You don't want mainframe to be siloed.

So tools like Zowe enable you to, as proven by Muharrem's team, take mainframe source code off the platform and use tools like SonarQube, SonarLint, which are the ones that are actually being used to do code analysis and code reviews on your Java or other distributed languages. Now it's possible with mainframe.

So again, the key thing, to go back, the word that stuck in my head this morning during the general session was heritage. Don't be a legacy organization; be a heritage. Leverage what you already have, and there are tools, techniques, and methodologies that are actually available to tie in your heritage to the modern world. That's the key message from where the current things stand from a DevOps on the mainframe world.

Matt Hogstrom: Yep. So as we're having the conversation, I think a real key point, it's already been said, I'll stress it, is there's not a thing that is the answer. If I'm driving a Jenkins pipeline to do build, Zowe brings aggregation or federation of REST APIs. We have CLIs. We have lots of different tools. We have dependency-based build for how we can integrate with those tools.

But one of the things we really didn't talk about, and at IBM, we're moving ahead very aggressively at how do we leverage automation like Ansible so that we're not just talking about I'm doing a COBOL build and I'm pushing it and it's getting compiled and loaded, but how do I change that whole process into something that's 100% automatable so that it's not just the building and the promotion of the artifacts?

Because the ability to really get the DevOps power is not just in development, but it's being able to take the packages that you're building or the iterations that you're releasing and push them through from test into QA, into pre-prod, into prod. And so you have to see, in that pipeline, what are the collection of tools that you're going to need? Zowe is certainly a key resource, but there's many others as well.

George DeCandio: Yeah. So I'll mention as well, coming in as the CTO of Broadcom, we have the same challenge ourselves, trying to modernize our processes and our different development teams. I was looking, we have over 250 million lines of COBOL code that we maintain - sorry, of assembler code that we maintain just as part of the products that our customers use. So a lot of assembler code.

But if you look at trying to improve DevOps across our teams, it's not a one-size-fits-all story. It has to come from the bottom, and the program we've implemented sort of enables that. We call it engineering excellence, but it's pretty much every agile iteration, the teams have to do something to improve their process, the same way they're doing a feature. They identify what that's going to be, and it may be different for each team.

So they measure before, they measure after, they see the improvement. Every iteration, we're spending time improving and moving towards a more agile DevOps environment. So I highly recommend these edicts from the top that everyone's going to do DevOps the same way. My experience with that is it doesn't work. I don't know if you guys are nodding your heads if you've experienced similar.

Venkat Balabhadrapatruni: I completely agree. There is no one size fits all. There is no silver bullet when it comes to DevOps, and that's independent of whether it's distributed or mainframe. Every organization is different. Every organization's priorities are actually different. So assessing, analyzing where you are, which is exactly what George was referring to, is the engineering excellence that we did. We had every team go review: what's your bottleneck? What is it you want to improve on? And then set goals and make sure that they measure themselves and continually improve.

George DeCandio: And some of the things that teams can't improve themselves, they need help externally.

Venkat Balabhadrapatruni: Absolutely.

George DeCandio: And that's where we need to be there to get them that help, to enable them to make the changes that they really want to make.

Matt Hogstrom: Can I just ask another qualifying question? How many people are in development only, or how many people do systems programming? So if you do development only, raise your hand. Okay. And are you on the systems or infrastructure side of the house? Okay, a couple of you.

One of the things we do at IBM, design thinking, and one of the challenges is kind of figuring out an empathy for the other people in the organization. So one of the barriers or challenges that we have typically in many organizations is the developer's not the first person that gets called when something goes wrong. It's the operations staff. So the operations staff is like, "Ah, I'm going to control as much of this as I can because I don't want to have to explain why we had an outage." That's not productive.

So understanding what some of their constraints are, as well as helping them understand what yours are. I tell you, my very first job, speaking of assembler, was a batch assembler programmer back in 1852. I was writing a set of programs, and I wasn't getting any love from the operations team. I think they hated me. So I volunteered. I went down, I said, "I will work in operations, second shift from 4:00 to midnight for three months with no pay. I'll just do that in addition to my day job."

When I did that, all of a sudden, mounting tapes and grabbing them from the library and seeing the production schedule and whatnot, I went, "Holy smokes." I was totally making their lives miserable. So by actually entering their world, I was able to work a lot better with them. Then after I finished my three-month stint, never waited for a tape mount after that. So just something to be aware as you're trying to do this cross-organizational thing. You mentioned outside help. Because the infrastructure teams really do want to change, and they want to kind of unleash you guys. We have to figure out collectively how do we change some of the contracts we have?

Venkat Balabhadrapatruni: Talking about outside help, one of the reasons why we're going down the open source route is essentially to create that community. To what Matt was saying, how it came about: Broadcom, IBM, Rocket coming together to create that community to enable you guys and actually help you guys essentially.

Matt Hogstrom: Actually, we should say Phoenix Software in there. They just-

Venkat Balabhadrapatruni: Phoenix as well, yes.

George DeCandio: So we're out of time. Thank you guys very much for your time and the insights you've shared here. I'm hoping many of you will join us to do a deeper dive in Zowe over in Montblanc 1, which is just down the hallway here in just a few minutes. Thank you all.

Panelists: Thank you.