Thomas Lawless

Thank you for joining us. We realize we're the last breakout session of the day, and we are standing between you and all the great activities planned for tonight: the book signings, the Lightning Talks Dinner. So we're going to do our best to give you guys a little bit of insight into our developer experience program and our enterprise IT transformation.

My name is Tom Lawless. I've been with IBM for 23 years, and I'm a Senior Technical Staff Member, or an STSM for short, and I lead our CIO Developer Experience initiative.

Mirella Batista

Hello. My name is Mirella Batista. I am also an STSM within the developer experience in the CIO. I've been at IBM a little bit less than you have, a little bit over two years, but I do have 25 years of experience as a software engineer. I am a self-taught coder. It has been a long and interesting journey, but it's brought me here to the stage to present to you. So thank you for being here today.

Tom and I are going to share with you some of the things that are going on in the CIO at IBM, and how we're transforming how we deliver enterprise IT with a focus on developer experience. We're going to talk about the what, how we are changing the way we work, how we're overcoming challenges, and how we are measuring impact.

Thomas Lawless

So what does the IBM CIO do? Well, we run the IT that runs IBM. So we provide enterprise tools and services that support accounting, human resources, sales, client support, as well as our internal developer productivity tooling and developer tooling. We're a large organization, globally distributed, made up of thousands of developers.

So to truly understand the scope of our developer experience program or initiative, we have to define what developer means in our context. So we talk to our teams a lot about how we are all developers. It doesn't really matter what your job role is. You might be an application developer, you might be an SRE, you might be a data scientist, an architect, or you might be in a maybe less technical role like product owner, scrum master. We are all developers within the CIO, and our experience program needs to be inclusive of all of our developers. So that's really a simple, straightforward, concise statement that we can make about the intention and scope of our program.

We deliver hundreds of enterprise tools and services, and they are comprised of tens of thousands of GitHub repositories. So, Mirella, how many GitHub repositories?

Mirella Batista

I have no clue.

Thomas Lawless

No clue. So we estimate roughly 20,000 GitHub repositories, but the truth is, we don't know. We don't know exactly how many repositories that we manage as an organization. So I'm not alone in not having a clue. It's been a real challenge for us, not just for Mirella, but it's been a real challenge for us as an organization over the last several years, because it's hard for us to define what our developer experience program really needs to target if we don't truly understand the scope and breadth of the technology that our organization uses.

So who remembers the Log4j vulnerability from a few years ago? I'm sure if you didn't live through it, you at least heard about it. So in our organization, what we should have been able to do is we should have been able to run a report: how many repositories are using a vulnerable version of Log4j? And we should have got a list. We should have been able to use that list to run automation that would have automatically remediated those vulnerabilities, and at the very least, push that back to the repositories as a pull request so that the owning team could have at least reviewed it, merged it, and got it out into production.

But we didn't do that. We couldn't do that because we don't know how many repositories that we're dealing with, and we don't have deep insight into what those repositories are and the composition of those repositories in terms of open source. And that really is the best example we have about why it's so critical that we get our hands around our source code so that we can truly understand what we're dealing with.

That really becomes the core of our developer experience initiative. Our mission is to help our teams continuously improve by removing waste from our system. So Log4j, again, as an example: because we didn't have the data, we had to run it as a program. We had to go to every team and have them go to every repository that they own, investigate whether it used Log4j, if it did, was it a vulnerable version of Log4j, and then report back. That took weeks. Then we had to drive a manual remediation effort, again taking weeks. It's a tremendous amount of waste for an organization of our size.

So when we talk about continuously improving developer productivity by reducing developer waste, developer waste in our context means any redundant and/or manual effort that takes our developers away from delivering value back to the business through feature development.

And over the last several years, we've developed a pattern for the way that we roll developer-centric initiatives out to the organization. And we call it the developer experience pattern, because that's pretty straightforward. So we break it down into five categories.

The first is governance. Who loves business processes? We do. We have a ton of them. We have business processes for everything that you could imagine. But when we talk about governance in this context, we talk about all the processes that a team has to execute in order to get something into production. So think business case review, but also think cybersecurity, data privacy, all of the checks and balances that are required.

Our processes have tended to be somewhat manual. Not only manual, but repetitive. So not only do you do this once, you wind up doing it every quarter. So think about recording your cybersecurity stance quarterly for every repository that you're responsible for. Think about the manual effort that that takes, the redundant effort that that takes. That is what we need to remove.

And our mantra around that has become: eliminate, simplify, automate. So, one, eliminate. Do we even need that process? Where did it originate? What purpose does it serve? Can we just get rid of it? Let's eliminate it if we don't need it. Simplify: if we need that process, does it really need to be this complex? And we tend to like complex processes. So can we simplify it? And then ultimately, we have to be able to automate it.

So whenever we roll a new program out to developers from a governance point of view, we go forward with at least a plan on how we're going to get to an automated governance process, if not deliver an automated governance process from day one.

Platform. We have a pretty broad definition of platform within our organization, and it's really every tool or service a developer touches that brings something into production. So it's not just our hybrid cloud platform. It's our source code management system. It's our cybersecurity tools that do code scanning, vulnerability scanning. It is our, give me another example, Mirella.

Mirella Batista

Source code management.

Thomas Lawless

Source code management, right? It's everything that a developer touches, in every situation possible.

We want one tool wherever possible so that we can cut down on, again, redundancy. Redundancy for us equals a lot of the developer waste. Our platform winds up being opinionated, which is okay because that helps us accelerate value. But we also need to automate our opinionated usage of the platform, which brings us into the third category: automation.

Now automation is one of our longer-running programs for our developer experience program. We have been working on a centralized CI/CD system for a couple years now. And the purpose, or the goal, of our system is to, again, cut down on the redundant work that our teams face when creating DevSecOps automation. It's been one of the larger areas of redundant effort.

So we've had plenty of options for CI/CD tooling over the years. And what we're trying to do now is we're trying to centralize all that to one platform. Excuse me. So think about it this way: it's not just one execution system, it's really one pipeline. So a developer doesn't keep the definition of their CI/CD automation in their repository. We keep it in a centralized repository. And when we need to support a new feature, we do that through inner source, which brings us to advocacy.

I'm sorry, Mirella, can you talk about advocacy for a minute?

Mirella Batista

Yes, of course. So advocacy: all of these tools and new ways of working and everything that Tom has been discussing, we need to help our developers in adopting these tools and upskilling them, and really changing the way that they think about the approach to the work so it supports the new ways of working. So we're focusing on education efforts, we're building communities, and we are encouraging reuse through our advocacy programs.

Thomas Lawless

Yeah. And advocacy is where we build culture too, right? Intentionally build culture. Because what we found over the last several years is that if we're not intentional about building the engineering culture, the organizational culture that we want to see, we see culture pop up all over the place at the team level, at the portfolio level. And that, again, winds up driving redundant effort, which we're trying to reduce as much as possible.

So the final category is analytics or metrics. And we look at this probably from three points of view. Right, Mirella?

One is, our mission is to help developers continuously improve, so we want to be able to provide that feedback to our developers about areas where they can improve. And we want to do that in a way that doesn't force context switching. We don't want them to go to a code quality tool, to a vulnerability management tool, to a tool over here, a tool over there. We want to bring all that data into one place and put it in context of the work that they do, because we know context switching is a productivity hit.

We want to be able to also measure the impact of our initiatives. Are we really helping our developers be more productive? Are we really helping them deliver value faster to our business?

And then finally, this is a new area that we're starting to dabble in. Do we understand the cost of running these features in production? Do we understand what the value the feature brings versus the cost of running it? Not only development, but also for an extended period of time, and does it make sense? If it costs X to run it and we get Y value, if X is more than Y, then it probably doesn't make sense to do it. So getting that deep insight into the operational cost of what we do is becoming more and more critical.

So I'm going to talk for a couple minutes about two initiatives that we're running. And I like these two initiatives because they really represent the spectrum of the complexity that we deal with. So on one side, we have transforming z/OS developer experience. This is a modernization journey. We're a big consumer of mainframe, and we have applications. I think the oldest one we found, Mirella, was like 50 years old, right? Literally, it was written 50 years ago. Can you imagine?

And on the other side of that, we have generative AI. So generative AI is brand new. We feel it's so important to the organization. I'm sure you all do as well. It needs to be successful. But they represent really two very different initiatives. But we want to apply our pattern, or the developer experience pattern, to helping our organization be successful at both.

So transforming our mainframe developer experience: again, governance. We run our most critical workload on mainframe. That is where the majority of our audits happen, because that is where we have to enforce our financial controls. So being able to automate that, not just the governance, but also automate it through platform management as well as DevSecOps automation, changes the way that we approach auditing as we go forward. And we've heard a lot about agile auditing today, continuous compliance. So that is where we need to get to for our z/OS developers.

But it's also about modernizing the tools that we're using on mainframe. So source code management, for example: historically, up until about a year ago, I would say most of our mainframe code sat on the mainframe using mainframe-specific source code management systems. Some areas just on the file system with no real version control at all. That represents a lot of risk for us because, one, it's our critical workload, our most critical workload. And two is because we're not investing in the protection of our intellectual property in that area. We're investing it in GitHub, and we need to get all of our code into GitHub so we can reap the value of that investment.

So we're driving a migration effort to get all of our mainframe source code into GitHub so that then we can also modernize the tools that they use, like getting off the green screen. Now, there's nothing wrong with the green screen. A lot of our developers love the green screen. They've become very productive with the green screen. But we can give them more modern IDEs that do things like syntax highlighting, and that for our maybe less-experienced mainframe developers help them be much more productive.

And then finally, really introducing DevSecOps practices. Once we get the source code into GitHub, then we can really, truly use our centralized CI/CD system and our one pipeline, our one way of doing it.

Now, on the other side of that, we have generative AI. Generative AI governance. AI governance is really still emerging, and we don't want our teams to go off and figure this out on their own. We want to do this in one place, and we want to do it in such a way that we can evolve it as the industry evolves with us.

We have the opportunity here to talk about opinionated platform abstractions, finally. A lot of what we deal with is existing code. We have mainframe code that goes back decades. We have Java code that goes back to early 2000s in some cases. And even if that code was written perfect at the time, over the years it's accumulated technical debt. So we spend a lot of our time talking about technical debt with our teams and the cost, the bubble cost, of getting onto a common CI/CD platform so that we can then reap the benefits of that long term.

But this is an area where we can get out in front of it. We can do what we think is right from the beginning, providing the right abstractions, the right automation, and put ourselves in a position where we can really scale this quickly so that our organization can be successful.

And then finally, upskilling our developers. Most of our developers are software engineers. We're not all data scientists. So how do we educate our developers on things like prompt engineering, and how do we do it in an effective, productive way?

Mirella Batista

So as you can hear, there's a lot of change going on within the CIO, and change brings challenges. So we are focused on, especially all of this is happening at a 113-year-old company with a globally distributed workforce, which is also multi-language. So we want to make sure that we enable communication across the globe, across the world, across the languages. We want to make sure that we have alignment throughout our development team so that we're focused on obtaining the right outcomes. And we need to promote the habits that lead to a culture where developers can thrive while adopting our new ways of working.

Oh, you changed the slide on me. I wasn't even looking.

So after all of this, there's a lot of change. And all of us in the CIO want our developers to succeed, and we want to help them succeed. And I am incredibly passionate about helping my fellow engineers in this. We are addressing a lot of their needs in an empathetic approach. I'm a firm believer in demonstrating organizational empathy.

So our z/OS developers in particular face a pretty big challenge while they're doing their regular work. We're asking them to upskill to Git and GitHub because these are brand-new skills for them that this population, the majority of this population, has not worked with before. We're asking them to adopt CI/CD pipelines, again, something brand new. And at the same time, we're asking them to migrate their applications to these tools and these ways of working.

So what are we doing? We're going for communication that catches their attention. We want to effectively communicate the changes, the things that they can do to stay ahead of the changes. One of my favorite things that we're doing is that we created a myth-busting video series for z/OS developers where we take around 12 of the most common misconceptions or confusions. I had one question that was, "Is GitHub the same thing as CI/CD? Which one do I choose?" And we're busting those myths.

We spent a lot of time working on the script for these videos so that the tone is positive and encouraging. It takes into consideration that these are very smart people, and we are helping them just through a little bump in the road.

Another thing that we're doing is enablement sessions, traditional-style live enablement sessions. But my favorite one of these is, again, for z/OS developers, where we created a panel of early adopters. These are people that have been working for about a year already in the migration of their apps to GitHub and adoption of CI/CD pipelines. They were peers of our audience, our attendees, and they got to talk about the challenges that they faced and how they overcame them, and how much support and help they got from the organization.

And finally, and this is where I'm going to totally geek out with you all, we are investing in learning, finding better ways to learn and understanding how we learn better. And one of my favorite things that's going on right now is a series of workshops based on the learn, do, teach approach. This is for the z/OS developers. Again, we are working with a globally distributed population. Our first cohort is in Japan and Australia.

So during the learn-do-teach workshop, these participants are learning through traditional-style instructor-led workshops. Us on the East Coast, the instructors and I log onto our laptops at 7:00 p.m. at night on a Monday night. And our participants and our attendees in Japan and Australia, it's just the morning for them. So we're teaching into the future, which I'm really nerdy about.

So with those once-a-week sessions, a two-hour session once a week, we're introducing the topics that they're going to be exploring during the rest of the week. The topics in this case are common Git commands, introduction to GitHub, oh my gosh, merge conflict resolution.

Then what we're doing is we're giving them a do assignment. This is learning by doing. During the learning-by-doing portion, we give them an assignment where they're asked to work on a daily basis throughout the week, hands-on experimentation with what they just learned. But we're doing it with real code. We're asking them to migrate their real z/OS apps during the rest of the week after the session.

And finally, the last part is learning by teaching each other. It is commonly known that if you really want to learn something, try teaching it to someone else. So we have these participants that at the end of the workshop, we're going to ask them, or we're asking them, to pick from a list of predetermined topics, think trunk-based development, PR practices, merge conflict resolution, to research these topics and come back and give us a five-minute talk on each topic.

Now, I said our first cohort is in Japan. Some of them don't speak perfect English, but I want all of them to participate in this. So we are going to do lightning talks in Japanese with a translator whenever necessary. What we want to do is for them to experience not only that they can research and learn on their own, but they can teach others and support others when they practice the lightning talks for the rest of the cohort.

So at the end of this workshop that I totally geeked out about, we are going after several outcomes. We want to have empowered learners who have gained the necessary skills to successfully migrate their applications. These are z/OS applications. We're going to have at least partially migrated applications by the end of the workshop because we're working on real code.

And finally, we're building community, because when these participants... Oh, one thing I forgot to mention. They are supported by a global network of mentors that we have created. We have mentors in the U.S., in India, in Europe, and we have mentors in Japan. We have a Slack channel so that as our participants are working on their tasks, any question, they ask the question in the Slack channel, and around the globe, around the clock, there'll be someone there to answer.

So that is also teaching them: you can reach out, ask for help. We have a huge community, a worldwide community that can support you. And that is one of our outcomes as well.

Which brings me to another of our areas of enablement, which is community building. We want to build an engineering culture through community where we encourage reuse. We want these engineers to be able to talk to one another and say, "Hey, I have this idea." "Oh, wait, that idea already exists." Or, "This is where you look to see if this already exists." So we reduce the duplication of work.

We want to encourage automation, in particular for z/OS engineers. These pipelines that they are adopting, they don't have all the features that they might need. So what we want to do is, "Hey, you want to ask for a feature? Awesome. Jump on board and help us develop it and contribute to the future." That was reuse. That was inner source contributions. Inner source. I totally changed topics.

Add automation to the pipelines. Use automation whenever possible. Increase productivity and communication. Know that there's somewhere where you can go and ask. You just don't stay in the immediate area of your team. There's a greater community where we can talk to each other and break down the silos.

Thomas Lawless

So we'll leave you with a few metrics, a few data points. We currently service 800 enterprise tools and services, and that's reaching probably about 90% of our overall portfolio.

5,000 GitHub repositories. About 25% are what we expect, but it is growing. And we're starting to take away other options now, like other CI/CD options, to help teams migrate into the strategic initiative. So that number's climbing pretty quickly.

We've executed 14,000 production deployments in the last six months, which is pretty good from our point of view. The teams that are using our common CI/CD platform are seeing around an 80% reduction in change lead time.

And inner source is so important to our initiative, because as we drive teams towards a common platform, we have to give them the capability of contributing the automation that they need. And it helps us build the engineering culture that we want, one of collaboration, one of contribution. We've gotten hundreds of inner source contributions. We are probably the second most successful inner source project within our company at this point.

Mirella Batista

So how can you help us? You can help us in the following areas. You've heard what we're trying to do. You've heard the challenges that we're facing. How to achieve alignment with a large organization? Do you have ideas that we can put in play? How do you approach new developer-centric initiatives? And how do you encourage behaviors that build a healthy culture?

And with that, we want to thank you so much for your time, for attending this session, for holding on before dinner. And please connect with us. You have our QR codes. We would love to continue the conversation, and we're happy to answer any questions that you all have after the... Let them clap. Okay? Okay. Clap now.