Servers Are Not the Hardest Thing to Orchestrate
Mayank Prakash started his career as a Graduate Engineer Trainee at the Hewlett-Packard. He worked in the HCL JV starting with the entrepreneurial Frontline Solutions start-up venture and was soon after selected to join the Senior Management Trainee Programme and deputed to incubate ERP implementation capabilities working with the Big 4 Consulting firms.
Prakash was the International CIO of Avaya, then the Group CIO of iSoft and later the CIO of Sage Group in UK.[3] Prakash was hired by Morgan Stanley as a Managing Director, Tech and Data, as part of the executive team of Morgan Stanley Wealth and Asset Management.
In 2014, Prakash left Morgan Stanley to join the Department for Work and Pensions as the Director General in charge of technology, replacing Andy Nelson. In this role, Prakash combines his predecessor's job as CIO with business transformation, security and data responsibilities, and works directly for Sir Robert Devereux. As of 2015, Prakash was paid a salary of between £195,000 and £199,999 for this role, making him one of the 328 most highly paid people in the British public sector at that time.[6]
The British Computer Society welcomed the appointment, and Morgan Stanley was sorry to see him leave. GDS Chief Mike Bracken described the hiring of Prakash as the turning point for Whitehall's ability to hire the best digital leaders across industry.
Credited with quietly turning around one of the largest IT estates into a lean digital delivery machine on a massive scale, TechUK and DigiLeaders recently recognised the strength of industry partnerships and the scale of digital transformation being delivered at DWP resulting in DWP becoming #8 on Twitter attracting a million digital techies and leaders from all sectors.
Chapters
Full transcript
The complete talk, organized by section.
Mayank Prakash
My name is Mayank Prakash. I had trouble saying my name as a kid, and so here is how I learned it. If you say, "My uncle," and just stop before the L.
But the deal is, do stop before the L. It's not funny if you call me your uncle, because I have a few white hairs now, and I'm touchy about these things.
So we're going to have a conversation about DWP.
I presume most colleagues don't know what DWP is. I certainly didn't till I joined DWP. The best way for me to describe DWP is every one of you in the room is my customer. And you know what? Your parents, your uncles and aunts are my customers as well. The people you live with on the street you live on are my customers as well.
If you're a child and your parents are separating, you're a customer of DWP. If you're of working age and finding work, you're a customer of DWP. If you're disabled, you're a customer of DWP, so that we can explore the potential of your ability as the mark of a civilized society. If you're a carer looking after disabled people, you're a customer of ours. If you're retired, which increasingly people are, we are all living longer lives, you are a customer of ours.
And we, 80,000 people in the "we" at DWP, come to work every day inspired by making sure they can make a change to the lives of 22 million active citizens we serve.
If we were on FTSE 100, we would be a FTSE 5. But we'd be different from the other four on FTSE 5, because we don't come to work to generate money. We come to work to make a difference to 22 million lives. Those people are in the lives of our employees. And so that inspires people to come and achieve what is our potential collectively.
I could describe DWP in many different ways, so let me do this with a bit of fun. I've got a suggestion. Have people heard of Elon Musk's BFR? Yeah? Have you heard?
Well, for those who haven't heard, Elon Musk has come up with this BFR rocket. I'll not do the full form here. And it's apparently going to take mankind to multiple planets.
I've got that problem solved for you. If you change all the money we spend every year, that's 170 billion pounds, into pound coins and stack them up, you'd get to Moon four times. So, that's how much money we spend.
If we were a country, we would be the seventh largest GDP in the world. We manage 1.75 trillion pounds of assets. That is everybody's pension in the country. And nobody has the patience to have a little error or a bug go through in their pension statement. I wonder why.
So how does all this run? It runs on 50 million lines of code and counting. We can count at least 50 million lines of code. Those 50 million lines of code have been written over the last four and a half decades.
Add a bit of perspective: Mars Curiosity Rover. It runs on, take a guess, more lines of code or less?
Higher. Higher? More lines of code? That's Gene's guess. What do you think? More or less?
Less. Less? You're right.
It's very rare for Gene to be wrong, but on this occasion, you're right. So Mars Curiosity Rover runs on two million lines of code.
We run a massive estate. We are rewriting that estate at the pace of about a million and a half lines of code every year. Now, when I say that, most Fortune 100 CIOs sit up and listen because that throughput is very rare in any large organization.
But you can do the maths. We can't sustain building our future at a pace if we just replace at the rate of one and a half million. So you've got to go up 10 times, and that's the mission we're on. That's a revolution that we at DWP Digital are leading as we transform UK's largest IT estate.
I could describe it as large and complex in many different ways, and every day today is a day where we exchange 10 million records with 500 organizations in 22 countries. I could describe us as large and complex with the fact that we run Europe's largest virtual IP contact center. I could use any facts and figures, and you'd find us large and complex on that scale, and we're on a mission to transform it.
So what is fundamentally behind that transformation? We used to buy technology as contracts. We were at the bottom of this pyramid. And we've just gone up this stack. So we got to know the technologies that sit behind those service provider relationships. Then we said, "Well, that's not good enough. What's the purpose of these things? What services do they provide?" And so we got to understand what difference do they make to our business.
And then we went up and said, "Well, that's all very nice, but it's automating processes. I am sitting on a huge amount of data. How can I gain insight out of that data to make a real difference and reimagine those services?" So we did that.
And then we went up and said, "Hmm, there is this thing called design." And we have banned the word "and" in between IT and business. Because how could you ever describe a business by excluding technology from it?
And so we design to achieve business outcomes rather than deliver to business requirements. So if anybody wants to deliver business requirements, you're missing a trick. Believe in your potential. Look in the mirror. Ask yourself, do you believe in your potential? Because if you do, never deliver business requirements. Your job can be done by an automated robot, and I would bet can be done better than what you would do.
And so the value proposition is in standing up and adding value, designing solutions that help creatively solve business problems. And that means we're not in the business of delivering IT. We're in the business of creating products, products which help us reimagine customer experiences, which help us make our business far more efficient and effective, and which help us improve outcomes for society.
Now, I'm not a commercial business. If I was a commercial business, it would be to not see competition in the rearview mirror, which would be the case for several of you. So that's what we are doing, and we are on a journey.
This has completely changed how we work. I get people to walk up to us and say, "Well, surely you've made some mistakes." Of course, we fail every day. But when I'm on release 95 of something, I've dealt with all those failures that you're going to find. And so when every fortnight we come up with a new release, very quickly you mature a product, and that is one of the secrets of our success. And it's the obvious thing to say, so I will say that.
We are only as good as the talent in our teams. And this is really, really difficult to crack because, let's just be honest, it's a rare group to meet where you can raise your hand, where so many people raise their hand to say they're hands-on and techy.
I am rare as a chief information officer of a large organization who is hands-on and techy. It's very rare to find people who are hands-on and techy. In our business, in the tech business, we very often see new technologies come in every three to five years. You know that.
And what happens? Very often people are lazy. They know yesterday's technologies and don't know today's technologies. And very often they are in management roles, in senior management roles, and they are the hurdle to making transformation happen.
And so work with colleagues who know their craft, and that doesn't just mean the team around you. Choose leaders who know their craft because you're going to have more fun reimagining what can be done with technology.
So having said all that, I take credit for Diptesh's work. Diptesh is a rebel with a cause. And so without further ado, Diptesh. Welcome.
Diptesh Patel
Thank you very much. Thank you, everyone. Thank you.
I'm often asked, what is hybrid cloud? And what does it mean to DWP? And what we've done is we've brought together all of our traditional hosting capabilities with some new technology.
So we've got traditional data centers with heritage technologies. Unsurprisingly, we still have mainframe. We've got our own private cloud, and we've also got, as you'd expect, hyperscale public cloud provision.
And so what we're doing is we're bringing this together to help our colleagues within different teams, product development teams, to look at how they can start their journey in transforming their applications through refactoring, re-architecting, to move to our target state of moving to a cloud-native service solution.
And we've been doing that for some time. And actually, the tooling, the technical aspects, that's one part of it. But what we've really learned is, and which I'm going to share today, is some of the cultural changes that we need to, or have had to, make and we're going to have to still continue to do.
So we are moving and adopting an approach where we're moving away from these silos, these autonomous teams, and starting to really focus on business outcomes. As Mayank highlighted earlier, we touch every person's life. So we've got pensions, we've got children, we've got health. And so we're working in a way to be able to really integrate in with each of those areas and create that culture and build that culture of DevOps culture.
So we're looking at end-to-end reuse, join-up, rather than different siloed functions. And that's really the key for us to adapt and move to a more agile, more DevOps culture, DevOps-focused approach.
So working across different teams, we've used dynamism. There's a lot of hard work that needs to go into all of the transformational work we're doing. And I think some of the things I'd like to share that we've learned is around we are still operating in a mode where we have two different types of service.
So our traditional agile DevOps teams are really focused on an agile delivery rather than perhaps looking at some of our traditional ITSM services. So our mixture of infrastructure means that we have two different modes of operation. And as time goes on, we'll start to look at how we can transform from moving to that ITSM approach, that waterfall approach, into agile. And we're doing that, and that's iterating and iterating and iterating.
And from that, and I'm sure many of you may have felt this as well, we have lots of challenges back around working in our operations team. It's around how can we deliver in a continuous integration, continuous development, when you're going to be impacted by having to create change controls and nodes?
But it's something that we're learning, and we're having to change the culture to be able to move to that model of agile delivery.
We're also introducing site reliability engineering to then again help to ensure that live service, or as a world-class service, is at the heart of everything we do. So rather than pushing things straight from pre-production into production, we're helping that journey through the introduction of services such as site reliability engineering.
DevSecOps is emerging now. Security is absolutely key to the way that we deliver. The sheer enormity of the services we are delivering across the department, as Mayank highlighted, it's absolutely important and critical that we remain a reliable and secure operation.
So I just mentioned world-class service, and what we are doing is, again, to focus that culture, is to deliver guardrails, provide that security, and help our DevOps teams to be able to integrate and consume and focus on delivering a lot quicker.
So we're doing that through some technical services. So everything we develop as a product has the word "secure" in it, so secure compute, secure developer tooling, secure access. And we're doing that to enable our DevOps teams to be able to quickly be able to deliver some real good business outcomes and value.
With that in mind, we also look at how we create the application observability, so making sure we've got the logging, the monitoring to ensure that our services remain world-class and highly available.
So training and development, collaboration, all of this is really key in building that culture. So we have a fantastic team that are on this journey of moving from a traditional approach, re-engineering, re-architecting applications.
So we're really encouraging collaboration. We have digital hubs across the UK. And so we've gone through that, breaking those silos by enabling that constant communication through using chat, documentation as code, using different shared tooling, wikis, social intranet. So a problem being experienced in one development team in one hub can be shared, and we can deliver that business outcome through building that collaboration.
So as Mayank mentioned, one thing that we've really learned, and it's really important for you guys out there, is that the DevOps culture is not just about doing this from a leadership perspective. We're absolutely supporting and working with our DevOps teams right at the engineering stage to say, "How can you help to make the change? How can we really adapt and move forward in building some really good business outcomes?"
So one thing we've learned is we need to be able to support our teams right at the grassroots to be able to develop much more quicker and be able to develop those services for citizens into making sure that our customers, our external citizens, have the best, most reliable service.
So we have built an engineering community, and that community is a practice where we invite colleagues all across our locations to really collaborate. So rather than sharing emails, we bring everyone together, and we start to focus on how we can all share and learn those skills.
So internal events, unconferences. We don't sit there just talking and presenting. We do actually get teams together and look at how we can do hackathons at these events to help teams to be able to learn from their experiences and try and resolve some of their challenges that they have. And we do that regularly across all of our locations.
So something common that we always hear is that this isn't a job title. DevOps is a culture, and we are, as we go through this journey, working with colleagues to say, "We can change. We've done this before for a number of years. We used to just configure a data center box, a server in a data center. But actually, what can we do different?"
It's not about going into a DC now, plugging in a cable. You can do this remotely. And that collaboration, and what I've highlighted, is helping our own teams to be able to change and to adapt, to be able to believe that we can move into an agile DevOps culture and have a better experience.
Thank you, Mayank.
Mayank Prakash
Nice. Don't be fooled by the calm exterior of Diptesh. He is trouble. He used to be trouble for the Home Office before he joined us.
So how do we distinguish people who we want to work with? Because we like troublemakers, we like the rebel in people, we like people who want to break the norm and want to achieve their potential.
How do you find those who really create value versus those who create noise within the organization? And here are the four things we look for.
So are you curious? Is your curiosity going to take you to new places? Are you going to pursue that with a hunger?
Are you outcome-focused? Remember I said earlier, if you want to deliver requirements, your job will be done by a robot. So are you outcome-focused rather than delivering requirements?
Can you design things? Do you like creating things? Do you like solving problems? Can you design things?
And are you going to be able to lead together as a team to deliver something? Because we do like hiring really smart people, and I am sure I speak not just for myself, but for colleagues. We get a buzz out of working with smart people, but we get a buzz out of working with smart people. And so there is no room for individual contributors because together we drive the impact and the outcome that we drive.
So with that, we've got a few minutes, and we deliberately set that up so that you can ask questions. If they're difficult ones, they're for Dip. If they're easy ones, I'll take them. And so let's open the floor up. What would you like to talk about? We can talk about anything that you'd like.
Who'd like to get us started?
Q&A
Q: Thanks for a great presentation. I've really enjoyed your journey from where you were to where you are today.
I wanted to ask about the ops side of DevOps. A lot of the presentations this week have focused about value creation from idea to deployed feature. How have you involved your operations engineers and the people that deal with issues in the cultural change that you've done at DWP?
A: Sure. Thanks. So as I indicated, we are on a journey, and one of the things we're doing is, as I talked about the bimodal operation, is to really integrate with our existing service operations team.
So the traditional approach of a service design and transition, we're supplementing that and changing that, augmenting it to include, effectively, a readiness assessment to hand over from non-production to our site reliability engineering function.
So we're building that function to really focus on servicing production operations, so moving services from pre-production into production. And that really will provide some of the additional checks and balances to make sure that we are delivering that world-class service.
So rather than our devs taking it straight through, working hand-in-hand, so it's not a case of another operations team embedded somewhere else. They'll actually coexist, work in those hubs, those SREs, with the development teams to then take it into operations. And we've started that now. We've been doing it for about a few months, and we're now iterating that as we go across different journeys.
A: This is one of the difficult challenges. It's easy to do DevOps in a contained environment and see the upside of that velocity and agility. It's far more difficult to do it on an enterprise scale. Certainly, when you are protecting a nation state, it's far more difficult to do that at that scale.
And we want to have our cake and eat it, too. And so we want to have the agility to have fortnightly releases out and sometimes more frequent. We have now increased the frequency to weekly for several of our platforms and services. But on the other hand, we have no chance of taking them down or making a mistake.
And we balance that by increasing automation and by making sure that colleagues stay embedded and are joined up as a team. But there may be segregation of duties in that embedded team so that we have enough balance between agility and getting it right.
And you can see it, you can see the result. Three years back, if you wanted to get a pension statement out of us, it would take two months because you would write to us or call us, and don't laugh, it's true. And two months was a good two months.
And here's why it was two months, by the way. You would write or call us to say you want a pension statement. We would say, "Well, really, we are not going to give it to you because you may not be who you say you are." So we'd write back to your address that we've got with a six-digit code we'd send you.
You would then write to us back or call us with that code to say, "I am really who I said I am. I do want it," or, "I don't want it." And let's presume in all of this, no post got lost, which doesn't happen. And so you then call us back, and then we'd put you in a queue, which is a backlog, which at times was six weeks to post this back out to you.
So two months was a good deal.
Now, eight million people in UK are using UK's fastest-growing digital financial service, which is Check Your State Pension online. On average, it takes 61 seconds to do it online, and 94% of the people who use it love it. They don't like it. They love it. We want to create products and services that people love.
So that's what we do.
Who else would like to ask questions? Yeah.
Q: Hello, this is a question for Dip. You mentioned about you've set out some technology governance guardrails...
A: Yes.
Q: ...to keep your teams in check. How do you manage the no-blame culture? Or do you shout at people? How do you actually manage that?
A: So probably what I didn't mention is that we don't want to stifle innovation, and that is one of the challenges around DevOps teams or devs want to have access to all the tooling, and we're not doing that. We're providing those guardrails to help folks to be able to onboard quicker and also provide that secure, that governance, that cost control, and management.
But when you start to look at the size of all of the services we've got, it's quite extensive. So by providing those guardrails with some services, it absolutely does get contentious at points.
So someone will say, "Why are you limiting my VPC to instances?" Or, "Why are you putting in these constraints?" And we frequently have engineering teams on Slack disagreeing and saying, "Well, actually, if we're providing that standard shared service, why does it need to be constrained?"
But we're working through it, so we're capturing all that demand, and as we start to get to these conferences and collaborate, we do start to iterate it. We're encouraging more, let's just say, we're being a bit more flexible and, for example, creating more sandbox environments for teams to be able to embark on their own testing and innovation so they can come back and say, "Look, we've proven this out. Actually, we can increase this, or we can change this and iterate it."
So rather than providing a central guardrail, we're taking and collaborating with all of the teams, the developers teams. So if they say, "Actually, we could reuse this, this would be really, really valuable," we'll take it, we'll adapt it, and then we'll keep iterating.
So it is an interesting question because we do have that frequent conversations. Our Slack board's full of disagreement, but hey, that's part of the journey. And that's welcome, right?
A: If you get two techies to agree on the same thing, then you probably don't have the right talent. They need to have three opinions at least.
And by the way, this starts from the top, right? Because let's just be honest, we are going to make mistakes if we go at the pace at which we are going, and we've got a choice.
We personally, in DWP Digital, are inspired by Red Arrows, the formation flying of Red Arrows. And the way they do it, to cut a very long story short, is after each formation flying, they sit down and have a good, honest debrief about the 40 things they didn't do well.
And then they get some of the people who have watched the formation flying come in and say how amazing was their formation. Because nobody noticed the 40 things they got wrong, and they get amazing feedback, but they pay attention to learning because their lives depend on that learning.
The people we serve are people we deeply care about in our lives, so we care about getting this right, and it has to start from the top. Not a single person will get fired in the organization because they made a mistake as long as they learn.
But let's also be honest, we don't live in romantic worlds. I have lots of colleagues who I don't choose to work with. Like every organization, you don't choose colleagues you work with.
And so it is really, really important that you provide that cover for your team. Stand up, stand tall, so when they make mistakes, you create space for them to learn rather than be blamed.
Who'd like to go next? There.
And we've got time for one more question after this, so I'll queue that question up for whosoever raises their hand. Please.
Q: So are you hiring right now? And if so, which roles and why?
A: Ah. Are we hiring right now? Yes, we are hiring right now. We are always hiring, and like any good large organization, we are constantly talent hunting.
What will give you a little bit of feel for us, we are an organization of several thousand people in DWP Digital, and we have, over the last 18 months, hired 400 people. We are going to hire another 1,300 people in the next... Well, as soon as we can hire them.
The reason why we don't put a timeline on it is because this is not a waterfall project that we're going to hit a milestone on. We are going to be very tight about who comes into the organization. We typically get 12 applicants for every role that we want to advertise.
But it is difficult to get in. Most people who apply are unsuccessful because we look for skills. We will look for skills, but we will also look for values. You have to fit in both ends of the spectrum to join us.
But we are inspired by the people who join us. We love the rebel and the troublemakers who join us, and we are inspired by what we can do together.
Did that answer your question, or did I duck it?
Q: Yeah.
A: You imagine a role and we are hiring. We are hiring people who run very large parts of a country. We are hiring people who are developers. You pick a technology, and we probably use it. You pick a tech stack, and we probably have it. Yeah.
A: We've got a number of developer roles recruiting at the moment, so feel free to have a look.
A: Yeah. And that's the fascinating thing. We love experimenting with new technologies, and you combine new technologies with our scale and ambition, and you combine that with our purpose, and you get a magical mixture, which brings us to work.
All right. I'll take one more question since I promised.
Or maybe I'll keep my promise to Gene and team and wrap up because I'm out of time. So thank you very much. Thank you for coming. Thank you, Dip. Thanks.