Stefania Chaplin

Today our session is "From Telco to TechCo: Striking a Balance Between Security, Innovation, and Productivity."

Hi, I'm Stefania Chaplin from GitLab, and I am going to be talking with Henry about some of these topics. A bit of introductions: I can start. I'm a solutions architect at GitLab specializing in DevSecOps, working with countless organizations through their journeys and all the challenges and the good, the bad, and the productive. Before that, I was a Python developer as well.

Henry, maybe if you want to say a little bit about yourself and what you're up to at VMO2.

Henry Tze

Yep. My name's Henry Tze. I'm a lead cloud security engineer in VMO2. I've been with the company for three years now. When I first started my role in VMO2, I was a DevOps engineer, just like many of you perhaps. I loved infrastructure and I hated security. But that's in my title now. It gets more powerful when you get that in your title.

I'll tell you a little bit more story about my life in Virgin Media. The first time I went live with one of our products, a front-end hosting product, nothing special -- CloudFront, S3, whatever -- done. But it took me nine months to get one product to go live in Virgin Media, because they were using data center security to apply to cloud security at that time.

Before I went live, I needed to have a 20-page design document: everything I do, diagrams, what services I use, what's the data in and out. Twenty pages. I needed to send that to a board of security. They had seven teams in security teams that would have a board during their weekly meetings. They needed to allocate how many security teams needed to help me through my process to go live with my product.

But at the end of the day, I've done all my CI/CD pipeline, security scan, deploy infrastructure, environment progression, dev, stage, production -- everything is there. I just don't know what else they can pick on. But of course, because it's Virgin Media, they have a process. There are a lot of people wasting FTE cost anyway. But anyway, seven teams there.

So that's kind of my role. I love security -- well, I love to actually comply with security, but not really into that security. I'm different. I'm coming from a very engineering background. So I moved two years ago. My boss, Stefano, told me: come and join the pirate. He created a digital security team, which is not a traditional security team like I had been working with before in my previous role.

So I said, get me on that crew, because I want to really change the way that developers and engineers look at the way that security teams are. I've been through the pain. I've been through the hell, and I want to make a difference. If I want to make a difference, telco is a very good company to make a difference in because there's so much process.

That's how I became a lead security engineer. I do CI/CD pipeline, infrastructure as code, everything as code, and now it's going to be more platform engineering, because digital security actually has more influence into the whole company instead of just helping a digital department. Now my team is actually supporting 13 different departments.

Stefania Chaplin

It's really interesting in terms of your journey to get where you are, because I think some of the best security people are people who have been on the other side, and then they're like, okay, this is all very painful; let me see how I can improve the system.

You discussed the digital security team and then advancing that into platform engineering. I think you touched on this a little bit with your 20 pages of process, but what are some of the challenges that you've encountered going from a traditional telecommunications company to more of a technology-driven organization?

Henry Tze

The first challenge: there are a lot of people in enterprise. There are many, many different specialists. We use a tool called Remedy -- BMC Remedy. Do you guys know Remedy? You know as well?

Stefania Chaplin

I know of it, yeah.

Henry Tze

That's so much pain. I don't know how people actually create categories in Remedy tickets. On the back of the Remedy ticket, you need to declare what you want. It could be a firewall rule change, I want that permission -- everything is going to be a ticket. On the back of it, there will be approvers. Depending on how critical your ticket is, some tickets need to go through three or five approvers.

The people who need to get involved in every single ticket is immense. Every single ticket that has a ticket catalog has a column called SLA. Every single ticket has its own SLA. It's from three days up to five days. If I just want to make a firewall rule change in a data center, I need to go through that. That's not just three days. Sometimes if I mistakenly fill in some of the forms incorrectly, I get rejected -- not on the first day. It will be at the end of the SLA when they actually have time to look at my ticket.

So that is a challenge: there are so many people involved and so many, I would say, redundant processes in place that slow people down when they look at productivity. I change the firewall rules; I need network engineers, I need security engineers, I need my line manager. What else do I need for the firewall rules?

At the end of the day, if everything is approved, there will be an engineer with some permissions who goes to a console: okay, add it. The outcome is literally two seconds. But I need to have five days of SLA and process. Just changing one thing -- why does it need to be so complicated?

That's one of the biggest challenges I have in telco. Of course, my seven security teams, that's another one. Security teams have always been a blocker, and they just don't really know about cloud. I know we're quite a big company. You think we are multi-cloud and on-prem, hybrid, whatever fancy name we have, but we still have a side of the people who are working heavily in data centers. Virgin Media started in 2007, so to be fair, about 20 years of data center knowledge, experience, and everything. They've got a process for everything.

So that's most of my challenges: SLA processes, too many people involved in so many things, and at the end of the day everything needs specialists to complete. I don't know why. You need to go first line, second line, and they don't know what, find a specialist -- how many chains of people to get one thing done. This is why I really hate it, but I want to try to fix it.

Stefania Chaplin

It sounds like definitely a lot of processes, a lot of approvals, a lot of bureaucracy, a lot of different documents. You mentioned security. How do you go about striking a balance between innovation and security on your transformational journey?

Henry Tze

Right now, you need to switch face on security, not the engineers. What I have done for the last two or three years now is establish the trust. Our team has been very good at taking on project after project. We rolled out an Okta project to all VMO2 internal users. It was about 40,000 users to be onboarded.

We actually have something embarrassing because it's Virgin Media O2, right? Which means Virgin Media and O2 with the merger about one and a half years ago. Things got very complicated. To get one identity platform, we actually had about 11 different ADs.

Stefania Chaplin

Wow.

Henry Tze

Across multiple data centers, multiple companies. There's a program that we run just to consolidate all of them into one Okta. That's a big piece. We're taking on a lot of stuff which is organization-wide to make the difference, simplify everything we can, centralize control, and give them a single pane of glass on everything they really, really wanted.

The way that we strike balance is what I would lead with: everything as code. I think that's just a common language. In presentations, English is a single language, but in engineering terms it's going to be everything as code. It doesn't matter which country you're in. The code will be the same. Terraform is Terraform. Maybe comments will be a different language, but the code itself, it doesn't matter which language you use; it's quite universal because it's widely adopted across the industry.

That is one thing. Balance is meant to be broken anyway, from my point of view. When you're thinking about balance between innovation, security, and productivity, in innovation you're always thinking of the DevOps engineers, the developers. They're making the features and making something happen for the company, building features for the company to go live with their feature, grab more revenue, get more customers in.

But security, if they only got one role, security only got one role: security. But security team wants developers to do compliance, security, and everything. I think that's not fair for all developers and engineers because there's so much stuff. For a security professional to be professional, it probably takes them five years, three years intense training, security and compliance training. There's so much to learn to become a security professional.

But for developers, I just want to go live with my features, to be fair. I just want to go live with my features. Why do you ask me to do all the compliance and security?

So how I strike the balance is I get the security team to be more hard-working than the developers. Security team needs to be application developers, they need to be infrastructure engineers, they need to be cloud engineers as well. If you show me your hundred tick boxes of your checklist of compliance and you want me to build it as a developer, but security team cannot show me that you can build it as well, that's not fair.

So I get the best people from AppSec, the best people from infrastructure into my team. We show them that we can build good infrastructure, good applications. Then once we find the good foundation, we share that -- share the love to all our developers, forming a central place, kind of a library of everything as code. Everything we build, we are not selfish. We give it to you.

Because I always find one thing about innovation. If you've been tasked from your stakeholders saying, build that feature out, the first thing you go for is your research phase. You go to Google or whichever cloud you're working with. The primary cloud we're using is GCP, so we rely heavily on the GCP documentation. The first thing you do, you search.

But when you do research, the example to get you started is very general, because it's supposed to be like that. Everyone's supposed to get started with the general documentation. But when we come into a telco context, we want compliance, we want security, we want specific ways to do things. It's impossible. There's no Stack Overflow that has that kind of detail. There's no template out there that can contain everything that we need.

So we as a security team build all of it. It doesn't matter what you build, from a bucket to a virtual machine, all the way to a Kubernetes cluster, to even CI/CD pipeline, how to build container, how to build a cluster. Every single part, we have security involved. So everything delivered to you is a better baseline. Whenever you do your innovation, the first phase research, instead of going to official documentation, go to our documentation. You can get all the pipelines or the blueprints that you want for getting started.

One thing I found out is really hard: we have 13 departments. They can vary from front end, middleware, back end, data, advanced analytics, data science. They have so many use cases. The security team is only 30 people, but we've got 13 departments to manage, and more departments are getting onboarded. The only way we can supply is baseline templates and ways of working that enable you to do things, giving you all the tooling that you want.

We are starting creating a platform, becoming platform engineering as well, not just DevOps security. On the platform engineering side, GitLab is one DevOps platform, but what we call it is: we give our end users a solid platform to develop on.

I've got five different tools in my tech stack because I find tool fatigue is very hard for everyone. If you join a company and there are 32 tools you need to learn about, that is going to be very hard for your onboarding. It's not going to get you anywhere productive within your first month.

So there are five tools that we use internally. I call it "GEGVO." The first G is GCP, our primary cloud. E stands for one of our self-service platforms; there's a tool called env0. It's a self-service platform. You can choose a template that you want. It's an infrastructure vending machine. We've got so many templates already embedded into it. Whenever you want a GCP project or AWS account, whenever you want to create new products, new pipeline, everything, it's a vending machine.

The next G is GitLab. We use it as source control and also all the CI/CD and security scanning capability. V stands for secret management, which is Vault. We use Vault all the time. I don't think any application in the world can live without secret management nowadays. The final is O, which is Okta.

Using these five tools, I am able to provide a platform for any product to go live. You can go end to end by yourself. We do a lot of platform engineering, enabling self-service automation.

I actually created automation that took our whole team, because there are people in the team that are no longer relevant for digital transformation. We need to accept that. If you have 1,000 people in your company, I would say easily you can chop half of them out through automation. I know it's cruel, but telco is cruel.

If we don't change from telco to techco, telco will die. Every time you do a renewal on your contract, you negotiate a lower price. You scare us by saying, "BT is doing a better deal," and then we need to do the retention discount. So telco, the way that we are doing it, is dying. We need to switch to a technology company. How do we make use of customer data? How can we provide a better customer experience through analytics and so on? We need to transform or we will die. It's quite crucial, but there's definitely very good motivation for us to become better in DevOps in Virgin Media O2.

Stefania Chaplin

It's really interesting, some of the topics you've talked about around innovation. If anyone was in the seminar yesterday -- seems so far away -- this is one of the things we talked about. Within your team, if you wanted to get a team onboarded onto these five tools, it would take something like two weeks, and then you got it down to 20 minutes. If you think about the amazing increase in efficiency, that's definitely going to have positive effects.

In terms of your platform, you've mentioned using GitLab for DevSecOps. Can you talk a little bit about your experience with GitLab and also how it's contributed to your successes?

Henry Tze

Cool. One of Gene Kim's examples yesterday, if you guys were in, was the two-hospital example. Everyone actually started with the same equipment. It's the way they're interpreting and using the tools that are different.

We adopted GitLab three years ago. Because it's telco, we had to purchase the self-managed instance to put in a data center to do our first version of GitLab. That was horrendous. But people were using it. They liked it. They started getting used to the tools, having CI/CD on that. Then we moved to GitLab.com two years ago, when we got budget and had so much momentum already. People really wanted to use CI/CD.

The first thing I like about GitLab is just a source control system, to be fair. That's the first thing that GitLab really does: source control. Because I do love everything as code. Everything I build is code. The only way I can share my knowledge and everything is as code. So choosing GitLab as a source control: fantastic. I don't need to manage anything on GitLab.com.

For me, I do transparency as well. I love one of the principles you guys have in GitLab: transparency. Every single bit of code builds up the whole infrastructure in our multi-cloud estate as code. Everyone who joins the company will have access to the code. We don't say, "Guys, this is top-secret project, NDA project, you cannot see it." We don't do that because GitLab provides that transparency. People can search about the code and wiki and all of that.

Nowadays, everyone starting onboarded: everything is code. It doesn't matter if you have experience in Terraform, Java, Go, and everything. You can get started right away. There are examples on pipeline, how to build code, deploy to Go applications, containers, serverless applications. They're all there. GitLab has become a way that I can shelve every single thing I've done as a library.

That's only a starter, right? Once you have code, what do you do with code? It's security scanning. GitLab has quite a few predefined templates for security scanning. It doesn't matter whether it's infrastructure scan, application scan, dependencies. I don't want to sell too much, but all the scanning tools.

In a way, because I need to work with so many departments -- departments can vary from 15 people all the way up to 500 people, and the skill set is totally different. I have people who really want to go to a new world with us from the data center, but they don't even know Git or source control. If I ask them to run the pipeline, they don't even know Git.

So I use GitLab to bridge the gap. If people have passion and are willing to learn, there is always a WIP project for them to use in our GitLab. Internally, I do a lot of video recording every single step. I try to form that into a tutorial, into three minutes. Documentation is quite good on GitLab; they have wiki and all of that. I'm able to use a lot of GitLab's tooling because they've been doing DevOps for a long time now. If I'm a new company joining, they already have a lot of tools already in place; why don't I just utilize it? The fewer tools, the better.

I'm not trying to give them all the fantastic tools. I know we can buy so many different tools which are really good at specific stuff, but with GitLab, we're actually teaching the concept of DevOps, the culture of DevOps. You need to get people to start learning and be willing to accept DevOps first before you can go to a different level.

I think with GitLab nowadays, when we're using all the 600 users now, they are kind of oriented in the DevOps ways of working. Everything they do in GitLab, they first think: I'm doing the DevOps stuff. Everything they do in GitLab, they feel some pride on that.

For a new starter, as you mentioned, historically it was two weeks to onboard a new department into our company. Of course there is a lot of paperwork: cost center, cost charge, everything like a legacy company. Once they get onto technical onboarding, I know I'm doing everything as code and two years ago, but it was scattered around. This is a learning curve for me. First I need to understand the code. I need to build a department out.

Last year, I formed a project called DOP -- not the Italian potato tomato, but the actual DOP, Department Onboarding Platform. I took my learning from two years ago and mushed them all together into one single pipeline.

Now I was able to do a technical onboarding for every single new department from two weeks, which required five engineers, down to one engineer. They don't even need to be technical. Just fill in the form, and then you have a new department, and all five products I just mentioned will all have a slice of the product for the department to fully control. Then we will apply guardrails on top. Of course, we need to provide that full autonomy for them to go for productivity.

So that's how we onboard: everything as code, a platform that onboards them easily, giving them the best experience we can offer through automation.

Stefania Chaplin

It's really interesting, some of the things you've touched on, because at GitLab we have our values, of which transparency is one of them. We talk about: GitLab is a tool, you can't buy DevOps; it doesn't work like that. It's very much about the cultural change and about having these reusable components we call inner source, and having a collaborative environment so that when you see the process, it's like: can I make that better? How can I automate it? How can I improve it?

I'm a bit conscious of time. I've got two final questions, and also if anyone else has any questions, please come find us at the GitLab stand in the expo hall. We'll be here for the next two days.

When you look back on your journey, is there anything you wish you had known when you started, and how would that knowledge have affected your approach?

Henry Tze

It's a skill gap, really the skill gap. We've been hiring crazy. We're telco; technology needs a lot of people to build. We've been through a frenzy of hiring. We hire contractors, we hire full-time.

What was the question again? Sorry.

Stefania Chaplin

Looking back on your journey, is there anything you wish you had known when you started?

Henry Tze

The skill gap -- trying to bridge the skill gap. I don't know that the skill gap is something I really tried to bridge from day one. I wish that I knew all the departments have different ways of working as well. If you don't actually go and understand each department, how they work, it's not going to succeed.

You need to spend a lot of time not just doing the drawing on the top organization level and bringing it down waterfall style. You need to talk to every single department and get them a forum every single biweekly or monthly. I actually have 13 departments now. I try to delegate some of the stuff for my other colleagues to do, but I actually have a monthly call with every single department to listen to their pain points.

When I started, I didn't know the demand. Now I understand the demand. Some of the engineers are really passionate -- not crying in front of me, but they tell me all the pain. Like: guys, the platform is not really good at that point. Can you put somebody to improve the service capability? The security scan is not scanning or just fails. What can we do on that?

Also, the whole cloud perimeter now in GCP: they have a really good service control called VPC Service Controls. If you're using GCP and not using VPC Service Controls, I suggest you start investigating that, because as soon as I put in that control on the top of three, VPC Service Controls is a perimeter that guards every single GCP API that you can think of in their platform.

Once I put it on there, all my data team is happy. When you deploy into the cloud, you need credential access key, secret key, service account key. Those get leaked, right? You need to accept that that's a risk; it's going to get leaked. So with that capability on the perimeter, I don't care that I give you my service account key. It doesn't work outside the perimeter or the conditions that we set up.

So the key now -- I'm not going to throw all my keys, but anyway...

Stefania Chaplin

Everyone gets Henry's key. Yeah.

Henry Tze

Yeah, yeah. It's not going to be usable for you if you're not inside certain conditions that we defined. I wish I knew all the features that the cloud, GitLab, Vault, Okta -- all the best stuff they actually have on day one, because I needed to go through the discovery to find out all the best features that I can take and apply to my end users. That was the hardest bit.

But the thing is, every single platform has thousands of features. How can you find a relevant feature, apply it to a use case for your company, or down to a department, or even further down to a product team? That is the hardest bit.

Stefania Chaplin

I'm conscious of time. I'm going to try and slip this in. It's very topical, what you were talking about: the buzzword of the year. In terms of AI-assisted DevSecOps and generative AI, how do you see this technology playing a role in your organization, and how do you envision integrating with GitLab?

Henry Tze

Literally, if AI is a role, I want to hire it. We try really hard: 13 departments, 600 users, or maybe more coming our way. We're trying to form a security champion program, but running a security champion program is very hard work. You need momentum, delegations, and constant feedback from the team. That is so hard work to do.

Because GitLab is releasing AI capability, one of the most popular things from my end users to the security team is: how do I fix security vulnerability? You have your fancy scan highlighting all the CVEs and everything: critical, medium, and high. How can I fix it? Because people have a knowledge gap on fixing the security vulnerability.

Using GitLab's new feature, I think it's still in beta, every single vulnerability has been highlighted. You can see "explain this vulnerability." It's my favorite AI feature.

Stefania Chaplin

We've got a couple of things, yeah. We have tried it out for security training, and then we have "explain this vulnerability." So it's like: okay, what is the vulnerability? It's the problem with this variable in this line. Here are some suggestions how to fix it, in a format that's consumable to the developers who are remediating it.

Henry Tze

Last time I used the AI-suggested explain, it gives you exactly what the vulnerability is, how to exploit the vulnerability, and also the inline code that I can copy and paste into my code. That is the way I foresee it: if AI is a role, I'll hire it as my security champion.

Because everyone in VMO is using GitLab, everyone has access to the tools to start fixing their own stuff, because I won't have time to go to 600 users. It's better to get AI to do that job, because developers want to be self-sufficient. They don't want to ask people to do things for them. They want to be self-sufficient, fix their own stuff themselves, and also have full flexibility on their platform.

So AI is going to be a big role. If everything is code, the next thing is going to be AI. I really want to invest some more time to see how I can scale myself into multiple formats.

Stefania Chaplin

Awesome. Thank you everyone for joining. Thank you so much, Henry. If you do have more questions, like I said, come find us. We'll be at the GitLab stand. Hopefully everyone has a better understanding about how to strike a balance between security, innovation, and productivity.

Henry Tze

There's one more thing. Thirty more seconds.

When the ticket was a chain of people, five or six people to approve. Now, because we are doing everything as code, if you're willing to learn -- you're an audio engineer or whatever, you're willing to learn -- we expose everything at organization level: how do we set up GCP, the guardrails, everything as code.

If you're willing to learn, you can make a change yourself. All you need to do is learn how to make a change, and then you submit a merge request, and then one of the approvers will be approving the change. That's one of the ways that I really found everything as code helped my company a lot, because now everyone can make a change instead of going through Remedy. Everyone willing to learn will be able to make a change themselves. So it cut down from a five-day SLA to possibly 15 minutes for every single change, from organization level all the way down. So that is one thing.

Stefania Chaplin

Really empowering everyone to get...

Henry Tze

Empowering everyone.