Maks Kiamos Shah

I'm Maks Kiamos Shah, and I'm from Syngenta, and I'm going to talk to you about learning how to scale DevOps in R&D agriculture.

I'm going to start to talk a little bit about my situation. I was told I'm the first life sciences presenter, and so I'm going to explain a little bit about my world, agriculture and R&D and what that's like. And I'm then going to actually focus on the use case that we wanted to present.

Actually, no. How many people here work in agriculture? If I asked this question 200 years ago, half of the room would have raised their hand. Two hundred years ago, we had 500 million farmers providing food for a billion people. Today, we have 500 million farmers providing food, feed, and fuel for seven billion people. That's a big step between the last 200 years.

Syngenta's only been around since 2000, but it's a company that has its roots back to 250 years ago. So we were kind of there at the beginning of this big change of agriculture. And it's an industry that's very much in transition.

Besides the ever-growing challenge of always trying to feed more people with less, we also have other things that have come into play that we've become more cognizant of. So adapting to climate change; the desertion of rural communities, people leaving the farms and coming into the cities; and of course, last but not least, the transition from a fossil fuel-based economy to a biological-based economy.

Our newest board member, Louise Fresco, did a really good job of describing or debunking some of what's going on in agriculture, the myths and the truths, in her interview at Le Point magazine on June 13th. I'd advise you to go look at it. It really gives a good understanding of what the reality of agriculture is today and what sustainable agriculture will mean in the future.

Back to Syngenta. Our headquarters is based in Switzerland, but we are owned by a Chinese company called ChemChina. We are 27,000 employees, and we are in 90 countries, and we have about $13.5 billion in sales last year. Most of our products, our services, and our solutions are sold through distributors, so it's a B2B2B business.

The other thing that's important to realize about agriculture, and this is not just Syngenta, this is pretty much everywhere, is that it's government-subsidized in some parts of the world and not others. North America, for example, and EMEA have a lot of government subsidies. Inside of LATAM and APAC, none at all, or practically none at all.

A little bit about me. I've been in life sciences for the last 30 years. My experience also falls nice and neatly into three decades: environmental, pharmaceuticals, and now agriculture. It's pretty obvious life sciences has been my life, and you can find me on LinkedIn if you want to find out more about agriculture, R&D, IT, and trying to apply DevOps in it.

I was asked to talk a little bit about what R&D in life sciences looks like, particularly in agriculture, because, again, it's not something that's often presented here. Most of our product life cycle, on average, lasts about nine years. From the point in time when we actually find a candidate chemical to when we actually have a registered product on the market, it's a nine-year cycle.

Obviously, one of the things that's important to us is actually failing early so that we can reduce the number of candidates to get to the right one as quickly as possible, but then also reducing the size of that cycle or shortening it. We have a couple of constraints. Most crops only grow once a year and have one growth season. The other thing is pest invasion is not a predictable thing. It happens sometimes, but not always.

In other words, you kind of have to have the right conditions to run an experiment. If you don't, you have to wait a whole year. That adds a year to your development cycle. That means a year later to actually going onto the market.

We have a chief information and digital officer, and we're about 750 people in the IT and digital space. I'm the domain head for product development. A domain head is kind of like a tribe lead. I'm focused on a particular area.

My team and I mostly work with biologists who do research. We design the IT and digital tools they use to do their research with, and there's about 2,500 of them. There's about eight of us. And we have no internal DevOps. It is all outsourced. For every one of my people, there's 10 over on the supplier side. All together, we have a community of about 100 people working on these solutions for scientists to do research with.

We mostly work with system integrators and some specialists. We have four system integrators and some specialist suppliers. As of 2018, all of our systems are sitting on the cloud. Our systems landscape consists of about 50 or so applications or solutions. Some of them are off the shelf, some of them are software as a service, some are platform as a service, and there's a lot of custom build as well.

We started our Agile DevOps journey in 2014, and we've been progressing it ever since. We started off with what we like to call the matrix. Basically, we had tribe leads, but no tribes underneath them. Tribe leads were taking their resources out of the services and working with people out of the services, but they were a dedicated group, so we were emulating a little bit of a tribe.

But we were still very much, up until 2014, running a very Waterfall, ITIL-style organization. It's been a few years since that, but it's been a great ride.

The story I'm going to talk to you about today, or the use case I'm going to talk to you about today, has to do with R&D portfolio management. The question that came to us, or the problem we were trying to solve, was we wanted a rolling monthly view of expected registrations and launches over the next three years for every product combination in every country.

What we originally had was something quite different. We had a setup where we could run one report every six months, and it was a very laborious process. It was obviously static. It was a printed report. People got it, and you waited until the next six months to get the update for it.

On the IT side, we had a couple of issues. One was obviously we were missing some capability, visualization and business analytics. But we couldn't just add something on to actually make it all work, because the key platform that was holding all our portfolio information and that we were using to manage our portfolio data was actually a problem. It was old. It was an off-the-shelf solution; however, it was customized to death. There's just no other way to explain it. It had become very fragile and obviously obsolete.

It was so customized and so fragile that we basically avoided making any upgrades or any changes. We would do a change every six months, but that was really under duress. We really had to. Otherwise, we avoided it.

Before I talk in more detail about what we ended up doing, I just wanted to give a little bit of context. I inherited the project. It was not one that I developed. At the same time, I was running my own attempts at DevOps delivery in other areas of R&D as well. There were about three replacement projects we were doing there.

This project was actually a revelation for me, and that's why I wanted to present it. It came over, I got it in the middle of phase two, where they were actually already starting the replacement. I didn't change a thing, obviously, for two reasons. One, it was near completion of phase two, so you're not going to go in. But what I found, or what I inherited, was actually something that was working really well.

They had done quite a few things that were different in this project. They had started off with an Agile delivery, and then they had found a supplier who already had a pipeline in place that was actually specific for the key system we wanted to implement, which is Planisware. They had basically piggybacked on that. It was great. We left it as is. We didn't touch a thing.

A couple of things happened, though, by the end of the delivery of this first phase. They started making different kinds of decisions. Not all the capabilities could release, and as an old Waterfall organization, or an ex-Waterfall organization, you're sometimes faced with these choices: do we actually delay the launch and put in the capability that was requested, or do we not delay the launch and just leave that capability to be delivered later?

Lo and behold, the project team with the product owner and our partners in R&D said, "No, we go live with what we have. The rest we'll deliver through." It was the first time that had actually happened, and it was a revelation for everyone that we could actually really work this way, and that there was confidence to continue working this way.

The other thing at that point was that I was faced with, do we hand this over to the supplier team that we always use or not? And we decided, no, actually, we're not going to do that. What we're going to do this time is we're going to use the same team and the same supplier and just give them the operational management. So we created the run-build scenario using the supplier who actually did the dev, and we've never looked back since.

Just to talk a little bit of particularities: when I say we run a DevOps pipeline or we run a DevOps operation, we run a DevOps operation up to a point. Our infrastructure and our security is still handled by a central team, and it still runs Waterfall and ITIL. But pretty much up until that, we've gone fully DevOps on this.

What we ended up doing was we kept actually the same software we had from the beginning. We had started with Planisware. We didn't do a good job the first time eight years ago, but that didn't mean that the tool wasn't a good tool. We know it's actually a decent system for obvious reasons. It's the standard. If you're in R&D portfolio management in life sciences, this is the tool that everyone uses.

So we pretty much kept that as it is, but we just went for a newer upgraded version, and this time we didn't customize anything, we just configured as much as only. We then added a couple of other things to it. We put in a data warehouse so that we could actually archive old views and versions of our portfolio. And then on top of that, we put QlikView so we could actually start doing some basic analytics and visualization.

All that is managed by two suppliers. And then, of course, it sits on our Amazon cloud, and that's managed by us, or a team like mine. This is one of our heaviest integrated applications or platforms. We have about 11 web services and integrations going from cloud to cloud and handled by different suppliers as well. So it's quite complex as an ecosystem in terms of people working together, but we are able to make it work.

What was the value that we created? We achieved our goals. We were asked to create transparency around registration and launching schedules over the next three years for all our countries, for all our product combinations, and we did that, and everyone was really satisfied.

But it wasn't just achieving the goal. It was the fact that we realized something that hadn't happened before, in the sense that, as the quote says, we got greater clarity on data that we didn't have at that level. And the bonus was actually that this was rolling. You could pick up and look at this every two weeks if you wanted to. You could check back and you were working with real data. It was live, it was real-time data and real-time decisions, and that was quite transformative.

Since then, we've added a registration module. We've actually also hooked up other systems. So now we can actually connect; we have total line of sight of what we've planned in the portfolio to actually execute and what actually lands in the market when. That allows us to make decisions with confidence that we weren't able to make before. Some of these decisions are in the millions.

There were obviously some IT benefits. That's not why we did this, but we realized all of a sudden we actually had a very convenient way of moving over to DevOps, even though we were not a company that was able to actually hire developers and bring it in-house. And the benefits were quite staggering.

We had 50% improvement in quality and in the speed of delivery. Actually, in the speed of delivery, it was more. We had gone from releasing every six months to now being able to release every six weeks. And the quality was up. We had very few redos. Our redos are now below 5%, which is quite impressive.

The other thing that was interesting was the 30% reduction in operational tasks and manual operational tasks on the IT side. I had never experienced the launch of an operational-debt-free system before. This was my first one. And it changed everything for me to the point where I was going to do this everywhere.

Hence DevOps is the new black. Any project I now start, any platform I now look at in terms of operations, how I'm going to run it, I am doing my best to move it over. My team is great help in this. And what we discovered over the few we've done, so we've actually launched four or five now, is that we've got a little bit of a formula that we apply.

Remember at the beginning of the presentation, I said, "And then there were six"? Well, we found what we call the six levers, or the six things you kind of have to get right. If you've got these, then you know you're on the right track and you're guaranteed a better success.

One is the product owner. One thing that was wonderful about this project was our product owner was a single product owner. We didn't have a committee. But he also had a single view of priorities. He was able to get his entire organization behind him and be consistent in terms of what is important, what the priority of that importance is, and what value that delivers to the business.

The other thing we put together with that was we did have a small interdisciplinary team in-house that actually manages and runs the platform and makes sure that all the connections of all the different parts of the organization and the partners work.

The third thing that we did was we stuck with the same supplier team. It wasn't just good enough to stick with the same supplier. You really had to keep the same team. If you could keep the same team, then you really could create this run-build effect.

The other thing, and this was the idea of the product owner, was that because we would have these little IT meetings with our product owner, we said, "Okay, here are some design principles we need to follow." He really took them on. He worked with his organization and he established a set of about 12 principles on sustainability and simplicity.

Before any user story meeting, before any requirements discussion, before any meeting that had anything to do with what are we delivering, he would put the slide up and say, "These are our 12 principles." He would remind people what we had agreed, and then he would go into the discussion that he needed to go into in terms of prioritization.

This probably, I would say, out of the six, and I know people know this, is one of the most important levers that you can have: a solid product owner who understands how to prioritize, who understands what's more valuable than what, and who is able to bring people along with them.

The other thing, and some people call this a cheat, is that we used a DevOps supplier that already had a CI/CD pipeline. So we piggybacked on what was there. And I know it's cheating, but it worked. I recommend it if you want to try this more than anything else from ground zero and you do not have an internal capability. This is probably the easiest way to go.

The other thing that we also did and do is that 20% of the story points go to managing down operational and technical debt. No matter what the deadline is, no matter what is going on, we hold to that, and we hold to that always.

What did we find about this formula? This works actually really well in situations where we have full replacements, so we're replacing a technology. It works well for a custom build or a commercial off-the-shelf solution. It can work with outsourced suppliers, even if your hosting environment is something you manage internally. And it does work even when there's heavy integration involved.

The other thing is it works very well if you can start with an Agile project. Trying to apply this model on a Waterfall project will not work. You do need to start with Agile. The community we were working with, or the user group we were working with, was about 1,000 scientists in 120 countries. So on that scale, this worked quite well.

What I wanted to ask people today, or what help I do need, is on some other use cases I'm working with. They're combinations. What I mean by combinations is I'm trying to pull in, for example, a custom build which we already have in place, but now I'm going to complement that with a platform as a service.

The big change for me, and this sounds funny, is that I'm actually working with an internal DevOps team, which was set up in another part of our organization that I can actually partner with. So I'm going to go through my first internal DevOps team experience, even though I've been doing this for the last four years. That's a new experience for me. Being able to combine those effectively is also something I'm interested in anybody having any stories or advice on.

The other is a little bit bigger. Here we have about 15 custom-built solutions that work together and a commercial off-the-shelf situation. As well, the community size is much larger. We're looking at 3,000 people. This is a much bigger play.

This is also an application that's not only used by our internal scientists, but by scientists outside as well. That's a big change for us because most of everything we've built up until now has always been used internally for our scientists. Now we're actually going out into the real world with non-Syngenta scientists, and we rely on the fact that we're able to train people to use our applications. So this is a bit of a difference for us as well.

The other thing that also adds complexity to the mix is that we're doing partial technology replacement. There are some pieces that we're keeping because they were delivered in the last two or three years. They were designed to work on the cloud, they run well on the cloud, and they were great. But some other components of this platform -- I should have said this at the beginning -- are actually the output of a 10-year transformation program. Parts of the platform are still 20 years old, parts of it are 10 years old, and parts of it are more recent.

It's come to us with some technical debt and some operational debt as well that we're trying to clean up now that we're moving over to a DevOps model here as well. These are some of my more challenging conversions, and like I said, if anyone's out there willing to give some advice or share some stories, I'd be more than happy to.

That's me. Thanks for listening. I'm hosting a Lean Coffee this afternoon, so obviously I can't dictate the subject there. But I am free for lunch, so if anyone wants to come and join me after this, because I know I'm standing in between you and lunch, and I apologize for that, I'd be more than happy to share some ideas and talk to you. Thanks.

Q&A

Actually, we do have a couple of minutes before lunch is officially starting. Does anybody have any questions or anything that actually you'd like to start with now, or remarks?

Audience: What was your biggest challenge?

Maks Kiamos Shah: My biggest challenge? This is going to sound funny. My biggest challenge was actually trying to -- when we started this, we were one of the first teams to actually start working with Agile. So being a small Agile island in a sea of Waterfall was difficult in the sense that we were trying to do something that was going pretty much against all the processes and all the typical ways of working, even to the standard suppliers we were using to deliver this.

I was very, very fortunate. I had the support of my direct line manager at the time that said, "You know what? Try this, do this. Let's see what happens." Like I said, it became the inspiration for the rest of us. I'm one of six tribe leads, and we've all now gone out and we experiment with how we deliver DevOps for our own organizations, whether it's designing tools for chemists to do molecular modeling, to people like me who are looking at field trials and all the way through.

So it was difficult. Sort of the license to break process, I guess, was the biggest challenge. Because you talk to people and they don't understand what you mean. Everybody knew and understood Waterfall. Scientists knew and understood, my biologists knew and understood Waterfall. My biologists did not know and understand Agile, CI/CD. I was speaking in tongues. So a lot of people granted me a lot of room to maneuver and a lot of faith.

The challenge may have been I was speaking in tongues. The advantage was I had the support of a lot of people: a product owner who understood what I was talking about, and a management team that was willing to try something different.

Audience: What do you think is going to be most difficult about getting from the Agile MVP to the hybrid ops at that scale?

Maks Kiamos Shah: What was the question? Oh, go ahead.

Audience: What do you think is going to be most difficult going from the MVP to this hybrid DevOps scale?

Maks Kiamos Shah: Scale. Yeah. To be honest, I don't know. I'm actually relying on -- and we have a series of workshops coming up in the next month with the in-house DevOps team. They run one of our largest platforms. They actually run our platforms that Commercial uses to go out and interact with our customer base.

I'm relying on the fact that they, right now, service about 32,000 users. So I'm going to piggyback on their experience. And I think that's the other thing that's been wonderful in all of this: there are pockets emerging everywhere, and we're all open to working together.

Even though I feel like I'm one of the first, I can actually talk to somebody who's done something completely different than me and has a lot to teach me, and maybe I can teach them some things about operating on a smaller scale. But yeah. I'm going to cheat there too. I cheat a lot, by the way. I find people who are smarter and better than me. I find teams who've delivered more than me and are more capable than me, and I work with them.

Audience: I've got one. How are you measuring the outcomes? You only talked about your outcomes from your previous projects as a nice surprise benefits or examples, but are you now looking more strategically at what outcomes you're hoping to achieve from these developments?

Maks Kiamos Shah: The question was how do we measure these outcomes, if anybody didn't hear, at more the strategic level. Right now most of the metrics we have planned are very much at the tactical level, things like the DORA metrics.

In terms of the value proposition for the actual solution, or what it's actually delivering, we're still using business cases. That's the model we're funding. In this particular one, case A, for example, most of the business case is based on the reduction of manual work people have to do, and freeing up time. That's what's being delivered there.

We also have a simplification of process going on. That's one we're struggling, actually, with to show measure-wise, because it's not really a matter of number of steps. What we're toying around with there is a productivity measurement more: how many people have stopped doing manual or tactical work? And we're trying to present it in that way. It's obviously very popular with the users. It is a more difficult strategic discussion.

The second one, on the other hand, is a little bit easier because it's directly related to the productivity of the nine-year pipeline we showed. We actually have very specific metrics in terms of conclusiveness of scientific results and things like that. So that one's actually quite easier to link.

Yeah. And it says, "Please wrap up now," and it's a wrap. Thank you very much.