Host Intro (Michael Dahl)

Hello, ladies and gentlemen, and welcome to our next speaker, who represents the company Grundfos, with solutions which most likely have an impact through all of our everyday lives, whether it is pumping, the circulation of drinking water, or water-borne heating, or for that matter, anything else that may be pumped.

Being the world's largest pump producer, Grundfos solutions are integrated in a multitude of environments and structures, hence controlled by a rapidly increasing number of apps and APIs. Grundfos is highly dependent on complete component transparency throughout the value chain in the integration with suppliers to customers.

I'm pleased to introduce to you Kim Hyldgaard, lead security architect, who will share with us his reflections on how to navigate this landscape from a supply chain AppSec perspective. Kim, over to you.

Kim Hyldgaard

Thank you, Michael.

So supply chain attacks, I want to reflect a little on that. I can promise you I'll not have the golden solutions for you, but at least I can tell you a little on what we're doing at Grundfos and how we see the world in that respect. So my name is Kim Hyldgaard. I'm a lead security architect with Grundfos.

So just for you, Michael gave a small introduction to Grundfos, and as Michael said, we are the largest manufacturer of pumps in the world, and we pump basically water. So we take water into your houses and away again. We are very much obliged to complying with the UN goals six and 13. Bringing clean water and sanitation to the entire world is our main goal, you could say.

So we were founded back in 1945, where Poul Due Jensen produced the very first pump. And you see a picture there in the right-hand side corner. And the reason for producing that was basically after World War II, a lot of farmers had a problem getting water for their fields, and this was the solution to that problem. And he has a quote back then, "The world is full of problems that can be solved in a better way." And this is really the way we live also today when we invent and come up with new solutions.

So in brief, there's some numbers here for you on Grundfos. As I said, we're the largest pump manufacturer in the world. We produce around 16 million units every year. And these pumps are of various sizes and used for a lot of different purposes, always centered around water.

So for you who do not really know how a pump looks like and how it could look, I've brought a few pictures on the sides of this slide here. So if you look at the right-hand side, in the lower part, there's sort of a long oval rod there, and that's a submersible pump. So this is where you drill down in the ground and put this pump down, and it'll bring you groundwater for drinking or for watering your agriculture plants and stuff. So that's a submersible pump.

On the left-hand side, in the lower part, we have a wastewater pump, which is basically bringing dirty water back to the cleaning facilities. The small red pump on the left-hand side is a circulation pump, so that is circulating water inside your houses for heating. So for your radiators and for your underfloor heating, we use that pump there.

Some of the other pumps are for circulating water. Could be in big cities, could be to do a flood prevention facility where you basically move water around in the city. Could also be for industrial purposes where you would, in a dairy, a brewery, or other industrial process facilities, want to move water around. So there's a lot of different areas, a lot of different applications, and I will not go through that and bore you with that. But now at least I hope you have seen at least some of the pumps here so you know how it looks like.

So what I'm going to talk about here today, the supply chain attacks. I want to dive into just a little to show you understand the side that we're coming from and what we see when we talk about supply chain attacks.

We have the pump and the controllers, and we have added cloud and services for the pump. So it helps us improve reliability, make our pumps even more efficient. We have pumps going as low as, I think it's five watts, so they're really efficient. But bringing extra value-added services on top of the pumping is what we work a lot with these days.

So on top, what we call the digital offerings are typically cloud applications, where we bring these different functionalities to our customers. And such a cloud application, a digital offering, of course, comes in a technical architecture.

And my headline here is simple might be complex, because even though we have a service technician here on the left-hand side who's using his iPad to do some servicing and some monitoring of a pump installation, there's a lot of things that goes behind that. And that is the supply chain. So everything that leads into this entire architecture is what you could call a supply chain or part of the whole supply chain.

So in the lower part here, we do have the pumps, the circles with the small arrows in are pump signatures. Then you have the controllers, SCADA systems, third-party gateways bringing data into the internet and cellular network, depending on which technology we're utilizing. Through this network, you bring the data into the clouds and do have cloud-to-cloud communication as well. So it's a really huge architecture. And this architecture I've drawn here is not in particular specific to our business. This is sort of the way IoT works. So that's pretty common to companies where there are physical devices involved.

So when we then look at the supply chain, what is actually coming into this? Then there's so many things because, of course, Grundfos do not make all the different things in this architecture. So we do have hardware suppliers, obviously, for the smartphones, the servers, the laptops that is being used. You have hardware components suppliers for all the things that goes into our products, especially the different chipsets, storage, communication, security processors, radio communication chipsets and stuff that we just utilize inside our products.

So on the software side, we have a lot of software components, which is the cloud itself comes with a lot of software, operating systems, protocol stacks, UI components, and the entire tool chain we use to develop software: automation control, compile, build, test, deploy. All these tools that are there as well is part of our supply chain. And of course, since we do have physical products, we also have production facilities where we put everything together.

So all this is what you could call the supply chain. This is a lot of different suppliers supplying things to us. And everywhere you could have a potential security attack. So I will dive into the software part because this is the typical place where you think of these supply chain attacks. But in principle, a compromised component or a production facility could also be compromised and give us some problems. But in this one, I would focus on the software parts.

So one angle I want to take here as well is the legislation and standards landscape. And currently, not so much legislation is in place in the different regions of the world, in different countries. There's a lot of standards, but not much in the legislation. But we can see the trend that this is coming and it's increasing. I cannot talk about supply chain attacks without mentioning the executive order from President Biden last year, where he is talking about enhancing software supply chain security. He's mentioning the famous SBOM, the Software Bill of Material.

In Europe, we have the NIS2 Directive also addressing security of supply chains. And part of NIS2 is that water is now becoming a critical part of our infrastructure. It wasn't that in Denmark where we are situated, but it will be now with the new NIS2. So it is important, of course, our cybersecurity efforts.

I've also brought a few of the standards that we look into. That's the 62443, which is the Industrial Automation and Control Systems Security Standard that also have requirements on externally provided components, and that is what supply chains is all about. You're getting components from the outside, outside your direct control.

The IoT Security Foundation, that's a framework we utilize as well. It has an entire section on secure supply chain production and how you actually put in secret software, firmware, and these kind of things into your devices in a secure way. These are some of the standards that we work with and that we adhere to in one way or the other.

I'll get a little back to this about the suppliers, because obviously, as you saw also in the previous, on the architecture slide, we cannot live without a supply chain. There will always be somebody supplying something to somebody else. We will not in Grundfos ever make tablets and laptops and cloud software and all that. So these foundational things, we have to rely on suppliers. So it's just how do we actually handle this?

One of the crucial points in working with or mitigating risk in relation to the supply chain attacks is the inventory overview. And why is it important? Tanya Janca said it very excellently: you really need to know what you're protecting, otherwise you cannot protect it. So an inventory is really, really important. And I've put some keywords here on what is important with your inventory.

And it's not by coincidence that I put automatically obtained on the very first bullet here, because it has to be automatically obtained. You have to do this automatic. You cannot do this manually, especially if you work with open source components, UI libraries that are updated very frequently. You cannot go in and do all these dependency lookups and find it automatically or manually. It'll simply not work.

So next thing is that it has to be complete. If you only have half an inventory, then you don't really know what the rest is, and you have no chance of looking into that. It has to be current. We cannot work with an inventory which is one year old.

It has to be reliable, so the information that is obtained in this inventory has to be something you can trust. It has to be searchable. And that's for the inventory to be sort of easy to work with. You really have to be able to search through it when you're looking for something. It has to provide overviews, and it has to provide details, because you need both when you are working with it in your everyday life.

And then I've written here that it has to be possible to integrate into your CI/CD pipeline, which is basically coming back to this piece about having things happening automatically. Every time there's a manual process, there's always the risk that it's not being done. So you really want to carefully do or take care that you're doing things automatically. Otherwise, it will simply not work in the long run. And this inventory would then be used to help you in making decisions on what to do with your software components.

So inventory is important. So I have brought a few practical examples so you basically can see how we work with it. And this is getting really down to practical examples. So I'll go to a software composition analysis tool that we use. We have a lot of different tools we use when we work with cybersecurity, and this one here is quite nice to work with. There's a lot of tools out there, and I'm not saying which tool is better than the other, but this is at least a tool that we're using.

So the first example I've brought here is a recent one here in January, where there's been found eight different packages with a vulnerability in these URL parser libraries. It's actually deriving from the Log4j incident, where if you use multiple libraries, parser libraries, to parse different kinds of things, then you could actually utilize these components here and lead your customers in the browser to go to a fake site or an evil site.

So basically, what I want to do is I want to check if we're using this piece here. So I go to my inventory tool, or in this case, the composition analysis tool, and I could just go and have a look for, for instance, the Video.js. That's a technology that we use. Just to be sure, let's just video. No. So we are not using that.

In this tool here, we are scanning just around 300 projects. We do have around 30 scrum teams working for us with software. So it's quite massive. I really need this search tool here. Otherwise, I have no hope to find anything.

So then I can see here the PHP, probably not because we don't use that technology. Python might be something that could be of interest here. Let me just check the Flask component. What do we have? We do have something Flask-y here. I can see there's a Flask Security, Flask on Chain, Flask Users. So different components in this Flask. There's a CORS, there's a Login, there's a Principal, RESTful, Talisman. No. So not that particular component are used here. So that is pretty nice. And that was actually quite quick to search through this.

And this particular, I have a link here from the Hacker News, but you could get this intel from wherever you normally get this. There's a lot of user groups you can subscribe to get this intel and figure out, just to know the vulnerabilities so you can have a chance to mitigate it.

So my next example here is a real supply chain attack. It's a UAParser.js, where in some of the versions here, it was a new version released. Yeah. Actually, the UA parser is a component in the user browser, which just checks something about what language do you use, where are you in the world. Very simple and very much used component in JavaScript.

The attacker actually took over the account of the author, and then he managed to put in, the attacker, a new version of this package. And this package contained a small thing, but it extracted credentials from Linux and Windows machines, but also installed a Bitcoin miner. So not something you want to have running here. And this is basically a supply chain attack. If you're using this component, then suddenly you get something in that is attacking you. In this case, it's attacking the user, but of course, Grundfos wants to protect our users, so we want to be sure.

So let me go back to my search tool here and just search for the UA parser. In this case, it didn't find anything, but sometimes the packages are called a little different in different formats here. But I do use the UA parser, and we do use it in different projects. I can see here the number of projects that's being used, and I can see I don't have these. It was 7.29, which was the vulnerable version, and also 8 and 1.0, and they're not here, so we're good, at least now.

So if I did find the 29, which was the vulnerable version, then I have a good way of actually going in and looking or helping the developers. Because when I go in here and look at the component, not at the version, I can see all the different versions here, and I can see the security risk up here on the right-hand side. So if I scroll down, I can see which versions we're using in different projects. And I can go in here, and I can actually see this 29 version. I can see this is the security vulnerability. I can also see it in the version 8.0 and in the 1.0.

So they are here. And yeah, as you can see here, we are using a component here which actually have some problems. So here, I could actually go into the development teams, figure out which projects are using it, and then request the developers to move up to a newer version if these security vulnerabilities apply to what we do and they would actually become vulnerable in the instances we have. So helping out and saying, "Okay, please move to 24 all the way up to 31." That is really, really cool. So you have an overview, and you are a little more in control with this kind of tool.

So the last example I have brought here is the lower part here, which is a little strange thing, but of course something like this can happen. It's also rather new. The two also JavaScript packages called Colors and Faker, which are small packages that are widely used. There's an author of these packages who grew tired of actually supporting these versions.

So 1.4.0 and 5.5.3 are versions that are okay. But he actually released a new version, which in Faker's case called 6.6.6, and in Colors it was 1.4.1. And when this loads into your browser, it's just writing liberty all over the screen and prints a lot of gibberish. Not really that harmful, but of course, not a nice situation to be in. So also a supply chain attack, but basically by the author himself here.

So again, back to the search tool. Really, really a bit, so color package. And I can see I have it here, Colors 1.3.3, 1.4.0. So we don't have this version called 1.4.1. And maybe I should just check for the other one as well. So here, Faker is 5.5.3, which is actually the good version and not that 6.6.6 version.

So in this case, we're sort of good. Of course, I should alert my software teams. I can see which projects are using it, so I can go to them and say, "Okay, please do not update to the newer version." But also consider that you might get into problems with this particular component, because if the author of this component is really angry and he doesn't want to update or maintain this anymore, maybe we should look for something else.

So with this tool, and very quickly, I could get sort of an overview of what happened in these cases. So very, very important. As I said in the beginning, this is just one tool. There are many tools out there. It is just so important when you're working with the supply chain attack that you have a quick access to information.

So this was the practical examples for you. And this, we are using it very much, of course. When we had the UA parser incident, the Log4j, this is the go-to. Do we have it in our inventory? Do we use it? And what can we do about it?

So this is cool. But really, finding these supply chain attacks is really, really hard because sometimes it's actually the real author, as we saw in the last example here. And then figuring out whether they are compromised is really difficult. So I would love if we could do something in addition here.

So how can we get more authenticity into this third-party software? It could be the components, it could be build tools. Anything can be compromised. So if there's already a version out there, we saw 1.4.0, how do I know that it did not change? It could happen depending on which site you're using. So are the processes good enough there so we can ensure that does not change? Maybe we want a hash signature or whatever to verify it.

How do we know that a new version, this 6.6.6, is not really what we want to have? Because it was coming from the correct author. Maybe peer reviews, maybe something like, I don't know, a trust indication. So many have already used it. It's all about trust and trusting your suppliers. And that is somehow recognized to increase that. I don't have the answers here, but I would like to see something like that coming in.

Of course, intelligence databases, the coverage, and the speed of which they're updating is very important, that we get even better there and we get all the sources in. So the support in our everyday life, and that's what I also said, we have 300 projects here, so it's very necessary that we are able to get information fast and we can get overviews of things because we cannot dive into details and just do a lot of things manually. We need to have these automated things, maybe indications of a lot of activity going on in some of the packages, and we can look more into that.

And then I would also think that our supplier cooperation, not just the open source, but also the closed source, which we are buying, components we're buying, we would start putting up requirements for the suppliers, because it's really needed to do more in this area if we're going to do even more mitigation. And we will see more of this attack since the core companies which were hit before, they're starting to protect themselves. So all the suppliers of components into your company, that's really a way in.

And we have seen that also with the SolarWinds, just to take a giant example. So how do we actually mitigate this? There's still a lot of work to do. I think what we could hope for is the tooling is helping us here. And as my last statement here, I don't want to have false positives because you can really, really spend a lot of time on this. So that is basically it.

So that was my last slide. So thank you for listening in. Contact me if you have any questions or something you would like to discuss. I'm also always open for that.

Host Outro (Michael Dahl)

Fantastic, Kim. And yes, that is certainly an open invitation here. There's a man that knows his details and reflections, so you're more than welcome.