Erik Maziarz

My name is Erik Maziarz. I'm coming from AGCO Corporation. Somewhat of a generic-sounding name, but it's an agriculture company. We build different kinds of agricultural farm machinery: tractors, combines, implements, dairy devices, feeding implements, and a lot of different things.

A little bit about me: I started off, aside from some odd jobs here and there, in the Air Force. I worked in the packing and crating section. We shipped out supplies, worked in the warehouse or woodshop, and did a lot of work in the sawmill and woodshop. I moved from there to UPS, where I worked as an application developer. I worked on the website and an email communication application, so a lot of straight-up development work there. From there, I went to AGCO, where it has been a mixture of development, project management, and overall system design. It is not a mess, but a lot of different things.

About a year and a half or two years ago, we started with a project to bring a lot of our engineering teams into a global system setup, because AGCO is a company that has so far grown by acquisition. They will buy different companies: Massey Ferguson, Gleaner, GSI, Fendt, Valtra. Those are all globally situated companies, and from there AGCO integrates them into its setup.

Some of the problems with that are that they did not start from the ground up as part of AGCO. They are all independent companies, and being independent companies, they have their ways of doing things, different cultures, and different methods of basically doing stuff, as you would expect. It is not a startup where you have five people and they all have the same vision and plan on doing things the same way. For years, since I think 1990 or 1991 with the first couple acquisitions, they have been doing things their own way, and that has been an established process for a few decades now. Not breaking them apart from those, but trying to integrate them into a global environment where the same standards and processes are followed, has been what I have been working on.

Tractors, combines, and those kinds of vehicles are pretty complicated nowadays. They are not just vehicles that pull stuff around the field. They have GPS systems in them. They have autonomous systems that can help plant crops or plant seeds. You send them out in the field, and they have swarm vehicles that we have developed that basically plant and sow a field on their own. You have air conditioning and heating in some of the cabs. Automatic milking, automatic feeding systems, automatic systems for the farms: all of those require software. Part of what we have been doing is trying to standardize. There are a couple pictures of some of the advanced-looking things that we have in our fleet, in our vehicles, and things that we work on, sell, and have.

My group is the Electronic Functional Group. Basically, we are in charge of the overall global direction that our groups take in the software and electronics engineering group. We manage the overall application lifecycle management, the source code traceability portion of things, and the overall way that the teams work from a software development standpoint.

Like I said previously, a lot of these teams would do things independently. They would develop code, requirements, and testing. That was all standalone. Site by site, they would communicate their requirements and design specifications via email and Excel spreadsheets, and that worked when you could walk to the next building or to a cubicle over from you and talk to your manager about this stuff. But given the direction that we have been going, trying to develop a platform strategy where the same software can run on the same machines, you have to have some sort of interconnectivity between our sites.

You have teams in the U.S., Europe, and South America. Most of the engineering teams are in Europe or the U.S., but trying to communicate between them about new software builds being ready to test or implement, or requirements for what is going into a new product, is difficult. It does seem like something that a company that has been doing business this long should be doing by now. They do it sometimes well and sometimes not, and that is where my group is working.

A couple of years ago, a project came to us to start migrating our different sites into a global platform, a global environment where everyone did their development, their resource tracking, and their whole software development lifecycle in the same environment. We needed that because as far as software builds go, a lot of the time it was someone's laptop or someone's desktop underneath a desk that was running those builds. Once a week, once every two weeks, or whenever a sprint cycle was done, someone would download or check out the code, run a build, publish it, and that site would have access to that build. Then maybe they would remember to send it out to other groups, and if those groups had dependencies, hopefully they would get what they needed to start integrating and be able to build their software with those builds.

There are multiple computers that go onto each of these machines, each of these devices, and it is a highly dependent system. There are a lot of people that depend upon something down the chain or up the chain giving something to them.

One of the goals we had with this was to spread information that was known among one group but might not be known among others, such as different testing methods. With embedded hardware, or embedded systems, which we use for our vehicles, it is a little bit different than software. Like I said earlier, I came from UPS, where we did software development. We had web applications, and you would build it, deploy it, it would go to an environment, and that was basically that. It was dependent upon your application. It was a pretty standard process.

Embedded systems, from what I have been able to learn so far, are a lot different and a lot more complex than web applications. One script may not be able to do what you want it to do for these applications. We have terminals that are built for these machines now. The terminals are basically part of the armrest apparatus that is in these machines, where there is a lot of very detailed information. Depending on the sites that we had, they were doing those drastically differently. Part of the first couple of sites that we migrated were these terminal applications, and basically we were trying to cross-pollinate ideas about what were good processes with migrating and what were not, what worked and what did not work.

As I mentioned earlier, we had multiple sites situated around the globe. Most of them numerically are in Europe, with a couple in the U.S. We have some non-engineering sites in other parts of the earth, but they do not really contribute as far as the engineering situation goes.

On the next slide, this is not an everyday situation, but some of the processes that we had would lead to this situation as far as builds would go. We would start with a team in Linnavuori, Finland, having a build published to Suolahti, and that would go to France and Beauvais, and they would all just wait for a build to be completed, sent, and distributed. It was a bit of a mess. It does seem like something that should have been taken care of before or done better than it was, but that is how the situation was. It was not clean. It was not efficient. It relied a lot on manual intervention by teams. That is what we were trying to get rid of. Some sites had independent contracts with external teams, and they would have their own setups that would work for them. But as far as the company went, that was not something that was in place as a standard. Part of my team's responsibility was to try to determine and establish a standard process that we would follow.

Some of the concerns that we had were that we were working with multiple groups, multiple cultures, and multiple sites. It was not just one site that had a project being done. They have multiple projects. They have anywhere between 10 and 50 engineers; maybe the smallest sites had five to 10. It is a lot of people working together, and it is just not what they are used to. If any of you are familiar with Little League sports or baseball, you have an all-star team at the end of the year where you take the best kids from all the teams and all of a sudden they have to work together. You have conflicting personalities and conflicting ideas on how to work together. They are all good independently, but trying to get them to work together is an effort. You have to convince them of the best way to do things given their new situation, and that is a lot of what we had to do here.

Different sites had their own processes for doing things. What we were trying to do was: if a team had a build that was done via maybe one gigantic script that had a lot of hard-coded dependencies, and that is how they did things, part of our job during this project to migrate them away from that was to break it apart into modular pieces. We use Electric Flow as our continuous integration environment, and that is how we broke it apart into modules, or procedures, which would help them reduce the impact that changes would make. It would also make it so that other groups and sites could consume those changes, or consume the kind of knowledge they had gained from how they would complete their builds and incorporate it into their own.

Some teams had their homebrew setups. They would use Jenkins. They would use just manual execution of scripts, or they would use SVN or Git, check out their code, and run scripts. Talking with one of the teams recently, for a Linux-based terminal build, every two weeks it would take upwards of half a day to get the environment set up for the build, check everything out, and copy everything. A lot of manual steps are not something we want to have. You want to be able to click a button, or give a couple of predefined parameters, and tell it to go. This was a lot of human interaction, which is something you want to avoid.

For some of the earlier sites we worked with, we were trying to decide which ones were different enough from other ones so that an initial foothold at those sites would be visible to other people, and we could get them at least able to buy into the idea that it was going to be useful to get into this global ecosystem.

Some of the biggest issues we had were trying to get them to buy into it. They had existing setups that would work for them. It was how they were doing it for years. How do you go to a team and tell them, "Okay, the way you're doing things, I know it works for you. I know you guys can do it. You have steps you can follow, and you know how to do it, but we need to change that and have you do the same thing, but in a different way. There won't really be any immediate results, but that's something that you just got to do." Unfortunately, from my side, I can tell that they see someone from corporate coming to tell them, "This is what is going to be the new standard for you guys to follow. You might not see why it is going to be beneficial right now, but that's what we're going to do." Up until somewhat recently, there has been a lot of pushback and resistance to that. But the more momentum we have been getting with additional teams and sites on board with more projects, it has been going better as far as teams being receptive to getting into this ecosystem.

I mention ecosystem because we have been trying to get a lot of the teams and sites on board as far as the whole application lifecycle management, build, continuous integration, source control management, and a bunch of different things. It is a whole environment, really.

We started off by going to a couple of pilot sites. It was a multi-phase project, but the first pilot sites were ones that were complicated or diverse as far as the environment, type of build, number of builds, and complexity. The first site we went to was for one of our flagship or new developments, a new combine being developed from the ground up as far as AGCO goes. It is actually the first combine that has been developed completely in-house. All the other software builds that we had were for existing models. This was something that was not released yet, so we were catching it somewhat early in the process. They had been spending the past few years defining requirements and everything for it, but the build process and builds for it had really just been ongoing for the past year and a half.

It was a very complex thing. That took, I want to say, a couple of weeks to get down fairly well. There were some unique challenges, given all the different types of applications that were used in the build. I think MATLAB caused an issue: invoking MATLAB would spring up a user interface, which had some issues with the Electric Flow agent we were using. We had to run that as a process instead of a service, just Windows issues with that. Aside from that, it was just a complex thing to do. A lot of steps were involved, and a lot of different variations in the build.

Some of the challenges that we had were that you had people who were not super intimate with the existing build processes. They knew how to follow steps to do the build, and they had a checklist of things to do, but they could not explain it very well. They were not experts on it. Trying to migrate a build from someone's knowledge who was okay with it to get it implemented completely was a challenge. From my side, having to work with these people to get their builds from a somewhat documented process to something that was working was a challenge. It really just took a lot of time working with these guys to help them understand what they were trying to do and get it working in the system.

Part of what we were trying to do also was the communication of the teams. They would send a build via email. Sometimes, really with external teams, they had a site set up where they would publish a build. Part of what we set up was a common file server for builds to be published to, so that teams that had dependencies on these builds would be notified of new changes so they could incorporate those in testing, or for their testing, or for development purposes in general.

Going back to the testing portion of things, one of the things we were able to do, given how earlier these builds were being done manually or kind of separate from the common environment, was that once we incorporated these, we were able to have a common step invoked that would trigger a set of testing. Some of the testing for embedded or hardware systems is a HIL system, or hardware-in-the-loop, where you need some complex or powerful computing machines to run these tests. We were able to trigger these after our builds were done, and then import these or link these with our application lifecycle management system, Polarion.

At the end of the day, a daily build would run, and that would trigger the test run in Polarion to be executed. It would connect to Polarion, pull down all of the test cases in a test run, execute those test cases based on a testing framework that they had set up, and import the test results. That might seem like a fairly trivial thing or a basic thing that should be done. It simply was not for us. We had one of our sites, Suolahti for Valtra, that was ahead of the curve as far as that goes with our company. Given how those steps were now visible in this global system, other teams and sites were able to see that this is something they were invoking with their builds. Hesston, Jackson, and the sites in Kansas and Minnesota, and Grubbenvorst in the Netherlands, started to incorporate that also, because they all use the same ALM system. They are able to incorporate that same testing framework because they all use the same system now, and they are able to have their testing done automatically based on nightly builds, which is something that had not been done before. It was all manual.

I cannot say how much time that saved, but it is something that happens automatically now, and from what I have heard, it has been a boon to their development process. It has been helpful just by virtue of having what Suolahti was doing visible to everyone else in the company as far as development goes. All those arrows pointing to the global file server show that builds are immediately available and immediately visible to the other teams. That is one of the other advantages of having this global, visible system.

We started off with the pilot sites by implementing a couple of the complex or different builds. Like I said, we have some of these terminal builds which are done on Linux systems, which only a couple of people that do the terminal applications are familiar with. Some of the more complicated Windows-based builds, like the flagship combine build that I mentioned, were done in the first couple of sites. Then we moved on to additional sites, simply incorporating more of our engineering sites. From there, they have been able to kind of, not evangelize, but talk to the other people at those sites and get them to bring their builds into the system.

It has been a slow process getting people to buy into it because they have something that works. Why would you want to change? It is hard to get them to buy into it, and it has taken a couple of years because the people we had working on it were me and me. That was about it. We worked with Electric Cloud. We had some of their professional services people working with us to help get the teams migrated and do training. I would say if you can devote people for resources and training, that would help getting people to buy into this kind of setup, getting them to transition from one system to the next.

That is where we are at now. We have had additional teams at these sites come into the system. They have been approaching us, my team, to help get their builds implemented, which has been nice. Some of the sites still have a lot of ways to go to be fully implemented, but it is getting better. A couple of informal surveys looked at the time spent compiling or setting up builds. Builds were done via someone's local desktop or laptop. We now have dedicated infrastructure that is part of our actual network. It is a bit more powerful. Things run more quickly. That has been something that people like. Basically, it is process improvements that they have enjoyed once they get things implemented.

Where we are at now, we are still working on getting people off of their existing setups, which is taking time, given resources and training. Some of the complaints we had initially were that the way things were done before, they could kind of isolate things via Windows batch files or scripts located in source control or on servers. Getting those extracted into procedures was a bit of a process. That is what we have been working on training teams with. You have a couple of build experts at each of these sites, and they have been helping different people who work on different projects get their projects implemented also.

The biggest thing is the testing. The automated testing that occurs at the end of the builds is not really necessary to have the build done in this ecosystem to have the testing done, but they like just the whole thing being an automated process. Getting them in for the testing kind of sets everything else up from the beginning to get them to want to start using it.

The deployment pipeline is something else we wanted to start working with, given how we are not just a web application. The deployment process for embedded systems is a bit different. We have some other kinds of projects in place for that. They may or may not use our continuous integration environment, but that is something I have been trying to figure out how to incorporate. It is an ongoing process. There is just no good way of doing it with the way our setup is. It is something I want to be able to use, but I have not really been able to justify or find a way to get it implemented yet. The deployment pipeline process may or may not be something we can implement in the future, but as far as testing and the actual build process, that has been something that has been working really well for us.

That is basically it. That is how we were able to start from no kind of shared environment and go forward to a 25% integrated environment. It is not all the way there yet, but that is where we are at.