David Blezard

Hello, everyone. My name is David Blezard. I am giving this talk today from one of the classrooms at the University of New Hampshire because I figured this was an appropriate venue to use, given that my work is in higher education, in technology supporting higher education. Really, everything that I do, everything I work on, is in support of what happens in this kind of a room. It is about the actual learning that goes on.

I am based at the University of New Hampshire. I have been there for a number of years, and most of the work I'm going to talk about today focuses on things that were specific to UNH. UNH is the lead institution for the University System of New Hampshire. It's a Research I institution, about 15,000 students. But I'm now part of an IT organization that has been consolidated across all of USNH. So Keene State College, Plymouth State University, and Granite State College, the other schools that are part of the University System of New Hampshire, are also areas that I now do work and have responsibility.

This is a chart of the interest in the term DevOps as measured by Google search over time. The term DevOps was coined in October of 2009, so that's where the chart starts, and it goes up to a peak of 100%, the highest level of interest that Google ever measured. You can see that August, September 2018 is about the 50% mark. That is where, over the last two years, we have had a doubling of interest in DevOps, at least as measured by Google searches.

I'm showing you this to have you compare it to this next chart. This is a month-by-month number of items that have been posted on the website for EDUCAUSE about DevOps. EDUCAUSE is the premier organization for IT in higher education, all about the uses of technology, both in running the organization as well as technology in the classroom and in the learning environment.

While this trend has gone up over time, it is certainly not going up at the same rate that it is across the broader technology spectrum as measured by Google. We hit a high of five items one month. That was huge. And I have to honestly say that many of these dots here are me, and things that I've done, presentations I've given, talks I've been involved with, because I've been involved in trying to promote DevOps ideas into the higher education space. But it really hasn't connected very well.

I think one of the reasons for that is the way that DevOps is often marketed. If I think back to 2016, when I and some colleagues first went to a DevOps event down in Boston, we were hearing stories about how a 70-person worldwide developer team all got together and on the same page and used continuous integration and pulled all their work together in that way. And I'm saying, when I have two people working on a project, that's huge.

We've got to get out the door first. If you don't release your code tomorrow, your competitor's going to do it instead, and you've got to beat them to market, so you better not just release once a day, but twice a day or five times a day or 10 times a day. And I'm just like, this doesn't make sense to me, right? Is there competition in higher education? Sure, but it's not the same thing. We're also not a cloud-native organizational structure by default, although things are certainly shifting in that way.

What is going on in higher education, though, is a lot of transformation in other ways. This is another snapshot from EDUCAUSE of the term digital transformation. I was going to make a chart of that one too, but when I actually saw the numbers here, I'm like, I'm not going to waste my time, because the fact that there are over 2,000 presentations and over 1,500 articles and so on makes the point pretty easily: yeah, there's a lot more interest and a lot more discussion around digital transformation and what that means.

So what is digital transformation? Well, it's defined as a series of deep and coordinated cultural, workforce, and technology shifts, and these lead to new educational opportunities, operating models, transforming an institution's operations, strategic direction, and value. There's a lot of things here that connect with DevOps right away. We're talking about culture. We're talking about technology. We're talking about value delivered by the organization and how we do things. So there are definitely similarities and connections.

For all of you who are DevOps technology companies, DevOps tool providers, DevOps service providers: if you want to reach higher education customers, you need to talk to us in our language. This actually works much better than just sort of talking about DevOps and go fast and go quickly and get stuff out the door and so on.

There are plenty of challenges in higher education, though. As I was putting this presentation together, this was an email that came out from our chancellor for the University System in New Hampshire, went out to all employees. It was just so spot on the things I wanted to relay that I figured I would copy and paste and use this here.

We are feeling the effects of a baby bust that happened about 20 years ago, a decline in the population in the United States in the number of births. There are fewer 18-year-olds coming into traditional college environments this year than there were last year, and there will be fewer next year. If you happen to be in the Northeast of the United States like we are, it's even worse around here.

There are changes in terms of what is the value of higher education and that being discussed. How do we deliver education? Do we have to be in a classroom like this, or can we be online? And if we're going to be online, what does that mean? Should we be teaching in the same ways, in the same modalities, in the same topics, or should we focus more on making people ready for business and marketplace needs for workforce? Competition is going up and the funding is not.

New Hampshire ranks 50th in state funding to higher education, and if we doubled the amount of money that the New Hampshire state government gives to higher education, we would still be 50th. So that isn't likely to change anything anytime soon. Then you put on top of this COVID-19. We had to close down in the spring like everybody else. A lot of students went home. When they went home, they weren't on campus using the dining halls and the residence halls and other services and so on, so there was a lot of money that had to be paid back. So there are really big challenges that UNH, the other USNH schools, and higher education are facing as a broad general trend, and even more so now because of the pandemic.

With that background laid out, I want to talk to you about our transformation and our work related to DevOps, and how we've gone from where we were to where we are today. I'm going to be focusing on work that was done by our team responsible for our core UNH websites and web applications.

This is what our environment looked like. We had three actual physical server boxes that everything ran on: databases, website applications, load balancing. If we had dev and test environments, they lived on there. A lot of times we didn't. It hosted static websites. It hosted stuff in Drupal 6, stuff in Drupal 7. We had a little WordPress thrown in there because, hey, why not? Somebody wanted it, so sure, we ran it. ColdFusion was what we did development in, despite the fact that ColdFusion often ran out of control and broke everything, and because it was all shared infrastructure, everything went down.

Then we started working on some maintenance and improvements of this. We had one awful deployment that was a Sunday morning death march that was supposed to be an hour long, that turned into five hours and failed and had to be rolled back, and so on. This led to us focusing in on one key thing more than anything else, and that was just pure uptime. We set up internal monitors and external monitors and just started religiously every month looking at our uptime and doing whatever we could do to make that number go up.

It took a while. I'm going to fast-forward to 2014. This is the new improved version that we got to, which was based on virtual machines. We were able to separate out functional environments, separate out technology stacks, so now when that ColdFusion stuff ran out of control, it would still crash the ColdFusion servers and all the ColdFusion-based applications would go down. But our websites that were in Drupal, our static websites and so on, they were still alive. So that was an improvement. We could also clone out proper dev, test, and production environments.

I would not have put it in this term at the time, but really what we were doing over all this time, and really what we were doing a lot of it from here on forward, was paying down technical debt. I would have talked about getting rid of legacy systems and out-of-date stuff and unreliable stuff. But really a lot of things had been built because, okay, that was easy, let's get it done and move on to the next thing. Sure, you can have a website for your lobster research, because why not? Is there a strategic reason for it for the university? I don't know. Nobody was really thinking about things in that way. So we had over 400 some odd websites. We pared that down to about 200. Still a lot, but a lot better. We started standardizing and having structural reasons for things, and making sure that when we did something, we did it well so that we didn't have to go back and redo things.

In 2016, we added Docker. We really started looking at containers as Docker was coming on board. It was in late 2016 that we first got our Drupal infrastructure for our web applications running in Docker. That was great because now we could take that idea of isolation of individual environments and build that out to the highest degree. It took us a long time. It took us about a year to build up enough knowledge about Docker and Docker Swarm, and maybe looking at Kubernetes and maybe not, and back and forth and working with pipelines and how do we deploy this and how do we manage this and how do we restart things. All the stuff we had to figure out took a long time.

But now we're in a place where all of our websites run in a containerized environment, and now not just if one individual application framework has a problem, it's now isolated to the individual website. So if suddenly our business school website gets a bunch of traffic, it might get overrun, it might get overwhelmed. We could do scale-out to help it, but it's not going to affect everybody else because it's isolated. So that's a huge improvement.

I've already talked about how long it took us to roll some of these things out and do some of this work, and there's a reason for that. Our budget throughout this entire initiative was $0. Well, not entirely. We did pay for an external monitoring tool. That's the only thing we actually bought. So we wanted something good, but it had to be cheap, which means it was not going to be fast. It took a lot of time and energy and effort to learn things and work on slowly implementing things. But the result of this was that we've really built ourselves an environment and a set of tools that meets our needs very effectively and very uniquely.

We've brought on a lot of tools. I've already mentioned some of these, like GitLab and Docker and Docker Swarm. Rundeck we've brought in for running utility processes on a regular basis or on-demand, so things that had been cron jobs before now become modular reusable pieces. We've started to do a lot more with automated testing and some static code analysis and some dynamic app testing. We have started to get centralized logging in place and centralized alerting. We're starting on centralized configuration management.

GitLab was just an example of us getting really lucky. We started using GitLab as a central code repository because it was a good code repository system. We wanted to get off of Subversion, and so we moved into GitLab. Then GitLab added CI/CD capabilities right around the time we were trying to figure out CI/CD capabilities. We were able to start using that. Suddenly the thing that had just been our source code repository was now our pipeline tool as well.

All of this was done with a $0 toolkit as much as possible. For development environments, coding standards tests, static analysis with PHPStan, unit tests, code repository, CI/CD, testing, hosting, and utility tools, we relied on open source and internal effort. We were not buying a big enterprise DevOps platform. We were assembling something that fit what we needed.

The second part of the journey is the data. Somewhere in the woods of New Hampshire is our core institutional data. Actually, it lives in a number of enterprise systems. In higher education we have lots of uses of data for academic and web work: learning management systems need students, faculty, courses, and enrollments; course evaluation systems need students, faculty, courses, and enrollments; directories need students and faculty/staff; department websites need faculty/staff and courses; exam scanning needs students. Why ask over and over and over?

Before, when we needed data from a core system, every individual application had to have a custom pathway built, a custom feed of data to go in and get those exact elements we need for that one thing and bring it back out. That was a lot of work and a lot of duplication.

One very specific example: we wanted some basic faculty/staff data from the HR system as we were working on building out our college and school websites. Just basic who you are, what department you're in, your name, your address, your office location, your phone number, your email address, some basic things like that. It took us eight months to get that because it had to be custom-built for that one function.

What we've done is set up what we call our data hub, where we now don't feed data to each application individually. We pull that core institutional data into an environment that is available to our web applications so that we can get to it. We use MariaDB there as the back end for it. In this data hub environment, we have all this stuff sitting so that when we have a new application come along, if it needs course data and faculty data and student data and so on, we already have it.

Now, we do still have processes in place. There are still approvals that go into effect to make sure that we're doing the right thing. We're not selling our data to somebody, we're not giving it away, we're not exposing things we shouldn't. But as an example, another time we needed to get data that was coming out of that data hub environment, it took us two weeks to go through those approval processes, and then we could get on moving with things. Much more effective, much more responsive, gets people what they need in a much better way, much more quickly.

We've taken this idea of working with data from a central place and built that as a core principle for our environments. We have one infrastructure in Drupal now for all of the different websites for each of the individual schools within UNH. There are 15 unique websites: the Marine School, the Business School, the Engineering School, Health and Human Services, Liberal Arts, and so on. All of those pull data from central sources: HR data out of our HR environment, research information about faculty research interests and their personal bio or CV, what's going on on campus from our central calendar system, what our degree programs are and the requirements for them, so that information is not duplicated. It is brought over as the one official copy of that information, never goes stale, never goes out of date. Also news items are built in another thing.

All of this goes into this one infrastructure. We maintain one infrastructure. We reuse that infrastructure over and over again. If we suddenly come along and buy a new college somehow, we could roll out a website for that college in about a day's work. Getting all the data fit into our environment, we have it there, roll out the website, and it would be pretty functional with their programs and their staff and so on. Also, because we've got one infrastructure that's shared, anytime we do fixes and enhancements, they get rolled out across all these sites.

This has also led to a much better user experience for our students and our staff. This is what our websites looked like before, pulling all of the headers from all the different websites. You will see sometimes About at the beginning, sometimes About at the end. We have How to Apply. Sometimes we do, sometimes we don't. Sometimes we have things about giving, sometimes we don't. Sometimes we call it People, sometimes we call it Staff, sometimes we call it Faculty. It's just all over the place.

Now, with one infrastructure, we have one consistent look and feel. Here is Liberal Arts. Here is Marine School, Marine Sciences. Here is Health and Human Services. They have the same structure, the same look and feel. You think you're in one organization that actually knows what it's doing because they all have the same information presented in the same way. It's much better service. It's accurate, it's up to date. If there's a change to the requirements in a degree program, as soon as that's rolled out, it goes onto the website immediately. If somebody's office moves and it gets updated in the central sources, it's on the website immediately.

While we are not releasing 17 times a day, we are releasing much, much more frequently than we used to, where it used to be we'd work on a website and then maybe not get back to it for months and months. Now, over the time we've had this in place for the last couple of years, we've been releasing about every 10 days, new functionality or upgrades to these websites.

So what's next? As I was putting this presentation together, I was looking back through some old things. I found this presentation from that event we went to down in Boston in May 2016. This was actually given by Jayne Groll from the DevOps Institute, and it was this slide about DevOps as a journey. It starts with a single step, it has no finish line. It connected back with me when I was doing this, and that is exactly where we are. We're not done. We never will be done. We just want to keep improving from here.

What are some ideas? How are we thinking about doing things new and different going forward to continue that digital transformation? This is another thing from EDUCAUSE. It's a list of all the possible kinds of services you might have across higher education. If I look across all of these, across the institutions, the ones I've highlighted here are places we do development. Everything I've been talking about so far is in that very last item down there, web services. Those are websites. We do development there. We do research-related development. We do development around integration to connect tools together. We do development around reporting.

But across a lot of these things, we don't develop. We don't build our own learning management systems. We don't build our own alumni systems. We don't build our own facilities systems. We don't build our own HR systems or finance systems. We buy them and use them. And so we have this sea of applications that we deal with. My current role is as a director of application administration across USNH, and so I'm thinking about how do I take these DevOps ideas and apply them across this kind of a space.

Just because DevOps has dev in the word, and implies development, doesn't mean that the ideas, the idea of having shared responsibility for an entire system and service, being able to measure things and use those measurements to drive improvement, to automate things, to do more reliable, more effective release and delivery of services, all of these things still make sense. If I have an application, it requires our database services, our system administration services, our identity services, our security services to all work together. A lot of this still makes a connection and makes sense in terms of talking about how we should run and manage things.

There are also specific pieces of things, and I'm going to give you three really quick examples here of things that we're just starting to experiment with. Automated testing: while we're not developing our learning management system, why can't we test our learning management system on a regular basis? When the developer releases new updates, and this is a SaaS product, so they release stuff all the time, how do we know that it still works? Especially if we make configuration changes. We have some things where there's a certain feature, there's no real way to turn it off, so we use some JavaScript or CSS to hide that button away so people can't use that feature. How do we know that's the case? How do we really know it's up and running and functional? If we can easily write functional testing using something like Behat or Selenium, we can get something like this.

Here is a proof of concept for validating logins for our learning management system. On the left, we have a situation where it goes through and it does a login test that passes. It does a login test with a bad password that fails just like it should. Great, that's a pass. Then we try to do something where we should hit a certain webpage and redirect, and we didn't. By comparison, on the right, you can see a successful test. Again, very simple, but this is the kind of thing that we're starting to build out for some of our core applications so that we don't have to validate them ourselves by hand over and over again. We can set this up to run when we need it, or even better, let's run it every 15 minutes and just know that everything works.

Applying the concept of doing things in code instead of doing it by hand: a lot of web applications, you go in, you click the button to add something, you type it, you hit save, you go next, again, again, repeat, typo, fix, change. You're doing all this stuff, and there's no permanence to it. It's kind of boring if you've got to do a lot of manual data entry stuff. We had an opportunity with our media hosting environment, where we set something up new for our undergrad research, our conference in the spring when they couldn't do it in person. We gave them this as an online platform to share all the student work.

It worked really well. We were able to build out that configuration in code. There were APIs available. We could basically build this, upload it into a test environment, get the department to look at things. They had some changes. We tweak the source files. We upload again. Looks good. Exact same thing gets loaded in production much more quickly, and we've got it as a permanent artifact we can reuse over and over again. Applications that allow this kind of modality of working instead of just everything must be done through button clicks are much better, something we definitely want to encourage.

Also, taking the idea of microservices, little tiny add-on functionality. One simple example: faculty members want a course in our learning management system to do some things so they can try something out. We call these sandbox courses. We don't normally want faculty to be able to create their own courses. That might lead to problems. So instead, if we had a simple button that they could click on to request a sandbox course, we capture the information, it does the work for them. That today happens by a person intervening. They submit the form. Somebody actually goes in and does the work. Let's automate it. Let's speed up delivery. Simple little services.

Dealing with students who have incompletes in a course: at the end of the semester, for whatever reason, they can't finish the coursework, they're allowed to continue on beyond that, but the course closes down. The faculty member has to ask me to open it back up. Why can't we have a microservice that shows the faculty a list of their courses? Once you pick a course, it shows you a list of the students. You pick a student, you submit a request for an incomplete reopening of the course to that student. It just goes and does it. Little microservices to give better functionality.

To close things out, this is my definition of digital transformation: taking away the rough edges, taking away the technology problems and weaknesses, taking away the over-engineered, over-designed solution that has been the way we've always done things, and the silos that separate, keeping us from supporting what we're really trying to do, our real jobs, our support for higher education and to support learning. If we do that well, that is what we want to accomplish. And to me, that sounds an awful lot like DevOps.

Thank you very much. I hope you enjoyed the presentation. If you'd like to talk more about it, please reach out. Good day.