Host Intro (Gene Kim)

As you may know, our first DevOps Enterprise Summit was in San Francisco in 2014. It was intended to be a conference not for the unicorns, the Facebook, Amazon, Netflix, Googles, and Microsofts. It was meant to be a conference for horses, by horses. No unicorns allowed. And this was despite the fact that so many of our friends were at the unicorns.

There were many surprises, but without a doubt, the biggest surprise to me was which session was the most talked about. And it was the session from Mark Schwartz, who was then the CIO of the US Citizenship and Immigration Services. And I think it's because we all thought we knew what world-class bureaucracies looked like, but then Mark showed us how little we knew, and he described the sometimes bewildering processes that are necessitated by being a part of a massive federal agency responsible for such an important mission.

I'm so honored that we have a team from USCIS presenting this year, describing the amazing work they're doing, showing that the mission that Mark Schwartz helped start way back then still lives on. David Blair is USCIS Delivery Chief of OIT. James Lloyd is Branch Chief of the Innovation Branch. Melinda Solomon's part of the Agile Training Organization.

And on a personal note, I'm so grateful for the important work that these people do. My parents were immigrants from South Korea. My wife and boss, Marguerite, her parents were refugees from Vietnam. And neither of us would be here without the work of this incredible agency and their dedicated public servants. So please welcome the team from USCIS.

USCIS Mission Video

US Citizenship and Immigration Services is the agency that administers our nation's lawful immigration system. Our mission is to safeguard the integrity of this system by efficiently and fairly adjudicating requests for immigration benefits while protecting Americans, securing the homeland, and honoring our values.

As part of the Department of Homeland Security, we are 19,000 government employees and contractors working at more than 200 offices. We ensure America meets its international humanitarian obligations by screening refugees and applicants for asylum. Our officers carefully vet those who apply to come to the United States to live and work by confirming they meet all eligibility requirements as required by law.

On an average day, we adjudicate more than 26,000 requests for various immigration benefits. We ensure the employment eligibility of more than 80,000 new hires in the United States through the E-Verify system, and we welcome nearly 2,000 new citizens at naturalization ceremonies.

In fiscal year 2018, we had a $4.5 billion budget supported almost entirely, 97%, by fees. In that year alone, we naturalized 757,000 new American citizens, accepted more than 600,000 new lawful permanent residents, and processed 2.1 million applications from aliens seeking to work legally in the United States.

Every day, we work closely with our DHS and interagency partners on behalf of the American people to ensure the safety and security of our immigration system while allowing immigrants whose requests are granted to enjoy the privileges and advantages of lawful presence in the United States.

David Blair, Jim Lloyd, and Melinda Solomon

David Blair: Great. Fantastic. So, welcome to the Push and Pull of Partnership. This is intended to be an honest conversation between IT and the business. We're here from Citizenship and Immigration Services.

Hi, I'm David Blair. I run the agency's digital transformation program. I'm also the chief of the Transformation Delivery Division within the Office of Information Technology.

Jim Lloyd: And I'm Jim Lloyd. I'm the branch chief of the Innovation Branch within the Field Operations Directorate, so I'm one of David's customers on the business side.

Melinda Solomon: And I'm Melinda Solomon, and I run the Agile Training Program at Citizenship and Immigration as a contractor. And I'm really today just a USCIS storyteller.

David Blair: So as you saw from the video, we run the largest immigration service in the world, and so that requires a lot of work, and sometimes we have some pretty big challenges. One of the biggest challenges that we have is paper. We have a lot of paper. Rooms in buildings across the country full of paper, and it comes in the form of forms that people fill out to request benefits, and then a whole bunch of supporting documentation to justify their request for that benefit.

Yeah, when you think about all the files that we receive on a daily basis, if you take each one of those, stack them on top of each other, they would reach twice the height of the Statue of Liberty. It's pretty crazy when you look at this. We're in the 21st century. We witness these technical marvels every single day, and we're still dealing with rooms like this. I always tell folks, I say, "You may not find the file that you're looking for, but you'll find the why we need to modernize, and that's for sure."

Jim Lloyd: Yeah, and on the business side of things, it's important for us to communicate to folks outside our agency that we do view these files as the lives of people. And many times more than just a person, an entire family may be waiting on that one mother or father to get status, to get the job, to get health benefits or whatever. So paper has been an inefficient way of doing business for a long time. Files get lost. Paper falls out of files. USCIS, as the record keeper, we don't always have the file. Sometimes other agencies are fighting for the file. So it's a paper problem that we're trying to solve in an effort to meet our mission and serve the people we serve better.

Melinda Solomon: So when it came time for USCIS to start leveraging IT to solve some of these problems, this paper issue was really the logical place for us to embark on a digital modernization.

David Blair: Yeah, that's right. Again, traditionally a heavily paper-based organization. We've had individual field offices and service centers. They try to modernize. They try to kind of streamline where possible. But again, their tools are really go out and purchase the technology or kind of tweak local SOPs and policies. But that kind of thing really causes some additional problems, where you have different offices operating in different ways.

Really, it causes problems in terms of sharing information with our DHS partners. Simple things like verifying the identity of an applicant. It's problems that we shouldn't have in this agency, and that's really the genesis of the transformation program. This program was meant to take the agency, go beyond these paper-based processes to a fully integrated electronic operating environment. And the platform that we were building was the electronic immigration system, or ELIS, essentially.

And who are our stakeholders anyway? We've got each of the operational directorates. They're all responsible for different lines of business. They've got folks like immigration officers and records clerks and background checks clerks. Each of these are folks that have their very complex jobs, have to pay attention to policy changes. And building one system to kind of satisfy every single office and every directorate within USCIS was definitely more challenging than originally anticipated.

Jim Lloyd: Right. And our external interests are large and loud as well. I think a lot of folks think of citizenship and immigration in terms of the immigrant population that come to us. But there's usually a US citizen or a US resident filing for that person. We serve them as well. There are United States corporations bringing people in, skilled labor. The US farmer bringing people in. We serve the US military. We have a lot of US citizens applying for international adoptions. All of these people are usually well represented with attorneys, community-based organizations. They'll go to their local congressional point of contact as well. So we have a lot of folks involved and a lot of voices involved in what we do.

David Blair: Yeah. And because we're a part of the federal government, we have a lot of oversight entities that we're beholden to as well. DHS Acquisition and Investment wants to make sure that we're using our money wisely, and the Government Accountability Office and Inspector General wants to make sure that we're making ethical choices with the money that we have. And then, of course, it's Citizenship and Immigration, and so congressional committees are always interested in what we're doing.

Melinda Solomon: Yeah. So getting all of these stakeholders on the same page is pretty challenging. And really, the initial goal was really just to get the business and IT on the same page. And we hadn't really done it this way before, and it ended up a little bit like an arranged marriage. Everyone had really good intentions, but everyone had a lot of assumptions about what the other really wanted.

David Blair: Yeah, I'd say that we were very much like your typical kind of government organization. We operated in very strict hierarchies. We had teams that were operating in silos, little collaboration. And operating in this manner caused us to lose time. We had these self-imposed kind of handoffs where we'd have one group that was responsible. Only them could engage with users to elaborate requirements. And then once they were done and had all the requirements that they could possibly think of, they would kick it over the fence to a dev team. And then the dev team, when they were done coding, kick it over to yet another independent test and evaluation team, and kind of so on and so forth.

And so again, operating in this manner caused us to have to de-scope quite a bit and become very focused on the schedule and not focused on what was most important, which was really delivering business value and making sure that our business operations was functioning well.

Jim Lloyd: Yeah. And from the business perspective, we felt back then everything was schedule driven. The business requirements were secondary. We knew that the IT folks obviously have a schedule, they have deadlines, things like that, but the requirements just seemed to come secondary to the schedule. And something that, I don't know if it's unique to government, maybe you can relate to this, but we didn't hold the purse. You've heard the phrase, "The power of the purse." We didn't go to IT and say, "Hey, build something, and we'll pay you for it." They were building something for us. They were paying themselves for us, for what they were building. And so it just kind of added to our feeling of a lack of control, lack of power, that we didn't hold the money. We were just getting whatever the kitchen was cooking.

David Blair: That's funny.

Jim Lloyd: Your kitchen.

Melinda Solomon: But we thought that with David representing the IT community and Jim representing the business, that surely we were on the road to success. We had spent a lot of time planning and strategizing, and we were ready to celebrate when we launched our initial version of the system. Did it really end up that way exactly?

David Blair: That's right. Yeah. Again...

Jim Lloyd: He was driving.

David Blair: Okay. Yeah. Set me up. I'm going to keep my mouth shut now. But certainly what was meant to be this big celebration was really a spectacular kind of disaster. Of course, everyone was committed, right? Everyone was working hard, long hours and focused on the mission, right? But again, commitment wasn't enough. And when we become so focused on the schedule and not focused on doing the right thing, we become the problem rather than the folks fixing it or giving you guys solutions.

Jim Lloyd: Something we contributed to this perfect storm on the business side as well, was that for a few years prior to turning everything on, we had been telling our field offices that this great solution is coming. We're going to solve the paper problems. Everything's going to be digital. It's going to be like the IRS. It's going to be like when you go to the doctor. And when we turned everything on, we did it with our most publicly visible product line, which is naturalization, and it just didn't work.

Certificates didn't print. People weren't showing up for appointments because there were scheduling issues. And so we had people maybe flying from one island to Honolulu for an interview, or someone driving over rocky, snow-filled mountains to get to Anchorage, taking time off work, renting a hotel, family with them, all the excitement building to the family member coming out of the office with their naturalization certificate, and they came out empty-handed because the system didn't work.

And so not only did we fail the people it's our mission to serve, another problem we had is we failed our internal users, and they were trusting in us to give them something great, and now trust is very low, and our system, ELIS, really became a four-letter word in some of our offices, unfortunately.

Melinda Solomon: Yeah. So here in our story at this point, we're $800 million into this project and the program is in breach. So it makes sense perhaps to take a strategic pause. And I think our business and IT stakeholders need to go to maybe the DevOps version of marriage counseling.

David Blair: Yeah, the breach was a blessing in disguise. And of course, we're government and have these traditional hierarchies that exist everywhere. For example, in management directorate, this is where our CXOs reside. And at that time, the program was housed within the program offices. And of course, Jim, your folks in FOD and others in the business, who we call the business, again, they had their own kind of hierarchies.

And so the breach allowed us to really take a step back, kind of reorganize into these business-oriented portfolios. The portfolios were meant to align with each of the operational directorates, making sure that we were always serving their needs, that we were working in conjunction with their priorities. And then we assigned these portfolio managers, folks that previously really didn't feel like they had the authority to make decisions. Here, not only did they have the authority, but we put them together again with folks in the business to make sure that we are properly aligned, and we're getting back to delivering business value and maybe just starting to earn some of that trust that we lost.

Jim Lloyd: Yeah. This reorganization on the IT side really partnered my team up with some very specific people. And something David did for us that, if I were to point to one moment in time where the ship started to make a U-turn and things were looking good, it was these field office visits that we were making. It wasn't that managers were going out to the field because it was winter and folks wanted to go to LA and San Diego. It was the guys in DC who write the code. David was sending those guys out to the field so that they could see their product in use.

They sat in the office with real applicants and real officers and watched real interviews in the live environment. And so not only were those guys getting to see their hard work in use, but my field officers and clerks were able to talk to them and give suggestions to them and say, "Hey, this would be better if it did this." And the guy would say, "Well, I wrote the code for that." And he'd make a fix. They'd literally go to lunch while the new code was released into the production environment, come back, hit refresh, and it worked. Their voice was now feeling heard in a very, very real way from the mail room clerk all the way to the top.

David Blair: Yeah. I think that, again, another fundamental shift: our folks that, again, in the past had been working in these kind of silos and these artificial handoffs, needed to know that there are real ramifications for delivering products and services that aren't high quality, that aren't meeting the needs of our users. And so, as you mentioned, we sent folks out to the field, and we went to LA to witness a 4,000-person naturalization ceremony in the Staples Center. And we saw when folks were pulled out of this ceremony.

They're there. They're thinking this is one of the greatest days of their lives. They took time off of work. They brought their family members with them. But because of the problems that existed in the system, there's no choice but to pull these folks out and say, "I'm sorry, we're going to have to reschedule you." And our folks, they took that to heart, and instead of running back to DC, heads hanging low, they actually came back inspired.

And they shared those stories with their team members and folks not only within their own team, but else across all teams. And the result was a kind of a cultural change across the organization. We saw folks really working at a much higher productivity. And again, there was this motivation that had once been kind of, I would say, like when you put a fire out, and in the past, we operated on this very cancerous operating model. So turning that around and getting folks inspired to do the right thing was definitely a game changer.

Jim Lloyd: From the business's perspective, and this isn't something that was written down in any formal plan, this is just a business guy talking to a bunch of IT folks who know far more than I do about IT. The secret sauce for me was in these field visits after office hours when my team and David's team, or David and I, would have dinner together, talk in the lobby of the hotel. We talked about work, but we also talked about our spouses, our kids, our grandkids. There was something that humanized those guys on the other side of the table when those personal conversations were taking place, and it very quickly ceased to be an us and them sort of situation and became a we. And we really felt like one team.

I'm not saying David buys me a Christmas present or anything. I've been waiting for one. He doesn't even hit like on my Facebook posts. But we talk daily. His team, my team, we talk daily, and because everything is changing on the fly constantly, I really feel like if there's not that daily constant conversation, some of his team, you're here, you know, there's something wrong if that communication's not there.

Melinda Solomon: Now, that level of communication and positive relationship and significant trust between IT and the business was not something that we could achieve without really investing in aligning our IT practices to be able to leverage tools and pipelines in a modern way.

David Blair: Yeah, I think by, I don't know, 2014 or '15, we'd already migrated to the cloud. But I think that now that Jim and I are Facebook besties, we really needed to elevate our game. We needed to bring in the tools and practices that were going to raise our level of maturity. And so we started to devote the basics, devoting capacity to working down our technical debt, our quality issues. I think about three years ago, we were somewhere around 50% code coverage, and of course, we had to work to bring this up well over 85%, but that became a best practice, right?

We had to improve our pipelines. We had to make sure that we were bringing testing and automating testing wherever possible, bringing performance testing and security testing into the pipeline. But I think, again, one of the fundamental changes, we needed to get new products, new services out to Jim and his folks on a daily basis. And so we needed to do so in a way that was going to minimize any impact to operations. So we, again, worked to incorporate these zero downtime deployments, or ZDD.

Jim Lloyd: Yeah. If you look at that second bullet on the business side, you're probably thinking, "Is this guy going to stop talking about communication?" In every marriage, there's always one person that wants to talk, right? All the time. So that's me.

Some of you, most of you probably flew here. Some of you might have even flown first class because you're from those cool companies. Something that you probably haven't experienced because you're not a government employee that's the coolest is you get to be part of boarding group nine. That's right. And you get to sit in the very back by the toilet, and it's awesome.

But that's kind of where we were in this development process. The business felt like we were in the back of the plane. We knew where we were headed, but we didn't get our Coke and peanuts until everybody else did and let us know when the plane landed. What David did for us, though, is he really kicked open the cockpit door and brought us right to the front, and he didn't have to do that.

He didn't have to open his door and show us the instrument panel, but my team now, they all have Jira access. We all have access to the Kanban boards, even though we're on the business side. And maybe that's how you do business already, but we're in the backlog grooming meetings. We decide what gets brought to the top and prioritized. So bringing us in and giving us access to everything, it's a risk on his end, but for us, it just makes everything so transparent. It eliminates risk for us. Makes us feel very comfortable. Even when things go wrong, there's no surprises. And so, if you can give your business folks as much transparency as possible, I know it's risky, but I think it pays off.

David Blair: Yeah. I would also say, Jim's not our only customer. And so again, in the days when reliability wasn't really our friend, we had lots of problems with that system. We really need to incorporate or take some other drastic measures in order to get product out the door into the hands of the business without kind of bringing the city down, without darkening all the lights. And so, essentially we started to decouple this monolithic application into microservices. And of course, that gave us the ability to really scale more elegantly and solve our reliability issues.

I think really, I can deploy one microservice, again, without disrupting business operations. And I think that once we started to make those architectural and design changes, then we started making tweaks, and I talked about zero downtime deployments. But we added a piece there. We added the ability to toggle on features. And what this meant was, again, new development we didn't have to keep hidden in a different branch. The goal was to get this new functionality into the hands of Jim's folks in the field so that we can get feedback and iterate on that feedback.

And so what happened was when we started to incorporate these feature toggles, Jim would identify, "Hey, I have a person over here in Yakima or Phoenix, and we're just going to toggle it on for that one person." And so we could, because we had the tools and again, the maturity to do so. And once he was comfortable, he said, "Okay, let's open it up to the entire field office," or, "Let's continue to open it up to the district or the region." And so again, the zero downtime deployments, the feature toggles, gave the business a voice in not only the timing of this rollout, but the scale in which we would roll things out.

Jim Lloyd: Exactly right. And just to emphasize what David already said, I'd like to maybe dip my toe in the pool before I jump in. So for him to give the business a say in how things are released, because he's got a huge team and they're releasing new functionality and updates every day, I need to be able to give my field offices a heads up when something new is coming, and if it's something big, I might even need to train folks.

To train someone in Yakima or Boise, no big deal, I'll give you a week. But Los Angeles, New York, Miami, Chicago, the big offices, they don't want something new tomorrow. They need time to train. And so the toggles gave the business the opportunity to say when and where new things were turned on. And again, lower risk, higher comfort level for us in this partnership.

Melinda Solomon: So now that Jim feels like he's being heard and IT, David, feels like he can truly be responsive to the needs of the business, we were in a position where we really needed to be able to have a culture where innovative experimentation was safe and something that people strove to do.

David Blair: Yeah, I would say that when we had all the media attention and congressional inquiries, and we were in breach, we had a lot of negative attention, and it felt like a very punitive kind of culture. So, you're right. Creating that culture where folks could experiment, and it could fail, and they were safe to do so, really resulted in some fantastic wins and things like what we call streamlined process.

And we have a group of individuals, all they want to do is renew a green card. It's something that we know who these folks are. We've given them their green cards before. This is a low-risk population. What we did is we just used automation to kind of take over the tedious and menial tasks that we had once had a lot of people spending a lot of time doing. And that freed up the smarter officers who have the experience to do the complicated adjudicated decisions.

And the result was fantastic. We had a backlog of like half of a million of these green card filings that were kind of piling up in a virtual shelf, so to speak, and we were able to reduce those down in just maybe a few months to around 70,000. And from a customer's perspective, they originally had been waiting well over 10 months just to get this green card renewal. But through automation, we were able to reduce that processing time to around 30 days, and most folks were getting, I wouldn't say most, but a lot of folks were getting their cards within two days. So, fantastic results.

Jim Lloyd: And from the business side, things were never forced or crammed down our throats. Like the second bullet says, we were given control over the experiments. So even just a couple of weeks ago, we're rolling out a new product line now. We went to Baltimore, and we did just two interviews, two cases, people seeking permanent residency. There were some problems. Nothing the applicant would have known about because our guys, David's team, fixed everything right on the spot.

A couple of days later, we did two more cases in Seattle. All of the Baltimore problems were already identified and fixed. We got to Seattle, a couple more problems were discovered, but those were fixed before we got to Chicago a couple of days later. So when my boss asks me, "How are things going? How is everything going?" I can say, "Well, we found some issues, but they were fixed." Well, is there a problem? Are we going to hear about this from the media? No, no. The risk is almost nil when we're testing at such a low volume.

And as you can see, David Blair, if you read his resume, was a Navy SEAL, but he's so soft at heart. He greets me every morning with love and affection. And awe. We know who's on the left and who's on the right, come on. I really value that. And I will say, the one point I'm trying to make from the business side today is that we're getting things done in a government environment, and it's happening in my opinion because there are strong business relationships here. It's not really about titles with us.

David Blair: Yeah, I would tell you that back again when we didn't have this relationship, we spent a ton of time, 10 years, a ton of money, and the result was, I think we were able to release six products representing around 12% of the agency's workload. And when you contrast that to just the recent or the past three years when we have this solid relationship, we've been able to get product out the door consistently. We were able to reduce the size of our resources and the cost and ultimately, we now have a platform that's digitally processing nearly half of the agency's entire annual workload. So again, there was magnitudes greater than it once was, the first 10 years of this spectacular disaster.

Melinda Solomon: So now at Citizenship and Immigration, we're no longer trying to recreate a paper-based process or simply digitize forms. That's not where our mindset is anymore. Now our goal is to truly enhance our enterprise agility. We want to achieve transformational outcomes that result in true business value.

But even though we've had some significant success, we still have some challenges that we're trying to overcome.

Jim Lloyd: I can take the first one there. Even though I don't supervise, my team does not supervise all of the contractors. The government has a lot of contractors. David's team has a lot of contractors. They don't like to say no. So on the business side, when we say we want something and my folks love gold plating, they love moving the goalpost, whatever phrase you want, they ask for the moon, not thinking in terms of what it's going to cost. Is it going to move this project back further? And so one of our issues, one of my issues, is telling my own people, the business, "Hey, settle down. These folks are not always going to say no. We need to know our limits so that we get what we've asked for and in a timely manner. Otherwise, we're just hurting ourselves."

David Blair: Yeah. It's funny because we talked about solving or getting past a lot of organizational culture issues, and we still have some. Again, in technology, we have folks that they really want to get the latest, the coolest, the bright and shiny thing, right? And then figure out what problem we can solve afterwards. And so that's something that we really need to change and make sure that we are a business first kind of mindset. And of course, in the federated environment, there's a lot of CYA mentality. And so it's just a matter of, again, DevOps principles. It's ownership and accountability. That's it.

So we're enthusiastic to talk to you guys further at the speakers' session tomorrow afternoon. We'll be there. And thank you so much for your time.

Jim Lloyd: Yeah. Thanks.