Thank you. Good morning. So I am with Promontory Interfinancial Network, and I will have to start out with one little caveat. You're welcome to tweet, but if you do and you see @Promontory, that is not us.

That is not our company, and it is a very important, we'll say, consulting firm with some very important people, and I don't think they'd want me talking on their behalf. So tweet away. I am @simonpstorm, but here we go. So Promontory Interfinancial Network is a relatively small financial services company.

We've got 175 people, 75 in IT. But what we do have is we have 3,000 banks that use one or more of our services. So roughly half the banking community is using us in one way or another. And we've made a very successful jump into continuous delivery.

We got a little bit of outside help, and they helped us actually tremendously. But essentially, we now have one product that has got a full pipeline behind it. So we have push-button deployments all the way to production. We have blue-green deployments so that we can fail back easily.

And we even have infrastructure as code so that we can rebuild our environment as we go through the process. So I'm going to start where a lot of presentations end, and that's with the credits. And so these are the people at my company that made this happen. So I'm the guy that gets to get up here and talk about it, but really, these are the ones that did all the work.

And the key thing to note here is that this really was a true DevOps team. This was the first time that we actually brought the group together, and I will admit that I was one of those people on the development side that loved nothing more than throwing my code over to the ops guys and letting them deal with it. Get to go home on the weekend, all that good stuff. But working with this group and having a chance to lead them really gave me a great insight as to what their world was like.

And then the most important thing that came out of it was that I really got a good handle of when we work together, how much work we could get done. And we did, in probably about a year, we got more accomplished than the five years before that. Huge leaps and bounds. And it really was a great team, and we've now kind of disbanded it.

We've gone back and everybody does DevOps, no matter where they are. We kind of got rid of this little silo, and as a whole organization, we're doing pretty well. Now, one of the things that comes out of having this work behind us is that we started to win back a little bit of time. So all of a sudden, we were free to try to figure out kind of that next layer.

And so where we really applied a lot of our ingenuity was towards audit. And being in financial services, it's a major part of our lives and something that you just have to accept. And one of the first things that I do encourage everybody, and it's probably the most important point of the entire talk, is you can't look at audit as being some big adversary or a big inconvenience that's preventing you from doing your job. Auditors are really there for a great reason.

They have a tremendous amount of value, and one of the things is they have the luxury of going from one company to another, finding out what does work, what doesn't work, and so this is a great opportunity for you to talk to them and find out what are the best practices, how can I make this better, and best of all, maybe they can help steer you away from a very common pitfall. So we've changed our mindset. We look at auditors as partners. We work with them, and it's gotten much better for us.

But if you're going to start to bake audit into your process, you got to start early. So we have kind of that mindset, and we get into the auditor's shoes from the very beginning. And we always think, what are they going to ask? What's that next question?

And one of the things that did not win me very many awards was the whole concept of if it's painful, do it more often. And so you can become very unpopular very quickly if you tell your entire team that audit is painful, so we're going to do it all the time. That was not popular, but now that they sort of see where we're going with it, they actually really do embrace this, and it's a challenge, and they're smart people. They love problem-solving.

So we really address auditing kind of at the very beginning and all the way through our development and operational life cycle. So taking a small step back, all the way to the Bible. So the first time I got involved in an audit, I was talking to a guy, and I really wish I remember his name because he was fantastic. I have no doubt in my mind that this auditor made me a better manager.

He made our organization a better organization. And what he did is he sat down, and he explained to me that he wanted to see our Bible, our SDLC, and he wanted to be able to flip through to any part of it and say, "For this task on your list of projects here, prove to me that you accomplished this task." Now, that was a tall order for us. That was a pretty big thing, and I've always kind of been working towards trying to figure out how to do it. And my first pass was not so successful.

I created the mammoth SDLC. I mean, this thing was awesome. It had RACI charts, traceability matrices, when meeting invites needed to go out ahead of meetings, and how early documentation had to go out. It had everything.

I thought it was the best until earlier this week, and then I realized that MD 102 might've been ahead of me, but I was a close second.And, so doing what I did, I successfully brought the entire development and the entire IT organization to a grinding halt. And we realized that we just didn't have everything we needed to make this happen. So we raced through a couple of more iterations. We did a slight version of it.

We jumped to Agile. We were trying to figure out what was the problem, and finally it was DevOps and continuous delivery that got us what we needed to get to, to make all of this work. So, first thing that we've done now is we've gone, and I mentioned that we did this just for one of our applications. We wanted to talk to our examiners and our auditors ahead of time and say, "This is what we're thinking." And I came up with a presentation.

It was a mere 186 slides, took me four hours to go through it, and I decided to save you guys the pain and distill it down to just 25 items that are useful when you're preparing for an audit. So we'll start out with some common sense stuff and some Agile. So first of all, socialize your plans. Don't surprise your auditor.

You don't want them to come one year and you're doing a waterfall with six months releases and then have them come back and you're deploying every 26 seconds. That's a very big jump for them. So you got to ease into it and you got to get their feedback, because again, these guys have great ideas and they have great insights, and you want to leverage that. Also, don't risk your crown jewel.

So we used a product called Bank Asset Point. It wasn't transactional. There's no money moving through it. It doesn't have customer information.

It was a safe place for us to try out these practices, get our auditors' buy-in, also from a development standpoint, make sure that we were doing it properly. Now, the next thing you want to do is demonstrate that you're not just kind of flying by the seat of your pants. And so show that you have experience in this area, and that comes down to great conferences like this where you're learning from Disney and from GE and all these wonderful companies, as well as your local meetups and books and trainings, but keep it documented. So write it all down, and that way when they say, "Well, how do you know you're doing it right?" You can say, "Look, we're doing some great stuff because we're seeing this everywhere." Now we also had to do a little bit of Agile education.

That is no longer as big of an issue. A lot of our auditors are certified Scrum masters. That means they're not actually doing Agile, but they are aware of what's going on. But we still map the Agile SDLC to our waterfall just to show them that things aren't disappearing, they're just being reincarnated slightly differently.

And so we have this kind of exercise for all the steps, but again, just trying to make sure they understand that we're thinking through all of these steps. Now the big moneymaker and the winner for Agile is the shorter cycle time. And so this one is really near and dear to the auditor's heart. So you can explain then the scenario that you find a medium-ish vulnerability, it's not the end of the world, type of thing you would throw in the next release.

If you're doing six-month releases, it's probably not going to get into production for at least six months. But with Agile and two-week cycles, it's there in two weeks, and then with the beauty of continuous delivery, we can actually get it into production in a couple of hours if we want. And then last but not least is the risk. Auditors are risk, risk, risk.

It's all they're thinking about. So you got to translate what you're doing into a language that they understand, and so you have to explain how all of this results in less risk, less schedule risk, less quality risk, and less business risk. Oops. So now you've gotten through Agile, and usually they're pretty there, much with you.

Continuous delivery is relatively new, and so there's a bit more education that needs to happen here. The number one thing, if it's automated, it's auditable. This is huge. So you go back to the Bible.

They want to know that these tasks happened. I can actually go into Cucumber and tell them that for this release, this feature was regression test at this millisecond because it's a log file, and I have all that information. That's very, very powerful. I can also be sure to explain to them that we don't have the snowflake effect going on.

We know where our application is and that because it all goes through the same steps every time, we always have the same application and the same version of the application running in all of our environments. And then you start to pair that with infrastructure as code, and now you really got something special going because we all know how it happens. After a little while, your servers start to develop their own personalities, and they all start to do their own thing. We can just rebuild our infrastructure and show them that no matter what, we've got a nice stable state, they're in sync, they can be built on demand.

So if we have a problem, all we have to do is just rebuild it. We can either spend those three days trying to figure out what's wrong or rebuild it in 12 minutes. And then, of course, everything is documented and version controlled. So it's just win-win-win.

Now, one part, and I hope this is part of everybody's continuous delivery pipeline, is having some sort of static code analysis. We use a tool called Sonar. There's tons of them out there. They're all really, really good.

But have one of them, and that allows you to have some powerful information that you can show your auditor. So this is a graph, and we started doing it kind of on the right side of it. And I remember asking the team, "Well, didn't we pull these numbers a while ago?" And they said, "Yeah, but that was a year ago." And I said, "No, no, no, throw it in the graph and let's see what we get." This is what happened. We went from 17,000 issues down to about 4,000 and change.

This is because of knowledge is power. We didn't even know we had these issues. We had a one-year-old application. It was surprising to me that we had 17,000 issues.

It was surprising to them. But once we understood that we were doing things wrong, we were able to start to make the corrections and get them into a better state.And then last but not least, when you're talking about continuous delivery, is having that centralized repository. Everybody needs to have this because it's that single source of truth. That's where your application lives.

That's where your classes, your libraries live. That's where all your installs come from. That way, everything is running off the same version, and again, you don't have that snowflake effect of where everything starts to become a little bit different. Now, I think I just skipped over my friend automated testing, because this is a goodie, and this actually goes to one of my questions for the audience and for future.

So automated testing, I thought was going to be our great winner. Like, we did it. We got a lot of help, and we ended up with a magical 80% test coverage. We were so proud of ourselves.

And then the auditor came in and literally, I felt like he punched me in the gut because he said, "How do you know that your automated tests work? How do you know that they are not all passing every time because the developer didn't code them properly?" So we've unfortunately fallen back to having a spreadsheet of all of our user acceptance tests, and we have a QA person who goes through and verifies that they work. And so this is not ideal, but it is a reality. We don't have to do this for every release.

We do have to do it once. So I'd love to know if anybody out there has a solution to that problem. So we've gotten through the continuous delivery education. They're on board.

They're liking what they're seeing. But now you get to go that next layer deep and really start to show that you are understanding kind of the intricacies of things. So first of all, if you're doing Scrum, you do have to kind of get away from the board. And I'm a big Agile nerd.

I go to all the Agile conferences, and everybody loves the stickies. And what happened was, one time an auditor came into our office, and he saw a story, and it was sitting there, and it said it was in the column for ready for QA. And he pulled it off and he said, "How do you know this is going to get tested?" And the truth was, I did not know. I'm hoping the developer would be like, "Where'd my story go?" But the reality was it could very well not get tested, and so we went digital.

And digital now has traceability, it's got logs, it's got all the stuff that we need. And it opened up the doors. So one of the things, all the tools have lots of options. We just happen to be an Atlassian customer, but we use a couple of their plug-ins.

Group sign-off. This is a great plug-in. Everybody who needs to be aware of a release has a thumbs up, thumbs down button, and that will say that I approve this sprint going into production. Compliance, security, the product owner, management, we all have to approve that it's ready to go.

Then we took this, and we used another plug-in called Exporter. And this is awesome. We now automatically generate our documentation. We grab the stories.

It just does a big mail merge from Jira into Word, takes our stories, takes our sign-offs, and now whenever an auditor comes and they want to see something, we just say, "What sprint do you want? That one?" Boop, press the button, off we go. Fantastic. And then we do the same thing for our pipeline activities.

Going back to the Bible, we generate a pipeline report. We stick it in Nexus. We send it out in emails. We actually even send a copy over to Jira.

It has everything. Who pressed what button, when that happened, what test passed, what test failed. Every piece of information about the release gets caught up in this report, aggregated all nicely, and then sent out to all the places that it needs to be. This is a big win for us because again, we are now always ready for the auditor.

We're always ready to answer whatever questions they might have. Now, with all this data, you are able to start to capture more and more meaningful metrics. Another big concern of audits and auditor and examiners is really where is management in this mix? Do you have proper management oversight?

And so we can pull things out like this that show, look, we are tracking everything that's going on. We're reviewing it regularly. That strange little graph on the left is essentially how many times one of our stories goes in without getting a bug. And so you can see that our overall trend, we're getting better as a group.

Our consistency, not so good. And so now as a manager, I can say, "Okay, team, let's go figure out how do we make this more consistent. I'm aware of what's going on." You do this stuff to make the auditors happy, but in reality, this makes me happy because now I have the insights into my team so that we can write better code. Now, again, I mentioned earlier that I'm an Agile nerd, and I do believe that you should follow Scrum pretty much exactly as it's written.

As soon as you start to wander off, you get into the what's called ScrumBut, where we do Scrum but something else. But we did make one little change. We added a sprint planning review meeting. And this was a bit of a win for us as well.

First of all, it shows anyone who's interested that we are not following processes blindly. We are stopping, we are reflecting on them. We're making sure that they work well for us. This meeting is basically at the end of the sprint planning meeting.

We bring in compliance, we bring in security. We've also opened it up to our help desk, our client services. So anybody who's curious about what's coming up in that next two-week sprint, they can find out and at least be aware, if not, do a quick thumbs up, thumbs down for us. And so again, this is one of those little wins, too, because there's always that person who said, "I didn't know that feature was coming out." Well, did you go to the meeting?

No. All right. That was not our fault.So from there, the next kind of big piece for us is orchestration. In my company, and this is probably very specific to us, our QA department has the best understanding of the application, so it made perfect sense for them to be the ones to migrate the code.

Now, again, now we're talking that magical separation of duties requirement that doesn't really exist. We said we're not going to worry about this one, and we're going to solve that problem in the next slide. But for now, what we do is we allow our QA team to be the ones who are responsible for when code goes into test, comes out of test, goes into our staging or our pre-production environment, or when it goes into our production. So they're the ones pressing the button, making it happen.

One of the key points here is that we do everything with Active Directory. So one of the things that's happened is as the auditors saw all these wonderful tools that we had, like Puppet and Jenkins and Git and Stash, the first thing that they asked was, "Very neat. How are you sure that those tools are being used properly?" So we've hooked everything into Active Directory, and of course, the question then becomes, "How are you managing your Active Directory?" So you got to keep playing this game. But for now, with Jenkins, we can be sure that only somebody who is in our QA team Active Directory group has the ability to press the button to go to production.

Now, the separation of duties myth. For our purposes, I'm hoping that maybe a year from now, we won't have this step anymore, but it made great sense because you don't want to go too fast. And while our operations team was thrilled, absolutely thrilled, not to be part of migrations, they don't really like it when we're just plopping things into their environments without their knowledge. So we've created a couple of extra gates within Jenkins.

So for something to go into the staging environment, all our system engineers have to do is click a button that says, "Yes, you can come in." Same concept for production. We did the exact same thing for our application engineers. We allow them to have a chance to review exactly what's going in and say that the environment is ready for them. Small step.

It's about six lines of code in Jenkins. Really easy, but extremely powerful and extremely useful. And then, of course, emails. If you're in IT, you're used to thousands of emails.

We add to that. This is a great way for basically everybody else who's not involved in IT to be aware of what's going on. We email everybody, and it's useful. So Wednesday morning rolls around, I can see, hey, we're already in staging.

We're doing great this week. Now, one of the areas which all auditors and examiners are extremely worried about is your source code control. And that makes perfect sense because as we all know, garbage in, garbage out. So if something bad happens there, it's going to be a problem throughout the entire enterprise.

So it's great that we all use Git, but you all have to figure out on top of Git or Subversion or any of the other tools, how are you managing those permissions? And so in our world, we made a slight mistake in the sense that we let the development team set up Git. And what ended up happening was as soon as we showed this to the auditors and we went and said, "Well, what are the permissions like in Git and Stash?" We realized that just about everybody had an administrator access. And so you got to lock those things down.

You got to be aware of what's going on. We use Stash. Stash has fantastic privileges. Again, it's all hooked into Active Directory so that we have the controls in place, that everybody, both from the management side of things, but as well as on the audit side, everybody sees what we need to see.

Now, Stash also has some fantastic things that have helped us as an organization. So when you do a pull request, you should always have someone doing a code review. And we are guilty, back in the early days, of waiting until the very end of the release to do the code review, at which point it was way too late to make any changes. So we'd say, "Yeah, next time we're not going to do it that way, but it's good enough for now." So now we actually do code reviews as we're going through, and Stash has the ability to be very intelligent about it.

It can say, randomly choose three people to do the code review. And now you don't have to worry about collusion, where you basically do something bad and then tell three of your buddies, "I'm going to check this in and you're going to approve it for me." It's random, so it makes much more better. Much more better. It makes it better.

And then at the same time, you can also have Stash say, whoever has the best knowledge and the most check-ins in this area of the code, have them do the code review. So again, you can keep improving your own development practices and make sure that the right people are looking at things, and that it's getting done nice and early in the cycle. And this is another one. This is kind of where we actually get kind of the brownie points and the extra gold star is we wrote a custom Git hook.

And so I mentioned earlier that the development team are the ones who administer Git and Stash. So of course, we have to have a couple of local admins on that box. But a local admin in that application also has the ability to create more users. So what we did is we wrote a special hook that says the only people that can check in code and approve code are people who have an Active Directory account.

And we don't have any access to Active Directory. So we were able to get that security that we don't have the ability for somebody else to create a whole bunch of users and again, game the system. So this is not something that an auditor or an examiner is going to ask you, but it is something that when you show that to them, they just really say, "Wow, you guys are thinking about the problem, and then the next problem, and the next problem, and the next problem."So, we're doing great. We have a pretty successful implementation, but we still have some problems.

So as I mentioned earlier, when we did this presentation to our auditors and our examiners, it was, again, that 184 slides or whatever. The one mistake I made was I did not let them go to the bathroom, and they actually told senior management about that, so I'm forever being made fun of for that one. But the thing that they noticed was that permissions is such a huge problem, and so that's something that is near and dear to our heart. We all have all these wonderful tools.

They all do so many good things. But you got to make sure that they are properly locked down. As I mentioned, we use Active Directory. Active Directory is a bit of a dangerous tool because if it's used like it is in our enterprise, there's this tendency to copy profiles from one person to the other.

And so you bring in a new hire, you go grab another developer, you copy his profile, and now he's already got all the rules set up. Of course, that also means that now that person has the keys to the kingdom if you happen to copy a developer who's actively administering some of your applications. So we have an audit process that we're working on of trying to figure out, how do we review Active Directory on a regular basis? And then one of the other big ones to be aware of is this idea of continuous improvement.

So again, going all the way back to the Bible, the Bible was pretty much handwritten the first time, right? And so it didn't change very much. But if you're doing continuous delivery and continuous improvement, you're constantly changing all your processes all the time as you are getting better and better and better. Wonderful stuff.

You got to encourage your team to do that, but you also got to keep an eye on that, and you got to make sure that you're aware of what's happening, and then more importantly, from an audit standpoint, that you can show that you're aware of that. One of the examples is our Agile teams, they do their retrospective, and then they try to figure out what didn't work this sprint, and we'll make some changes. All of that for us gets documented. It all gets documented in Confluence.

There's a team board that says, "These are the rules that we're going to follow." And then management subscribes to that. And so every time there's a change to their rules, I get a little email that says they've now made these changes to how they define done or what steps need to happen before it can be handed off to QA. And that way, I'm always aware of what is going on with the team, and I've got a chance to, A, show that I'm aware, but also be aware, and be able to weigh in if there's a problem. So here's what I would love help with.

So first of all, that testing question from slide 10, that's a biggie. One of my big questions is how do you ensure and then audit that the right people have access to the right applications at the right level all the time? One of the problems, of course, is that if you do this quarterly, that basically means there's three months where things could be incorrect. And so I would love to know what other companies do to do that audit on a regular basis or, better yet, be notified as soon as something strange pops up.

And then one of the other big questions I have has to do with empowering individuals. So I love this idea that you get the people who understand the problem the best to solve it. But how do you make sure that they're not doing things that are against what your management team actually likes? And there's so many decisions that are being made every single day.

I don't want to know about everything. I just want to be the ones that might cause us a problem down the road. And so, again, that's another one I would love some help with anybody who's got any ideas on. So two minutes.

Couple questions if anybody has. Questions? If not, thank you all very, very much. Thank you, Simon.