We'll bring out our esteemed panelists. First, we have the auditor. No boos. All right.

This is Jim, James, the auditor. You can tell because of his sport coat and it's buttoned. Okay, next we have the counselor. You can tell because he's got jeans on, and his shirt's not tucked in.

He's probably the guy that fixes everything. Right. Next, we have clearly the DevOps guy because he's in a wolf pack of one. And we have a financial services guy who is the stakeholder because he also has a sport coat, but it is unbuttoned.

All right, so there's a little bit of false advertising in that this is not ask the auditors because we only have one, so I will start off. I am Josh Corman. I am not now, nor have I ever been an auditor. But I do care about the relationship between compliance and security, and this seems to be one of the bottlenecks.

So until we can get you over the hump of your audit challenges with DevOps, which is mentioned by nearly every speaker in every case study, that DevOps and audit is a challenge. I want to get that clear out of the way so we can actually focus on risk management and actual security risks since compliance is not security. But we tried to get as many voices as we could here. Essentially, our protagonist, the one you most identify with, is our DevOps guy here, Byron, and he just wants to go faster and faster and keep doing things and innovating.

Continuous process improvement. Yeah. The goals is to be able to move things quick, accelerate our deployments, and also make sure that we build in the requirements from the audit team into how we deliver our products. And as an operations guy, the big thing that I always run across and have seen a lot of people talk about is what I call learned helplessness.

We don't know how to work through audits or fit them into our products and our designs and how we deliver them. So one of the goals that we got together for this toolkit that we're working on is to try and help give people a roadmap where they can meet the audit requirements using CI tools, continuous delivery, DevOps practices, and deliver their products that way and accelerate. Yeah. Also, next brief intro.

Our practitioner, so to speak, or the stakeholder, Simon, is not just in financial services, but also a service provider to many. So you're probably one of the most heavily audited people in the room. Yeah, it's a tough situation for us. So I work for a company, and we have services that are used by 3,000 banks.

So in addition to having our own audit and our own regulatory exams, we also have 3,000 banks that come to us with their auditor hats on. And so while we are in the continuous delivery realm, I'm always trying to figure out how can we do this but still be able to present a good picture to our customers as well as to our examiners and our regulators. So it's definitely a challenge, and unfortunately, it's just getting harder and harder as more and more of the regulations are encouraging banks to evaluate their third-party service providers. So they're coming under the guise of due diligence, and they are asking us very, very hard questions.

And you've learned in your career before DevOps disruption to have a positive relationship with the auditor. So not all auditors are bad. There are bad auditors, but not all auditors are bad. So as an auditor, can you give us a thumbnail as to what motivates you and how we should embrace you?

So I think I would focus on we're always looking for the why, and we're trying to help communicate that the risk is addressed. And so we're looking for efficiency for you and be able to communicate that, the common phrase is an opinion, right? But I think the good and the bad auditor perspective would be the bad auditors are the ones that are doing checkbox, that aren't really driving value to you and giving you the feedback that you need, and the good auditors are the ones that are more on collaborating. And auditors, we mentioned examiners.

Auditors could be your internal audit department, those that are full-time with your business, external auditors that are there for regulatory requirements, or those that are doing assurance type of work. So all three should be able to provide you a positive feedback loop. Those are the good auditors. Those that are just looking at checkboxes and kind of give you that feeling of a waste of time, and you're not receiving any inputs afterwards, that would be the bad auditor.

And we'll get to Jeff in a second here, but I think it's Jez Humble says, "If it hurts, do it a lot, or do more of it." So we're going to release on GitHub today. No? You're releasing it today? Soon.

Soon. Soon. We're going to release it soon. A toolkit that causes you to get audited every single day of the week.

No. But one of the founding parts of DevOps, at least the cultural transformation part, is the empathy of having the once adversarial relationship between dev and ops, learning that if you can hug it out, you can have better outcomes together. So similarly, I guess this is a good transition to you, Jeff. We can all work together, right?

So how did you normalize, or what's your approach to normalizing the stability he wants, the valuable audit he wants, and the speed and disruption that Byron wants? Good question. The big theme, if anybody hasn't noticed here at the conference, has been breaking down the silos and the change management and getting people to work together and collaborating and all that stuff. So just like you can do that with dev and ops, or you can do that with sec dev and ops or dev, ops, and sec or whatever order you want to do that with, you could also do it with auditors.

You're not a bad guy, James. Cool.That by collaborating with your auditor and going shoulder to shoulder with your auditor, and understanding that they're in it, at least the good ones are, are in it to help your business, you can make those audits a lot less painful and frankly, a lot more valuable. So if you understand the why when James comes in and starts asking questions and looking for what practices and policies and procedures, and controls you have in place, then you can start reflecting those in your business a lot better. You can start proving to him, and proving to your business that you have those controls in place, that they're effective, and that they're working as intended.

And at the end of the day, what you really get is, that the risks that are really important to manage and mitigate for your business are actually getting managed and monitored effectively. So we're going to take... This is an ask the auditor section, so start thinking of your questions, if not coming to approach to ask your questions. But for one that has come up for the first two days is everyone seems to think that the elephant in the room is we lose separation of duties.

So what's the fact and the fiction behind, what are the common tension points for audit in a DevOps environment? That's to you. Yeah. So, separation of duties.

How many people are familiar with separation of duty? Okay. How many people have actually bumped up against that and said you can't do something because we have to maintain separation of duty? Okay.

I think more hands went up when you just said that last part. Yeah. Yeah. Something that I've learned from James, who's been doing this for a lot longer than I have, is that separation of duties is not actually written anywhere in any law or policy or regulation, at least from the federal government or- The auditor would say, I want to use the word all in that sentence.

Okay. Okay. That's another thing I've learned from James. But it has been a traditional control that people have used to prevent the bad things from happening that those regulations and laws are trying to prevent.

So going back to the point of trying to understand the why, understanding the why those policies and regulations, and rules are in place in the first place, you can actually walk those forward to different kinds of controls, than separation of duties, where you can achieve the same objectives without necessarily using the same controls, like separation of duties. So you can use your tools, you can use your processes, you can use automation to achieve a lot of those same objectives. This could be to you, James, but what's another common point of tension where DevOps collides with the preconceptions about audit? So I think if you had a...

So we have separation of duties. Change management is an area of confusion in the audit side of the business. And that's, I would, maybe that meets your criteria in the question, but I think I would highlight that because when we do DevOps and you have the different pipelines and you have the automated scripts, and then people are adding scripts and really improving that process. When the auditors, not the great ones, are asking questions, saying, "Hey, I need to see your change approvals," and they ask for 25 over the last 12 months, and then you send them and they're all automated script outputs, right?

So you have millions of these automated approvals, and then you're sending it. It doesn't actually address the risk, which the auditor has been assigned to kind of reconcile, right? Because that's not how your change management processes really function anymore. Peer reviews maybe have been modified.

So I would focus on separation of duties, change management, and it's really the lack of understanding of the intersection of the different pieces. And the other side, which is an interesting one, has to do with the interplay of the different parties. So as you introduce different technology, right, we go to the breaks and we walk around, we see lots of different tech that's helping us, enabling the process. But how do we know that the pieces are fitting together and how do we communicate that?

So that's one of the challenges of DevOps is the speed and integration. We're using so many different providers, so you're kind of building upon assumptions, and then how do you communicate that assumption in a way that somebody else can kind of gather that and achieve the result? Yeah. For you, you've gone through this process pre-DevOps and are morphing your practices to adapt to DevOps.

What was the biggest surprise or change for you? Yeah, I think it comes down to, we talked about, separations of duties and change management. The other one that has been a challenge for us has been management oversight, and that was something which I didn't understand until we got a little deeper into the auditing process, was how much value auditors and examiners place on management oversight. And so straight with the idea of continuous delivery and then even the scary continuous deployment, you have this idea that code is going out all the time and how could your executive team really know what's going on?

How does your compliance officer really know what's going on? And so that's something that you can't really automate. You have to actually figure out a way to make that work. And one of the things that we've done is we have our sprint planning meetings.

We created a sprint planning review meeting, and we sit down with those key stakeholders and explain what we're doing. So, to a certain extent, we've had to throttle how fast we really can go. But it is important, especially in the financial services area, to be able to show that everyone is on board with what you're doing. Now, you guys have built a toolkit.

Who's best to start to describe the toolkit? What's it called? All right, I'll take this one. It's called the DevOps Audit Defense Toolkit.

There's a Google Plus community that's out there where there's some good traffic and a lot of resources around how to do DevOps and audit together. How do you audit organizations that are using DevOps practices? And there's a link to, right now, a draft document that we think is going to go final here shortly. We've just got to put the finishing touches on it.

Yeah, and I want to highlight that the idea of the toolkit is to describe some of the challenge areas, configuration management, separation of duties, and these classics, and then translate what that would be in a DevOps. So it's that conversion between the two. And then if you're looking at your organization and it's mature, to go through that conversion process, it would be, so year one, you're sticking with the old representative of evidence, the classic controls. And then year two, you would do them both.

And then year three, you would be full turned over into the DevOps style of evidence. Yeah, there's a lot of hope for this. I think one of the, again, I mentioned earlier that we've got these 3,000 banks that are coming to us, and this is something where we need the whole community to come together because there isn't a common language. And one of the scary things that we see are these requests for information, and it's 50, 60 pages of questions.

And so having a toolkit out there which could become the de facto standard that if you can accomplish the goals and the controls that are laid out in that document and have a wide adoption, I think there's a lot of benefits that could come from that. But we are going to need every one of us to help chip in and work on this kind of project. Just a reminder, we have a microphone up in the front. Damon's walking around, and this is your one time to ask the one auditor.

Anybody else? Raise your hand if you want to ask a question, and I'll come find you. Or even a testimony of how you've solved one of these problems. Hold on.

Exercise two. I need it, actually. So do I. So I work in healthcare, and we're FDA regulated.

And we've been trying to work through the DevOps pipeline and pushing it through, and we're getting a lot of pushback from our internal Q&R on the gatekeeping and controls. So we go from check-in, testing, and we have all these gates that things have to go through. What sort of documentation and artifacting do auditors like to see for automated records? I guess that was one.

So when it comes to documentation, I think number one, it's substance over form. A lot of organizations, they question the procedures and the policies. Do they have to be a certain formal way in a Word doc or published? You need to have the procedures, and there needs to be a clarity of flow.

Because a lot of times, as I said in the beginning, there's a lack of education and an awareness as to how those gates work. And then when it comes to the type of evidence, what's nice about the automation that exists is that they're generally creating the audit logs by their very nature. And what we need to do is we need to add in simple file integrity into the logs, number one, that you're capturing, so you can represent that they haven't been modified. And then number two, need to highlight the points that are relevant.

So having gates are good, but if the gates are automated triggers based off of scripts, well, then you have to demonstrate how the different gates are designed and how they satisfy, and they have integrity in themselves, the requirements of the FDA space and that. But in other spaces, you would have compliance and fraud if you're in different spaces, or security. And so when we look at things at scale on an audit basis, and many of my clients are on the West Coast and working at scale, we focus on the gates and the systems that are distributing it out. And because if you focus on that, if I can make sure that is right and you've got good conversion and management of that process.

So say, for instance, you do a deploy or you do X to Y in the process. If the thing in the middle doesn't have stability or you can't vouch for that, well, the scripts didn't change, but we have no idea who has access to that one system that runs all the scripts. Well, then I have a problem trusting it. But if you can help bring confidence into the components that make up those gate process, then you'll have a much simpler conversation because then it's a sample of one.

Because I got to make sure that one thing works, and I do a unit test through there to confirm that it follows the logic that you documented, and then I don't need to see the 15 million approvals. So I hope that answers your question. Yeah. All right.

Yeah. Other questions? Can't see any other one. Anybody else?

Yeah. I'll jump into the question, actually, then. So we hear a lot about evidence gathering. That's the part that it's never the requirements that scares organizations.

It's the, we're going along with our normal work. Suddenly there's an audit coming up. It's the evidence gathering or evidence collection that is the all hands on deck scramble, drop everything, get this done. Everybody sweats.

Can you talk more about that process when it's either done well by a company or done not so well by a company, and any kind of tips or top advice for companies who even have to sweat that process? Yeah, I think, so number one, when you get the requests and things that come out, I would understand that the requests are based off of what the business decided should be tested. And so I think number one thing fundamentally this is that the audit is being done to address the risk that's hitting the organization and the controls that are designed to prevent it. So when you receive requests for evidence, it's designed to substantiate that those controls are effective and find weaknesses in there to help improve your own processes as a result.

So when you think about it from that level, that must mean somewhere in your own organization is aA listing of the control procedures, the objectives, and the type of things that are in scope. So I would suggest working with your own internal teams to gather that in advance, and you could introduce that into your systems on an automated basis. I know many clients where they will actually push it to a separate repository or have scripts pre-built so when the audit requests come, because sometimes they can be many pages long, from your vendors, the customers that are paying. But when an auditor comes in, remember, if they're hired by you, they're there to serve a purpose for you.

If they're from a vendor, then they're looking for certain verifications. And so I would look at, number one, try to get in a list in advance, and then you can then manage it into the future pretty easily. And then when it comes to the vendor request, on average, that's roughly a 40-hour hit to any organization. It's kind of the math.

If you get asked from a vendor, you add all the hours in for the meetings and the evidence collection, it's roughly 40 hours of impact. So that can add up very quickly to a sprint. And so for those, I typically see, I'll call it a gauge, like a person or a team that sits in front and has a consolidated set of evidence that they respond as a package. As I would answer it from both perspectives of both kinds of audits, one is the internal, where you're paying for it, and then one is the third party vendors that are coming.

And then that way you can bring efficiency, and you don't have to continually redo the same work that would happen as a result. So I hope that answers that question. Yeah. To piggyback on that a little bit, we've done some of the same thing, and you have to take advantage of the resources that you have.

So we all have all these wonderful engineers, and they want to automate everything. They want to be able to get to production as fast as possible. You also need to redirect their energy and focus on audit if that's an important aspect of your business. And so, we actually put it into everyone's job description that you are responsible for assisting with audits.

And that way, that is a focus of the automation that we're doing. So it's not just continuous delivery of getting software into production. It goes back to some of the earlier talks that we've heard over the last couple of days about what is important to the business. And to our business, getting through audits, exams, due diligence is critical.

And so you got to focus your energies of your engineers on that kind of work as well. And I would add that being in the engineering side, the best thing you can do is put it through some of your test and dev cycles so that you're not practicing in production. That way, you just get an insight into what's deliverable, and then you start building that into your product, which is kind of cool because you can look at it from risks. If you're in a web presence company, you have risks beyond the audit, but the same data that you collect is just as valuable, and if you can put that into your product through the pipeline, then whenever they come and ask questions, you have an answer, and it's a good thing to do.

And just to keep coming back to this idea of trying to get everybody speaking the same language, the audit isn't the risk. Getting through the audit isn't the risk. The audit is there to manage the risk and making sure that the business is doing the things that are important to the business- Right ... making sure that bad things don't happen.

So if you have that sort of perspective, then the approach that Simon's company took of making sure that compliance and audit are baked into everybody's job is absolutely the right thing to do because helping the business succeed, we've come from that. Helping the business win is part of everybody's job, right? Yeah, I'm going to say some of this in my talk tomorrow, but I think if you look at the patron saint, I think John Willis calls him the Shakespeare of DevOps. But essentially, Deming, who saved auto manufacturing with the Toyota supply chain model in Japan post-World War II, that's the godfather that leads to things like the goal, leads to things like lean and agile and what we're now encountering.

I think one more turn of the crank is essentially a software supply chain approach. And not only is that going to be better for the auditors that we have continuous governance, right? We talk about continuous integration, we talk about continuous deployment, but this idea of continuous governance where you know the quality the suppliers are using, the quality of the supply from those suppliers, where that code has made it, who pushed the button to deploy. That level of traceability is going to actually make it a lot easier once you normalize the interfaces with auditors.

And it just so happens to also make you go faster and reduce operational complexity and code complexity and elective attack surface and security debt and technical debt and all these wonderful things we're trying to get rid of. So I don't think it necessarily solves itself, but the one fear we talked about over lunch is if you let your auditor dictate your agenda, especially if you're in a highly regulated organization where you might have conflicting regulatory, one of the questions from HIP and high-tech, normalizing all these different audit and assessor and examiner nitpickers can probably distract you entirely. So it really has to subordinate to your business goals. We have four minutes left, so it's probably enough time for either one more question or some closing remarks.

Two in the middle. Here we go. Oh, oh, oh. Here.

Just thinking about separation of duties. Devs want to try a new tool, and the Windows guys in the operations have to do the installation of a Python library or something like that. So they're opening a ticket, and then 24 hours later, maybe they finally get it done, and it could've been done in 15 minutes. So I'm wondering how auditors are...

You say that separation of duties is not written anywhere, but I sometimes see a lot of- procedures that seem to reflect, be hiding behind that auditing requirement. Can you comment on that? Oh, they're definitely hiding behind it. Because think of it this way, we do physical inspections of servers because there's a requirement of understanding the risks, the threats that it attracted directly.

But if you're sitting on a hypervisor, there's a risk that you had before equal to the risk you have today. So, I guess to directly answer your question, so you're speaking about, it goes under the phrase of system utilities being installed and out in a production environment, would probably be the classification where an auditor would probably start spinning. And so what I would articulate there is that let's assume you're using a configuration management platform, and then that has to go through some kind of check. And I think earlier we had a speaker that was talking about their Puppet Lab deployments were considered standard change, and then that would enable that flow.

So, the idea is that if the Puppet deployment or the NEX deployment has a process, and it's verifiable, and we can have confidence in it, and then that push passes the other components, then I think then that's where you can have that acceleration in speed. And so the auditor, if you install a coding library, let's say, well, you install anything on there that could change the risk profile, then it should activate other things. So manually what was happening before to what could happen in the Puppet scenario, just to kind of draw that one direct example, I think that's what is automated, and it's a maturity level. You're not simply saying, "Oh, we're going to add this," and then nobody has visibility into it.

That's not what we're saying. We're saying, "No, everybody has visibility to it," and we added all this other stuff that made sure it didn't harm anything. Mm-hmm. And then we deployed it, and if it had a bump in the night, it was completely wiped back to the old version in five seconds.

And auditors don't hear that. They hear that, "Oh, well, we deployed, yeah, we've got operating systems, and we got all this stuff happening here. It's great. NetCat, it's beautiful." So that's what we want to avoid.

We want to emphasize the process that was there and then the integrity of the process such that we can trust the results. And then obviously the rollback speed is something that most people don't even understand. So I hope that answers that. And then there's one over to the left.

Yeah. Yeah. Yeah. Yeah.

So I would like to talk about or understand the potential pitfalls, very specific to organizations adopting DevOps when you audit them. What do you see as the top three potential areas where they fail? Well, we have one minute. Do we want to have one person just say one thing, and then I can try to answer that quickly, or do you want to answer real fast?

Well, I got one. From the operations DevOps side, the first thing that always fails is we look at auditors as adversarial. Yeah. We answer them with yes, no questions.

I've worked at places where they told me how to answer the auditors, and we weren't trusted. So culture and how you work with your auditor, I think, is a huge thing that causes many people to fail. Yeah, saying yes, no during an audit, and no, obviously we'd never want to hear. So this is how I do it, or this is why we do it, and how we manage that risk is the better response.

So more exploratory. If I had to say the quick three things, would be maturity of the organization. Sometimes I see organizations try to accelerate into high DevOps scenarios where they lack the script, they lack the testing, they don't have the gates being managed with integrity, they haven't done those verifications, and they don't have all the parties involved. So for me, that would be the number one fail point is because I get in there and, "Oh, we do peer review, and we do this, and we have no feedback loop.

Oh, and we have five scripts that we run." And security has no idea what you're talking about. So then the pyramid starts to fall down. And so I would maybe answer it in bulk. I think just to give you the third is surprising your auditor or your examiner.

So if they're coming every year, and one year you're a waterfall shop, and the next time they come back, and all of a sudden you're continuous delivery and agile, they're going to probably freak out a little bit. So you have to talk to them ahead of time. They're usually there. They're available to you, so reach out to them and make sure that they're aware of what you're doing before you do it.

And that way they've got a chance to bless it. So the same way that ops doesn't like it when dev surprises them with a new feature, you got to apply that same mentality to your auditor and your examiners. All right. We're getting the hook, but I guess that's a good point to end on, which is DevOps works because two different groups with competing incentive structures found a way to talk to each other, express empathy, and find a way to transcend.

So do the same thing with your auditor and build that relationship. Thank you for your time. Thank you.