Sam Guckenheimer and Topo Pal

- Sam Guckenheimer (~0:02): So I'm Sam Guckenheimer. - Sam Guckenheimer (~0:04): Welcome. This is six years in the making, that we have a chance to get all of the big four in one place for you to ask any question you want. - Sam Guckenheimer (~0:19): I'll be in the hall with a mic, so that we don't need to play the repeat the question game. - Sam Guckenheimer (~0:28): Topo, why don't you go over the ground rules? - Topo Pal (~0:30): Yes. - Topo Pal (~0:31): So welcome. And I think we want more people, and I expect that more people will come in after John's stuff is done in the other room. - Topo Pal (~0:43): But in the meantime, I think we got enough to start off. - Topo Pal (~0:47): For Sam and I, this is like our dream moment, and we had been trying this for about four years now. - Sam Guckenheimer (~0:54): Low expectations. - Topo Pal (~0:56): Yes. - Topo Pal (~0:57): So this is kind of, in our view, the last hurdle in our DevOps transformation, our DevOps journey. - Topo Pal (~1:04): We brought in business, we brought in security, we brought in ops, and everybody involved in the development life cycle for our DevOps transformation. - Topo Pal (~1:15): And I think this is the last one, last hurdle that we need to cross, to make it really happen. A couple of years ago, we, as a team in Jenkins DevOps Enterprise Summit Forum, we published a love letter, Dear Auditor, and I think that got good traction. It was kind of an olive branch to the auditors, "Please come and help us. - Topo Pal (~1:40): We really want to work with you." So the result of that Dear Auditor love letter movement, if you want to call it that way, is this. That we have all the people who can actually help us on the same stage. - Topo Pal (~1:56): And for us, it is, again, our dream moment. - Topo Pal (~2:00): So before we start, there are some ground rules. - Topo Pal (~2:04): For the camera, we know that they are not focused on the audience. They are focused on actually the stage here. - Topo Pal (~2:14): And we are not going to introduce ourselves. - Topo Pal (~2:17): All you need to know is that they are the actual people who can help you and help us. And we are just two developers, right? - Sam Guckenheimer (~2:25): That's right. - Sam Guckenheimer (~2:25): And we are going to ask some dumb questions. - Topo Pal (~2:28): And then you are going to ask any questions that you may have in your mind in terms of audit and compliance and how the DevOps transformation is taking place. So the ground rule is you raise your hand to ask a question. Do not introduce yourself or where you are from, unless you want to get into trouble. - Topo Pal (~2:47): Raise your hand ahead of time and one of us will bring you a mic. - Topo Pal (~2:50): Right. And then ask the question. Do not think about whether the question is relevant or not. - Topo Pal (~2:56): If you have a question, please ask, and they're going to try to answer the right way. - Topo Pal (~3:02): Before we start, we can do some icebreaking. - Sam Guckenheimer (~3:06): Sure. - Topo Pal (~3:07): Right. So we have some questions. You want to ask the first one? - Sam Guckenheimer (~3:11): Yeah, so true or false, audit adds value to software delivery. - Topo Pal (~3:15): I think this is the question for you guys. - Sam Guckenheimer (~3:18): Who thinks audit adds value to software delivery? - Topo Pal (~3:22): Don't feel stressed out that all of us are staring at you. - Sam Guckenheimer (~3:25): Okay. So for those of you who didn't raise your hand, who wants to argue that it's of no value? - Topo Pal (~3:37): It's okay. - Sam Guckenheimer (~3:38): No dissenting opinions. - Sam Guckenheimer (~3:39): So those of you that did raise your hands, who wants to argue that it's valuable?

Q&A

01Does audit add value to software delivery?

- (~3:50) Yeah, thank you. - (~3:52) So I'm not saying who we are, but we're in the medical software. It's like our software is a medical device, regulated. - (~4:02) And the regulations basically just want to ensure that the software can be used safely and as intended. - (~4:09) Like that's basically all of that comes to that, right? And we find that in order to ensure that, we need to have our own internal control so we can keep our customers happy, and we care about the patients at the end. And so we're totally aligned with what really the regulations want. - (~4:29) Of course, there's a little bit more bureaucracy when you deal with particular regulations, but that's what we're struggling right now with. - (~4:37) But we think definitely that auditing helps us achieve the goal we are after anyway. - (~4:46) Go ahead. - (~4:49) So I actually raised my hand to say that no, I did not think it added value, but I want to make sure that I understand the question correctly. - (~4:56) Adds value to software delivery, was that the question? - (~4:59) Yes. - (~4:59) Okay. I think that auditing adds value. - (~5:01) I just don't know that value is in the software delivery necessarily. - (~5:05) That's a not very strong opinion, held very loosely, and I'm willing to be convinced otherwise. - (~5:11) So I'll take a first pass at convincing. - (~5:14) One of the largest churns in software delivery is rework, right? - (~5:20) The amount of times as... And I said this earlier, I consider myself a recovering developer. I'm on the advisory side and help translate between the auditors and technologists. - (~5:32) The amount of times I, as a dev, reworked the same stuff was astronomical. Now, if audit is part of that internal audit, IT audit, and compliance and risk controls are part of that, they're going to catch it. Right? So if I make a mistake that impacts a medical device or EMR, and they can catch that compliance thing before it ever gets out of my user story into code, that really helps my software development life cycle. - (~6:01) Because all of a sudden I'm not-Wasting efforts, things like that. Because I can only know so much as a developer. - (~6:09) I'm not going to be a compliance expert, but we have compliance experts who can help with that. - (~6:13) That's my shot, at least. - (~6:15) I would expand it a little to say it's smart auditing. - (~6:18) Auditing that's understanding the methodology, understanding what it is you're trying to do, and adapting the audit approach to that. - (~6:25) And so not bringing old-fashioned controls and techniques, but actually working with the methodology, working with the way that you're working, to essentially not be part of the quality check, because obviously quality needs to be part of the dev team, but where you've got a lot of overlap between what audit's doing and what quality's doing. - (~6:45) I have another concept on this that I talk to developers about, and that is, what if you, as a developer, your annual performance was based on the quality of the software that you delivered. Would you feel the same way? Would you care about quality more? - (~7:01) Would you want to get all the help that you could possibly get in doing your job? - (~7:06) When you take it down to that level, all of compliance is based on quality. - (~7:12) And if we all own quality equally, that really helps bring everything up to go quicker and faster, and that's what we all want. - (~7:27) I'm interested to ask the panel, for the audit profession, and compliance professionals, what do you think is the biggest behavioral change that you're seeing is needed or you're seeing is happening or is not happening? - (~7:46) I'm interested around the culture and the behavior. - (~7:53) I'm not one of the auditors in the room, and I want to address that and maybe be powerful since I'm on the other side of the fence. - (~7:58) I'm a security person first and also a recovering developer. - (~8:04) What I've seen, at least from a cyber side, is that along the lines of what the folks were talking about here on the last question is, we have a lot of people with the same skill sets, especially in internal audit, that are all trying to achieve the same means to an end. Yet throughout most of my 20-year career, they were pitted against each other. - (~8:31) At places, particularly where there's cost pressures, I'm seeing a lot of motivation to pool resources on, I would say, probably activities that were a little further down the chain, and they're realizing that as they shift left together, there's a lot of economies of scale to be had. - (~8:49) And especially when you're talking about some things that are in the security space where there is a war for talent right now, I'm sure we're always battling each other on that front.

02What behavior change is needed in audit and compliance?

- (~9:01) There's a lot to be gained there. So I see the culture shifting to be more inclusive with both groups and realize a lot of those efficiencies for a variety of different drivers. - (~9:12) So you used the term internal audit. - (~9:15) I know that a lot of people have questions around that. - (~9:17) What is internal audit and external audit, and how do they play together? - (~9:21) I think you started to broach that question. - (~9:23) Yeah. Sure. So we're saying auditors very generically. Auditors are a lot of different things. - (~9:30) The main distinction is internal audit and external audit. - (~9:33) External audit are the people who are coming in and reviewing your financial statements and giving an opinion over those financial statements over whether the information contained in those is fair and true. - (~9:44) And that is presented to the market, to the SEC, or whoever your investors are, to help give them some comfort about the accuracy of those financial statements. - (~9:54) Those internal auditors are kept at arm's length. They need to be independent. - (~9:58) They're called external for a reason. - (~10:01) And so the role that they can play in this process, especially in the front end of the process, is relatively limited. - (~10:07) However, they will audit, in particular, systems that help generate those financial statements or feed into those financial statements. - (~10:18) Your internal auditors, and I'm saying that very loosely now to bring in your compliance groups, your IT compliance, your security teams, anything that's internal to the company, they still may audit you. - (~10:31) However, they don't have those independence restrictions. - (~10:34) They're not reporting out to the market. - (~10:36) And so they can give a lot more advice, and they can be in the room a lot more with you as you're creating and designing controls or setting things up. - (~10:43) So they're the two main distinctions. - (~10:45) There's a lot more distinctions, but that's the two different paths. - (~10:48) And internal audit can play a lot more of a helpful role along the way, whereas your external auditors need to be kept at arm's length. - (~10:56) Just to add to what Matt was saying there. - (~10:58) I think to expand the external too, a lot of it's obviously external auditors versus the financial statement auditors. - (~11:06) But also, I'm sure a lot of you issue SOC 2 reports or SOC 1 reports where these products might not be in scope for financial audit purposes for your external auditor, but you might have it in scope for the product you deliver or the device that you're creating. - (~11:20) So a lot of times, those also are also some of your audience on the external audit. - (~11:24) But just to jump on to add to your point, how has the mindset changed? From an auditor's perspective, I think where I've seen the whole DevOps models really pushed us, is really auditors have to get away from a checklist-based audit and really are forced to really understand the business and the flow of transactions through the systems. - (~11:46) I think more than ever, we need to really understand how transactions flow and how things work through the whole life cycle of the system to understand where the key risks are, and then working closely with the IT organization to understand how are they mitigating, how are they grading regression test or unit test that mitigate some of those risks that we really care about. - (~12:05) And just to bring that home, when we start, and Amy could probably give some great soundbites on this, when we scope an- The audit for a product to do a level two type of internal review.

03How do internal and external audit differ?

- (~12:15) The first thing as auditors, whether you're the business risk or the tech risk, that we ask for is we go to the architects and we say, "Show us your proposed architecture and walk us through the tech stack and all the services that go into it." That's where the risk starts. - (~12:30) Let me jump in and say the question was about culture. - (~12:33) So does anyone want to do a good culture versus bad culture? - (~12:40) I can do a quick soundbite on that. So from my perspective, a good culture is how do we get to a place of yes? - (~12:46) There should be yes on all sides, right? - (~12:49) The business has an objective. IT, you want to get us there. - (~12:52) You know some of the best ways to do that, and compliance wants to enable that. - (~12:56) So how do we get to our state of yes? And bad culture would be you coming to me and me saying no. I don't ever want to have to say no. - (~13:04) That's not a good feeling. It's not a good feeling for you, and it doesn't make sense for enabling the business and for what you guys do. - (~13:10) With technology, there are a lot of great ways to make things happen. - (~13:13) There's a dozen different ways to perform a control and mitigate a risk. - (~13:17) Let's do it together and figure out how. - (~13:18) Yeah. The other important piece about culture, and this is a gong I've been banging for the past several years and very much today, is about empathy and understanding, right? - (~13:29) So how many people in this room would consider themselves a developer? - (~13:35) Okay. So if I asked all of you 10 years ago what a UX person did, you would've said pretty pictures. - (~13:43) Pretty picture people. But now, if I ask you, what's a journey map? - (~13:47) Yeah, that's something you know, right? - (~13:49) We absorb that into our culture. Now, if you go and ask the average internal auditor, "Explain to me DevOps and Agile," they'll say, "Developers have access to production." - (~14:03) Right? - (~14:04) And segregation of duties. - (~14:05) And segregation duties and no documentation. - (~14:07) It is on our onus to have empathy for what their problem is and how we communicate it, and inverse, right? - (~14:14) That community opening up and going, "Tell me," as you were saying, "through the architecture, how does this work?" And you can go through and talk about a lot of different things around culture about what is the behavior we want when something goes wrong, right? What's the behavior we want when something goes well, things like that. But to me, that uncommon understanding and not seeing them as the office of no, right? Which is often the perceived perception.

04What does good culture look like?

- (~14:38) Yeah, looking at it as a partnership. - (~14:41) So I think I am the one that raised hand for both of it. Does it add value to software delivery? - (~14:47) I raised hand saying yes and no. Right? - (~14:50) So here is my experience. - (~14:52) So when we started doing continuous delivery into production, right? - (~14:55) The first thing is we go to the infrastructure team and say, "Look, we want to do this continuous delivery." They go, "You need to go talk to your audit and compliance people." What did they tell me? They told me, "You know what? You cannot do that. - (~15:09) You need to have segregation of duties or compliance," it goes on. - (~15:13) And I ask them, "Why does it matter?" Because all we are trying to do is make sure that wrong, malicious, whatever, things does not get into production. I ask them, "So why can't I just show you what is the compliance route? How did it go from dev to production?" The answer was... - (~15:31) Right? So we have to go from that place to a place where we are today, and that was a very, very rough journey. So does it add value to production? - (~15:39) I think it's in context. It's in context of the people that you are working with, in context of how much they are aware of DevOps and this continuous delivery, right? - (~15:48) You guys have beer in your office? Maybe that could help. - (~15:52) So you got to take down these mental silos, right? - (~15:55) If everybody has the common goal of delivery and providing value for the business, and if leadership on the business side stands up and says, "We need this technological approach to drive value for the business and to grow revenue and to get us to where we want to be strategically," DevOps is now a strategic priority for the business. So don't just think about the compliance people as the burden being on your shoulders to influence them. Look who influences the compliance people other than the regulators. Have the leaders of the business and the CEO and whoever - (~16:29) else you need, like COO type of person, make it their priority. - (~16:32) Yeah, it starts at the top. - (~16:33) Let me just- All right ...say, what do you say to the gentleman's experience where you're in a situation where your internal audit is saying, "No, you can't." - (~16:47) How do you work with that? - (~16:48) Well, if you happen to have an external auditor, they could be an ally, right? Because when the external auditors come in, they're looking at the work that the internal auditor does as well. - (~16:57) I spend a lot of time, as all of our competitors up on the stage, just talking to clients from a mere educational element and explaining to them what is DevOps work? What's this new type of process? - (~17:09) How are organizations utilizing this approach to transform? - (~17:13) So once you play that a couple times, the repetition makes it start to sink in and people ask intelligent questions. - (~17:20) You'd be surprised. - (~17:21) Hello. - (~17:22) I have a question here, but before that, one last bit of it. - (~17:25) I think the problem that you're saying is felt by many developers. - (~17:30) Developers do not directly talk to auditors. - (~17:34) That's exactly one point that I wanted to add, is we don't know who the external auditors are. We have no ally. - (~17:41) You got eight people right over here that are here to help you now. - (~17:44) Good to know. Yeah, and I think that's also just finding those folks within your organization. We talk a lot at this conference about the engineering-focused change and modernization, right? - (~17:53) That same exercise has to happen within your organization on the compliance side as well. And so it's finding those allies within your organization, and then it's taking the time and energy to also educate them as well. Because a lot of them, unfortunately, haven't, in a sense, to use Gene's words, seen the light as it comes to this here. And unfortunately, those first conversations ... can be rough, but it's sticking with it. - (~18:20) It's helping them understand the impact to you and your business and the need for them to be supportive of that end goal. - (~18:30) I think it's actually nice that you guys went and asked. - (~18:33) I have clients where they tell us, "We are going DevOps. - (~18:35) You're the auditor, get comfortable with it." - (~18:38) So we've sat down with the head of engineering saying, okay... - (~18:41) And this comes back to my earlier comment, where we really have to understand the business and really what are the key KPIs that you guys are committing to your clients, and also from your own products, like how are you recognizing revenue, all those things. So we sat down with the head of engineering and said, "Okay. If you're going to go down this model, here are the key things you cannot get wrong." And so we sat down with them, and they created unit tests or regression tests that specifically test that functionality every time you push code, and - (~19:06) that's how we got comfortable with it. - (~19:08) Yeah. I'm also a security guy by trade, and we have this experience that I think has gotten much better in recent years where security is considered, and we now have that DevSecOps name or whatever acronym you feel like using around that. - (~19:21) But in the same way that we would say, "Hey, dev team, you should probably talk to the security folks earlier in the SDLC to make sure that you bake in the right things." - (~19:29) And I say, maybe that means cutting a ticket so that every six weeks or whatever periodicity makes sense, you have a conversation. - (~19:37) It can be that tactical, and I'm exaggerating for the sake of storytelling. - (~19:40) But hard coding those conversations with the people, in this example, IA or whatever auditors we're talking about, at least in the beginning of our journey, can enforce the right behaviors and the muscle building between those two silos. - (~19:54) Just change the culture from the top. - (~19:55) Those of you that have influence, the way I always say it to people is, look, you need to change the mindset from it being any compliance function, whether that's kind of internal audit or otherwise, change it from being a tax to being a competitive advantage. - (~20:11) If you kind of live that mantra, that brings together the culture. - (~20:14) All these great points people are saying, that kind of seems to resonate because nobody likes to pay taxes. - (~20:21) One, two, three.

05Are COBIT and older governance frameworks still relevant?

- (~20:25) So much of governance, or at least a lot of the conversations I wind up in, at some point start to mention COBIT from ISACA. I'm wondering to what extent any of you are active in that. And in particular, even COBIT 2019 is still based on plan, build, run, which is essentially a statement of waterfall development. - (~20:52) I'm developing the opinion that plan, build, run is no longer suitable as a basis for governance. - (~21:01) So the question is two parts. Is COBIT relevant? - (~21:04) Should it be changed? - (~21:06) So we talked a little bit this morning about you have a control objective at a high level, and then how you implement that may vary depending on your delivery model. - (~21:18) I'd say the holistic principles that you just nicely articulated, you could apply those to a DevOps world. - (~21:26) It's just repeating itself. In Microsoft's case, what is it, 85,000 deployments a day you guys do? - (~21:33) So, it's about the application and how you implement the principle. - (~21:39) Even like accounting and basic auditing standards, they're principles with a loose guidance of how to apply them. - (~21:45) The facts and circumstances that you apply them to will dictate if you met the objective. - (~21:50) Yeah, I would agree with that. I think the one thing it does is it really pushes the boundaries of the tools you're using because you can still do that same waterfall approach as long as you're using the tooling to kind of automating the whole process through that. - (~22:02) I will add, let's talk about the straight-up frameworks for a second, right? - (~22:07) COBIT has its issues from a, to your point, the somewhat dated methodology. Now people have found ways to satisfy those aspects of it in a modern realm, and you can also see the progression. - (~22:21) So like what is the latest, is it ITIL V4 is what they're calling it? It's basically DevOps, right? - (~22:27) You look at that framework and you're like, "Yep, connect the dots. - (~22:31) Cool." - (~22:32) There is an industry shift going on, right? I argue that right now we're at a key nexus point, right? I described it earlier as like we went from being a garage band, punk band in the garage to now we're on a major label, right? We're at that sort of nexus point, where I think a lot of the things are starting to change. - (~22:53) You wouldn't have us up here having this conversation unless this wasn't the moment. But when you look at that moment, there's this uncomfortable pivot, right? Where some of those things of like, oh, that framework that I said we're going to follow doesn't match the thing. - (~23:08) And I think you are at that point where it is shifting and because it's being more iterative, it's much more about these interrelationships and understandings. - (~23:17) Well, and I would look at that from the perspective of that's verbiage on paper, and it may function that way, but you have to look at it from the other side of what's the risk that it's trying to mitigate. All those frameworks are driving towards risks. - (~23:32) The risks haven't changed. Our verbiage that we use around them, the processes and procedures, the controls we put in place to mitigate those risks, that's what's changed. And so articulating that back and pushing back on them saying, "Here's the risk still, and this is how we're mitigating it." - (~23:47) And the best way I've explained this to coders before is I think most of us probably work at organizations that have data sovereignty or regulatory challenges when you're doing business at a global scale. - (~23:58) And you have so many different countries around the EU, around the world, and they have their own data postures, their own data rules. - (~24:06) If you go to certain jurisdictions, specifically in Europe, they may have some archaic data rules that have been on the books since the '60s or some since World War II, right? And how do you apply these old regulations that have never been updated to comply with the spirit of the law? - (~24:24) So you have to think about how do you comply with the spirit. - (~24:28) I think we had a question there. - (~24:30) So hello.

06Why can audit not publish one fixed DevOps checklist?

- (~24:32) I guess where my struggle is, is with the way that this engagement is. - (~24:41) And let me explain that. - (~24:44) The developer community is expected to get non-functional requirements, very clear. This is what you do. - (~24:52) This is the chain of custody. This is the segregation of duties. - (~24:55) These are all the requirements that you need to fulfill. - (~24:59) And that's very clear, right? These are all the rules that you must follow. - (~25:07) In contrast, the auditors that we heard from this morning, four of you, said that you believed in DevOps values, and you thought that DevOps was a good framework for audit. - (~25:24) After that, a lot of conversations that I had, people were saying, "That would be an interesting video to play to my auditor because I can bet my auditor probably will not agree with that." - (~25:38) So why is that a- The internal auditor, you mean Why is that a critical mass that needs to be achieved? - (~25:46) Why is there not a clarity there? Why is there not a standard body for auditors who could be convinced and who could say, "These are our standards, these are our rules. - (~25:56) These are the rules that we live by, just like the developers are supposed to live by a certain set of rules. - (~26:03) These are the things that we will accept. - (~26:05) These are the frameworks we will accept. - (~26:07) These are the changes to the COBIT rules that we're going to make because we understand that it's outdated." Rather than achieving a critical mass or talking to enough people and making enough change happen, or making sure that your auditor likes you or having to buy them beer. Why is that the conversation? - (~26:29) Your auditor should be buying you beer. - (~26:31) I just want to be clear on that, but you're taking a binary view as a coder, as a programmer would, to give me the NFRs, and I'll implement them. But that's not how it works because your data sensitivities are different. - (~26:47) Your different services are different. - (~26:50) You may get some levels of coverage from a SOC report from your cloud provider for some services, others you may not. - (~26:56) So there's all of these factors that come into play, and depending on the type of functionality and the type of compliance mechanism that you have in place, somebody mentioned medical devices, somebody may have to comply with the confidentiality principle from a SOC report or HITRUST or PHI. - (~27:14) How you comply with all those, depending on the tech stack that you as a developer want to use, and every tech stack and technology is uniquely different. So the approach on how to comply is never going to be the same with all these different differences. - (~27:28) And we want to empower you, the developer, to use the best approach that you believe from a technical lens is needed, and we have to partner with you to find the right answer to mitigate these risks. So it's never going to be give me the NFRs, and I'll implement them. - (~27:44) It's very much an art and not a binary science. - (~27:48) It ultimately comes down to the risks, and that's the shift in the audit world as well, from really, as mentioned earlier, that real checklist approach to what are the true risks in your organization. Some of those historical controls or checkpoints that you had may no longer apply because they're not providing any value. And so it's really that interaction between the process owners, in this case, the engineers, and your auditors to have that open and frank conversation of look, this is our environment, this is how we understand it, these are our risks, and this is how - (~28:21) we've determined how we're going to address those risks. - (~28:24) And so your auditor may have an opinion of whether yes or no that those risks are addressed. But now you're having a conversation around the substance of how you move forward to actually do this versus just the no or the hesitation that you've gotten in the past. - (~28:39) I think, and I use this verb on purpose, we empathize with the frustration. I think the eight of us on the stage here now are here because we want to be part of the solution with you and have our colleagues where we work be part of that solution, but it will take time. But as Sam and Topo opened up this conversation with, we'd like to think that DevOps Enterprise Summit is a signal and hopefully an inflection point to the outcomes that we're all hoping for together. - (~29:07) The other thing I'd say is, let's be real. - (~29:09) If you're a developer and you're at this conference, you're the minority, right? You're the minority of developers. You're passionate about this. - (~29:17) You're changing this. It's then you on the advocacy to go the other way. - (~29:21) So this year was a turning point for me. - (~29:24) I presented at two ISACA conferences. - (~29:27) ISACA is a bounding organization around IT audit. - (~29:34) I presented, including Guy Hubert, who is a-- His best title in the world, he's a risk futurist at Atlassian. - (~29:41) He's their chief head of risk. And we were both presenting about Agile and DevOps at an IT audit conference. And then I go, and I speak at DevOps World, and I go, "You guys should talk to the auditors. - (~29:56) We should bring this together." So just as much as it is them educating, it's also us as an advocacy group for tearing down a silo, go into those communities. And have those conversations because it is as much about that learning and understanding. - (~30:14) And they do have groups like this just as much as we have meetups and things like that. - (~30:21) So maybe next time you should invite some developers to come to your conference. - (~30:24) He needs to ask a question, Topo. He's been waiting patiently. - (~30:28) All right. So my question is kind of two-part.

07How do old regulator handbooks map to DevOps?

- (~30:30) First of all, have you had success having conversations about modern DevOps practices with the United States government, particularly the OCC or FFIEC on the financial side? - (~30:41) And then secondly, if you have, what has made that successful, and how have you reconciled their handbooks, which can be as much as 15 years old and very focused on waterfall methodologies? - (~30:52) And they come in and they have very prescriptive thoughts on what your risk and compliance and audit focus needs to be based on those handbooks. - (~31:00) So I have an interesting story on this one. - (~31:03) Because, again, I'm a financial auditor. I went to the technical side, and now I'm a risk manager. But we as an audit business, and if you think about how old auditors are, we have a several hundred-year-old profession. - (~31:17) We went through a massive transformation in terms of how we do our work, changing things, using enabling technology to really shake up how we do audits. - (~31:27) From a qualitative standpoint, from a values standpoint, and that's a big transformation. So a lot of the government agencies recognizing that the audit community has gone through a lot of transformations, have reached out to us, and I've found myself in multiple agencies having a discussion about transformation. What does transformation look like? How do you transform? - (~31:50) So our consulting practice is everybody's trying to help the government transform, and depending on the agency, they may have different commitments that they've made in terms of what they want to do. But that is being driven by the business. - (~32:03) When your business drives your transformation, the risk side of the house comes along with that and has to look at themselves internally and come to a conclusion and say, "Well, if this is what our leadership wants, what do I have to do, or what do we as a government IT organization have to do to modernize to meet the needs of the business?" So that should never be just on the lap of the IT audit and the compliance group, but it's really a business objective. And we have this concept internally that the cloud is not a technology strategy. - (~32:38) The cloud is a business strategy. So when you look at it through that lens, and we talked before about the tone of the top, that opens the doors, and the conversation just really changes. - (~32:48) But specifically to the OCC and FFIEC, those handbooks are old, and they are definitely out of date, and I think that's acknowledged. We've been through a couple of exercises where we actually take those principles or those requirements and map them through to a methodology like SAFe, and we say, "Hey, this is what you're trying to achieve here, and this is where we achieve it over here." And it's not the same thing. - (~33:11) It's not one for one, but we're achieving that objective. - (~33:14) So it's a bit of a slow process. You have to go through, you have to do that mapping, and then you very much have to take your regulator on that journey with you. So whoever's in the trenches with you, whether it's remediating a finding or doing an audit and actually work through and say, "Hey, it's not that we've just ignored it. We just think it's covered in a different way," and just kind of go one for one through. But yeah, it doesn't cover them off neatly, and so it's kind of on you either as the technology organization-- Probably as the technology organization to actually say - (~33:43) how you're achieving those objectives. - (~33:45) So you kind of pull it back to, rather than the specific activity, what's the objective that activity is trying to achieve, and here's where we achieve it, even though it looks and feels different. - (~33:55) I'll just add, too. I'm definitely not in our federal practice, but at least once a month, I am pulled into a pursuit with our federal practice because they're asking for our approach for Agile and DevOps. Right? So who's the design arm that the federal government spun up at some point, something 20, I can't remember. But they had built up- 18F. - (~34:18) 18F, right. They built up an Agile standard methodology that they rolled out from design thinking through CI/CD. - (~34:25) And every single proposal I get from Fed at this point is, "Talk to us about your DevOps principles and your Agile methodology, and what is the tools that you're using?" That is becoming standard expectations from Fed. Now, you brought up an interesting point, which is, look, we say DevOps like it's a single thing, right, and it's got a single definition. And what I'm going to do to implement DevOps in an organization that is a mainframe-based legacy application is inherently different than a functional programming Go cloud-native - (~34:59) Kubernetes thing. Right? But all of my tasks are going to be in a task management system like Azure DevOps or Jira. Right? I'm going to follow a standard process. - (~35:10) And even when I'm doing a legacy waterfall thing, I can still use those tools. We've been referring to this as a waterfall with modern tools to enable those sort of scenarios. - (~35:23) So we need to be really conscious of what are the delivery models within DevOps, not just going, "I put it on the DevOps, so now I'm Agile." - (~35:34) I just want to share real quick that it's not just the civilian side that's asking for support in how to transform into this way of working. - (~35:40) But we see requests coming in in the military side as well, which is exciting. - (~35:45) We do a lot of labs where we'll bring in the folks from specific government organizations and talk through it, white space it, do the mapping. - (~35:55) Again, you're going to try to take a substance over form approach, like was articulated in terms of the objectives. - (~36:02) And again, it's about getting all the people together, but that requires tone from the top. - (~36:09) Hi. So, I was wondering if the community as a whole, the audit community, the external audit community, to be precise, has a community of practice developing as well around either DevOps and/or Agile and/or Lean.

08Is there an audit community of practice around DevOps?

- (~36:25) And I'm asking because to put a shout-out to both KPMG and PWC quite a few years ago when we started our firstImplementation of an application that came with a tool set that did implement continuous deployment capabilities, though our processes didn't allow for it. - (~36:47) They were the ones that actually helped us teach our internal auditors, right? - (~36:50) So that was great. - (~36:53) And so now as we're really adopting it wholeheartedly at the enterprise, our internal auditor, at least there was one small group that actually understood what we were trying to do, and they've been really helpful. But I'm wondering if there is some community of practice or some organization, informal organization or formal, that we can point other people to. - (~37:15) Because it's great to say, "Yeah, go to your external auditor," but as other people have pointed out, number one, not all the teams have visibility to their external auditors. I did. I was lucky. - (~37:25) Also, we're heavily regulated, like that gentleman over there various government bodies who I was really happy to hear you say you've been mapping it to SAFe, but our arm of the Fed clearly doesn't know that based on the questions we've got this year. - (~37:43) So it would be really awesome if we could sort of point them. - (~37:48) So I think my colleague over here, Yosef, has an answer, but I want to repeat, she said KPMG and PWC. Just to be clear, that's the words I heard. Okay, your turn, buddy. - (~37:59) I heard Arthur Andersen. That's... Oh! - (~38:04) Just kidding. We got to have a sense of humor here. - (~38:06) But there is a consortium of internal auditors, it's called the IAA, and they do talk about cutting-edge technology. - (~38:14) Now, I know if Gene were here right now, he would stand up, grab the mic out of one of our hands and say, "Hey, when Sarbanes-Oxley came around, we actually helped write one of the chapters in the internal audit approach to testing controls under a COSO perspective." And Gene and a lot of his colleagues helped create that for the Institute of Internal Auditors for compliance. - (~38:36) Maybe there's another chapter that has to be added to evolve it. - (~38:40) But I would say cloud is definitely talked about within the internal audit space and within the industry. - (~38:49) Goes- I have a question over here. - (~38:52) Yes. He was raising hand. - (~38:55) You go first and then we come... - (~38:58) What guidance do you have if there's an IT team wanting to go to compliance for the first time, make an assumption that a couple of times it didn't go well, they just shut everything down.

09How should teams bring compliance along in the first 90 days?

- (~39:07) What would you recommend over a 90-day period to bring the compliance team on the journey? What should we do? - (~39:13) We'll buy them beer, we'll do all of that stuff, but what else can we do? - (~39:17) Can we bring them into a dojo setting? - (~39:19) Can we bring them to the development teams? What would you advise? - (~39:23) Show them this YouTube video. - (~39:26) We're all internal auditors, we're external auditors or security professionals that deal with DevOps and compliance every day. So it works. So that's the first thing. - (~39:37) You have to make sure people understand that it's something that can be done. - (~39:41) Then it has to be a strategic priority. - (~39:43) In other words, it's something that they must do for a business reason. - (~39:47) Once you overcome those two obstacles, then you're into, okay, well, now how do we get it done? - (~39:53) Yeah. - (~39:53) You've got to break through the barrier and then get into the implementation. - (~39:57) I don't mean to sound maybe patronizing. I hope this doesn't sound that way. - (~40:02) But I would broker those conversations with an alignment to the broader mission, right? If both groups have the shared mission and we can at least agree on that, that's the starting point for how do we get it done together, which Yosef was saying. - (~40:13) It's oversimplifying it, but I find that that's helpful. - (~40:17) Where we seem to be most successful is when those compliance folks are part of your delivery and deployment team of rolling this out, right? - (~40:24) They're in the planning sessions from the start, that they're brainstorming with you around those risks and how you guys are going to define those risks, right? They're part of the process from the start. - (~40:36) Where it becomes more challenging is you've developed your pipeline, you've developed your process, and now you bring the compliance folks in, you say, "Certify this. Tell me that it's good so we can go forward." It's much easier to kind of guide the path with them there from the start. - (~40:51) Yeah, we have a couple of examples where we've worked with them kind of from those early stages, and you get the internal auditors or whomever in the room as early as possible. - (~41:01) And probably our biggest success story is they gave multiple feedback points along the way. And so when we first asked them in, they were like, "Really? You're asking us in? Okay." - (~41:11) And then it wasn't like, "Hey, we've done this. - (~41:14) Give us feedback." It was like, "Hey, we've done a little bit. What do you think? - (~41:17) We've done a little bit more. What do you think? We've done a little bit more. - (~41:20) What do you think?" And so we weren't getting to the end and saying, "Hey, isn't this great? Can you sign off on it?" We were giving them multiple kind of iterations along the way to give that feedback back and give their input. - (~41:30) I think too, very few places except for the military is truly a hierarchical organization. - (~41:38) Right? Raise of hands, are you in a matrixed organization? - (~41:43) I know that I am. I know that this guy is, right? - (~41:48) Finding those advocates, right, and an interest person who can be your advocate that is aligned into that organization that you can build a relationship with that may not be the lead of the organization, but is an influencer, and have them bring it along, right? It's kind of like bringing your friend to a party who's then going to spread the word to the other friends to come to the party, as opposed to, well, we went to the head of internal audit and they said no. - (~42:17) Right? Finding those advocate groups. - (~42:19) And everything we just talked about, it's cultural and soft skills and change management. It's not technical in nature. - (~42:25) It's how do you get buy-in to be an enthusiast and then go to the next level of being a person of change, and not just a change advocate, but being a driver of change within your organization. - (~42:39) I think one thing that the audience may have gotten from us is that we keep repeating, bring them in early, bring them in often. - (~42:46) That sounds a little bit blunt. There might be a strategy where you say, "Hey, we've got a few experimental projects over here. - (~42:53) We're going to just see what happens." Invite these guys to these three small things, very low impact, requests from the business, not the big one, with massive strategic implications. - (~43:02) You wouldn't, for the first time, broker the conversation there. - (~43:04) But build the muscle small. It's just like going to the gym. - (~43:07) You're not going to go bench 350 your first time there. - (~43:10) Work your way up to it on smaller gigs. - (~43:12) And just engage them as partners. Have that kind of attitude. - (~43:17) I think the fact that we've got folks up here that are a mix of auditors and non-auditors, and we obviously work very closely together, speaks to that and speaks to the success of that. - (~43:26) And I would just add making sure that you can ask them, what are they really concerned about? You're going down this path. - (~43:33) Help us understand what are you guys real-- Why is this a roadblock for you guys? - (~43:36) What are some of the requirements you are stuck on? - (~43:39) And let us work together to figure out what those are. - (~43:41) So let's try to close this out with a great story. - (~43:44) You have to be able to effectively drive change, to be an unbelievable salesman or storyteller. - (~43:51) So internal auditors don't always want to do a lot of work. - (~43:55) They tend to be, if it's the old-fashioned way, they're checklist-based. They may be doing the same thing every year unless the risk profile changes in the organization. Paint a picture. - (~44:06) What if, internal auditor, we create a paradigm where you will see in a dashboard all the exceptions for whatever the specific control is, and you could actually see not just the exceptions, but how they've been resolved by clicking through a screen in front of you from your desk. - (~44:24) Is that a future that you think would help protect the business and drive efficiencies? Everybody's going to say, "Where do I sign?" - (~44:32) So paint a picture of what the future life of an internal auditor or an internal compliance specialist is going to look like in a data-heavy DevOps world where you could drive compliance through automation. - (~44:46) My question's around PCI.

10How do PCI, production access, and automated approval work?

- (~44:48) So I know you guys brought up the having the segregation between dev and the production. - (~44:55) In DevOps, we're doing the opposite, right? - (~44:57) We're having developers giving access to production. - (~45:01) Do we expect that PCI requirement to change? - (~45:04) And then also, if we're being audited now, is having the controls around the deployments and then also having logs going to an external system sufficient to pass an external audit? - (~45:18) Yeah, so I'll take a stab at that. I'll start with first the segregation of duties thing. No dev actually wants access to production. That's terrifying. - (~45:28) It's horrible, right? The actual value is, can you put controls in place, so I've got guardrails to work within, right? - (~45:36) And then that system creates the ability to have tracking of all of those things, right? - (~45:43) So if all of a sudden I have the guardrails in place, the system will control whether I have multiple people approving pieces, right? So I don't see that going away. - (~45:54) I actually see the guardrails improving. - (~45:56) I've told this story multiple times today, but it makes sense to me. - (~45:59) It wasn't the FBI that stopped people from stealing music. - (~46:05) It was iTunes. - (~46:08) It wasn't the FBI's compliance and governance and scaring everybody into it. It was Spotify, right? Now, if you apply that to our DevOps model, well, yeah, if I'm using a connected tool chain that's enforcing my compliance policies about segregation of duties and tracking of that end-to-end piece of it, then I'm living the compliance because it's supported by my tool set. I don't see that going away. - (~46:36) I actually think it gets more so because infrastructure as code should not be developed by the same people who are writing software to run our infrastructure. - (~46:42) Before you come in, quickly, you mentioned fully automated systemic change of production systems, and then you also said about multiple people approving it. - (~46:54) So can you kind of... - (~46:56) Let me try to talk about this. So if you think about your chain of command on a deployment, if security has to sign off on a deployment, if DevOps have to sign off on the deployment, all the different stakeholders involved have to systematically say, "Yes, this is really ready to deploy in multiple environments," you've just made your world better than it is in Waterfall, by design, by definition. - (~47:17) And all of that has complete audit trail because it's all systematic. - (~47:21) That's number one. Number two, when you think about fundamental access to production, in the old world, there would be a few individuals that would always have access to go in, and maybe their access was logged, hopefully, and somebody was monitoring it, maybe. - (~47:36) Today's world, the individuals have the potentiality to have access to production, where they will go and request a temporary 24-hour or six-hour time span code that gives them access to it so that it's not standing access, it's potentiality to get access, but they have to open a service ticket and state why they're going in, and then it's removed. - (~47:59) That is a new world. - (~48:01) On-prem, you don't necessarily have that unless you've invested in those capabilities. - (~48:04) And even better, it's actually nobody who has access to it. - (~48:08) I'm using a GitOps type system where I commit a thing, it gives me just-in-time tokens, and it opens up for the automation only, and you govern the automation like a human. - (~48:18) As who has access, et cetera. But the key is being deliberate about that. So one of your other questions was like, hey, if I've got all this, what I call data exhaust from the system that gives me the logs and the traceability, yeah, but this is a human issue. - (~48:32) I'm not probably going to trust that you have the data exhaust. - (~48:36) I'm going to want to trace bullet through that whole thing, and that's the gaining the trust of that organization. - (~48:41) I'm still confused about multiple people approving ... production deployment and a fully automated, how do they line up? - (~48:48) Yeah. - (~48:48) Where do the people come in? - (~48:50) So I'll give it and then I'm mic hogging. - (~48:54) I think there's a couple of things there. - (~48:56) One, which is I had a customer who's like, "I want my dog to be able to approve builds." And his rationale is it should be fully automated, right? - (~49:04) The reality is that is not the case for most things. - (~49:07) There are many things that can be fully automated based off of a risk threshold, right? There's other ones which may be notification based and someone, let's just say in ServiceNow, gets something that somebody has to approve and then kicks it back to another system to actually build it. - (~49:23) There are still some human controls you might decide to be in that system. Some of them might be automated controls that it passes all of the tests, right? - (~49:34) I ran through Sonar Qube and it got a low enough threshold, it passed a certain level threshold on my unit tests. - (~49:40) I'm not impacting more than X systems. - (~49:43) Lo, it passes through the workflow and says, "Yeah, go ahead." But that was something that this guy approved the workflow of, that this was an agreed upon setup. How about the guardrails? - (~49:55) I think what's important here in that conversation is organizations defining their process and their risks, right? - (~50:03) Different organizations have different risks, different thresholds, and there very well could be cases, right, that require no approval, up to multiple levels of approval. But it's how you define and understand your environment, how you lay that out and say, "Look, this is how we look at our world. - (~50:19) This is what's critical, this is what's important, and these are things that are just kind of day-to-day operations that are lower risk for us," right? - (~50:26) Each of those defined kind of thresholds are going to have potentially different approval requirements. - (~50:31) And so it's really back to how do you understand your own environment, how do you determine those risks within your environment, and then how do you explain that thought process to your internal and external auditors so that they can weigh in on, yes, we agree with this or we don't. But now you're having a conversation on risk, on process, versus, no, you can't do this. - (~50:55) And I mean to add to that, I mean, you really, to understand the environment, what can a developer do in production? If the code's already compiled and sitting in GitHub, I can make configuration changes in production, but you have a configuration management tool that runs every 15 minutes, Chef or the overrides, everything is a production anyway. - (~51:11) So worst case, they might be a configuration's out of date for 15 minutes in production. So again, it comes back to understanding the risk and understanding the environment. - (~51:19) And I think that too, this is a mind shift scenario. - (~51:22) You heard, at UHG the other day on the main stage, they said they went from doing a quarterly release to two releases a day. That also means that my risk blast radius became so much smaller. And so on the risk and internal audit side, they're willing to accept a different risk because what's the big deal? I've got a blue-green deployment. - (~51:45) I hit a certain threshold, I rolled it back. - (~51:47) The risk of that and the blast radius of that became much smaller than big bang release we do every quarter that in fact, that impacts- Big bang- ... three, five systems ... is higher risk. If you have a release once a year, your risk profile is through the roof. The incremental releases, the more releases you have, the less risk you are. - (~52:10) Hi, so, it's a hypothetical situation, right?

11Should old controls be revisited as a system?

- (~52:16) So we heard about some of the questions were around code pair, some of the questions were around the higher level, what do we do about it, right? - (~52:22) If you think about, let's say some large organizations, right, where all the policy, standards, controls were written in an era where they were waterfall, where there wasn't automation, right? - (~52:36) And all of a sudden they are doing projects to product. They're product-based now. - (~52:43) They are waterfall to agile. They are now DevOps, right? - (~52:46) They're adopting cloud. - (~52:48) What would be your suggestion? Let's say we've been through your two steps, and we are trying to figure out how to implement the controls. - (~52:56) Would you look at that individually or would you step back and say, do we just need to-- because it changes your risk profile, changes your responsibility model, changes everything, right? - (~53:05) So would you step back and say, are those policy standards and controls and implementations the right ones, or would you do it one by one and... - (~53:15) You really want to look at it as a whole. - (~53:19) It's kind of everything you just described because if you look at things in silos or in isolation, you may not be covering off that risk. - (~53:26) So you've got to say, okay, we're moving to this new process. - (~53:30) Here's what that new process looks like. - (~53:31) Here's where the risks exist in that process. Here's old controls that we had. - (~53:35) Are any of those still good or not? If there are some that are still good, pull them forward. Then look at the remaining risk exposure, controls to close any gaps, and then going around and making sure you're updating those policies and procedures. - (~53:49) Because as boring as it is, when an auditor comes in and policies and procedures aren't updated, they're going to ding you on that straight away. - (~53:56) Doesn't matter how good the controls are. - (~53:57) If the policies and procedures aren't updated, you've got an issue. - (~54:00) Yeah. Sorry, one more question if I can add on. - (~54:03) Who do you think in your mind is this approach, who owns recommending this approach in terms of owning, saying, this is how we should approach compliance- Oh, so, so- ... because we are doing all those four things. - (~54:16) So there's the spirit of the law, but somebody has to own compliance at the end of the day. So when we came out with COSO 2.0, which came out three years ago, four years ago, I forget already, when it became effective, it was an update to the Sarbanes-Oxley initial framework. So what all public companies around the world had to go through and do is they dusted off all the initial controls documentation that somebody did five years before and most likely hadn't changed unless there was a new process, and they had to now look at - (~54:48) all their control mappings and all their policies in light of COSO 2.0. And what people learned when they went through that exercise is that-Why did we not go back and change this? - (~54:58) And why were we forced into a way of operating just because at a point in time, this is how we did the mapping? - (~55:04) There's nothing that says you cannot adapt and change how you comply with the control objective from a compliance lens. - (~55:10) So somebody has to take it upon themselves to transform within the organization and challenge the way we've done things to make it better. - (~55:19) And that doesn't mean that you're not complying. - (~55:21) It means you're redefining how you're complying. - (~55:25) There's nothing against that. - (~55:27) And in terms of ownership, that can be challenging. - (~55:31) It may be an IT compliance group who kind of quarterbacks the process. But there's no easy answer on that one. - (~55:38) But it's a case of finding someone and kind of pinning them to it, especially in this transformation state, and saying, "Hey," maybe it's your VP of business agility or whoever's driving that. - (~55:48) And now they're not going to do it, obviously, but having someone who's on the hook for it, and then to Yosef's point, looking at that at the ongoing process and making sure you don't just blindly follow these policies that were written down and keep iterating on those as you change and as you grow. - (~56:07) We have a question here. - (~56:09) Hi, good evening.

12How do teams practice failing an audit?

- (~56:12) So as an engineer, I practice failure. - (~56:14) I deliberately break things in test. - (~56:17) I practice major outages. How do I practice failing an audit and- Is there any kind of techniques, or am I really stepping into a hornet's nest? - (~56:29) So there's a concept that we refer to as a defendable position. So if you want to take an approach for something, and it's not exactly how maybe your process is laid out, but you still believe that it complies with the spirit of the law, make sure you could defend it. So if you want to go through an audit and have an approach that's set up, and I'm audited and she's audited because all the 54 applications we support go through internal audit, go through ISO readiness audits. So we're audited, and we're sort of the auditors, and we have - (~57:00) audit applications. So we deal with this literally all the time. But if you take a position, you got to have a story to tell. So when you go through that life cycle, why are you willing to deploy code if you know that there's known frailties in it? - (~57:14) Because nothing's perfect. There's always frailties. - (~57:17) We know that there could be tons of different things that fail from a functional perspective. But you take a risk-based approach as to why you're willing to go to production with all that. That's a defendable position. - (~57:27) I think one way to look at it as well is maybe, and almost building on your question as well, when you ask who owns compliance? - (~57:34) Well, in this space, who owns the DevOps process, right? - (~57:38) Those are the owners of those controls of that process. - (~57:42) And just like you're trying to constantly improve your product, similar from a compliance standpoint. - (~57:48) It's another thing to regularly look at and say, "How are we doing this? - (~57:53) Is this the right way? Is this the best way? - (~57:55) Is there a better way that we can do this?" And unfortunately, from a compliance lens, Yosef mentioned it, controls stay static until they're forced to change. - (~58:05) You look at an organization, they say, "We've been doing this since we implemented for the last 10 years." And that's the mind shift that also has to happen, is how do we constantly improve the way we're meeting our compliance objectives? - (~58:18) And so to take the very engineering response to that. So we created chaos engineering and all of that stuff to kind of break our own stuff. - (~58:30) So the same rationale I've been trying to push is compliance as code. - (~58:34) Not policy as code, that's configuration and everything, but compliance as code. - (~58:38) So pairing a developer with a risk person to say, "Yeah, try and write a test that commits code and then approves your own pull request." Try and break the system, and then you can continually run that on every single build. - (~58:57) And that really means just like a content expert would be helping the developer with the functional requirements around a user story, this would be a risk or compliance person who's helping you write the test that proves that the control works. - (~59:14) And tactically, if you're bringing in these risk and compliance folks early on in the process, you can have them go HAM on a QA environment, on a lower environment. If you want to fail an audit and fail it fast, have them look at one of your lower environments and say, "Here, come in, take a look, tell me what I'm doing wrong." - (~59:32) Sorry, follow-up. I don't know if you can hear me. - (~59:36) Quick follow-up to that. Compliance as code. - (~59:39) If you think about securities-- Sorry, it's related. So think about security. - (~59:45) You have secure coding, you have threat modeling that we're starting to teach developers. Is there something like that from a compliance perspective that you guys, you said compliance as code, but is there something like that? - (~59:54) Well, from an audit perspective, actually, we try and get developers to push bad code because they always pan to these downstream monitoring controls that will catch anything. - (~60:04) So we're saying, okay, well, if you can push this code, let's see if it falls out on the back end. So, I mean, happy to talk to you. - (~60:11) I work in the ad tech space a lot, and so one of the things are they'll never charge for duplicate clicks. I'm like, okay, now push code that all of a sudden now you're going to start recognizing revenue on duplicate clicks. - (~60:21) It's so hard to push code. And so some of these things, we definitely try and see if we can test that monitoring that they have on the back end that would detect some of these things that we really care about in the bad code. - (~60:31) So also, just following up into that, where reliance on compensating controls can go wrong in a major way. If you always point to the same compensating controls, in addition to where those controls serve as a primary code, you have so much weight now and so much dependency on it, and if that breaks, you're failing a lot. So pushing things and being reliant on other compensating controls, you have to really be methodical in terms of the frequency that you do it and think about what we call aggregation risk. - (~61:04) What's the risk if this control now breaks to everything else in the system? - (~61:09) Hello.

13Is the three lines of defense model dead?

- (~61:11) Is the three lines of defense model dead? - (~61:13) Because I hear a lot talk about IA being involved in consulting and doing things that, where I come from, you don't do as internal audit. What's your perspective on that? - (~61:24) I've got one. I think the three lines of defense are working much more collaboratively with each other. - (~61:32) It doesn't mean that it's necessarily dead. - (~61:35) In fact, I think it's actually strengthening it. - (~61:40) I'm seeing projects in the last 12 months, not just even with the pipeline, but just in the whole security portfolio, where we've got projects that are chartered, paid for by the security organization and the internal audit function, and they're working on it together. And I'll give you one specific example. - (~61:56) We're doing a security assessment where both parties are paying for it within that same enterprise, and we're doing a data-driven assessment. We're doing a traditional kind of subjective type of interviews and things on top of it. And then we're taking that same body of work and we're going to go and write an audit report over here with it and a security roadmap over here for the security organization. They've saved a ton of money. - (~62:23) They haven't wandered out of their own boundaries and their three lines, and everybody's benefiting because they're collaborating and they've adopted that partnership type approach that we've all kind of been intimating. - (~62:32) So the three lines, for many, many years, if not decades, they were silos. - (~62:37) Every single one was siloed. They didn't talk to each other. - (~62:40) It was a checklist-based approach, and that was it. Organizations today, with transformation, with the speed of technology, the collaboration that's needed, the knowledge share that's needed, the technical skills that are needed, you're still maintaining that objectivity in terms of being able to apply those principles, but it's much more flat, and those silos are really just from the budget lines. - (~63:04) It's about speed. If you don't move faster, you die in this culture, right? - (~63:11) And so they're doing it out of survival, and you don't need to break down your fundamental role in the organization to move faster. - (~63:18) You just have to move more collaboratively with each other. - (~63:23) Yeah, I have a question.

14How are audit finding severity and common controls judged?

- (~63:24) So once a year, we have external auditors who come in and audit us for ISO 9001 and TL 9000, and they come and audit various products that use different styles, whether it's Waterfall, Agile, and the DevOps styles. - (~63:44) And then they have some findings, and then through some kind of a magical formula, they come up with major non-conformances, minor non-conformances, and sometimes things that we need to improve on. - (~63:58) My question to you is, and specifically on the DevOps style, in your thoughts, what is that criteria that they use to say, "Oh, okay, this is a major non-conformance versus maybe a minor non-conformance?" - (~64:18) So every firm has their own approach in terms of classifying errors and findings and translating them. So the language you just used, it's probably specific to a particular audit firm. - (~64:32) But in a nutshell, there's different criteria that's typically risk-based that leads you to a level of severity, however it happens to be defined. And there's different factors. - (~64:46) There's the frequency of an error. - (~64:49) There's the severity of an error. There's should there have been a compensating control? Was there bells and whistles going off and you actually had the logging, you actually had the monitoring, but nobody did anything? - (~65:00) So it really depends on really the facts and circumstances of the findings that potentially could lead to different buckets of severity. - (~65:07) And especially with ISO, they have the specific domains. - (~65:10) I'm more familiar with ISO 27001. - (~65:12) So a major non-conformity is you cannot meet one of the criteria. - (~65:16) And so that could potentially pull your certificate. - (~65:20) So there's some of those things, what they're looking at, and then minor non-conforming, you might not have not been in compliance with your own policy. - (~65:27) And then areas for improvement typically is just like, we're adding value to your business. - (~65:32) But if you do have an internal audit department, they should come in before the auditors into a readiness exercise. - (~65:39) So last question, go ahead. - (~65:41) Just to add on to that. So I understand the determination of risk is going to be very specific to the company, to the application, to the environment. - (~65:51) But are there any specific controls that would be common that will determine risk? We seem to be throwing some examples around, separation of duty. You talked about vulnerabilities. You talked about test automation. - (~66:06) If you do this, these 10 are the most common that everyone in their pipeline should include to judge the risk, and then everything else will be kind of context. - (~66:17) So I got this standard answer I give to this question. - (~66:19) It's called background checks. - (~66:21) Because background checks are in every single one of these rules, and typically there's a standard approach that all the background check agencies utilize. And it is really a joke, but it's facts and circumstances. The spirit is the same. - (~66:35) The application is going to be different depending on the type of cloud you're using, depending on your data, depending on your application. - (~66:42) So I'd love for there to be a unicorn way that it's always going to be a checklist, but we talked a little bit before the checklist is sort of gone now, and we take a risk-based approach, and it may vary. - (~66:54) Come find me after and we can double-click on that. - (~66:57) Yeah. - (~66:57) But there are going to be some things that are always going to be common, especially around access. And so if you're being controlled by the tool, you can't have access to go in and change the configurations of that tool. And so regardless of what those configurations are, that's going to remain true across the secure pipeline. - (~67:15) Similarly, controls around who can change those configurations and making sure the changes to the tools themselves. - (~67:20) So more on the ITGC side of the tools, there's definitely commonality in those controls. Now, exactly what they look like is going to change, but around security access, locking down those pipelines, that's going to be consistent. - (~67:34) We've got a little bit over one minute left.

15Closing invitation

- (~67:36) Sam- Caleb, you can take the last word if you want. - (~67:39) I was actually going to- Okay, so for those of you in the room, we have a session tomorrow where you can go deep privately and ask anyone here specific questions, and you can say, "My goodness, our case is so messed up. - (~68:01) What should I do?" - (~68:03) And for those of you who aren't in the room but are watching this on YouTube, if your situation is so messed up that you're not doing compliant DevOps and you think that audit is blocking you, we would like you to reach out to anyone on the panel to say, "How do we fix it?" - (~68:29) Let's give a hand to our panel.