Q&A Workshop (Topo Pal, Gene Kim, Yosef Levine, John Waters, and Audience)

Topo Pal: Good afternoon. Hope you had a nice lunch. After this lunch, this is the session you want to be in, so call your friends, text them, and ask them to come over here. This is not a typical workshop. What we want is a very open discussion around this particular topic: audit and DevOps and cloud and everything that is going on across the industry.

Topo Pal: There are some ground rules before we start. Number one is: we do not introduce ourselves. You know who they are; there is no point introducing themselves again if they are on the program. The same goes with you. We do not want to know who you are, where you come from. It is just the questions and the answers that matter. When you ask a question, there will be mics going around. Just raise your hand and the mics will go to you. The first thing you do is ask the question, not introduce yourself or where you are from.

Topo Pal: We really want to have that rule because we really want an open discussion. I think this is very critical. I have been in multiple DevOps conferences and many other kinds of conferences, but I think this is the first conference, and this is the first time we have auditors talking to us face-to-face directly, taking questions. So thank you very much. We really appreciate it.

Yosef Levine: Our pleasure. We do have one ground rule: any discussion or perspectives that we share, they are not meant to be authoritative guidance on behalf of Deloitte. It is just us sharing perspectives.

Gene Kim: I do not need to be on camera, but I really appreciate this because even the last five minutes of the earlier session, that video is so important. I will be sending that to all of my friends. I think the video that comes out of this is probably the most important video that will come out of this entire conference because we are getting experts to opine, not officially, but through their experience.

Yosef Levine: Sharing perspectives.

John Waters: Lessons learned from the journey.

Gene Kim: So thank you. And I am so glad they moved the camera, because otherwise I would have to make a disclosure that if you ask a question and then you get up, we could no longer protect your privacy. We would have to edit you out or whatever. Thank you to the AV crew for moving the camera.

Topo Pal: Apart from them, only myself is on the camera, and I will disclose that actually I changed my T-shirt. I had a branded T-shirt, so I went back to my room and changed my T-shirt. Whatever I am going to say here is just from my own perspective as a DevOps enthusiast.

Yosef Levine: This is a therapy session now, guys. Therapy.

Topo Pal: Just to start as an icebreaker, I will pose this question. We all in this room are developers, engineers. We write code and send it to production. The basic question is kind of stupid and dumb, if you look at it: why do you need to audit the whole software delivery process? If you can just briefly say, and then we can go along from there.

Yosef Levine: It is like building a building. When you build a building, you can go and inspect it when it is done, but if you do not know the strength of the materials, and they do not pass certain standards, the whole building is going to collapse even if you follow the plans. It is about the material. When you correlate that to a healthy software development life cycle, it is all about the journey, not necessarily the final product.

John Waters: Yosef covered it well. What are the steps you are taking along that journey to say this is now something I can rely upon to do my work, or other people can rely upon to do their work? It is a step beyond just testing, not just the functionality of the product, but the entire environment in which the product relies. You might have a great application that functions exactly like the business wants it to, but if it is in an insecure environment, that is not a lot of value when the breach occurs.

Topo Pal: I am a core developer, so I have been writing code for long. You get in a room with developers and engineers and architects and the business, and things get heated up pretty soon. Then all the developers ask one simple question: "What are your requirements so that I can code?" From an audit perspective, what are your requirements?

Yosef Levine: You could slice this a bunch of different ways. Standards and regulations translate into control objectives and control processes that have to be included in your software. How you solve that for PaaS is subtly different than how you solve it for IaaS, because in a PaaS environment hopefully a lot of that is taken up by the SOC report of the specific cloud provider. You do not necessarily have that if you are dealing with an on-prem scenario. You could look at requirements in terms of control objectives that need to be met, but generally auditors are not going to be specific or opinionated in terms of how you do it, so long as throughout the organization it can be consistent and be solved the same way. We are looking for that consistency. From a speed, performance, and reliability standpoint, John will probably care a lot more, and that has an indirect relationship to my controls because how you do it potentially has an operational direct implication on meeting those objectives over time.

John Waters: Yosef and I make a fairly good pair because his team leads our technology controls department for the audit function for Deloitte. My team and teams I work with are responsible for meeting those requirements. Sometimes we hit gray areas around the spirit of the control objective, whether confidentiality, integrity, availability, and so forth. We have to think from a technology perspective about how we meet that objective. Sometimes there is latitude in the response, but that also results in IT professionals saying, "No, I want them to tell me what to do," whereas control auditors are looking for the overall outcome and less on how you get there, but certainly more on consistency. That gray area causes discomfort for IT folks, but if auditors are very prescriptive, IT folks may feel less empowered to innovate or make technology choices. It is a balance and sometimes a joint journey. Having those discussions earlier in the software development life cycle is critical, because if they are not occurring until your last sprint before release, there is a high opportunity for rework.

Yosef Levine: If you take a look at your SOC principles of a SOC report, you have a ton of control objectives within the trust principle of security. Encryption is one way to meet certain control objectives. Not all encryption is created equally with the different services available through PaaS or IaaS. The control language is not specific in terms of the type of encryption, encryption at rest, encryption in transit, keys, certificates. But the type of encryption you have in place may solve other control objectives or operational or regulatory needs. Through healthy dialogue, the approach you take just to solve encryption could solve a vast multitude of other areas just by talking more.

Topo Pal: You mentioned control objective. Can you explain what that means, who writes those controls, and where they come from?

Yosef Levine: Control objectives typically come with a control standard. SOC 1, SOC 2, SOC 3 have different trust principles, and within those trust principles, as promulgated by a regulator, there are specific control objectives meant to be met, similar to NIST or ISO. There are high-level objectives and sample control activities that are somewhat responsive, but those are just samples and need to be tailored based on fact and circumstance.

Audience: In larger organizations there is often a 20% bubble doing DevOps and succeeding beautifully, but as it penetrates the rest of the organization and runs into entrenched ways of working, people say, "You cannot change that. It is a control point for our audit." Sometimes those are manual stops introduced to show something is being judiciously assessed. How do we disassemble those things, assess whether they are mythical, or find a better way to comply?

Yosef Levine: Communication is key. Be empowered to speak opinion in a respectful way. John and I do not always agree, but we know we have each other's backs. From an auditor perspective, the best interest is not passing the audit; it is complying with the objective. Auditors can empower an organization to transform because everything has to be safe and secure and conform with regulatory needs. But you have to think about the spirit of the control objective. Could there be multiple ways to comply with the spirit? Peeling back that onion and having dialogue through that lens creates a healthier outcome.

John Waters: In release pipelines, you may start in development looking for lots of iteration and true CI/CD. As you move closer to release, you introduce additional controls, but those controls are not intended to be onerous. For example, a production release might involve IT approving in the deployment tool, a product owner doing the deployment, and perhaps controls approving the release. But this is just a notification and a green-light/red-light approval, not getting a bunch of people in a room with lots of formality to slow it down. A lot of what you describe sounds like muscle memory: that is how we did things, so that is how we continue to do things, without assessing new technologies that still achieve the same objective.

Yosef Levine: Internal audit has to buy into the business strategy. The business strategy is not the same old way. The business strategy is to evolve, transform, and grow. If you phrase the dialogue as trying to achieve common objectives, hopefully the outcome is what John is referring to.

Audience: In The Phoenix Project, an IT security person has an epiphany that some controls were less important because there were compensating downstream controls. Do IT security people sometimes lack a holistic end-to-end view and hold too tightly to a control matrix they downloaded from the Cloud Security Alliance? How do we help them see the big picture?

Yosef Levine: To implement any control framework and even do an external audit, you start with understanding the nature of the business, transaction, or process. Often an auditor can be stuck on the rigidity of being compliance-minded versus process-minded. You audit a business process that has controls built into it. That is a mindset change. Auditors tend to be highly conservative, but we are evolving. You have to take the auditor back to core principles: understanding. You cannot have control objectives or an activity unless you understand the process.

John Waters: When we first pivoted toward a DevOps culture and cloud, we went through some of the same experiences. Core security tenets tend to transcend DevOps and cloud. It was incumbent on us to partner and educate. Just-in-time education can be too late, because people may have emotionally or organizationally worked themselves up to a point where they say, "This cannot be right. We must do it the way we have always done it."

Yosef Levine: We had a situation where the controls people and cloud architecture people had done sessions and change management exercises, but coders and product architects forgot, force-forgot, or were too busy. We needed refreshers. We level-set everybody and got to an answer where you mitigate risk, still comply with standards, and come up with another iteration of code to get things where they need to be before you need to meet the control objectives.

Topo Pal: Do you think in classic organizations there is a problem of mistranslation of actual requirements, what needs to be done, and how it can be done in a modern, better way?

Yosef Levine: Yes, yes, and yes. Positive intent. Our CTO says assume positive intent and have a discussion. Everybody is well-intended, but when you explain something to an auditor, CFO, COO, or IT person, you have to know how to translate. You need one person or a group of people who know how to have a meaningful story with the appropriate stakeholder group so everybody can be level-set.

Audience: I deal with external auditors looking at financial regulations. When you have the conversation and are at a point where you are going to get a finding, it is either a fail or not a fail. Is your industry moving toward a model where learning, failing, and turning it around quickly is okay?

Yosef Levine: Before you get to a failure, there is an exception. Once you pull apart the exception and understand the facts, was there a compensating control? Was it picked up by management through self-monitoring? Is there a waterfall approach of multiple exceptions because something was undetected? Those considerations are based on audit standards. The nature, extent, and severity of how that translates into a positive or negative control statement in a public audit report depends on the organization, the nature of the exception, the extent of the failure, and how pervasive it is. Use the language of exception, not pass or fail. Then look at qualitative factors: upstream, downstream, repetitiveness, whether it was isolated, who detected it, whether the organization documented and investigated it, and whether it was self-corrected. That can be a positive storyline.

Audience: Would it be worth having a minimum viable common language which both sides understand?

Yosef Levine: In our world, this was recognized early. All of our technologists, coders, and engineers are forced to take a day of audit for non-auditors, to have vernacular squared away and concepts more fully understood. Coders are developing products for auditors, so it is important they speak the language of the business. In any IT environment, you are serving a customer, and your customer is the business. We teach our coders the language of audit because they are developing products for us. You have to invest in creating that learning so people speak the same language in relation to the business. On the flip side, our auditors who are working on products take Agile training, product owner training, and cloud training, and they know how to use VSTS. We try to bridge the gap in both directions.

John Waters: Bringing internal technology controls into the journey very early was essential. Control objectives are intentionally written at a higher level so they are not brittle. As technologists iterate and chase new technologies that suit the business need better, the controls do not continually break and are not overly prescriptive. If that dialogue occurs very late, it is almost like we are speaking different languages.

Audience: You spoke about privacy by design and security by design. From an auditing point of view, how do you still do that if auditing is linked with each product release and DevOps makes it hard to audit every release? And if the control team should engage early, do we invest more in the control team or train engineering to enable them?

Yosef Levine: Yes, you need a more significant investment in controls because you are not just testing at the end. Agile means having decision makers in the room at the right time, which means auditing and controls are no longer a compliance exercise; it is direct value to the business. Because it is value to the business, you have a new story to tell to your COO and CFO when you want to hire more people. You are not asking for more money for back-office IT security. DevOps is organizational and business transformation. On the other side, controls can become a capability that an engineer can download: configurations preconfigured to be responsive to controls. We are trying to democratize controls through automation and hardened parts of code that get ingested into applications. If key control-sensitive elements become shared, configurable components, access and roles can become plug-and-play. Then it is a configuration exercise, not a coding exercise, and you do less ongoing control testing because you did it by design.

John Waters: Some of these ideas scale naturally to very large organizations, while smaller organizations may find them a heavier lift. Look at which activities require judgment by a person and which can be automated. You can bring DevOps discipline to controls testing and automatically generate evidence: for a server, who has admin access, when credentials were last rotated, what the organizational policy is, and whether it is in compliance. Today that can be a manual discussion. From a DevOps perspective, it is low-hanging fruit, easily automatable, and evidence can be generated through a portal.

Yosef Levine: On frequency, it depends on the nature of the release. Not every release of code is created equally. Deloitte created categories of release types. Every release fits into those categories depending on significance and nature. If there is massive new functionality, re-platforming, or significant architecture change, we do more testing. If it is middle of the road, it varies. If it is very low, we may do no testing at all.

Gene Kim: Imagine I am a junior auditor and I find a developer deploying into production, obviously bypassing segregation of duty control. How would you respond to assuage my concern?

Yosef Levine: If you are auditing me, I should have a deeply defined control exception process that includes quantitative and qualitative evaluation criteria, because management has to conclude on severity. If whoever is in charge of Sarbanes-Oxley or compliance did not create a detailed program for evaluation of control deficiencies, that in and of itself is a control deficiency. Built into any control environment is a process to manage exceptions of controls, similar to having a policy to manage policy exceptions.

Gene Kim: A valid response might be, "By design, an SOD control is not part of our control design. That objective is being taken care of by a different set of controls."

Yosef Levine: It could be, depending on the nature of the organization and how you have taken the principle of segregation of duties and implemented it within your unique control environment.

Topo Pal: Product development teams feel they have enemies around: ops hates us because we change production, security hates us because we do not know security. Is it the same story, that auditors hate us too?

Yosef Levine: I do not think so. If you have common interests and stakeholders are aligned, internally that has to do with culture. It has to do with how the CEO, financial leadership, and risk leadership have created a culture aligned with business objectives. It is not an auditor concept; it is culture. You get people together more often so they understand what everybody does and why things are important. That open dialogue and trust remove noise from the system.

John Waters: Operations and control folks have a bit of natural tension, but it is supposed to be healthy. Is your objective to achieve a new high score in the number of findings? Or is your objective to release code that is of value to the organization and its clients? A healthy partnership between the controls team and DevOps team is bi-directional education, not holding product back.

John Waters: On whether developers can access production, it depends on your environment and regulations. But who wrote the code? It is less about the last step in the journey and more about the controls along the way. No organization wants anyone, regardless of label, to deploy to production without insight. What controls did you put in place before the final step in the release pipeline so you have reasonable assurance that controls were met?

Yosef Levine: Segregation of duties historically is about people, but it is really about process as much as people. Process can be automated and coded by design. You can have segregation of duties in a smaller organization where steps of process have to be met that create a controls environment that looks like segregation of duties from a process lens versus a people lens. That will not work in every organization, especially with deep data sensitivity, but facts and circumstances dictate the variability that can be used.

John Waters: Also consider who has access to production systems, what responsibilities and rights they have, what data they can access, and what compensating controls are in place. Encryption at rest and in transit are basics. Organizations are moving to encryption in use, where sensitive fields are encrypted in the database so DBAs or developers with database access cannot see private health information in plain text.

Topo Pal: In reality, organizations often come down to who has access to production and who does what based on title, as opposed to what they actually did.

Yosef Levine: That is not what the standard says. That is an on-prem way of thinking. In an on-prem environment, without DevOps operating, that can be correct. When you have a mature DevOps mentality and cloud, the principles do not change, but how you apply them can vary.

John Waters: Depending on your industry, external regulators can still say, "These need to be separate teams."

Yosef Levine: If a regulation affirmatively states that, the law will always come after technology changes. If your regulated industry has rigid technology laws, escalate to your regulation department and talk to the regulator. Ask for a no-action perspective or clarification that may eventually lead to a change in regulations.

Audience: At what point should we involve control teams and audit teams in DevOps transformations?

Yosef Levine: Your first or second meeting with the business. From day zero, blank whiteboard, you have security, cloud architecture, and controls in the room. You are not even at sprint zero. You are at: we went out to lunch and had an idea and think we can get budget. How do you think about it conceptually? If that has not happened, it is not necessarily bad, but it can be painful to level-set everybody.

Audience: Engineers often cannot talk directly to the auditors and feel they only get interpretations through internal compliance. Is there any direct education or public dialogue so engineers understand what actually has to happen?

Yosef Levine: As a regulated profession, we have standards evolving around cloud and information security. In any auditing firm, you have financial competencies and internal control specialists from business process and technical systems sides. If you are in an audit process and only speaking with a financial auditor, bring technical auditors into the discussion. If they do not deeply understand DevOps or cloud, ask them to bring cloud experts. You are giving the auditor a chance to bring true value and to help you communicate better. Most midsize and large auditing firms have that capability. It may just be an ask and how you phrase it.

Topo Pal: The reverse is also true. Internal auditors may believe the DevOps things are good and better, consistent, repeatable, and believable, but external auditors may say there is a book written back in the day and somebody should change that book. Who is going to rewrite the book?

Yosef Levine: If you are not having a meeting of the minds, it is a relationship problem versus a regulatory problem. Escalate it to the partner in charge of the audit or the head of internal audit. Even at Deloitte, we have our own internal auditors. I end up teaching them the cloud. We engage in meaningful dialogue so people understand how to take audit principles and apply them to the cloud and DevOps. Do not lead with, "No, you are wrong." Offer another position and empower people with knowledge so they can see through the same lens. Every industry is disrupt or be disrupted. That is not a technology strategy or a regulatory strategy. It is a business strategy. You still have to protect the organization, but there is nothing to protect if you are bankrupt.

Audience: With policy as code, do we need a domain-specific language or meta-language for audit? If policy can be defined as code, translation error goes away.

Yosef Levine: I will give a parallel. In the US we have XBRL, an XML-based standardized way, with taxonomies, of filing financial information. It started when auditors and engineers got together in consortiums. The concept was a common language for comparative financial information between industries and similar clients. We have tried to do it. It takes time, effort, and energy.

John Waters: Getting to a DSL is next-level thinking. We see architecture teams creating Word documents that get published and then sit in Slack, SharePoint, or elsewhere while people do their own things. Tools like Azure Policy, AWS Organizations, Puppet, Chef, and similar policy-as-code or remediation tools can put true teeth behind those documents. We are using that as evidence to internal controls teams: this is 100% sampling size, this is our control objective, this is our policy that either audits it or denies and prevents deployment, or fixes configuration drift. A provider-neutral DSL may be a next step, but as an industry we are at the initial cusp of adopting policy as code.

Audience: How should evidence change when controls are automated and built into pipelines instead of being paper documents, diagrams, screenshots, emails, and demos?

John Waters: We are enabling controls teams to go into a portal, select a product, geography, life cycle, and pull information on demand for a given environment. They do it themselves.

Yosef Levine: It is called read-only access. It is the best thing in the world. DevOps does not have to do anything other than sign off on the permission to allow them in.

John Waters: There are ancillary artifacts like architecture documents, and where possible we provide links to master repositories. Auditors may do additional sampling, but that should be the exception, not the norm. The norm should move toward a DevOps approach to control automation and evidence gathering.

Yosef Levine: For design and implementation, if cloud architecture and controls redefine a security policy with hardened configurations, the process of coming up with those configurations and publishing them into the master repository becomes a control in and of itself that gets tested. The automated process is when teams pull that and put it into code.

Audience: Sometimes after remediation, auditors require seeing a control operate for 60 days before validating it. That seems at odds with fail-fast. Do you see that changing?

Yosef Levine: It depends. There is a concept of runway of controls: the period in which a control is set to operate. The frequency of a control is directly correlated with your situation. If a control is quarterly, like a quarterly access review, and it failed, you have to wait until the next quarter to revalidate it. If it is an automated control that transpires multiple times a day, typically we pull a sample of one, maybe a few if conservative. But that assumes change management controls governing your automated controls have been thoroughly tested. You should ask, "This is not a manual control; it is an automated control. Help me understand why that level of runway is important."

John Waters: Part of that period is proving this was not just a one-off hotfix and that the organization will not immediately regress. If you have repeatability and can show you do it many times, that can be part of the discussion. If there is not a lot of repeatability, it can be time-based.

Topo Pal: In a future where everything is automated, does that affect the workforce in the audit community?

Yosef Levine: Just as today there are not enough people who understand DevOps, there are not enough auditors who understand technology and DevOps. The more we automate and make audits go quicker, I am not worried about jobs for auditors. I am worried about people who truly understand transformational technology and how to secure it, because it will always evolve. Ethical AI controls could suddenly exist in five years and we will have to figure out how to test them, just like we had to figure out how to test DevOps. The future is bright for controls, but adaptability is important.

Audience: Agile, Lean, DevOps, and Christina Maslach's burnout work emphasize human aspects: joy of work, satisfaction, trust, autonomy. When you audit, are you seeing interest in human dimensions like workplace engagement and the principles behind the human side of work?

Yosef Levine: In the UK, auditors are in the headlines a lot: what is the value of an audit, are auditors missing things? The profession is thinking about the value proposition of an audit from a financial lens and whether other elements of a client's organization, separate from audit standards, should be considered as part of responsibility to the investing public. That inflection point is being driven from the UK, India, and other jurisdictions. On a micro level, with peer reviews, can you assume a peer review is an effective exception mechanism if people cannot speak out with trust? That is part of the control environment. People have to feel empowered. On my team, I want the quietest person to debate me because if you are in a risk or transformation role, everybody has to have a strong voice. If someone is not empowered to reject a PR because another person has more seniority, a savvy auditor should look at whether PRs actually go through iteration or whether everything is just pulled into master. It is not just whether the PR occurred, but whether it is an actual healthy control.

Audience: In complex, highly regulated organizations, when DevOps is done well, auditors can become major change agents. Is the audit community incorporating these principles into learning and certification programs so they can be champions earlier?

Topo Pal: Going forward, as Gene has promoted good practices across the enterprise for years to make it happier, better, sooner, faster, and safer, how can the DevOps community work with the audit community better and come to the same table?

Yosef Levine: As an auditor, when I go to a client and do an audit, I do my best to stay ahead of issues and problems and clearly communicate with management and my client about findings. We are not there to get you. We are there to validate what you have done. That is a core principle of auditing. At Deloitte and our competitors, we are educating professionals in the cloud. If you see a communication breakdown or somebody does not get it, go to the chief audit executive, CFO, or whoever owns the relationship and say, "We need some help. We need additional perspective." Escalate it. You are never alone. You just have to know when to pull the "I need help" card.

John Waters: We have auditing for non-auditors for our IT team. There is also an aspect of educating on the technology controls side: the steps that lead up to a SOC 2 or ISO 27001 certification, and what training can occur from technology controls teams to DevOps teams. The technology controls team can say, "Here is how the watch ticks, and if you see that over here, you should expect it to show up here and here. If you do not, that is a finding." If it is, "I have the secret answer and I am not going to tell you," that creates unhealthy behavior. People go to opposite sides of the room, and we want to bring them together.

Topo Pal: With that, thank you very much. Really appreciate it.

Yosef Levine: My pleasure.

Topo Pal: It was awesome. Thank you. And thank you for all the questions.