Clarissa Lucas

Welcome to DevOps and Internal Audit: A Great Partnership. Before we dive into the material, Rusty and I will tell you a little bit about ourselves, what we do, and where we work.

My name is Clarissa Lucas, and I am an IT audit director at Nationwide Insurance, which is located in Columbus, Ohio. I focus on auditing our technology organization and providing IT audit support to our business auditors. I am not your typical IT auditor, though. In fact, up until the end of 2018, I was not an IT auditor at all. I have been an operational and business auditor as well as a risk management leader, and recently got the pleasure of joining the IT world as an auditor. This non-traditional background and journey into IT auditing has enabled me to bring a fresh perspective and challenge our customary way of doing IT audits. This has resulted in more meaningful views of risk and sustainable remediation of gaps. When I am not getting all geeked up on risks and controls and audit procedures, I enjoy spending time with my family and staying active by running and lifting weights. Rusty?

Rusty Lewis

Good morning or good afternoon, everyone, depending on where you are. My name is Rusty Lewis, and I am an IT staff auditor, also at Nationwide, working with Clarissa in our internal audit department. I joined Nationwide nearly two years ago, but prior to that, I worked for PricewaterhouseCoopers for two years in their IT audit practice. Similar to Clarissa, when I am not equally excited about IT audit risks and controls, I enjoy spending time with my wife and our dog, Scooby. I also enjoy recording a podcast with my brother-in-law, where we discuss our passion for video games, technology, and pop culture.

Now I will turn it back to Clarissa to give a brief introduction about Nationwide.

Clarissa Lucas

Thanks, Rusty. Let me share my screen. All right.

Rusty and I both work at Nationwide Insurance, which is located in Columbus, Ohio. Nationwide exists to protect people, businesses, and futures with extraordinary care. We accomplish this through a diverse portfolio of products, including financial services and property and casualty insurance. Our financial services business includes offerings such as life insurance, mutual funds, and retirement plans. The property and casualty business includes commercial lines such as agribusiness and standard commercial insurance, as well as personal lines like auto and homeowners insurance. We also offer some less obvious products, such as life insurance for your pets and insurance for your sports cars. We are here for our members and the things that matter most to them. In fact, Nationwide is number one in 457 retirement plans and is a number one pet insurer and writer of farms and ranches.

Extraordinary care does not end with our members. It also extends to our associates, partners, and communities. Nationwide has been named a Fortune 100 Best Company to Work For and Best Workplace for Diversity. Now that you know who we are and what we are about, let us talk about the objectives for today’s session.

As internal auditors, we work closely with our clients and often receive questions as our clients are looking to make a change or implement a new process or tool. They want to make sure they get audit’s stamp of approval before proceeding. We often hear questions like, "How can we pass an audit when we move to a DevOps operating model?" Or, "What are audit’s requirements for segregation of duties when I am implementing DevOps practices?"

Incorporating DevOps practices into the delivery pipeline can have significant impacts across the three lines of defense, from developers to internal auditors. At the end of this session, you will have learned to change the way you think about risks and controls, understanding why the controls are performed, and selecting the best control or controls for a specific risk in a given situation. You will also learn how DevOps can positively impact your experience during an internal audit.

We will accomplish the learning objectives by defining audit’s role, exploring the impact of DevOps on our way of thinking, walking through examples of controls that may look and feel different in a DevOps environment from what you are used to, and taking a focused look at key risk and control considerations. In our examples, we will focus on the risks mitigated by the controls. Then you can apply this thought process to other risks and controls and use it to mitigate those risks in a way that makes sense for your organization and operating model.

To understand how DevOps can impact your experience during an audit, you first need to understand audit’s role in risk management and how that impacts you in your role. To do that, let us first introduce the three lines of defense.

The first line of defense owns the risk and executes controls to manage the risk. An example here would be the development teams. These are the individuals who are writing code, testing the code, and reviewing and approving any changes. The second line of defense is responsible for policy creation, defining a risk tolerance, and monitoring adherence to those policies and those tolerances. An example of a second line of defense, building off of the development teams, would be information risk management. And the third line of defense is internal audit.

Internal audit provides assurance to the audit committee of the board as well as to senior management through independent assessments of risks and controls. We accomplish this through seeking to understand what could prevent the organization from achieving its objectives, and we consider a multitude of risks. We then evaluate the actions that management is taking to mitigate those risks within established tolerances.

To tie this together, we conduct integrated audits to evaluate business risks and controls and the systems or applications that support those business processes. Examples of IT controls evaluated could include change management, user access, and interface integrity, and we will also look at adherence to that information security policy and any supporting standards.

So part of understanding who we are as internal audit is also understanding who we are not. We are not IT risk management. We are not compliance. We are also not the external auditors or a regulatory body. We are not the development teams or operations. It is important to note that internal audit does not tell management how to manage a risk. Internal audit does partner, though, with the first and second lines as well as external parties as we provide assurance on the organization’s ability to meet its objectives.

Shifting to DevOps is a transformation of both thought and practice, and it requires partnership and collaboration between the three lines of defense. For example, if your policies require segregation of duties between developing code and promoting it to production, the three lines of defense need to work together to understand whether this works with a DevOps model. If it does not, and the organization truly wants to move to DevOps, the policy may need to be revised to allow for a different method of controlling the risk that is currently mitigated by segregating those duties.

A little context about Nationwide’s internal audit team. We have about 70 internal auditors with a wide variety of experiences, degrees, and certifications. We focus a lot of our professional development on increasing our business acumen and better understanding the areas that we audit. For Rusty and I, it is technology. Our technology audit team provides assurance over the technology supporting key business processes at Nationwide.

So I have touched on the audit part of the DevOps and audit partnership. Now Rusty will get us into the DevOps side of the partnership. Rusty?

Rusty Lewis

Thanks, Clarissa. This picture is a visual representation of how some practitioners may see the DevOps movement. There is a lot of items at stake, and things can go very badly really quickly. While modifying the development process, incorporating DevOps into the delivery pipeline can also have significant impacts on assurance providers and security practitioners, and this may cause people to think of DevOps as a bull in a china shop, as this image here depicts.

Continuous delivery challenges the enterprise to balance control execution with speed, and maximizing automation can change how traditional controls are performed and assessed. As an example, this gives all of us an opportunity to think about segregation of duties differently than we have in the past. Typically, segregation of duties is thought of as two separate people in a single process: one person developing the change and another promoting it to production. We may begin to think of duties being segregated through automation and with the potential higher reliance on other controls.

Moving to DevOps has a strong value proposition, and while it may change the way we think about risks and controls, at a baseline level, the risk and control environment are relatively similar to the control expectations of today. I will walk through some specific examples in a few slides, but a few quick examples for now would be the expectation that access is restricted as appropriate and changes are tested prior to being migrated to production, regardless of whether today’s change management process is followed or DevOps processes are followed. As Nationwide adapts the development process, internal audit is also on the same journey to adapt the way we assess controls.

Now, we have talked a little bit about risk. So what truly can go wrong using the example of change management? The primary risk associated with change processes is compromised systems or data availability, integrity, or confidentiality, which can have adverse impact on business operations. Having a strong control environment in place helps you prevent this from happening by preventing the items listed here on the slide, including unauthorized changes introduced into production and introduction of security vulnerabilities. Internal audit performs audits to evaluate controls and determine whether these items are well managed. Next, we will walk you through how internal audit currently evaluates these risks and how that may change as we adopt DevOps practices.

Now currently, our testing focuses on these three key controls. The objective of the approvals and authorizations control is to ensure that all changes related to an application being audited are logged, prioritized, authorized, tested, and approved prior to release into production. Depending on the total number of released changes during our scoping period, we typically select a sample between one and 30 changes for detailed testing, and we will then request documentation evidencing approvals for sample changes at various stages of the change management process.

Our second control, segregation of duties, focuses on verifying that appropriate separation of duties exists within the change management process to prevent individuals from writing, testing, and promoting their own code to production without independent checkpoints. Lastly, our third control, metrics reporting, really focuses on monitoring and reporting activities to determine effects of changes on business operations.

Now, let us take a closer look at the second control, segregation of duties. So we have to ask the question: Why are duties typically required to be separated? Let us look at an example. In the old ways of thinking, or traditionally, a developer is not allowed to promote their own code to production by themselves. They would need to have someone else from, say, operations or quality assurance promote it for them. And so then the question again becomes why? Why do we need to separate these actions from one another?

Well, it is to make sure the code is not malicious or fraudulent in nature, or to make sure the code does not create disruption to systems or data issues when migrated into production. And so what does the person promoting the code do to ensure that these things do not happen? They review the code, and this helps in detecting and fixing defects early in the process. It also helps to maintain a level of consistency in design and implementation of the code. The review also allows for uniformity and understanding, which will help the interchangeability of team members in the case of non-availability of any team member.

So let us say the organization’s risk appetite allows for detective controls, or controls that focus on finding and fixing after an event or incident has occurred, rather than preventative controls, or controls that focus on checking before an incident or event has occurred. What sort of actions or controls could mitigate the same risk here?

Now, there are a number of examples here on the slide, but I want to focus on automated reviews or tests. So what if we wrote a script for an automated test that would review the developer’s code and promote it to production once it has been tested, reviewed, and approved? The automated test would then enable developers to more efficiently and quickly review code and on demand. Now the duties are separated, not between two humans, but between a human or developer and a digital worker or bot or code. We also have a mechanism now for providing immediate feedback and learning to the developers and from promoting code without extended wait times.

Now I will turn it back to Clarissa to take another look at a previous slide outlining our three key controls of focus.

Clarissa Lucas

Thanks, Rusty. A few minutes ago, Rusty walked us through these traditional change management controls. He just showed us how segregation of duties can be achieved using DevOps principles. Take a look at this slide again, this time focusing on the approvals control at the top of the slide. As a reminder, the objective of the approval control is to ensure that all changes perform as expected and that they do not cause problems when they are introduced into production.

Now let us look at an example of how change approval controls might differ in a DevOps environment. We ran into this recently in one of our infrastructure platform audits. In this instance, individuals have access to both develop a change and move it into production. We are currently working with management who is mitigating the risk that changes can be made without going through the enterprise change management process. One way to mitigate this risk is to work with the change management governance team to evaluate the types of changes, to determine which require approvals for each change, and which could be part of a change pre-approval process.

That pre-approval process is where you would gain pre-approval for the type of change, and these are going to be typically your low-risk routine changes, and all changes that fall into that category can be made and logged without obtaining a separate approval. For changes that are routine, the changes are logged and management monitors the logs to ensure that only routine, low-risk, pre-approved changes are made without gaining that separate approval. Changes that are not routine are also going to be logged, and management can monitor those logs to ensure that these changes have a corresponding approval.

Now, Rusty and I have walked through detailed examples of controls that look and feel different in a DevOps world. This slide shows additional risk and control considerations specific to the DevOps environment based on industry best practices. This may be different from the controls that are currently in place in a non-DevOps operating model. In the middle column, you see the risks. These are the things that we are all worried about, not just as internal auditors, because they represent what can go wrong. Controls on the right-hand column are the activities that are performed by management. These are what internal audit assesses to determine whether those risks in the middle column are managed within established tolerances.

Now, I will not read each of these bullets. Instead, I want to walk through an example of how a control mitigates a specific risk and how we would evaluate that control in an internal audit. I will use the example of the risk of implementing untested or unauthorized code, which could lead to increased operational failures, poor transaction processing, or unreliability of the system itself. A control that can be used to help mitigate this risk and manage this risk is automated production deployment.

In this instance, the code is automatically moved to production as long as it meets predefined requirements such as positive test results, completion of a peer review or authorization, or completion of some other control or suite of controls that mitigates the risk that the code will not perform as expected or that it will introduce vulnerabilities once it is moved into production. The automated deployment control will also reject code that does not meet these predefined requirements, thus reducing the risk of untested or unauthorized code getting deployed into production.

How would we as internal auditors test this? Since this is an automated control, we would check the configurations of the control to determine whether it is designed and configured to mitigate the risk. To say that differently in less audit-heavy terms, we would review the configurations of the deployment mechanism to see whether it is set up to only move code into production if it meets the requirements, and that it is also set up to reject any code that does not meet those requirements. We would also test through observation or review of evidence after the control has run one instance where the code meets the requirements to see that it does get promoted as expected, and we would also look at one instance where the code does not meet requirements to make sure that it gets rejected as we expect.

This might sound a little different from the audit experience that you are used to. You might be used to your auditors requesting pages and pages of documentation for a sample of 30 or more transactions, such as code deployments or changes. On the next slide, we will dig a little bit deeper into the impact to your audit experience.

As organizations are changing their software development processes, internal audit is responding by adapting our control assessment practices to better fit the world of DevOps. We now have a better understanding of internal audit’s role in the organization and their objective, which as a reminder is to provide assurance to key stakeholders through evaluations of risks and the controls mitigating those risks. We have also shifted our mindset to focus on the risk and how best to address that risk in a given situation, rather than looking at a checklist of controls to implement.

Now let us look at how DevOps can change your experience during an audit, and why DevOps and audit are truly a great partnership. When you have automated controls, we can look at a smaller sample for our testing because the automation removes the human error element. If it is set up correctly, the automated process will execute as designed each and every time. This means that you spend less time being audited. I know we are not in the same room right now as we enjoy this virtual experience, but I know somewhere somebody is just cheering right now.

And as if that were not enough of an advantage, there is more. Audit can also extract evidence directly from source systems, resulting in fewer requests for evidence, which means decreased interruption to your daily work. And with greater reliance on automated controls with audit trails comes greater assurance through full population testing. This is great news, and I will bet you even like your auditors a little bit more now. Rusty is going to bring it home with a question on everyone’s mind.

Rusty Lewis

So the question Clarissa is referring to, and one that we are actually frequently asked is, when we begin leveraging DevOps practices, or we have been for quite some time, how do we actually pass the audit? The short answer is, there is really no magic formula or one-size-fits-all approach. Over the last 30 minutes, you have learned that internal audit’s purpose is to provide assurance on the organization’s ability to achieve its objectives. We do this through understanding the risks to achieving the objectives and evaluating controls in place to mitigate those risks in alignment with the organization’s risk tolerance.

As you are transforming your delivery pipeline and implementing DevOps practices, bring your internal audit friends along for the ride. Help them understand how you are mitigating risks. Do not be afraid to challenge them if they are stuck in an old way of thinking. For example, thinking that segregation of duties is the only way to mitigate a particular risk. Show them how the controls you have in place also mitigate those risks, and how they can see evidence that the controls are being performed appropriately.

Now we have walked you through a few examples of suites of controls that can mitigate risks differently than what we are used to. Earlier, we also encouraged you to bring your internal audit friends along for the ride. I can only imagine, especially in our virtual presentation environment, that some of you were scratching your head questioning how auditor and friend could ever be used in the same sentence. But the sooner internal audit can connect with operations and development teams, the sooner that relationship becomes less about confrontation and opposition. Instead, it can be founded on trust with a focus on collaboration, creating camaraderie with your peers, and risk mitigation.

Now, on behalf of both Clarissa and I, we want to extend a very special thanks to each and every one of you that joined us today for our presentation. And a special thanks to Gene Kim and everyone at IT Revolution for allowing us to present this year at the DevOps Summit. We also do not want the conversation around DevOps and its impact across the three lines of defense to end here. Our contact information is listed on the slide deck, and we would encourage you to reach out to us directly via email with any questions you might have. Thanks again, and enjoy the rest of the DevOps Summit.