Amélie Koran

Hello all. Welcome to my talk today, Beyond Firefighters Versus Safety Matches, where we will talk about growing the DevSecOps pipeline.

Mandatory forward-looking statements. To be honest, being required to show these during a presentation for a few seconds is a lot less painful than having to get a talk approved as a government official, so I'll take the mulligan for you all.

So hello, I'm Amélie Koran, senior technology advocate here at Splunk. I've been with them since 2019. Prior to that, I spent a decade as a public servant, split up by a year at Disney. It happens, I literally needed a vacation, I guess. And by vacation, I mean working on the future of technology for one of the largest media conglomerates in the world. Bonus points for being able to say I went to Disneyland for work as research. During my time in the government, I worked for the Interior Department, Treasury Department, and left service as the CTO for the US Department of Health and Human Services, Office of the Inspector General. Oh, and I helped start the US Digital Service while on rotation at the Office of Management and Budget at the White House. So that, too. I'm mainly a security person in my career and tend to circle a lot in that community, including speaking and volunteering as a staff at a number of conferences, including multiple Security BSides conferences and as a DEF CON goon, which brings me to my topic today.

Three years ago, I stood up at ShmooCon in Washington, DC, adjusted my soapbox and told the general establishment that the way they were trying to handle an important shortage or shortcoming was wholly wrong and why they should listen to me. It made it easier that I was surrounded by friends and colleagues when I essentially did my best impersonation of Martin Luther nailing my treatise on the door of the Catholic church in Washington, DC. Most of that establishment tended to say that what you needed to do was to wave a magic wand, and you can create or find talented security people for every organization, and that everything was fine. Or that you can fix this by adjusting curriculum in schools, asking a passing UFO if they had any security pros they could lend planet Earth, or just wait it out until the rest of everybody's demand was met and start from there if you didn't want the fight for the talent.

This, in fact, was not fine. I thought cooler heads would prevail. People would think a bit deeper about this problem, and maybe it gets to solving some of those issues, either through policy or direct action. Oh, no! Excuse me. Excuse me. All right, move on. Nothing to see here. Please, disperse. Nothing to see here. Please.

I know this is odd, but the more we force folks down a given, and albeit narrow, path, there's going to be some pushback and some pressure differential, a kind of a back pressure, if you will. Some will consider this some level of gatekeeping, but it's more of ensuring that those for which our recent policy to get as many people into cybersecurity to solve the pipeline problem may hit kind of a bottleneck. We may need some specialty talent. We may be forcing too many people down one path, creating some resistance, and possibly lowering bars for skills and salary that companies are willing to offer for some of the important expertise. It's good that we are catching up a little bit to the perceived talent shortage by addressing this earlier in education, training, and transitioning from other roles, but we need to do it smartly. This is similar to my astronaut problem from my prior talk, where the prestige role may have a limited payoff and a short lifespan, even after taking a lot of work and effort to get there.

So conversely, with the rise of DevSecOps, which is not DevOps plus security, the idea that we may be putting too much on one skill set or role may also prove to be somewhat problematic. While there may seem to be limitless supply of folks who say they can do this job or want this job, this actually isn't the case. Plus, the same skills gap we find in security, we also find in these roles. And with that, are you then putting water or gasoline on that fire? It's not like there's an endless supply of these folks either, but many may find it easier to put a cape on than claim specialty in the particular role.

So we want the skills pipeline to ever be refreshed and sustainable, but we don't want too much generalization to lose the depth of some skills and overloading of some roles. How do we find that happy medium? Or do we want to? Is DevOps and DevSecOps something that's realistic and sustainable? Is it, at the very least, what we want and actually need? Or is it just a fad everybody wants to do, like look at Tamagotchi and Beanie Babies? What's the care and feeding required to make this work? Namely, what are the skills we need to generally focus on? Or are we running down the wrong path?

Just like DevOps isn't stacking development on top of ops in the relation to the workload and responsibility, but it's sharing and enabling some of these capabilities in a single role.

DevSecOps is moving and sharing skills and responsibility among a single role, but where does all of this, the rest of the stuff that got peeled off of the traditional stack, end up? This is where you allow for some specialization and the ability to focus and optimize and make efficient rather than having to build it all into one role, person, or responsibility.

In short, it's okay not to be the wizard. It's okay to be an apprentice as well.

DevSecOps is, again, not DevOps plus security, but also a balance of the various forces at play within the organizations for technology.

Part of the finding of the balance between the as is and the to be, or the present and future state, is an assessment of the talent and your skills within the organization and what parts of it exist to make the transition easy, and which gaps are there that need to be supplemented or built up. Sometimes this is merely training those who are already there in some new skills. Others, it's finding entirely new resources to augment your team. The challenges with the skills augmentation is finding ways to add and plus up the folks you already have without overloading them with work and responsibilities.

DevSecOps is about shared and balanced, not less and overloaded.

So think of it this way. As you see here with the diagram being a tight Venn diagram where the skills and responsibilities generally overlap fully, there's still some specialty roles within each main category, but it's a tight weave elsewhere. You can approach a more comprehensive overlap as it begins to appear as just one circle, but there will always be just a few responsibilities that the original team may need to solely own or as such, but may not have to actually do too regularly. Those diagrams on the right are typical organizational patterns I've observed in my multiple orgs that I've worked at, for, and in. Each are not quite at equilibrium, which is a goal here, but where the forces from each team are balanced, but as well as the motivations of the team and overall strategy and organizational leadership are in pull. Some are led by a strong development arm, but others may have a subservient operational and security teams. In other cases, there's some movement and conjoining where they touch, but not a core shared DevSecOps culture. This is as much a process and strategy issue then as well as much of a cultural and personnel issue.

So anybody might recognize these classical looking fellas, Tantalus and Sisyphus. You probably have heard of the latter, but maybe not so much about the former. We often talk about how tasks seem Sisyphean, all the work, dedication, effort, but we don't get our payoff. To note, this was a penance that was put onto Sisyphus by Hades as he cheated death, which in itself is a good story. I highly advise reading the backstory. The former, Tantalus, anybody see here where the name of the root comes from? Tantalize. His penance, which wasn't from exactly as doing something as appearing as noble as Sisyphus, was being placed in a pool of water neck deep, and when thirsty, having the water move away from his mouth so he couldn't drink, and above the pool was a tree with fruit from which he wanted to grab and eat, being moved out of reach every time he grabbed. By the way, his offense was revealing the secrets he learned while hanging out with the gods to the mortals. Killed his son, fed him to the gods to see if they knew if they were eating his son, kind of a nasty thing, and then stole ambrosia, the nectar of the gods, and also gave that to the mortals. Kind of sounds like an IT vendor, right?

Tech seems a lot like these two regardless. So either a lot of work to see it just rolled back to zero or being tempted by a thirst or hungry just to see it always out of reach. And these were penances. Seems like every day in tech.

With the move to DevSecOps or even DevOps for that matter, you are using the desire for speed and optimization to play a little bit with your ego.

As mentioned earlier, our pipeline for tech is still a constrained workforce, but market demands put the desire from orgs and their leadership to put more features and capabilities out quicker and quicker. So they see the adoption of automation and CI/CD methodologies in their orgs as a way to increase and satisfy those demands, but not realizing you need the right folks setting it up, running it, and monitoring it.

To not bring out the proverbial sacred horses out of the stable, but if the lessons from "The Mythical Man-Month" have taught us anything, you can't throw more people at a problem or expertise and have it sped up or solved properly. It's a matter of matching needs with the staff and ensuring that you will find a balance that allows them to best apply their skills to a problem. So the push for a universal SRE or even a pluggable developer to take on every aspect of the design, development, delivery, and operations of a service or application is only going to get you tired SREs and developers.

Work the strengths, but not the strengths to death. Develop the teams of talents and cross-train where possible or have complementary skills, but never, ever throw it on the backs of small teams to build, secure, and operate. Okay. It's great to be all soft and conceptual, but where's the data to back up this warm and fuzzy talk? Okay.

During my initial presentation, I found major gaps between the Bureau of Labor Statistics, BLS, estimates for roles and expertise given a job category in their occupational outlook handbook, which is forward-looking supposedly, but maybe misses the actual realistic demands versus what the economy actually is supporting day to day. To note, my ex worked at BLS, so I have a little idea how the place works after 10 years of marriage and half a decade of them coming home and sharing their own frustrations and questions. If anything, I look around my neighborhood in Northern Virginia and say, "Well, there's a lot of data centers being built, people doing a ton of online shopping," and really wonder if wind turbine service technicians and solar installers are the actual jobs of the future.

In the top 10, you do have information security analysts, so they may not be too far off. With a median income of 103K per year and those installers ranging from 46K to 56K per year, so I wonder what's going to draw more folks into a career when it comes to wanting to secure their future income.

When I researched this three years ago just before they updated their job outlook, it was barely above 100,000 people with this job. Now it's listed 131,000 with a decade growth of about 41,000 more folks, a 31% increase. So gives you a pause for a moment. It was 100K in 2016 to 120,500 in 2026, so it's not a lot.

Now BLS has computer programmers and software developers in two different job series, but for simplicity, I will focus on the latter. While they aren't even listed in the top 10, they supposedly make up over 1.4 million or 1,469,200 jobs with a suspected 316,600 expected job growth in the next decade, which is a 22% increase. Unless software developers subsume the security folks, not only can you see a growth issue with the needs for supplies, but a major difference in skilling, like nearly a one to 15 ratio. Slightly larger if you want to toss in computer programmers, which brings it to nearly a one to 17 ratio. So that's a lot.

Throw in what's termed as network and system admins, and you get 374,000 jobs with a 16,000 growth in the next decade of about 4%. There's some overlap of all these with computer systems analysts job descriptions, but these would most likely be your architecture business consultants and some of the testing and requirements folks, or even agile coaching types of skills, which are 632,000 people with nearly 47,000 improvement on employment for an outlook of about 7% growth. There's definitely not going to be enough security people, according to government outlooks, to keep pace with all the folks building and deploying apps. All of this will ride on software developers. Is this a good or a bad thing? It's a lot of numbers. I think it's incredibly bad for many reasons. So what does our industry actually have to say about this? Just because a cursory look at what the government thinks we need and the actual reality of the situation seems way, way out of whack.

My initial talk sourced a really cool site that also sourced their numbers based on the actual open requisitions for security positions. I think it's reliable. It's been cited by a number of folks in industry and academia before and controlling for the unicorn clause, which I dictate as having one person for a req with every skill under the rainbow and is going to conferences and speaking with peers, students, and others, and is at least grounded in some sort of reality.

Oddly, this project, Cyberseek, is also supported by NIST and DHS through NICE. So there's a lot to ask why the government stats and tools differ so greatly in what they are representing. So let's see. Number slide. Yeah, the right number slide. So now, besides needing bodies for roles, the other ask of executives and other leaders is the skills gap. So what does that all mean?

In short, you can get the job, but can you do the job? Sort of what the question that's asked in "Joe versus Volcano." So while you can fill a slot, will you be actually able to do the job effectively and correctly?

If we are already seeing a massive skills gap, what's to say asking a developer and operations person to suddenly take on those roles? They can learn it, but with how much rigor than they can actually may have their skill overlap their original skill sets. Will we expect a junior or newly minted DevOps or DevSecOps engineer to provide the same level of expertise as a senior one? And with DevOps and security folks making up three sliders on their expected work coverage, are all the levers at the same level and you kind of plan and compensate for that? Also, for all the new and awesome stuff that DevOps folks are asked to add, such as AI, ML, and other bits of automation, you can kind of assume that everybody in the leveling for junior, mid, and senior are going to be able to grasp and exploit these new demands and let them alone secure them.

The last numbers from 2019 in my talk, there were 716,000 employed security people with 314,000 openings. The Evans Data 2020 Worldwide Developer Population and Demographic Study notes that the total number of software developers globally is about 24.5 million. This graph here demonstrates a similar stratospheric growth in employment, just with the kind of growth expected in the next couple of years. Now, it's nowhere in the realm of the number of people whose main source of income is retail sales, farming, or other professions, which are some of the larger employed roles people maintain globally, but it's still a pretty high percentage. That's still 3.1% of the entire global workforce. Even then, this encompasses career, contract, self-employment, subsistence, and other temporary work when you look at the workforce.

So Ian Malcolm says, "I'll tell you the problem with scientific power that you're using here. It didn't require any discipline to attain it. You read what others have done. You took the next step. You didn't earn the knowledge for yourselves, so you didn't take any responsibility for it. You stood on the shoulders of geniuses to accomplish something as fast as you could, and even before you knew, you had patented, packaged it, and slapped it on a plastic lunchbox, and now you're selling it. You want to sell it." We're told as individuals or executives that we will resolve our problems through the application of fancy new technologies or exploiting new advances if you only buy this product or service. Kind of sounds a little bit like Jurassic Park. No, just stop it. It doesn't work that way. It never has, never will. If you're a traveling folklore seller heading from town to town, or in this case, company to company, industry to industry, peddling the new thing they can make some money off of, but you're left wondering what this actually does once it's in your hands because they didn't tell you there were extra steps that were needed or didn't even work actually in the first place. Now, I know this is virtual, but I bet three-quarters of you are nodding your head in sheepish agreement to that statement.

The sequel film, "The Lost World," I'm sure you've all seen it also has another gem of wisdom from Dr. Ian Malcolm when they naively wonder in the magnificence of what they've been demoed or promised. When your plans go awry and we're left with our consequences at odds with our wonders and expectations, with only folks who've been through the proverbial s**t to offer the warning, "Ooh, ah, that's how it always starts." Then later it's running and screaming, "So what are we supposed to do?"

Well, take the word from somebody that says our problems will be solved with an elixir or actually think deeply about what we're about to get ourselves in. Who would you trust?

So my experiences. Wow, 15 slides in and I'm actually sharing IRL experiences.

This talk was built nearly on a quarter-century of experience doing a lot of this stuff. So you can listen to me, you can ignore me, whatever. But here are some nuggets of what has and hasn't worked. I don't have all the answers, let alone that I can actually give you an answer, but I'm giving you a heads-up here. Consider what we're doing, essentially, down the same path. So looking at the problem, thinking we can just do the opposite of what we've done wrong in the past. It needs something new, though. Now, I don't mean to nuke it from orbit, meaning it's the only way to be sure. However, the issues with it being organic, in short, if you can't shoehorn a change in, and you don't have the right option, is that it's going to grow to the environment it finds itself in. So if you're not fostering a good environment, it's going to just be a disaster. If your culture, policies and procedures, technology, and business itself isn't set up to allow you to grow prize-winning flowers, you're going to end up with ditch-weed quality dandelions. If you don't have an organization that is willing to change and recognize that they need to change, and has the patience for it, I bet that is just going to be sadness.

For an overarching case, I will take my immediate past, the federal government, that primarily everybody sees as this antiquated, with legacy systems and thinking, that's been most receptive to change and modernization, actually.

It came down from two directions. The actual fact that there were many systems and solutions that were just at or rapidly approaching end of life, and even some well beyond that. The nature of this came from how the government tended the work. With changing leadership, weird budgeting, rotating crops of contractors, and if it's not broke, don't upgrade it.

Conversely, if you want to give the Obama administration some level of well-earned kudos, its introducing of the Digital Services Playbook, which definitely leaned into Agile and DevOps ideas and methodologies. Some contractors for projects were doing it already, but it was never really formalized, like previously with Cloud First as a mantra and initiative from within government.

We saw the advent of 18F, born out of the Presidential Innovation Fellows, and the US Digital Service, which was a joint initiative between OMB and the Office of Science and Technology Policy within the White House to approach getting UX, DevOps, and Agile into the top levels of project initiation and management, where 18F were more of the consultative doers and initiators of such stuff.

Some agencies were very resistant to these efforts. Sometimes a little worm tongue, if you're a "Lord of the Rings" person, from contractors who had vested interest in keeping the status quo, but also knew they didn't have to bench talent to keep up with sea change. So here's a brief on the several problem models I proposed in my last talk and how they're applied to DevSecOps. So the accountant problem. The commoditization of your career or job, where the value that companies and recruiters put on the skills required and the expected availability of these candidates. This is the point where the market feels that there's no need to fight for the skill set. This has happened in many tech roles, but for different reasons, but namely due to the disconnect between hiring managers, HR, and understanding of required skills. The seal problem. This is also titled as the rockstar and ninja problem as well. There's been an additional shading to this, especially in the security realm, over-professionalization of the career space that isn't quite mature enough to realistically support it. This is often a system that's gamed, where people are more bolstered by certs and credentials without a demonstration of applicable skills. So it's juggling and diluting the market for talent, confusing HR and managers, and creating an arms race to be noticed by professionals and job hunters. So trying to stand out often elongates the hunt for roles, but also makes employers gun-shy if they end up having a spate of bad hires, simply only looking at credentials rather than the history of delivery and achievement beyond the paper.

The astronaut problem is a bit more exacerbated in the DevSecOps world as compared to the security-only world, where being perceived an expert in a particular skill set is then burdened by which specific tool or platform you're expert in, be it tools offered by Google, Amazon, Microsoft, or others. The uniqueness is only as good as its application, and in the pace of how quickly technology, and any technology, changes and matures, to be extremely deep and not particularly broad may give you a career of reinvention or the possibility of being a one-trick pony.

And then the firefighter problem. This is ultimately where we all go in. Best of intentions work ourselves out of a job, either due to stress or boredom after a while. It takes a toll on us, and eventually we need something else. Some will leave, some will pivot, but often a lot of what we do is being pushed to automate things. However, as we start to do these savers, we still need people with these skills and minds to do all this, and it's a little unique. Not everybody can be an SRE, just like not everybody is a malware analyst, but not everybody wants to be those either. But those skills are a need, and there's very few times you're going to ever be able to catch every case and give it an auto-handler.

Mission impossible. No. Move fast and break s**t. No on that as well. This is gap analysis. Capture the desire to make change, do good, and see improvements, but also know that we need to ensure that we aren't just going through the motions, burning people out, and not delivering quality solutions. The way could be to look at this as a stepwise maturity model, of which there are some roadmaps, and decide if this is prescriptive or advisory.

Some orgs do not take well to rote checklists, but as part of maturity, having a documented and repeatable process that folks can follow and refer to, especially in cases where stuff may go sideways, is a sign of maturing and the way to take the next step up. What makes up the right talent for your objectives? Well, do you need a generalist? Do you have highly specific problems to solve?

One of the ways to bridge some of these self-assessment gaps is to look forward to short-term or contract work in some cases. There's plenty of guns for hire, as it were, those that have specific skill sets and are being well-regarded for being good at it.

But also realize they're our scalpel in this modernization surgery and will be fine with moving on after their initial work is done. Just make sure things are well documented before they are handed off, and that your regular staff know how to take it from there. I know this from experience. It's one of the largest worries an executive taking on much of this transformational work is wondering if they need to find the unicorn and try to woo them on staff or build actually a solid team of folks that can handle most things and do a lot of the others, but highly specialized work is on somebody else who may be more transient and a little bit more expensive. It's also up to leadership to be very engaged and box a lot of this work and set up expectations between you and that specialist.

As part of what you need to do to make sure you're balancing the work and talents is to look at where you are and where you need to go. As mentioned before, who knows this from a fear of missing out fad, or that IT folks are scrambling in order to match another industry or sector in competition, or it's actually making things better given resources, talent, technology, and advances and so forth.

Not everything has to or will move over to DevSecOps. There's never a perfect end game. Plus, not everything fits well in a shared management responsibility model. It's like asking the tiger cage cleaner to also be their veterinarian. Rarely do the skills and expertise cross over the same responsibilities in real life. For sure they can help and say if that cage cleaner was interning and this was part of either hazing or part of the intern's responsibilities, but it would take some time to spin up that level of expertise where that task was easily interchangeable.

This is also why there's multiple tiers and help desks in SOCs. The scaling and cost at certain levels do not beget everybody being a tier three grade expert or needing to be on hand for all the harder issues or events to handle. There's also some issues that will just hang on until they die, such as having an application portfolio that the sunsetting of an app or service isn't refactoring, replatforming, or rebuilding, but just setting a sunset date and letting it die with decreasing DME and support until folks are weaned off of it. It's not pretty. You can't tie it up into a bow, but you will need to analyze what work streams can be shared and which ones will and should remain on their own individual ones.

This is a jumping-off point. While I've been around a bit, seen different companies, different sectors, and types of products and services they offer, the way to address this is highly individual. As I highlighted with the Venn diagrams of doom earlier, where you start from is highly individual, both the size of your org, its maturity, and the cultural structure of them.

How you navigate that is up to you, your teams, and leadership. The speed is also part of how the organization will adjust these bubbles and circles and their interactions. This may be both cultural, having the right people in place, but also risk tolerance for the change that they may have.

Getting you to a proper staffing and talent in each of these areas will help you reach that mark, getting your ops, dev, and security teams working closer and better together. But it's a long road. It may branch out. It may change. It may just be a step into something else.

We know, especially in tech, that nothing is static, and there will be something after DevSecOps that we'll be asked to chase. It's an imperfect model, and those models will change along with society, cultural, and technology, as well as our own education pipeline, our goals, and the desire for folks to want to be in tech overall.

Market demands will offer branching opportunities, but you will get to a point once groups are resourced and aligned properly, or at least in a working capability, and it'll make it easier to address what's next. For some, it can be growth, others positioning for an M&A activity, others looking to split functions or just plain stabilization.

Oddly, the last bit never gets enough credit, as sometimes just getting to a place where it's not always a fire drill can be an admirable thing in itself.

So where do you go? It's up to you.

So thank you. Be safe, stay healthy, and always be learning and exploring.