Host Intro (Gene Kim)

Gene Kim: John Rauser is director of engineering at Cisco. We finally got to meet two months ago at the conference, and I was so excited by his presentation. In fact, this is certainly going to be one of the experience reports that we're going to track for many years to come.

He leads engineering supporting Cisco's Zero Trust offering, but you recently jumped into exploring how AI is being integrated across multiple products across the Cisco portfolio. John, can you introduce yourself, tell us about what you presented on in Vegas two weeks ago, and maybe even tell people about the vast scale that is Cisco? By some reports, it's the sixth-largest company in the world now.

John Rauser

John Rauser: That's right. Thanks, Gene. Great to be here. Really appreciate the invite to GeneCon, the well-named GeneCon.

Cisco is traditionally a hardware company. You all might remember back in the early 2000s when it was actually the most valuable company in the world at one point, for a very small amount of time, and it built its business selling internet hardware. But the company has been going on a transformation to a software company, and so I'm part of that transformation.

I'm director of engineering in one of the software groups at Cisco. We're building cloud security software, which is basically security software to protect the large enterprise. We build in the cloud, 100% cloud software. We're part of that transformation, changing the revenue from a one-time sales model to a subscription-based model, and we're very excited about all the opportunities in AI to power our portfolio.

Across Cisco, like you said, it is a very large company. We're in so many different product categories. We sell in networking, security, of course, but also applications monitoring. I don't know if you saw the recent acquisition: Splunk is going to be part of our portfolio very soon. So there's a huge amount of opportunities across all these products and across all these different go-to-market strategies around AI.

That's where my first interest was: what are we all doing in all these different areas, and how is it different when you look at, say, the networking portfolio or the security portfolio versus a different product portfolio? How are they thinking about these things differently? That's what I sought out to explore and talked about in my presentation. You can go listen to the recording. I found a few patterns or models that different groups are using to think about how they're going to deliver their products and tried to put names on those.

Host Intro (Gene Kim)

Gene Kim: Yeah, in fact, they were the Judge, the Vizier, and the General.

John Rauser: The General, right. That's right.

Gene Kim: It was a fantastic presentation. By the way, so excited that you presented that. Joseph is managing director of AI and ML and emerging technology at EVT. As I mentioned in the presentation that we gave together, I've known you for 10 years, and I've had so much fun working on that, not just the presentation, but on a series of projects that you've been helping teach me what I need to learn to help solve problems I want to solve, and how to pick among the vast ecosystem that is surrounding AI. Can you introduce yourself and tell us about what you presented two months ago?

Joseph Enochs

Joseph Enochs: Absolutely, Gene. I'm very excited, and I resonate with John on the perfect naming of the show here. It was great. Love it, and thank you again for having me.

Joseph Enochs, again, director of emerging technologies, AI and ML at EVT. The talk that we spoke about focused on a couple of things, not to mention the fun that we've had over the last decade and the fun that we had when we had a meetup earlier in the year. We were participating in a charity event, sat down for lunch, and started talking about some of the fun things that we were doing in generative AI, and that's all that Gene and I talked about.

Gene was sharing with me some of the things that he'd like to do with the DevOps talks. He's got well over a thousand DevOps talks, and he wanted to be able to synthesize those into something that was a single page that he could read. Right now, I think he's probably watched as much or as many of those videos as anybody, but taking the time to rewatch all of them takes a lot of time. Being able to have a single document that summarizes those can help Gene take those and get those folks up to the plenary stage as quick as we can.

In that scenario, I think we talked about that. We built that, and then we also talked about some of the things that were happening out with Google and the OpenAIs of the world, the leaked document that they had, and we shared some of the things that we had built along with our own agents. There was Perplexity, which is like a multimillion-dollar thing that was built by Perplexity, great tool. If you haven't seen it, go check it out. But then there were some open source things that we had built that, with some elbow grease and some open source tooling, we could just deploy those.

I think the interesting thing there was the same thing that happened to Perplexity, where these tools were coming out, happened to Gene and me on our quest. We were doing a lot of retrieval augmented generation, dealing with like a 3,000-word context window. Then GPT-3.5 comes out, it's got a 16K window, and guess what? We didn't have to do all those chunking and all the exciting machinations of code that we were working on. We could just dump the whole thing in there, and it would give us a wonderful summary.

Fun times. I'm happy to be here. Maybe just a quick note about myself, a couple of projects that I work on. I was the chief architect for LA County's COVID tracking system, which was a big PRADA project, and I've done a lot in the utility space with helicopters and drones using computer vision to help avoid the next wildfire.

Q&A

Gene Kim: Awesome. Let's start with John. One of the things that really made an impression in your presentation was that you reached out to other engineering groups across Cisco and discovered that there was this vast, insatiable appetite for things AI-related inside of Cisco. But you also said that it is actually this technology leadership community that might be most well suited to help bring these capabilities to market across large, complex organizations. Can you talk about why you believe that's true and maybe substantiate that claim? What about your own experiences make you believe that?

John Rauser: Awesome. Thanks, Gene. Those are a couple good questions. First of all, Cisco looks at AI and the opportunity in AI as two main things. Again, I talked about this in the presentation, but we look at traditional AI, what people are calling traditional AI, but we like the word predictive AI. We're using machine learning, data science to make predictions about what's going to happen next. That's your classical or traditional space in AI, which is now augmented with the generative AI space and using language models to power applications.

At Cisco, we've been building with traditional AI or predictive AI for a long time, and a lot of those people are now looking at this and seeing the opportunities here and exploring those. What I wanted to first understand talking to all those groups is, what are you doing? But then second of all, what are the challenges that you're having with building this stuff? Because I think there's a big sort of intellectual debt between what we can mock up or what we can get going really quickly using an API with OpenAI or something like that and actually putting it into production.

Actually, Gene, there was a really interesting thread that I saw you posted on Twitter just the other day about Amazon's chatbot, the professional services chatbot, and Corey Quinn interacting with that and kind of getting the goods from this chatbot, which is really interesting to see what you can pull out of there.

Gene Kim: Just to be very specific, Corey Quinn, Cloud Economist, he's spoken at Enterprise Technology Leadership Summit. He went to the Amazon Q chatbot offering and basically asked, what is the offering that has the least amount of customers or the least amount of adoption? He asked, what are the offerings that actually increase the price, not decrease the price? I was thinking, holy cow. I wonder if they really game-planned that before they went into production. Is that why you brought that up, John?

John Rauser: That's why I brought that up, because I think there's a big difference between mocking that up and then shipping it and putting it into the hands of customers and letting them play with it and do whatever they want with it. I think that is where all the learning is going to happen over the next few years.

I liken it to when we started working with databases or websites or even just the internet, and we started using all of those things in a very naive way. The way that we used databases early on: clearly you need to protect your database from injections, but when we first started using them, we didn't think about that stuff. The way that the internet was designed in its original form, it wasn't secure at all. So there's all kinds of things I think we're going to learn about running AI in production, securing AI in production, that we might have a sort of naive view of today, if you know what I mean. There's going to be learnings there.

Gene Kim: One of the things that I would just want to confirm, because I was inferring it, is that what was remarkable in your talk is that you were surveying the landscape not just out of curiosity, but it seemed like you were really trying to position you and your teams to say, how can we best help not just your group, but groups across Cisco bring these capabilities to market? Did I interpret that correctly?

John Rauser: Yeah, I think so, Gene. I definitely look at what is the platform opportunity in AI and enterprise AI as well. How are we going to enable development teams to move fast by providing them with platforms and services like a framework to securing your AI, or models, or frameworks to running your AI in production? Definitely that's something that I've been interested in figuring out: what are those platform plays, and how can we get people moving faster in enterprise AI, where we have so many more requirements and so much more going on than just the hackathon stuff that a lot of people are doing today?

Gene Kim: Awesome. Joseph, you're in a position where you get to see a lot of initiatives that organizations are working on. To what degree do John's observations mirror your own observations?

Joseph Enochs: A hundred percent, I would say. There was a question earlier in the networking session about how safe do we need to be, what does safe mean, and how safe do we need to be? I really do think it has to be with the use case. If you have external-facing customers that are looking at this use case, in that scenario you should really be very, very, very safe. Data leakage or questions that you would have thought people wouldn't ask, they're going to ask. Those responses need to really be curated and logged and things of that nature.

If you're putting something out to your customer community, you've got to be very safe. If you're putting something internally, again, once you've got access to the large language model, depending on who you've given access to, they can ask anything they want. Dependent on how you have done your retrieval augmented generation in the background with role-based access control, they may be able to access any data that they want.

It's very important that we really know what these chatbots are capable of, that we have a curated way of making sure that these question and answers are not giving back things that are toxic or PII or things of that nature. The guardrails all the way along the way have to really be well thought out. For us, whether you're on the auditing or policy side, step back to that use case and outcome side. Depending on who your end customer is and who you're going to give access to these things, you're really going to have to take a close look at how you safeguard the question and answering these bots. Again, take it on a use-case-by-use-case basis, because you don't want to expose things to risk. There are plenty of really interesting examples of that in the news lately.

Gene Kim: One of the themes in Joseph's presentation was the vast rate of change in the space right now. John, you also mentioned this as well, viscerally. It seems like organizations are forced to invent things which six months later are now being offered by vendors, which might result in whatever was being built being thrown away, abandoned sometimes. Joseph, you had some sense of equanimity about this, sort of that's the way the world is when life moves as fast. Can you talk a little bit more about what strategies one should have in mind as engineering groups are trying to get capabilities to market on whether to build, buy, wait? You first, and then John, I would love your reactions as well.

Joseph Enochs: That has happened so many times over this past year. We've all had an outcome that we wanted to reach, and we really wanted to use the technology at our fingertips to experiment with those things. Then the technology came right into our fingertips so that we could use it.

When we're doing product management, I would say the same sort of thing for any use case that we're building internally as we would externally. You have that same strength, weakness, opportunity, threat, the traditional SWOT analysis for a product that you're bringing out. You have to really embrace that things are moving fast. There are some components that are going to be overcome by events, but for us, it's really attaining that outcome. We want to attain the outcome. We want to attain that objective.

As we're writing these pieces and parts of our code, we really want to put those pieces and parts that we know are going to be critical for us and really focus on those, and the parts that we anticipate through research and through talking to the community, talking to our cloud providers and things of that nature, what their roadmaps are going to be, really try to get a sense of what things are going to be overcome by events. Either stub those things out or not necessarily invest so much in them because they know that they're coming. Again, focus on that signal that you can, on the outcome that you need internally, but understand that some of these things are going to be overcome by events.

When we were working on these things, we thought retrieval augmented generation for this particular use case was going to be great. Lo and behold, now we've got 128,000-token windows on GPT-4 Turbo and 200,000 on the Anthropic group. How could we have anticipated that? You've got to embrace the change. You've got to continue on with your efforts and focus on those outcomes, understand what's specific to you, but understand that some of these technologies are going to be overcome by events, and don't invest a tremendous amount in things that are going to be on the roadmaps for the large cloud providers and for the OpenAIs of the world.

Gene Kim: John, I have to imagine you have a slightly different perspective. It's not just one group going, oh, we have to do a platform shift. When you aggregate that across scores of groups, that's real money, real effort. What's your take on this?

John Rauser: There are two things I want to say here. The first is that there's definitely a searching right now, a searching to figure out what the use cases are for all this stuff. There's a recent O'Reilly report that just got published a couple days ago where they say the thing that's holding companies back the most is just that, finding a good use case. There are a lot of people trying to explore that.

I think enabling teams to try to find use cases is important. Even if it involves using a technology that then becomes no longer relevant or built into the system, built into the platform, that's fine. The second thing I want to say is that I think there's something really invigorating about working on a frontier like this. I think it's important for technology organizations, engineering organizations, to be exploring a frontier. This is a great one, and it happens to have a lot of interesting applications. The power and the energy that comes with exploring and allowing people to do new things is great. It's important, and it gives people a lot of motivation just in their day jobs that maybe have nothing to do with AI at all.

We just ran a hackathon with all 1,000 engineers in our organization. There's about 1,000 in our specific organization. I have to tell you, it was incredible to see how people got excited about this stuff and the things that they came up with, finding those use cases, finding those different ways that we can apply AI. There's a lot of fun stuff going on there, but there's a few gems as well that we got to pull out of that. I think it's very important to be engaging that kind of stuff as a group.

Gene Kim: John, my question to you is, listening to your talk, I felt like you were saying it's a great time to be in the game for those technology leaders who want to jump in and help create platform-like capabilities. Jump in, the water's great. Is that what you're saying? Even if so, and even if not, what advice would you give to this community given your experiences exploring and searching for a niche for what engineering leaders can and should be doing to create value for everyone?

John Rauser: I do think, thank you for capturing that, I would say jump in, the water's warm. That's a good metaphor. But definitely with an open mind and not that sort of naive approach that, hey, we can just ship this integration that's really just three lines of Python code. There's so much that has to go on around that: the guardrails, the security, the DevOps. That stuff is going to be critical.

Exploring that as well with your organization, building that competency, sharpening that saw, is going to be really critical for everybody. So yeah, get in there, Gene. That's my advice. Everyone, run a hackathon. AI hackathons are hard. They're not the same as a regular hackathon, but there's so much fun and so many ideas get thrown around. It's really great.

Gene Kim: Joseph, any advice you would give to this community? Final points?

Joseph Enochs: I've always said this: treat your data as treasure. It's the most valuable asset that you have. Definitely treat your data as treasure. All these things are going to be swirling around you with AI, but you're the people who have your data, and that's your unique value proposition, is your data.

My second thing is personal devices. These personal devices are going to change. Like I said in my talk, we're not talking about just cell phones. We're talking about the pens, the pendants, the glasses. These personal devices are going to be coming to an organization near you. Be prepared for them.

Gene Kim: Fantastic. Thank you so much for your great talks. Looking ahead for the Enterprise Technology Leadership Summit, we're going to have experience reports like we've always had around technology transformations that we talked about, the second one around career leadership careers, and then the third one will be generative AI, focusing on what we need: stories from the frontier to show us what is possible.

John, Joseph, thank you so much for sharing with this community what is possible and how we can replicate those outcomes.

John Rauser: Thank you.

Joseph Enochs: Thank you, Gene.

Gene Kim: John, Joseph, I have to tell you, from my perspective, it's been so fun actually doing this GeneCon format, very stimulating and learning a ton. With that, I saw Jeff come up on stage, and I believe Steve Spear will be coming up on stage momentarily if I read the agenda correctly.

Jeff Gallimore: You actually had a few more minutes if you wanted to go with Joseph and John.

Gene Kim: Oh, we actually had 30 minutes for this period? Get them back on. I actually had a whole question. Sorry. This helps to have a copilot.

John, Joseph. Okay, we're going to pretend like the conclusion, we're just going to move to the end and we're not going to repeat the conclusion.

John, it was so interesting to me the fact that you actually came from a security background. You're supporting a security product, and now you're in this position of bringing those competencies and expertise to securing these AI capabilities. Can you talk about what was the biggest surprise? To what extent were people that you were working with thinking through security, to what extent were you able to help educate them on what the surface area was and what things you could actually help them with?

John Rauser: Great question. I'm glad I got to come back too, Gene, because I didn't get to get a selfie with you guys. I just have to get that. Okay, there we go.

We just finished running a really interesting virtual event, an AI security summit. Some of the things that we talked about there were around the threats that are going to materialize in AI. I read a great paper just the other day, and you should definitely go look this one up. Some researchers figured out a way to exfiltrate private data by just repeating the same word over and over again: book, book, book, book, book. Eventually it diverges and spits out private data. We don't know why, and we don't know how, but it does that. Now they put protections into place since then, but go check out this paper.

One of the great ways of understanding something, understanding a subject matter in technology, is looking at the security aspects of it. You break down the threat model for a new technology. You can learn a lot about how that technology works, not just from a security perspective, but from a usability perspective, from a DevOps perspective, and so on. That's why the security approach to understanding AI, I think, is really valuable and generates a lot of new knowledge about how AI actually works. Definitely go check that stuff out.

There are other frameworks that are coming out. There's the NVIDIA AI security framework. There's the OWASP Top 10. These are all great ways of understanding things, and as these new threats materialize, we're going to have to meet them. We're going to have to figure out how to meet them, and that's going to be the job of security professionals, security tooling, and so on. Lots more to come there, Gene.

Gene Kim: It seems to motivate this position, right, that we don't want every development group to have to be a functional expert, and not just security, but also security as it pertains to LLMs, which represent just another vast piece of surface area that needs to be protected. It seems like this is another place where you just want platforms to hide that complexity and someone else to be that first line of defense. Is that part of your thinking and your thesis?

John Rauser: Absolutely. Yeah, it is, if we can. But there's certain things that we have to engage with the development community on, the DevOps community, and we have to push those up. That's what the OWASP Top 10 is all about: here's, as a developer, what you have to think about when you're building with these things. That is sort of shifting left some aspects of it. Then there are other aspects that we can shift down into a platform. We have to take all those opportunities. We have to look at it holistically, I would say.

Gene Kim: Awesome. Joseph, in your client work, I have to imagine this resonates with you.

Joseph Enochs: Completely. Depending on where your end user is and regulation and policy, we deal a lot with regulated industry and utilities, and we deal a lot with air-gapped types of environments. Whether you have customer-facing data, internally facing data, or secure regulated data, not everybody can do round trips up to the cloud. Not everybody can have their data sitting up on the cloud. There are a lot of industries that have to have them in their facilities, and they can't send these data out.

We're talking about data exfiltration. John mentioned that paper on something as simple as repeating words over. But there have been other papers released that went through Notion and built LLMs to exploit these other tools, and 86% of these applications were vulnerable. One of the things that they found out was, on the other 14%, which is the ones that I'm interested in, how did they stop these people from exfiltrating data? They had very specific and precise processes that they did for the LLMs that they chose, the way they did role-based access, the way their question and answering were.

For us, not just looking at the frontier closed models, but some of these open source models that we can run in these isolated environments and truly put guardrails around how they respond. I'm really excited because I think Swyx is going to be coming up talking here in a little bit, and he's the AI engineering, and he talks a lot about how these guardrails and things function. I can't resonate enough with John, and thank you for sharing NVIDIA and OWASP. I'm excited to actually take a close look at those. It definitely resonates with us in large enterprise, especially in regulated industries.

Gene Kim: Here's my last question. There's this famous graph that Joseph presented. It's like how many papers are being published, how many announcements are being made, all the work. I put a link to the tweet of the exploit. Catherine Lee is one of the researchers that found this. Apparently they were sitting on this for 90 days and notified OpenAI of this. They waited 90 days, and when they didn't fix it, it's published, kind of common in the security community.

How does one reasonably keep up with this rate of advancement? I'll be honest, I have not read more than a handful of papers on OpenAI. Mostly I'm relying on videos. Can you give advice to this community in terms of what's the best way to gain some competency of information that's relevant, maybe in two tracks: for the people who want to benefit from it, and maybe the second track of how do you create this new competency and potentially lead this effort to create services for other engineering groups? John, let's start with you.

John Rauser: If you haven't already, you've got to get a community of practice going in your enterprise. We have a channel with about 2,000 people on it, maybe more now, and we're just sharing stuff all the time: papers, talks, podcasts. That is actually my primary jumping-off point to find things. You've got to get that going in enterprise. I'm sure there's a ton of people that would love to be a part of that. They want to be a part of that. Maybe you have it already; breathe some life into that, invigorate that and get that going. That would be my first thing.

Secondly, it's a lot of searching and looking on X, formerly known as Twitter, that kind of thing. There's too much to keep up with, though. Finding good newsletters, finding good weekly summaries, things like that. I can share a couple in Slack that I follow, and that will get you started.

Gene Kim: Fantastic. John, Joseph, maybe if you'd be willing to help volunteer, let's create one here as well in the ETLS community, and would love your help in shepherding that. Joseph, your advice.

Joseph Enochs: There are a couple YouTubers I love: AI Explained, David Shapiro, if you haven't checked him out, he does really great stuff. Papers with Code, if you're wanting to be more of a practitioner, take a look at Papers with Code, very good resource. You can register for Hugging Face's daily papers. They release their top papers, so you can sign up for some of these newsletters.

If you really want to get extra credit, you can go open source and get GPT Researcher, put a little elbow grease in it like our teams did to get it whittled down to what you want to do. Then you can ask it and it will go out and search for you. You can even put a cron job to have it run daily or nightly and ask for specific things, and you get those in a nice, wonderful PDF format that comes in for you. If you want to get extra credit, you can build something like that. But definitely love the communities of practice for sure.

John Rauser: That's the right answer right there. Get AI to do it for you.

Gene Kim: Awesome. This is terrific. Personally speaking, I've had so much fun being able to solve all these problems that are now within reach because of generative AI. I just want to thank John for really stating explicitly the same community that helped catapult DevOps capabilities into the enterprise is likely the same community of leaders to do something similar for generative AI capabilities. John, I'm so glad Jeff, thank you for reminding me that we had another 10 minutes. It was so well spent. John, thank you. Joseph, same to you.

John Rauser: Thank you, Gene.

Gene Kim: To everyone, look for the new generative AI channel on this Slack instance.

Joseph Enochs: Awesome. Thank you so much.