Hope Lynch, Logan Donley, and Stephen Magill

Hope Lynch: Welcome to today's VendorDome, where we have Stephen Magill and Logan Donley, and our topic is "Can you accelerate delivery without compromising security?" Everybody wants to go faster, but what does going faster well look like? First, let's get our introductions done. Logan is here with me from CloudBees. Logan, tell us a little bit about yourself.

Logan Donley: I started here at CloudBees as a solution architect, and I have to apologize, I'm a bit sick right now, so my voice is probably not super strong, but hopefully you can hear me. Now I'm a technical evangelist and work with Hope, and we work on helping our customers understand what we can do with modern DevOps practices.

Hope Lynch: Thank you. And Stephen is here with us from Sonatype.

Stephen Magill: I've been working in software security ever since doing my PhD in that area 15-plus years ago. My initial work was in static analysis and reasoning about programs. Since then, I've gotten more and more interested in development workflows and the processes that you can put into place at large organizations to actually apply tools in the right way and have a big impact in terms of software security. That's been my focus here at Sonatype. I've been lead for the last three State of the Software Supply Chain Reports, which is an annual report that Sonatype puts out, and so we learn a lot each year with that reporting about how the community is managing open source, how enterprises are managing their use of open source. I'm really excited today to talk about this topic and what people out there can do to really substantially improve their security.

Hope Lynch: I'm your moderator, Hope Lynch. I also work at CloudBees, senior director, platform. My background: I've been a product manager for IoT platforms, managed systems engineering teams when they were called systems engineering teams, managed platform teams, done web development, done corporate strategy, and one of my passionate things that I've done and still continue to sneak in where I can is anything related to transformation, digital agile business transformation. Throw your questions into SpeakerTrack. We want this to be a back-and-forth conversation if we can.

Progressive Delivery

Hope Lynch: This rapid innovation, the way we're discussing it today, is tied to progressive delivery. Logan, what is a good definition or description of progressive delivery?

Logan Donley: Progressive delivery is moving away from having a large release where we roll out one big bundle with, let's say, 10 features out to our production environment. Instead, we look at it from a feature perspective, where I can roll out each feature individually to a small cohort of users. We can start with beta users, or we could do a random 10%. If Stephen, Hope, and I have all written a new feature each, and my code is actually bad and once it's out in production it's going to cause an outage, if we rolled it out completely then 100% of our users can't access the application. If instead we used a progressive approach where we started by rolling out to 10% of our users, it sucks for the 10% who can't access it, but the other 90% can keep using it and it's not a total failure. But the problem is now Stephen's and Hope's features are beholden to mine until I figure out why mine is busted.

Logan Donley: If we do something with feature flagging, each of our features can be wrapped behind a feature flag. This is effectively an if statement if you're not familiar with feature flags. You wrap your new features behind an if statement, and the conditional that determines that is pushed outside of the code, so it can be updated on the fly. You can determine based on who the user is: if they're a beta user, they'll see this; if it's a random percentage, they'll see it. The beautiful thing is we can just shut it off. If my thing breaks, we roll it out to some percentage of users, we see some errors, we turn it off, and instantly Stephen's and Hope's features can be used by customers. I can go figure out why my thing is wrong, and nobody's angry at me for breaking anything.

Hope Lynch: Progressive delivery, at least in my last organization working on an IoT platform, was a fantastic thing to be able to utilize because it was such a critical service for our end users. If something did go wrong, we could isolate it, maybe roll it back just for a specific set if they had some configuration that was just in conflict. Progressive delivery perhaps is one of those things that does speed up innovation. If you've known me long enough, we're going to talk about Agile at some point. That iterative process that you go through, you learn, figure out what changes to make, adjust, and implement quickly. Progressive delivery for Agile is such a good match and helps innovation speed up.

Stephen Magill: What you were talking about, Logan, was the ability to have more reliable releases, quicker rollbacks, and preserve functionality in the case where there's breakage. Do you also see progressive delivery improving delivery time, delivery frequency, shortened delivery times? Even when things work, does it help things flow faster?

Logan Donley: It depends on the organization. That is definitely the idea: yes, it should speed that up. There's a cultural component. If you're going from monthly releases to being able to push anything, your team needs to build up confidence that yes, we can push things faster without as much testing. You always want to be cognizant of the fact that you don't want your users to be guinea pigs. I'm sure we've all experienced an app where every time you log in everything has shifted around. You don't want that experience, so make sure things are properly tested beforehand. Once you get the organization behind that way of working, if Stephen writes a new feature with code I want to reuse but for whatever reason it's blocked and we can't push it to production yet, we have an opportunity to put it behind a feature flag, not enable it in production, and keep it off of the long-lived feature branch. We can speed up the standard deployment cycle. There are fewer delays waiting on large merges and blocking behind other people's changes.

Logan Donley: It is very dependent on your organization. At Sonatype as well as CloudBees, a lot of our stuff is heavily self-managed focused, and that doesn't really work where I can't release a new feature and send it to 5% of users and ask them to try it out. I'm going to lose that 5% of users pretty quick. It doesn't work in that world quite as much, but for the SaaS world, our feature management solution itself is a SaaS, and so we heavily use that there because we have that flexibility.

Hope Lynch: We are seeing more of a shift to SaaS. It depends on the industry segment, but there's more comfort using cloud services. If it's well managed, you get access to features in a timely way and it improves things without manual deploys. Hopefully you're not in the scenario where everything in the UI changes on you and flips back the next day and you have whiplash. If someone is new to this and says, "We're doing continuous delivery; why do we need progressive delivery?" how do progressive delivery and continuous delivery work together, and how are they different for developers?

Logan Donley: Progressive delivery is, I've heard it described as continuous delivery plus-plus. Progressive delivery is really about introducing feature flags, and that lives in your source code. You're introducing that at the start of the cycle when you're writing your code, then after the thing is in production you can toggle the flags on and off. You still need the same delivery mechanism to get your code deployed into the production environment. You still need the whole CD process in place. We're adding on both on the authoring-of-code side, as well as at the end toggling on those flags.

Stephen Magill: We've mentioned advantages in deployment frequency and development speed. That ends up being really important from a security perspective as well. We talk a lot about how quickly you can remediate issues and respond to incidents, and a lot of that has to do with the capacity you have to push changes quickly. Whenever you have to remediate something and respond quickly to an incident, if you have that muscle built up where on a regular basis you're pushing things quickly, it becomes not an exception to what you normally do, but just part of standard day-to-day.

Hope Lynch: Chris asks, "How do you deal with this when you have legacy code that maybe doesn't have nicely decoupled components?"

Logan Donley: Legacy code depends heavily on the stack. You don't need to go all in on feature flagging all at once. You can intersperse it where possible. There are components where it's never going to make sense. Think about a database migration. You can't really just toggle that on and off. That's true of larger applications where it's too complex to break it down such that any new feature can be toggled on and off. Start where it makes sense. One challenge when people adopt these practices is trying to take on too much at once. It's a huge ask when you've got hundreds of thousands of lines of code to figure out how to deal with that all in one fell swoop.

Hope Lynch: Quinn Daly asks: when is the right time to remove a feature flag and all the associated old code from your code base, and how do you ensure the old code is removed along with the flag?

Logan Donley: At CloudBees, we try to make that as automated as possible by detecting when that stale point is. Once a flag has been in action, whether turned on or off for everybody consistently for some amount of time, and there hasn't been any variation, then it's usually safe to remove it. If everybody has seen this new feature as true for a week, then we can probably safely start to remove that. Lifecycle management tools come into play to figure out: I've seen that it's been in action, it's stale, here are all the references to it, go ahead and remove those. It's important that you don't clutter your code with thousands of flags that are no longer in use.

Hope Lynch: Gene asks, "It's amazing how feature flags have taken off in the industry. What else can we learn from dynamic runtime environments where we enable the runtime to be truly dynamic? What else could we enable versus things we view as static right now?"

Stephen Magill: One thing that comes to mind is managed services. It's becoming more common to not manage your own database, but rather use a managed database service, a managed message queue service, and things like that. That takes a component and says, I'm going to shift from a static view of this, knowing exactly what version it is and manually updating it myself, to it becoming this dynamic component that offers this API and this service. We have a contract in terms of what interaction with that service means, and that's all that's required. The more you can shift away from specific code into contracts of interaction between components, that's the key thing that keeps the level of abstraction correct for the problem you're trying to solve.

Security, Speed, and Dependency Management

Hope Lynch: With all of these things happening so quickly, there's so much more speed coming to more teams. How secure can we be with progressive delivery and automation? Is it inherently less secure for any reason?

Logan Donley: That's always been my biggest question. My assumption would always be if we're moving faster, we're giving up something for that. Increased velocity does seem inherently risky. But we're using the same CD processes, so maybe it's not. I'd like to hear what Stephen has to say.

Stephen Magill: It's a common view that there's a trade-off between efficiency, speed, and security. What we found in the software supply chain report, when we did a survey of development practices and outcomes at large enterprises, was four clusters. A couple correspond to that view: people moving really fast and not caring about security, and people who achieve security by slowing down the process and introducing checks along the way. That is an effective way to get security, but it slows things down. But there was a cluster managing to achieve both high security and high development velocity. In that group we saw a lot of leveraging of tooling, adoption of CI/CD practices, agile practices, and leveraging of policy, centralization, and uniformity of the build pipeline. If you have a standard for what a pipeline looks like in your organization, and how you're going to leverage tools to do those checks instead of manual processes, then you can raise the bar in terms of security while still having a very agile and efficient development process.

Hope Lynch: What are your best practices for day-to-day dependency management?

Stephen Magill: My biggest best practice for day-to-day dependency management is just do it. There are a lot of places where that's not something that's thought about. It's not part of maintenance or planned for every sprint: let's revisit dependencies, let's see if there are updates to do there. But it is super important in terms of decreasing security risk. Most disclosures of vulnerabilities happen when someone in the white-hat security research community discovers something, works in private with the project that has that vulnerability to get it remediated, gets a new version pushed out, and then after the community has had time to adopt that updated version, then it's disclosed. If you're staying up to date and adopting those new versions, by the time that vulnerability is disclosed, you're already covered. That day-to-day practice is really important.

Hope Lynch: How can organizations know what they actually have? If they have a large enterprise and need to take on remediation, how can they get their hands around that?

Stephen Magill: It can be hard at large scale when you have thousands of repositories to even know what you have. The first step is tooling to generate software bills of materials for everything that you have, so SBOMs. That gives you the start of that inventory. There's additional information you'd like on top of that. If you have 2,000 repositories, not all of those are active applications. Not all became production applications. A lot are small projects or proofs of concept. Knowing which is which, among the applications knowing which ones are in production, which are internet-facing outside the firewall, all starts to give you a sense of where your security risk lies. When you do have an incident, something like Log4j, which did not follow the typical responsible disclosure timeframe because the exploit was posted on GitHub to the surprise of the world, then you have the information you need to remediate and prioritize remediation. You know these are the apps most at risk based on network connectivity, the role they play, touching customer data, and things like that. Then you can focus on actual remediation rather than getting stuck figuring out what you even have.

Hope Lynch: Is it typically developers keeping track of these things, or do most companies have shared services or a security team actively monitoring and assigning tickets?

Stephen Magill: This has to live at a level above individual development teams. Development teams know a lot about their code base and their application and which portions are security critical, so they should have a role in the inventory process. But it's really an organization-wide concern, and you want some amount of centralization of that inventory.

Hope Lynch: Top-down mandates like "be secure" are not effective if you don't have processes in place and have not planned it into the organization. How do you get leadership to shift and support the tooling, implementation, and conversations required?

Stephen Magill: A top-down mandate on its own does not do anything. If you say we have to be secure and prioritize security, but you don't give developers time each sprint to address security issues, don't have a process to prioritize security issues, don't do architectural reviews, and don't give the security team a voice in application design where relevant, then it's not going to do anything. Those structural aspects of supporting security do have to come from leadership. You need to say: yes, we want to focus on security, and these are the processes we're going to put in place, this is the support we're going to give you to make sure we accomplish that.

Hope Lynch: And support sometimes means the dollars.

Stephen Magill: Often the dollars. It's not just money for tooling either. It's the people. There are organizations with a 1-to-100 ratio of security professionals to developers. It is hard to get good security people.

Hope Lynch: Security is not one size fits all. What different types of solutions do organizations and security professionals need to look at depending on the response types needed?

Stephen Magill: The needs are very different at a large enterprise versus a smaller team. There are good lightweight accessible tools for small and medium-sized teams, and more open source security tooling than there used to be. At Sonatype we have Sonatype Lift, a SaaS product that integrates with GitHub. If you're on GitHub and starting to build your software, or even a medium or large team, you can use a managed service without installing anything or worrying about compute costs, and get insight into the security of your code. Where it shifts is when you have a massive inventory of applications and code, even non-application code, and management becomes the primary bottleneck. It is not monitoring this code base for security issues; it's figuring out where all the code is, making sure it's all covered, making sure processes are uniform. With tens or hundreds of development teams, it takes a lot more effort to get uniformity and full coverage. Enterprise tooling gives you that enterprise-level visibility: these are all my applications, this is where the risk is.

Stephen Magill: At that level, setting up something at the gate comes into play. Large enterprises often set up a repository server, something like Nexus Repository, that's proxying access to outside open source. That's a great place to inject additional policy: check things as they're pulled into development pipelines, make sure they adhere to our policy, catch zero-days and security issues as they happen. At that scale, it's about having multiple points of control so you can get full coverage, because it's hard to get that with just one point of integration.

Hope Lynch: Do you feel the state of security in organizations is improving a lot, or is it at the same level it's been?

Stephen Magill: There has been substantial improvement, but there is still a long way to go. In last year's Software Supply Chain Report, we found that by and large people were doing a good job managing the dependencies as they were updating them, choosing good versions and secure versions. But they were only actively managing 25% of their dependencies. The other 75% never saw an update over the year timeframe we looked at. There is a lot of room for improvement in practices. At the same time, we are seeing the security process work, the timeframes for remediation come down. In the open source community, the average time it takes a package to update its dependencies when new versions come out has dropped, and that helps with the transitive dependency problem. When Log4j happened, within two days more than 50% of the downloads from Maven Central were the patched version. It then leveled off, so you have a cohort that is not getting the message. But that fast-mover cohort was probably smaller 10 years ago. There has been improvement, but there is still room for more.

Hope Lynch: Gene asks about stories of engineers successfully rationalizing dependencies. What should inform what components should and shouldn't be used?

Stephen Magill: One thing we've seen is cases where there are 15 different XML parsing libraries in use at an organization. I didn't even know there were 15 XML parsing libraries. You can get into situations where pretty much everything that can be used is being used. It's easy to get there because everyone has their toolkit, new developers join and bring their favorite packages, and dependency bloat is real. Having a process saying, "This is our tech stack. We'll make exceptions when reasonable, but we're going to try to stick to this collection of packages," helps a lot because then it's less to monitor for your security team. The more familiar you get with the packages you're using, the more on top of security and update you can be.

Logan Donley: It is always a balancing act. We want to stay productive, and being productive might mean bringing more dependencies that you're familiar with, but that can come with security risk because you're adding additional dependencies.

Hope Lynch: Billy Hudson asks, "How and where are people recording metrics for resolving dependency issues from discovery to resolution?"

Stephen Magill: We call that timeframe MTTU, mean time to update, when it is just an update and not necessarily security relevant. If there's a CVE against the version you're moving from, we call that a remediation and look at mean time to remediate. We measure that as the time between when a new version of a dependency becomes available and when a version of an application that uses that dependency gets pushed that has updated that dependency: the timeframe between availability and use in your application. Across all the dependencies you incorporate, that gives a sense of how quickly you're reacting. If you have a regular process for reviewing dependencies, that will naturally lower that number. It's not just a fun metric to track. It's useful. We have found that MTTU is correlated with security. Packages that are better at managing dependencies have, on average, less security risk. It is also correlated with time to remediate. If you keep update time low, then your remediation time will be low, because if you're up to date, a lot of times you're already patched when a vulnerability gets disclosed.

Large-Scale Incidents, SBOMs, and Tooling

Hope Lynch: You mentioned Log4j. For large-scale security incidents that make it to the news and everyone is scrambling, were there things certain organizations did really well, or places others stumbled?

Stephen Magill: That emphasized the importance of having inventory of applications, knowing what you have. Organizations that could very quickly run a search or query and find out whether they were using Log4j, how pervasively, and what versions, had a significant leg up in the remediation process. If it takes several days to even figure out whether you're using it, that obviously slows you down.

Logan Donley: A lot of our customers had that same issue. It is a matter of having a good process in place where, using the right tools, you are looking at that SBOM and enforcing that whenever you're releasing a new version of your software, you're passing all those gates. You're not accidentally skipping when there is a big vulnerability. The companies that already had that in place were able to get past this and upgrade faster because they had that bill of materials to understand where they were vulnerable. For software they might not have had that bill of materials for, when they next run and try to release new software, they can figure out that they do have this vulnerability and resolve it.

Stephen Magill: While CloudBees isn't doing direct security scanning itself, it plays a critical role in security because the CI and CD process is a critical point to monitor from a security standpoint. Tooling that generates the software bill of materials will probably run as part of CI. Policy is also typically enforced as part of CI or maybe CD, where you look at that SBOM and say if any of these things have a medium or above vulnerability, or high and critical, whatever threshold, we're not going to allow a release that violates this policy. That gets orchestrated as part of CI, and once it's in CI, you know it's happening. It's not something you have to remember; it's automated.

Hope Lynch: Chris asks, "Are there tools that can help you build a software bill of materials?"

Stephen Magill: Sonatype's lifecycle products will generate a software bill of materials, and you can export it in common formats. There are also open source tools that will build an SBOM. They tend to be language specific: a tool for Java, a tool for Go, and so forth. They are out there. Then it's a matter of working it into your process, making it part of CI and making sure it goes somewhere. It doesn't help to generate the SBOM and throw it away. That's fine if all you want is a release gate saying we can't release if we have vulnerabilities. But if you want to persist that information so you can quickly respond to zero-days like Log4j, you need to think about where you're storing them.

Hope Lynch: Log4j was about six-ish months ago. Are there other things that have happened since then, maybe not as newsworthy but still significant?

Stephen Magill: There was SpringShell, which did get press, not as much as Log4j, maybe because it's not quite as pervasive, although Spring is very widely used. We didn't see quite the same rapid adoption or full adoption of the patched versions. Sonatype has live dashboards for Log4j and SpringShell, and it's interesting to compare the two. The adoption curve for the updated version of Spring Boot levels off at a much lower level than Log4j. For Log4j, something like 30% to 35% of downloads from Maven Central are still vulnerable. It would be great if that were lower, but with Spring Boot, it's much larger than that, well over 50% of downloads still vulnerable. Dependency management is not where it should be.

Hope Lynch: What do you think contributes to that?

Stephen Magill: It could be a perception about the severity of the vulnerabilities. When people don't update, a common rationale is, "The way I'm deploying it or the way my application is configured, we're not vulnerable to this." Sometimes that's true. You do have to be careful of spending more time justifying not upgrading than it would take to just upgrade. Often it is best to go ahead and upgrade. It could also be the profile of the applications: are they in production or more internal applications? Although Spring is widely used for externally visible applications, so I would expect it to be more critical.

Free-Form Q&A

Hope Lynch: Logan, why do you think there are organizations that have not yet fully explored or implemented progressive delivery?

Logan Donley: It feels scary to give people flexibility to push code into production that is not as heavily vetted. You would hope you have as many tests in place, but when you're focusing on increasing speed of innovation, realistically if we're saying let's get to production faster, shortcuts might be taken. There is concern: is this going to be risky? I know I can turn it off, but could there still be problems? This isn't helped by the fact that a lot of people, probably even CloudBees people and maybe myself, have said testing in production with feature flagging. You want to qualify that by saying we're testing whether this function is good, whether it is converting as well as we want, whether people are getting value out of it, not whether this thing breaks or not. That is not the type of testing in production you want to do. There are misconceptions out there. At the end of the day, it is the same thing people have been doing for a long time before feature flags were called feature flags. People still did remote config with if statements calling out to a database. It has been around for a while, but the fact that it is a cool new thing still scares people a bit.

Stephen Magill: There is a certain element of adhering to the level of discipline required to support that process that has a positive impact on code quality. We see this for deployment frequency as well. The data shows that when you deploy more frequently, you are generally more secure and have better business outcomes. Is it all due to just pushing more versions of your application out there? No. Most of it is probably the discipline and processes required to get to that level of engineering excellence. A whole lot goes into that that improves quality across the board. The same thing is probably true for progressive delivery. If you have to engineer your code so you can support a feature being on or off, there is a certain amount of robustness you have to build in, and that robustness is probably beneficial outside that particular feature flag.

Logan Donley: You also need additional tools in place. Just being able to toggle something on and off is cool, but if you're going to be testing things in production in the sense of seeing whether the new feature is successful, you need metrics. You need analytics tools to see whether the events you expect are happening. You also want good server metrics and telemetry. There is not much value in feature flagging if you don't have the right tools in place to benefit from it.

Stephen Magill: Measuring the right things is important too. You may have telemetry in place, but are you collecting the right telemetry? Feature flag experiments are a great opportunity to learn, but you need to be collecting the data.

Hope Lynch: Eric asks, "At what level of depth should dependency be? If supply chain is deep, what is the best practice?" How far down do we go?

Stephen Magill: In some sense, even a very deep vulnerable component exposes you to risk. It doesn't matter whether you directly call it or call some library that calls some library that calls this vulnerable method. That poses a big challenge for remediation because you're inheriting that vulnerability, but you're not responsible for the maintenance of the package that's bringing it in. Ultimately, it's up to the community as a whole to make sure that we're all updating quickly so those transitive vulnerabilities get patched as quickly as possible. This speaks to being careful about what dependencies you choose. If you pull in a dependency that is not great about updating their dependencies, then you'll be exposed to that transitive risk for much longer. When you're thinking about which XML parsing library to use, one thing to consider is how on top of their own dependency maintenance they are.

Hope Lynch: What trends do you see around security and these vulnerabilities?

Stephen Magill: One trend we've seen recently that's kind of new is malicious code injections. It's not all accidental vulnerabilities anymore. There are people working to inject vulnerabilities into popular open source projects. That is a new landscape for the attacker, and it has required new responses on the defensive side. One thing we've done to address that is having an artifact proxy or artifact server, something like Nexus or Artifactory, where you can monitor where you pull in dependencies. We have a product called Firewall that lets you know about dependencies you're pulling in that seem to have had malicious commits lately. You can quarantine and set a policy: I'm not going to allow the use of this new version until the community has vetted these suspicious commits. That's one way to approach that problem. It has opened up a new landscape of things we have to worry about defensively, so it's important to think about what technology you have in place and what technology you can put in place to protect against those.

Hope Lynch: How are SREs impacted by progressive delivery? Is there a benefit for SREs?

Logan Donley: It depends on the definition of SRE, but I will take it to mean actual Google SRE: trying to keep up a massive web-scale application. Progressive delivery can help from the perspective that we're no longer having to deal with rollback in the same way when bad things go into production. Even if you have a good canary deployment where you're rolling out progressively on the server side, not the feature side, if you're releasing multiple features at a time, one bad feature can still block the rest from functioning and you have to roll back the entire thing. Progressive delivery helps everybody involved: developers get code out faster, PMs get information about new features, and SREs don't have to worry as much about the actual rollback itself, just using metrics from their tools to determine when it's appropriate to turn flags off.

Hope Lynch: Feature flagging sounds like it could be a dream come true for SREs if the organization has the processes aligned properly. Any SRE impacts you've seen with security?

Stephen Magill: There is always the difficult question of what responsibilities live where. Especially with DevOps and DevSecOps, the answer more and more is developers own that one. Is some amount of SRE falling on the developer's plate, or are there security issues best monitored by SREs? That negotiation has to happen to figure out whose responsibility is what. There are advantages to specialization: having people in charge of SRE, and even on the ops side experts who can help get things set up, follow best practices, and educate new developers. But there is also value in having one person own things end to end. The answer for each organization is different, but it is a conversation worth having if you haven't had it.

Closing

Hope Lynch: We are pretty much at time. This has been a fantastic conversation. We started around progressive delivery, worked through security, some of those large-scale vulnerabilities that everyone has seen, really doubled down on dependency management, dependency management, dependency management, and just do it. There are four letters there; we're only going to use three. Any parting thoughts?

Logan Donley: There are always a lot of cool new processes and techniques out there. It is important to consider whether this is appropriate for our use case. Progressive delivery is an amazing new way of delivering software if all the key things align. It's worth exploring those things as they come out, but you don't have to go all in. You don't need to make a big decision that this is how we're doing everything from now on. Just experiment and see what works.

Stephen Magill: I would echo that. You have to do what's right for your organization. You also have to consider the people and processes as much as the tooling. It really starts with getting the right culture in place and the right practices.

Hope Lynch: Thank you again. Thank you, audience, for being so engaged. Great comments and great questions, and wonderful VendorDome. Thanks again.