June 2024
From: Brian and Tobias
Subject: The fragility of identity and authentication in AI security
Hi all,
Welcome back to the June edition of the B&T infra newsletter! Hope you all had a relaxing 4th of July weekend.
We’re taking a left turn this month into the world of cyber. Many have analogized the LLM to the computer, and we agree. We’re at the beginning of a new era in computing that requires not just new infrastructure, but new security tooling. As LLMs become more adopted, so too will the security businesses protecting use of LLMs. As a result, Security x AI is a space we have been exploring deeply, thinking about what the right entry point is from an investment standpoint. There is an insane amount of noise and activity, and we are still so early. It’s disorienting and hard to figure out who and what to invest behind.
One area we’ve honed in on as particularly interesting (although definitely still with lots of question marks) is who can see what data from AI applications. It doesn’t matter if the LLMs output answers that sound good if sensitive or irrelevant information ends up in the hands of the wrong people, be them customers or employees. We’ve come to this area after tracking AI security over the last ~2 years. Let’s dive in.
The first chapter: the inputs
After the release of ChatGPT, there was a flood of AI security startups, most of them being what we would describe as input-focused. The prime example of this was startups focused on “data loss prevention” (DLP). Big companies were worried about their employees using ChatGPT to ask questions about confidential or sensitive data and exposing that data to the OpenAI API. These companies were concerned that OpenAI could use sensitive data to train their models and then spit back that information to other users (and potentially competitors) with some clever prompting. OpenAI has ensured that ChatGPT conversations are not used to train their models, but still, companies like Samsung and the major financial institutions banned or severely restricted internal access to ChatGPT.
So, there was a wave of startups offering firewalls to detect PII or other sensitive data in real time and block it from hitting the OpenAI API. These startups are input-focused because they protect against bad user inputs into the model provider’s API. Examples include Arthur, Prompt Security and Liminal, although there are many others competing in what is now a rather red ocean.
The current problem: the outputs
In talking with security and ML leaders, the most concerning security issue around AI is no longer inputs-driven, but outputs-driven. In other words, leaders care more about what the models will say, and how to mitigate that risk and protect against unwanted responses. The nature of LLMs is that they are non-deterministic – the model will output different responses to the same input, making it unpredictable and causing hallucinations. These models are also black boxes, so diagnosing why the model says what it says is a challenge. In a world where this class of model and expected advancements like multi-modality (e.g. audio and visual outputs) become pervasive, making inputs secure is not enough. Security leaders need to evaluate outputs as they arise as well.
We think of two basic categories for these kinds of security risks:
A lot of companies are tackling the first of these problems, primarily through protection against “prompt injection.” This is an area where HiddenLayer has shined. Protect AI has a number of products in the same vein, focused on improving AI model security and the integrity of the AI supply chain. There is also exciting research around AI model explainability and interpretability, which should significantly help identify the data that is causing bad outputs and remediate those issues. Anthropic’s research into “Golden Gate Claude” – a version of Claude biased heavily to mention and promote the Golden Gate bridge – is an interesting early example of this. We see this as a big opportunity, and we’ve met companies like Haize Labs and Womp Labs in recent months tackling this problem.
The overlooked problem: identity and authentication
The other type of problem – outputs that are not inherently bad but end up in the hands of the wrong people – is the one we’ve been spending the most time on over the last few months. We think it’s been underappreciated and under-resourced, and that there may be big opportunities to build businesses with that problem at the center.
Over the last several months, we have been surveying security leaders on their AI-related concerns. The one that has risen to the top has been access and permissions for information coming out of AI systems. Imagine an internal AI application that sits on top of a bunch of different data sets and documents, powered by RAG (retrieval-augmented generation). The application is a querying engine that allows employees to ask questions to an AI system instead of searching for answers amidst troves of documents. This is what Glean has enabled out of the box, although some companies are building solutions like this themselves.
Different people should experience this application differently. Interns should not be allowed access to a lot of information. Managers should get tailored information when asking about direct reports. Different departments should see different work from home policies. The list goes on. There should be an identity aware AI application that takes into account who is on the receiving end of information so that the user always gets the right information. This is both a security problem and an effectiveness/usability problem. Without this layer, AI apps risk surfacing privileged information, as well as unhelpful or incorrect information.
Crowded, but unsolved
There are existing solutions to this problem. Knostic is the notable up-and-comer that’s getting some buzz. We’ve also anecdotally heard from security leaders that Robust Intelligence has a good offering to solve this problem.
Perhaps the more important players, though, are the vector database businesses and companies like Glean – businesses that already own the part of the stack where authentication and access control can happen. The Vector DB businesses all have some access control functionality. For example, Pinecone has RBAC (role based access control), ABAC (attribute based access control), and ReBAC (relationship based access control) capabilities. Glean simply transfers existing permissions to its product, so that when a user queries something, the only documents queried are ones that user has access to already.
Some CISOs find these solutions sufficient, but there are others who have built totally homegrown solutions because they’ve found corner cases for access control that aren’t covered by these solutions. We’ve spoken to two very notable tech companies with high-importance and high-visibility AI projects where Glean and Vector DB access controls were insufficient. One thing we’re trying to define more specifically is why and where these deficiencies occur. What kinds of access control do these tools cover well, and what do they cover badly?
One issue we have heard regarding solutions that simply transfer over existing permissions is “permission drift.” Organizations don’t have a good handle on who has permission to access what. Think about your own organization. There are probably lots of old Google docs and sheets that you shouldn’t have permission to anymore but that you and the rest of your team have forgotten about. One founder told us recently that over time, views of internal documents go down, but the number of people with access to the same documents goes up.
This problem becomes exacerbated when these documents are put behind an LLM application. Now, those documents you would have never searched for are just as easily queried as the document you were editing yesterday. Whatever permission drift exists in the organization gets exposed quickly when an LLM shines a light on it.
The bigger picture, though, is how we give LLM apps the context of the user on the other side. Human beings speak with the context of their audience. We could be asked the exact same question by two different people and have wildly different answers. Our point of view is that LLMs, for both internal and external use cases, and for both B2B and consumer applications, are missing this context. GenAI’s true value and demand for adoption will increase dramatically once this context is better introduced and informs LLM outputs. There are likely many startup opportunities related to this theme, from security solutions for enterprises adopting AI to personalization solutions for consumer AI apps.
Right now, approximations of this context seem limited and crude, and we’re excited about exploring opportunities to solve this problem. If you know anyone you think would have a strong perspective on this problem, we’d love to talk to them.
As always, we appreciate your thoughts and feedback!
Until next time,
B&T