June 2023
From: Brian and Tobias
Subject: Real budget, and real concerns, for AI at work
Over the last two months, Tobias and I have spoken to over 50 AI and ML leaders at scaled enterprises – companies ranging from leading technology businesses like Uber, Square and PayPal to more traditional enterprises like JP Morgan, AT&T, and Genentech. After discovering some real aversion while doing diligence on startups building infra solutions on top of OpenAI, we wanted to understand what, if any, appetite ML teams with real budgets have as it pertains to AI and startups. Here are our findings.
Concern #1: Security
The top concern with using LLMs is definitely data security and privacy.
The most notable model providers (e.g. OpenAI) offer their model through an API. So, if you want to use an OpenAI model for a company-specific workflow or operation, that inference happens on OpenAI’s servers, not yours. This is a non-starter for many larger enterprises. We spoke to multiple people whose companies have terms of service that prohibit them from sharing customer data in any form with a third-party service. It’s one thing to store customer information in an AWS database, but another thing entirely to send that information to another company for processing. Similarly, some companies (mostly in financial services and healthcare) need to retain all data on a private cloud or on-prem environment, also prohibiting the use of a service like OpenAI. We spoke to an AI professor at a leading U.S. university who regularly consults for large enterprises on AI projects. In his opinion, “no one” will use the API services at scale because of data security issues and the increasing ease of re-training open source models. This is an extreme view but reflective of the concerns over data among large companies.≥
Additionally, companies are concerned about potential new attack vectors these LLMs create. OpenAI has already confirmed a data breach, and some companies are worried about their own employees leaking sensitive data or IP to OpenAI through ChatGPT. Both of these concerns have resulted in large enterprises banning ChatGPT, and have added to enterprises’ interest in alternatives to third-party API services for their LLMs.
Concern #2: Cost
After data security, the next concern AI leaders talk about is cost, both for training and inference. We have already written about the cost concerns of incorporating LLMs into existing products. Even excluding training costs, inference costs are often prohibitive for startups and ugly for large companies. In fact, we know a CEO of a publicly traded company who wants to invest in LLM R&D and inference but can’t get the cost implications past his Board. One AI leader we spoke to at a $400M revenue software business said that in order to incorporate LLMs into a consumer-facing product, they would have needed to increase price 15%-20% just to make the economics work. The VP of AI at a multi-billion dollar tech company told us that inference costs needed to be 1/10 of what they are now for his team to use LLMs at scale in their products.
The core problem is that the API services are, on the whole, less expensive than standing up a model on your own infrastructure using an open source model. This is the direction that many companies worried about data security are exploring, but deploying resources against that kind of project and setting up the infrastructure to support it, not to mention bearing the inference costs of running the darn thing, are hard to stomach. AI leaders are faced with a Catch-22 in the choice between model services and setting up an open source model on their own infrastructure: efficiency, ease of use, and cost vs. data security, ownership, and specificity.
Concern #3: Model Selection
This tradeoff is the crux of the issue when companies are thinking about which models to use, and for which use cases. Tech companies using LLMs for internal workflow automation and other use cases that don’t touch sensitive data are often opting for API solutions like OpenAI. In fact, we heard from a few large companies in recent weeks that they were using Azure’s enterprise-grade OpenAI API for these kinds of AI workloads, one example of how the OpenAI-Microsoft alliance is playing out.
However, for many large enterprises (especially in financial services and healthcare), the API solution is a non-starter. Additionally, for building applications that interface with customers and incorporate customer data, companies are wary of third-party solutions and are considering open source or building their own models from scratch. Only the most sophisticated tech companies (FAANG style) are deploying resources to do de novo LLMs internally. Our guess is there are fewer than 100 companies in the U.S. that are in the midst of training their own LLM right now. Even some of the largest companies we spoke to said they were still undergoing an evaluation of training costs before taking the plunge.
So, open source is going to be a big deal. We’ve also written about this belief in prior newsletters, but our conversations have confirmed this belief. We’ve heard amazing feedback on LLaMA and the new FLAN-T5 model from Google with an unbelievable 11 billion parameters. The models themselves seem to be less of a barrier to organizations adopting LLMs, but rather the time and energy of managing and standing up an open source deployment is what scares them. This status quo is not sustainable if we believe LLMs will become ubiquitous – something needs to give between the API services not passing muster and the open source models being a pain to deploy and manage. Both of these problems represent fertile ground for potential startup activity, and we’re looking to invest in startups on both sides of this coin.
Challenge for startups: Trusted Incumbents
When speaking to enterprises, they are much more likely to rely on existing vendors to help them develop AI capabilities than turn to a new hot AI startup. We saw this again and again with Databricks, which many AI leaders specifically noted. The same of course is true for Azure and Microsoft more broadly. JP Morgan recently published a CIO survey on Generative AI. They asked CIOs about vendors that they expect to spend more money with because of Generative AI. The results aligned with our conversations:
The top three here are the big cloud providers: Azure, GCP, and AWS, with Azure out in front by a pretty healthy margin. In fourth is Databricks, which just made news with its splashy acquisition of MosaicML. This comes about a month after Snowflake acquired Neeva. At Snowflake’s Summit at the end of June, the company also unveiled a bunch of Generative AI-related features including a new partnership with NVIDIA to help customers more easily access GPUs and its own LLM to help with document generation and analytics.
One implication of all of this is that the notion that LLMOps will replace MLOps is nonsensical for the enterprise. Delivering value through data, analytics, and AI means interfacing and playing with your data in its native environment, which means integrating into the existing workflows and tools already embedded in the organization. This makes a company like Databricks so well positioned. If all of a company’s MLOps is already flowing through Databricks, then it becomes the trusted and most logical party to layer on and offer generative solutions. The MosaicML acquisition helps further this capability. An analog can be made to Snowflake, which controls so much of the data analytics stack for lots of big companies. As Matt Turck’s tweet below makes clear, this is a massive market and startups could emerge with a unique niche, but overlooking incumbents here because MLOps is now somehow irrelevant is a big mistake.
The implications for startups:
Amidst all of these conversations, we ground our learnings in the world we occupy: seed stage startups. What are the key takeaways we’ll bring to our work with existing portfolio companies and new founders we meet?
- Wrapper solutions don’t have good buyers. Many of the startups we speak to, at both the application and infrastructure layers, are “wrappers” on top of existing models like GPT-3. Unfortunately, these solutions will lack enterprise-scale buyers because of all the challenges outlined above. If a large company already has reservations about hooking into the OpenAI API, it will be all the more concerned about becoming a customer of a startup that uses that very same API under the hood. People talk about these sorts of companies lacking defensibility, which is a concern as well, but an equally important one is how to get adoption with large companies.
- Security and cost saving value propositions are the easiest to sell. For AI solutions, startups need to have a cost-saving or security value proposition. We’ve written about how this is the case in software broadly given the macroeconomic conditions, and this is only heightened for AI startups.
- Building in this market requires deep infra knowledge. As the Databricks and Snowflake activity, along with the JP Morgan survey, show us, incumbents are really well-positioned to win big in this AI movement. Building something new that provides true value will require a deep understanding of which problems incumbents are deficient and strategically disincentivized from solving. We are looking for founders who meet this criteria and teach us a ton about how to solve problems we may or may not be aware of.
- Now is the best time ever to sell AI solutions to the enterprise. Despite those words of caution, now is still the best time (maybe ever) to be starting an AI company. The level of optimism and enthusiasm to adopt AI solutions from leaders of large companies is seemingly unprecedented. They all want to know what use cases and technologies their peers are interested in, and when we share our insights from calls with other AI leaders, they immediately perk up and pay attention. The mere fact that we’ve been able to get dozens (perhaps soon to be 100+) AI leaders to respond and talk to us from a not-so-extensive outbound sequence says it all.
As always, please let us know if you have any feedback or others in your network who should be added to this distribution. We’d love to meet them.
Until next time,
Brian & Tobias