Back to Home

November 2024

From: Brian, Tobias and Gaby

Subject: The future of on-device inference and its implications for consumer AI

In our initial investment memo for Etched, we highlighted a telling shift: for the first time, at least in decades, CEOs were bragging about the names of the chips they were purchasing—on golf courses, no less. That was over 18 months ago and the conversation has only intensified. 

At Apple's most recent product release, a new line of chips played the starring role. It practically felt like an NVIDIA event, with an Apple touch. There’s good reason for this emphasis. Apple is the sleeping giant of AI and hardware will lead to their great awakening. This newsletter looks at why and what the implication may be.

Consumer AI: a disappointment so far

Many have bemoaned the lack of awesome consumer AI applications. Outside of ChatGPT and Perplexity, AI has fallen short. Just two years ago, many were predicting that Her-esque experiences, i.e. personalized AI companions, were just around the corner. Character AI has come closest to this, but isn’t a sticky product and ended in a “reverse acquihire” to Google.

The core reason why we have yet to see an explosion of AI consumer applications is hardware: the phones and laptops we all use every day, that we rely upon to deliver applications to us, do not yet possess the requisite hardware to support real-time, multimodal inference. The best models are too big to run on a device without being costly to the end user. For example, Replika Pro is $19.99/month and still far from compelling. Latency, realism, and functionality are all lacking. It feels more like LARPing in SecondLife than chatting with Samantha, the AI from Her.

And so, even for consumer apps, we’ll repeat a refrain we’ve reverted to over and over again in various editions of this newsletter: hardware is the core bottleneck in AI. For consumer and on-device inference, it comes down to chips in your phones and laptops.

This fact, which seems simple at surface level, actually has far reaching implications for how AI will impact our lives and how the market landscape will develop. First, let’s start with the most important company in all of this: Apple.

‍

Apple: the sleeping AI giant

Over the last two years, Microsoft, Google, Meta, and to a lesser extent AWS have all been discussed as key players in the AI age, either due to owning compute or training their own foundation models. One overlooked incumbent is Apple. Apple is the biggest company in the world and somehow the least discussed mega tech company with respect to AI. And yet, if we believe powerful AI will ultimately make its way onto phones and laptops, Apple will be at the center of the action. 

Apple is fully aware of this and positioning itself for the AI age. Apple recently introduced its new M4 series of chips for laptops, stating in the press release, “the M4 family of chips brings incredible performance for… AI workloads” and claiming that its chips are 4x faster than comparable Intel products. Apple has made similar moves with its iPhone processors, most recently its A18 chipsets, which are specifically designed to accommodate Apple Intelligence. We also know that Apple plans to continue optimizing for LLM inference with more die space for matrix multiples. 

Not only will Apple keep getting better, they are aggressively diversifying their chip strategy from just being vertically integrated to also partnering. Apple and Amazon announced at re:Invent that Apple is planning to use Amazon’s AI accelerators. Apple’s native AI product, Apple Intelligence, is also powered by OpenAI, which was announced in June. Ben Thompson has written about how this dynamic is reminiscent of Apple’s incredibly lucrative partnership with Google – Google pays Apple ~$20B a year to be the preferred search engine on its devices. We could be headed for something similar with OpenAI.

We also believe AI will be a new boon to application development and consumer apps. As hardware improves, developers will build amazing new AI-powered applications for phones and laptops. Apple will use the app store playbooks and take a toll on the AI consumer ecosystem. Apple truly owns this “last mile” for consumer AI – other tech giants can battle it out for the best cloud offering and the best foundation models, but today Apple stands alone dominating consumer hardware and the ultimate AI delivery mechanism. Competition will emerge in this category, and we also believe AR/VR and wearables, combined with AI, will create incredible experiences. Many companies may go after that opportunity, but Apple is undoubtedly out in front, and by far the best consumer hardware company in the world.

‍

Implications for startups

While Apple owning on-device seems daunting, it actually opens doors for startups. A few interesting downstream implications include:

Margins of AI app developers are going to go up: In a world of on-device inference, app developers do not need to pay for compute – consumers will host the compute themselves instead of cloud providers. Sure, we may enter a world where really large workloads are sent to the cloud and the rest is done locally (“local by default, cloud when you must”), but a performant on-device app will require some amount of inference to happen on the device itself. This means the margins of AI applications will go up, as companies’ hosting COGS will decrease. This should incentivize more developers to build AI applications delivered to devices and encourage continuous and self-reinforcing growth of the consumer app ecosystem.

‍Breakthroughs in multimodal AI (e.g. avatars, companions, voice bots, game generation): One popular use case in consumer AI is digital avatar companions. Currently, avatars are brought to life by orchestrating an intricate pipeline of models and API calls, but this form factor is severely challenged for a number of reasons. In addition to hardware, models are also a limiting factor here. Mimicking the mouth during speech, rendering facial expressions, cost, and latency are also big issues with current state of the art technology. As a result, better hardware, along with software and model compression/distillation will need to see massive innovation. We think this is a particularly rich area for startups.

‍On-device security: As inference on device increases, we could face entirely new security risks. Customized, per-user models that live on-device would be a goldmine for malicious attackers, and they may identify creative ways to exploit that information. Although we’re not entirely sure what startups in this space could look like, it’s a net new attack vector we have our eyes on.

‍The phone wars will heat up. We can imagine a whole new world of AI consumer apps if on-device hardware gets good enough for high-speed inference. This could be an app explosion unseen since the dawn of the mobile era, and companies with resources are not going to let Apple jog to an uncontested victory. Google will undoubtedly have something to say in this battle, owning the entire stack, from cloud to foundation models to personal devices. Watch out for OpenAI as well – Sam Altman has been rumored to be working on an AI device with Jony Ive (although rumors are it’s not a phone).

‍

Implications for startups

Amidst all of this, the question we ask ourselves as seed investors is: how does this impact us, and where should we invest? Despite the daunting behemoth of Apple standing in the way, we think better on-device inference is the key unlock to a proliferation of a new wave of successful consumer apps. That opportunity is massive, and there may be ways for startups to tackle it and succeed. Some interesting spaces and startups include:

‍

In this space, better infra will lead to better apps, which will inevitably require build out of better and more complex infra, and so on. We believe the move to on-device will create opportunities up and down the stack, and that the catalyst will be better inference hardware in your phones and laptops.

Additionally, we’re thinking about this recent tweet from Renee Shah at Amplify:

‍

We agree, and would note that all the companies mentioned here from the mobile generation are consumer companies. Two of the three AI ones mentioned are B2B. We think consumer AI businesses will spawn great infra, just like they did with mobile. However, we need better hardware to unlock this wave.

‍

As always, please share any thoughts and feedback with us, and if you know any teams building in this space, we’d love to meet them!

‍

Until next time,

B&T&G

‍