What will be the personal GenAI device?
What will be the personal GenAI device?
A primer on the GenAI smartphone opportunity
Hey all, I just returned from a writing hiatus to spend time with my family. I hope you enjoy my most researched piece to date about the potential of personal AI devices. Note that I have invested in many of the companies discussed below, which I disclose at the end of the post.
The potential of personal AI devices
I made a point to learn about the hardware aspects of AI this year. Historically, I've focused on the software side, but I realized that hardware is the major constraint for generative AI today. Currently, there are not enough servers in data centers to train the next frontier model, and the processors in existing computers and portable devices are not powerful enough to run these models. However, jumping straight into the world of semiconductors and devices was overwhelming. The supply chain is complex, and most books focus either on geopolitics or the electrical engineering aspects of wafers and circuits. I was specifically looking for a product & business overview.
I ended up with a hodgepodge of materials of varying quality and depth.
So, I gave up on old-school research and turned to ChatGPT. With ChatGPT, I could ask about anything, wherever my mind took me. When I encountered jargon like RISC (reduced instruction set computer), I could get an immediate explanation instead of parsing through Google results. Instead of listening to podcasts and watching videos, I talked to ChatGPT. That wasn't the first time I used the voice capabilities, but it was my first time having a lengthy conversation about a complex topic with ChatGPT. The transformative experience was strolling around San Francisco while talking to ChatGPT. I had a research collaborator in my pocket that I could bring anywhere. I didn’t have to be glued to a screen. All it takes is a few taps to start talking to an AI assistant that knows more about semiconductors than 99% of the world.
When OpenAI demoed GPT-4o, it showcased another subtle yet significant improvement in user experience: the ability to interrupt ChatGPT and redirect the conversation by talking. OpenAI models are overly chatty. So when I’m talking to ChatGPT while my phone is in my pocket, I have to either listen to the entire response or pull out my phone to interrupt it manually. This might seem like a mundane quality-of-life upgrade, but it's incredibly more natural. It feels more human. After spending hours talking to ChatGPT, the potential of personal AI devices dawned on me. It is not a conceptual idea anymore but a lived one.
Types of personal AI devices
Personal AI devices are pocket-sized pieces of hardware through which people primarily interact with an AI assistant. Desktops, laptops, and AR/VR headsets don’t make the cut. Being pocket-sized is important because it allows the user to carry the device anywhere. This aspect of being pocket-sized is what made the iPhone iconic: Your life in your pocket.
The opportunity is immense: The iPhone launched the mobile era and generated $91B in sales for Apple last year. With the smartphone market valued at $485B annually, investors are eagerly hunting for startups aiming to create the iPhone of the GenAI era. More than $250 million has been invested in the makers of the distinctly neon orange Rabbit R1 ($199) and the Apple-esque Humane Pin ($699 + $24/mo subscription + $100 for a shiny chrome finish).
Despite getting awful reviews, it is laudable that both companies attempted to create new form factors. I am looking forward to seeing more ideas. However, I think we will likely see upgrades to the devices that we already have today: smartphones and wearables like in-ear headphones, glasses, and watches.
Smartphones have an established form factor that can fit enough memory and computing power for GenAI models. They are the most essential piece of personal technology today — we carry them everywhere to take pictures, call loved ones, reply to work emails, and binge-watch videos.
What’s new in this category: large foldable phones such as Samsung Fold and Pixel Fold are becoming more popular. Users get a larger screen, and manufacturers can pack in more battery and processing power.
Earphones are interesting because audio is central to communication. Until we get brain-computer interfaces, audio is the most efficient way to communicate. While we can use gestures and wave our hands, it’s inefficient and weird. The tiny form factor of in-earbuds makes it impossible to fit enough battery and processing power to make them viable AI devices. While over-ear headphones can pack more, they are not pocket-sized.
What’s new in this category: Companies are experimenting with new designs such as Google-spinout Iyo's One buds, which are noticeably larger to fit in a larger battery, ten microphones, and standalone internet connectivity.
Smart glasses are interesting because they capture what our eyes see. Vision is fundamental because sometimes it is easier to communicate with images than words. This is more so for people whose language is not the primary language the models are trained on, such as myself, who grew up speaking Tagalog. With flowers blooming in San Francisco summer, I recently passed by flowers I wanted to identify. Instead of attempting to describe it — lamp-shaped orange flowers with hairy buds in San Francisco — I sent a picture to ChatGPT, which identified it as Orange Clock Vine. Using the textual description, I got California Poppy instead. A picture is worth a thousand words.
What’s new in this category: Meta is the leader in this space with the Ray-Ban smart glasses and is reportedly exploring designs combining both earphones and cameras into one device. Google, which was way ahead of its time with Google Glasses 11 years ago, is also developing a new device today.
Smart watches are not interesting because they are only good for tracking biometrics. Outside of that, they are primarily used as an extension of the smartphone, so they are unlikely to be personal AI devices.
What’s new in this category: more sensors.
What will be the personal AI device that everyone uses? I bet that it will be smartphones, with wearables relying on them as the processing unit. We already see this pattern today with our earphones and watches paired to smartphones via Bluetooth. Wearables augment what we can do with a smartphone, but wearables are useless without a smartphone.
What are GenAI smartphones?
While AI has already been part of our daily smartphone interactions, native GenAI experiences are new. Smartphones today need an upgrade to become personal AI devices or what industry analysts coin as GenAI/AI smartphones. An AI smartphone is defined as having:
A GenAI assistant integrated with phone controls and OS to interact with mobile apps.
Capability to process on-device foundation models and funnel complex workloads to cloud-based models. There are a couple of reasons why on-device AI matters:
Local processing of sensitive data: Processing data locally can keep data private. This is important for apps that handle sensitive data, such as messaging apps with end-to-end encryption.
Offline access: Users can access AI features even without an internet connection. This is useful for applications that need to work offline or with spotty connectivity.
Cost savings: Developers can reduce inference costs by offloading execution to consumer hardware, ideally leading to lower mobile app prices.
Hardware capable of delivering the features above. IDC defines this as a phone capable of >30 TOPS (trillion operations per second).
According to market research firm Counterpoint, of the 1.2 billion smartphones shipped in 2023, less than 1% met the definition of an AI smartphone. 2024 will be the year of AI phones, with vendors launching their first AI phone models. Samsung launched the Galaxy S24 in January, and Google and Apple are expected to release theirs in the coming months. Supply chains are rapidly being reconfigured to ramp up production of AI phones, which are expected to comprise 43% of smartphone shipments by 2027
Who is positioned to win the race?
While high-end specs are the minimum requirement to play in this space, winning requires a deep ecosystem of developers, applications, and other personal devices. Mapping these factors against the key players, Apple and Google are best positioned to win the space. However, their paths to winning differ.
Apple is the leading smartphone maker in the US, with over 50% market share. It also has a wide install base of Apple TV and Mac computers — macOS holds 20% of the laptop and desktop market. Many are looking towards WWDC 2024, when Apple will share its GenAI plans and OpenAI partnership. We may not yet see the next iPhone then, but in a few more months, Apple is expected to launch the first “Ai”Phone. From a business perspective, these will drive customers to upgrade sooner and Apple could also add an Apple Intelligence service fee.
While Apple will win with its vertically integrated ecosystem, Google will win via software and ecosystem lock-in. Its line of Pixel phones has less than a percent of the global market and ~5% of the US market. But Google's play is software. Android holds ~70% global OS market share. During their recent I/O conference announcement, Google committed to making Gemini a core part of the Android experience. Gemini Nano, arguably the #1 on-device model, will seamlessly integrate with the #1 smartphone OS via the Google Edge AI SDK. While Google does not directly charge for the SDK, ease of use and “free” on-device inference incentivize developers to build with its SDK instead of managing cloud AI costs or a separate foundation on-device model. If developers want to use more powerful models in the cloud, Google will make connecting to larger Gemini models easy. More Android developers = more apps = more Google Play app fees and Google Cloud revenue.
Samsung is also in a strong position as the largest smartphone vendor globally, barely beating Apple. It has the most extensive range of smart home devices and is even more vertically integrated than Apple. The Samsung group of companies has its own battery and chip fabs. However, if Samsung wins, Google also wins since many of the key AI features in its latest flagship Galaxy S24 phone rely on Google’s AI.
Where can OpenAI play?
OpenAI is purely on the software layer with GPT models and ChatGPT. Their advantage is that they can provide a seamless experience across ecosystems and devices. Users own a range of devices. For example, I have a Pixel Fold, a Dell work laptop, and a personal Lenovo laptop. I’d like to interact with the same AI assistant that knows my data regardless of where I interact with it — and that could be ChatGPT. The counter here is that Google and Microsoft are playing the same game: the Gemini and Microsoft Copilot apps are available on iOS and accessible via any browser.
A deep partnership between OpenAI and Apple makes sense because both complement each other’s gaps. OpenAI has a market-leading AI assistant, and Apple has a broad portfolio of devices that seamlessly work together. Alternatively, Apple could acquire OpenAI. OpenAI’s latest valuation of $80 billion is “just” less than 3% of Apple’s $3 trillion market capitalization.
Who could win among the smaller private companies?
While I believe that Google, Samsung, and Apple will dominate the AI phone market, there is one private company I think could carve its own niche: Nothing Tech. The UK-based company is founded by OnePlus co-founder Carl Pei. In case you didn’t know, OnePlus is one of the largest phone brands globally, although primarily known in Asia.
Nothing makes well-reviewed and beautifully designed mid-range wireless earbuds and smartphones. If I were not attached to Pixel, I would use Nothing devices. Product aside, what makes them unique as a company is how fast they launched products and grew despite cutthroat competition. Four years since being founded in 2020, Nothing has already sold 3 million devices worth ~$560 million across two generations of phones and three generations of earbuds.
Nothing also thinks smartphones will be the personal AI device that everyone uses. Their CEO recently shared what I think is the clearest vision of what the GenAI era means for smartphones: the opportunity and challenge of moving away from an app-centric paradigm to one in which AI is the super app. Check out the video tweet in which Carl elaborates more on Nothing’s approach and early design concepts for their AI phone.
If OpenAI wants to accelerate its hardware play, Nothing is a prime acquisition candidate. It is the only hardware company actively integrating ChatGPT into the user experience — it was the first to integrate ChatGPT as the default assistant for wireless earbuds. Its phone OS also features ChatGPT-specific widgets and functions.
Challenges & opportunities in AI smartphone components
As AI smartphone production ramps up, companies that can solve the current hardware constraints are best positioned to rise with the market. The three primary constraints are processors, memory, and battery.
The brain of every smartphone is the SoC (system on chip), which includes the CPU for general tasks, GPU for graphics and complex calculations, NPU for machine learning, DSP for signal processing, and connectivity modules for cellular, Wi-Fi, and Bluetooth. For simplicity, think of the SoC as the processing power. When an LLM is needed, it is loaded into the device's RAM. The challenge is fitting large models into the available memory without compromising performance and battery life.
To illustrate this, you can see a teardown of the iPhone 6 internals below. The battery occupies most of the space, with the mainboard housing the SoC and memory.
Over the years, users have demanded faster, more responsive devices capable of handling multiple tasks, running more complex apps, and maintaining long battery life. This includes seamless video calls (FaceTime), quick photo and video editing for social media (Instagram), doomscrolling videos (TikTok), and efficient navigation with real-time updates (Doordash and Uber). To meet these needs, smartphone manufacturers have been cramming more powerful SoCs, additional memory, and larger batteries into phones every year. Since 2016, the physical volume of smartphones has increased by over 50%. Running GenAI models on-device is even more resource-intensive, so we’ll likely see phones become even larger and more manufacturers experimenting with the foldable format.
Key constraints to manufacturing AI smartphones
1. Processing Power / SoC / “the chip”
Traditional CPUs and GPUs are often inefficient for AI workloads, leading to the integration of NPUs in modern smartphones. NPUs, designed for neural network computations, enable faster and more power-efficient AI processing. This specialized hardware supports real-time inferencing, enhancing features like advanced photo editing and real-time translation.
Key players: The best SoC manufacturers are Qualcomm, MediaTek, and Apple (for its own devices). MediaTek used to be the mass-market player but has recently made a big push to make premium chips with its Dimensity 9 series. Reports suggest they are also partnering with NVIDIA to produce smartphone chips.
2. Memory / RAM
AI applications require substantial memory resources, particularly LPDDR (Low Power Double Data Rate). LPDDR is a type of DRAM designed for mobile and portable devices. It offers high-speed data transfer and low power consumption, making it ideal for smartphones. For simplicity, just think of it as RAM. The size and power of a model that can be run on a device are constrained by RAM. It is the minimum gating factor. It is recommended that devices should have at least 8GB of RAM available to run the 7B parameter models, 16GB to run the 13B models, and 32GB to run the 33B models. The standard Llama-3 model has 70B parameters.
Quantization is a process that reduces the precision of the numbers representing the model parameters, which can significantly reduce the memory footprint of AI models. For example, a 7B parameter model typically requires 16GB of RAM. If quantized to half its size, the RAM requirement could potentially be reduced to 8GB. This reduction makes running larger models on devices with less memory feasible, albeit with a trade-off in accuracy.
Interestingly, the phones with the most RAM are all Chinese phones with configurations up to 24GB, such as the Honor 90 GT. The leading US smartphones today have much less: iPhone 15 Pro Max has 8GB while Pixel 8 Pro and Samsung S24 have 12GB.
Key players: Samsung, SK Hynix, and Micron are the largest manufacturers of DRAM. RAM for smartphones is commoditized; there is no material difference between Samsung’s and SK Hynix’s. What matters more is that they can manufacture the latest generation of LPDDR.
3. Battery & Power Consumption
AI applications on smartphones are highly compute-intensive, significantly impacting battery life. For example, using an AI de-noise filter in Adobe Lightroom required 30 times more battery power than a standard noise filter. Battery life, which is what users care about, is determined by chip efficiency and battery capacity.
Chip design is crucial for battery life. Two major design architectures dominate the market: x86 and ARM. ARM architecture, known for its energy efficiency, powers nearly 99% of mobile chips. Qualcomm and MediaTek rely on ARM's design to design its chips. In contrast, Intel's x86 architecture, traditionally used in PCs, is less optimized for energy efficiency. This fundamental difference is why ARM-powered devices, such as Apple's products, often outperform their x86 counterparts in terms of battery life. Finally recognizing the advantages of ARM's low-power architecture, Microsoft has adopted ARM/Qualcomm as the platform for next-gen AI PCs.
Battery capacity is measured in mAh (milliampere-hour). Batteries are more commoditized than memory with batteries from different manufacturers interchangeable with one another. For many years, there have been no new developments until recently phone vendors started using lithium-ion batteries housing silicon-carbon anodes, which have more capacity than traditional graphite anodes. Chinese phone maker Honor uses this technology to pack a 5,600 mAh battery inside their latest flagship phone Magic 6 Pro. In comparison, iPhone 15 Pro Max has 4,422 mAh while Pixel 8 Pro has 5,050 mAh.
Key players: China's Amepretex Technologies is the largest phone battery manufacturer, followed by South Korea's LG Energy and Samsung SDI. It is worth noting that Japan's TDK supplies both Honor and Apple.
Is model performance a constraint?
No. Frontier models have passed the Turing test and continue to progress rapidly. Take OpenAI's GPT model as an example, from GPT-4's first release to GPT-4o's earlier two weeks ago. In 14 months, context length increased 16x, became multimodal, and became smarter, while dropping prices by ~80%.
Investment screen & disclosure
I’ve noted below what I have invested in. The stock price returns below are not mine, unfortunately.
Curated reads
Commercial: Apple’s WWDC 2024 AI Plans