I’m very grateful to everyone who read along, especially those who have been here since the beginning, when I first published a report on foundation models and the business landscape around them (before ChatGPT). Four editions later, and after the biggest one yet in January 2026, your readership, feedback, and pushback made the work better and sharpened my thinking.

The timing feels bittersweet because I have a run of long-form pieces lined up, including the recent essay on whether there is an AI bubble and the now forever-draft of The Future of SaaS, along with a pile of other unpublished analysis. In a way, that feels true to the spirit of Generational. I always wanted to publish only when there was something worth saying. Something worth your time.

Thank you again for reading, and for all the support. If you’d like to stay in touch, you can find me on LinkedIn.

The standard approach is to examine demand and supply separately. On the demand side: is there real economic value being created, or are people buying assets purely because they expect prices to rise? On the supply side: is capacity being financed in ways that can survive if assumptions change, or is the capital structure fragile?

Bubbles can form from either side. Tulip mania and Crypto 2021 were demand-side bubbles—speculative buying untethered from any use value. The 2008 housing crisis was partly a supply-side phenomenon—real demand for housing existed, but the financing structures couldn’t survive a modest price decline. The dotcom bust had elements of both: real demand for internet connectivity, but overbuilt capacity financed on revenue projections that never materialized.

For AI infrastructure, the demand side is the easy part. The evidence for real economic value is substantial and growing. The interesting question is on the supply side: whether the capital structure financing the buildout can survive the assumptions embedded in it.

Demand: Where the Evidence Points

The case for genuine demand rests on observable business impact, not projected future value.

What these numbers represent is companies disclosing AI-driven financial impact in earnings calls—statements made under securities law, not marketing materials. The tripling of companies reporting AI benefits since early 2023 reflects actual deployment reaching the income statement. When Walmart says GenAI replaced work that would have required 100x the headcount, or Estée Lauder reports 31% ROI improvement on media campaigns, these are claims with consequences for executives making them. The investor relations teams and other internal teams vet and validate what executives say publicly.

Consumer adoption data reinforces this. Over half of US adults have used GenAI, with workplace usage nearly doubling within a year. At 900+ million weekly active users, ChatGPT alone has reached a scale where “will people use this?” is no longer a relevant question.

One way to contextualize where AI sits developmentally: think of it like human capital formation. Humans absorb roughly $350,000 in rearing and education costs over 20 years, then produce approximately $2.8 million in lifetime earnings over a 40+ year career. AI spent 2022-2025 in its equivalent “education phase”—massive investment in training foundation models. We’re now entering the “working phase” where deployment generates returns. The important difference: humans retire and their knowledge doesn’t transfer perfectly. AI systems can operate indefinitely and improve continuously.

This framing helps explain why the traditional bubble indicators aren’t flashing red. Economists look at five macro signals that preceded the dot-com crash: extreme valuations, declining corporate profits, elevated credit risk, high cash burn, and rising leverage. These tend to move in sequence—valuations run hot first, then fundamentals deteriorate, then credit markets crack.

The current picture shows valuations elevated but not extreme (95th percentile vs. the 99th percentile at the March 2000 peak), with the other four indicators in healthy territory. This is what you’d expect if demand is real rather than speculative: companies can support elevated valuations because underlying profits are strong, not because everyone is buying in anticipation of selling to a greater fool.

The demand side of the AI market doesn’t look like a bubble. It looks like a genuine technological transition with measurable economic impact.

Supply: What’s Being Built

Meeting this demand requires building across the entire AI value chain—from semiconductor fabrication through data center infrastructure to the software layers that deliver AI to end users.

The chain spans multiple industries with different economics. Semiconductor fabrication is concentrated (TSMC, Samsung) and capital-intensive. Data center buildout requires coordinating power systems, servers, networking, and physical facilities. Cloud providers and operators sit in between, purchasing compute capacity and selling it to software companies and enterprises. Each layer depends on the others. Constraints anywhere ripple through the system.

The buildout currently faces real bottlenecks. In a survey of 149 senior data center professionals, 92% cited utility capacity as a barrier, with 44% facing 4+ year wait times for grid connections. Permitting issues affect 86%, fiber availability 85%, chip access 81%.

The industry response has been to route around constraints rather than wait for them to clear. 62% now plan on-site power generation as primary contingency—a shift from treating utilities as default to treating them as backup. When Elon Musk built Colossus 1, he brought in 35 semi-trailer-sized gas turbines. For Colossus 2, he bought a power plant overseas, disassembled it, and shipped it to the US for reassembly. These reflect how tight the energy market has become.

These constraints explain why shortage pricing exists today. The question for financing is whether they persist long enough to support the debt structures being built around them.

Supply: The Financing Structure

The AI buildout requires approximately $2.9 trillion in data center capital expenditure through 2028. The hyperscalers are spending aggressively and so is OpenAI. CEO Sam Altman references $1.4 trillion in compute commitments over eight years. Dario Amodei describes ambitions for hundred-billion-dollar training clusters. Google’s infrastructure leadership talks about doubling compute every six months.

Hyperscalers’ operating cash flows can fund roughly half of this. The remainder—about $1.5 trillion—requires external capital. This isn’t unusual. Large infrastructure buildouts routinely rely on outside financing. What matters is whether that financing is structured appropriately for the assets it’s funding.

The capital stack breaks down as follows: $200 billion from corporate debt issuance (companies borrowing against their own balance sheets), $150 billion from securitized assets (loans packaged and sold to investors), $800 billion from private bilateral credit (direct loans from non-bank lenders), and $350 billion from other sources including private equity.

The $150 billion in securitized assets and $800 billion in private credit deserve particular attention because of their scale and characteristics. Private credit consists of loans extended by asset managers, pension funds, and specialty lenders rather than traditional banks. Private credit has grown dramatically over the past decade, partly because post-2008 banking regulations pushed riskier lending outside the regulated system. The loans are typically held to maturity rather than traded, which means problems don’t surface until borrowers actually struggle to pay. Securitized assets carry a different risk: loans packaged and sold to dispersed investors who may not fully understand the underlying collateral dynamics—a structure that should sound familiar from 2008.

The nearly $1 trillion in external debt expected to finance data centers through 2028 carries a specific assumption: that the underlying assets—primarily GPUs and related infrastructure—will retain enough value over a 5-6 year loan term to support the debt. This assumption is standard for infrastructure lending, where assets like buildings, fiber optic cables, or aircraft depreciate slowly and predictably over decades.

GPUs don’t work that way.

The Depreciation Problem

When lenders finance infrastructure, they model whether cash flows provide enough buffer to service the debt. In a worst-case default scenario, they assess how much they can recover by liquidating the assets. Ideally, loan-to-asset value stays below 100% so lenders can recoup their principal.

For most infrastructure, this is straightforward. Commercial buildings depreciate over 30-40 years. Fiber optic cable lasts 25-30 years and, because improvements happen in endpoint equipment rather than the cable itself, the same physical glass can carry more data over time as technology advances. Aircraft operate for 20-25 years. These timelines give lenders comfortable cushion against typical 5-year loan terms.

GPUs follow a different pattern. NVIDIA releases a new architecture roughly every 12-18 months, and each generation delivers 2-4x better performance per watt than its predecessor. Unlike fiber, where new technology makes existing infrastructure more valuable, new GPU technology makes existing chips obsolete. Operators must replace the GPUs themselves.

The practical result: GPUs have competitive lives of 2-4 years before they’re outperformed enough to lose pricing power. An H200 purchased today will compete against B200 chips delivering twice the performance. Within 2-3 years, it will face Rubin delivering nearly 9x the compute—an efficiency gap that makes older hardware economically uncompetitive.

This matters for financing because the loans assume 5-year asset lives while the assets themselves become economically obsolete in 2-4 years. The only way the math works is if shortage pricing persists—if demand so exceeds supply that even older GPUs command premium prices throughout the loan term.

How the Financing Breaks

To see where the stress point lies, consider the economics of a GPU-based data center at different pricing scenarios.

The table models a single-GPU data center using a financing structure similar to CoreWeave’s: 85% debt, 10% interest rate, 5-year amortization. Three price points matter:

• Market price is what operators currently charge customers for compute, set by supply and demand in today’s constrained environment.

• Breakeven price is the minimum rate needed to cover debt service plus operating expenses (primarily electricity). Below this, the operator loses money and defaults on debt.

• Equilibrium price is what pricing would look like if supply caught up with demand and GPUs competed purely on performance per dollar. At equilibrium, older chips must price at parity with newer chips on a $/TFLOPS basis, which means steep discounts to their original cost.

At today’s market prices, all GPU generations show healthy margins. This shortage premium is what makes the leveraged financing structure work.

At equilibrium pricing, the picture fractures. H200s and newer chips still cover their costs. But A100s fall deep into negative margins—well below breakeven. Operators holding A100 debt would be unable to service their loans from operating income.

The key question is how long shortage pricing persists. Risky loans in the private credit and securitized asset stack is a bet that equilibrium arrives after the debt matures. If equilibrium arrives earlier, operators face a choice between default and refinancing at unfavorable terms—and refinancing requires finding new lenders willing to take the same bet on even older equipment or invest more money to purchase the latest GPUs.

NVIDIA’s excellence is, paradoxically, the problem. The company’s relentless execution—B200 ramping now, Rubin shipping in late 2026, Rubin Ultra in 2027—means product cycles have compressed to roughly 18 months for a new generation. Each generation delivers dramatic efficiency improvements. NVIDIA’s engineering prowess might be driving the obsolescence cycle faster than lenders modeled.

The hyperscalers themselves are starting to acknowledge this. In January 2025, Amazon reversed course—shortening server depreciation from six years back to five. The company’s 10-K was explicit about why:

This is notable because it goes against the trend of other hyperscalers: Meta, Alphabet, and Microsoft have extended depreciation schedules from 3 years to 5 to 6 years since 2020, Amazon looked at the same data and moved the opposite direction.

Two Forces Accelerating Supply Equilibrium

Several developments are accelerating the path toward compute equilibrium, each working through a different mechanism but producing the same effect: more compute capacity available per dollar, which erodes shortage pricing.

Algorithmic efficiency is compounding faster than most models assumed. DeepSeek provides the clearest data points. Their V3 model, released December 2024, trained with 2.6 million GPU hours compared to 30.8 million for Meta’s Llama 3 405B—roughly 12x more efficient for comparable benchmark performance. The headline “$5.5 million training cost” is debatable, but the efficiency differential was real and reflected genuine architectural innovations.

By December 2025, their V3.2 release achieved results that benchmark against GPT-5: gold medal performance at the International Mathematical Olympiad, top-10 at the International Olympiad in Informatics, second place at ICPC World Finals (programming). Cost to run the Artificial Analysis benchmark is $54 vs Grok 4’s $1,900—35x cheaper for better results. On January 2, 2026, DeepSeek published a paper introducing “Manifold-Constrained Hyper-Connections,” a training architecture that addresses scaling instability—one of the key obstacles to training ever-larger models efficiently. Expect this technique to appear in their next major release.

The implication for infrastructure financing: efficiency gains reduce compute demand per unit of AI capability. If capability-per-dollar improves 10x annually, the compute needed to serve a given market shrinks proportionally. That’s positive for AI adoption and negative for GPU pricing power. Shortage premiums compress not because demand falls, but because less supply is needed to meet it.

Energy constraints may ease faster than expected. Power availability is the most commonly cited bottleneck for data center expansion, but the nuclear pipeline is materializing on an aggressive timeline. In the past 18 months, hyperscalers signed agreements for over 10GW of new nuclear capacity: Microsoft’s $16 billion deal to restart Three Mile Island by 2028, Google’s 500MW Kairos Power small modular reactor (SMR) fleet targeting 2030, Amazon’s $20+ billion Susquehanna campus conversion plus 5GW X-energy SMR pipeline. Regulatory timelines compressed after May 2025 presidential executive orders mandated 18-month maximum Nuclear Regulatory Commission (NRC) reviews for new reactors, down from historical 7-year processes.

If power becomes abundant sooner than expected, the primary constraint on supply expansion disappears. More data centers can be built and operated, which increases available compute and pressures pricing toward equilibrium. The power scarcity premium embedded in current rates erodes.

Both forces point the same direction: toward compute abundance arriving sooner than lenders modeled. Efficiency shrinks demand per unit of capability. Energy expansion grows potential supply. For AI adoption, these are positive developments. For financing structures betting on persistent scarcity, they compress the timeline.

Evidence of Stress in Specific Companies

Broad credit markets remain healthy—spreads are tight, capital is available, no systemic distress is visible. But within the specific segment of data center and GPU financing, stress signals have emerged.

CoreWeave’s stock has declined approximately 60% from peak levels. The company exemplifies leveraged GPU economics. Financials from their Q3 2025 report:

• + $997 million gross profit

• - $631 million depreciation

• - $310 million interest expense

• - $1,412 million quarterly debt repayment

• + $3,129 million in new debt issuance

I believe the stock price reflects investor reassessment of whether shortage pricing will persist long enough for the model to work.

Oracle’s situation is different but related. The company maintains investment-grade credit ratings, but its bonds due 2035 trade at yields closer to BB-rated (junk) than BBB-rated (investment grade) paper. CDS spreads have reached levels last seen during the 2009 financial crisis. The market is pricing specific risk around Oracle’s $248 billion in off-balance-sheet data center commitments—obligations that don’t appear on the balance sheet but represent real financial exposure.

To be clear: the hyperscalers will survive this. When Alphabet extended depreciation to six years in 2023, it boosted pre-tax income by $3.9 billion. When the writedowns eventually come, Alphabet absorbs an accounting hit—painful for the stock, but not existential. Alphabet has $96 billion in cash. The same applies to Microsoft, Meta, and Amazon. These companies have fortress balance sheets. They can afford to be wrong about depreciation schedules.

The risk sits one layer down: in the operators who financed GPU acquisitions with debt rather than cash flow. When CoreWeave faces the same depreciation math, the $310 million in quarterly interest payments don’t pause for accounting adjustments. The hyperscalers take writedowns. The leveraged operators can’t service their debt.

These aren’t signs of a broad bubble. Credit conditions overall remain accommodative. What they indicate is targeted concern about the gap between GPU economic lives and the financing structures built around them. The stress is appearing exactly where the depreciation thesis predicts—in leveraged operators holding older equipment against long-dated debt.

What This Is and Isn’t

The AI infrastructure situation doesn’t fit the pattern of a classic demand-side bubble. Demand is real, growing, and generating measurable economic returns. Traditional macro indicators show one warning sign (elevated valuations) against four healthy readings (profits, credit spreads, cash burn, leverage). The comparison to tulip mania or dot-com speculation mischaracterizes what’s happening.

The risk is narrower and more technical: financing structures built on the assumption that GPUs will retain economic value—the ability to generate cashflow—for 5-years, when competitive obsolescence arrives in 2-4 years. This works only if shortage pricing persists—and several forces are working to end shortage pricing sooner than the financing models anticipated.

If the risk materializes, the result wouldn’t look like a market-wide crash. It would look like credit impairments concentrated in GPU-backed lending, distress among leveraged data center operators, and margin compression for providers running older hardware. The stress would likely surface in credit markets first—in private loan defaults, securitized asset writedowns, and distressed refinancings—before rippling into equity markets as the hyperscalers eventually take the writedowns. This isn’t a prediction that stock prices won’t fall. It’s a view on sequencing: credit cracks first, equity follows. CoreWeave’s stock and Oracle’s bonds suggest markets are beginning to price this scenario.

The investors exposed to this risk are not the ones betting on AI. Its the ones betting on scarcity.

A Note on Michael Burry’s AI Thesis

—the investor who identified the subprime mortgage crisis before it unfolded and the person whom I’ve watched & read about a lot when I started my finance career—has been publishing detailed analysis of the AI buildout on his Substack, . Having now gone through his writing carefully, we’re seeing the same thing—where we differ are more about emphasis than fundamental disagreement.

Burry’s analysis rests on several pillars:

• Capital Cycle Theory. Burry presents historical data showing that stock market peaks tend to occur mid-buildout, often before capital expenditures peak. His chart of S&P 500 net capital investment divided by GDP shows this pattern across the dot-com boom, housing bubble, and shale revolution. The telecom bust wasn’t about demand being fake—internet demand was real. The problem was supply massively overshooting: by 2002, less than 5% of the fiber infrastructure built during the bubble was actually lit.

• Depreciation mismatch. Hyperscalers have extended depreciation schedules from 3 years (2020) to 5-6 years today, while NVIDIA’s product cycles have compressed to roughly one year. This inflates earnings and overstates asset values. Burry documents this from 10-K filings and estimates double-digit billions of earnings overstatement for hyperscalers over 2026-2028. When Alphabet extended to 6 years in 2023, it boosted pre-tax income by $3.9 billion.

• Economic obsolescence. Burry distinguishes physical utilization from economic value creation. NVIDIA’s CFO noted that A100s from six years ago are “fully utilized.” Burry’s response: “Just because a widget is used does not mean the widget is profitable to a degree that it is worth more than residual value.” The A100 uses 2-3x more power per FLOP than the H100. Blackwell is reportedly 25x more energy efficient than H100. An older chip can be running at full capacity and still be economically unviable.

• Demand relative to CAPEX. Burry cites Menlo Ventures data showing application-layer GenAI revenue at $37 billion this year versus $400 billion in chip spending. ChatGPT has 900 million weekly users, but only 5% pay. His concern isn’t that AI has no demand—it’s that supply is being built far in excess of what current revenue can support. As he puts it: “One can believe AI is transformational and also believe that some public and private AI stocks are very overvalued.”

• Credit risk. This is where our analyses converge most directly. In “Unicorns and Cockroaches,” Burry writes: “The big target for private credit now is AI data centers... there is a duration mismatch of catastrophic proportions between the asset and the loan.” In his Q&A piece, he adds: “There is a lot of leverage behind the data center buildout, and I believe bank and non-bank entities will be at risk if that buildout busts.”

Burry is clear about what he’s not saying: “I do not predict that Nvidia, Meta, Microsoft, Amazon, Alphabet are doomed. They all should survive. I believe some will survive like 2000 Microsoft did, some will survive like 2000 Cisco did.”

Where Burry and I align:

Where we differ in emphasis:

On demand, we weight the evidence differently. Burry emphasizes consumer app revenue and conversion rates. I weight enterprise deployment—the S&P 500 earnings disclosures are claims made under securities law, which I find harder to dismiss. But if he’s right that demand is weaker than I assess, that makes the credit thesis worse, not better. Less demand means faster arrival at equilibrium pricing, which means faster erosion of the shortage premium that makes the financing structures work.

One area Burry has developed that I haven’t fully explored: circular financing. He describes how tech giants invest in AI startups who then buy from them, and how “off balance sheet special-purpose vehicles” and “badly structured but easily obtained financing” are goosing GPU demand. To the extent this is true, it amplifies credit risk—loans underwritten against demand that’s partially self-referential are even more exposed when capital stops flowing.

One final thought. Burry made his name by poring through individual mortgage loan files—finding the misaligned incentives and mispriced risk that aggregate data obscured. He saw what the rating agencies and institutional investors missed because he did the granular work.

The same opportunity exists here. The nearly $1 trillion in private credit and securitized assets projected to finance GPU infrastructure contains specific assumptions about depreciation schedules, collateral valuations, residual values, and covenant structures. Someone with access to those documents—the actual loan files, not the marketing materials—could identify which pools are most exposed and how the risk is distributed.

Burry notes he plans to address financing, circular deals, and “a curious liability hiding in the financial statements” in his Part 3. I’ll be reading carefully.

That work remains to be done. I’m curious what he finds.

This article is a summary of the report’s key findings, with screenshots and the AI agent market map. For the full report & database:

• Multimedia and PDF versions of the report: link

• AI agent market map: link or head to the bottom of the article

Key Themes from the 2025 AI Trends report

From Experiment to Economic Engine

This is the fourth edition of my annual AI Trends report. I started writing these in 2022, before ChatGPT launched, and watching this technology evolve from academic curiosity to economic force has been something else.

This year’s report is titled “Crossing the Chasm” because that’s what the data shows: generative AI has moved from early adopters into mainstream production use. It’s no longer about potential—it’s generating real revenue, displacing real labor, and running into real physical constraints.

What follows are the findings I think matter most.

1. AI Agents Are Matching Human Professionals

The shift from AI-as-assistant to AI-as-worker is showing up in benchmarks now. The GDPVal benchmark tests whether AI can produce professional-grade deliverables—things like 3D engineering models and financial analyses. The latest results: GPT-5.2 matches or beats human experts (with an average of 14 years of experience) 70.9% of the time.

The economics are stark. For a range of knowledge work, AI agents are now over 20x cheaper for basic tasks and more than 3x cheaper for advanced work like programming and creative writing.

This is already affecting hiring:

• Employment for workers ages 22-25 in highly exposed roles (software development, customer service) has fallen sharply

• Job postings for AI-exposed roles are down 12% relative to less exposed roles

• When surveyed, companies report AI-driven headcount reductions most often in Engineering (42%), CS & Support (27%), and Marketing (26%)

2. Energy Is the Bottleneck

For all the focus on algorithms and chips, power is emerging as the binding constraint.

Epoch AI found that while chip manufacturing could theoretically support an 80,000x increase in training compute by 2030, power constraints cap that at around 10,000x. Energy is the tightest limit by a wide margin.

This isn’t just theoretical. 92% of senior data center professionals now cite utility power availability as a major barrier. 44% face wait times of four years or more for grid connections. Some are building their own on-site generation—gas turbines, even small modular nuclear reactors.

3. Adoption Is Faster Than Any Technology We’ve Measured

By mid-2025, 55% of US adults had used a generative AI product. That’s a faster adoption curve than the internet or the smartphone.

The products driving this:

• ChatGPT: 900 million weekly active users, >$12B ARR. It’s become the default brand for AI the way Google is for search.

• Claude Code: $1B revenue run rate six months after launch. This is the clearest commercial proof that autonomous agents can replace high-value professional work at scale.

• Sora 2 & Nano Banana: High-fidelity video and image generation moved from novelty to genuine creative tool for millions of users.

4. Acquisitions Are the Exit Path, Not IPOs

AI exit activity increased 44% from 2023 to 2025, but the biggest value is going to Big Tech acquirers, not new public companies.

The interesting dynamic: incumbents are using licensing and acqui-hire structures to sidestep antitrust scrutiny. Microsoft’s deal with Inflection AI became the template—they hired the key employees and licensed the technology without a formal acquisition. It effectively transfers the most important assets while avoiding regulatory review.

5. Is This a Bubble? It’s Complicated.

Valuations are high—95th percentile on the Goldman Sachs bubble framework. But other indicators that typically precede crashes (profit decline, credit risk, leverage) remain healthy. High valuations alone don’t cause crashes; deteriorating fundamentals do.

The new risk factor is asset lifespan. The telecom bubble left behind fiber optic cable with a 25-30 year useful life. The AI boom is built on GPUs with 2-4 year lifespans that depreciate fast as new chips release. That makes the capital stack more fragile and more dependent on continuous external financing.

I’d call the fundamentals healthy but the financing structure worth watching closely.

6. Where the Next Wave of Startups Will Come From

The framework I use: map job tasks by automation feasibility against total wages paid for that role. High feasibility + large wage pool = attractive target for agentic AI companies.

This worked well in practice. The high-potential roles I flagged in last year’s report (software developers, financial managers, legal professionals) attracted the most VC and produced the fastest-growing startups—Cursor, Harvey, etc.

Closing

The data on adoption, labor displacement, and infrastructure strain all point the same direction. The open questions are about pace and constraints—how fast energy infrastructure can scale, how financing structures hold up, and which verticals see the most disruption next.

There’s a lot more in the full report: deeper dives on scaling laws, sector-by-sector breakdowns, and a market map of 1,800+ AI startups. What I’ve covered here are just the findings I thought deserved the most attention.

Full Report

Agent Market Map: 1,800 companies and the top 10% that matter the most

I’ve written before about AI at work — how it affects productivity, tasks, and decision-making. But broad adoption changes the kind of evidence we can rely on. For the past few years, most conversations were narrow: a study on call centers here, a coding experiment there. Useful glimpses, but they described isolated use cases. Without ubiquity, there was no way to see the larger pattern.

That’s shifted. ChatGPT now has more than 700 million weekly active users, sending around 18 billion messages each week. If current trends continue, it should cross 800 million by October. More than 70 percent of those interactions are outside of work. People use it for tutoring, recipes, fitness, or personal writing. That consumer side makes the scale visible, but the real question is what happens to work once AI is this embedded.

On average, each user sends about 26 messages a week, three to four per day. It’s no longer a trial tool. It’s routine. Think of the cadence: grocery shopping once or twice a week, coffee nearly every day, driving almost daily, checking a phone dozens of times a day. ChatGPT is not yet as constant as phone checks or text messages, but it has already settled into the rhythm of weekly life.

This is the turning point. Once a technology reaches that rhythm, we can stop speculating in fragments and start analyzing at scale. And the results for work are more complex than a simple story of automation. Customer support agents close tickets faster. Junior analysts complete more tasks in less time. Writing-heavy roles see measurable gains. But experienced developers in some trials actually slow down with AI, and productivity improvements depend heavily on task design and integration. AI is not just substituting labor — it is reshaping how tasks are done, who does them, and what counts as scarce skill.

That’s why it makes sense to revisit the economics of AI now. Not just through anecdotes or consumer adoption curves, but with benchmarks like GDPVal and studies that measure its impact on output and labor directly.

If you want to dig into that data, join us at The Economics of AI at SF Tech Week. We’ll look at how AI’s spread into everyday life and work is reshaping productivity, automation, and the labor market. RSVP here. Seats are limited so DM me or leave a comment if you’d like a seat.

This update reflects how quickly the space has moved in practice. Agents are no longer a handful of demos. They run in production at well-known companies and cover routine work in software development, customer support, finance operations, and other structured processes. Cursor, Lovable, and the likes are now well-known products seeing rapid adoption globally. ChatGPT now has agent mode and Codex while Claude Code is increasingly used for non-coding use cases. The best models are new, often only a few months old, and the frontier keeps shifting as context windows, tools, memory, and supervision improve. I also expanded the dataset to 19,000 tasks and 830 occupations—about 3,000 more tasks and 30 more occupations than last year—and new wage and employment data.

There is now a credible usage signal as well. Anthropic’s Economic Index aggregates 1 million conversations and maps it across the 22 major occupation groups. That lens matters because it tells us where teams already route tasks through models on the job, not just where models test well. Capability and usage together make the map more useful: one axis for agentic automation potential, the other for observed adoption patterns.

The picture that emerges is straightforward. Computer and Mathematics land in the high-potential, high-usage corner. Teams already push real tasks through agents there, which is visible in day-to-day engineering productivity and support operations. Management, Business Operations, and Office or Administrative work sit in the high-potential, lower-usage corner. The wage pools are large and adoption is still early. That is where new products and companies should grow1.

Because usage data is grouped at the major-occupation level, practical targeting needs more detail. The detailed occupation-level view pairs potential with wage pools across roughly 830 roles and shows where a product can attach to outcomes and budget. Most roles sit in smaller pools. A few dominate and match what we see in the market: software developers; customer service, accounting and audit, information systems management, project management, business operations. These are the places where agents translate cleanly into tickets closed, tests added, reconciliations completed, approvals moved, and minutes saved—and where pricing can anchor to time or output rather than seats. The fastest-growing agent companies lined up with this list, which suggests the framework has been a useful guide rather than a retrospective label.

There is also new & convincing evidence on how AI, and consequently agents, is affecting the job market. Using high-frequency payroll records through mid-2025, researchers examined employment by age and by exposure to generative AI. Since late 2022, employment for workers aged 22–25 declined in the most exposed, automating occupations such as software development, while older cohorts in the same jobs continued to grow. Customer service shows a similar split. Occupations where usage is primarily augmenting do not show the same decline. This does not settle every long-run question, but it is a timely signal that agents are already taking a share of entry-level tasks in the proven corner.

Last week, I wrote about a high-level view of how to invest in AI across public and private markets, and across active and passive approaches.

This can be the roadmap for active private investing. Use the same discipline, now with sharper signals. Start with wage pools by role and overlay observed usage as frontier models turn over faster and policy controls mature. Judge products by whether they finish work inside systems of record and expose measurable outcomes. The practical takeaway is the same as last year, now with better evidence: act where potential and usage already overlap, and build into the gaps where potential is high and usage is beginning to rise.

Curated reads:

• Anthropic Economic Index

• Six Facts about the Recent Employment Effects of Artificial Intelligence

Subscribe now

This guide focuses on equities: public stocks and private company shares. We are not covering credit, convertibles, or land for data centers. Think of what follows as a menu you can choose from based on your appetite for selection and liquidity.

There are two dimensions to the menu. Public vs. private tells you where a company trades and how liquid it is. Active vs. passive tells you how much selection you want to make. Private deals can be compelling but are illiquid. Public markets are accessible and easy to trade. Within public markets, passive means tracking an index at low cost. Active means leaning into a theme or specific companies you believe can beat the market. That said, I won’t recommend specific companies in this article.

Public Markets Overview

From the beginning of 2023 to the end of August 2025, the S&P 500 rose 69%, roughly 22% annualized over that period. This is several times higher than the long-run average gain of 10% annually. The Magnificent 7 (MAG7) — Google, Microsoft, Amazon, Meta, Tesla, Apple, NVIDIA — contributed 40% of the gain. While the economy also grew and favorable business policies were enacted during the same period, most of the gain was driven by the business of generative artificial intelligence.

A more precise view of how AI is affecting the public markets can be gleaned through an index of companies for whom AI is a core strategy. I have been maintaining a list of 100 stocks, and collectively it has increased 157% versus 69% for the S&P 500 and 107% for the Nasdaq. Looking at the chart below, the lines broadly follow the same patterns since most of the gains are driven by the same several stocks like the MAG7.

Passive strategies for the public markets

There are two paths to passive public investing: AI-themed ETFs or the broad market.

Most AI-themed ETFs bundle the obvious AI infrastructure companies — chips, cloud, and model platforms. They will rise faster when AI excitement is high but also fall harder when it cools. They usually charge slightly higher fees, although still negligible, and often lean heavily on a few big stocks. If you pick one, favor larger funds with at least about $1B in assets so it’s easier to buy and sell. While almost any tech-oriented ETF is now AI-driven, there are more AI-focused ones like:

Broad index ETFs like the S&P 500 or Nasdaq-100 are also good choices. They’re simple, diversified, and have plenty of AI exposure given the MAG7 are driven by the AI narrative. The broad indexes also expose you to the rest of the economy that will benefit from AI. AI agents can draft replies, pull records, route approvals, and close routine tickets, while robots take on physical work in warehouses, roads, and factories. Morgan Stanley analyzed every S&P 500 company by mapping employee roles against tasks AI agents and robots could take on. Their estimate: AI could add about $920 billion to S&P 500 pre-tax profit, or roughly 28% of projected 2026 earnings. While this isn’t a forecast of full adoption by that date, it is a measure of potential impact. The key point is that AI’s gains extend well beyond a few big tech names, and a broad index lets you capture that diffusion at low cost and with less single-stock risk.

Active strategies for the public markets

I will neither discuss specific companies here nor how to evaluate a business. There are already plenty of resources for that. Instead, I will go through three AI themes I am investing in. I like thematic investing because it helps narrow the consideration to a handful of companies instead of hundreds. You pick a wave, learn how the business works across the value chain, and build a small basket of companies you like. The three themes are: smart glasses, autonomous vehicles, and agentic software.

1. Smart glasses (AI glasses)

   The interface is shifting from tap-and-type to look-and-ask. Smart glasses are becoming mainstream, with millions of Meta Ray-Bans sold. They are appearing in major retail stores priced like smartwatches. That matters because price bands and shelf space change the demand curve. With AI, glasses have new practical capabilities: hands-free capture, quick messaging, navigation prompts, translation, and “show what I’m seeing” without fishing for a phone. The new Meta Display glasses, with a built-in display, add another layer of practical capabilities: navigational guidance, visual feedback, and more.

   

   How to build exposure:

   1. Devices & platforms: broad consumer pull and software upside. Examples: Meta, Google (soon), Samsung,

   2. Retail and eyewear design partners: for consumer brand awareness and distribution. Examples: Essilor Luxottica, Warby Parker.

   3. System silicon & vision: shared components that power multiple brands. Examples: Qualcomm, Ambarella, ON Semiconductor.

   4. Optics & materials: inputs that scale with units. Example: Corning.

2. Autonomous vehicles

   Autonomous vehicles, specifically Waymo, are rapidly expanding to major cities. In California, monthly driverless trips climbed from ~12k (Aug-2023) to ~708k (Mar-2025) as service zones expanded. In San Francisco, robotaxis overtook Lyft in gross bookings by January 2025 and reached >25% ride-hail share by April 2025, even while pricing carried a ~41% per-mile premium — a sign of demand outpacing early supply. In a forward-looking move, Lyft announced a multi-year partnership with Waymo in September, which includes Lyft investing to construct a purpose-built AV fleet management facility. Live commercial service now spans six U.S. cities, with several more in active testing in the U.S. and Tokyo.

   

   How to build exposure

   1. Autonomy platforms & fleet operators: the software and operations layer that turns vehicles into a service. Examples: Alphabet (Waymo), GM exposure via Cruise, Tesla (if Robotaxi launches)

   2. Network & demand side: aggregators that funnel riders and freight to autonomous supply. Examples: Uber, Lyft.

   3. Compute & perception: components every platform needs. Examples: Nvidia, Mobileye, Qualcomm.

3. Application software (agents inside everyday workflows)

   Agents are moving from assist to automate inside office suites, CRM, service desks, design tools, finance, and HR. They draft, retrieve, route, approve, and increasingly finish well-scoped tasks. Distribution is already installed — millions of paid seats — so adoption can scale without changing buying behavior. The economics tie to time saved and cycle-time reduction, which is why vendors lean on clean add-ons per seat and usage meters per task or minute. AI agents are now handling tasks that take longer and are growing in complexity, with the average time per task roughly doubling every seven months — a practical signal that their scope is widening quickly.

   

   How to build exposure

   1. Horizontal suites with named agent SKUs: large installed bases and deep workflow hooks. Examples: Microsoft 365 and Teams, Google Workspace, Atlassian, ServiceNow

   2. Backoffice (IT, finance, and HR) where outcomes are measured: agents tied to reconciliations, tickets, approvals, payroll, filings. Examples: Intuit, Workday, Oracle Fusion Apps.

Private markets overview

Private market exposure reaches earlier in a company’s life with less information and less liquidity. Most vehicles require accredited investor status under SEC rules. Accredited investor status exists to protect less sophisticated investors from high-risk, illiquid investments. Investment platforms will verify if you are accredited, though occasionally some rely on self-declaration. You qualify if you meet at least one of the following:

• Income: $200,000 individual or $300,000 joint in each of the two most recent years

• Net worth: over $1 million, excluding a primary residence

• Professional certifications: Series 7, 65, or 82 in good standing

• Company role: director, executive officer, or general partner of the issuer

About 20% of U.S. households meet accredited thresholds. There is also a “higher tier” of Qualified Purchaser status that opens more investment opportunities but applies to just 2% of households.

Private markets matter because more value formation now happens before listing. The number of publicly traded companies in the U.S. has fallen from approximately 7,000 in 2000 to approximately 4,000 in 2024, shrinking the investable universe for most investors. At the same time, private companies are growing, both by number and by market value. U.S. private companies now outnumber public companies by over 6.5 to 1. The aggregate estimated value of private companies in the U.S. surpassed $10 trillion in the first quarter of 2025. The number of so-called “unicorns” (private companies valued at $1 billion or more) in North America increased from 20 in 2016 to over 1,000 in 2024. At the same time, the median time from a private company’s initial financing to its IPO has increased, on average, from six years to eleven years. Some high-quality scaled businesses are delaying public listing indefinitely, even after reaching profitability (e.g., Stripe).

Passive strategies in private markets

The choices here are limited, primarily private funds that accept small checks like AngelList’s rolling funds, publicly listed private equity firms, and publicly listed venture funds. However, only a few are focused on AI. The most notable ones are:

Active strategies in private markets

Company selection matters more here than in public markets. Publicly listed companies are already the winners and leaders in their industries, so they tend to ride thematic waves. In private markets you are judging whether a specific company will become one of those leaders. Uncertainty is greatest in the earliest rounds (seed to Series A). By late stage and pre-IPO (Series D onwards) there is more evidence in revenue history, customer adoption, and margins. Unless you have unique information and access, put most of your attention on late stage. In this section, I will go through mechanics more because it is more complicated than investing in publicly listed stocks.

Two axes shape how you can participate. Direct vs. indirect describes the path to ownership. Direct means the company issues or transfers shares to you and your name appears on the cap table. That is common for employees, advisors, and investors. Indirect means you invest through a vehicle that holds the company’s shares. Single-deal SPVs are the usual structures, and your rights are those of the vehicle (not the shares directly). Primary vs. secondary describes what is being sold. In a primary, the company creates new shares and receives the cash. In a secondary, an existing holder sells to a new buyer.

For most individual investors the practical route is indirect secondaries. Primaries via vehicles do appear around marquee rounds, but secondary blocks are more common. For example, I saw SPVs for recent primary raises for marquee companies Anthropic, Replit, and Databricks. But I’ve always had opportunities to buy secondaries. Unfortunately, investing through SPVs often involves a 2% management fee and a 20% carry (a share of profits taken by the SPV owner). Some platforms instead charge an upfront 5% brokerage fee, like EquityZen.

Investing direct makes things more complicated because shareholder rights matter more. You also have more to consider: RSUs, stock options, preferred shares, and other terms. Unless you are a professional investor, you don’t want to deal with all the legalities. For example, I directly invested in a startup through a SAFE that then refused to issue the shares after they raised more money. I had to resort to months of tense conversations and eventually legal threats before getting my money back. If I had invested indirectly through a platform or SPV, they would have handled that.

There are various options to access indirect secondaries and primaries:

• AngelList is the original retail-accessible route with a large network of investment syndicates and emerging managers. It is useful for single-deal vehicles in primaries and some secondaries. The strength is breadth of leads and deal flow. The practical challenge is selection because syndicates vary in diligence, communication, and cost.

• EquityZen focuses on late‑stage secondaries with standardized vehicles and a flat, transparent fee schedule. The company list is curated and workflows are predictable, which makes it a strong default for late‑stage exposure.

• Hiive is a live marketplace for direct secondaries with visible bids and asks. It offers price transparency, but approvals can fail and indicative prices can shift during approvals.

• Forge arranges negotiated secondaries at larger scale. Its strengths are sourcing, issuer relationships, and the ability to aggregate blocks into a closing. Tradeoffs include per‑side fees, more process, and higher minimums.

While late stage shifts the risk away from company specifics (i.e., idiosyncratic risks), I still advise doing company-specific analysis. Use measures you can infer from available information, and Sacra is often a great resource for this. An example analysis would be the ones I published for Databricks and Scale AI before I invested through SPVs. It gave me conviction to invest, and so far it has paid off with ~200% gains in one to two years.

Practical takeaways

• Decide your lane first: Public vs. Private for liquidity, Passive vs. Active for how much selection work you want to do.

• Default option for most people: broad, low-fee index ETFs (S&P 500 or Nasdaq-100). You still get meaningful AI exposure with diversification.

• If you want an AI tilt without picking stocks: use AI-themed ETFs, but favor larger funds ($1B+ AUM), check fees and top holdings.

• If you want to be active in public markets: pick one theme (e.g., smart glasses, autonomy, agentic software), learn the value chain, and build a small basket. Write a half-page thesis before buying anything.

• Private exposure is optional and only for those who qualify and accept illiquidity. Start with late-stage secondaries via reputable platforms rather than early seed bets. Write a one-page thesis before buying anything.

• Know the mechanics in private deals: most retail paths are indirect vehicles. Deals can be delayed or cancelled.

• Check what kind of equity the SPV is buying: Avoid employee options and multi-layer SPVs with egregious fees

Curated reads:

• Meta Connect 2025: The evolution of AI glasses

• Lyft and Waymo launch partnership to expand autonomous vehicles

• How are people using ChatGPT? A study of 700 million weekly active users

I also incorrectly predicted the trend might not endure as scrutiny mounted. Well, the trend has not only continued but evolved with more complex hybrids. Two fresh examples: Meta's $14.3 billion investment in Scale AI and Google's $2.4 billion licensing deal with Windsurf.

In this update, I'll cover recent developments with an updated deals table, how they align with last year's patterns, key differences and insights, per-head fee calculations ($100M/head), and a measured take on employee outcomes. Here's an updated version of the table from my 2024 piece, incorporating the completed quasi-acquisitions.

Subscribe now

Details of what happened: How Windsurf & Scale AI deals follow and differ from previous patterns

These new transactions echo the quasi-acquisition playbook I outlined last year: Big Tech pays premium fees for talent and tech access without full ownership, bypassing heavy regulatory reviews while delivering returns to investors. Here's how they align:

• Avoiding Standard Acquisition Structures: Just like Microsoft-Inflection or Amazon-Adept, both deals sidestep Hart-Scott-Rodino (HSR) Act filings by not crossing "control" thresholds. Meta's 49% non-voting stake in Scale keeps it below 50%, classifying as an "equity investment" under ASC 323—mirroring how non-exclusive licenses in 2024 deals avoided asset-transfer triggers. Google's Windsurf agreement is pure licensing, with no equity, echoing Character.ai's structure.

• High Rewards and Competition Driving the Deals: As I noted, the AI arms race demands speed. Meta gains Scale's data-labeling tech for Llama models, much like Amazon accessed Adept's datasets. Google outbid OpenAI for Windsurf's technology, securing a competitive edge—intense rivalry I predicted would fuel more such moves.

• Investor Returns via License Fees: These ensure VCs get multiples without blocking deals. Scale investors (e.g., Accel) saw ~2x returns on the $14.3B investment, similar to Inflection's 1.1-1.5x. Windsurf's $2.4B fee provided ~1.6x on its ~$1.5B valuation, aligning with my point that fees compensate for lost independence.

• Access to Key Assets: Beyond talent, fees buy datasets and infra. Meta licenses Scale's vast labeled data, akin to Inflection's 22,000 GPUs.

While following the core patterns, these 2025 deals introduce twists that reveal evolving strategies—hybrid elements and more transparent (yet still limited) disclosures. This gives us deeper insights into Big Tech's adaptability.

• Hybrid Structures for Partial Control: Unlike 2024's pure licenses, Meta-Scale blends investment with integration. The $14.3B gets a stake that lets Meta influence without full ownership. This differs from Inflection's clean split, providing "effective control" via board seats (Wang joins Meta AI leadership while retaining Scale board control).

• Increased Scale and Bidding Wars: Fees are bigger: $14.3B for Scale dwarfs Inflection's $650M, reflecting AI's maturation. Unlike 2024's quieter negotiations, these had public rival bids, per reports—highlighting how competition inflates costs but accelerates consolidation.

How much does Big Tech value top AI talent? The $100M headline number

A key metric I touched on last year was cost justification via talent. Let's quantify with per-head calculations, using data from disclosures and reports The denominator is strictly the number that moved to the buyer. Amazon and Microsoft look “cheap” relative to Google and Meta. Since Meta’s acquisition included an equity stake, we attribute 14% of the $14.3B to the ~10 employees who joined Meta. 14% is to reflect the recent 14% layoff by Scale - though arguably 14% is conservative because many of Scale’s top customers are moving most of their business away.

What these deals mean for startup employees

On average, the cost per AI talent acquihired is $59 million. Unfortunately, most of that money goes to the investors. Employee experiences from acquired companies have been mixed—often providing financial and career upsides for those poached by Big Tech, but with disruptions like uncertainty or layoffs for others.

For poached talent: In Scale, ~10 top staffers transitioned to Meta with enhanced packages, including access to vast resources. One engineer posted on Reddit: "It's a step up—better pay, stability, and impact at scale." Windsurf's ~30 hires to Google received packages worth double their equity value, with immediate signing bonuses; many signed on-site during a secretive July 11 meeting. Similarly, Inflection's ~70 staff joined Microsoft with competitive comp, including equity equivalents and Adept's ~66 to Amazon got higher salaries (~$300K+ base for engineers).

For those not poached: Outcomes varied. Scale laid off 14% (~200 employees) post-deal, citing "restructuring for efficiency" after missing projections. Windsurf's remaining ~120 staff initially felt "abandoned" after Google's selective hire, with frustration over non-accelerated vesting —but Cognition's July 15 acquisition waived vesting cliffs and ensured financial participation for all, turning uncertainty into a "great outcome" per Windsurf CEO Jeff Wang.

Overall, these deals cushion transitions for many via Big Tech's resources, but remnants face flux—layoffs or pivots, though often mitigated by payouts. No widespread backlash.

Wrapping up: Takeaways and what is next

These deals confirm quasi-acquisitions' staying power, with twists like hybrids reshaping the playbook. But regulators are watching closely. The FTC's January 17, 2025, compendium includes a "Joint Statement on Competition in Generative AI Foundation Models and AI Products," warning: "We will continue to scrutinize partnerships and investments that may enable dominant firms to exert undue influence or gain privileged access in ways that could undermine fair competition". They've probed similar deals, like seeking info on "serial acquisitions". We might see one or two more this year before the FTC or SEC adds restrictions—such as expanded HSR scrutiny on "quasi-mergers." For those who want a quick exit, the time is now to join a hot AI startup. But which one?

July 20, 2025 update: Corrected the number of Scale AI employees joining Meta to ~10.

In this interview, I speak with Zep’s CEO, Daniel Chalef. Zep, built on top of Graphiti, powers AI agents with knowledge graph memory built from user interactions and business data. I first came across Zep more than a year ago. I was looking for a memory solution beyond vectors and they stood out as the only one considering the temporal nature of data. Graphiti is now one of the fastest growing open source projects — check it out.

Key learnings

• AI memory can't be generic—it has to understand your business: Daniel learned that memory systems need to be domain-specific because a mental health app tracking emotional states and therapeutic progress has completely different memory requirements than an e-commerce agent remembering purchase patterns. The entities and relationships that matter are fundamentally different across domains. Generic memory solutions fail because they treat all context as equivalent.

• Customer intimacy beats feature completeness in AI: While most B2B companies focus on features, Daniel focuses on customer intimacy—the agent's ability to respond accurately and personally. This drives conversion and retention because generic or hallucinated responses erode trust. For AI products, trust is the entire game, and personalization is how you build it.

• User preferences have expiration dates that embeddings can't track: Vector databases treat information as immutable, but real user data changes and conflicts over time. When someone says "I love Adidas shoes" but later "these Adidas shoes fell apart," temporal knowledge graphs can invalidate the old preference while vectors just store both as independent facts. Time-aware context is crucial for AI products.

Subscribe now

What is Zep?

Kenn So: I reached out because I'm a fan of what you're building at Zep. Super appreciate you taking the time. I was looking through LinkedIn — you wore so many different hats, which is very interesting. I'll poke around that during the interview. To get started, can you share about yourself, the company, and the open source project?

Daniel Chalef: So I'm an engineer turned founder. Zep is my second startup. It's also my second open source startup. In between the two startups, I took on many different roles—led very product-focused ML and engineering groups, as well as marketing, as well as corporate development teams.

Interesting set of experiences. The through line to my career has been I've always worked on very technical products that were being sold to technical buyers—engineers, data science leaders, ML leaders, etc.

Zep is near and dear to my heart because it's actually an agglomeration of a bunch of my experiences across both ML as well as some of my marketing experiences. This will become apparent as I unpack the vision for Zep and where we're going.

You can think of Zep as a memory layer for the agent stack. We think more expansively around what memory is and how we can help our customers.

Memory—we need it because without memory, we can't adequately personalize responses for an agent attempting to solve a problem for a user or respond to a user. If they don't have that context, they don't have the personalized context, then they're going to hallucinate at worst or at best respond generically.

And that doesn't do what's important for our customers, which is solve problems and drive customer intimacy. We work with a lot of companies who really view customer intimacy as a key to building a successful agent.

What I mean by customer intimacy is that the agent is able to respond accurately in a personalized way. Why intimacy is important is because it drives conversion rates and drives retention. Because if an agent responds generically or hallucinates, that erodes customer trust, which is the antithesis of customer intimacy.

We primarily work with B2C companies, many at scale. You can see why there's that intersection now with customer intimacy. Everybody from consumer mental health apps all the way through to coaching apps, through to e-commerce, through to—we're even working with, we keep it relatively generic, big consumer goods companies.

Where Zep is different is that we view memory as being expansive. The types of companies that we work with have many touch points for the user beyond just the agent. So there might be email, support cases, billing data, data coming from SaaS applications, so actual events—a user did X or a user did Y.

We allow developers to send us a stream of both chat messages and business data. We call what I just described this broader context—we call it business data.

We then integrate it together on a knowledge graph. It's not just any knowledge graph, it's a temporal knowledge graph. So we're able to understand how user state changes over time. Preferences might change over time, and there might be conflicting information that we later receive that we then have to integrate into the graph.

By using the temporal metadata that we capture, we're able to correctly integrate conflicting information into the graph. For example, invalidate an old preference:

I love Adidas shoes, but because my shoes fell apart, I might be angry. So Zep understands the emotional valence, understands the context, understands the temporal dimension, and is able to say, oh, that old preference is now invalid.

We essentially view Zep as more broadly than just memory. It's a customer data layer for the agent stack.

When we think about memory, we also have a very strong perspective that memory is not one-size-fits-all. Memory is very much a domain-specific implementation.

The types of things that you care about as a developer of a mental health application are going to be very different to the types of things you care about as an e-commerce agent developer.

So we allow developers to define custom domain-specific entities and edges in the Zep graph. They can capture specific things—things like media preferences or shoe preferences and retrieve those more deterministically from the graph with higher data quality and higher accuracy.

How it all started

Kenn So: There's a lot to dig in there. I saw the entity types blog post, which was great. But I want to take a step back into maybe the beginning of Zep as well. From LinkedIn, I saw you started March 2023. Walk me through that. Those were really early days post-ChatGPT.

Daniel Chalef: It was crazy because it was just after the seminal agent paper. When I started really thinking about this was before we called agents "agents."

I'd been using language models back to BERT and BART and others back in 2017, 2018. I'd been thinking about them for a long time and independently thought that we could be building something that could work autonomously.

I started to build an application that would take boring old policy and process documents and turn them into living and breathing—not only, it was during the vogue of the time of GPT-3.5, and there were all the chat over documents applications, Q&A over docs, and I was like, oh, come on, there's no defensibility here.

I wanted to build something that was far more compelling—an agent that would actually take and consume documents and turn them into a process.

There's no state. These applications, the models were stateless. I would have to determine what goes into a prompt. Prompts were small. The token windows were very small then, like 4,000 tokens. I had to work out how to structure that, and I built something that would basically summarize conversations, and I open-sourced it, created LangChain connectors for it, did a bunch of work with LangChain, and actually became a top 10 contributor.

That's how I got to memory—thinking about memory—at least. As I started thinking about the opportunities for what we would be building, I started realizing that there are all sorts of—given my experience with ML and at scale—there are all sorts of pipelines that we need to build to make memory into something and then retrieve memory. Memory shouldn't just be the conversational domain. My application is emitting data as well. So what do I do with that?

How do I contextually recall the right data? I can deterministically pull in data from a Postgres database. But sometimes I don't know what data I should pull in and when. So that non-deterministic part was—the things I don't know at development time and situations that I can't anticipate.

That is where agentic behavior really came into the fore and where having a memory service that allowed agents to recall both past interactions and business data was really useful. One thing led to another, got into YC, raised capital, built a service, a multi-tenant service.

People started using the product like you and started getting a lot of enterprise inbound as well. So we have tons of pilots now with large enterprises and, yeah, it's been exciting.

Why graphs beat vectors for memory

Kenn So: Going a little bit technical, there's this philosophical debate around knowledge graph versus vectors and knowledge graph that was even in the early days of 2023. Was the knowledge graph structure something that you honed in early on?

Daniel Chalef: We knew pretty early on that it was going to be very difficult to capture enough context, temporal and relational context, in a vector database. Vector databases, embedding vectors, exist independent of each other and are, in many respects, immutable. There's no way that you can understand causal relationships between vectors in a vector database.

We initially started with vector search, but knew that it was inadequate and attempted to work out ways that we could understand causal relationships, understand temporality at more depth. Having worked with graph databases in the past and graphs at scale, I knew that the intersection of graph and semantic would be a very powerful approach to memory.

What Zep does today, by virtue of leveraging Graphiti, is it uses both BM25 and Semantic Search to find subgraphs within the broader user graph or user record, and then uses traversal to retrieve relevant recent memory, so that an agent can then reason over the causal and temporal nature of memory that it is presented with, which allows agents to do things like root cause analysis, because not only are they getting, "I can't log in," oh, their user's account is suspended, but they're also getting, "we couldn't charge the credit card."

And why couldn't we charge a credit card? Because the credit card is expired. And you can get that with the graph, because you can look at the relationships.

Kenn So: The thing that drew me to Zep is how you have productionized knowledge graphs for AI. The graph construction , you can kind of hack it together, but building the pipeline to constantly update the knowledge graph is hard. And that's something you’ve built.

Daniel Chalef: Yes. And it was fun building it. We decided to build the heart of Zep as open source, the Graphiti graph framework. We knew it had broader applicability. So we have users building sales intelligence companies. Why? Because mapping companies, the relationships between companies, the employees of companies, how their roles change over time, how they might move to a different company is really useful on—a temporal knowledge graph is really useful for that use case.

Teaching customers to use the right tool

Kenn So: Well, let's go down this thread of technical products. Do you find your customers using Zep along with a more traditional database of storing all the chats and pull them both together? Or do you find them just pretty much sticking with Zep to feed all the contextual information into a model?

Daniel Chalef: We have customers who attempt to overuse Zep. They want to use it for relational data. They want to use it for high-frequency event data. They want to use it for RAG. And, you know, we're often telling them, if the data structure is known at development time, if the data itself is known at development time, and there are simple pipelines to update that data, and it is something that is either an immutable data structure, or is an append-only log of sorts, use a relational database.

You can more deterministically retrieve your data from a relational database, and it doesn't need to be on a graph.

Just because graphs are cool, doesn't mean you need to use a graph for everything. Ditto, we have customers who try and do RAG with Zep, and it's not designed for that when you need to recall the original chunks of your data. Zep can do that, but it's not really designed for that. It's probably better for you to use any number of RAG databases if your data is not dynamic, i.e. it never changes. Then you might as well just vectorize it and put it into a vector database. So we spend a lot of time educating customers.

When open source goes viral

Kenn So: It's super interesting. There are two topics I want to move onto. One is, it seems like Graphiti is on a rocket ship. What's happening there? And how are you keeping up with it?

Daniel Chalef: Oh, man. We're not keeping up with it. There are way more issues than we can handle. It's been challenging.

I think it's just captured imagination. You get into, with an open source project, you sometimes get into a strong virtuous cycle. Somebody might build something cool. And other people think it's really cool and it goes viral on X or wherever and lots of people star the repo, then other people write about it and you might get onto Hacker News or something and you get even more stars.

So you become a project of the day and the more people find out about it because Github emails everybody saying this is the project of the day and then you might stay up on project of the day over an entire weekend and then other influencers might hear about it and then they build things with it and you just get into a positive cycle around a project. And then it gets to a certain size where it's got some gravity to it, and it just keeps rolling.

Kenn So: It does. Now it has 11K stars. Has that translated to enterprise business pipeline?

Daniel Chalef: It has. We have some very large companies who reached out to us struggling with the temporal nature of memory, and thought that a graph database would be better, and then came across Graphiti, and then came across Zep, and now we're talking about pilots with them.

Kenn So: That's exciting.

Daniel Chalef: Yeah.

What's coming next

Kenn So: You're going to be very busy. So what's next for Zep? What can you share about what we should be looking out for?.

Daniel Chalef: There's a lot of work to do when it comes to enterprise adoption. So we're putting a lot of effort into that. We're SOC 2, Type 2 compliant. We have a BYOC offering. And so there's a lot of engineering and infrastructure work that we're spending time on.

That said, we recently rolled out the ability to customize Zep for your domain. And so there's a lot of work going into that. We actually rolled out this past week, the ability to build custom edge types as well.

Kenn So: So you can build custom relationships between nodes now. Interesting.

Daniel Chalef: Yeah. And that's experimental. We're constantly working on latency. Recall latency is down to 200 milliseconds for our hot path, memory.get API. So, we're working with voice agent platforms at the moment on that.

Wearing multiple hats as a founder

Kenn So: We’re almost out of time, any call to action for the readers?

Daniel Chalef: I would just make it checking out Graphiti. A lot of our marketing focus has gone into Graphiti, and Zep kind of comes in as a byproduct. I don't know if you saw my talk last week.

Kenn So: At the AI Engineering conference? I haven’t. I saw you were presenting but I didn't get to attend your talk.

Daniel Chalef: You missed it — the talk was apparently voted best of the GraphRAG track. I have a trophy that I have to go fetch.

Kenn So: Ha, nice. Actually, another (last) thing I really want to ask you is you've worn many different hats, head of marketing, corporate development, founder, engineer. How has that led you to think about how to build Zep?

Daniel Chalef: It's given me pretty deep insight into how to market Zep. I've been close to the go-to-market function, so how to sell Zep as well, what enterprises need. I've worked in—got a lot of enterprise DNA, worked in enterprise software companies, coupled with an ML and engineering background, understanding both how it works, but also what our customers might need and what resonates with them has been very, very effective.

Kenn So: Lots to unpack there and we could go for another 30 minutes. But we can end it here. I really enjoyed the chat. Thanks for taking the time, Daniel.

Related posts:

In this interview, I speak with Kapa.ai’s CEO co-founder, Emil Sorensen. Kapa helps companies answer technical product questions from their customers & developers, instantly and accurately. I first met Emil when they were presenting their hack in the early days of 2023, not long after ChatGPT came out. That hack evolved into the company that is now serving 150+ customers including companies with world-class engineers like OpenAI.

Key learnings

• Focus Beats Breadth: Emil emphasized the power of staying laser-focused on technical documentation rather than expanding into general-purpose chatbots. This narrow focus allowed Kapa to build deep expertise and better serve their specific audience, competing effectively against broader platforms like Intercom and Zendesk.

• Build Your Own Evaluation Framework: One of Kapa's biggest advantages is their comprehensive evaluation suite developed over two years. This "secret sauce" allows them to quickly test new models and features against their specific use case, staying grounded amid AI hype cycles.

• The Last 30% is the Hardest: While talented in-house engineers can build a RAG system to 70% quality, the remaining 30% to reach production-quality involves security, rate limiting, front-end interfaces, and operational excellence that require focus and time.

• Quality of Data Sources Determines Everything: The biggest determinant of answer quality isn't the underlying model but the quality of documentation and data sources. Kapa turns away customers whose documentation isn't ready.

• "I Don't Know" is a Feature, Not a Bug: Users highly value when AI systems admit uncertainty rather than hallucinating answers. This builds trust and makes the system more reliable for technical use cases.

• Small Teams Can Build Big Things: With just 15 people, Kapa serves 150+ companies and has answered 5+ million questions. The key is finding talented people who thrive in high-impact, chaotic startup environments.

There's so much more to learn and unpack from my chat with Emil. Read on below to see what's next.

Subscribe now

Kenn So: Thanks for taking your time, Emil. This is two years in the making, so I'm excited we are here today. For the audience, let's start with the usual - what is Kapa? What's the origin story? How did you get to start Kapa and what's the journey been so far?

Emil Sorensen: At a high level, Kapa is a platform that makes it really easy for technical companies to build RAG assistants and deploy them where their users or employees have questions. What this looks like in a lot of cases - take someone like Docker that uses Kapa - is an AI system that lives on their documentation. For other folks like OpenAI, it might mean they have an active Discord community, so a version of Kapa lives in there to help their end users navigate the thousands of pages of content that the OpenAI team produces.

Now, two years in, Kapa is used by more than 150 companies, has answered more than 5 million technical questions, with lots of cool companies like Reddit, Docker, OpenAI all using Kapa.

Kenn So: Super exciting to hear 150+. And you've got some of the most technically minded companies who are at the AI frontier using Kapa. Was that how you envisioned it right from the very start, or how did it evolve over time?

Emil Sorensen: One thing I don't believe in a ton is this concept of a founder waking up one morning and seeing the light and coming up with a mission. I'm much more in the school that it's exciting to work on stuff that's used by people.

It's a good bridge to maybe the founding story, which is actually pretty straightforward. Finn and I are very close friends. We both met in London for our first Master's degree, thinking we would go study finance at LSE, which we did. Fortunately, just before that I'd spent a summer in San Francisco interning at a startup. Still no idea how I got the gig, but I was working as a self-taught engineer writing some terrible code.

I came back and met Finn in the line for picking up our student cards. I was telling Finn about this startup experience, and he was like, "Oh wow, that sounds awesome!" Our friendship started there. We got excited about all this stuff and looked at each other a couple of weeks into this program going, "What are we doing trying to be bankers in London? This is a horrible idea." But we saw the degree through and in parallel convinced ourselves to go study computer science instead when we were done.

We went to do a conversion master's in computer science, which apparently in London you can just study for a year.

We didn't have any ideas when we were done with our degrees in 2019, but around that time we'd been tinkering with embedding models and fine-tuning early GPT versions to try to write abstracts. None of it worked, but our intro to deep learning class, they used OpenAI's reinforcement learning work - I think it was Dota where they showed how you could train models. We were looking around this stuff, and they were teaching us how to write basic neural nets from scratch. The attention paper was coming out around this time, but there wasn't much to it.

Kenn So: So early though, 2019. GPT-3 might still be training then and only a few people know about it. How did you kind of get into those early explorations?

Emil Sorensen: It was just around this time that the attention paper was coming out. There was a little bit of chatter around this, but the only motivation we had for tinkering with this was maybe it could write our abstract, which it couldn't. It was horrible. And then we moved past it.

Kenn So: How'd you go from that to McKinsey?

Emil Sorensen: Very personal reasons. At the time I was living in London, but my girlfriend, now wife and soon to be mother of our first child, lived in Copenhagen. If you're kind of ambitious in Copenhagen, there isn't this big scene of cool tech companies to go work for - you go work for McKinsey. As a lack of inspiration, I would have started a startup then, but Finn and I didn't have any good ideas. So he went to work as an engineer for a couple of years while I was at McKinsey.

Kenn So: What then led you to go from McKinsey to deciding to start Kapa?

Emil Sorensen: For a while Finn and I - because I was at McKinsey for three years before we quit - we really tried actively on weekends and early mornings to think about what startup we wanted to build. We wanted to try to build something together, but it was like, "What are we going to build? We should do some customer interviews." But when you're working 80-hour weeks flying everywhere as a consultant, you just don't have time for this.

We met over summer in Austria to go mountain biking for a week together, and around then Finn started flirting with the idea that maybe we should just quit and give ourselves a bit of time. We left that vacation going, "Maybe quitting first of December doesn't sound too bad." We said, "You know what, it's now or never. We're in our late twenties. If we don't do this now, it's going to be way harder." And then we just quit.

We showed up and started brainstorming mid-January. We bought a whiteboard, and the day after we bought that whiteboard - timeline context, this was mid-January, and by early December ChatGPT had launched. Ada embeddings came out December 13th or 14th. Around this time we were reading Hacker News, and people were beginning to say you could finally build these systems. They didn't call them RAGs then, but these systems where you pull data back from a database. Super lucky with the timing, to be honest.

The day after we bought a whiteboard and started brainstorming without any ideas, we were really lucky to have two friends that each had co-founded dev tools. One was the guy we met while studying computer science in London - he had a YC-backed computer vision company. Another friend, Nick, here in Copenhagen who I worked with at McKinsey, co-founded Medusa.js, a big headless commerce platform with tens of thousands of Github stars.

Both of them called us on the same Tuesday saying, "Hey guys, we know you quit, you're looking for ideas." Both were complaining about the same thing: their engineers were spending so much time on Discord and Slack communities answering technical questions that maybe with this new ChatGPT stuff might be solvable now. "Could you guys try ramming our documentation and some GitHub issues into some sort of assistant?"

We spent time building a prototype and did that. That's the origin story.

Kenn So: Such an amazing founding story. So Kapa started there, then you went through YC two years ago. There's so much to unpack in between then and now. Maybe let's start with the product. How has the product evolved?

Emil Sorensen: The interesting thing is at its core, not much has changed. We still try to do one thing really well, which is: you give us all your technical content - all your developer documentation, all your API references, all your GitHub issues. And we'll do the best job possible of letting you ask questions about that and generating an answer.

Whether that's using RAG, whether that's using some reasoning-based approach, whether that's using a Claude model or an OpenAI model, or open source or fine-tuned model - we don't really care. All we care about is whatever does the best in our evaluations, and then we'll ship that. We initially struggled with what to call this, but we've settled on "answer engine" as a concept. Question comes in, answer comes out. Our users don't really care what we do under the hood.

That's our North Star. Of course there's been a lot that's happened since then from a product perspective - integrations, platform features, all this stuff - but really at its core, our job is just to give really accurate answers based on the content you give us.

Kenn So: Having interacted with Kapa since the early days of the industry, its fun when I still encounter it in new AI projects I am working. It's still the same experience on Discord, still gives you accurate answers, still says "I don't know" if it really doesn't know. But I am sure a lot has changed under the hood for the past two years. What were the major developments in terms of what made your product harder to build, or maybe made it easier?

Emil Sorensen: Let me break it down into interesting data sources, interesting integrations, and then interesting approaches to the actual models themselves.

From a model perspective, without a doubt, the models themselves getting better is definitely a big improvement for Kapa. Initially, I heard "GPT wrapper" so many times, but that meme is finally starting to decline as people see this is just another piece of infrastructure you need to build on top of and tame, much like you do with a Postgres database.

The tricky part to navigate is, if you're very deep on Hacker News or Twitter, you will freak out as a founder in this space if you don't have anything to evaluate these things against. For us the biggest lift was very early on making a bet that the secret sauce for a company like Kapa is our approach to evaluations.

Building trust in our own eval set so we can constantly, whenever people are panicking about DeepSeek or seeing that Anthropic is releasing a new citations API, very quickly test that for our use case to see if this is actually helpful or not. One thing we really care about is a model's ability to say "I don't know."

Kenn So: That's really interesting. Are there specific instances where evals kept you grounded? Were there instances where you saw the hype was real versus ones that were duds?

Emil Sorensen: The Anthropic team really figured something out with Sonnet 3.5, that was very good. To lift the lid on what's very top of mind for us now, GPT-4.1 is a phenomenal model, but it takes a lot of taming to get it right. For grounded RAG use cases, it's great.

Some very hyped things have been open source models. And it's a bummer as an engineer, you want to see open source catch up. But they've just constantly lagged behind, at least on our evals, by a year to year and a half in terms of performance.

Kenn So: Yeah, evals really help you keep grounded amidst all the hype. If you don't have a particular use case, it's really hard to know what you're going to ground yourself in as a general bot.

Emil Sorensen: Exactly right. Today, when someone like a larger technical enterprise evaluates Kapa, usually the main competition we're up against is a crack team of really good engineers internally that might go, "Hey, it makes sense for us to build some capabilities in-house around this."

We've been able to crowdsource our evaluation suite over close to two years to understand how a model like Kapa is supposed to behave.

It's one of these things where it's very easy to build a prototype that gets you 70% of the way there. It's very hard to get that last 30% to get it to a point where you feel comfortable putting it into production.

Kenn So: That's a point I want to unpack later - you are up against crack engineers who could probably put a RAG model against their data sources. But let's go back to earlier topics. You mentioned data sources and integrations. Maybe we could go there next.

Emil Sorensen: If you think about Kapa, it's data sources in, question comes in, answer comes out. One of the things you can play with is the actual model and how a question comes in and retrieval generation. The other thing you can do is think about data sources.

A good example here is something like a PDF. We work with a bunch of semiconductor companies and all their documentation lives in data sheets and PDFs. Horrendous data format. We've tried now three or four iterations over the last two years of building a PDF converter that we were satisfied enough with the quality to fold into our product.

We've tried coming back to this every half a year, and it's only now in May 2025 that it's gotten to a point where combining OCR and VLMs has finally gotten good enough that we're like, "Okay, this thing we've been working on for two or three months to ingest PDFs is finally good enough to make its way into Kapa's integration."

The reason I say that is because we care about very specific things. Unlike a general system, we care about ability to cite, ability to preserve hierarchy within a big document. Some of these data sheets are hundreds of pages. Once you have folks you're working with, it's easy to go find examples and build a specific eval suite and work against that to build your very niche integration.

It's that story time and time again - how do you think about GitHub? How do you think about Zendesk support tickets? How do you think about all these other technical sources? Which is cool because we can afford to do that because we're so focused on just working with technical companies. But if we positioned ourselves as a general-purpose chatbot that's also supposed to do support for e-commerce companies, there's no way we can invest this amount of time into parsing through and structuring 100+ page long technical PDFs.

Kenn So: That's a good segue to the point of you being up against world-class engineers who can probably do 70-80% RAG off the bat. What's that 10-20% that you're so focused on? What makes it really hard and tricky that you're focusing on?

Emil Sorensen: I don't think there's any big secrets other than it's just the engineering slog around it. What we often see is a team of really good engineers can definitely build something with pretty good answer quality. Sure, we can poke at edge cases where Kapa is better, but very quickly, this team will have spent half a year building something like this.

Then they realize, "Oh, at some point you also have to take this to production. How do people want to consume this?" They want to consume this as a chat widget on your website. Well, do you also have a bunch of front-end engineers who are ready to build that? How are you thinking about security? You can't just expose your chatbot for anyone to inspect the network request because they can hammer it with a million requests and bankrupt you overnight. You have to think about proxies and rate limiting and reCAPTCHA.

There's all this other stuff that becomes table stakes. To be honest, I'll try to check my own bias, but if all you're trying to do is build a system that's able to answer questions about your product that has access to your docs, it probably doesn't make sense to use your very expensive engineers that are able to build large language models to be building that.

Instead, what you should do - and we can talk about a few examples - is do what a company like Docker does. They have some really talented, very capable engineers that are thinking about how to build Docker support for MCP servers. Go do that! That's really good. Or how do you build a Docker Desktop AI assistant that can look at your docker-compose file and take action?

What they do is say, "Well, we still want this exposed as a chat interface, but we don't want to build and maintain something like Kapa. Let's just use Kapa as a tool call in that case." That's the product approach which is just part of a stack. For most companies, that's probably the path to go.

Kenn So: When you work with customers, what does onboarding look like? Is there any sort of fine-tuning or evaluation that you do for each customer? I'm curious because even within technical companies, there's a wide range from hardware to software. Even within software, you have map APIs versus JavaScript front-end types. How do those change how your product is experienced?

Emil Sorensen: On the onboarding aspect, that's really company dependent. We let folks self-onboard that are very capable and really want to do that. We also lean in a lot because one thing we've realized is how you set up an instance of Kapa - it's not obvious. Just to say, "Do you want to let us slurp up all your GitHub issues?" We've seen a lot of cases now, so we like to share that advice and help folks make sure they get everything configured correctly, because that is the biggest determinant at this point of the quality of answers you'll be getting.

That also answers the second question a bit, which is we try to make Kapa as general-purpose for technical questions as possible. How it performs - by this point, wasn't the case two years ago, but by this point the biggest determinant of that quality is really the quality of data sources.

Unfortunately, just before this I was talking to a customer who decided not to move forward with Kapa, and they were very honest. They said, "We thought we were at a place now with our documentation where this could be helpful enough to answer questions, but we need to come back in a couple of months. We have some work to do on our documentation. It just doesn't cover what our users want to ask."

We try to build some of this into the product to make it easier to find these gaps, but that really is super important.

Kenn So: How do you assess if something's good enough from a documentation perspective, from a data source perspective?

Emil Sorensen: By now, the whole Kapa team can take a look at a set of docs pretty quickly and just kind of sniff it out. But the TLDR is, you don't really know before you try it.

Kenn So: You mentioned trying to help fill that gap for your customers - "Hey, this is probably an area where you need to fill up the docs or some data sources." Hey, this is a good segue into the different modules and maybe vision into what Kapa could be. Because you can do so much with the data you have. You should be able to say, "Based on the questions, this is an API doc area where you need to fill out, or this is where you have the most pain points in terms of which endpoints." I'm curious - you have so much you could extract from the data. What's next?

Emil Sorensen: That's a great question. The short answer is whatever is important to the company that's using Kapa. For most folks, the first thing they get started with from an analytics perspective is starting to build some workflow habits around just on a weekly or monthly basis plugging some holes in their docs.

We try to design the product to lend itself to those workflows so they can go, "Oh yeah, seems like May 2025 is the time I focus on my getting started guide because a lot of folks seem to be struggling with that, or maybe it's time to focus on my React SDK docs because there's really high uncertainty in Kapa's ability to answer questions around this topic" - it says "I don't know" a lot.

For other teams, they take it a bit more at a strategic level, so they'll use it more as a tool to say, "Hey, we're thinking about launching on-prem finally. Do we have any data to support that users are increasingly interested in this or less?" It can also become more of a research tool.

But to be honest, this analytics thing on unstructured data produced by LLMs - that's still a tricky problem to solve. I don't think we've completely cracked the right approach yet.

Kenn So: This is a question I should have asked at the top of the interview - who are you primarily selling to? Which personas? How has it evolved? I imagine it's engineers at first, and now maybe for more mature teams, you have developer experience teams.

Emil Sorensen: You nailed it. That's pretty much it. There are some slight variations. Usually the core unifying thing is it's whoever owns and cares about onboarding documentation, customer education. Turns out this title usually is different from company to company. It's not super unified, but developer experience, head of documentation, technical writer, that kind of stuff.

Kenn So: I'm curious what you've learned from watching them use your product. Has it changed, or is it still fundamentally the same? I know they're a little bit more strict and critical when it comes to the output and will point out inaccuracies more.

Emil Sorensen: That's a good question. Generally, this is just a complete mega trend now of people adopting LLMs into their workflow. Obviously I'm biased because the people I talk to use these in the context of Kapa but folks have multiple tools like ChatGPT, Gemini, and Claude open, and they each have a different mental model of who they like. Claude's ability to write emails, but ChatGPT's a little bit better at meeting notes and so on. So everyone's more comfortable with it.

That's a misconception a lot of companies have when they think about rolling out a system like this - "Oh, people aren't comfortable with LLMs. What if it hallucinates?" Because no system is perfect. But what's the alternative?

A really interesting insight is people really like it when the bot just admits defeat and says, "I don't know." This is one of these things that by sheer luck in prompting that first week or two, pre-launching Discord deployment, whoever stayed up late that night and wrote the first couple of meta prompts had this over-indexed on "I don't know." But that's really just a core product feature since.

Kenn So: It is such a different experience back then of models saying they don't know, because everyone's trying to get the models to do everything and anything, and there was a lot of hype. That was my experience as well. I built a POC internally, and the first thing leaders tried to test was like, "Can it say I don't know, or I can't answer that?" It's a user trust feature that's more subtle.

Maybe we can shift a little bit into building. Any guidance or tactics and insights about how you were able to grab all those amazing 150+ customers, including some of the best ones? I know one tactic I really love, even from two years ago, is you build demo chatbots for prospective customers. That was really impressive.

Emil Sorensen: With the Langchain example, Harrison at the time - obviously we were all in the Langchain Discord and following what's going on and using Langchain first and all this stuff. Early on it's like, "Hey, this seems like a really helpful thing. Let me just send something over to Harrison." He's like, "Yeah, that's cool. Let's deploy it." That definitely helped.

Other things that have really helped - it's been really, really narrow in our focus. A lot of people right now - which I get is natural - think about, "Well, okay, Zendesk has this AI offering and Intercom has this." And they're fantastic. Intercom is fantastic, Zendesk maybe less so. But these systems have to cater to such broad audiences, whereas we really speak to the persona that uses Kapa, which is one that cares about technical documentation.

That has really helped us to also when someone's saying, "I'm not trying to solve your support team's problem or McDonald's support chatbot problem" - some other smart folks are doing that. And they're very different LLM problems you have to solve for that. What I'm trying to do is this.

Kenn So: Okay, maybe since we're talking about how to get customers, shifting gears again into building your team. I'd love to just talk about how you're building your team right now in terms of the skills that you're looking for and any philosophies you're applying to it. Is there anything specifically for AI startups?

Emil Sorensen: It's not going to be anything super novel. We're just married to the YC playbook, to be honest. In terms of hiring, it's very merit-based. It's very big on cultural fit. Startups are not for everyone. The biggest learning here is really for both sides when evaluating whether or not to join a startup like Kapa.

We have people come in and work with us for a couple of days, and we pay them a bunch to do that, which is definitely not the norm here in Europe where we're mostly hiring out of. Just to say, "Hey, this is going to be more chaotic than your job at Google. The amount of impact you can have will be 100x outsized compared to what you're able to do now." Some people just thrive in that context, and other people just like going back to their eight weeks of vacation a year here in Europe and 37-hour work weeks.

That's been a really important element too. In terms of AI focus, maybe an interesting thing from an engineering perspective is you have a bit of a split in terms of engineering preference. Most of our engineers - all of our engineers are phenomenal. They're really very high - I'm sure every founder will say this - but very high talent density team. But we have a couple of guys who just hate dealing with LLMs because of their non-deterministic nature and would get so much more joy shipping a new integration or shipping a new feature on the platform because you see the output, whereas other folks that are more research-minded just love wrangling in this nasty LLM space and can easily spend one or two weeks and don't get deterred when they don't see scores moving up or something like that.

Kenn So: That's so interesting. And then, how big is the team now?

Emil Sorensen: About 15, some part-timers here and there.

Kenn So: That's a lot of customers for 15 people. I know it's software, so there's that ratio thing. But it feels like you've gotten so much done with 15 people. Was keeping it lean kind of intentional? Because you've also raised venture funding, so you have the money to continually hire and potentially drive growth.

Emil Sorensen: I don't know, maybe let's see, the jury's out, maybe I'm completely wrong here - but I don't think more people solve startup problems, at least not necessarily in the AI space. You're seeing that proven out with the recent Windsurf acquisition and all this stuff. What you really need is just a team that communicates very well across the commercial side that's getting constant product input and the engineering side that's building stuff.

Kenn So: I think so. But at some point, you also need to add more people because there's company stuff that you need to keep track of things. Maybe that's a good segue into what's next for Kapa, for you and Kapa. 150+ customers, 15 people, really great customer logos. What's next?

Emil Sorensen: We hold ourselves to very ambitious goals, very high growth rates. So it's just continuing to deliver on those and doing whatever is rational to keep moving in that direction.

It's not the most exciting thing, but it's really just keep making sure Kapa is the best version of what it is, which is a thing that answers questions about your products, and spending a ton of time doing customer interviews to understand as these models get better every week, what new points of value we can essentially build around that to keep making Kapa better.

Kenn So: Any sort of preview you can share an area you’re going to expand to next?

Emil Sorensen: There's always the question around how much do you lean into something like support. That's still very open-ended because that's also a freaking rabbit hole to lean into.

There's a question around the underlying paradigms. The amount of times I've heard people say "RAG is dead" and it turned out to be false, so I'm not going to say that. But I'm very interested to see how paradigms are starting to change. We published a bit of research back in January and February, very publicly talking about our experiments using reasoning models combined with RAG that had some really interesting results, but was at a point where the models we were using just weren't experienced enough with tool calling to consistently call tools at the right time. It's changed very fast.

So that's the space we're looking very much into. But other than that, it's really just helping people that want to build product assistants do that in a really good way.

Kenn So: Okay, last part. How are you using AI personally?

Emil Sorensen: The most personal anecdote is probably for the first time since starting up two years ago - not a very inspirational thing to say - but I finally took some time off with my wife. We're about to have a baby in two or three months, our first kid, which I'm very excited for. There are lots of founder thoughts like how do you balance building a company and being a good dad.

Part of that is we wanted to take some time away. So we did a nice road trip a couple of weeks ago for a week, and Claude's advanced research was phenomenal for that. Very vanilla use case, but that's crazy.

Kenn So: I really like the latency when using something like that too. It makes me feel that the output is earned, that I have to sit and wait for like 10 minutes.

Emil Sorensen: That's a great point. It feels like so much of our experience is we expect immediacy, but the research forces you to kind of wait for that output. It's a little bit like photographing with film - you have to be a little bit more intentional with every shot.

Kenn So: Yes, it's similar to the Starbucks story of customers don't like it when the coffee is instantly available. They like to see grind the coffee and pull out the steamer, even though the taste is exactly the same.

Emil Sorensen: Yeah, pretty much.

Kenn So: Awesome. Let’s wrap here. Emil, thanks again for the time!

Emil Sorensen: This was great. Thanks for having me.

Grain is an AI that grows with you. Through conversations, it'll remember what you talk about, your preferences, your projects, and key moments in life. As I wrote in my previous essay, memories are essential for users to perceive AI not just as tools, but as agents. After writing that, I felt the urge to build—to make the idea real. The result is Grain. I will be writing more about the journey of building this soon, but for now, here’s more about Grain.

Relationships often matter because of the shared memories we build over time. Grain aims to become that kind of digital AI companion. It won't be perfect—it will hallucinate like most AIs—but it’ll try to know you. And whenever you want Grain to forget, you can easily delete specific memories or clear them all.

Because memories grow over time, switching from other AI tools can be hard. So, Grain lets you import conversations from other apps like ChatGPT to pick up seamlessly where you left off.

• What Grain isn't (yet): It’s not built for deep knowledge work or complicated data analysis (like ChatGPT does), nor for extensive web research (like Perplexity does)—though it can do the usual web searches and help with productivity tasks. It also is a browser app for now.

• What Grain is: It's for anyone who wants an AI companion that genuinely tries to remember conversations.

If you’re curious about a simple comparison, ChatGPT doesn’t recognize I chatted with it today. I've had the memory feature turned on since it was available. But Grain did.

If you are curious, give it a go. Expect quirks since I am not a full-time developer. Feedback, feature ideas, and bug reports are always welcome at kenn@grain.computer.

Subscribe now

This limitation is becoming apparent in AI systems as other aspects continue to progress. Since OpenAI’s o1 model release in September 2024, AI models have become remarkably good at reasoning through complex problems. They can analyze data, write code, and engage in sophisticated PhD-level discussions. But they have a fundamental limitation - they can't remember what they've learned from one conversation to the next without retraining . Like a brilliant person with no memory, they solve the same problems repeatedly without building on past experience.

As these reasoning capabilities become standard for the industry, what will set AI agents apart isn't their raw processing power, but their ability to maintain and learn from memory. This isn't just about storing information - it's about building systems that can understand context, remember past interactions, and apply those lessons to new situations. The memory layer will be a key differentiator in how AI agents understand and respond to us.

This article examines memory from multiple angles: how it shapes human cognition, how we're implementing it in AI systems, and how companies are building the technology to make AI agents that learn and remember.

Subscribe now

Memory in humans

Personal: Memory is central to who we are. Our autobiographical memory - the collection of personal life events - creates a narrative connecting our past to our present, giving us a coherent sense of self. Research shows an interesting two-way relationship between memory and self-image: what we currently believe influences which past experiences we recall, and those memories in turn shape how we see ourselves today. Memory also shapes who we are to other people. Through our shared experiences, people remember us as their mother or father, their best friend or their enemy, the funny one at work or the quiet neighbor next door. Their memories of us are just as important as our own memories in creating who we are.

Social: People with strong memory capabilities - who easily recall facts, names, or past conversations - are often seen as intelligent and knowledgeable. Remembering information signals that someone is well-educated and quick-thinking, and it often goes hand in hand with problem-solving abilities. These individuals might also come across as reliable and attentive: think of a colleague who never misses a meeting detail or a friend who remembers every birthday - it shows conscientiousness and dependability. On the flip side, frequently forgetting things can have subtle social costs. When someone often forgets important details or asks the same questions repeatedly, others might question their organization skills or mental sharpness.

Practical: Memory underlies many of our cognitive functions. Far from being just a storage system for past events, memory actively helps us think about the future and make decisions. Drawing on past experiences lets us simulate possible future scenarios and apply lessons learned when facing choices. In fact, memory is involved in everything from problem-solving and communication to social skills like empathy - suggesting it's woven into nearly every aspect of complex thought. Without memory, we'd struggle to weigh options or learn from mistakes.

Memory in AI Agents

User Experience

When we look at AI agents, there are clear parallels between human conversation patterns and AI user experience. For an interaction with an AI system to feel natural, it needs some form of memory. Just as we remember what was said earlier in a conversation, an AI needs to retain context from previous exchanges to respond coherently. Without memory, the AI treats every query in isolation, leading to disconnected exchanges.

That's why it was frustrating to repeatedly copy-paste past conversations into ChatGPT and Claude when I want to bring in context from previous discussions. It feels like talking to someone with no memory of our past interactions. While ChatGPT now has a long-term memory feature that makes this better, Claude still doesn't have this capability.

Here’s another example, a voice assistant interaction: if you ask "Who called?" and the assistant says "Eric H.", you'd expect that following up with "Call him back" would make it dial Eric. Without short-term memory of context, you might get an annoying "Whom should I call?" response, forcing you to repeat information. This is why design guidelines emphasize maintaining conversational memory - it creates a smooth user experience and lets you reference earlier information efficiently. Memory transforms an AI from a basic question-answering tool into a conversation partner that can use pronouns and implicit references meaningfully because it "knows" what they refer to.

Memory also enables meaningful personalization. When an AI agent remembers your preferences, history, or goals, it can tailor its responses specifically to you. A personal assistant that remembers your favorite restaurants or previous travel plans can make more relevant suggestions. A recommendation system that recalls your past ratings can better predict what you'll like. In technical terms, long-term memory lets agents store historical data and user profiles, which directly improves their ability to personalize interactions. Instead of one-size-fits-all responses, a memory-enabled agent adapts to each user, maintaining continuity across sessions (it "remembers" you from last time) and awareness that extends beyond single interactions. Studies of conversational agents in healthcare show that such personalization leads to higher user satisfaction and engagement. Users stick around longer in conversations that acknowledge their personal context and don't make them repeat themselves. The dialogue quality improves because the agent's responses are contextually appropriate and consistent over time.

Memory in AI doesn't just produce better task performance – it shapes how users emotionally relate to and trust the system. An AI agent capable of remembering and learning over time tends to feel more human and empathetic.

For example, if a virtual assistant consistently recalls a user's preferences (like their food allergies or schedule constraints), the user will trust its suggestions more and feel a greater sense of rapport, much as we trust friends who remember our likes and concerns. Continuity builds relationship: a chatbot that references a user's previous messages ("I remember you mentioned last week you were feeling anxious. Are things better now?") demonstrates care in a way, thereby encouraging the user to confide and engage further. This human-like quality increase users' trust in the AI.

In contrast, a lack of memory can make an agent seem robotic or inattentive, undermining trust. Users often grow frustrated or disengage when they have to repeat information or when the AI fails to "recall" context that it should know. It breaks the illusion of interacting with an intelligent partner and reminds the user that the agent is just a machine following scripts. Especially in applications like healthcare, counseling, or personal assistants, trust is paramount – and trust is reinforced when the AI shows it remembers past interactions accurately. By retaining history, AI agents can also avoid inconsistent or contradictory responses, further improving credibility. All these factors contribute to a more user-friendly experience. Users are more likely to continue using, relying on, and recommending an AI service that remembers them.

Technology: AI Agent Framework Refresher

At a conceptual level, an AI agent has six core components, modeled after the human mind. Here's a simple breakdown of each:

For a deeper technical exploration, see:

1. Perception: Converts raw data into formats the AI can process. Like human senses, it handles tasks like computer vision and natural language processing. Tools like Unstructured.io help convert formats like PDFs into data the AI can work with.

2. Working Memory: Think of this as the AI's mental workspace - temporary storage and active processing. Like how we juggle thoughts or calculations in our head, in AI this is similar to the context window in large language models, holding immediate inputs and retrieved information for processing.

3. Procedural Long-term Memory: This is where "how-to" knowledge lives. In AI, it includes the implicit knowledge built into the model during training, particularly post-training, like how to maintain response boundaries, follow instructions, reason through problems, and use tools.

4. Declarative Long-term Memory: Stores facts and events. In AI, this uses databases - think knowledge graphs for facts and vector databases for abstract data like image patterns.

5. Motor: Handles interactions with external systems. For AI, this means performing actions like sending emails or managing files, usually through APIs.

6. Orchestrator: Coordinates all the other components, managing:

   • Retrieval: Getting data from memory into working memory

   • Synthesis: Converting temporary data into permanent storage

   • Storage: Converting data into machine-processable formats

   • Planning: Organizing tasks and decisions to align with broader goals

Technology: Long Term Memory

While AI agents use different types of memory as shown above, let's focus on the challenge of declarative long-term memory - storing and retrieving information across conversations. When I first wrote about this topic 18 months ago, the working memory (context window) was quite limited - 4,000 to 8,000 tokens at most. The natural solution was external long-term memory using vector databases and improving how we orchestrated RAG. Since then, context windows have grown exponentially - Google's Gemini now handles up to 2 million tokens.

This expansion of working memory might seem to solve our problems - after all, we could theoretically fit 170 hours of conversation in Gemini's context window. But dumping everything into working memory fails for several reasons:

1. Just like humans can't perfectly recall hour-long conversations, raw context doesn't create useful memory

2. The quality of retrieval degrades with too much unfocused context

3. Processing large context windows remains computationally expensive

The core challenge in declarative long-term memory is providing the right context so the model can give the best answer for the user. This breaks down into three key problems:

• Synthesis: How do you summarize memories?

• Storage: How do you retain them efficiently?

• Retrieval: How do you recall the right information at the right time?

There are generally three approaches to building these systems:

1. Plain text

2. Vectorization

3. Synthesized chat logs in structured form (table or knowledge graph)

Each approach has trade-offs, and they can be combined for hybrid RAG. For simplicity, plain text isn't practical because searching through raw text is inefficient. Let's focus on vectors and structured data.

Vector Storage: Technically, this approach converts text into numerical vectors using embedding models. Each chunk of text becomes a long list of numbers (usually 1000+ dimensions) that represent its meaning. When you need to find relevant information, you calculate how similar these vectors are to each other - like measuring the distance between points in space.

It's like having a huge pile of clothes and using similarity to find what you need - you might search for "something red and soft" to find your favorite sweater. It's quick to set up but can get messy when you have lots of items. Vector search is great for finding semantic similarity but doesn't understand relationships between pieces of information.

Knowledge Graphs: Technically, this approach stores information as a network of nodes (entities) connected by edges (relationships). Each node and relationship can have properties that describe them. When you need information, you follow these connections to find what you're looking for.

It's like an organized closet where everything has its place - shirts with shirts, pants with pants, and you know which items go together. It takes more work to organize initially, but finding things is more systematic. Knowledge graphs excel at following explicit relationships but require more structure.

Hybrid Systems: The most sophisticated systems combine both approaches: use vector similarity to find relevant content quickly, then use graph relationships to understand context and connections. This gives you both semantic search and structured relationships, though it requires more engineering effort. For example, when searching for project information:

• Vector search finds semantically relevant documents

• Graph traversal shows how these connect to people, deadlines, and dependencies

• Together, you get both content and context

For a more technical comparison of vectors and knowledge graphs, see:

I lean toward knowledge graphs because they match how we think when we're being deliberate - in terms of categories, things, and relationships between them. This reflects what psychologists call System 2 thinking (slow, conscious reasoning). Though admittedly, most of our day runs on System 1 (fast, automatic responses). When I see my parents, I automatically greet and hug them. I don't consciously think "I see two people, these are my parents, parents deserve respect and love, therefore I should greet them." This aligns with industry practice that for 80% of cases, System 1-style processing (like vector search) works fine. For the 20% of cases needing more careful thinking, we need System 2-style processing (like knowledge graphs).

Recent research from Amazon and CMU offers valuable insights into how different memory approaches stack up. Their findings show that dumping everything into the context window (CoT LLM) performed much worse than any RAG system, despite Claude 3 Sonnet having a 200,000 token window. Graph-Only RAG was only slightly better than Text-Only RAG (vector RAG), showing about 5% improvement. Interestingly, agentic/self-reflective systems ReAct and Corrective RAG performed worse than expected. However, combining methods proved powerful - Text & Graph RAG and HYBGRAG (which adds self-reflection to Hybrid RAG) showed significant accuracy improvements.

For most applications, basic vector RAG works well enough. It's when organizations want to build exceptionally smart and personally tailored applications that they need structured data or knowledge graphs. While implementing knowledge graphs can be challenging, companies like Google and Glean have successfully deployed them to give them a competitive edge.

For a detailed exploration of knowledge graphs and how they fit into Google and Glean, see:

Another interesting study is done by Zep, one of the memory startups we'll discuss below. They compared their specialized memory system to the "dump everything in context" approach. Their system scored 18% higher while using only 1/10th of the processing time and 1/100th of the context tokens. For users, this means more relevant answers, faster. For developers, it means lower costs since they're not paying for extra input tokens.

Business

The memory layer is becoming a key focus among the most popular AI apps. OpenAI has been rolling out enhanced memory features that let ChatGPT store explicit memories and reference past conversations. This is a shift from their original approach where ChatGPT would start fresh with each conversation, though they did add basic memory capabilities in February 2024.

Similarly, Gemini incorporated memory features in November 2024. While OpenAI and Google are likely building their own memory systems (a technically challenging task), it's worth looking at startups offering memory as a service, particularly those implementing hybrid graph RAG. While any database could theoretically serve as memory, we'll focus on those using sophisticated hybrid approaches: Zep and Mem0. We'll also look at Letta, which treats working memory as a core feature in their agent framework, and Neo4j, the go-to graph database for AI products.

Other interesting companies and projects in the space but not profiled below: Cognee, WhyHow, Memary, Kuzu, Memgraph,

Zep (Zep 3K stars, Graphiti 2.1K stars)

• Background & Founding: Zep was founded in 2023 in San Francisco as part of Y Combinator’s Winter 2024 batch. It emerged to build a “memory layer” for LLM applications, allowing AI agents to retain and recall past interactions.

• Funding: The company has raised $3.3 million from Engineering Capital, Step Function, and founders & leaders at Vercel, Google, and several well-known AI companies.

• Founders: Zep was founded by Daniel Chalef (CEO) – an engineer who previously founded KnowledgeTree and held senior roles at SparkPost. Chalef’s background spans engineering, data science, and enterprise software, guiding Zep’s technical vision.

• Product & Technology: Zep offers a long-term memory service that developers can plug into their AI agents to store conversation history and retrieve relevant facts on the fly. Initially, Zep’s system extracted “facts” from chat logs and used a specialized RAG pipeline (semantic search + reranking) to surface relevant context. To overcome issues with purely semantic retrieval, Zep developed Graphiti – an open-source library for building temporal knowledge graphs from chat history. Graphiti dynamically captures entities and relationships from conversations (including changes over time), and supports hybrid search that combines vector similarity with graph traversal queries. This approach lets an AI agent recall context not just by semantic match, but also via connected knowledge (e.g. remembering that “X is Y’s manager” or how a user’s preferences changed). Zep’s memory API returns these relevant snippets or facts in milliseconds, without requiring the full chat history in each prompt.

• Differentiation: Zep stands out for its temporal knowledge graph approach. Instead of just using vector embeddings or key-value stores, Zep automatically maintains a structured graph that updates as conversations develop. This captures context shifts and relationships over time (like noting that "the user previously preferred X but now prefers Y") - nuances that simple vector search might miss. By combining graph-based memory with semantic search, Zep aims for more accurate and explainable context retrieval, especially for handling changing facts and historical context in ongoing conversations.

Mem0 (25k stars)

• Background & Founding: Mem0 was founded in 2023 and is based in San Francisco. The company is backed by Y Combinator and launched to tackle the stateless nature of LLMs by providing a persistent memory layer for AI applications.

• Funding: The company raised a seed round of about $500K in April 2024, led by Y Combinator.

• Founders: Taranjeet Singh (co-founder & CEO) and Deshraj Yadav (co-founder & CTO) lead the team. Singh’s background includes engineering and product stints at Khatabook (YC S18) and Paytm, as well as founding an AI tutoring app and co-creating the open-source EvalAI platform. Yadav was previously an AI Platform lead at Tesla Autopilot and is an AI/ML infrastructure expert who created EvalAI during his grad studies.

• Product & Technology: Mem0 provides an open-source memory layer that makes AI assistants stateful. It stores user interactions, preferences, and other context so that an AI agent can “remember” past sessions. Under the hood, Mem0 uses a hybrid datastore architecture combining three components: a vector store, a key–value store, and more recently a graph store to represent relationships between entities, . When an AI query comes in, Mem0’s engine automatically extracts important information from prior interactions and uses a blend of graph traversal, vector similarity search, and key-value lookups to fetch the most relevant memories. This ensures the LLM is injected with the right contextual info (e.g. past user queries, corrections, or personal facts) without the developer having to manually prime each prompt with all history. The design goal is to improve personalization and reduce repetition, while minimizing prompt size by externalizing memory storage. Mem0 is open source and has a cloud offering of managed mem0 with premium features. The open source project has garnered over 22,000 GitHub stars and 500,000+ downloads.

• Differentiation: Mem0 distinguishes itself through a unified hybrid-memory architecture and open-source approach. By natively combining graph, vector, and key-value stores, it handles both semantic and symbolic aspects of memory out-of-the-box. This differs from solutions that use only vectors or only graphs - Mem0 believes combining techniques leads to more personalized and cost-effective AI interactions.

Letta (15k stars)

• Background & Founding: Letta emerged from UC Berkeley’s AI research community in 2024. It was spun out of the Berkeley AI Research Lab (BAIR) and came out of stealth in September 2024. The founding team had been researching AI agent memory before forming Letta as a company.

• Funding: Letta raised a $10 million seed round in 2024, led by Felicis Ventures. The round also included participation from Sky9 Capital and Essence VC, as well as prominent angel investors such as Jeff Dean (Google DeepMind’s Chief Scientist) and the CEOs of Hugging Face, Runway, MotherDuck, dbt Labs, Anyscale, and Hex.

• Founders: Dr. Charles Packer and Dr. Sarah Wooders co-founded Letta. They met during their PhD research in the Sky Lab at UC Berkeley, where they worked under professors Ion Stoica and Joseph Gonzalez who now serve as advisors to Letta. Packer and Wooders are AI researchers who co-authored the MemGPT paper – which introduced the concept of self-editing memory for LLMs – and have deep expertise in machine learning systems.

• Product & Technology: Letta is building an end-to-end platform for AI agents with long-term memory. It has a popular open source project and is building a hosted service where developers can deploy stateful AI agents via an API. The platform is model-agnostic – developers can bring their own LLM while Letta handles the memory and agent logic. A key component is Letta’s Agent Development Environment (ADE), a web interface that lets developers design, debug, and monitor their agents’ reasoning steps and memory content. This ADE emphasizes white-box memory: developers can see exactly what information is stored and used at each step, and even edit the agent’s memory or prompts in real-time. Technically, Letta’s memory system builds on the MemGPT concept of self-editing memory: the LLM agent can write to and read from an external memory dynamically during a conversation. For example, after each user interaction, the agent can update its knowledge base (adding or modifying facts) which will persist into future sessions. By maintaining this stateful context, Letta-enabled agents aim to avoid resetting every conversation – they “remember” the user’s past instructions, preferences, or corrections.

• Differentiation: Letta stands out by focusing on agent-centric memory and developer tools, backed by academic research. While other memory layers act as infrastructure, Letta provides a complete agent platform - essentially a turnkey way to deploy AI agents that learn over time. This comprehensive approach means they handle issues like agent derailment, reliability, and prompt management alongside basic memory storage. Their focus is on making it easy for developers to create stateful AI agents, rather than just providing memory storage. This makes them complementary to lower-level storage solutions - they could potentially integrate with vector or graph databases from other providers.

Neo4j (14k stars)

• Background & Founding: Neo4j, Inc. is the oldest and most established company in this group, known as the pioneer of graph databases. It was founded in 2007 by Emil Eifrem (CEO), Johan Svensson, and Peter Neubauer. Neo4j’s early vision was to make data relationships a first-class citizen in databases.

• Funding: Over more than a decade, Neo4j has raised approximately $580 million across multiple venture rounds. Notably, it closed a $325M Series F in 2021 at a valuation above $2 billion, one of the largest investments in database history. Investors in Neo4j have included GV (Google Ventures), One Peak, Morgan Stanley Expansion Capital, Creandum, and others.

• Founders: Emil Eifrem, the CEO, is credited with sketched the first code of Neo4j and evangelizing the power of graph databases. Co-founder Johan Svensson has served in engineering leadership, and co-founder Peter Neubauer (who has since pursued other ventures) helped bring the database to market. The founding team’s background was largely in enterprise software and open-source development.

• Product & Technology: Neo4j is a high-performance graph database management system that stores data as nodes (entities) and edges (relationships) with properties. In the context of AI and memory layers, Neo4j serves as a platform to build and query knowledge graphs – structured representations of facts that an AI can draw upon. Developers use Neo4j’s Cypher query language (or its GraphQL and Python integrations) to query complex relationships efficiently. For example, one can retrieve a subgraph of all information related to a user’s query, enabling multi-hop reasoning (traversing connections between concepts). Neo4j ensures ACID-compliance and scalability for large knowledge graphs, which is important for enterprise deployments. Recently, Neo4j has actively positioned itself in the GenAI space by promoting GraphRAG (Graph-augmented RAG) architectures. The idea is to combine Neo4j with traditional vector-based retrieval: a vector search might find relevant documents, and Neo4j’s graph can then add context by linking those documents to related entities or facts.

• Differentiation: Neo4j's role in the memory layer space is as foundational technology rather than a specialized AI memory service. They stand out through maturity and capability - they're a proven platform with global enterprise adoption, suitable for production knowledge graphs with millions of nodes and relationships. Many other companies in this space use or integrate with graph databases like Neo4j under the hood. Unlike startups focused on chatbots or LLM agents, Neo4j is a general-purpose database adaptable to many uses (from fraud detection to recommendations, and now AI context storage). This means more integration work - organizations need to build their schemas and data pipelines - but it offers unmatched flexibility and powerful graph analytics through their Graph Data Science library.

Subscribe now

Curated reads

• Academic: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Bases

• Commercial: HP to Buy Parts of Humane, Maker of the Ai Pin, for $116 Million

• Social: Microsoft Study Finds AI Makes Human Cognition “Atrophied and Unprepared”

Subscribe now

Table of Contents

• Summary

• AI Trends Report

  • Section 1: Macroeconomics

  • Section 2: Public Markets

  • Section 3: Industry Deep Dives

  • Section 4: Private Markets

  • Section 5: Product & Models

  • Section 6: Scaling Laws

  • Section 7: AI Agent Opportunity

  • Section 8: 2025 Outlook

Summary

The story of AI in 2024 is of power law and agents. Power law because the monetary gains are concentrated in a handful of companies, even in public markets. There’s also an increasing agglomeration of capital among the few who can continue to continue the path of AI scaling laws.

Beginning with the big picture, generative AI or foundation models are the fastest adopted generational purpose technology (GPT). In 2 years, 40% of the population have used a generative AI product. It took personal computers over 15 years to get similar adoption. Smartphones was the latest GPT to get rapid adoption and it took 4 years to get to 40%.

Generative AI is also a powerful technology with major implications in how we live and work. I evaluated 16,000 tasks across 800 jobs to understand what is the automation potential of different jobs across the spectrum of generative AI technologies. Agents are the next frontier of automation and we should be ready for that.

The rapid adoption is reflected in stock market returns with the Generational AI Index returning 127% since the beginning of 2023, compared to 85% for NASDAQ and 58% for S&P. As you’ll see in the full report, 75% of the AI Index gains come from just 7 companies.

Most of the gains are also concentrated in the AI value chain, notably the chips and energy supplied to data centers. Data centers are the “currency” to play in the generative AI game. If scaling laws continue, there will continue to to be chip and energy shortages in the near to medium term.

In the private markets, there’s also herding (i.e. concentration) among venture funds in a handful of companies. Different venture pursued different strategies, reflecting different risk profiles. If the generative AI boom continues, then Sequoia, a16z, and Lightspeed stand to make generational returns for their LPs and themselves.

The pace at which the AI industry advanced in 2024 made 2023 look pale in comparison. Even now as of writing in December 13, OpenAI has 5 more days in its Shipmas series. Google just released Gemini 2.0 two days ago. Microsoft also released Phi-4.

But the entire AI boom seems to centrally hinge around scaling laws continuing. There are lots of chatter about whether we’ve faced a scaling “wall”. We have not but we are facing a more complex scaling dynamic. There’s still plenty of room to scale. In the diagram below, I sketch out what scaling laws look like today.

Even if scaling laws continue, there is the question of whether it is worth it to continue pursuing it. o1 likely costed billions to train. Is a $10 billion model worth training? What about $100 billion? These are local GDP level numbers. One way to frame the ROI question is to look at how generative AI can augment and automate jobs. There’s ~$40 trillion of wages paid each year, of which ~$11 trillion are paid in the US. A $100 billion model still sounds extreme, but at least not insanely irrational. In the full report, I provide a framework of how to think about the best opportunities for AI agents and the 12 areas with the most potential.

AI Trends 2024 Report

If you prefer a PDF copy: link

If you’d like to view it online with media & links: link.

Subscribe now

Section 1: Macroeconomics

Section 2: Public Markets

Section 3: Industry Deep Dives

Section 4: Private Markets

Section 5: Product & Models

Section 6: Scaling Laws

Section 7: AI Agent Opportunity

Section 8: 2025 Outlook

AI agents are all the hype today, so you’ve probably already read about them. But in case you are interested in understanding the inner workings, check out my previous essay How to Create a Mind.

In this article, we will take a business and investor perspective on AI agents, rather than a product and engineering one. With all the buzz about how “AI agents will revolutionize everything,” it is essential to have a framework to identify the areas with the most potential. To build this framework, I analyzed the automation potential of approximately 16,000 tasks across 800 jobs.

Motivation

When ChatGPT was released in December 2022, people found it to be remarkable. It was a fun tool, quirky, yet in many ways useful. Few considered how it might replace anyone in their jobs. The perception shifted when GPT-4 was released in March 2023, with benchmarks showing it is better than the most humans on exams that resonate with us — SAT, GRE, LSAT, AP.

While the benchmarks are useful, they do not directly translate to how well AI can perform in our jobs. A more helpful framework focuses on specific tasks — the jobs-to-be-done — in our roles. By looking at specific tasks, we can gain more nuanced insights into how much of our jobs is automatable by AI. That also indicates how much opportunity there is for the companies building AI startups (or how replaceable we may be). If AI can do our jobs, then the market opportunity is much larger than the $500 billion US software market — it is 20 times larger.

To precisely identify where the market opportunity is, I analyzed ~16,000 tasks across ~800 jobs from the US Bureau of Labor Statistics. Much credit is due to the tireless government researchers who have profiled all those tasks and jobs, even differentiating between computer programmers, software developers, and web developers.

Analysis

I used GPT-4o to evaluate the tasks (because o1 was too expensive) using a rubric. Below is a simplified version of it:

• No-Automation Exposure: AI cannot perform any aspect of this task.

• Low Automation Exposure: AI could complete 0-50% of the task components at high quality.

• Moderate Automation Exposure: AI could complete 50-90% of task components at high quality.

• High Automation Exposure: AI could complete 90-100% of task components at high quality.

• Full Automation Exposure: AI can complete all task aspects at high quality without oversight.

While others have run similar task studies, many did not specify what an “AI system” is. In my analysis, I distinguished the automation potential of base models, copilots, agents, and robotic systems. This differentiation allows us to be more precise about what we are measuring and where the opportunities are.

Findings and learnings

The incremental automation from base models to copilots is modest. Copilots, broadly — not just Microsoft’s offering — are impressive, but many feel they are not as transformative as initially marketed. Marc Benioff’s tweet captures this sentiment well. The next significant shift in AI is agents that autonomously execute tasks across tools and contexts, as in real jobs. AI agents are poised to be the next big thing — not just by hype, but supported by data.

The Agent Framework

Automation potential is one dimension in understanding how fast companies can achieve ROI from AI agents. Computer programmers are among the most automatable roles, which explains why coding tools like GitHub Copilot and Anysphere/Cursor are gaining traction quickly. Another key dimension is total wages paid, reflecting market opportunity. While proofreaders are among the most automatable roles, there are only around 5,500 in the U.S., earning about $280 million combined. In contrast, roughly 1.7 million developers collectively earn close to $230 billion.

Mapping 800 jobs across these two dimensions reveals roles with the highest potential for AI agent automation. The chart below shows the frontier of jobs ripe of agentic automation. It is no surprise many of these roles align with high-growth startup areas, where VCs are offering top valuations.

• Software developers: Poolside at $3 billion

• Lawyers: Harvey at $1.5 billion

• Customer Service Representatives: Sierra at $4.5 billion

I am still digging through the data but have already found some interesting patterns that I will be publishing in December as part of the 3rd annual Business of AI report. Subscribe if you want to get a copy of it.

Subscribe now

Curated reads:

• Academic: Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

• Commercial: OpenAI Shifts Strategy as Rate of ‘GPT’ AI Improvements Slows

• Social: Donald Trump’s EPA pick wants to ‘make America the AI capital of the world’

One of the most impactful applications of generative AI is a knowledge copilot for humans. A knowledge copilot intelligently reasons through contextual information, providing a delightful and practical user experience. I first wrote about this topic a year ago, let's explore the industry and technological advances since then.

A knowledge copilot is a user’s thought partner, optimized for retrieving information from a knowledge base, reasoning through context, and synthesizing responses

Reasoning: With the launch of Open AI's o1 model last September, it's shown that models can reason as well as they can summarize and rewrite. GPT-4o can't correctly count the number of letters 'r' in the word "strawberry," a reflection of the state of AI models before o1. While it is arguable that what the o1 model does is not the same logical reasoning that humans do, it does a good job outperforming human experts at PhD-level science questions.

User Experience: The first copilots were pure chatbots, limited to text exchanges. Rapid improvements in copilot UX have occurred since then. Both OpenAI’s ChatGPT and Anthropic’s Claude now display code, apps, and documents—much like collaborating with another person today. ChatGPT’s advanced voice mode introduced an entirely new modality. With Anthropic’s new computer API, soon we'll have copilots controlling our screens.

Context: For general conversations, the relevant context might be the knowledge stored within the model's weights. For knowledge-intensive uses, the relevant context resides in private documents (i.e., knowledge base) and user information. Retrieval-Augmented Generation (RAG) is the only way to insert the ever-changing and growing knowledge base. Individually, we create new documents daily. In an enterprise, employees create thousands of new documents each day. Fine-tuning or dumping the entire knowledge base isn't practical.

Over the past year, copilot deployments have grown from pilots to enterprise-wide. With a much larger knowledge base to sift through, RAG systems struggle to pull out the right information. It is notable that whenever RAG gets brought up in industry events I’ve attended, the vocal engineers express their frustrations about how it's not pulling the right information. The quiet ones wince and commiserate.

As knowledge bases grow, using vector similarity becomes more fragile. The same words appear in multiple unrelated documents, making it difficult to identify the most contextually relevant documents for the copilot. Knowledge graph-augmented RAG (graph RAG) has emerged as a trend to improve accuracy. This approach was popularized by Microsoft’s GraphRAG paper and open-source project. New graph database open-source projects like Kuzu and the resurgence of Neo4j further point towards the trend.

Why context is the most important now

Using the wrong information as context leads to inaccurate or misleading copilot responses. That is the number one risk, and increasingly so, for organizations adopting AI. Inaccuracy erodes trust—the trust that gives us comfort to not have to constantly review copilot responses, and the trust that deters the adoption of software that many well-funded companies are building.

While better model (perhaps o2 in six months) and new user experiences (like having Claude guide my mom on how to find the settings menu on her phone) are always welcome, the bottleneck for knowledge copilots is setting the right context.

Knowledge graphs can help set the context

Knowledge graphs help because they are computer-readable representations of information that reflect the way humans think: entities (persons, things, events), relationships (spouse, author, attendee), and properties (age, date). The way we think is by entities and their relationships, not documents (unless you’re a lawyer): Kenn->writes->Generational. Generational->gains->Subscribers. Subscribers->seek->Knowledge. Knowledge->enhances->Learning. Learning->leads_to->Happiness. Happiness->increases_with->Subscribers.

Subscribe now

While that is a tongue-in-cheek nudge for you to subscribe, it reflects the logical context that AI models can reference to reason through.

However, incorporating knowledge graphs into RAG is challenging. The first few steps are already complicated, requiring engineers to figure out:

• How to parse raw documents (e.g., what about tables and drawings inside PDFs)?

• How to chunk the parsed raw documents? Limit it to 100 tokens for all chunks? Or do we create rules to accommodate paragraph endings, sections, etc.?

• How to augment & transform each chunk (Anthropic just introduced contextualization by pre-pending the previous chunk’s summary as context for the next chunk)?

• What embedding model to use?

Indexing knowledge graphs is more complicated because it requires computationally intensive processing of the entire knowledge base. It involves intelligently rewriting the knowledge base into another format (graph generation) and then also having to figure out how to make it readable by the computer (graph indexing). There are several steps involved in generating a knowledge graph and different ways to index it, each with its pros and cons.

Knowledge Graph Generation:

1. Document Preprocessing: Preparing raw documents by cleaning and formatting them to ensure that the text is in a usable state for extracting knowledge.

2. Ontology and Schema Definition: Defining the structure and types of relationships that will be captured in the knowledge graph, essentially setting the rules for how information is organized.

3. Entity Extraction: Identifying key entities (such as people, places, or concepts) from the text to populate the nodes of the graph.

4. Relation Extraction: Extracting relationships between entities (such as "works at" or "is located in") to form the edges of the graph.

5. Attribute Extraction: Identifying additional details or attributes about entities, like an entity’s age or location, which enhance the richness of the graph.

6. Deduplication and Resolution: Merging duplicate entities (e.g., “AI” and “Artificial Intelligence”) and resolving conflicts to ensure the graph is coherent.

7. Updating: As new information becomes available, the knowledge graph needs to be continuously updated to stay current. This involves re-running processes like entity extraction, relation extraction, and deduplication to integrate new data without disrupting the existing structure. The updated data is then re-indexed, ensuring that both the graph generation and indexing remain synchronized, enabling accurate and up-to-date access by computers.

Knowledge Graph Indexing:

1. Property Graph: A graph where each node and edge has associated properties, making it easy to search for specific attributes or relationships. It's flexible and supports rich metadata but can become complex when scaling, especially with large datasets and multiple property relationships. Example: (User:John)-[:FRIENDS_WITH]->(User:Jane)

2. RDF Triples: A structure used to represent data in subject-predicate-object form, enabling a formal semantic structure. It offers a standardized way to represent relationships, ideal for web-scale data, but its rigid structure can make modeling complex, real-world scenarios more challenging. Example: Kenn->writes->Generational

3. Vectors from Graph Embeddings: Translating graph nodes into vectors allows for efficient operations like similarity searches. It excels in complex tasks like clustering, though it sacrifices the explicit relationship semantics found in the original graph. Example: A node for User:John is converted into a vector [0.24, 0.87, 0.56]

4. Text and Text Embeddings: Textual data is transformed into machine-readable format for easy search and comparison within the graph. While it integrates well with structured data, it can lose contextual nuances, leading to reduced accuracy in some cases. Example: The text "John is a software engineer"

With LLMs, knowledge graph generation can be streamlined by using the same LLM for each step. But it still requires substantial engineering work to make the graph generation pipeline robust. User experience will also define the schema, which can be difficult if the application is broad.

Incorporating graphs into RAG is worth it

With knowledge graphs, there are four types of index data (there’s more, but we’ll stick with four) that can be used to retrieve the context. Given how complicated it can be to incorporate graphs, especially in a hybrid approach that combines multiple indexes, is it worth it?

It is. One of the more recent popular graph RAG frameworks is called LightRAG, which incorporates both knowledge graphs and semantics through text vector retrieval. The architecture below illustrates how it works.

Researchers compared LightRAG to other RAG systems—Naive RAG (basic RAG that engineers often dislike), RQ-RAG (breaking down user queries into manageable sub-queries), HyDE (writing hypothetical responses to pull similar documents in the knowledge base), and GraphRAG—using different datasets from areas like agriculture, computer science, legal texts, and mixed topics. The datasets were large, containing hundreds of thousands to millions of words. To evaluate the models, the researchers looked at how well they answered questions based on completeness, variety, and usefulness.

LightRAG came out significantly ahead of the others, especially in providing complete and varied answers, thanks to its ability to combine detailed and broad retrieval through graphs and vectors. It was particularly effective with more complex and larger datasets, like legal documents, where it was better at creating rich, well-connected responses. Based on the results, a more sophisticated RAG is worth it.

Semantic/Document RAG

• Naive RAG: Naive RAG segments texts into chunks and stores them in a vector database. It retrieves chunks based on similarity to the input query. This method is simple and fast but lacks the depth to understand relationships between the retrieved pieces, making it less effective for complex topics.

• RQ-RAG: RQ-RAG breaks complex queries into multiple sub-queries, making each search more targeted and precise. By decomposing queries, it can retrieve information more accurately, especially when dealing with multi-faceted or ambiguous questions.

• HyDE: HyDE generates a hypothetical document based on the query to guide the retrieval process. This approach allows the system to find information that best fits the imagined answer, making it particularly useful when the original query is vague or lacks specific keywords.

Knowledge graph RAG

• GraphRAG (from the Microsoft paper): GraphRAG uses graphs to represent entities as nodes and relationships as edges, forming communities of related information. It retrieves data by traversing these communities, which is ideal for understanding complex interdependencies. Unlike LightRAG, GraphRAG doesn’t combine graph and vector retrieval, focusing solely on graph traversal for deeper contextual understanding.

Semantic-Document + knowledge graph (hybrid) RAG

• LightRAG: LightRAG integrates graph structures and vector representations, allowing for both precise, low-level retrieval (specific entities) and broad, high-level retrieval (general themes). This combination makes it versatile and ensures comprehensive, context-aware responses. LightRAG can also quickly adapt to new information, without the need for full system rebuilds. Unlike GraphRAG, LightRAG’s dual system of graph and vector retrieval makes it efficient and capable of handling diverse queries.

The complexities of RAG is an opportunity

However, knowledge graphs are just one component of a RAG system. LightRAG is already a relatively simple hybrid RAG strategy, but it remains complicated to productionize, especially as part of a larger RAG system that incorporates additional types index data into a complex retrieval system combining multiple metrics and steps.

Combine that with parsing different file formats and types, figuring out the right chunk size, transforming the prompt, and choosing the embedding model—you have a system that can serve as a technical moat for companies. It also means that building an enterprise-grade system can be a standalone business.

Two examples of this are Google (of course) for consumer search and Glean for enterprise search. While we don’t have details on how Glean’s systems work, we do have some for Google Search. Google’s Knowledge Graph contains 1,600 billion facts (relationships, details about entities, etc.) about 54 billion entities. It is used both to rank search results and to power a UX called knowledge panels. As you can see, the entire system is complex, with the knowledge graph forming a small yet important part of it.

Subscribe now

Curated reads

Academic: Graph Retrieval-Augmented Generation: A Survey

Social: Former OpenAI Researcher Says the Company Broke Copyright Law

Commercial: Demo of Claude autonomously researching and orchestrating a sunrise hike by the Golden Gate bridge

Subscribe now

In this interview, I speak with Spellbook’s Scott Stevenson (CEO and co-founder) about building AI products. Their product uses AI to review and draft contracts and they have always been at the forefront of adopting the latest models to legal work, such as launching the first agentic product for lawyers, Spellbook Associate. Spellbook is used by over 2,600 law firms, professional services, & in-house teams including the likes of Addleshaw Goddard (Global Law 200), Nestle (Fortune 100), and BDO (top 5 auditing firm). Spellbook recently raised a $20 million series A from Inovia Capital and strategic investor Thomson Reuters.

Key learnings

1. Embrace the "Skepticism Window": Capitalize on the period when a new AI technology faces skepticism but shows promise. Scott's experience with agentic AI mirrors the early days of GPT models, suggesting that this skepticism often precedes rapid adoption and improvement.

2. Focus on Hard Sub-Problems: In complex AI systems like Spellbook Associate, solving difficult sub-problems (e.g., manipulating a 100-page legal document) is crucial before tackling higher-level tasks. This approach ensures a solid foundation for more advanced features.

3. Leverage Existing Workflows: Spellbook's success partly comes from integrating with lawyers' familiar processes, like track changes. This reduces adoption friction and addresses potential concerns about AI errors.

4. Sequence Feature Expansion: Start with core workflows and gradually broaden functionality. This strategy allows for mastering specific use cases before expanding, as seen in Spellbook's evolution from single to multi-document workflows.

5. Ride the Foundation Model Wave: Build on top of rapidly improving foundation models to benefit from their advancements. Scott emphasizes the importance of capturing upside from new model releases like GPT-4 and o1.

6. Value-Based Pricing in High-Stakes Industries: In fields like law where professional time is extremely valuable, flat-fee pricing can be more appealing than usage-based models. This approach simplifies billing and emphasizes the product's value proposition.

7. Balance AI Aggressiveness with Cost Control: Encourage aggressive AI usage in development while monitoring for excessive resource consumption. Scott mentions allowing developers to use AI freely but intervening when necessary to prevent runaway costs.

The Origins of Spellbook

Kenn: Welcome Scott! Maybe to start, I'd love to hear the history of Spellbook leading up to the launch of Spellbook Associate. Along that journey, I'm curious how the problems you're trying to solve have changed and, correspondingly, how the products you built changed as well over time.

Scott: Sure, I'll start with the problem, which has always been the same. We started, technically, almost 5 plus years ago. There are three co-founders, and we all had our own stories. I was an engineer, and I had my own small business. One day I got a legal bill that took half the cash out of our bank account. That was a moment for me where I thought, "Wow, this is a huge problem." Everyone has that experience of getting their first legal bill and thinking, "Holy cow, what did I just pay for?"

We set out to solve the inefficiency of legal work, that sort of value-to-cost equation not feeling good. My co-founder Daniel was a lawyer, so he saw the problem from the other side. He went to law school for years, racked up student loans, and then he graduated and started practicing. He realized he actually hated the work, which involved many nights until midnight with ten Word documents on the screen, trying to make sense of them and copy-pasting between them all.

That was the problem we're trying to solve: basically, the legal document drudgery required primarily in business and transactional work. This includes contracts, employment agreements, setting up companies and minute books, doing VC transactions, and all these things that create so much friction. That problem has always been the same.

Evolution of the Product

Scott: When we started, generative AI wasn't around. We started with a templating tool, basically a glorified templating tool where a lawyer could build a template for an employment agreement and then use the same one for multiple clients. We sold that to around 100 firms, and we actually had over a hundred landing pages. We sold many variations; we had literally over a hundred variations of our messaging and tweaks on our product to sell it and angle it in different ways. We even had one that was direct to businesses.

It did okay but never really worked that well because legal work is ultimately unstructured. Lawyers can't easily put their work into a template. Everything is pretty bespoke a lot of the time, and often a lawyer is working on someone else's paper, someone else's contract. You're almost never working on a brand new contract.

As an engineer, I saw GitHub Copilot, and this was probably early 2022 or late 2021. I thought, "Whoa, this thing is amazing." The early version of GitHub Copilot would auto-complete your code based on what you've written. What amazed me about it is you can use it in any situation without any setup. You didn't have to create templates in advance or anything like that. It just worked out of the box all the time, and it's in your existing workflow. For us, that was an "Aha!" moment. I thought, "This is what lawyers need."

We launched the public version one at the end of summer 2022. This was before ChatGPT, before any other generative AI company launched to lawyers. Within three months, we had more revenue than our past three years. It was just this explosive moment. We had 30,000 lawyers on our waitlist.

Kenn: It's really interesting to hear the journey. I'm curious, when you launched v1, was it mostly around helping generate or complete the next set of text similar to what GitHub Copilot does, or make edits? But then you've evolved it to also have benchmarks, broadly integrating the lawyer's workflow. I guess I'm curious, was it just a function of what your customers were telling you to build, and how much of that is also driven by what you've seen developing in the foundation models becoming smarter?

Scott: Yeah, so I think a bunch of things have driven our new features. Let me share my screen. I think it's helpful for you to see. So yeah, we started with autocomplete. Then we originally had a bunch of spells that would find issues for you. That kind of became our second thing, issue spotting. Here's an example of our negotiate feature: I can give it an agreement, and it will tell me things that I might want to improve with the agreement. It can even personalize suggestions, so it learns over time the sort of things that I like to negotiate for.

This became possible with GPT-4. This sort of fine-grained document analysis wasn't really doable or very good with GPT-3. When GPT-4 came around, this became more possible. This was something our customers really wanted. We just listen to our customers, see what they want, and yeah, the models advanced very quickly.

We do leverage a lot of off-the-shelf models. We do many things internally as well, but our general attitude is that foundation models are moving so fast. Most startups should build on top of them most of the time so that you capture the upside when something like GPT-4 is launched.

The Birth of Spellbook Associate

Kenn: What led to the creation of Associate? You already have a successful product sitting on top of a Word application. What led to building Associate, which automates much more of the work? Is that something you saw as an evolution of what the work will be for lawyers? Or is this something you've heard lawyers want, which is interesting too if that is the case, because that decreases their hourly billings?

Scott: I think since we launched Spellbook, the very first request we got was, "Can it work on multiple documents?" So even in 2022, we had customers asking. Spellbook works in Word, really on a single document at a time. We basically pushed the single document workflows as far as we could go. We built out just a ton of things that you can do: summarize changes, draft new clauses, draft full documents. We built out as much as we thought we could do at the single document level.

There are all these multi-document workflows that lawyers want to be able to do, too. You can't really do that in the Word sidebar, so the big motivation for us is getting to a multi-document workspace.

I think also we just have faith that agentic AI is the next big thing. It feels so similar to what GPT-3 felt like, and GPT-2 even, where everyone was skeptical at first. And then all of a sudden, it actually works really well. Agents feel like they've gone through that exact same thing: immense skepticism, a lot of "Oh, your examples are cherry-picked," and then, "Oh, actually, it's working 80% of the time. Oh, actually, it's working 90% of the time. Oh, actually, it's working 95% of the time."

We like to lean into that skepticism when something is new and it shows promise, and people aren't really ready to accept it yet. I think that's when you can be early. You know, those are all the signs of being early to a market. For us, we try to be really fast. We want to be two years ahead of the market, and two years ahead of all of our competitors is our goal. Moving fast is one of our competitive advantages as a startup.

Product Evolution and User Feedback

Kenn: When I saw the Associate product, it's like you're starting to build into how you break down a task or request into subtasks and then execute those clearly, because in a multi-document setting, my wife would ask the junior associate to update the name of the company, but it's actually spread across many docs, and they often miss one doc to update. That's really painful. So I really like how you set up the Associate product.

I'm curious, since it's early access, if there have been any surprises in terms of how you've seen your users use it, and any pitfalls that you've learned so far, being ahead of the market.

Scott: I think the main thing we've learned is that users want consistency. Yeah, it's cool to be able to do anything, but users want a sort of consistency. Focusing on a couple of really specific, popular use cases has been really good for our user experience.

I think of it as more of a sequencing. I think it's like you should nail your core workflows at first, and then once you have those nailed, you should broaden it and broaden it. I don't think there's one set point. I think it's really a sequencing of wanting to start and make sure you have really solid use cases out of the gate, and then you want to expand from there.

Another thing I think about is, if you think about Spellbook Associate, there are really kind of two layers of planning. There's the top level of, "Oh, I'm going to look at this document," or "I'm going to revise that document." And then there's once it gets to a document, actually conducting all the changes it needs to make. That second part of actually going to the document and making the changes is still an immensely hard problem that agents help solve.

We have found it's really good to focus on these hard steps. It's almost like there's an agent within an agent, and there's no point having a high-level agent if you can't do those steps really well. The step for us is taking a long 100-page legal document with instructions and manipulating the document. That's really, really hard.

Kenn: I want to shift gears a little bit into user perception and user adoption. I think Spellbook is one of the most well-adopted legal tech AI tools, and lawyers typically are the ones that are most sensitive to any errors due to the liability nature of their job. How have you seen them adopt these tools? Is that a real concern you've seen from your users, and how do you mitigate it? Even a simple typo in the date is a material thing when you draft contracts.

Scott: It hasn't been a problem for us, and that's because all of our workflows are designed to use track changes and to have this user step where they're accepting or rejecting a change. The same with Associate – it's changing the docs, but it's using track changes. You still have to go through and accept or reject the changes.

We're never altering something without the diff. Someone, when they saw our Associate video on Twitter, said, "Oh, this is what makes agents make sense." They're like, "This is also why Cursor works, and this is also why Devin works, because there actually is a really good, known UX already for accepting and reviewing changes."

That's why this works. Because lawyers do this all day already – they go through changes and accept and reject them. And that's what developers do all day – they go through pull requests and accept and reject them or review their own changes. So that's what makes it work well and what really puts the onus on the user to make sure that what they're reviewing... We've never had any kind of issues or worries about it because of that.

Kenn: Interesting. Now that you mentioned track changes, it sounds exactly like committing code to a repo. You have the lineage, you have the diff. It mirrors a lot with engineering. How does this apply to the Associate product? Does the user have to go through each doc and approve the changes? Is that the workflow you're envisioning for Associate?

Scott: We actually have two versions. We have one that lives on your desktop and opens up Word and uses that Spellbook UI that I showed you, and you have a sidebar. Then we have another fully browser-based cloud version that just does everything in the background and hands you back a doc. Then you basically have to open the doc and go through it normally.

We're getting feedback on whether they like that pure cloud experience or if they want this on their desktop talking to Word. There are a bunch of trade-offs there that we have to think about. But both of those workflows still require someone to go in and accept the changes.

Pricing and Cost Considerations

Kenn: Being sensitive to time, I know you have a hard stop. I want to shift to the last question, which is not often talked about – and feel free to keep it confidential if you don't want to talk about it – how do you think about pricing and cost for Associate? I think it's been a top concern, at least for my day job at Smartsheet. We're always worried about AI cost blowing up. How do you price a product like Associate? And how do you think about costing for it?

Scott: It's a great question, and it's very difficult because the costs are so dynamic. We've decided to go with basically a flat fee. We have a basic product and a premium tier that has more features. We actually don't do any usage-based billing at all today. We might at some point.

We just try to make sure that we're taking into account margins and usage. Generally, if you think about the value of a lawyer's time, lawyers are billing $400 to $800 to $1000 an hour somewhere in that range. So it doesn't take a lot for us to make up for that for them. We can make a pretty good case with flat-fee billing.

We've thought about usage billing all the time, but it's just hard because things are so dynamic. Generally, we tell our dev team to use AI very aggressively and not to worry about cost. That's been totally fine. There's been one or two moments when I'm like, "Hey, you're auto-running this huge job every time someone opens a doc – we can't do that." I will monitor the charts. But generally, the margins have been good.

That's simplified things for us and for our customers. It is something we think about, too. It's really hard because the costs are so variable. But our goal is to provide so much value to our customers that hopefully, our value-add is enough that we don't have to worry about it too much.

Kenn: Okay, that sounds good. That's a great point about lawyer per hour being so high – the value is high. Scott, I super appreciate the time.

Scott: Amazing, great. Thanks so much, Kenn. It was great to chat.

Thank you for reading! If you enjoyed this interview, who should we interview next? And if you haven’t subscribed, subscribe below to get the latest posts.

Subscribe now

In the past few months, Microsoft, Amazon, and Google have engaged in quasi-acquisitions of foundation model companies Inflection, Adept, and Character.ai, respectively. Instead of a traditional acquisition involving stock purchases, these Big Tech firms opted to hire key AI talent from these startups and pay the investors license fees. In this article, I analyze these deals through a corporate development lens, offering details and insights beyond what has already been covered in the media:

• Why Microsoft, Amazon and Google pursued those deals

• Why they resorted to an unusual deal structure

• Why they paid so much money

• Who else could acquire and be acquired

• Why quasi-acquisitions won’t become a trend

Subscribe now

Overview of the deals

Why acquire?

Speed is the primary reason why Microsoft, Google, and Amazon spent hundreds of millions (perhaps billions) on these deals. They are rushing to become market leaders because the rewards are high, competition is intense, and there is a data flywheel effect that accumulates over time.

Reward for winning is high. The revenue potential of the generative AI market is enormous, and the pace of growth is unprecedented. For example, in June 2024, Sam Altman shared in a company-wide meeting that OpenAI’s annualized revenue is $3.4 billion—121 times more than the $28 million revenue the company generated in 2022, making it the fastest-growing company in history. It was estimated that 55% of this revenue comes from individuals (ChatGPT), and the other 45% from businesses, suggesting that both B2B and B2C markets are lucrative.

Competition is intense. With such a large prize at stake, competition is fierce. All the big tech companies are spending billions to build data centers to support AI models. Microsoft’s exclusive relationship with OpenAI drives both Azure spend and access to OpenAI’s latest tech, which is integrated into Microsoft’s other products. Amazon has a deep relationship with Anthropic through a $4 billion investment. If AI cloud spend goes to Microsoft, it might also attract other cloud spending. Google’s search dominance is at risk if it doesn’t build an AI layer on top of its search results, and its cloud prospects might also suffer. Google committed to investing $2 billion in Anthropic.

In addition to these investments, the Big Tech companies have been investing approximately $15 billion each in cloud infrastructure every quarter, and they expect this to increase over the next several quarters.

• Alphabet/Google (7/23/2024): “Our reported Capex in the first quarter was $12 billion. Looking ahead, we continue to expect quarterly Capex throughout the year to be roughly at or above the Q1 Capex of $12 billion. We continue to invest in designing and building robust and efficient infrastructure to support our efforts in AI given the many opportunities we see ahead.”

• Microsoft (7/30/2024): “Capex, including financing leases, were $19 billion. Cloud and AI-related spend represents nearly all of our total Capex. We expect Capex to increase on a sequential basis, given our cloud and AI demand as well as existing AI capacity constraints.”

• Meta (7/31/2024): “We anticipate our full year 2024 Capex will be in the range of $37 billion to $40 billion. We currently expect significant Capex growth in 2025 as we invest to support our AI research and our product development efforts. Llama 3 is already competitive with the most advanced models and we’re already starting to work on Llama 4. The amount of compute needed to train Llama 4 will likely be almost 10 times more than what we used to train Llama 3.

• Amazon (8/1/2024): “For the first half of the year, Capex was $30.5 billion. Looking ahead to the rest of 2024, we expect capex to be higher in the second half of the year. The majority of the spend will be to support the growing need for AWS infrastructure as we continue to see strong demand in both generative AI and our non-generative AI workflows.”

Data flywheel. As discussed in my past essay on data moats, there is a data flywheel in AI products. User data helps build better models and products. Researchers estimate that LLMs will have been trained on all public internet data by as early as 2026. Proprietary user data becomes even more important in building better models.

Now that we know why the big companies made these acquisitions, there are two questions: why the quasi-acquisition structure, and why such large amounts of money?

Why not a standard acquisition structure?

The structure of the quasi-acquisitions consists of two parts:

1. Talent Acquisition: The big tech companies hire the founders and key employees of these startups, selectively acquiring the talent they want. Microsoft hired over 80% of Inflection AI’s team, Amazon hired 66% of Adept’s team, and Google hired a smaller fraction of Character AI’s team.

2. Non-exclusive technology licensing: Instead of acquiring the entire company, the Big Tech firms obtain non-exclusive licenses to the startups' models. The license fees are then used to pay investors a negotiated return over time.

Why not just buy the shares like a regular acquisition?

This is to avoid further regulatory scrutiny, which would be triggered by an HSR filing. An HSR filing (Hart-Scott-Rodino Act) is a pre-merger filing to both the FTC (Federal Trade Commission) and DOJ (Department of Justice) and is required when it meets certain thresholds such as a $120 million deal size. After filing, the agencies will have 30 days to do an initial review. During this period, the agencies can approve the transaction or request for more information. Companies must substantially comply with the data requests. That takes weeks to prepare. Agencies can take weeks to months more to review the new information, which could lead them to challenge the deal. Given the strategic importance of speed and certainty to close, Big Tech firms (like all firms) prefer to avoid this scrutiny, especially when the FTC announced in January 2024 that it had already launched inquiries into AI deals done by them. While the HSR Act applies to acquisitions of assets and voting securities, the definition of assets conveniently excludes non-exclusive licenses and people.

Why not just give large bonuses to talent and avoid paying license fees?

Investors would block the acquisition. If key talent is hired, the startup’s value plummets. The license fees give investors a satisfactory return on their investments. Since the founders are also owners, license fees serve as exits for them. For example, in the Microsoft-Inflection deal, Series A investors will get 1.5 times their $225M investment, and Series B investors will get 1.1 times their $1.3B investment. From the date of the funding announcement to the date of the acquisition announcement, Series A investors achieve a 25% IRR, and Series B investors, 14%. Google is more generous with a 2.5x return. Adept investors get their money back. It is likely that these returns correlate with consumer product presence:

• Adept is targeting enterprises and has not publicly released a product yet

• In March 2024, Inflection reportedly had 1 million daily active users

• In July 2024, Character reportly had 6 million daily active users

Why that much money?

If the acquired companies were struggling, as media reports suggest, how did the teams at Microsoft, Amazon, and Google justify the high “license fees” they paid? It’s not because of the models themselves. They obtained non-exclusive licenses to models that will be outdated in six months. With a gutted team and license fees going to investors, the acquired companies have no resources to continue building frontier models. So it’s a mix of other factors:

• Talent: The primary driver behind these deals is the acquisition of top AI talent. These startups have assembled teams of the best researchers and engineers, and Big Tech companies are willing to pay a premium to bring them in-house. This talent acquisition accelerates their own AI development efforts and potentially prevents competitors from gaining an edge.

• Dataset: While the models might not be the primary focus, the datasets these startups have accumulated can be invaluable. These datasets, often fine-tuned for specific tasks or domains, can provide a significant advantage in training and refining the acquiring company's models. It was reported that Amazon will gain access to Adept's dataset, which likely includes valuable proprietary data related to web browsing, productivity tools, and other user interactions as part of the deal. Unlike public web text, user interaction data is hard to obtain.

• GPU access: Acquiring startups that have secured significant GPU capacity can help Big Tech companies overcome potential bottlenecks and accelerate their AI initiatives. Inflection, for example, had secured access to a cluster of 22,000 NVIDIA H100 GPUs, one of the largest in the world, which further incentivized Microsoft's acquisition.

• Users: Users provide a testing ground for new AI features and products, and their engagement data can further enhance the acquiring company's models. Additionally, a loyal user base can translate into potential revenue streams and market share. Character AI, with monthly traffic rivaling Gemini, presented an attractive target for Google in this regard.

  

• VC approval: Venture capitalists who have invested in these startups expect a return on their investment. The high price tags associated with these quasi-acquisitions ensure that investors are adequately compensated, preventing potential roadblocks.

Who else could be quasi-acquisition targets?

According to The Information, there are seven AI startups that are struggling and may look for buyers: Reka (raised $56 million, valued $300 million), Imbue (raised $232 million, valued >$1 billion), Cohere (raised $980 million, valued $5.5 billion), Pika Labs (raised $141 million, valued $470 million), Ideogram (raised $97 million, valued $580 million), AI21 Labs (raised $336 million, valuation $1.4 billion), and Essential AI (raised $65 million)

But who could potentially buy these companies? Acquiring model-centric companies only makes sense for cloud service providers. Given the additional scrutiny the recent deals have attracted, it is unlikely they would pursue another high-profile acquisition. IBM and Oracle, the 4th and 5th largest US cloud providers, are unlikely to acquire talent to start building their own frontier models. Two application-layer companies, Salesforce and Adobe, could be potential buyers. Salesforce has cloud ambitions and a broad surface of applications to deploy AI talent. Similarly, Adobe also has a wide range of applications, including its family of Firefly models. But neither Salesforce nor Adobe has antitrust concerns regarding AI. They would likely buy the stocks directly or pursue a typical acquihire.

Interesting reads:

• Seven generative AI startups that may look for a buyer

• Google’s Gemini Live is here

• MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

Subscribe now

In this interview, I speak with PolyAI’s Shawn Wen (CTO and co-founder) and Devidas Desai (SVP of Product) about building enterprise AI products and how communications & AI have evolved with generative AI. PolyAI develops enterprise conversational assistants that engage in natural conversations with customers to resolve their issues. These assistants understand customers regardless of their phrasing or manner of speaking. While voice interaction has gained recent popularity due to GPT-4o, PolyAI has been at the forefront of this technology since 2017. The company recently raised a $50M Series C round led by Hedosophia and NVentures (NVIDIA's venture arm), bringing their total funding to $120M from prominent investors including Khosla Ventures, Georgian, Point72 Ventures, and others.

Subscribe now

Table of Contents

• Key learnings

• Introduction and journey to PolyAI

• Evolution of AI and impact of GPT-4

• Changes in communication, customer service, and PolyAI’s approach

• Building AI assistants: technical aspects, enterprise concerns, and advice

• Future plans for PolyAI

Key learnings

1. Enterprise AI adoption requires balancing multiple stakeholders: While customer experience heads traditionally were the main decision-makers, generative AI brings security, IT, branding, and legal teams into the conversation. This complicates the sales process but also creates opportunities for education and addressing diverse concerns.

2. Enterprises often hold AI to higher standards than humans: As Devidas noted, AI assistants may be expected to stay rigidly on-topic in ways that human agents are not. This creates challenges in making AI seem natural while still meeting strict enterprise requirements.

3. Customization and control are critical: Some enterprises have extremely specific restrictions, like forbidding an AI from stating basic facts unrelated to their business. AI systems need to be highly configurable to meet these idiosyncratic needs.

4. Practical solutions trump theoretical ideals: Shawn emphasized the importance of being practical rather than trying to build the perfect technical solution. Time-to-market, packaging, marketing strategy, and sales execution are as important as the underlying technology.

5. Layered safeguards are necessary: PolyAI uses multiple layers of protection, from general content filters to project-specific customizations. This allows tailoring the AI's behavior to different levels of enterprise risk tolerance.

6. Iterative testing with clients is crucial: Finding edge cases and potential issues requires extensive testing, both internally and with clients. This process accumulates knowledge over time that can be applied to future projects.

7. Transparency about limitations builds trust: Being open about the current state of generative AI technology and its limitations, while showing a clear roadmap for addressing concerns, helps enterprises feel more comfortable adopting these solutions.

8. Self-serve capabilities are becoming important: With generative AI, some enterprises want more control in maintaining or even building their own assistants. Providing tools for this can be a differentiator.

Introduction and Journey to PolyAI

Kenn: Thank you both for joining me today. I'm excited to talk with you about PolyAI. To start, could you each share how you found your way to the company?

Shawn: I'm Shawn, co-founder and CTO of PolyAI. I was there at the very beginning with Nikola and Eddy when we co-founded the company in November 2017, about 7 years ago now. The three of us met at the University of Cambridge. I actually met Eddy during my undergrad, as we're both Taiwanese, so our history goes back about 15 years. Eddy and I went to Cambridge to pursue our PhDs under the same supervisor, and Nikola happened to be in the same year as well.

We were actually the last batch of Steve Young's students. Steve Young is a pioneer in speech recognition who created the HTK open source toolkit. It was initially sold to Microsoft in the early 1980s, then open-sourced, with Microsoft continuing to maintain it. It was a traditional n-gram based and hidden Markov model toolkit for speech recognition. Since then, Steve moved on to building dialogue systems and doing research in that area.

I think the reason we're doing what we're doing now is very much because of Steve, both on the research side and the entrepreneurship side. Steve actually started three companies and sold them to Microsoft, Apple, and Google respectively. So because of the research topics we worked on with him, and his entrepreneurship history, it pointed us towards wanting to do something together.

Before graduation, we all went to big tech companies - I went to Google, Nikola to Apple, and Eddy to Facebook. But we started to feel it was a good time to do something together. So we all came back to the UK and started the business here. That's a brief history of the company's founding.

Kenn: That is cool. I did not know that about the founding team’s history with Steve Young. Thank you for sharing that background. Devidas, how about you?

Devidas: My story isn't as cool as Shawn's, but I've been in the collaboration and communication space since I started in product management, and the field really fascinates me. Joining PolyAI allowed me to stay close to communication, but leverage conversational AI to improve customer and caller experiences.  Of the companies I met with, it sounds cliché, but honestly the quality of people I met with helped me make the decision to join PolyAI. The strength of the technology team was just super impressive, which is what essentially brought me here. And it's fantastic to be here.

Evolution of AI and Impact of GPT-4

Kenn: That's great context. Shawn, you mentioned studying for your PhD. Machine learning was a very different world back then, from both an academic and industry perspective. How has the AI field changed from your perspective?

Shawn: It's a very interesting question. I started to get into AI and machine learning in my third year of undergrad. I actually started in electrical engineering, then gradually shifted towards AI because I found it more interesting. I began with speech recognition, which involved a lot of signal processing. As I mentioned before, it was hidden Markov models and n-gram based models. These approaches are actually very Bayesian, and that continued to be the case when I went to Cambridge, which is traditionally a big Bayesian camp. You have people like Zoubin Ghahramani there doing a lot of Bayesian-based approaches, graphical models, Gaussian processes, and so on. It's very mathematically heavy and intellectually interesting.

That kind of model was actually popular for quite some time, and deep learning was just something in the background because the compute wasn't there. But many of the algorithms and optimization techniques we're using these days were already there. I think the shift started around 2013-2014 in the speech recognition field. In those days, if you had an improvement of word error rate by 0.5%, you could publish a paper. But Microsoft trained a deep neural network model and achieved a 4% reduction in word error rate, which was huge because for several years no one had managed to achieve that. It blew everyone out of the water, a bit like OpenAI with ChatGPT these days, but people don't remember that because it was more in the academic world.

Afterwards, you see a bunch of people starting to train bigger neural networks and applying them to different use cases - first ImageNet, then people moved on to different problems like NLP. Deep learning has progressed a lot since then, and a lot of the benefit is because the compute power caught up. People started to realize it's very difficult to figure out exact, beautiful mathematical equations to solve real-world messy problems. So people started engineering these large machines and piling data into them, and magically, you get something really powerful.

I think the reason large language models surprised a lot of people is because previously, if you were training small language models, it was really about garbage in, garbage out. It wasn't really that helpful. And magically, at some point, once the model reaches a certain scale and the data reaches a certain scale, it reaches that tipping point and boom, it just starts to work really well. I think this caught a lot of people by surprise.

Kenn: GPT-4o is a big leap from the early days of neural networks that you sketched out. How has GPT-4o affected PolyAI or your thinking about building products in this space?

Shawn: GPT-4o is super exciting, and this kind of multimodal model is at the forefront of technology innovation. I think it's going to make things potentially even easier for everyone. We're super excited about it. In fact, we're actually in touch with OpenAI to make sure that we can get private access once they finally release it. It's unfortunate that they recently announced they have to push it back for another month.

We're very excited about it, but at the same time, I think getting these models right is probably quite tricky. That's probably the reason they're pushing back the timeline. Even for the initial announcement, they're only going to release it to a few trusted partners to test it. If you think that hallucination in GPT's text version is already dangerous, you can totally imagine that hallucination in the voice channel is a completely new world.

We haven't actually gotten our hands on testing the system yet. But we do see that for building consumer-facing products, which I think is what they're trying to do, they will make a lot of huge progress. I think for enterprise applications, I wouldn't think that you would actually gain so much quick adoption yet, just because the end-to-end modality makes it even harder to understand what's actually going on. If GPT-4o suddenly starts screaming at you in the voice channel, you don't exactly know what's happening or what input is causing the trouble. I think that would actually make enterprises worry. So we're super excited, but we're also cautiously thinking about what would be the best way to use it because we know that a lot of our enterprise customers would have concerns.

Kenn: It's a good distinction. As a consumer, I'm super excited about getting my hands on GPT-4o. Have you had prospective customers or customers reach out to you asking if you can build something like GPT-4o for them?

Shawn: We haven't heard that yet. I think people still don't quite understand what it means, especially the end-to-end modality of it, because they haven't actually gained access to it yet. Once they gain access to it, I do imagine that people will start to ask about it. But there's no way for them to try it yet. And I think in the documentation or the announcement, unless you're a techie, which the majority of our clients aren't because they're actually operating contact centers, they don't quite get what's coming yet.

Kenn: It's interesting that the models continue to progress at a very fast pace. How do you think about the rapid progress of these models? And how do you future-proof yourselves, or your tech stack? Because I'd imagine that multimodal capabilities will change your tech stack quite a bit. But also from a more strategic perspective, how do you think about the progress of these models and how that relates to the problems you're solving for your customers?

Shawn: I think the models will continue to evolve, and they're going to evolve at a very rapid speed because now there are more and more companies jumping into this kind of large model training and multimodality. Therefore, I think these will continue to progress super fast. As a company, our philosophy is always that these are all new tools that we should incorporate into our product or offering. Not one single technology could be for everyone. I think it's very important to actually find the right technology and tooling for the right customer.

Our goal has always been to continue to invest and innovate on our own technical stack, but at the same time, we should keep an open mind to incorporate any of the latest technology into our offering. Because at the end of the day, our goal is to make the best, human-like voice assistants that enterprises would be comfortable adopting and using, and that callers would be happy with as well. So to us, it's really about how we can continue to incorporate these new technologies fast enough into our product offering.

Devidas: I would just add that from a product standpoint, we are building this as a true platform. Each of the components that power the experience, whether it's the listening piece, the reasoning piece, the speaking piece, or the biasing piece, we can offer the best in breed. Some of the best-in-breed pieces could just be PolyAI tech, which is proprietary, and a combination of that just provides an amazing caller experience because we are the thought leaders when it comes to bringing this together and building proprietary tech on the available technology so that we are thinking about the caller experience.

From a product strategy standpoint, we know what we are good at. We know what are some of the areas that we have an upper hand on, and we want to make sure that we continue to build on that proprietary tech, make it a part of the platform to solve for amazing caller experiences.

Changes in Communication, Customer Service, and PolyAI's Approach

Kenn: Devidas, you've been a veteran of the communications space. How has communications changed, either from a personal or business perspective? And how have developments in AI changed or shaped your view of communications from a product perspective?

Devidas: Those are both excellent questions. In general, I think the changes have been very sticky because the technological changes have actually also changed user behavior. From a communications and collaboration standpoint, previously, communications were very simple and single-channeled. We're talking about either an email communication, an IM, an SMS, or a phone call, and they all had different jobs. That's exactly what users were expecting. If I want to email you, I'm going to email you, and I would expect your email back. The lines were quite thick.

Then we started moving towards modalities coming closer, where you started having the concept of messages having attachments or mentioning people in your emails, and the lines started getting blurrier. This is where you bring multiple modalities together. Then we moved over to web conferencing, which led to UCaaS (Unified Communications as a Service) products, bringing messaging, video, and phone together. It almost doesn't matter where the user is or what the other side is using - you have all of the modalities, and you can switch between them to talk to people. It's about the speed of communication.

Now we're having AI-driven communications. All of these changes are really impacting user behavior and changing user habits. People are getting more and more used to spending less time on communication channels and making them super efficient. Previously, users were expecting simple, basic communication channels. Then we started moving towards multi-channel communications. Then users started expecting 24/7 instant responses, which brings AI into the picture because you don't necessarily want to power those communications with humans all the time.

Now we're in this phase where people are expecting 24/7 instant responses that are personalized. You don't necessarily want to go over the same details over and over again. When you reach out to a rep or an AI, you expect the other side to know who you are, what your preferences are, and just pick up the conversation where you left off.

It's been fascinating to see how the tech has evolved based on users' expectations and vice versa, and how some of these changes are very sticky. AI is really helping provide quality communications to users. Separately, it's helping businesses allocate human capacity to the highest ROI customer profiles. The way we see it, AI is not replacing anything. It's actually making the spend on customer service, customer interactions, communications, or collaboration in general more efficient. You're getting a higher ROI because you're now in a position to determine which conversations you want bots to handle versus humans, and how personalized you want those exchanges to be.

I'm pretty sure we'll soon see a world where it really doesn't matter as a user or caller who I'm speaking with, whether it's an AI-powered bot or a human. As long as my experience is great and it does the job, I really don't care whether I'm talking to a human, because I don't want to spend too much time on a phone talking to a customer service professional anyway.

Kenn: That's a good segue to finally talking about what PolyAI is at a high level and what problem you’re solving. Could you elaborate on that?

Shawn: PolyAI has been quite consistent in terms of what we're building. During our PhDs, we were doing research on conversational systems, and the company is really an extension of that. We continue to build voice assistants, primarily over the voice channel. The same technology can be used for building text-based assistants as well, but we've been focusing on developing a really good voice assistant because putting all these technologies together requires a lot of focus.

Our primary market is in enterprise contact centers. A lot of these are communications that happen that normal people might not pay attention to. But whenever you have an issue and you call a contact center, especially in the Western world, the quality of service is often not good enough. This is because of labor shortages, increasing labor costs, and the fact that there's still a requirement to staff enough people in a contact center, but fewer people are willing to do that kind of job. Or if they do want to do it, it's often a temporary job - you come in, train for 3 months, and 6 months later, you're out.

There's also the challenge of effective communication between callers and offshore contact centers. When you have an issue with your bank and you call, you might speak with someone from a different part of the world who may not share the same cultural context or communication style. This can sometimes lead to misunderstandings or difficulties in fully grasping the caller's situation and concerns. As a result, many customers find it challenging to connect on a personal level during these interactions. This is one of the reasons why many companies initially moved their contact centers offshore, but are now considering bringing them back to their home countries in response to customer feedback. However, the underlying issues persist - service quality is declining, there's a shortage of people willing to take on these roles, and contact centers continue to face significant challenges.

We think technology can solve this kind of problem. In a contact center, voice calls actually need a dedicated agent listening to the phone call and speaking to the user on the phone. While chat or digital channels like emails are much easier because you can parallelize the work, voice is especially challenging for contact centers. But a lot of people still want to call when they have urgent issues or when they've tried to do something online but couldn't.

PolyAI is trying to help solve that problem. We want to place AI agents into contact centers. The AI agent should be the best tier 1 agent that the enterprise can have, representing their brand. They need to communicate naturally and sound like what the brand wants them to sound like. We're not intending to just automate the entire workflow because naturally, there are certain kinds of use cases that AI is not good enough for yet. Like a lot of emotional cases, some cases that require lots of empathy, and some use cases that require very complicated transactions. These are things where tier 2 human agents will have a very strong capability to help, and that's where AI can step out and hand the call back to the contact center.

A lot of people have been thinking about AI taking people's jobs. I think partially that's true, but it's also not true because there are already problems with contact centers - they just cannot staff enough people. And arguably, training AI is also not free. You actually need to supply it with a lot of data, and there should be a lot of humans in the loop as well. As new technology continues to evolve, there will always be new jobs being created, even as some jobs go away.

Kenn: That's interesting. I grew up in the Philippines, and I know turnover rates in contact centers there are 50% every year.

Shawn: Even in the Philippines, yes. It just tells you how tough the whole business is.

Kenn: I think it would be helpful to sketch out who the different users and personas involved are. As an end user, I just talk to someone. But I know in the enterprise, you have corporations that may or may not outsource it to someone, and they may have their own requirements. Then you also have the end users. How would you sketch out the problems and how you're solving for those different personas?

Devidas: Maybe I can talk about this a bit. I'll answer this in two ways. There's been a historical persona who we've been selling to, and then I feel like generative AI is definitely changing the landscape in terms of who's involved with respect to the buying process and the maintenance of the conversational assistant.

Historically, and this is still the case, customer experience heads and heads of contact centers are still the primary people we sell to as far as enterprises are concerned. I feel like generative AI is changing that landscape a bit and is bringing in a few more personas that are involved in the buying process. Because with generative AI, you're no longer just solving for customer experience, cost efficiency, or improvement of CSAT. With generative AI, you have to worry about how safe the solution is. So that brings security and IT into the mix.

With generative AI, you also want to make sure that, especially for a voice assistant, the voice is on brand and you are appropriately representing the brand that the assistant is taking calls for. So that brings in someone like a CMO or head of branding into the mix as well. It's been a really interesting shift where we are seeing more people coming to the table with respect to making decisions, and it's fascinating. Some of these interactions are very exciting in terms of us either educating them or answering good questions.

As Shawn touched upon, one additional persona that we're seeing come into the mix is that there are teams and jobs being created for every conversational assistant that is deployed or sold or built for a particular brand. There are people responsible for monitoring the calls, making sure that the assistant is on point, and reviewing calls to see what could be improved. They either make those improvements themselves, or if they don't want to be hands-on, they work with us to make those improvements. This alludes to the point that Shawn was just making, which is that it's partially going to make some jobs redundant, but at the same time, it's introducing new jobs and new skill sets that haven't been in the industry for a while. And that's a shift that we're seeing which is here to stay.

From a persona standpoint, those are the ones that we're seeing. One of the other trends that we see is that with generative AI, it is becoming a bit simpler and easier for customers to actively maintain their own assistants. In some cases, they can even build their own assistants, which was previously quite tough with intent-based models. We're seeing a lot of demand for that as well, which is, "Hey, I want to self-serve. I want to self-care for my own assistant because I know my business the best." And that's totally valid. We are able to provide for that, and we are seeing that shift as well. So it's been overall a fascinating experience. Generative AI has definitely changed the landscape with respect to conversational AI.

Kenn: That's really interesting. I didn't think about the other personas now getting involved in customer service agents as well. It seems like there's been a significant shift.

Devidas: It's still early days, but I think we're seeing enough of it to know that this is going to be a very logical trend going forward.

Building AI Assistants: Technical Aspects, Enterprise Concerns, and Advice

Kenn: You touched on something interesting about voice assistants. Before, with any customer experience assistants, you had to script every workflow. But now, I don't know how your team is building it, but I'm curious how much you can share. It seems like you're leaving it more to the AI system, which needs to know more about the company and the policies, but you also have to prevent it from hallucinating. So there's less and less control. Can you share how you're building it and controlling it to make sure that it's on brand, on policy, and aligned with the business?

Shawn: That's a very good question. This has been something we've been working really hard on since generative AI and large language models came along. If you try to build these kinds of voice assistants, you can actually build something quite impressive quite quickly. But it doesn't really do certain things well. For example, if you ask ChatGPT to make a restaurant reservation, it would actually pretend that it's making an API call in the background and tell you it's been done. So it hallucinates a lot.

I think GPT is pretty good at pattern matching. How we think about large language models is just to incorporate them as one of the building blocks into building the voice assistant. It's a super powerful building block. What it can do really well is pattern matching. So if you provide some information in the prompt and then the task is to ask GPT to pattern match what users say to what is included in the prompt, it can usually do it quite well. And that's usually how we handle FAQ questions from a large source of knowledge base.

When you start to build more complicated flows that require several steps of transactions, then there will be some design elements involved. You also need to build some guardrails on top of it. What we do is basically build a programmatic approach overlaying on top of GPT. Think about it like this: if you're just prompting GPT to do certain things, it's equivalent to handing GPT 100 pages of contact center instructions and saying, "Hey, read it and then interact with the user directly." If you were newly onboarded as a person, you're not going to read all these manuals before you jump into a call, and you're going to hallucinate yourself because you just want to survive in the conversation. That's what GPT does when you don't give it proper training or plans.

What we effectively do is say, "I'm not going to just give you 100 pages of these instructions and guidelines. I'm going to group them into different categories for you." For example, if the user is asking about a payment issue, you go into this particular payment issue flow. You fetch all these instructions or manuals, and those manuals are also paginated for you. So that means if a user asks about this question, the first step is you need to ask about these 5 different questions, and then you flip to the next page, then you answer another 3, and so on. By paginating these instructions and giving it a step-by-step progression into a conversation, you're kind of guiding GPT. It's more of a copilot kind of design we're doing here, rather than relying on GPT to just go on autopilot.

Kenn: That's really interesting. Let's say I'm a head of customer experience and I go to PolyAI. What do prospective customers usually ask you about? How do you describe the product to them, and what does it take to get set up? And what concerns do you usually hear from customers, particularly as they think about deploying at enterprise scale?

Shawn: From the hallucination aspect, it's not even about hallucination they're worried about. Some of these enterprises are super strict in terms of what the bot can say and cannot say. For example, there's this telco company we work with that doesn't even allow the model to say who their CEO is, even though it's a customer service bot for that particular company. They also don't allow the bot to say what the capital of France is. It's not allowed to say Paris because it's considered "out of scope." So that's the level of scrutiny you'll probably run into with enterprises. It depends on the enterprise, but these are the extreme cases we've seen so far.

Devidas: Sometimes you can say that AI assistants are held to a higher bar than human assistants. For example, if I press a human four times to tell me what's 2 plus 2, the human assistant could totally be like, "Alright, it's 4. Now what can I help you with?" And that's totally okay, but it probably wouldn't be okay if an AI assistant did that. So sometimes the bar is much higher for AI compared to humans.

To extend Shawn's point, some of this is quite synonymous with generative AI. When you talk about generative AI, you start thinking about safety and whether the assistant is on point. Typically, the three dimensions that we see our customers having concerns about are: Is the assistant providing accurate responses? Is the assistant detecting any threats if someone's trying to change its behavior, and how is it managing that threat? And lastly, how is the AI treating my customer's information? Because you end up sharing PII (Personally Identifiable Information) in these conversations in order to help the caller, and how is that being managed?

At PolyAI, we take these concerns very seriously. If you see our roadmap, there are clear items addressing each one of these concerns, and we're actually on a good path with that.

Kenn: Interesting. To the extent you can share, how do you red team or test against these concerns? For example, making sure the voice assistant doesn't say who the president of France is?

Shawn: That's a problem that everyone will have to face. As we've gotten very deep into selling to enterprises, we've developed several layers of safeguards. The first layer is that we actually put in a content filter. This filter is basically the highest level, filtering out anything that is dangerous, biased, sexist, or otherwise harmful content. The model would be triggered if any of this content is included, and the system or assistant would basically just refuse to answer or have a particular behavior. Some of our clients want the call to be handed off back to a human agent in this scenario.

Then going downwards, you have these very specific requirements about what you shouldn't share outside of what you're supposed to do in the contact center scope. We have an automatic evaluation framework that allows our developers or designers to write stories about particular conversations. Every time we're about to launch a new system, we run all those examples against the system to make sure that all the examples pass. That's another way to safeguard it.

One of the major challenges is how do you actually find those examples of what you're not supposed to do. In that particular case, it requires years of accumulated relations. It requires several projects where users do a lot of extensive testing. You need to give it to your clients to do a lot of extensive testing as well. And then you have to decide a boundary where you put those examples in.

So we have different layers of safety guardrails. One is that hallucination and harmful information should be excluded. The second layer is that you are allowed to say freely what is not harmful. But then you have another layer that you are only allowed to answer questions related to the contact center. And then the most customized level is that you are only allowed to answer specifically for that particular project or account. We don't have any clients at that level yet, but there are definitely some customizations we'll have to do for some future clients, I imagine.

Kenn: This has been great. We've touched on a lot of enterprise concerns. Is there anything else you think we haven't discussed about what enterprises are looking for? How would you advise people trying to build AI products for the enterprise? What are some of the lessons you've learned in your journey building PolyAI?

Devidas: From a product standpoint, the trend we're seeing from enterprises is that almost every enterprise champion we've interviewed wants to adopt generative AI, but they're a bit nervous about it. That nervousness stems from a range of concerns. It ranges from "I believe in it, I want to get this done in my company. How can you help me convince the business that this is a good thing?" all the way to "How do you take care of safety? How do you take care of prompt injection? How do you take care of hate speech?"

Essentially, that's exactly why I said a lot of this is education for our customers and just being very transparent about how we are solving for some of those problems. From a solutioning standpoint, it ranges from us addressing the concerns or giving our champions enough ammunition to feel good about the fact that this makes absolute sense, and they're going to put their name behind it and push for it within the organization.

What is working well for us is just being open and transparent about how we are solving for some of these concerns. Customers understand that generative AI is fast-moving, there isn't a foolproof solution yet. It's about trying to stay ahead of the curve, trying to constantly evolve, and then showcase data on what the assistant is doing well, where it could improve, and then have tools in place to keep that improvement going on an ongoing basis. That's how they feel pretty good, strong, and safe about adopting the solution.

Shawn: From the technical founder's point of view, I think you just need to be super practical. Everyone wants to build the next Google or Facebook. Everyone wants to build a platform. But in reality, it's probably not going to happen that way just because if it's actually that much of a market, then the big players would jump in and they would already take the market. You're never going to be able to win that battle because they have way too many resources.

I think we just need to be practical. Technical founders like myself sometimes are not very practical. You always want to solve the theoretical best problem, like what is the best revenue model, like a search engine. If you can actually build a business model that can print money, that would be the best. Everyone wants to build that, but only a couple of people manage to do it.

What we learned through the journey is that we just need to be practical about go-to-market. It's not just about technology. It's about the product, how you package it together. It's about the marketing strategy. It's about the sales executions as well. You just need to find the right way to sell the right product to the right people. And that requires a lot of learning and a lot of trying over a long period.

Kenn: That's good advice. In San Francisco, I attend all these AI events, and it's mostly about the technology and building prototypes around it. Less about the go-to-market side of it.

Future Plans for PolyAI

Kenn: Before we end the interview, congratulations on the huge $50M series C round and $500M valuation. What's next for PolyAI? What are each of you excited about for the future? And do you have any call to action for people reading this interview?

Devidas: I think we're going to keep things simple. We want to keep our product strategy living and breathing. We're going to spend a lot of time on product discovery, keeping our eyes on the competition, and accordingly evaluate our strategy on an ongoing basis. We're going to keep the focus high. We're going to use the funding round to address some of the gaps in the right skill set areas. But all in all, we're going to keep it simple. We're going to focus, have a clear product direction. We're going to differentiate on our strengths, and we'll continue to invest in that. We're going to continue to believe in and increase our amazing team. We have a fantastic team that can make anything happen, and we're going to continue to support them and keep addressing the gaps wherever necessary.

Shawn: For me, it's about continuing to expand PolyAI's reach. Even 6-7 years down the road, we're still not taking enough calls yet. I just want the systems to take more calls, to be able to solve more real-world problems that callers and contact centers are facing. Because this kind of pain is real. Every time I actually phone into a doctor’s office, I hope there will be some sort of system that can actually just help me get through what I want rather than waiting on the phone for 30 minutes, someone picks it up, doesn't understand what I want, transfers me to someone else, I wait for another 15 minutes, and so on. I think those kinds of experiences for humans in this new world should be in the past. People shouldn't spend their time phoning their doctors, their banks. They should just have a system that can understand them and send them to the right place.

Kenn: That’s awesome. And with that, let’s wrap. Thank you both for your time. This was a great conversation. 

Subscribe now

I made a point to learn about the hardware aspects of AI this year. Historically, I've focused on the software side, but I realized that hardware is the major constraint for generative AI today. Currently, there are not enough servers in data centers to train the next frontier model, and the processors in existing computers and portable devices are not powerful enough to run these models. However, jumping straight into the world of semiconductors and devices was overwhelming. The supply chain is complex, and most books focus either on geopolitics or the electrical engineering aspects of wafers and circuits. I was specifically looking for a product & business overview.

I ended up with a hodgepodge of materials of varying quality and depth.

So, I gave up on old-school research and turned to ChatGPT. With ChatGPT, I could ask about anything, wherever my mind took me. When I encountered jargon like RISC (reduced instruction set computer), I could get an immediate explanation instead of parsing through Google results. Instead of listening to podcasts and watching videos, I talked to ChatGPT. That wasn't the first time I used the voice capabilities, but it was my first time having a lengthy conversation about a complex topic with ChatGPT. The transformative experience was strolling around San Francisco while talking to ChatGPT. I had a research collaborator in my pocket that I could bring anywhere. I didn’t have to be glued to a screen. All it takes is a few taps to start talking to an AI assistant that knows more about semiconductors than 99% of the world.

When OpenAI demoed GPT-4o, it showcased another subtle yet significant improvement in user experience: the ability to interrupt ChatGPT and redirect the conversation by talking. OpenAI models are overly chatty. So when I’m talking to ChatGPT while my phone is in my pocket, I have to either listen to the entire response or pull out my phone to interrupt it manually. This might seem like a mundane quality-of-life upgrade, but it's incredibly more natural. It feels more human. After spending hours talking to ChatGPT, the potential of personal AI devices dawned on me. It is not a conceptual idea anymore but a lived one.

Types of personal AI devices

Personal AI devices are pocket-sized pieces of hardware through which people primarily interact with an AI assistant. Desktops, laptops, and AR/VR headsets don’t make the cut. Being pocket-sized is important because it allows the user to carry the device anywhere. This aspect of being pocket-sized is what made the iPhone iconic: Your life in your pocket.

The opportunity is immense: The iPhone launched the mobile era and generated $91B in sales for Apple last year. With the smartphone market valued at $485B annually, investors are eagerly hunting for startups aiming to create the iPhone of the GenAI era. More than $250 million has been invested in the makers of the distinctly neon orange Rabbit R1 ($199) and the Apple-esque Humane Pin ($699 + $24/mo subscription + $100 for a shiny chrome finish).

Despite getting awful reviews, it is laudable that both companies attempted to create new form factors. I am looking forward to seeing more ideas. However, I think we will likely see upgrades to the devices that we already have today: smartphones and wearables like in-ear headphones, glasses, and watches.

1. Smartphones have an established form factor that can fit enough memory and computing power for GenAI models. They are the most essential piece of personal technology today — we carry them everywhere to take pictures, call loved ones, reply to work emails, and binge-watch videos.

   • What’s new in this category: large foldable phones such as Samsung Fold and Pixel Fold are becoming more popular. Users get a larger screen, and manufacturers can pack in more battery and processing power.

2. Earphones are interesting because audio is central to communication. Until we get brain-computer interfaces, audio is the most efficient way to communicate. While we can use gestures and wave our hands, it’s inefficient and weird. The tiny form factor of in-earbuds makes it impossible to fit enough battery and processing power to make them viable AI devices. While over-ear headphones can pack more, they are not pocket-sized.

   • What’s new in this category: Companies are experimenting with new designs such as Google-spinout Iyo's One buds, which are noticeably larger to fit in a larger battery, ten microphones, and standalone internet connectivity.

3. Smart glasses are interesting because they capture what our eyes see. Vision is fundamental because sometimes it is easier to communicate with images than words. This is more so for people whose language is not the primary language the models are trained on, such as myself, who grew up speaking Tagalog. With flowers blooming in San Francisco summer, I recently passed by flowers I wanted to identify. Instead of attempting to describe it — lamp-shaped orange flowers with hairy buds in San Francisco — I sent a picture to ChatGPT, which identified it as Orange Clock Vine. Using the textual description, I got California Poppy instead. A picture is worth a thousand words.

   • What’s new in this category: Meta is the leader in this space with the Ray-Ban smart glasses and is reportedly exploring designs combining both earphones and cameras into one device. Google, which was way ahead of its time with Google Glasses 11 years ago, is also developing a new device today.

4. Smart watches are not interesting because they are only good for tracking biometrics. Outside of that, they are primarily used as an extension of the smartphone, so they are unlikely to be personal AI devices.

   • What’s new in this category: more sensors.

What will be the personal AI device that everyone uses? I bet that it will be smartphones, with wearables relying on them as the processing unit. We already see this pattern today with our earphones and watches paired to smartphones via Bluetooth. Wearables augment what we can do with a smartphone, but wearables are useless without a smartphone.

What are GenAI smartphones?

While AI has already been part of our daily smartphone interactions, native GenAI experiences are new. Smartphones today need an upgrade to become personal AI devices or what industry analysts coin as GenAI/AI smartphones. An AI smartphone is defined as having:

1. A GenAI assistant integrated with phone controls and OS to interact with mobile apps.

2. Capability to process on-device foundation models and funnel complex workloads to cloud-based models. There are a couple of reasons why on-device AI matters:

   • Local processing of sensitive data: Processing data locally can keep data private. This is important for apps that handle sensitive data, such as messaging apps with end-to-end encryption.

   • Offline access: Users can access AI features even without an internet connection. This is useful for applications that need to work offline or with spotty connectivity.

   • Cost savings: Developers can reduce inference costs by offloading execution to consumer hardware, ideally leading to lower mobile app prices.

3. Hardware capable of delivering the features above. IDC defines this as a phone capable of >30 TOPS (trillion operations per second).

According to market research firm Counterpoint, of the 1.2 billion smartphones shipped in 2023, less than 1% met the definition of an AI smartphone. 2024 will be the year of AI phones, with vendors launching their first AI phone models. Samsung launched the Galaxy S24 in January, and Google and Apple are expected to release theirs in the coming months. Supply chains are rapidly being reconfigured to ramp up production of AI phones, which are expected to comprise 43% of smartphone shipments by 2027

Who is positioned to win the race?

While high-end specs are the minimum requirement to play in this space, winning requires a deep ecosystem of developers, applications, and other personal devices. Mapping these factors against the key players, Apple and Google are best positioned to win the space. However, their paths to winning differ.

Apple is the leading smartphone maker in the US, with over 50% market share. It also has a wide install base of Apple TV and Mac computers — macOS holds 20% of the laptop and desktop market. Many are looking towards WWDC 2024, when Apple will share its GenAI plans and OpenAI partnership. We may not yet see the next iPhone then, but in a few more months, Apple is expected to launch the first “Ai”Phone. From a business perspective, these will drive customers to upgrade sooner and Apple could also add an Apple Intelligence service fee.

While Apple will win with its vertically integrated ecosystem, Google will win via software and ecosystem lock-in. Its line of Pixel phones has less than a percent of the global market and ~5% of the US market. But Google's play is software. Android holds ~70% global OS market share. During their recent I/O conference announcement, Google committed to making Gemini a core part of the Android experience. Gemini Nano, arguably the #1 on-device model, will seamlessly integrate with the #1 smartphone OS via the Google Edge AI SDK. While Google does not directly charge for the SDK, ease of use and “free” on-device inference incentivize developers to build with its SDK instead of managing cloud AI costs or a separate foundation on-device model. If developers want to use more powerful models in the cloud, Google will make connecting to larger Gemini models easy. More Android developers = more apps = more Google Play app fees and Google Cloud revenue.

Samsung is also in a strong position as the largest smartphone vendor globally, barely beating Apple. It has the most extensive range of smart home devices and is even more vertically integrated than Apple. The Samsung group of companies has its own battery and chip fabs. However, if Samsung wins, Google also wins since many of the key AI features in its latest flagship Galaxy S24 phone rely on Google’s AI.

Where can OpenAI play?

OpenAI is purely on the software layer with GPT models and ChatGPT. Their advantage is that they can provide a seamless experience across ecosystems and devices. Users own a range of devices. For example, I have a Pixel Fold, a Dell work laptop, and a personal Lenovo laptop. I’d like to interact with the same AI assistant that knows my data regardless of where I interact with it — and that could be ChatGPT. The counter here is that Google and Microsoft are playing the same game: the Gemini and Microsoft Copilot apps are available on iOS and accessible via any browser.

A deep partnership between OpenAI and Apple makes sense because both complement each other’s gaps. OpenAI has a market-leading AI assistant, and Apple has a broad portfolio of devices that seamlessly work together. Alternatively, Apple could acquire OpenAI. OpenAI’s latest valuation of $80 billion is “just” less than 3% of Apple’s $3 trillion market capitalization.

Who could win among the smaller private companies?

While I believe that Google, Samsung, and Apple will dominate the AI phone market, there is one private company I think could carve its own niche: Nothing Tech. The UK-based company is founded by OnePlus co-founder Carl Pei. In case you didn’t know, OnePlus is one of the largest phone brands globally, although primarily known in Asia.

Nothing makes well-reviewed and beautifully designed mid-range wireless earbuds and smartphones. If I were not attached to Pixel, I would use Nothing devices. Product aside, what makes them unique as a company is how fast they launched products and grew despite cutthroat competition. Four years since being founded in 2020, Nothing has already sold 3 million devices worth ~$560 million across two generations of phones and three generations of earbuds.

Nothing also thinks smartphones will be the personal AI device that everyone uses. Their CEO recently shared what I think is the clearest vision of what the GenAI era means for smartphones: the opportunity and challenge of moving away from an app-centric paradigm to one in which AI is the super app. Check out the video tweet in which Carl elaborates more on Nothing’s approach and early design concepts for their AI phone.

If OpenAI wants to accelerate its hardware play, Nothing is a prime acquisition candidate. It is the only hardware company actively integrating ChatGPT into the user experience — it was the first to integrate ChatGPT as the default assistant for wireless earbuds. Its phone OS also features ChatGPT-specific widgets and functions.

Challenges & opportunities in AI smartphone components

As AI smartphone production ramps up, companies that can solve the current hardware constraints are best positioned to rise with the market. The three primary constraints are processors, memory, and battery.

The brain of every smartphone is the SoC (system on chip), which includes the CPU for general tasks, GPU for graphics and complex calculations, NPU for machine learning, DSP for signal processing, and connectivity modules for cellular, Wi-Fi, and Bluetooth. For simplicity, think of the SoC as the processing power. When an LLM is needed, it is loaded into the device's RAM. The challenge is fitting large models into the available memory without compromising performance and battery life.

To illustrate this, you can see a teardown of the iPhone 6 internals below. The battery occupies most of the space, with the mainboard housing the SoC and memory.

Over the years, users have demanded faster, more responsive devices capable of handling multiple tasks, running more complex apps, and maintaining long battery life. This includes seamless video calls (FaceTime), quick photo and video editing for social media (Instagram), doomscrolling videos (TikTok), and efficient navigation with real-time updates (Doordash and Uber). To meet these needs, smartphone manufacturers have been cramming more powerful SoCs, additional memory, and larger batteries into phones every year. Since 2016, the physical volume of smartphones has increased by over 50%. Running GenAI models on-device is even more resource-intensive, so we’ll likely see phones become even larger and more manufacturers experimenting with the foldable format.

Key constraints to manufacturing AI smartphones

1. Processing Power / SoC / “the chip”

Traditional CPUs and GPUs are often inefficient for AI workloads, leading to the integration of NPUs in modern smartphones. NPUs, designed for neural network computations, enable faster and more power-efficient AI processing. This specialized hardware supports real-time inferencing, enhancing features like advanced photo editing and real-time translation.

Key players: The best SoC manufacturers are Qualcomm, MediaTek, and Apple (for its own devices). MediaTek used to be the mass-market player but has recently made a big push to make premium chips with its Dimensity 9 series. Reports suggest they are also partnering with NVIDIA to produce smartphone chips.

2. Memory / RAM

AI applications require substantial memory resources, particularly LPDDR (Low Power Double Data Rate). LPDDR is a type of DRAM designed for mobile and portable devices. It offers high-speed data transfer and low power consumption, making it ideal for smartphones. For simplicity, just think of it as RAM. The size and power of a model that can be run on a device are constrained by RAM. It is the minimum gating factor. It is recommended that devices should have at least 8GB of RAM available to run the 7B parameter models, 16GB to run the 13B models, and 32GB to run the 33B models. The standard Llama-3 model has 70B parameters.

Quantization is a process that reduces the precision of the numbers representing the model parameters, which can significantly reduce the memory footprint of AI models. For example, a 7B parameter model typically requires 16GB of RAM. If quantized to half its size, the RAM requirement could potentially be reduced to 8GB. This reduction makes running larger models on devices with less memory feasible, albeit with a trade-off in accuracy.

Interestingly, the phones with the most RAM are all Chinese phones with configurations up to 24GB, such as the Honor 90 GT. The leading US smartphones today have much less: iPhone 15 Pro Max has 8GB while Pixel 8 Pro and Samsung S24 have 12GB.

Key players: Samsung, SK Hynix, and Micron are the largest manufacturers of DRAM. RAM for smartphones is commoditized; there is no material difference between Samsung’s and SK Hynix’s. What matters more is that they can manufacture the latest generation of LPDDR.

3. Battery & Power Consumption

AI applications on smartphones are highly compute-intensive, significantly impacting battery life. For example, using an AI de-noise filter in Adobe Lightroom required 30 times more battery power than a standard noise filter. Battery life, which is what users care about, is determined by chip efficiency and battery capacity.

Chip design is crucial for battery life. Two major design architectures dominate the market: x86 and ARM. ARM architecture, known for its energy efficiency, powers nearly 99% of mobile chips. Qualcomm and MediaTek rely on ARM's design to design its chips. In contrast, Intel's x86 architecture, traditionally used in PCs, is less optimized for energy efficiency. This fundamental difference is why ARM-powered devices, such as Apple's products, often outperform their x86 counterparts in terms of battery life. Finally recognizing the advantages of ARM's low-power architecture, Microsoft has adopted ARM/Qualcomm as the platform for next-gen AI PCs.

Battery capacity is measured in mAh (milliampere-hour). Batteries are more commoditized than memory with batteries from different manufacturers interchangeable with one another. For many years, there have been no new developments until recently phone vendors started using lithium-ion batteries housing silicon-carbon anodes, which have more capacity than traditional graphite anodes. Chinese phone maker Honor uses this technology to pack a 5,600 mAh battery inside their latest flagship phone Magic 6 Pro. In comparison, iPhone 15 Pro Max has 4,422 mAh while Pixel 8 Pro has 5,050 mAh.

Key players: China's Amepretex Technologies is the largest phone battery manufacturer, followed by South Korea's LG Energy and Samsung SDI. It is worth noting that Japan's TDK supplies both Honor and Apple.

Is model performance a constraint?

No. Frontier models have passed the Turing test and continue to progress rapidly. Take OpenAI's GPT model as an example, from GPT-4's first release to GPT-4o's earlier two weeks ago. In 14 months, context length increased 16x, became multimodal, and became smarter, while dropping prices by ~80%.

Investment screen & disclosure

I’ve noted below what I have invested in. The stock price returns below are not mine, unfortunately.

Curated reads

• Academic: An Introduction to Vision-Language Modeling

• Commercial: Apple’s WWDC 2024 AI Plans

• Social: Why the OpenAI board fired Sam Altman