This article is a summary of the report’s key findings, with screenshots and the AI agent market map. For the full report & database:

• Multimedia and PDF versions of the report: link

• AI agent market map: link or head to the bottom of the article

Key Themes from the 2025 AI Trends report

From Experiment to Economic Engine

This is the fourth edition of my annual AI Trends report. I started writing these in 2022, before ChatGPT launched, and watching this technology evolve from academic curiosity to economic force has been something else.

This year’s report is titled “Crossing the Chasm” because that’s what the data shows: generative AI has moved from early adopters into mainstream production use. It’s no longer about potential—it’s generating real revenue, displacing real labor, and running into real physical constraints.

What follows are the findings I think matter most.

1. AI Agents Are Matching Human Professionals

The shift from AI-as-assistant to AI-as-worker is showing up in benchmarks now. The GDPVal benchmark tests whether AI can produce professional-grade deliverables—things like 3D engineering models and financial analyses. The latest results: GPT-5.2 matches or beats human experts (with an average of 14 years of experience) 70.9% of the time.

The economics are stark. For a range of knowledge work, AI agents are now over 20x cheaper for basic tasks and more than 3x cheaper for advanced work like programming and creative writing.

This is already affecting hiring:

• Employment for workers ages 22-25 in highly exposed roles (software development, customer service) has fallen sharply

• Job postings for AI-exposed roles are down 12% relative to less exposed roles

• When surveyed, companies report AI-driven headcount reductions most often in Engineering (42%), CS & Support (27%), and Marketing (26%)

2. Energy Is the Bottleneck

For all the focus on algorithms and chips, power is emerging as the binding constraint.

Epoch AI found that while chip manufacturing could theoretically support an 80,000x increase in training compute by 2030, power constraints cap that at around 10,000x. Energy is the tightest limit by a wide margin.

This isn’t just theoretical. 92% of senior data center professionals now cite utility power availability as a major barrier. 44% face wait times of four years or more for grid connections. Some are building their own on-site generation—gas turbines, even small modular nuclear reactors.

3. Adoption Is Faster Than Any Technology We’ve Measured

By mid-2025, 55% of US adults had used a generative AI product. That’s a faster adoption curve than the internet or the smartphone.

The products driving this:

• ChatGPT: 900 million weekly active users, >$12B ARR. It’s become the default brand for AI the way Google is for search.

• Claude Code: $1B revenue run rate six months after launch. This is the clearest commercial proof that autonomous agents can replace high-value professional work at scale.

• Sora 2 & Nano Banana: High-fidelity video and image generation moved from novelty to genuine creative tool for millions of users.

4. Acquisitions Are the Exit Path, Not IPOs

AI exit activity increased 44% from 2023 to 2025, but the biggest value is going to Big Tech acquirers, not new public companies.

The interesting dynamic: incumbents are using licensing and acqui-hire structures to sidestep antitrust scrutiny. Microsoft’s deal with Inflection AI became the template—they hired the key employees and licensed the technology without a formal acquisition. It effectively transfers the most important assets while avoiding regulatory review.

5. Is This a Bubble? It’s Complicated.

Valuations are high—95th percentile on the Goldman Sachs bubble framework. But other indicators that typically precede crashes (profit decline, credit risk, leverage) remain healthy. High valuations alone don’t cause crashes; deteriorating fundamentals do.

The new risk factor is asset lifespan. The telecom bubble left behind fiber optic cable with a 25-30 year useful life. The AI boom is built on GPUs with 2-4 year lifespans that depreciate fast as new chips release. That makes the capital stack more fragile and more dependent on continuous external financing.

I’d call the fundamentals healthy but the financing structure worth watching closely.

6. Where the Next Wave of Startups Will Come From

The framework I use: map job tasks by automation feasibility against total wages paid for that role. High feasibility + large wage pool = attractive target for agentic AI companies.

This worked well in practice. The high-potential roles I flagged in last year’s report (software developers, financial managers, legal professionals) attracted the most VC and produced the fastest-growing startups—Cursor, Harvey, etc.

Closing

The data on adoption, labor displacement, and infrastructure strain all point the same direction. The open questions are about pace and constraints—how fast energy infrastructure can scale, how financing structures hold up, and which verticals see the most disruption next.

There’s a lot more in the full report: deeper dives on scaling laws, sector-by-sector breakdowns, and a market map of 1,800+ AI startups. What I’ve covered here are just the findings I thought deserved the most attention.

Full Report

Agent Market Map: 1,800 companies and the top 10% that matter the most

Subscribe now

Table of Contents

• Summary

• AI Trends Report

  • Section 1: Macroeconomics

  • Section 2: Public Markets

  • Section 3: Industry Deep Dives

  • Section 4: Private Markets

  • Section 5: Product & Models

  • Section 6: Scaling Laws

  • Section 7: AI Agent Opportunity

  • Section 8: 2025 Outlook

Summary

The story of AI in 2024 is of power law and agents. Power law because the monetary gains are concentrated in a handful of companies, even in public markets. There’s also an increasing agglomeration of capital among the few who can continue to continue the path of AI scaling laws.

Beginning with the big picture, generative AI or foundation models are the fastest adopted generational purpose technology (GPT). In 2 years, 40% of the population have used a generative AI product. It took personal computers over 15 years to get similar adoption. Smartphones was the latest GPT to get rapid adoption and it took 4 years to get to 40%.

Generative AI is also a powerful technology with major implications in how we live and work. I evaluated 16,000 tasks across 800 jobs to understand what is the automation potential of different jobs across the spectrum of generative AI technologies. Agents are the next frontier of automation and we should be ready for that.

The rapid adoption is reflected in stock market returns with the Generational AI Index returning 127% since the beginning of 2023, compared to 85% for NASDAQ and 58% for S&P. As you’ll see in the full report, 75% of the AI Index gains come from just 7 companies.

Most of the gains are also concentrated in the AI value chain, notably the chips and energy supplied to data centers. Data centers are the “currency” to play in the generative AI game. If scaling laws continue, there will continue to to be chip and energy shortages in the near to medium term.

In the private markets, there’s also herding (i.e. concentration) among venture funds in a handful of companies. Different venture pursued different strategies, reflecting different risk profiles. If the generative AI boom continues, then Sequoia, a16z, and Lightspeed stand to make generational returns for their LPs and themselves.

The pace at which the AI industry advanced in 2024 made 2023 look pale in comparison. Even now as of writing in December 13, OpenAI has 5 more days in its Shipmas series. Google just released Gemini 2.0 two days ago. Microsoft also released Phi-4.

But the entire AI boom seems to centrally hinge around scaling laws continuing. There are lots of chatter about whether we’ve faced a scaling “wall”. We have not but we are facing a more complex scaling dynamic. There’s still plenty of room to scale. In the diagram below, I sketch out what scaling laws look like today.

Even if scaling laws continue, there is the question of whether it is worth it to continue pursuing it. o1 likely costed billions to train. Is a $10 billion model worth training? What about $100 billion? These are local GDP level numbers. One way to frame the ROI question is to look at how generative AI can augment and automate jobs. There’s ~$40 trillion of wages paid each year, of which ~$11 trillion are paid in the US. A $100 billion model still sounds extreme, but at least not insanely irrational. In the full report, I provide a framework of how to think about the best opportunities for AI agents and the 12 areas with the most potential.

AI Trends 2024 Report

If you prefer a PDF copy: link

If you’d like to view it online with media & links: link.

Subscribe now

Section 1: Macroeconomics

Section 2: Public Markets

Section 3: Industry Deep Dives

Section 4: Private Markets

Section 5: Product & Models

Section 6: Scaling Laws

Section 7: AI Agent Opportunity

Section 8: 2025 Outlook

In the past few months, Microsoft, Amazon, and Google have engaged in quasi-acquisitions of foundation model companies Inflection, Adept, and Character.ai, respectively. Instead of a traditional acquisition involving stock purchases, these Big Tech firms opted to hire key AI talent from these startups and pay the investors license fees. In this article, I analyze these deals through a corporate development lens, offering details and insights beyond what has already been covered in the media:

• Why Microsoft, Amazon and Google pursued those deals

• Why they resorted to an unusual deal structure

• Why they paid so much money

• Who else could acquire and be acquired

• Why quasi-acquisitions won’t become a trend

Subscribe now

Overview of the deals

Why acquire?

Speed is the primary reason why Microsoft, Google, and Amazon spent hundreds of millions (perhaps billions) on these deals. They are rushing to become market leaders because the rewards are high, competition is intense, and there is a data flywheel effect that accumulates over time.

Reward for winning is high. The revenue potential of the generative AI market is enormous, and the pace of growth is unprecedented. For example, in June 2024, Sam Altman shared in a company-wide meeting that OpenAI’s annualized revenue is $3.4 billion—121 times more than the $28 million revenue the company generated in 2022, making it the fastest-growing company in history. It was estimated that 55% of this revenue comes from individuals (ChatGPT), and the other 45% from businesses, suggesting that both B2B and B2C markets are lucrative.

Competition is intense. With such a large prize at stake, competition is fierce. All the big tech companies are spending billions to build data centers to support AI models. Microsoft’s exclusive relationship with OpenAI drives both Azure spend and access to OpenAI’s latest tech, which is integrated into Microsoft’s other products. Amazon has a deep relationship with Anthropic through a $4 billion investment. If AI cloud spend goes to Microsoft, it might also attract other cloud spending. Google’s search dominance is at risk if it doesn’t build an AI layer on top of its search results, and its cloud prospects might also suffer. Google committed to investing $2 billion in Anthropic.

In addition to these investments, the Big Tech companies have been investing approximately $15 billion each in cloud infrastructure every quarter, and they expect this to increase over the next several quarters.

• Alphabet/Google (7/23/2024): “Our reported Capex in the first quarter was $12 billion. Looking ahead, we continue to expect quarterly Capex throughout the year to be roughly at or above the Q1 Capex of $12 billion. We continue to invest in designing and building robust and efficient infrastructure to support our efforts in AI given the many opportunities we see ahead.”

• Microsoft (7/30/2024): “Capex, including financing leases, were $19 billion. Cloud and AI-related spend represents nearly all of our total Capex. We expect Capex to increase on a sequential basis, given our cloud and AI demand as well as existing AI capacity constraints.”

• Meta (7/31/2024): “We anticipate our full year 2024 Capex will be in the range of $37 billion to $40 billion. We currently expect significant Capex growth in 2025 as we invest to support our AI research and our product development efforts. Llama 3 is already competitive with the most advanced models and we’re already starting to work on Llama 4. The amount of compute needed to train Llama 4 will likely be almost 10 times more than what we used to train Llama 3.

• Amazon (8/1/2024): “For the first half of the year, Capex was $30.5 billion. Looking ahead to the rest of 2024, we expect capex to be higher in the second half of the year. The majority of the spend will be to support the growing need for AWS infrastructure as we continue to see strong demand in both generative AI and our non-generative AI workflows.”

Data flywheel. As discussed in my past essay on data moats, there is a data flywheel in AI products. User data helps build better models and products. Researchers estimate that LLMs will have been trained on all public internet data by as early as 2026. Proprietary user data becomes even more important in building better models.

Now that we know why the big companies made these acquisitions, there are two questions: why the quasi-acquisition structure, and why such large amounts of money?

Why not a standard acquisition structure?

The structure of the quasi-acquisitions consists of two parts:

1. Talent Acquisition: The big tech companies hire the founders and key employees of these startups, selectively acquiring the talent they want. Microsoft hired over 80% of Inflection AI’s team, Amazon hired 66% of Adept’s team, and Google hired a smaller fraction of Character AI’s team.

2. Non-exclusive technology licensing: Instead of acquiring the entire company, the Big Tech firms obtain non-exclusive licenses to the startups' models. The license fees are then used to pay investors a negotiated return over time.

Why not just buy the shares like a regular acquisition?

This is to avoid further regulatory scrutiny, which would be triggered by an HSR filing. An HSR filing (Hart-Scott-Rodino Act) is a pre-merger filing to both the FTC (Federal Trade Commission) and DOJ (Department of Justice) and is required when it meets certain thresholds such as a $120 million deal size. After filing, the agencies will have 30 days to do an initial review. During this period, the agencies can approve the transaction or request for more information. Companies must substantially comply with the data requests. That takes weeks to prepare. Agencies can take weeks to months more to review the new information, which could lead them to challenge the deal. Given the strategic importance of speed and certainty to close, Big Tech firms (like all firms) prefer to avoid this scrutiny, especially when the FTC announced in January 2024 that it had already launched inquiries into AI deals done by them. While the HSR Act applies to acquisitions of assets and voting securities, the definition of assets conveniently excludes non-exclusive licenses and people.

Why not just give large bonuses to talent and avoid paying license fees?

Investors would block the acquisition. If key talent is hired, the startup’s value plummets. The license fees give investors a satisfactory return on their investments. Since the founders are also owners, license fees serve as exits for them. For example, in the Microsoft-Inflection deal, Series A investors will get 1.5 times their $225M investment, and Series B investors will get 1.1 times their $1.3B investment. From the date of the funding announcement to the date of the acquisition announcement, Series A investors achieve a 25% IRR, and Series B investors, 14%. Google is more generous with a 2.5x return. Adept investors get their money back. It is likely that these returns correlate with consumer product presence:

• Adept is targeting enterprises and has not publicly released a product yet

• In March 2024, Inflection reportedly had 1 million daily active users

• In July 2024, Character reportly had 6 million daily active users

Why that much money?

If the acquired companies were struggling, as media reports suggest, how did the teams at Microsoft, Amazon, and Google justify the high “license fees” they paid? It’s not because of the models themselves. They obtained non-exclusive licenses to models that will be outdated in six months. With a gutted team and license fees going to investors, the acquired companies have no resources to continue building frontier models. So it’s a mix of other factors:

• Talent: The primary driver behind these deals is the acquisition of top AI talent. These startups have assembled teams of the best researchers and engineers, and Big Tech companies are willing to pay a premium to bring them in-house. This talent acquisition accelerates their own AI development efforts and potentially prevents competitors from gaining an edge.

• Dataset: While the models might not be the primary focus, the datasets these startups have accumulated can be invaluable. These datasets, often fine-tuned for specific tasks or domains, can provide a significant advantage in training and refining the acquiring company's models. It was reported that Amazon will gain access to Adept's dataset, which likely includes valuable proprietary data related to web browsing, productivity tools, and other user interactions as part of the deal. Unlike public web text, user interaction data is hard to obtain.

• GPU access: Acquiring startups that have secured significant GPU capacity can help Big Tech companies overcome potential bottlenecks and accelerate their AI initiatives. Inflection, for example, had secured access to a cluster of 22,000 NVIDIA H100 GPUs, one of the largest in the world, which further incentivized Microsoft's acquisition.

• Users: Users provide a testing ground for new AI features and products, and their engagement data can further enhance the acquiring company's models. Additionally, a loyal user base can translate into potential revenue streams and market share. Character AI, with monthly traffic rivaling Gemini, presented an attractive target for Google in this regard.

  

• VC approval: Venture capitalists who have invested in these startups expect a return on their investment. The high price tags associated with these quasi-acquisitions ensure that investors are adequately compensated, preventing potential roadblocks.

Who else could be quasi-acquisition targets?

According to The Information, there are seven AI startups that are struggling and may look for buyers: Reka (raised $56 million, valued $300 million), Imbue (raised $232 million, valued >$1 billion), Cohere (raised $980 million, valued $5.5 billion), Pika Labs (raised $141 million, valued $470 million), Ideogram (raised $97 million, valued $580 million), AI21 Labs (raised $336 million, valuation $1.4 billion), and Essential AI (raised $65 million)

But who could potentially buy these companies? Acquiring model-centric companies only makes sense for cloud service providers. Given the additional scrutiny the recent deals have attracted, it is unlikely they would pursue another high-profile acquisition. IBM and Oracle, the 4th and 5th largest US cloud providers, are unlikely to acquire talent to start building their own frontier models. Two application-layer companies, Salesforce and Adobe, could be potential buyers. Salesforce has cloud ambitions and a broad surface of applications to deploy AI talent. Similarly, Adobe also has a wide range of applications, including its family of Firefly models. But neither Salesforce nor Adobe has antitrust concerns regarding AI. They would likely buy the stocks directly or pursue a typical acquihire.

Interesting reads:

• Seven generative AI startups that may look for a buyer

• Google’s Gemini Live is here

• MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

Subscribe now

I made a point to learn about the hardware aspects of AI this year. Historically, I've focused on the software side, but I realized that hardware is the major constraint for generative AI today. Currently, there are not enough servers in data centers to train the next frontier model, and the processors in existing computers and portable devices are not powerful enough to run these models. However, jumping straight into the world of semiconductors and devices was overwhelming. The supply chain is complex, and most books focus either on geopolitics or the electrical engineering aspects of wafers and circuits. I was specifically looking for a product & business overview.

I ended up with a hodgepodge of materials of varying quality and depth.

So, I gave up on old-school research and turned to ChatGPT. With ChatGPT, I could ask about anything, wherever my mind took me. When I encountered jargon like RISC (reduced instruction set computer), I could get an immediate explanation instead of parsing through Google results. Instead of listening to podcasts and watching videos, I talked to ChatGPT. That wasn't the first time I used the voice capabilities, but it was my first time having a lengthy conversation about a complex topic with ChatGPT. The transformative experience was strolling around San Francisco while talking to ChatGPT. I had a research collaborator in my pocket that I could bring anywhere. I didn’t have to be glued to a screen. All it takes is a few taps to start talking to an AI assistant that knows more about semiconductors than 99% of the world.

When OpenAI demoed GPT-4o, it showcased another subtle yet significant improvement in user experience: the ability to interrupt ChatGPT and redirect the conversation by talking. OpenAI models are overly chatty. So when I’m talking to ChatGPT while my phone is in my pocket, I have to either listen to the entire response or pull out my phone to interrupt it manually. This might seem like a mundane quality-of-life upgrade, but it's incredibly more natural. It feels more human. After spending hours talking to ChatGPT, the potential of personal AI devices dawned on me. It is not a conceptual idea anymore but a lived one.

Types of personal AI devices

Personal AI devices are pocket-sized pieces of hardware through which people primarily interact with an AI assistant. Desktops, laptops, and AR/VR headsets don’t make the cut. Being pocket-sized is important because it allows the user to carry the device anywhere. This aspect of being pocket-sized is what made the iPhone iconic: Your life in your pocket.

The opportunity is immense: The iPhone launched the mobile era and generated $91B in sales for Apple last year. With the smartphone market valued at $485B annually, investors are eagerly hunting for startups aiming to create the iPhone of the GenAI era. More than $250 million has been invested in the makers of the distinctly neon orange Rabbit R1 ($199) and the Apple-esque Humane Pin ($699 + $24/mo subscription + $100 for a shiny chrome finish).

Despite getting awful reviews, it is laudable that both companies attempted to create new form factors. I am looking forward to seeing more ideas. However, I think we will likely see upgrades to the devices that we already have today: smartphones and wearables like in-ear headphones, glasses, and watches.

1. Smartphones have an established form factor that can fit enough memory and computing power for GenAI models. They are the most essential piece of personal technology today — we carry them everywhere to take pictures, call loved ones, reply to work emails, and binge-watch videos.

   • What’s new in this category: large foldable phones such as Samsung Fold and Pixel Fold are becoming more popular. Users get a larger screen, and manufacturers can pack in more battery and processing power.

2. Earphones are interesting because audio is central to communication. Until we get brain-computer interfaces, audio is the most efficient way to communicate. While we can use gestures and wave our hands, it’s inefficient and weird. The tiny form factor of in-earbuds makes it impossible to fit enough battery and processing power to make them viable AI devices. While over-ear headphones can pack more, they are not pocket-sized.

   • What’s new in this category: Companies are experimenting with new designs such as Google-spinout Iyo's One buds, which are noticeably larger to fit in a larger battery, ten microphones, and standalone internet connectivity.

3. Smart glasses are interesting because they capture what our eyes see. Vision is fundamental because sometimes it is easier to communicate with images than words. This is more so for people whose language is not the primary language the models are trained on, such as myself, who grew up speaking Tagalog. With flowers blooming in San Francisco summer, I recently passed by flowers I wanted to identify. Instead of attempting to describe it — lamp-shaped orange flowers with hairy buds in San Francisco — I sent a picture to ChatGPT, which identified it as Orange Clock Vine. Using the textual description, I got California Poppy instead. A picture is worth a thousand words.

   • What’s new in this category: Meta is the leader in this space with the Ray-Ban smart glasses and is reportedly exploring designs combining both earphones and cameras into one device. Google, which was way ahead of its time with Google Glasses 11 years ago, is also developing a new device today.

4. Smart watches are not interesting because they are only good for tracking biometrics. Outside of that, they are primarily used as an extension of the smartphone, so they are unlikely to be personal AI devices.

   • What’s new in this category: more sensors.

What will be the personal AI device that everyone uses? I bet that it will be smartphones, with wearables relying on them as the processing unit. We already see this pattern today with our earphones and watches paired to smartphones via Bluetooth. Wearables augment what we can do with a smartphone, but wearables are useless without a smartphone.

What are GenAI smartphones?

While AI has already been part of our daily smartphone interactions, native GenAI experiences are new. Smartphones today need an upgrade to become personal AI devices or what industry analysts coin as GenAI/AI smartphones. An AI smartphone is defined as having:

1. A GenAI assistant integrated with phone controls and OS to interact with mobile apps.

2. Capability to process on-device foundation models and funnel complex workloads to cloud-based models. There are a couple of reasons why on-device AI matters:

   • Local processing of sensitive data: Processing data locally can keep data private. This is important for apps that handle sensitive data, such as messaging apps with end-to-end encryption.

   • Offline access: Users can access AI features even without an internet connection. This is useful for applications that need to work offline or with spotty connectivity.

   • Cost savings: Developers can reduce inference costs by offloading execution to consumer hardware, ideally leading to lower mobile app prices.

3. Hardware capable of delivering the features above. IDC defines this as a phone capable of >30 TOPS (trillion operations per second).

According to market research firm Counterpoint, of the 1.2 billion smartphones shipped in 2023, less than 1% met the definition of an AI smartphone. 2024 will be the year of AI phones, with vendors launching their first AI phone models. Samsung launched the Galaxy S24 in January, and Google and Apple are expected to release theirs in the coming months. Supply chains are rapidly being reconfigured to ramp up production of AI phones, which are expected to comprise 43% of smartphone shipments by 2027

Who is positioned to win the race?

While high-end specs are the minimum requirement to play in this space, winning requires a deep ecosystem of developers, applications, and other personal devices. Mapping these factors against the key players, Apple and Google are best positioned to win the space. However, their paths to winning differ.

Apple is the leading smartphone maker in the US, with over 50% market share. It also has a wide install base of Apple TV and Mac computers — macOS holds 20% of the laptop and desktop market. Many are looking towards WWDC 2024, when Apple will share its GenAI plans and OpenAI partnership. We may not yet see the next iPhone then, but in a few more months, Apple is expected to launch the first “Ai”Phone. From a business perspective, these will drive customers to upgrade sooner and Apple could also add an Apple Intelligence service fee.

While Apple will win with its vertically integrated ecosystem, Google will win via software and ecosystem lock-in. Its line of Pixel phones has less than a percent of the global market and ~5% of the US market. But Google's play is software. Android holds ~70% global OS market share. During their recent I/O conference announcement, Google committed to making Gemini a core part of the Android experience. Gemini Nano, arguably the #1 on-device model, will seamlessly integrate with the #1 smartphone OS via the Google Edge AI SDK. While Google does not directly charge for the SDK, ease of use and “free” on-device inference incentivize developers to build with its SDK instead of managing cloud AI costs or a separate foundation on-device model. If developers want to use more powerful models in the cloud, Google will make connecting to larger Gemini models easy. More Android developers = more apps = more Google Play app fees and Google Cloud revenue.

Samsung is also in a strong position as the largest smartphone vendor globally, barely beating Apple. It has the most extensive range of smart home devices and is even more vertically integrated than Apple. The Samsung group of companies has its own battery and chip fabs. However, if Samsung wins, Google also wins since many of the key AI features in its latest flagship Galaxy S24 phone rely on Google’s AI.

Where can OpenAI play?

OpenAI is purely on the software layer with GPT models and ChatGPT. Their advantage is that they can provide a seamless experience across ecosystems and devices. Users own a range of devices. For example, I have a Pixel Fold, a Dell work laptop, and a personal Lenovo laptop. I’d like to interact with the same AI assistant that knows my data regardless of where I interact with it — and that could be ChatGPT. The counter here is that Google and Microsoft are playing the same game: the Gemini and Microsoft Copilot apps are available on iOS and accessible via any browser.

A deep partnership between OpenAI and Apple makes sense because both complement each other’s gaps. OpenAI has a market-leading AI assistant, and Apple has a broad portfolio of devices that seamlessly work together. Alternatively, Apple could acquire OpenAI. OpenAI’s latest valuation of $80 billion is “just” less than 3% of Apple’s $3 trillion market capitalization.

Who could win among the smaller private companies?

While I believe that Google, Samsung, and Apple will dominate the AI phone market, there is one private company I think could carve its own niche: Nothing Tech. The UK-based company is founded by OnePlus co-founder Carl Pei. In case you didn’t know, OnePlus is one of the largest phone brands globally, although primarily known in Asia.

Nothing makes well-reviewed and beautifully designed mid-range wireless earbuds and smartphones. If I were not attached to Pixel, I would use Nothing devices. Product aside, what makes them unique as a company is how fast they launched products and grew despite cutthroat competition. Four years since being founded in 2020, Nothing has already sold 3 million devices worth ~$560 million across two generations of phones and three generations of earbuds.

Nothing also thinks smartphones will be the personal AI device that everyone uses. Their CEO recently shared what I think is the clearest vision of what the GenAI era means for smartphones: the opportunity and challenge of moving away from an app-centric paradigm to one in which AI is the super app. Check out the video tweet in which Carl elaborates more on Nothing’s approach and early design concepts for their AI phone.

If OpenAI wants to accelerate its hardware play, Nothing is a prime acquisition candidate. It is the only hardware company actively integrating ChatGPT into the user experience — it was the first to integrate ChatGPT as the default assistant for wireless earbuds. Its phone OS also features ChatGPT-specific widgets and functions.

Challenges & opportunities in AI smartphone components

As AI smartphone production ramps up, companies that can solve the current hardware constraints are best positioned to rise with the market. The three primary constraints are processors, memory, and battery.

The brain of every smartphone is the SoC (system on chip), which includes the CPU for general tasks, GPU for graphics and complex calculations, NPU for machine learning, DSP for signal processing, and connectivity modules for cellular, Wi-Fi, and Bluetooth. For simplicity, think of the SoC as the processing power. When an LLM is needed, it is loaded into the device's RAM. The challenge is fitting large models into the available memory without compromising performance and battery life.

To illustrate this, you can see a teardown of the iPhone 6 internals below. The battery occupies most of the space, with the mainboard housing the SoC and memory.

Over the years, users have demanded faster, more responsive devices capable of handling multiple tasks, running more complex apps, and maintaining long battery life. This includes seamless video calls (FaceTime), quick photo and video editing for social media (Instagram), doomscrolling videos (TikTok), and efficient navigation with real-time updates (Doordash and Uber). To meet these needs, smartphone manufacturers have been cramming more powerful SoCs, additional memory, and larger batteries into phones every year. Since 2016, the physical volume of smartphones has increased by over 50%. Running GenAI models on-device is even more resource-intensive, so we’ll likely see phones become even larger and more manufacturers experimenting with the foldable format.

Key constraints to manufacturing AI smartphones

1. Processing Power / SoC / “the chip”

Traditional CPUs and GPUs are often inefficient for AI workloads, leading to the integration of NPUs in modern smartphones. NPUs, designed for neural network computations, enable faster and more power-efficient AI processing. This specialized hardware supports real-time inferencing, enhancing features like advanced photo editing and real-time translation.

Key players: The best SoC manufacturers are Qualcomm, MediaTek, and Apple (for its own devices). MediaTek used to be the mass-market player but has recently made a big push to make premium chips with its Dimensity 9 series. Reports suggest they are also partnering with NVIDIA to produce smartphone chips.

2. Memory / RAM

AI applications require substantial memory resources, particularly LPDDR (Low Power Double Data Rate). LPDDR is a type of DRAM designed for mobile and portable devices. It offers high-speed data transfer and low power consumption, making it ideal for smartphones. For simplicity, just think of it as RAM. The size and power of a model that can be run on a device are constrained by RAM. It is the minimum gating factor. It is recommended that devices should have at least 8GB of RAM available to run the 7B parameter models, 16GB to run the 13B models, and 32GB to run the 33B models. The standard Llama-3 model has 70B parameters.

Quantization is a process that reduces the precision of the numbers representing the model parameters, which can significantly reduce the memory footprint of AI models. For example, a 7B parameter model typically requires 16GB of RAM. If quantized to half its size, the RAM requirement could potentially be reduced to 8GB. This reduction makes running larger models on devices with less memory feasible, albeit with a trade-off in accuracy.

Interestingly, the phones with the most RAM are all Chinese phones with configurations up to 24GB, such as the Honor 90 GT. The leading US smartphones today have much less: iPhone 15 Pro Max has 8GB while Pixel 8 Pro and Samsung S24 have 12GB.

Key players: Samsung, SK Hynix, and Micron are the largest manufacturers of DRAM. RAM for smartphones is commoditized; there is no material difference between Samsung’s and SK Hynix’s. What matters more is that they can manufacture the latest generation of LPDDR.

3. Battery & Power Consumption

AI applications on smartphones are highly compute-intensive, significantly impacting battery life. For example, using an AI de-noise filter in Adobe Lightroom required 30 times more battery power than a standard noise filter. Battery life, which is what users care about, is determined by chip efficiency and battery capacity.

Chip design is crucial for battery life. Two major design architectures dominate the market: x86 and ARM. ARM architecture, known for its energy efficiency, powers nearly 99% of mobile chips. Qualcomm and MediaTek rely on ARM's design to design its chips. In contrast, Intel's x86 architecture, traditionally used in PCs, is less optimized for energy efficiency. This fundamental difference is why ARM-powered devices, such as Apple's products, often outperform their x86 counterparts in terms of battery life. Finally recognizing the advantages of ARM's low-power architecture, Microsoft has adopted ARM/Qualcomm as the platform for next-gen AI PCs.

Battery capacity is measured in mAh (milliampere-hour). Batteries are more commoditized than memory with batteries from different manufacturers interchangeable with one another. For many years, there have been no new developments until recently phone vendors started using lithium-ion batteries housing silicon-carbon anodes, which have more capacity than traditional graphite anodes. Chinese phone maker Honor uses this technology to pack a 5,600 mAh battery inside their latest flagship phone Magic 6 Pro. In comparison, iPhone 15 Pro Max has 4,422 mAh while Pixel 8 Pro has 5,050 mAh.

Key players: China's Amepretex Technologies is the largest phone battery manufacturer, followed by South Korea's LG Energy and Samsung SDI. It is worth noting that Japan's TDK supplies both Honor and Apple.

Is model performance a constraint?

No. Frontier models have passed the Turing test and continue to progress rapidly. Take OpenAI's GPT model as an example, from GPT-4's first release to GPT-4o's earlier two weeks ago. In 14 months, context length increased 16x, became multimodal, and became smarter, while dropping prices by ~80%.

Investment screen & disclosure

I’ve noted below what I have invested in. The stock price returns below are not mine, unfortunately.

Curated reads

• Academic: An Introduction to Vision-Language Modeling

• Commercial: Apple’s WWDC 2024 AI Plans

• Social: Why the OpenAI board fired Sam Altman

The Business of AI Report analyzes the AI trends and events that happened over the year since ChatGPT was released through a business & strategy lens. It covers a wide range of topics from the fastest growing generative AI products to why San Francisco is the AI hotspot to OpenAI’s Dev Day. There is a link to the report in Slides format (with product GIFs) at the end of this post.

If you’re interested in meeting a curated group of 40-50 other people interested in the business of AI, check out the Enterprise GenAI Forum I’m hosting next week. The event will feature several lightning talks from people who have scaled AI companies. More about it here: event page.

Subscribe now

Subscribe now

Table of Contents:

• Modern data stack and business intelligence

• Modern BI problems & generative AI opporunities

• Deep dive: Text-to-SQL

• Notable companies: Incumbents & startups

• Company feature: TextQL

Subscribe now

Modern data stack and business intelligence

The modern data stack (MDS) emerged as a response to the growing need for managing and analyzing massive amounts of data generated by various sources like web applications, IoT devices, and third-party APIs. The traditional data stack, which often relied on monolithic, on-premises infrastructure and relational databases, struggled to keep up with the increasing volume, variety, and velocity of data. Consequently, the MDS was developed to accommodate these new data challenges more effectively and efficiently.

Two defining features of the MDS are its scalability and modularity. It leverages cloud-based services, distributed storage, and processing systems, allowing organizations to scale their data infrastructure more efficiently and cost-effectively. This approach also supports integration of point solutions for data ingestion, storage, processing, analysis, and visualization, making it adaptable to changing needs.

Modern business intelligence (MBI), built on top of the MDS, focuses on providing faster, more flexible, and scalable analytics solutions. Instead of using manual ETL processes for data integration, modern BI platforms employ automated data pipelines and ELT processes, speeding up data processing and transformation. Moreover, modern BI tools enable self-service analytics. Business users can analyze data without relying on data experts.

For example, a retail company might use the MDS to collect data from various sources like online sales, customer feedback, and social media interactions. They could use a cloud-based data warehouse like Snowflake to store and manage this data, and then employ data processing tools like Apache Spark or dbt for cleaning and transforming it. Finally, the company can leverage modern BI platforms like Tableau or Looker to analyze the data and generate actionable insights to optimize marketing campaigns and drive sales growth.

Overview of modern business intelligence

MBI comprises several components of the MDS architecture: data discovery, data modeling, metrics layer, end-user facing dashboard, and analytics.

1. Data discovery: This process involves identifying, collecting, and integrating various data sources relevant to the business. The objective is to obtain a comprehensive understanding of the data landscape while pinpointing gaps or inconsistencies.

2. Data modeling: This step transforms raw data into an easily analyzable and comprehensible format. Data modeling generally entails creating data structures, defining relationships between datasets, and ensuring data quality and consistency.

3. Metrics layer: This component deals with defining and tracking key performance indicators (KPIs) and other vital metrics. The metrics layer establishes a consistent framework for evaluating and measuring performance over time.

4. End-user facing dashboards & analytics: This aspect focuses on presenting data in a clear and actionable manner for end-users. Dashboards and analytics tools enable users to visualize data, explore trends, and make informed decisions based on insights.

5. Spreadsheets: Although not typically considered part of the MDS architecture, spreadsheets remain a widely used tool for data analysis and manipulation. They facilitate data organization, calculations, and charting.

Modern BI problems & generative AI opportunities

The modularity of MDS and MBI, while beneficial, has introduced new problems: disconnected tools and unmanageable data swamps. MDS often consist of disconnected tools that need specialized knowledge to integrate. The scalability of storage & compute also leads to a store-everything mentality creating unmanageable data swamps in SQL-centric data stores. With data being ingested from different sources, understanding the context becomes difficult. Tracing back entities and tables become increasingly perplexing with each step losing context. Sometimes even requiring another tool to decipher. Consequently, teams struggle to identify the source of truth, leading to ad-hoc, bespoke tables for answering specific questions. This creates "data debt" as these one-off solutions accumulate over time.

These problems present opportunities for generative AI to unlock the value of MBI.

1. Natural language querying (usually but not limited to text-to-SQL)

Pain point: Non-technical users struggle to extract insights from data stored in databases due to the learning curve associated with SQL or other query languages.

Status quo: Business users often rely on data analysts or engineers to write SQL queries, which is time-consuming and create bottlenecks in decision-making.

Use case solution: Generative AI converts natural language queries into SQL code. For example, a product manager asks, "What is the average revenue per user for the past three months?" The AI generates the SQL query, retrieves the information, and presents the result to the user.

2. Data documentation (table-to-text and SQL-to-text)

Pain point: Understanding and navigating complex data structures is challenging, especially when documentation is lacking or outdated.

Status quo: Data documentation is often created manually, which is time-consuming, error-prone, and difficult to maintain.

Use case solution: Generative AI generates dataset documentation, including descriptions of fields, data types, and relationships between tables. For example, when given a database schema for an e-commerce platform, the AI can create a document explaining each table (e.g., orders, customers, products) and their relationships.

3. Data munging / transformation

Pain point: Data preparation tasks such as classification, transformation, and cleaning are time-consuming and tedious.

Status quo: Data analysts and engineers spend a significant amount of time on manual data wrangling, which slows down the analysis process.

Use case solution: Generative AI automates data preparation tasks. For example, suppose an organization has inconsistent date formats across multiple data sources. The AI identifies the discrepancies, standardizes the date formats, and cleans the data.

4. Natural language analysis

Pain point: Data analysis using SQL is limiting, especially for more advanced techniques like clustering, sentiment analysis, or even simple financial analysis.

Status quo: Analysts and data scientists often need to be familiar with spreadsheet formulas or programming languages to perform advanced data analysis, which requires expertise.

Use case solution: Generative AI models and analyzes data without relying on SQL queries. For instance, an AI model can analyze customer transaction data to identify distinct customer segments based on their purchase behavior.

Deep dive: Text-to-SQL

Text-to-SQL, sometimes called natural language interfaces to databases (NLIDB), refers to the process of converting natural language queries into SQL queries that can be executed on a database. The primary goal of text-to-SQL systems is to provide a more user-friendly and accessible way for non-technical users to interact with databases, eliminating the need to learn complex SQL syntax. In this section, we will explore the historical evolution of text-to-SQL systems, highlighting key milestones and breakthroughs that have shaped the field.

If you observe the data space, you will notice a recurring trend: every couple of years a wave of startups emerge claiming to solve natural language data analysis. But data professionals have grown accustomed to being disappointed by products that are only right 50% of the time or only return outputs assuming your data is perfectly cleaned and your query matches the labels exactly. Catalyzed by OpenAI’s large language models, a new wave of these startups is spinning up again. With foundation models, maybe this time is different.

History

Rule-Based Systems (1970s-1990s) The journey of text-to-SQL began in the 1970s when computer scientists experimented with systems like LUNAR and CHAT-80. These early systems relied on handcrafted rules and keyword matching to translate natural language queries into SQL. While they demonstrated the potential of text-to-SQL, they struggled to handle complex queries and had limited scalability.

[LUNAR] has made considerable progress towards the ultimate goal of providing a general purpose natural language interface between men and machines. It is an experimental, research prototype of a question-answering system to enable a lunar geologist to conveniently access, compare, and evaluate the chemical analysis on lunar rock and soil composition that is accumulating as a result of the Apollo moon missions

Classical Machine Learning (1990s-2000s) The advent of machine learning in the 1990s brought new possibilities to the field of text-to-SQL. Researchers developed statistical models that could learn from data, allowing for more flexible and accurate query translation. One notable system from this era was PRECISE, which used statistical parsing to map natural language to SQL queries.

Deep Learning (2010s) The 2010s marked the arrival of deep learning, which had a transformative impact on text-to-SQL. Researchers started using neural networks, particularly sequence-to-sequence (seq2seq) models, to automatically generate SQL queries from natural language. These models were capable of handling complex queries and achieved impressive accuracy on benchmark datasets like WikiSQL.

Transformers & Foundation Models (2020s) The 2020s have seen the rise of transformer-based models, which have become the foundation of modern text-to-SQL systems. Pre-trained models like T5, BERT, & GPT-3 have been fine-tuned for text-to-SQL tasks, achieving state-of-the-art results. Off-the-shelf GPT-4 performs exceedingly well as well. These models are powerful because they're trained on vast amounts of text, allowing them to capture linguistic patterns and relationships. Another key development in this era is embedding analytical tasks in a conversational interface, reflecting that real-world analysis is done iteratively in practice.

Datasets & benchmarks

The introduction of benchmark datasets, such as WikiSQL and Spider, played a crucial role in driving progress in text-to-SQL research. These datasets provided large-scale, annotated examples of natural language queries and corresponding SQL queries, allowing researchers to train and evaluate their models in a standardized manner. In 2018, 11 Yale students at the LILY lab released the Spider dataset, the latest text-to-SQL performance benchmark containing 10K+ questions for over 200 databases across 100+ domains. It is currently the academic standard for measuring text-to-SQL models. The following year, the same lab released CoSQL, a conservational text-to-SQL dataset based on the original Spider dataset. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from 3k dialogues querying 200 complex DBs spanning 138 domains.

In April 2023, a team using the system DIN-SQL + GPT-4 reached the top of the Spider benchmark leaderboard with a score of 85%, outperforming the previous best by 5% points. This is a significant margin since the next 10 models perform within 5% points of each other.

Execution engines & state-of-the-art

While off the shelf LLM models provide a good baseline of syntactically correct SQL queries based on academic benchmarks, in practice queries are more complex. LLMs need to paired with an execution engine that evaluates, modifies, and validates the raw LLM SQL output. There are different approaches this from training a fine-tuned model, like PICARD, to a more flexible system, like DIN-SQL.

Query engine = LLM + execution engine

Since DIN-SQL is the state-of-the-art as of writing, we’ll focus on it. The acronym stands for Decomposed In-Context Learning of Text-to-SQL with Self-Correction and was created by Mohammadreza Pourreza and Davood Rafiei from University of Alberta. To identify where LLMs fall short, both researchers tested off-the-shelf LLMs like OpenAI Codex’s SQL outputs against the Spider dataset. The incorrect outputs were then categorized to understand where off-the-shelf results fail.

1. Schema Linking: LLMs often struggle with correctly identifying and linking elements of the database schema (like column names, table names, or entities) mentioned in the natural language queries.

2. Complex Queries: LLMs can struggle with more complex queries, especially those involving multiple table joins, nested queries, or set operations. They may not correctly identify all the tables required or the correct keys to join the tables.

3. Grouping: Queries that require a GROUP BY clause can pose a challenge. The model may not recognize the need for grouping or may use the wrong columns for grouping the results.

4. Invalid SQL: Sometimes, the SQL statements generated by LLMs have syntax errors and cannot be executed.

Armed with a grounded understanding of where LLMs fail, they designed the four modules that comprise DIN-SQL.

1. Schema Linking Module: This module identifies references to database schema and condition values in natural language queries. It's a critical preliminary step in most existing text-to-SQL methods. It helps to link the natural language query to the relevant parts of the database schema.

2. Classification & Decomposition Module: This module classifies each query into one of three classes: easy, non-nested complex, and nested complex. It then decomposes the query accordingly. This step helps to break down the problem into smaller, more manageable sub-problems.

3. SQL Generation Module: This module generates the SQL query based on the results of the previous modules. Depending on the complexity of the query, it may use different prompts for each query class. This step is where the actual SQL query is generated.

4. Self-Correction Module: This module corrects minor mistakes in the generated SQL queries. It's a final step to ensure the accuracy and correctness of the generated SQL queries.

Opportunities & challenges in practice

While an 85% score on the Spider dataset is impressive, there are still areas to improve the query engine and other parts of the MBI stack.

• Improved Text to SQL Models: While general models like GPT-4 perform well out-of-the-box, specialized text-to-SQL models will eventually beat them. These fine-tuned models are expected to deliver quicker response times, require fewer parameters, and offer broader context windows for understanding tabular relationships.

• Data Modeling and ETL: Initial solutions in this field will depend on existing data models and cleaned data to deliver analytics. However, it's anticipated that future products will be capable of extracting Common Table Expressions (CTEs) and tidying up SQL files exceeding 1,000 lines and refactoring them into staging tables. These products could potentially generate an ETL pipeline immediately upon indicating an API source for data, thereby streamlining data transfer into warehouses and optimizing the computation of necessary joins.

• Documentation and Dictionaries: Maintaining updated data dictionaries and documentation of past queries has been a persistent challenge for data teams. Given that most data in warehouses are inserted and then neglected due to the ease of data ingestion compared to its cleaning and comprehension, data refactors often render these dictionaries obsolete. The advent of self-maintaining data documentation could significantly benefit existing data teams, serving as a useful reference for language models to grasp and adapt to the specific terminologies used by distinct teams.

• End-to-End Decision Making: While text-to-SQL accuracy is a noteworthy development, the substantial value lies in enabling non-technical individuals to pose abstract questions like “What should I do today?” and allowing the product to deduce the user’s priorities, propose data-driven queries, interrogate the data, perform the analysis, and present decision recommendations with accompanying trade-offs. The new code interpreter module from OpenAI that analyzes CSV files using Python represents a fresh approach to this area.

Notable companies: incumbents & startups

Incumbents

• ThoughtSpot unveiled Sage (March 2023), initially powered by using GPT-3 and now GPT-4.

• Tableau/Salesforce announced Tableau GPT (May 2023), which is powered by Einstein, the AI platform powering Salesforce’s portfolio of products.

• Microsoft Power BI announced Copilot for PowerBI alongside a unified analytics solution Fabric (May 2023). The Fabric solution is their “full stack” MDS, combining disparate data subsystems from data storage to data connectors to PowerBI into a single UX.

• Google Looker has "Ask Looker" (January 2022), offering users the ability to find and visualize data insights through natural language queries. Surprisingly, nothing has been announced yet regarding the use of foundation models explicitly. But given Google Cloud’s momentum, we can expect something soon.

New crop of early-stage startups

The past couple of months have seen an explosion of activity as new startups experiment with the latest models. While the dynamics of generative AI gives incumbents the benefit of more data and wider distribution, the modularity of the MDS has given startups potential foothold to wedge into. Instead of creating a market map of all companies, which is expanding every day, we are instead highlighting four notable startups approaching the problem space in distinct ways:

• TextQL is building a fully virtual data analyst, Ana, that can navigate the MDS. It’s cataloging and enriching the semantic layer, then using that as a reference to write accurate SQL and Python to return business insights to end users. Approach: serve business users via a chat interface

  

• Equals is a modern spreadsheet that can do what we expect from Excel & Google Sheets but its also albel to pull from data warehouses and other systems via SQl or their native integrations. It recently announced Command+K, which allows users to smartly fill a spreadsheet using existing data on the spreadsheet. Approach: serve end users via a spreadsheet interface

  

• Hex is a platform designed to simplify the work of modern data teams by streamlining the entire analytics workflow. Hex users can mix and match SQL, Python, and no-code in its “polyglot” notebooks, with a built-in AI assistant that can generate, edit, debug, and explain code. Hex allows users to build dashboards, reports, or interactive tools and publish them with a click. Approach: serve data scientists in a notebook IDE

  

• Numbers Station is automating the data stack by first focusing on the data transformation steps. With natural language, users can build transformation pipelines and export it into dbt scripts. It can also perform intelligent data munging such as classifying the data of other columns and fuzzy matching records across data sources. Approach: serve data analysts & analytics engineers

  

Company feature: TextQL

TextQL is building a fully virtual data analyst that can navigate the MDS. It’s cataloging and enriching the semantic layer, then using that as a reference to write accurate SQL and Python to return business insights to end users.

Ethan Ding (CEO) was the data product manager at Tackle, where he grew his team of data engineers from 4 to 12 and built across snowflake and redshift, dbt and Astronomer, in-house data catalogs, and all BI tools. During his time there he ran every SQL report for the business to save engineering time for six months

Mark Hay (CTO) was a Senior Software Engineer at Meta for three years, where he worked on ML and Heuristic Abuse Detection Systems, processing over 50 million queries per second, maintained the largest Haskell Deployment in the world at over 7 million lines of code, and optimized ML infrastructure to save over $1.5m a year in computing cost.

Interview with Ethan Ding, co-founder & CEO of Generational

Q: What is TextQL?

Imagine a data analyst - they can document your data, write queries, ask you what you mean by your question, find the right dashboards, and create reports with provide recommendations for what you need to do. We’re building that, but our analyst doesn’t sleep, doesn’t need to context switch, and answers all of your questions pretty much instantly.

Q: Why did you start TextQL? What’s the origin story of eventually getting conviction to start a company? Especially in a space where there’s a lot of incumbents

While I was at Tackle, I was doing all of the SQL-monkey-work for the whole company - and I realized I could automate myself out of that role once I came across the improving performance of language model’s code generation abilities. Then I looked at the incumbents and challengers and realized that everyone was building with and for the models that existed already, instead of looking at the slope and building forward facing products.

When we forecast 10 years in the future, I don’t think the role of the data analyst, someone who works on top of a layer of cleaned dbt tables and communicates with business users about what they need, is going to exist anymore. Every single piece of that, from the question clarification, dashboard navigation, SQL writing and Python analysis will be automated. From there, we worked backwards to figure out how to build that future.

Q: How does TextQL work?

TextQL’s product has 3 parts, which can be thought of as the rails, the train, and the station.

The rails is the catalog and semantic layer, we document every dbt model and column in your warehouse and then enrich that information in the semantic layer. We then store all of that information, alongside an index of your BI tools within a data catalog.

The train is the query engine that uses a vector database to identify the correct output from your data assets to return to a user. It gets you the table and data you’re looking for.

The station is our python interpreter, it picks up the table output and analyzes it for leading indicators, linear regression, predictive analysis, and anything else an analyst can do in a notebook.

Q: Why couldn’t I just grab open source models out there or use ChatGPT?

While the models seem good enough on academic benchmarks, they don’t have infinite context windows. Relying on the metrics or semantics layer adds that additional guardrail, and we can return existing dashboards when that’s the right solution (as opposed to trying to generate new SQL every time).

Also - do you really want to rebuild everything we’ve built from scratch? There are a dozen integrations, metadata management, context window management, model tuning, validation test set question generation, validation testing, model selecting, vector indexing, and the list keeps going…

Q: Anything proprietary you’re doing differently than everyone else?

There’s no other team out there that’s enriching your documentation and metadata, indexing every single asset, storing that in a language model-friendly way, and then using that to power analysis on the other side.

Our secret sauce is our Capsule (name still pending). When we were taking a customer live, we realized a lot of domain-specific information easily maps onto a semantic layer. We invented a Capsule, a unit of domain-specific metadata that we can synthesize from every conversation you have with TextQL - making it more intelligent every time you interact.

Q: What’s your wedge into this market?

With a lot of our peers, they often market their value as sidestepping the data and BI team directly or indirectly. But all companies will have data teams that will be the custodian of the data. As someone who ran the data team at Tackle and run their SQL reports, a text-to-SQL solution is not enough. We will build deep integrations with their semantic layer (dbt, Coalesce) and data catalogs (Alation, Collibra), ensuring that the definitions we use are the same as the ones they’ve laid out. That model will win over the trust of data teams in the short run by delivering accurate summaries of their past work.

Q: Data analysis is probably one of the most crowded application space for generative AI, maybe 2nd or 3rd to writing assistants. What are you worried about?

I’m doing this because I think the future will be a world where anyone will be analyze their data without any technical skills - no more bottlenecks on the data team for other people to get the analysis they need. If someone else cracks the perfect experience that solves the problem for everyone - I’d be perfectly happy with that.

This is a massive opportunity and a massive problem, and I think we have a solid shot on goal here. Is it a layup? Probably not, but I’m excited never the less.

Subscribe now

PDF Version: Download

Since this primer was first written in August 2022, there have been major developments from massive funding for startups like Jasper to expanding applicability of foundation models to actions to the wildfire popularity of ChatGPT. OpenAI's release of DALL E-2 and ChatGPT made AI cross the chasm towards the general population. Interacting either products kindled the imagination of individuals across generations. 2022 is marked by generative AI applications but foundation models is a broader term with wider applications and larger opportunities.

This edition adds:

• Generative AI highlights and examples

• Notes on the top research labs, companies, and products

• Expanded market map with 180+ companies

• Practical learnings from prototyping GPT applications

• 2023 predictions

You can view the presentation or find a PDF copy below.

• Google Slides: Link

• PDF: Foundation Model Primer for Investors and Builder (Dec 2022 Edition) .pdf

5 MB

What are foundation models?

Foundation models (FM) are a class of machine learning models that are trained on diverse data and can be adapted or fine-tuned for a wide range of downstream tasks. The term “foundation” is controversial among some researchers, but setting aside disagreements over terminology,  these models already have had a significant impact. They are already used in large-scale applications in various areas including search, natural language processing, and software development.

Scale is a key element of these models. Scale is enabled by improvements in hardware, the emergence of powerful models that can be parallelized (transformers), and the availability of large amounts of data for training (typically by self-supervised learning). In Figure 2 below, we show some of the open data sources used in training a large language model. The fact that such models are trained on a broad knowledge base makes them adaptable to a variety of downstream tasks.

To illustrate their broad utility, we present a list of common NLP applications that foundation models are being used for today. Many of these applications (Q&A systems, chatbots, etc.) are well-known, but others, like code generation, are more innovative.

Why should investors and builders care

1. Foundation models are already powering the products we’re using daily: BERT is used in Google’s search engine. As we show in Figure 4 below, foundation models are not novel technologies looking for problems to solve. 

2. Foundation models accelerate product development: In areas where foundation models are available, the focus shifts from training models from scratch, to acquiring data to fine-tune models for specific applications and tasks. FMs are also helping software engineers write code (e.g. Github Copilot). 

3. Underlying technologies have made significant advances and models will only continue to improve: Google’s PALM was shown to be better than the average human across a wide range of language benchmarks from Q&A to code explanation to logical deductions. And researchers are just getting started: experiments with more efficient algorithms, new distributed computing tools,  and multimodal models continue to yield steady improvements. 

4. Foundation models could lead to more pegacorns: Deep learning breakthroughs led to startups with significant revenues (“pegacorns”). Most of the AI Pegacorns we uncovered are vertical or domain specific applications, not general purpose platforms We believe that foundation models will spur a similar wave of successful startups. 

Startup Ecosystem

Given the potential of FMs, investors have begun funding startups in this space, most notably in companies building copywriting software. In Figure 5 below, we list over 60 companies that have raised over $3.8 billion to date and have been publicly described as using or building foundation models, which include large language models like GPT-3,  Jurassic, etc.

Implications for product builders, entrepreneurs, and investors

• Experiment with foundation model APIs: Decent models are becoming more available through SaaS providers or model hubs. You’ll likely still need to tune these models for your specific use case and application. Fortunately, computer vision and language models can be fine-tuned with fewer data to reach the level of accuracy needed for real-world use cases. “Off-the-shelf” models will also improve over time as vendors deploy larger base models.

• Don’t confine yourself to text-based models or products: While NLP is the area where FMs have led to the most impressive applications, there are FMs in computer vision and speech that are beginning to lead to equally impressive applications.

• Revisit your assumptions on what’s possible to build: Chatbots were the rage a few years ago but state-of-the-art technologies failed to live up to the hype. This created a stigma for chatbot startups. But with foundation models, creating more useful and more robust chatbots is now possible. For example, the CoQA (conversational question answering) benchmark measures the ability of models to understand a text passage and answer a series of questions in a conversational fashion. Several models have beaten the CoQA human benchmark since 2019.

• Anticipate second-order effects: As applications built on foundation models grow, infrastructure, risk mitigation, and security needs will too. We’re still in the early phases of tackling questions like how to retrain models with new data efficiently, how to distinguish between machine versus human generated content, and what risk mitigation tools and processes will be needed.

Here’s the slide deck version of the guide where you can also find the spreadsheet of 60+ companies in the market map.

Subscribe now

Share

The shift of data to the cloud is massive. Gartner estimates that the database management system is $80 billion with the cloud segment contributing ~50% or $39.2 billion. The market grew by $14.5 billion (22.3%) in 2021 and 90% of the growth came from the cloud.

You can see the momentum of cloud players not only from the chart above but also from the shift in market leaders. Microsoft Azure, Amazon Web Services, and Google Cloud Platform (the cloud trinity) overtook Oracle, IBM, and SAP.

To put the opportunity into perspective: Startups often want to be in a $10B market because capturing just one percent means a $100 million revenue - the threshold to becoming a mature stand-alone company. Yet the cloud database segment grew by $13 billion in just one year.

One driver of this growth is software companies integrating with and building on top of cloud databases. Having connections to the cloud trinity data services are table stakes, but what I’m seeing in the startup scene is a quinternity with Snowflake and Databricks filling out the group.

Databricks and Snowflake as data platforms

Excluding the cloud trinity, Snowflake and Databricks are the largest cloud data vendors. Snowflake’s last fiscal year revenue was $1.2 billion. Databricks’ ARR on December 2021 was $800 million, up 80% YoY. Assuming the same growth, Extrapolating 40% growth since then (another half a year of 80% growth), the estimated current ARR is $1.1 billion. For simplicity, let’s say ARR = revenue (it never is), both Databricks & Snowflake command ~3% market share each in the cloud segment. As the cloud segment grows, I’d guess that the top five DBMS players will eventually be AWS, GCP, Microsoft, Databricks, and Snowflake.

In June 2021, Snowflake launched a program called “Powered by Snowflake” with five founding participants selling applications built on top of Snowflake. This excludes technology partners in adjacent data engineering spaces like data catalogs. The program gives companies dedicated support and co-marketing opportunities to build and sell data products using Snowflake. The list has grown >9x in less than a year to 47 today with use cases from education to manufacturing. This understates the scale because there are a lot of companies that directly integrate with Snowflake, like Kubit which provides product analytics for the modern data stack, but are not part of the program.

This makes Snowflake a data platform by definition: 3rd party software developers are increasingly technically and economically dependent on Snowflake. What about Databricks?

Compared to Snowflake, which made the product generally available in 2015, Databricks is newer to the data warehouse scene:

• 2019: started the Delta Lake project

• 2020: popularized the Lakehouse concept

• 2021: announced Databricks SQL general availability, with a vigorous response from Snowflake

While Databricks has catching up to do, I’m looking forward to the applications built on top of Databricks which has historically been treated as a data lake where unstructured data is funneled to. Think images, videos, audio, etc. The breadth of use cases expands exponentially vs. building with structured data alone.

What about OLTP vendors like MongoDB? They’re great but the value of OLAP databases like Databricks and Snowflake as platforms is bringing data from several sources into one accessible interface. Developers need to combine multiple data sources, like correlating click streaming data and customer information coming from Salesforce to highlight product qualified leads. In this context, OLTP is more a data source than a database.

SaaS 3.0: Data applications

Bastardly simplifying the technical SaaS stack, what we’re seeing is the third generation SaaS architecture. SaaS started out siloed, then became integrated via APIs, to now being integrated via data platforms.

This makes sense from a macro-efficiency standpoint. For n SaaS apps in the world, (n-1)!! - double factorial - integrations are needed to connect all apps in the SaaS 2.0 world. And n x m integrations in the SaaS 3.0 world, where m is the number of data platforms.

Software developers can dedicate more engineering efforts toward novel applications that can be built with centralized data. So far what’s bubbling in the startup scene are tooling for new GTM motions (e.g. product-led growth), new billing models (e.g. usage-based billing), and accurate real-time analytics (e.g. full sample product analytics). At the same time, it will be a bloodier competitive space for SaaS companies because data gravity will weaken, lowering switching costs.

But it is great news for the data platforms, they are going to eat a larger piece of the pie.

Till next time, Kenn

Subscribe now

Share