In this interview, I speak with Zep’s CEO, Daniel Chalef. Zep, built on top of Graphiti, powers AI agents with knowledge graph memory built from user interactions and business data. I first came across Zep more than a year ago. I was looking for a memory solution beyond vectors and they stood out as the only one considering the temporal nature of data. Graphiti is now one of the fastest growing open source projects — check it out.

Key learnings

• AI memory can't be generic—it has to understand your business: Daniel learned that memory systems need to be domain-specific because a mental health app tracking emotional states and therapeutic progress has completely different memory requirements than an e-commerce agent remembering purchase patterns. The entities and relationships that matter are fundamentally different across domains. Generic memory solutions fail because they treat all context as equivalent.

• Customer intimacy beats feature completeness in AI: While most B2B companies focus on features, Daniel focuses on customer intimacy—the agent's ability to respond accurately and personally. This drives conversion and retention because generic or hallucinated responses erode trust. For AI products, trust is the entire game, and personalization is how you build it.

• User preferences have expiration dates that embeddings can't track: Vector databases treat information as immutable, but real user data changes and conflicts over time. When someone says "I love Adidas shoes" but later "these Adidas shoes fell apart," temporal knowledge graphs can invalidate the old preference while vectors just store both as independent facts. Time-aware context is crucial for AI products.

Subscribe now

What is Zep?

Kenn So: I reached out because I'm a fan of what you're building at Zep. Super appreciate you taking the time. I was looking through LinkedIn — you wore so many different hats, which is very interesting. I'll poke around that during the interview. To get started, can you share about yourself, the company, and the open source project?

Daniel Chalef: So I'm an engineer turned founder. Zep is my second startup. It's also my second open source startup. In between the two startups, I took on many different roles—led very product-focused ML and engineering groups, as well as marketing, as well as corporate development teams.

Interesting set of experiences. The through line to my career has been I've always worked on very technical products that were being sold to technical buyers—engineers, data science leaders, ML leaders, etc.

Zep is near and dear to my heart because it's actually an agglomeration of a bunch of my experiences across both ML as well as some of my marketing experiences. This will become apparent as I unpack the vision for Zep and where we're going.

You can think of Zep as a memory layer for the agent stack. We think more expansively around what memory is and how we can help our customers.

Memory—we need it because without memory, we can't adequately personalize responses for an agent attempting to solve a problem for a user or respond to a user. If they don't have that context, they don't have the personalized context, then they're going to hallucinate at worst or at best respond generically.

And that doesn't do what's important for our customers, which is solve problems and drive customer intimacy. We work with a lot of companies who really view customer intimacy as a key to building a successful agent.

What I mean by customer intimacy is that the agent is able to respond accurately in a personalized way. Why intimacy is important is because it drives conversion rates and drives retention. Because if an agent responds generically or hallucinates, that erodes customer trust, which is the antithesis of customer intimacy.

We primarily work with B2C companies, many at scale. You can see why there's that intersection now with customer intimacy. Everybody from consumer mental health apps all the way through to coaching apps, through to e-commerce, through to—we're even working with, we keep it relatively generic, big consumer goods companies.

Where Zep is different is that we view memory as being expansive. The types of companies that we work with have many touch points for the user beyond just the agent. So there might be email, support cases, billing data, data coming from SaaS applications, so actual events—a user did X or a user did Y.

We allow developers to send us a stream of both chat messages and business data. We call what I just described this broader context—we call it business data.

We then integrate it together on a knowledge graph. It's not just any knowledge graph, it's a temporal knowledge graph. So we're able to understand how user state changes over time. Preferences might change over time, and there might be conflicting information that we later receive that we then have to integrate into the graph.

By using the temporal metadata that we capture, we're able to correctly integrate conflicting information into the graph. For example, invalidate an old preference:

I love Adidas shoes, but because my shoes fell apart, I might be angry. So Zep understands the emotional valence, understands the context, understands the temporal dimension, and is able to say, oh, that old preference is now invalid.

We essentially view Zep as more broadly than just memory. It's a customer data layer for the agent stack.

When we think about memory, we also have a very strong perspective that memory is not one-size-fits-all. Memory is very much a domain-specific implementation.

The types of things that you care about as a developer of a mental health application are going to be very different to the types of things you care about as an e-commerce agent developer.

So we allow developers to define custom domain-specific entities and edges in the Zep graph. They can capture specific things—things like media preferences or shoe preferences and retrieve those more deterministically from the graph with higher data quality and higher accuracy.

How it all started

Kenn So: There's a lot to dig in there. I saw the entity types blog post, which was great. But I want to take a step back into maybe the beginning of Zep as well. From LinkedIn, I saw you started March 2023. Walk me through that. Those were really early days post-ChatGPT.

Daniel Chalef: It was crazy because it was just after the seminal agent paper. When I started really thinking about this was before we called agents "agents."

I'd been using language models back to BERT and BART and others back in 2017, 2018. I'd been thinking about them for a long time and independently thought that we could be building something that could work autonomously.

I started to build an application that would take boring old policy and process documents and turn them into living and breathing—not only, it was during the vogue of the time of GPT-3.5, and there were all the chat over documents applications, Q&A over docs, and I was like, oh, come on, there's no defensibility here.

I wanted to build something that was far more compelling—an agent that would actually take and consume documents and turn them into a process.

There's no state. These applications, the models were stateless. I would have to determine what goes into a prompt. Prompts were small. The token windows were very small then, like 4,000 tokens. I had to work out how to structure that, and I built something that would basically summarize conversations, and I open-sourced it, created LangChain connectors for it, did a bunch of work with LangChain, and actually became a top 10 contributor.

That's how I got to memory—thinking about memory—at least. As I started thinking about the opportunities for what we would be building, I started realizing that there are all sorts of—given my experience with ML and at scale—there are all sorts of pipelines that we need to build to make memory into something and then retrieve memory. Memory shouldn't just be the conversational domain. My application is emitting data as well. So what do I do with that?

How do I contextually recall the right data? I can deterministically pull in data from a Postgres database. But sometimes I don't know what data I should pull in and when. So that non-deterministic part was—the things I don't know at development time and situations that I can't anticipate.

That is where agentic behavior really came into the fore and where having a memory service that allowed agents to recall both past interactions and business data was really useful. One thing led to another, got into YC, raised capital, built a service, a multi-tenant service.

People started using the product like you and started getting a lot of enterprise inbound as well. So we have tons of pilots now with large enterprises and, yeah, it's been exciting.

Why graphs beat vectors for memory

Kenn So: Going a little bit technical, there's this philosophical debate around knowledge graph versus vectors and knowledge graph that was even in the early days of 2023. Was the knowledge graph structure something that you honed in early on?

Daniel Chalef: We knew pretty early on that it was going to be very difficult to capture enough context, temporal and relational context, in a vector database. Vector databases, embedding vectors, exist independent of each other and are, in many respects, immutable. There's no way that you can understand causal relationships between vectors in a vector database.

We initially started with vector search, but knew that it was inadequate and attempted to work out ways that we could understand causal relationships, understand temporality at more depth. Having worked with graph databases in the past and graphs at scale, I knew that the intersection of graph and semantic would be a very powerful approach to memory.

What Zep does today, by virtue of leveraging Graphiti, is it uses both BM25 and Semantic Search to find subgraphs within the broader user graph or user record, and then uses traversal to retrieve relevant recent memory, so that an agent can then reason over the causal and temporal nature of memory that it is presented with, which allows agents to do things like root cause analysis, because not only are they getting, "I can't log in," oh, their user's account is suspended, but they're also getting, "we couldn't charge the credit card."

And why couldn't we charge a credit card? Because the credit card is expired. And you can get that with the graph, because you can look at the relationships.

Kenn So: The thing that drew me to Zep is how you have productionized knowledge graphs for AI. The graph construction , you can kind of hack it together, but building the pipeline to constantly update the knowledge graph is hard. And that's something you’ve built.

Daniel Chalef: Yes. And it was fun building it. We decided to build the heart of Zep as open source, the Graphiti graph framework. We knew it had broader applicability. So we have users building sales intelligence companies. Why? Because mapping companies, the relationships between companies, the employees of companies, how their roles change over time, how they might move to a different company is really useful on—a temporal knowledge graph is really useful for that use case.

Teaching customers to use the right tool

Kenn So: Well, let's go down this thread of technical products. Do you find your customers using Zep along with a more traditional database of storing all the chats and pull them both together? Or do you find them just pretty much sticking with Zep to feed all the contextual information into a model?

Daniel Chalef: We have customers who attempt to overuse Zep. They want to use it for relational data. They want to use it for high-frequency event data. They want to use it for RAG. And, you know, we're often telling them, if the data structure is known at development time, if the data itself is known at development time, and there are simple pipelines to update that data, and it is something that is either an immutable data structure, or is an append-only log of sorts, use a relational database.

You can more deterministically retrieve your data from a relational database, and it doesn't need to be on a graph.

Just because graphs are cool, doesn't mean you need to use a graph for everything. Ditto, we have customers who try and do RAG with Zep, and it's not designed for that when you need to recall the original chunks of your data. Zep can do that, but it's not really designed for that. It's probably better for you to use any number of RAG databases if your data is not dynamic, i.e. it never changes. Then you might as well just vectorize it and put it into a vector database. So we spend a lot of time educating customers.

When open source goes viral

Kenn So: It's super interesting. There are two topics I want to move onto. One is, it seems like Graphiti is on a rocket ship. What's happening there? And how are you keeping up with it?

Daniel Chalef: Oh, man. We're not keeping up with it. There are way more issues than we can handle. It's been challenging.

I think it's just captured imagination. You get into, with an open source project, you sometimes get into a strong virtuous cycle. Somebody might build something cool. And other people think it's really cool and it goes viral on X or wherever and lots of people star the repo, then other people write about it and you might get onto Hacker News or something and you get even more stars.

So you become a project of the day and the more people find out about it because Github emails everybody saying this is the project of the day and then you might stay up on project of the day over an entire weekend and then other influencers might hear about it and then they build things with it and you just get into a positive cycle around a project. And then it gets to a certain size where it's got some gravity to it, and it just keeps rolling.

Kenn So: It does. Now it has 11K stars. Has that translated to enterprise business pipeline?

Daniel Chalef: It has. We have some very large companies who reached out to us struggling with the temporal nature of memory, and thought that a graph database would be better, and then came across Graphiti, and then came across Zep, and now we're talking about pilots with them.

Kenn So: That's exciting.

Daniel Chalef: Yeah.

What's coming next

Kenn So: You're going to be very busy. So what's next for Zep? What can you share about what we should be looking out for?.

Daniel Chalef: There's a lot of work to do when it comes to enterprise adoption. So we're putting a lot of effort into that. We're SOC 2, Type 2 compliant. We have a BYOC offering. And so there's a lot of engineering and infrastructure work that we're spending time on.

That said, we recently rolled out the ability to customize Zep for your domain. And so there's a lot of work going into that. We actually rolled out this past week, the ability to build custom edge types as well.

Kenn So: So you can build custom relationships between nodes now. Interesting.

Daniel Chalef: Yeah. And that's experimental. We're constantly working on latency. Recall latency is down to 200 milliseconds for our hot path, memory.get API. So, we're working with voice agent platforms at the moment on that.

Wearing multiple hats as a founder

Kenn So: We’re almost out of time, any call to action for the readers?

Daniel Chalef: I would just make it checking out Graphiti. A lot of our marketing focus has gone into Graphiti, and Zep kind of comes in as a byproduct. I don't know if you saw my talk last week.

Kenn So: At the AI Engineering conference? I haven’t. I saw you were presenting but I didn't get to attend your talk.

Daniel Chalef: You missed it — the talk was apparently voted best of the GraphRAG track. I have a trophy that I have to go fetch.

Kenn So: Ha, nice. Actually, another (last) thing I really want to ask you is you've worn many different hats, head of marketing, corporate development, founder, engineer. How has that led you to think about how to build Zep?

Daniel Chalef: It's given me pretty deep insight into how to market Zep. I've been close to the go-to-market function, so how to sell Zep as well, what enterprises need. I've worked in—got a lot of enterprise DNA, worked in enterprise software companies, coupled with an ML and engineering background, understanding both how it works, but also what our customers might need and what resonates with them has been very, very effective.

Kenn So: Lots to unpack there and we could go for another 30 minutes. But we can end it here. I really enjoyed the chat. Thanks for taking the time, Daniel.

Related posts:

In this interview, I speak with Kapa.ai’s CEO co-founder, Emil Sorensen. Kapa helps companies answer technical product questions from their customers & developers, instantly and accurately. I first met Emil when they were presenting their hack in the early days of 2023, not long after ChatGPT came out. That hack evolved into the company that is now serving 150+ customers including companies with world-class engineers like OpenAI.

Key learnings

• Focus Beats Breadth: Emil emphasized the power of staying laser-focused on technical documentation rather than expanding into general-purpose chatbots. This narrow focus allowed Kapa to build deep expertise and better serve their specific audience, competing effectively against broader platforms like Intercom and Zendesk.

• Build Your Own Evaluation Framework: One of Kapa's biggest advantages is their comprehensive evaluation suite developed over two years. This "secret sauce" allows them to quickly test new models and features against their specific use case, staying grounded amid AI hype cycles.

• The Last 30% is the Hardest: While talented in-house engineers can build a RAG system to 70% quality, the remaining 30% to reach production-quality involves security, rate limiting, front-end interfaces, and operational excellence that require focus and time.

• Quality of Data Sources Determines Everything: The biggest determinant of answer quality isn't the underlying model but the quality of documentation and data sources. Kapa turns away customers whose documentation isn't ready.

• "I Don't Know" is a Feature, Not a Bug: Users highly value when AI systems admit uncertainty rather than hallucinating answers. This builds trust and makes the system more reliable for technical use cases.

• Small Teams Can Build Big Things: With just 15 people, Kapa serves 150+ companies and has answered 5+ million questions. The key is finding talented people who thrive in high-impact, chaotic startup environments.

There's so much more to learn and unpack from my chat with Emil. Read on below to see what's next.

Subscribe now

Kenn So: Thanks for taking your time, Emil. This is two years in the making, so I'm excited we are here today. For the audience, let's start with the usual - what is Kapa? What's the origin story? How did you get to start Kapa and what's the journey been so far?

Emil Sorensen: At a high level, Kapa is a platform that makes it really easy for technical companies to build RAG assistants and deploy them where their users or employees have questions. What this looks like in a lot of cases - take someone like Docker that uses Kapa - is an AI system that lives on their documentation. For other folks like OpenAI, it might mean they have an active Discord community, so a version of Kapa lives in there to help their end users navigate the thousands of pages of content that the OpenAI team produces.

Now, two years in, Kapa is used by more than 150 companies, has answered more than 5 million technical questions, with lots of cool companies like Reddit, Docker, OpenAI all using Kapa.

Kenn So: Super exciting to hear 150+. And you've got some of the most technically minded companies who are at the AI frontier using Kapa. Was that how you envisioned it right from the very start, or how did it evolve over time?

Emil Sorensen: One thing I don't believe in a ton is this concept of a founder waking up one morning and seeing the light and coming up with a mission. I'm much more in the school that it's exciting to work on stuff that's used by people.

It's a good bridge to maybe the founding story, which is actually pretty straightforward. Finn and I are very close friends. We both met in London for our first Master's degree, thinking we would go study finance at LSE, which we did. Fortunately, just before that I'd spent a summer in San Francisco interning at a startup. Still no idea how I got the gig, but I was working as a self-taught engineer writing some terrible code.

I came back and met Finn in the line for picking up our student cards. I was telling Finn about this startup experience, and he was like, "Oh wow, that sounds awesome!" Our friendship started there. We got excited about all this stuff and looked at each other a couple of weeks into this program going, "What are we doing trying to be bankers in London? This is a horrible idea." But we saw the degree through and in parallel convinced ourselves to go study computer science instead when we were done.

We went to do a conversion master's in computer science, which apparently in London you can just study for a year.

We didn't have any ideas when we were done with our degrees in 2019, but around that time we'd been tinkering with embedding models and fine-tuning early GPT versions to try to write abstracts. None of it worked, but our intro to deep learning class, they used OpenAI's reinforcement learning work - I think it was Dota where they showed how you could train models. We were looking around this stuff, and they were teaching us how to write basic neural nets from scratch. The attention paper was coming out around this time, but there wasn't much to it.

Kenn So: So early though, 2019. GPT-3 might still be training then and only a few people know about it. How did you kind of get into those early explorations?

Emil Sorensen: It was just around this time that the attention paper was coming out. There was a little bit of chatter around this, but the only motivation we had for tinkering with this was maybe it could write our abstract, which it couldn't. It was horrible. And then we moved past it.

Kenn So: How'd you go from that to McKinsey?

Emil Sorensen: Very personal reasons. At the time I was living in London, but my girlfriend, now wife and soon to be mother of our first child, lived in Copenhagen. If you're kind of ambitious in Copenhagen, there isn't this big scene of cool tech companies to go work for - you go work for McKinsey. As a lack of inspiration, I would have started a startup then, but Finn and I didn't have any good ideas. So he went to work as an engineer for a couple of years while I was at McKinsey.

Kenn So: What then led you to go from McKinsey to deciding to start Kapa?

Emil Sorensen: For a while Finn and I - because I was at McKinsey for three years before we quit - we really tried actively on weekends and early mornings to think about what startup we wanted to build. We wanted to try to build something together, but it was like, "What are we going to build? We should do some customer interviews." But when you're working 80-hour weeks flying everywhere as a consultant, you just don't have time for this.

We met over summer in Austria to go mountain biking for a week together, and around then Finn started flirting with the idea that maybe we should just quit and give ourselves a bit of time. We left that vacation going, "Maybe quitting first of December doesn't sound too bad." We said, "You know what, it's now or never. We're in our late twenties. If we don't do this now, it's going to be way harder." And then we just quit.

We showed up and started brainstorming mid-January. We bought a whiteboard, and the day after we bought that whiteboard - timeline context, this was mid-January, and by early December ChatGPT had launched. Ada embeddings came out December 13th or 14th. Around this time we were reading Hacker News, and people were beginning to say you could finally build these systems. They didn't call them RAGs then, but these systems where you pull data back from a database. Super lucky with the timing, to be honest.

The day after we bought a whiteboard and started brainstorming without any ideas, we were really lucky to have two friends that each had co-founded dev tools. One was the guy we met while studying computer science in London - he had a YC-backed computer vision company. Another friend, Nick, here in Copenhagen who I worked with at McKinsey, co-founded Medusa.js, a big headless commerce platform with tens of thousands of Github stars.

Both of them called us on the same Tuesday saying, "Hey guys, we know you quit, you're looking for ideas." Both were complaining about the same thing: their engineers were spending so much time on Discord and Slack communities answering technical questions that maybe with this new ChatGPT stuff might be solvable now. "Could you guys try ramming our documentation and some GitHub issues into some sort of assistant?"

We spent time building a prototype and did that. That's the origin story.

Kenn So: Such an amazing founding story. So Kapa started there, then you went through YC two years ago. There's so much to unpack in between then and now. Maybe let's start with the product. How has the product evolved?

Emil Sorensen: The interesting thing is at its core, not much has changed. We still try to do one thing really well, which is: you give us all your technical content - all your developer documentation, all your API references, all your GitHub issues. And we'll do the best job possible of letting you ask questions about that and generating an answer.

Whether that's using RAG, whether that's using some reasoning-based approach, whether that's using a Claude model or an OpenAI model, or open source or fine-tuned model - we don't really care. All we care about is whatever does the best in our evaluations, and then we'll ship that. We initially struggled with what to call this, but we've settled on "answer engine" as a concept. Question comes in, answer comes out. Our users don't really care what we do under the hood.

That's our North Star. Of course there's been a lot that's happened since then from a product perspective - integrations, platform features, all this stuff - but really at its core, our job is just to give really accurate answers based on the content you give us.

Kenn So: Having interacted with Kapa since the early days of the industry, its fun when I still encounter it in new AI projects I am working. It's still the same experience on Discord, still gives you accurate answers, still says "I don't know" if it really doesn't know. But I am sure a lot has changed under the hood for the past two years. What were the major developments in terms of what made your product harder to build, or maybe made it easier?

Emil Sorensen: Let me break it down into interesting data sources, interesting integrations, and then interesting approaches to the actual models themselves.

From a model perspective, without a doubt, the models themselves getting better is definitely a big improvement for Kapa. Initially, I heard "GPT wrapper" so many times, but that meme is finally starting to decline as people see this is just another piece of infrastructure you need to build on top of and tame, much like you do with a Postgres database.

The tricky part to navigate is, if you're very deep on Hacker News or Twitter, you will freak out as a founder in this space if you don't have anything to evaluate these things against. For us the biggest lift was very early on making a bet that the secret sauce for a company like Kapa is our approach to evaluations.

Building trust in our own eval set so we can constantly, whenever people are panicking about DeepSeek or seeing that Anthropic is releasing a new citations API, very quickly test that for our use case to see if this is actually helpful or not. One thing we really care about is a model's ability to say "I don't know."

Kenn So: That's really interesting. Are there specific instances where evals kept you grounded? Were there instances where you saw the hype was real versus ones that were duds?

Emil Sorensen: The Anthropic team really figured something out with Sonnet 3.5, that was very good. To lift the lid on what's very top of mind for us now, GPT-4.1 is a phenomenal model, but it takes a lot of taming to get it right. For grounded RAG use cases, it's great.

Some very hyped things have been open source models. And it's a bummer as an engineer, you want to see open source catch up. But they've just constantly lagged behind, at least on our evals, by a year to year and a half in terms of performance.

Kenn So: Yeah, evals really help you keep grounded amidst all the hype. If you don't have a particular use case, it's really hard to know what you're going to ground yourself in as a general bot.

Emil Sorensen: Exactly right. Today, when someone like a larger technical enterprise evaluates Kapa, usually the main competition we're up against is a crack team of really good engineers internally that might go, "Hey, it makes sense for us to build some capabilities in-house around this."

We've been able to crowdsource our evaluation suite over close to two years to understand how a model like Kapa is supposed to behave.

It's one of these things where it's very easy to build a prototype that gets you 70% of the way there. It's very hard to get that last 30% to get it to a point where you feel comfortable putting it into production.

Kenn So: That's a point I want to unpack later - you are up against crack engineers who could probably put a RAG model against their data sources. But let's go back to earlier topics. You mentioned data sources and integrations. Maybe we could go there next.

Emil Sorensen: If you think about Kapa, it's data sources in, question comes in, answer comes out. One of the things you can play with is the actual model and how a question comes in and retrieval generation. The other thing you can do is think about data sources.

A good example here is something like a PDF. We work with a bunch of semiconductor companies and all their documentation lives in data sheets and PDFs. Horrendous data format. We've tried now three or four iterations over the last two years of building a PDF converter that we were satisfied enough with the quality to fold into our product.

We've tried coming back to this every half a year, and it's only now in May 2025 that it's gotten to a point where combining OCR and VLMs has finally gotten good enough that we're like, "Okay, this thing we've been working on for two or three months to ingest PDFs is finally good enough to make its way into Kapa's integration."

The reason I say that is because we care about very specific things. Unlike a general system, we care about ability to cite, ability to preserve hierarchy within a big document. Some of these data sheets are hundreds of pages. Once you have folks you're working with, it's easy to go find examples and build a specific eval suite and work against that to build your very niche integration.

It's that story time and time again - how do you think about GitHub? How do you think about Zendesk support tickets? How do you think about all these other technical sources? Which is cool because we can afford to do that because we're so focused on just working with technical companies. But if we positioned ourselves as a general-purpose chatbot that's also supposed to do support for e-commerce companies, there's no way we can invest this amount of time into parsing through and structuring 100+ page long technical PDFs.

Kenn So: That's a good segue to the point of you being up against world-class engineers who can probably do 70-80% RAG off the bat. What's that 10-20% that you're so focused on? What makes it really hard and tricky that you're focusing on?

Emil Sorensen: I don't think there's any big secrets other than it's just the engineering slog around it. What we often see is a team of really good engineers can definitely build something with pretty good answer quality. Sure, we can poke at edge cases where Kapa is better, but very quickly, this team will have spent half a year building something like this.

Then they realize, "Oh, at some point you also have to take this to production. How do people want to consume this?" They want to consume this as a chat widget on your website. Well, do you also have a bunch of front-end engineers who are ready to build that? How are you thinking about security? You can't just expose your chatbot for anyone to inspect the network request because they can hammer it with a million requests and bankrupt you overnight. You have to think about proxies and rate limiting and reCAPTCHA.

There's all this other stuff that becomes table stakes. To be honest, I'll try to check my own bias, but if all you're trying to do is build a system that's able to answer questions about your product that has access to your docs, it probably doesn't make sense to use your very expensive engineers that are able to build large language models to be building that.

Instead, what you should do - and we can talk about a few examples - is do what a company like Docker does. They have some really talented, very capable engineers that are thinking about how to build Docker support for MCP servers. Go do that! That's really good. Or how do you build a Docker Desktop AI assistant that can look at your docker-compose file and take action?

What they do is say, "Well, we still want this exposed as a chat interface, but we don't want to build and maintain something like Kapa. Let's just use Kapa as a tool call in that case." That's the product approach which is just part of a stack. For most companies, that's probably the path to go.

Kenn So: When you work with customers, what does onboarding look like? Is there any sort of fine-tuning or evaluation that you do for each customer? I'm curious because even within technical companies, there's a wide range from hardware to software. Even within software, you have map APIs versus JavaScript front-end types. How do those change how your product is experienced?

Emil Sorensen: On the onboarding aspect, that's really company dependent. We let folks self-onboard that are very capable and really want to do that. We also lean in a lot because one thing we've realized is how you set up an instance of Kapa - it's not obvious. Just to say, "Do you want to let us slurp up all your GitHub issues?" We've seen a lot of cases now, so we like to share that advice and help folks make sure they get everything configured correctly, because that is the biggest determinant at this point of the quality of answers you'll be getting.

That also answers the second question a bit, which is we try to make Kapa as general-purpose for technical questions as possible. How it performs - by this point, wasn't the case two years ago, but by this point the biggest determinant of that quality is really the quality of data sources.

Unfortunately, just before this I was talking to a customer who decided not to move forward with Kapa, and they were very honest. They said, "We thought we were at a place now with our documentation where this could be helpful enough to answer questions, but we need to come back in a couple of months. We have some work to do on our documentation. It just doesn't cover what our users want to ask."

We try to build some of this into the product to make it easier to find these gaps, but that really is super important.

Kenn So: How do you assess if something's good enough from a documentation perspective, from a data source perspective?

Emil Sorensen: By now, the whole Kapa team can take a look at a set of docs pretty quickly and just kind of sniff it out. But the TLDR is, you don't really know before you try it.

Kenn So: You mentioned trying to help fill that gap for your customers - "Hey, this is probably an area where you need to fill up the docs or some data sources." Hey, this is a good segue into the different modules and maybe vision into what Kapa could be. Because you can do so much with the data you have. You should be able to say, "Based on the questions, this is an API doc area where you need to fill out, or this is where you have the most pain points in terms of which endpoints." I'm curious - you have so much you could extract from the data. What's next?

Emil Sorensen: That's a great question. The short answer is whatever is important to the company that's using Kapa. For most folks, the first thing they get started with from an analytics perspective is starting to build some workflow habits around just on a weekly or monthly basis plugging some holes in their docs.

We try to design the product to lend itself to those workflows so they can go, "Oh yeah, seems like May 2025 is the time I focus on my getting started guide because a lot of folks seem to be struggling with that, or maybe it's time to focus on my React SDK docs because there's really high uncertainty in Kapa's ability to answer questions around this topic" - it says "I don't know" a lot.

For other teams, they take it a bit more at a strategic level, so they'll use it more as a tool to say, "Hey, we're thinking about launching on-prem finally. Do we have any data to support that users are increasingly interested in this or less?" It can also become more of a research tool.

But to be honest, this analytics thing on unstructured data produced by LLMs - that's still a tricky problem to solve. I don't think we've completely cracked the right approach yet.

Kenn So: This is a question I should have asked at the top of the interview - who are you primarily selling to? Which personas? How has it evolved? I imagine it's engineers at first, and now maybe for more mature teams, you have developer experience teams.

Emil Sorensen: You nailed it. That's pretty much it. There are some slight variations. Usually the core unifying thing is it's whoever owns and cares about onboarding documentation, customer education. Turns out this title usually is different from company to company. It's not super unified, but developer experience, head of documentation, technical writer, that kind of stuff.

Kenn So: I'm curious what you've learned from watching them use your product. Has it changed, or is it still fundamentally the same? I know they're a little bit more strict and critical when it comes to the output and will point out inaccuracies more.

Emil Sorensen: That's a good question. Generally, this is just a complete mega trend now of people adopting LLMs into their workflow. Obviously I'm biased because the people I talk to use these in the context of Kapa but folks have multiple tools like ChatGPT, Gemini, and Claude open, and they each have a different mental model of who they like. Claude's ability to write emails, but ChatGPT's a little bit better at meeting notes and so on. So everyone's more comfortable with it.

That's a misconception a lot of companies have when they think about rolling out a system like this - "Oh, people aren't comfortable with LLMs. What if it hallucinates?" Because no system is perfect. But what's the alternative?

A really interesting insight is people really like it when the bot just admits defeat and says, "I don't know." This is one of these things that by sheer luck in prompting that first week or two, pre-launching Discord deployment, whoever stayed up late that night and wrote the first couple of meta prompts had this over-indexed on "I don't know." But that's really just a core product feature since.

Kenn So: It is such a different experience back then of models saying they don't know, because everyone's trying to get the models to do everything and anything, and there was a lot of hype. That was my experience as well. I built a POC internally, and the first thing leaders tried to test was like, "Can it say I don't know, or I can't answer that?" It's a user trust feature that's more subtle.

Maybe we can shift a little bit into building. Any guidance or tactics and insights about how you were able to grab all those amazing 150+ customers, including some of the best ones? I know one tactic I really love, even from two years ago, is you build demo chatbots for prospective customers. That was really impressive.

Emil Sorensen: With the Langchain example, Harrison at the time - obviously we were all in the Langchain Discord and following what's going on and using Langchain first and all this stuff. Early on it's like, "Hey, this seems like a really helpful thing. Let me just send something over to Harrison." He's like, "Yeah, that's cool. Let's deploy it." That definitely helped.

Other things that have really helped - it's been really, really narrow in our focus. A lot of people right now - which I get is natural - think about, "Well, okay, Zendesk has this AI offering and Intercom has this." And they're fantastic. Intercom is fantastic, Zendesk maybe less so. But these systems have to cater to such broad audiences, whereas we really speak to the persona that uses Kapa, which is one that cares about technical documentation.

That has really helped us to also when someone's saying, "I'm not trying to solve your support team's problem or McDonald's support chatbot problem" - some other smart folks are doing that. And they're very different LLM problems you have to solve for that. What I'm trying to do is this.

Kenn So: Okay, maybe since we're talking about how to get customers, shifting gears again into building your team. I'd love to just talk about how you're building your team right now in terms of the skills that you're looking for and any philosophies you're applying to it. Is there anything specifically for AI startups?

Emil Sorensen: It's not going to be anything super novel. We're just married to the YC playbook, to be honest. In terms of hiring, it's very merit-based. It's very big on cultural fit. Startups are not for everyone. The biggest learning here is really for both sides when evaluating whether or not to join a startup like Kapa.

We have people come in and work with us for a couple of days, and we pay them a bunch to do that, which is definitely not the norm here in Europe where we're mostly hiring out of. Just to say, "Hey, this is going to be more chaotic than your job at Google. The amount of impact you can have will be 100x outsized compared to what you're able to do now." Some people just thrive in that context, and other people just like going back to their eight weeks of vacation a year here in Europe and 37-hour work weeks.

That's been a really important element too. In terms of AI focus, maybe an interesting thing from an engineering perspective is you have a bit of a split in terms of engineering preference. Most of our engineers - all of our engineers are phenomenal. They're really very high - I'm sure every founder will say this - but very high talent density team. But we have a couple of guys who just hate dealing with LLMs because of their non-deterministic nature and would get so much more joy shipping a new integration or shipping a new feature on the platform because you see the output, whereas other folks that are more research-minded just love wrangling in this nasty LLM space and can easily spend one or two weeks and don't get deterred when they don't see scores moving up or something like that.

Kenn So: That's so interesting. And then, how big is the team now?

Emil Sorensen: About 15, some part-timers here and there.

Kenn So: That's a lot of customers for 15 people. I know it's software, so there's that ratio thing. But it feels like you've gotten so much done with 15 people. Was keeping it lean kind of intentional? Because you've also raised venture funding, so you have the money to continually hire and potentially drive growth.

Emil Sorensen: I don't know, maybe let's see, the jury's out, maybe I'm completely wrong here - but I don't think more people solve startup problems, at least not necessarily in the AI space. You're seeing that proven out with the recent Windsurf acquisition and all this stuff. What you really need is just a team that communicates very well across the commercial side that's getting constant product input and the engineering side that's building stuff.

Kenn So: I think so. But at some point, you also need to add more people because there's company stuff that you need to keep track of things. Maybe that's a good segue into what's next for Kapa, for you and Kapa. 150+ customers, 15 people, really great customer logos. What's next?

Emil Sorensen: We hold ourselves to very ambitious goals, very high growth rates. So it's just continuing to deliver on those and doing whatever is rational to keep moving in that direction.

It's not the most exciting thing, but it's really just keep making sure Kapa is the best version of what it is, which is a thing that answers questions about your products, and spending a ton of time doing customer interviews to understand as these models get better every week, what new points of value we can essentially build around that to keep making Kapa better.

Kenn So: Any sort of preview you can share an area you’re going to expand to next?

Emil Sorensen: There's always the question around how much do you lean into something like support. That's still very open-ended because that's also a freaking rabbit hole to lean into.

There's a question around the underlying paradigms. The amount of times I've heard people say "RAG is dead" and it turned out to be false, so I'm not going to say that. But I'm very interested to see how paradigms are starting to change. We published a bit of research back in January and February, very publicly talking about our experiments using reasoning models combined with RAG that had some really interesting results, but was at a point where the models we were using just weren't experienced enough with tool calling to consistently call tools at the right time. It's changed very fast.

So that's the space we're looking very much into. But other than that, it's really just helping people that want to build product assistants do that in a really good way.

Kenn So: Okay, last part. How are you using AI personally?

Emil Sorensen: The most personal anecdote is probably for the first time since starting up two years ago - not a very inspirational thing to say - but I finally took some time off with my wife. We're about to have a baby in two or three months, our first kid, which I'm very excited for. There are lots of founder thoughts like how do you balance building a company and being a good dad.

Part of that is we wanted to take some time away. So we did a nice road trip a couple of weeks ago for a week, and Claude's advanced research was phenomenal for that. Very vanilla use case, but that's crazy.

Kenn So: I really like the latency when using something like that too. It makes me feel that the output is earned, that I have to sit and wait for like 10 minutes.

Emil Sorensen: That's a great point. It feels like so much of our experience is we expect immediacy, but the research forces you to kind of wait for that output. It's a little bit like photographing with film - you have to be a little bit more intentional with every shot.

Kenn So: Yes, it's similar to the Starbucks story of customers don't like it when the coffee is instantly available. They like to see grind the coffee and pull out the steamer, even though the taste is exactly the same.

Emil Sorensen: Yeah, pretty much.

Kenn So: Awesome. Let’s wrap here. Emil, thanks again for the time!

Emil Sorensen: This was great. Thanks for having me.

In this interview, I speak with PolyAI’s Shawn Wen (CTO and co-founder) and Devidas Desai (SVP of Product) about building enterprise AI products and how communications & AI have evolved with generative AI. PolyAI develops enterprise conversational assistants that engage in natural conversations with customers to resolve their issues. These assistants understand customers regardless of their phrasing or manner of speaking. While voice interaction has gained recent popularity due to GPT-4o, PolyAI has been at the forefront of this technology since 2017. The company recently raised a $50M Series C round led by Hedosophia and NVentures (NVIDIA's venture arm), bringing their total funding to $120M from prominent investors including Khosla Ventures, Georgian, Point72 Ventures, and others.

Subscribe now

Table of Contents

• Key learnings

• Introduction and journey to PolyAI

• Evolution of AI and impact of GPT-4

• Changes in communication, customer service, and PolyAI’s approach

• Building AI assistants: technical aspects, enterprise concerns, and advice

• Future plans for PolyAI

Key learnings

1. Enterprise AI adoption requires balancing multiple stakeholders: While customer experience heads traditionally were the main decision-makers, generative AI brings security, IT, branding, and legal teams into the conversation. This complicates the sales process but also creates opportunities for education and addressing diverse concerns.

2. Enterprises often hold AI to higher standards than humans: As Devidas noted, AI assistants may be expected to stay rigidly on-topic in ways that human agents are not. This creates challenges in making AI seem natural while still meeting strict enterprise requirements.

3. Customization and control are critical: Some enterprises have extremely specific restrictions, like forbidding an AI from stating basic facts unrelated to their business. AI systems need to be highly configurable to meet these idiosyncratic needs.

4. Practical solutions trump theoretical ideals: Shawn emphasized the importance of being practical rather than trying to build the perfect technical solution. Time-to-market, packaging, marketing strategy, and sales execution are as important as the underlying technology.

5. Layered safeguards are necessary: PolyAI uses multiple layers of protection, from general content filters to project-specific customizations. This allows tailoring the AI's behavior to different levels of enterprise risk tolerance.

6. Iterative testing with clients is crucial: Finding edge cases and potential issues requires extensive testing, both internally and with clients. This process accumulates knowledge over time that can be applied to future projects.

7. Transparency about limitations builds trust: Being open about the current state of generative AI technology and its limitations, while showing a clear roadmap for addressing concerns, helps enterprises feel more comfortable adopting these solutions.

8. Self-serve capabilities are becoming important: With generative AI, some enterprises want more control in maintaining or even building their own assistants. Providing tools for this can be a differentiator.

Introduction and Journey to PolyAI

Kenn: Thank you both for joining me today. I'm excited to talk with you about PolyAI. To start, could you each share how you found your way to the company?

Shawn: I'm Shawn, co-founder and CTO of PolyAI. I was there at the very beginning with Nikola and Eddy when we co-founded the company in November 2017, about 7 years ago now. The three of us met at the University of Cambridge. I actually met Eddy during my undergrad, as we're both Taiwanese, so our history goes back about 15 years. Eddy and I went to Cambridge to pursue our PhDs under the same supervisor, and Nikola happened to be in the same year as well.

We were actually the last batch of Steve Young's students. Steve Young is a pioneer in speech recognition who created the HTK open source toolkit. It was initially sold to Microsoft in the early 1980s, then open-sourced, with Microsoft continuing to maintain it. It was a traditional n-gram based and hidden Markov model toolkit for speech recognition. Since then, Steve moved on to building dialogue systems and doing research in that area.

I think the reason we're doing what we're doing now is very much because of Steve, both on the research side and the entrepreneurship side. Steve actually started three companies and sold them to Microsoft, Apple, and Google respectively. So because of the research topics we worked on with him, and his entrepreneurship history, it pointed us towards wanting to do something together.

Before graduation, we all went to big tech companies - I went to Google, Nikola to Apple, and Eddy to Facebook. But we started to feel it was a good time to do something together. So we all came back to the UK and started the business here. That's a brief history of the company's founding.

Kenn: That is cool. I did not know that about the founding team’s history with Steve Young. Thank you for sharing that background. Devidas, how about you?

Devidas: My story isn't as cool as Shawn's, but I've been in the collaboration and communication space since I started in product management, and the field really fascinates me. Joining PolyAI allowed me to stay close to communication, but leverage conversational AI to improve customer and caller experiences.  Of the companies I met with, it sounds cliché, but honestly the quality of people I met with helped me make the decision to join PolyAI. The strength of the technology team was just super impressive, which is what essentially brought me here. And it's fantastic to be here.

Evolution of AI and Impact of GPT-4

Kenn: That's great context. Shawn, you mentioned studying for your PhD. Machine learning was a very different world back then, from both an academic and industry perspective. How has the AI field changed from your perspective?

Shawn: It's a very interesting question. I started to get into AI and machine learning in my third year of undergrad. I actually started in electrical engineering, then gradually shifted towards AI because I found it more interesting. I began with speech recognition, which involved a lot of signal processing. As I mentioned before, it was hidden Markov models and n-gram based models. These approaches are actually very Bayesian, and that continued to be the case when I went to Cambridge, which is traditionally a big Bayesian camp. You have people like Zoubin Ghahramani there doing a lot of Bayesian-based approaches, graphical models, Gaussian processes, and so on. It's very mathematically heavy and intellectually interesting.

That kind of model was actually popular for quite some time, and deep learning was just something in the background because the compute wasn't there. But many of the algorithms and optimization techniques we're using these days were already there. I think the shift started around 2013-2014 in the speech recognition field. In those days, if you had an improvement of word error rate by 0.5%, you could publish a paper. But Microsoft trained a deep neural network model and achieved a 4% reduction in word error rate, which was huge because for several years no one had managed to achieve that. It blew everyone out of the water, a bit like OpenAI with ChatGPT these days, but people don't remember that because it was more in the academic world.

Afterwards, you see a bunch of people starting to train bigger neural networks and applying them to different use cases - first ImageNet, then people moved on to different problems like NLP. Deep learning has progressed a lot since then, and a lot of the benefit is because the compute power caught up. People started to realize it's very difficult to figure out exact, beautiful mathematical equations to solve real-world messy problems. So people started engineering these large machines and piling data into them, and magically, you get something really powerful.

I think the reason large language models surprised a lot of people is because previously, if you were training small language models, it was really about garbage in, garbage out. It wasn't really that helpful. And magically, at some point, once the model reaches a certain scale and the data reaches a certain scale, it reaches that tipping point and boom, it just starts to work really well. I think this caught a lot of people by surprise.

Kenn: GPT-4o is a big leap from the early days of neural networks that you sketched out. How has GPT-4o affected PolyAI or your thinking about building products in this space?

Shawn: GPT-4o is super exciting, and this kind of multimodal model is at the forefront of technology innovation. I think it's going to make things potentially even easier for everyone. We're super excited about it. In fact, we're actually in touch with OpenAI to make sure that we can get private access once they finally release it. It's unfortunate that they recently announced they have to push it back for another month.

We're very excited about it, but at the same time, I think getting these models right is probably quite tricky. That's probably the reason they're pushing back the timeline. Even for the initial announcement, they're only going to release it to a few trusted partners to test it. If you think that hallucination in GPT's text version is already dangerous, you can totally imagine that hallucination in the voice channel is a completely new world.

We haven't actually gotten our hands on testing the system yet. But we do see that for building consumer-facing products, which I think is what they're trying to do, they will make a lot of huge progress. I think for enterprise applications, I wouldn't think that you would actually gain so much quick adoption yet, just because the end-to-end modality makes it even harder to understand what's actually going on. If GPT-4o suddenly starts screaming at you in the voice channel, you don't exactly know what's happening or what input is causing the trouble. I think that would actually make enterprises worry. So we're super excited, but we're also cautiously thinking about what would be the best way to use it because we know that a lot of our enterprise customers would have concerns.

Kenn: It's a good distinction. As a consumer, I'm super excited about getting my hands on GPT-4o. Have you had prospective customers or customers reach out to you asking if you can build something like GPT-4o for them?

Shawn: We haven't heard that yet. I think people still don't quite understand what it means, especially the end-to-end modality of it, because they haven't actually gained access to it yet. Once they gain access to it, I do imagine that people will start to ask about it. But there's no way for them to try it yet. And I think in the documentation or the announcement, unless you're a techie, which the majority of our clients aren't because they're actually operating contact centers, they don't quite get what's coming yet.

Kenn: It's interesting that the models continue to progress at a very fast pace. How do you think about the rapid progress of these models? And how do you future-proof yourselves, or your tech stack? Because I'd imagine that multimodal capabilities will change your tech stack quite a bit. But also from a more strategic perspective, how do you think about the progress of these models and how that relates to the problems you're solving for your customers?

Shawn: I think the models will continue to evolve, and they're going to evolve at a very rapid speed because now there are more and more companies jumping into this kind of large model training and multimodality. Therefore, I think these will continue to progress super fast. As a company, our philosophy is always that these are all new tools that we should incorporate into our product or offering. Not one single technology could be for everyone. I think it's very important to actually find the right technology and tooling for the right customer.

Our goal has always been to continue to invest and innovate on our own technical stack, but at the same time, we should keep an open mind to incorporate any of the latest technology into our offering. Because at the end of the day, our goal is to make the best, human-like voice assistants that enterprises would be comfortable adopting and using, and that callers would be happy with as well. So to us, it's really about how we can continue to incorporate these new technologies fast enough into our product offering.

Devidas: I would just add that from a product standpoint, we are building this as a true platform. Each of the components that power the experience, whether it's the listening piece, the reasoning piece, the speaking piece, or the biasing piece, we can offer the best in breed. Some of the best-in-breed pieces could just be PolyAI tech, which is proprietary, and a combination of that just provides an amazing caller experience because we are the thought leaders when it comes to bringing this together and building proprietary tech on the available technology so that we are thinking about the caller experience.

From a product strategy standpoint, we know what we are good at. We know what are some of the areas that we have an upper hand on, and we want to make sure that we continue to build on that proprietary tech, make it a part of the platform to solve for amazing caller experiences.

Changes in Communication, Customer Service, and PolyAI's Approach

Kenn: Devidas, you've been a veteran of the communications space. How has communications changed, either from a personal or business perspective? And how have developments in AI changed or shaped your view of communications from a product perspective?

Devidas: Those are both excellent questions. In general, I think the changes have been very sticky because the technological changes have actually also changed user behavior. From a communications and collaboration standpoint, previously, communications were very simple and single-channeled. We're talking about either an email communication, an IM, an SMS, or a phone call, and they all had different jobs. That's exactly what users were expecting. If I want to email you, I'm going to email you, and I would expect your email back. The lines were quite thick.

Then we started moving towards modalities coming closer, where you started having the concept of messages having attachments or mentioning people in your emails, and the lines started getting blurrier. This is where you bring multiple modalities together. Then we moved over to web conferencing, which led to UCaaS (Unified Communications as a Service) products, bringing messaging, video, and phone together. It almost doesn't matter where the user is or what the other side is using - you have all of the modalities, and you can switch between them to talk to people. It's about the speed of communication.

Now we're having AI-driven communications. All of these changes are really impacting user behavior and changing user habits. People are getting more and more used to spending less time on communication channels and making them super efficient. Previously, users were expecting simple, basic communication channels. Then we started moving towards multi-channel communications. Then users started expecting 24/7 instant responses, which brings AI into the picture because you don't necessarily want to power those communications with humans all the time.

Now we're in this phase where people are expecting 24/7 instant responses that are personalized. You don't necessarily want to go over the same details over and over again. When you reach out to a rep or an AI, you expect the other side to know who you are, what your preferences are, and just pick up the conversation where you left off.

It's been fascinating to see how the tech has evolved based on users' expectations and vice versa, and how some of these changes are very sticky. AI is really helping provide quality communications to users. Separately, it's helping businesses allocate human capacity to the highest ROI customer profiles. The way we see it, AI is not replacing anything. It's actually making the spend on customer service, customer interactions, communications, or collaboration in general more efficient. You're getting a higher ROI because you're now in a position to determine which conversations you want bots to handle versus humans, and how personalized you want those exchanges to be.

I'm pretty sure we'll soon see a world where it really doesn't matter as a user or caller who I'm speaking with, whether it's an AI-powered bot or a human. As long as my experience is great and it does the job, I really don't care whether I'm talking to a human, because I don't want to spend too much time on a phone talking to a customer service professional anyway.

Kenn: That's a good segue to finally talking about what PolyAI is at a high level and what problem you’re solving. Could you elaborate on that?

Shawn: PolyAI has been quite consistent in terms of what we're building. During our PhDs, we were doing research on conversational systems, and the company is really an extension of that. We continue to build voice assistants, primarily over the voice channel. The same technology can be used for building text-based assistants as well, but we've been focusing on developing a really good voice assistant because putting all these technologies together requires a lot of focus.

Our primary market is in enterprise contact centers. A lot of these are communications that happen that normal people might not pay attention to. But whenever you have an issue and you call a contact center, especially in the Western world, the quality of service is often not good enough. This is because of labor shortages, increasing labor costs, and the fact that there's still a requirement to staff enough people in a contact center, but fewer people are willing to do that kind of job. Or if they do want to do it, it's often a temporary job - you come in, train for 3 months, and 6 months later, you're out.

There's also the challenge of effective communication between callers and offshore contact centers. When you have an issue with your bank and you call, you might speak with someone from a different part of the world who may not share the same cultural context or communication style. This can sometimes lead to misunderstandings or difficulties in fully grasping the caller's situation and concerns. As a result, many customers find it challenging to connect on a personal level during these interactions. This is one of the reasons why many companies initially moved their contact centers offshore, but are now considering bringing them back to their home countries in response to customer feedback. However, the underlying issues persist - service quality is declining, there's a shortage of people willing to take on these roles, and contact centers continue to face significant challenges.

We think technology can solve this kind of problem. In a contact center, voice calls actually need a dedicated agent listening to the phone call and speaking to the user on the phone. While chat or digital channels like emails are much easier because you can parallelize the work, voice is especially challenging for contact centers. But a lot of people still want to call when they have urgent issues or when they've tried to do something online but couldn't.

PolyAI is trying to help solve that problem. We want to place AI agents into contact centers. The AI agent should be the best tier 1 agent that the enterprise can have, representing their brand. They need to communicate naturally and sound like what the brand wants them to sound like. We're not intending to just automate the entire workflow because naturally, there are certain kinds of use cases that AI is not good enough for yet. Like a lot of emotional cases, some cases that require lots of empathy, and some use cases that require very complicated transactions. These are things where tier 2 human agents will have a very strong capability to help, and that's where AI can step out and hand the call back to the contact center.

A lot of people have been thinking about AI taking people's jobs. I think partially that's true, but it's also not true because there are already problems with contact centers - they just cannot staff enough people. And arguably, training AI is also not free. You actually need to supply it with a lot of data, and there should be a lot of humans in the loop as well. As new technology continues to evolve, there will always be new jobs being created, even as some jobs go away.

Kenn: That's interesting. I grew up in the Philippines, and I know turnover rates in contact centers there are 50% every year.

Shawn: Even in the Philippines, yes. It just tells you how tough the whole business is.

Kenn: I think it would be helpful to sketch out who the different users and personas involved are. As an end user, I just talk to someone. But I know in the enterprise, you have corporations that may or may not outsource it to someone, and they may have their own requirements. Then you also have the end users. How would you sketch out the problems and how you're solving for those different personas?

Devidas: Maybe I can talk about this a bit. I'll answer this in two ways. There's been a historical persona who we've been selling to, and then I feel like generative AI is definitely changing the landscape in terms of who's involved with respect to the buying process and the maintenance of the conversational assistant.

Historically, and this is still the case, customer experience heads and heads of contact centers are still the primary people we sell to as far as enterprises are concerned. I feel like generative AI is changing that landscape a bit and is bringing in a few more personas that are involved in the buying process. Because with generative AI, you're no longer just solving for customer experience, cost efficiency, or improvement of CSAT. With generative AI, you have to worry about how safe the solution is. So that brings security and IT into the mix.

With generative AI, you also want to make sure that, especially for a voice assistant, the voice is on brand and you are appropriately representing the brand that the assistant is taking calls for. So that brings in someone like a CMO or head of branding into the mix as well. It's been a really interesting shift where we are seeing more people coming to the table with respect to making decisions, and it's fascinating. Some of these interactions are very exciting in terms of us either educating them or answering good questions.

As Shawn touched upon, one additional persona that we're seeing come into the mix is that there are teams and jobs being created for every conversational assistant that is deployed or sold or built for a particular brand. There are people responsible for monitoring the calls, making sure that the assistant is on point, and reviewing calls to see what could be improved. They either make those improvements themselves, or if they don't want to be hands-on, they work with us to make those improvements. This alludes to the point that Shawn was just making, which is that it's partially going to make some jobs redundant, but at the same time, it's introducing new jobs and new skill sets that haven't been in the industry for a while. And that's a shift that we're seeing which is here to stay.

From a persona standpoint, those are the ones that we're seeing. One of the other trends that we see is that with generative AI, it is becoming a bit simpler and easier for customers to actively maintain their own assistants. In some cases, they can even build their own assistants, which was previously quite tough with intent-based models. We're seeing a lot of demand for that as well, which is, "Hey, I want to self-serve. I want to self-care for my own assistant because I know my business the best." And that's totally valid. We are able to provide for that, and we are seeing that shift as well. So it's been overall a fascinating experience. Generative AI has definitely changed the landscape with respect to conversational AI.

Kenn: That's really interesting. I didn't think about the other personas now getting involved in customer service agents as well. It seems like there's been a significant shift.

Devidas: It's still early days, but I think we're seeing enough of it to know that this is going to be a very logical trend going forward.

Building AI Assistants: Technical Aspects, Enterprise Concerns, and Advice

Kenn: You touched on something interesting about voice assistants. Before, with any customer experience assistants, you had to script every workflow. But now, I don't know how your team is building it, but I'm curious how much you can share. It seems like you're leaving it more to the AI system, which needs to know more about the company and the policies, but you also have to prevent it from hallucinating. So there's less and less control. Can you share how you're building it and controlling it to make sure that it's on brand, on policy, and aligned with the business?

Shawn: That's a very good question. This has been something we've been working really hard on since generative AI and large language models came along. If you try to build these kinds of voice assistants, you can actually build something quite impressive quite quickly. But it doesn't really do certain things well. For example, if you ask ChatGPT to make a restaurant reservation, it would actually pretend that it's making an API call in the background and tell you it's been done. So it hallucinates a lot.

I think GPT is pretty good at pattern matching. How we think about large language models is just to incorporate them as one of the building blocks into building the voice assistant. It's a super powerful building block. What it can do really well is pattern matching. So if you provide some information in the prompt and then the task is to ask GPT to pattern match what users say to what is included in the prompt, it can usually do it quite well. And that's usually how we handle FAQ questions from a large source of knowledge base.

When you start to build more complicated flows that require several steps of transactions, then there will be some design elements involved. You also need to build some guardrails on top of it. What we do is basically build a programmatic approach overlaying on top of GPT. Think about it like this: if you're just prompting GPT to do certain things, it's equivalent to handing GPT 100 pages of contact center instructions and saying, "Hey, read it and then interact with the user directly." If you were newly onboarded as a person, you're not going to read all these manuals before you jump into a call, and you're going to hallucinate yourself because you just want to survive in the conversation. That's what GPT does when you don't give it proper training or plans.

What we effectively do is say, "I'm not going to just give you 100 pages of these instructions and guidelines. I'm going to group them into different categories for you." For example, if the user is asking about a payment issue, you go into this particular payment issue flow. You fetch all these instructions or manuals, and those manuals are also paginated for you. So that means if a user asks about this question, the first step is you need to ask about these 5 different questions, and then you flip to the next page, then you answer another 3, and so on. By paginating these instructions and giving it a step-by-step progression into a conversation, you're kind of guiding GPT. It's more of a copilot kind of design we're doing here, rather than relying on GPT to just go on autopilot.

Kenn: That's really interesting. Let's say I'm a head of customer experience and I go to PolyAI. What do prospective customers usually ask you about? How do you describe the product to them, and what does it take to get set up? And what concerns do you usually hear from customers, particularly as they think about deploying at enterprise scale?

Shawn: From the hallucination aspect, it's not even about hallucination they're worried about. Some of these enterprises are super strict in terms of what the bot can say and cannot say. For example, there's this telco company we work with that doesn't even allow the model to say who their CEO is, even though it's a customer service bot for that particular company. They also don't allow the bot to say what the capital of France is. It's not allowed to say Paris because it's considered "out of scope." So that's the level of scrutiny you'll probably run into with enterprises. It depends on the enterprise, but these are the extreme cases we've seen so far.

Devidas: Sometimes you can say that AI assistants are held to a higher bar than human assistants. For example, if I press a human four times to tell me what's 2 plus 2, the human assistant could totally be like, "Alright, it's 4. Now what can I help you with?" And that's totally okay, but it probably wouldn't be okay if an AI assistant did that. So sometimes the bar is much higher for AI compared to humans.

To extend Shawn's point, some of this is quite synonymous with generative AI. When you talk about generative AI, you start thinking about safety and whether the assistant is on point. Typically, the three dimensions that we see our customers having concerns about are: Is the assistant providing accurate responses? Is the assistant detecting any threats if someone's trying to change its behavior, and how is it managing that threat? And lastly, how is the AI treating my customer's information? Because you end up sharing PII (Personally Identifiable Information) in these conversations in order to help the caller, and how is that being managed?

At PolyAI, we take these concerns very seriously. If you see our roadmap, there are clear items addressing each one of these concerns, and we're actually on a good path with that.

Kenn: Interesting. To the extent you can share, how do you red team or test against these concerns? For example, making sure the voice assistant doesn't say who the president of France is?

Shawn: That's a problem that everyone will have to face. As we've gotten very deep into selling to enterprises, we've developed several layers of safeguards. The first layer is that we actually put in a content filter. This filter is basically the highest level, filtering out anything that is dangerous, biased, sexist, or otherwise harmful content. The model would be triggered if any of this content is included, and the system or assistant would basically just refuse to answer or have a particular behavior. Some of our clients want the call to be handed off back to a human agent in this scenario.

Then going downwards, you have these very specific requirements about what you shouldn't share outside of what you're supposed to do in the contact center scope. We have an automatic evaluation framework that allows our developers or designers to write stories about particular conversations. Every time we're about to launch a new system, we run all those examples against the system to make sure that all the examples pass. That's another way to safeguard it.

One of the major challenges is how do you actually find those examples of what you're not supposed to do. In that particular case, it requires years of accumulated relations. It requires several projects where users do a lot of extensive testing. You need to give it to your clients to do a lot of extensive testing as well. And then you have to decide a boundary where you put those examples in.

So we have different layers of safety guardrails. One is that hallucination and harmful information should be excluded. The second layer is that you are allowed to say freely what is not harmful. But then you have another layer that you are only allowed to answer questions related to the contact center. And then the most customized level is that you are only allowed to answer specifically for that particular project or account. We don't have any clients at that level yet, but there are definitely some customizations we'll have to do for some future clients, I imagine.

Kenn: This has been great. We've touched on a lot of enterprise concerns. Is there anything else you think we haven't discussed about what enterprises are looking for? How would you advise people trying to build AI products for the enterprise? What are some of the lessons you've learned in your journey building PolyAI?

Devidas: From a product standpoint, the trend we're seeing from enterprises is that almost every enterprise champion we've interviewed wants to adopt generative AI, but they're a bit nervous about it. That nervousness stems from a range of concerns. It ranges from "I believe in it, I want to get this done in my company. How can you help me convince the business that this is a good thing?" all the way to "How do you take care of safety? How do you take care of prompt injection? How do you take care of hate speech?"

Essentially, that's exactly why I said a lot of this is education for our customers and just being very transparent about how we are solving for some of those problems. From a solutioning standpoint, it ranges from us addressing the concerns or giving our champions enough ammunition to feel good about the fact that this makes absolute sense, and they're going to put their name behind it and push for it within the organization.

What is working well for us is just being open and transparent about how we are solving for some of these concerns. Customers understand that generative AI is fast-moving, there isn't a foolproof solution yet. It's about trying to stay ahead of the curve, trying to constantly evolve, and then showcase data on what the assistant is doing well, where it could improve, and then have tools in place to keep that improvement going on an ongoing basis. That's how they feel pretty good, strong, and safe about adopting the solution.

Shawn: From the technical founder's point of view, I think you just need to be super practical. Everyone wants to build the next Google or Facebook. Everyone wants to build a platform. But in reality, it's probably not going to happen that way just because if it's actually that much of a market, then the big players would jump in and they would already take the market. You're never going to be able to win that battle because they have way too many resources.

I think we just need to be practical. Technical founders like myself sometimes are not very practical. You always want to solve the theoretical best problem, like what is the best revenue model, like a search engine. If you can actually build a business model that can print money, that would be the best. Everyone wants to build that, but only a couple of people manage to do it.

What we learned through the journey is that we just need to be practical about go-to-market. It's not just about technology. It's about the product, how you package it together. It's about the marketing strategy. It's about the sales executions as well. You just need to find the right way to sell the right product to the right people. And that requires a lot of learning and a lot of trying over a long period.

Kenn: That's good advice. In San Francisco, I attend all these AI events, and it's mostly about the technology and building prototypes around it. Less about the go-to-market side of it.

Future Plans for PolyAI

Kenn: Before we end the interview, congratulations on the huge $50M series C round and $500M valuation. What's next for PolyAI? What are each of you excited about for the future? And do you have any call to action for people reading this interview?

Devidas: I think we're going to keep things simple. We want to keep our product strategy living and breathing. We're going to spend a lot of time on product discovery, keeping our eyes on the competition, and accordingly evaluate our strategy on an ongoing basis. We're going to keep the focus high. We're going to use the funding round to address some of the gaps in the right skill set areas. But all in all, we're going to keep it simple. We're going to focus, have a clear product direction. We're going to differentiate on our strengths, and we'll continue to invest in that. We're going to continue to believe in and increase our amazing team. We have a fantastic team that can make anything happen, and we're going to continue to support them and keep addressing the gaps wherever necessary.

Shawn: For me, it's about continuing to expand PolyAI's reach. Even 6-7 years down the road, we're still not taking enough calls yet. I just want the systems to take more calls, to be able to solve more real-world problems that callers and contact centers are facing. Because this kind of pain is real. Every time I actually phone into a doctor’s office, I hope there will be some sort of system that can actually just help me get through what I want rather than waiting on the phone for 30 minutes, someone picks it up, doesn't understand what I want, transfers me to someone else, I wait for another 15 minutes, and so on. I think those kinds of experiences for humans in this new world should be in the past. People shouldn't spend their time phoning their doctors, their banks. They should just have a system that can understand them and send them to the right place.

Kenn: That’s awesome. And with that, let’s wrap. Thank you both for your time. This was a great conversation. 

Subscribe now

In this deep dive, you’ll learn insights from conversations with many of Databricks’ customers and ex-employees. I want to thank Tegus for giving me access to their centralized expert call transcripts. With a platform as broad as Databricks, it is almost impossible to parse signal from the noise without primary research. If you’re curious about Tegus, try them out with this link.

Thank you to the 25 who joined in the past 2 weeks. Shout out to and for referring new subscribers!

Databricks is the data & AI company. Founded by the creators of Apache Spark, the company has evolved from its roots in big data processing to a data intelligence suite covering the entire data & AI lifecycle. The company is helping over 10,000 customers, including iconic organizations like Adobe, Toyota, FDA, Shell, Conde Nast, and many more.

Why Databricks is a generational company

• Product: Databricks' unified platform is designed to handle massive amounts of data, providing businesses with a seamless way to process, analyze, and gain insights from their data. The platform is based on open-source technologies giving customers optionality (which enterprises love) while also building proprietary optimizations across the entire stack to make the product faster, more stable, and cheaper to use than competing products.

• Market: Databricks’ $126B addressable market will grow 17% annually over the next few years, creating ~$30B potential industry revenue annually. Databricks is also well-positioned in the fast-growing generative AI market, which is expected to grow 87% annually, creating ~$15B potential industry revenue annually.

• Traction: Databricks’ revenue grew from $1M in 2015 to $1.6B in 2023, making it one of the fastest-growing companies in history. In spite of intense competition, the company is expected to continue growing by over 50% over the next two years.

• Team: The company is one of the best-rated companies to work for, with a 4.4 Glassdoor rating and a 4.0 Blind rating. Most of the founders continue to be actively involved in the company and are respected in both industry and academia. Ali Ghodsi is one of the most well-regarded CEOs and was voted one of the best CEOs by the tough crowd at Blind.

Contents

1. Origin — The Spark that started it all

2. History — Four Phases of Databricks

3. Pain point — Workloads, Personas, Architecture

4. Product — Data Intelligence Platform

5. Market Opportunity — $126B growing $25B a year

6. Competitive Landscape — Snowflake, CSPs, Startups

7. Team — One of the best

8. Financials — One of the fastest growing in history

9. Valuation — Historically rich, a deal looking forward

Origin — The Spark that started it all

It all started with a research project at UC Berkeley's AMPLab in 2009. A team of seven UC Berkeley academics - Ali Ghodsi, Andy Konwinski, Arsalan Tavakoli-Shiraji, Ion Stoica, Matei Zaharia, Patrick Wendell, and Reynold Xin - came together to work on Apache Spark, an open-source distributed computing framework designed to be faster and easier to use than Hadoop MapReduce, the dominant big data processing framework then. The key innovation behind Spark was the concept of resilient distributed datasets (RDDs) - a way to divide big data into smaller chunks that can be processed faster across multiple machines while ensuring data is not lost if something goes wrong. This allowed Spark to achieve speeds up to 100x faster than Hadoop by caching data in memory instead of reading/writing from disk. But the goal was not just speed - it was to make big data analytics accessible to a wider audience. Programming MapReduce is painful. Spark is easier for developers to learn and program.

In 2010, Spark was open-sourced under a BSD (Berkeley) license. This allowed a community of contributors to grow around the project beyond UC Berkeley. The community around Spark grew rapidly. In 2013, Spark became an Apache incubator project and was promoted to a top-level Apache project in 2014.

As Spark's adoption grew, the founding team realized there was an opportunity to build a company around it. In 2013, the founders decided to start Databricks. The company‘s first product is a cloud-based notebook interface that allows customers to use Spark without all the complications of setting it up. They also continued to lead Spark's open-source development along with the community.

Under Databricks' stewardship, Spark adoption has skyrocketed and become the de facto processing engine for data engineering, data science, machine learning, and business intelligence in thousands of enterprises worldwide.

As the CEO, Ali Ghodsi is the face of Databricks. But he didn’t want to become the CEO in the first place. Neither did Ben Horowitz, their first investor board member. In 2015, then-CEO Ion Stoica decided to go back to his professorship at UC Berkeley and step down. Ali, who was VP of Engineering at the time, was chosen by the other founders to take over as CEO. But Ali did not want to, he wanted to continue being an academic. Ben also doubted whether Ali would be a good fit. But everyone else in the founding team thought Ali was the right person. So Ali took the mantle in 2016 on a one-year probation. Fast forward to today, Ali is one of the most respected CEOs and Databricks is one of the most iconic companies.

History — Four Phases of Databricks

Phase 1: Growing and commercializing Spark (2013-2017)

In the early days, from 2013 to 2017, Databricks focused on growing the Spark community and building out its commercial product. Spark moved to the Apache Software Foundation in 2013 and graduated to a top-level Apache project in 2014, driving significant community adoption. In 2014, Spark demonstrated its performance advantages by handily beating Yahoo's record on the Graysort big data processing benchmark, taking one-third the time with one-tenth the computing power.

The company launched its commercial product in 2015, providing a managed platform with collaborative notebooks, automated cluster management, a user-friendly UI, and integrations with cloud storage like Amazon S3. This simplified the deployment of Spark for big data analytics and machine learning. Databricks also organized the first Spark Summit conferences to bring together the growing Spark community.

Phase 2: Becoming Data + AI (2017-2020)

From 2017 to 2020, Databricks entered a phase of rapid growth and expanded its vision to become a unified platform for data and AI. The company formed a major partnership with Microsoft in 2017 to integrate Databricks with Azure cloud as a 1st party service, exposing Databricks to Microsoft's large enterprise customer base. As a 1st party service, Azure Databricks is natively integrated with Azure's infrastructure, security, and services, providing a seamless experience for users. For example, I was able to launch Azure Databricks in a few minutes but it took me over 30 minutes to configure Databricks to run on GCP.

Databricks expanded its product to support the full machine learning lifecycle, from data preparation to model training and deployment. In 2018, it launched MLflow, an open-source platform for the ML lifecycle, and renamed Spark Summit to Spark + AI Summit, reflecting its broader focus. More crucially, in 2019, the company contributed Delta Lake to the Linux Foundation, planting the seeds for its Lakehouse strategy.

Phase 3: Data Lakehouse Platform (2020-2023)

From 2020 to 2023, Databricks pioneered the concept of the data lakehouse, an open architecture combining the best elements of data lakes and data warehouses. The Lakehouse Platform became the centerpiece of the company's strategy, and its product suite expanded to cover more of the data lifecycle. Reflecting this emphasis, Databricks renamed its conference to Data + AI Summit in 2021.

The company launched Databricks SQL service in November 2020, bringing data warehouse capabilities to the lakehouse and enabling a wider audience of SQL users. This became one of their fastest-growing products, growing to a $250M run rate ~3 years after launch. Databricks also launched other lakehouse capabilities like Unity Catalog for unified governance and Delta Sharing for secure data sharing.

During this phase, Databricks also actively acquired companies to get talent and accelerate its product roadmap:

• 2020: Redash (data visualization)

• 2021: 8080 Labs (data exploration) and Cubonacci (data science)

• 2022: Datajoy (Sales/AI solutions) and Cortex (ML production)

• 2023: bit.io (data developer experience), Rubicon (AI serving), Okera (data governance), Arcion (data replication), MosaicML (LLM training)

Phase 4: Data Intelligence Platform (2023-present)

In June 2023, Databricks repositioned itself as the Data Intelligence Platform to make GenAI a main part of its platform. A central piece of this is Databricks IQ, a knowledge engine that learns the unique nuances of customers’ data allowing users to interact with their data in natural language. Furthering the GenAI push, Databricks acquired generative AI startup MosaicML for $1.3 billion, open-sourced a large language model called Dolly, built many GenAI features (e.g. vector search), and invested in another genAI startup Mistral. These moves aim to make it easier for enterprises to build GenAI applications on top of Databricks' platform.

Pain point — Workloads, Personas, Architecture

Databricks’ platform is expansive, so I won’t cover the pain points the product suite tackles in detail. Instead, we’ll go through the primary workloads data and AI professionals manage.

1. Data Storage

In the context of data and AI workloads, it's about storing vast amounts of data in a secure, reliable, and accessible way for analysis and processing.

Key Persona: Data Engineers and IT Administrators are the primary personas involved with data storage. Their job is to ensure data is stored efficiently, securely, and in compliance with regulatory requirements. They are responsible for selecting the appropriate storage solutions (like databases, data lakes, or cloud storage services).

Challenges: Managing the exponential growth of data, ensuring data security and privacy, achieving high availability and disaster recovery, and optimizing costs associated with data storage solutions.

2. Data Management

Data management involves data cleaning, enrichment, classification, and governance to ensure high-quality data.

Key Persona: Data Managers/Stewards play crucial roles here. They focus on creating policies and procedures for data handling and usage, ensuring data quality, and maintaining data governance requirements.

Challenges: Ensuring access to quality data, managing data across systems and formats, and adhering to evolving regulatory requirements.

3. ETL (Extract, Transform, Load)

ETL is a process that involves extracting data from various sources, transforming it into a format suitable for analysis, and loading it into a final target database or destination. It's a foundational process for consolidating, cleaning, and preparing data for analysis.

Key Persona: Data Engineers are the key persona. They design and implement ETL processes, ensuring data is accurately extracted, transformed, and loaded into the destination systems for further analysis.

Challenges: Handling large volumes of data from diverse sources, ensuring the integrity of data through the transformation process, optimizing performance to reduce processing times, and managing the complexity of ETL workflows.

4. Data Orchestration

Holistic management and coordination of end-to-end data workflows, pipelines, and tasks. It encompasses a variety of operations, including but not limited to ETL, to ensure seamless data movement and integration across various platforms and systems.

Key Persona: Data and ML Engineers are central to orchestrating data pipelines. They design, implement, and monitor automated workflows that ensure data is processed and available where and when it's needed.

Challenges: Ensuring reliability and scalability of data pipelines, managing dependencies between different data processes, monitoring pipeline performance, and troubleshooting failures.

5. Business Intelligence

Business Intelligence (BI) involves analyzing data to extract actionable insights that inform business decisions. SQL is a programming language used for managing and manipulating relational databases, a common tool in BI processes.

Key Persona: Data Analysts are the primary personas. They use SQL and other BI tools to query, analyze, and visualize data, creating reports and dashboards that help businesses make informed decisions.

Challenges: Integrating data from multiple sources, ensuring data accuracy and consistency, designing effective data visualizations, and keeping up with the fast pace of business demands are significant challenges.

6. Machine Learning

Machine learning systems learn from data, identify patterns, and make decisions with minimal human intervention. It involves training models on data sets to perform ML tasks (like prediction and classification) and serving the models in production to inform decisions or power product features.

Key Persona: Data Scientists and Machine Learning Engineers are the key personas. They develop, train, and deploy machine learning models, working closely with data engineers to ensure they have the quality data needed for model training.

Challenges: Acquiring and preparing high-quality training data, selecting the appropriate algorithms and models, and managing the computational resources required for training and inference.

The roles within these infrastructures are highly interconnected, with data engineers laying the foundation for reliable data pipelines, data analysts uncovering insights from processed data, and data scientists building advanced predictive models. Machine learning engineers then operationalize these models, enabling their deployment and continuous improvement in production environments. Each of these personas uses a collection of tools that collectively form the data & AI architecture, which often is a complicated maze. Andreessen Horowitz’s blueprints provide the best overview of a unified data and AI infrastructure. In the images below, each box represents a software category and where it fits in the data lifecycle (sources to output). Databricks’ product suite covers most boxes except being the data source and having data labeling capabilities.

Product — Data Intelligence Platform

Databricks is a unified, open analytics platform for building, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale. The Data Intelligence Platform is structured into several layers, each providing specific functionalities that interconnect to form a comprehensive solution. Going through the detailed diagram above is outside the scope. Instead, we’ll go through the key products loosely mapped against the primary workloads discussed in the previous section.

How Databricks works (why not just directly use open source on AWS/Azure/GCP)

Databricks runs on their customer’s cloud infrastructure (AWS/Azure/Google Cloud aka cloud service providers or CSPs). Under the hood, when customers deploy Databricks, it sets up a workspace to manage and deploy cloud infrastructure on customers’ behalf. This includes setting up compute clusters or SQL warehouses that are configured with Spark.

Many of Databricks’ services are built on open-source. While any developer can run these directly in any of the CSPs, Databricks abstracts away the complexity of setting up, maintaining the infrastructure, optimizing the configurations, and putting a collaborative unified interface on top. A lot of the optimizations are underpinned by the Databricks Runtime, a set of software artifacts that run on clusters managed by Databricks. These optimizations include, but are not limited to:

• Proprietary enhancements that significantly improve the performance of Spark workloads, potentially offering gains of up to 5x over open-source Spark

• Databricks Enterprise Security: Enhances security with features such as data encryption at rest and in motion, fine-grained data access control, and auditing

• Databricks Runtime for Machine Learning: Includes machine learning libraries and tools, optimized for ML workloads. While it leverages Spark for data processing, it also includes libraries like TensorFlow and PyTorch for deep learning

• Photon: a high-performance query engine that runs SQL workloads and DataFrame API calls faster, delivering up to 12x speedups

Delta Lake (Data Storage)

Delta Lake is the optimized storage layer that provides the foundation for tables in a lakehouse on Databricks. It can store any type of file, such as images, documents, audio/video, etc., in its data directories alongside the structured data files. Users can create Delta tables that contain references/pointers to the unstructured data files stored in the data lake storage (e.g., Azure Blob Storage, AWS S3, etc.). Databricks Delta Lake differs from the open-source Delta Lake project in that it includes proprietary features and optimizations specific to the Databricks platform.

Unity Catalog is a unified governance solution for data and AI assets. The key features include a single place to administer data access policies that apply across all workspaces and a built-in auditing and lineage that captures user-level audit logs and tracks how data assets are created and used across all languages. It also offers data discovery tools that allow users to tag, document, and search for data assets.

Data Intelligence Engine

The Databricks Intelligence Engine, also known as DatabricksIQ/LakehouseIQ, is the platform's brain. It uses AI to understand the semantics of customers’ data, usage patterns, and org structure. This powers Databricks Assistant, an AI pair programmer, and improves in-product search by automatically describing assets in Unity Catalog. My favorite part is that DatabricksIQ will be available as an API, which can power all sorts of applications.

Databricks AI (Machine Learning)

Databricks AI is a comprehensive suite that allows users to build, experiment, and productionize machine learning models. It can be bucketed into two buckets: end-to-end (classical) machine learning and generative AI. The end-to-end ML capabilities encompass a full machine learning operations (MLOps) workflow with MLflow, which includes automated machine learning (AutoML), monitoring, and governance. The GenAI portion is anchored by MosaicML, which Databricks acquired. MosaicML developed efficient methods to reduce the cost of training and customizing large language models (LLMs), making these capabilities more accessible to a broader market. Customers can develop custom LLMs and serve them in production.

Delta Live Table (ELT)

Delta Live Tables (DLT) is an ETL and real-time analytics tool. It simplifies data ingestion and automates the creation of reliable data pipelines. DLT automates and orchestrates data ingestion, transformation, and management tasks, allowing users to define transformations using SQL or Python. DLT also supports data quality enforcement through expectations, which define the expected quality of data and specify actions for records that fail to meet these standards.

Workflows (Data Orchestration)

Databricks Workflows is a managed orchestration service designed to facilitate the definition, management, and monitoring of multitask workflows for ETL (e.g. with DLT), analytics, and machine learning pipelines. Workflows provide intelligent ETL processing with AI-driven debugging and remediation, ensuring end-to-end observability and monitoring of data processing tasks. Workflows can trigger based on schedules, file arrivals, or continuous runs to ensure that jobs are always up-to-date.

Databricks SQL (BI)

Databricks SQL is the collection of services that bring data warehousing capabilities in a lakehouse architecture. It has an in-platform SQL editor and dashboarding tools that allow team members to collaborate with other users. Databricks SQL also integrates with a variety of tools (e.g. Tableau) so that analysts can author queries and dashboards in their favorite environments without adjusting to a new platform.

Aside from these, there are two other products that are worth noting: Databricks Marketplace and Lakehouse Federation.

Databricks Marketplace

Databricks Marketplace is an open marketplace where anyone can obtain data sets, AI and analytics assets—such as ML models, notebooks, applications, and dashboards—without proprietary platform dependencies, complicated ETL, or expensive replication.

Lakehouse Federation

This is probably my favorite recent product. If customers are not going to consolidate their data on Databricks, they can still access their data through Databricks. The Lakehouse Federation capability can query against external data sources and currently integrates with PostgreSQL, Amazon Redshift, Snowflake (!), Azure SQL Database, Azure Synapse, and Google’s BigQuery. Databrick’s ambition of unifying the data estate go beyond just proprietary systems. With Delta UniForm (short for Delta Lake Universal Format), Databricks (and technically anyone using Delta) can read (and eventually fully manage) all popular Lakehouse formats.

Market Opportunity — $126B growing $25B a year

Databricks’ plays in the following analyst-defined markets:

• Database management system is a product used for the storage and organization of data typically with defined formats and structures.

• Data management software consists of tools to achieve consistent access to and delivery of data.

• Analytic platforms are data science platforms for analysts to analyze data and build models

Enterprises are expected to spend ~$30B more each year in these systems, representing a large opportunity for Databricks to grow into. Generative AI tools, in particular, are going to be a material growth driver, with the relevant GenAI categories doubling almost every year in the near term, adding ~$15B in revenue opportunity annually. There’s some overlap between the two figures but the point stands that Databricks has a large market opportunity to continue growing into.

These numbers are based on analysts surveying organizations on how much they spend on different tools. It is not a pie-in-the-sky TAM figure but is based on actual and projected spend.

Competitive Landscape — Snowflake, CSPs, Startups

Databricks’ key competitors can be grouped into three:

• Snowflake (or data platforms in general)

• Cloud service providers (Azure, GCP, AWS)

• Purpose-built tools

Databricks vs Snowflake

There is a rivalry between Databricks and Snowflake. While there are other data platforms, such as MongoDB, the narrative is primarily the competition between red and blue.

The rivalry intensified in 2020 when Databricks launched the Lakehouse architecture, encroaching into Snowflake’s territory. This even elevated to an unusual public tit-for-tat on official company blog posts when Databricks claimed to set the world record for a data processing benchmark, and Snowflake claimed foul because the assessment was not fair. Snowflake is also going after Databricks’ workloads. Snowpark first launched in 2021 allowing developers, data engineers, and data scientists to run non-SQL code. It supports Python and Scala for processing data pipelines, it also has a DataFrame API for data manipulation similar to Spark. Just from the name, Snowpark is a wordplay on Spark. In 2023, Snowflake launched Notebooks to address more of the ML workflow. While both seem to have a lot of overlap, there are key differences:

Databricks leans AI and engineers, while Snowflake leans toward BI and analysts: Databricks is the preferred tool for data scientists and engineers. Snowflake is for data analysts. Snowflake is also the preferred tool to just dump clean data. That said, both are trying to after each other’s core persona.

The quotes below are edited excerpts from Tegus’ platform.

• Customer / Director of Data Insights at a large telecom company — So basically, both of the tools have very large presence [in my company]…we allow the users to have a different choice. Generally speaking, in chief data office organizations, we prefer to use Databricks, because the majority of us are data engineers or data scientists or software engineers. We like that environment. But from the business unit, majority of them prefer to use Snowflake. For those hardcore, very power-intense, compute-intense calculation, transformations, we use Databricks. And then for those more business-oriented scenes, we use Snowflake.

• Customer / CTO at a large financial services company — You can actually get faster performance, I think, on Databricks if you know what you're doing. I think that's why the Snowflake released Snowpark because they understood that in some high-performance use cases where you need in-memory analytics, they were not as competitive.

  So I would say that's a clear winner for me is Databricks. But at the same time, the folks who are like, just click around kind of user, maybe for them it's okay. Like, maybe they're not as concerned with performance as much as like the data science folks who have like large pipelines and large data sets to analyze.

• Ex-employee & current service partner / Data engineering consultant — People think it's like Databricks versus Snowflake. It's actually very common for companies to have both where they use Snowflake for the data warehouse that powers all their business intelligence and all their analysts use that, and then they would use Databricks for more data engineering style work or like that the data scientists or data engineers or Python heavy users would want to go in and use.

Databricks has a broader product suite that is also modular: Adopting Databricks is easier because it be plugged into almost any data stack and used for any workload. A customer can have any data store, even if its Snowflake or AWS Redshift, and still use Databricks for data processing and AI/ML. The modularity also extends to Databricks’ open-source compatibility, having been built on top of major OS projects.

The quotes below are edited excerpts from Tegus’ platform.

• Ex-employee / Sales executive — As a seller of Databricks, one of the easiest ways for us to get a foot in the door was, "I'm not going to ask you to change everything you're doing. We'll fit into your existing user flow and take over that one part. Maybe you only want to use this for your data ingestion, or you want to use this for your ML models". That's definitely a big focus area, not just from the open-source perspective, but also from the ease of use and scalability perspective. It makes a decision maker's decision a lot easier if there are already existing connectors or plug-ins or simplified ways to get started with their existing ecosystem and just plug Databricks in there.

  And by the way, if the customers ever don't like Databricks, they can always revert back to an open source. They’re not locked into Databricks. The risk mitigation that came to that narrative compared to Snowflake was actually really significant particularly when we engaged IT in the sales process.

• Customer / CTO at a large financial services company — The other big thing is Databricks is built on a lot of open-source technologies so that I know that if something horrible happens and Databricks gets acquired by Oracle, like my Halloween scenario. I'll still have a way to run stuff using open-source software. So I'm actually feeling much better about like going in with Databricks because it's better to have , potential options in the future versus like Snowflake, which is locked in some kind of ecosystem, but then Databricks also gives me some capabilities on-prem too.

Databricks vs CSPs

The CSPs Microsoft Azure, Google Cloud Platform, and Amazon Web Services are coopetitors. Similar to Databricks’ broad plug-and-play platform, the CSPs can also do the same. AWS has Sagemaker for AI/ML, Redshift for data warehouse, Glue for ETL, and Athena for BI. Azure and GCP have their own counterparts. They’re all well resourced competitors but also partners since Databricks directly on top of customers’ cloud infrastructure. Among the three CSPs, Azure is noteworthy because it launched a direct competing product, Microsoft Fabric, while also offering Databricks as a 1st party service. Customer conversations point out that Databricks is more scalable and stable than CSPs’ native products and Microsoft Fabric is better suited for smaller workloads. That said, Databricks’ closest cloud partner might be its biggest competitor.

The quotes below are edited excerpts from Tegus’ platform.

• Customer / VP Cloud Architecture at a large insurance company — So we evaluated AWS data warehouse, Azure data warehouse, Google data warehouse and Databricks. And we found that Databricks is very cloud-agnostic. It has better scalability. On paper, on checklist, all the cloud providers do provide all those features. But when you go into deep analysis and depending on the use case, you will find that scalability in Databricks is more efficient.

• Customer / VP AI/ML Apps at a large media software company — So my company uses Databricks heavily for mostly the compute because earlier services like Azure or AWS do not have a very good compute layer and our clusters were failing all the time. And Databricks provided a very good stable alternative, especially since they're owner of Spark and our team uses Spark heavily. So the Spark clusters were very stable and reliable on Databricks. While the cost is slightly higher for Databricks, the experience and the stability we got were very much well worth it because all our time went into making the application rather than debugging the systems that we do not own.

• Global system integrator / Data architecture lead at a top 3 GSI — First of all, the Databricks is the god of capacity. Instead of EMR and SageMaker, Databricks is more robust. To give you an example. Azure has a data pipeline to do the ETL work, to curate the data, you need some logic to be created. But they have some bindings there. You cannot give everything. You cannot build everything in the Azure data pipeline. You need to bring Databricks if there is a complex pipeline. If there's complex logic is there, if there's a complex rule is there, it's really hard to implement that logic in Azure data pipeline. Same thing is for EMR or SageMaker.

• Customer / Staff solutions architect at a large telcom company — I think that probably, the simplest way to put it is that Fabric as it stands today is best designed for small to medium power -- like BI teams that are trying to expand their data capabilities without necessarily having the technical like deep dive know-how in order to administer a lakehouse. There's a number of features that are missing that, I think, are necessary for it to be an enterprise like data platform product.

• Customer / Principal data scientist at a large telecom company — I think we already make like some decision to move some workflows that require low latency to Fabric eventually. We think that approximately like 20% of all workflows will go with Fabric in span of next two years, maybe. With Fabric, we can go like even with Python or SQL to create those kinds of workloads. For future projects maybe like even up to 40% of all projects will go with Fabric.

Purpose-built tools

Aside from Snowflake and the big CSPs, Databricks also competes with point solutions, primarily startups. In a way, you can think of Databricks as bringing in together all the point solutions and optimizing it in a unified experience.

• Databricks AI Notebooks: Domino Data Lab, Hex, and many others

• Databricks AI MLFlow: Weights & Biases, etc.

• Workflows: Astronomer (Airflow), Dagster, Prefect, etc.

• Databricks SQL: Dremio, Starburst (Presto) etc.

• Delta Live Tables: dbt, Matillion

• Unity Catalog: Alation, Atlan, Acryl (DataHub)

• Delta Lake: Tabula (Iceberg), OneHouse (Hudi)

Team — One of the best

Databricks has ~5,500 employees at the end of 2023 and growing. It has a 4.4 Glassdoor rating and a 4.0 Blind rating, which is among the best. Databricks is a large company with an extensive leadership bench so we will only go through the C-Suite that Wall Street will focus on.

Ali Ghodsi (CEO and co-founder) is responsible for the company's growth and international expansion. He previously served as the VP of Engineering and Product before taking the role of CEO in January 2016. In addition to his work at Databricks, Ali is an adjunct professor at UC Berkeley and one of the creators of Apache Spark.

Andy Kofoid (President, Global Field Operations) brings nearly three decades of experience building high-growth software businesses. Andy is respondible for all aspects of the Databricks customer journey, from brand awareness to renewals. Prior to Databricks, Andy served as the President of North American Sales at Salesforce, a $12B+ business unit with over 8,000 employees.

David Conte (CFO) leads all financial and operational functions for Databricks. He has more than 30 years of finance and administration experience in multinational public and private companies within the technology industry. Most recently, David served as CFO at Splunk where he took the company public and helped it grow from $100M in annual revenue to more than $2B.

But it is also worth noting that six of the seven founders are still executives in Databricks and very much active in Databrick’s growth (go browse their blog posts). Matei Zaharia is CTO and a board member, Reynold Xin is Chief Architect, Ion Stoica is Executive Chairman, Patrick Wendell is VP of Engineering, and Arsalan Tavakoli-Shiraji is SVP of Field Engineering. The only one who is not an executive anymore (but still is an advisor) is Andy Kowinski, who went on to co-found another amazing AI company Perplexity AI.

Financials — One of the fastest growing in history

Databricks grew from $1M to $1.6B revenues (not ARR) in 8 years, making it one of the fastest growing companies in history, along with a healthy gross margin of 80% and marketing-leading NRR of 140%. Databricks expects to continue growing at >50% and be cash flow positive by 2025 / fiscal year 2026.

Valuation — Historically rich, a deal looking forward

Wall Street analysts will compare Databricks to Snowflake. The key difference between the two financial profiles is the mix of growth and margins. For Snowflake, while growth has decelerated to mid-20s % (which is still fast relative to most companies), it is has ~30% free cash flow margin. At their scale, that’ll be over $1B in cash they can reinvest annually into growing the company. In contrast, Databricks expects to burn cash for another 1-2 years but grow at a faster pace of over 50%. In today’s market, there is still a higher premium attached to growth. Databricks’ $43B series I valuation looks rich relative to near-term financials but is fairly valued or even cheap a few years out, considering the expected growth.

In this deep dive, you’ll learn insights from conversations with Scale’s customers, ex-employees, and competitors. I could do this thanks to Tegus, which centralizes expert calls into a single platform. Nothing beats primary research when it comes to understanding a company. If you’re curious about Tegus, try them out with this link.

Scale AI accelerates the development of AI applications through services and software. Its product suite has grown over time and can be mapped into the different layers of the AI stack: Data, Models, and Applications. Its core business is the suite of data solutions to collect, curate, and annotate high-quality data. Scale is trusted by top AI research labs (Open AI, Anthropic, Microsoft, Meta, Google, Cohere, Adept, NVIDIA) and the most iconic organizations as its customers (General Motors, Toyota, Etsy, Instacart, Chegg, US Army).

Why Scale AI is a generational company

• One of the fastest growing tech companies. Since 2018, Scale has been doubling every year and reached $760M in annualized run rate in 2023

• Trusted by the leading AI teams globally, including the top foundation model labs. This gives them a tailwind for the rapidly growing generative AI market

• Unmatched scale of human & software operations to annotate data at scale. Customers consistently cite Scale’s ability to handle large projects as its unique advantage

• Scale is consistently adding new products, even those that might cannibalize their core services business. This shows a willingness to keep innovating

Contents

1. Origins

2. History

3. Pain point

4. Products

5. Leadership & team

6. Market

7. Competitors

8. Financials

9. Valuation

10. Key debates

1. Origins

The story of Scale AI started with a simple and amusing problem — suspected that one of his college roommates at MIT was stealing his food. And he wanted to build a smart fridge camera to catch the alleged thief.

To build the computer vision model powering the smart camera, he turned to tutorials on Google's TensorFlow, an open-source platform for machine learning. He copied the tutorial code for training models almost word for word. His challenge was getting a labeled dataset of food images that the AI algorithm could learn from. The only way to do so back then was to do it manually. After painstakingly labeling tens of thousands of images, Alexandr finally trained a model that performed well enough. This experience underscored his insight that data is the code. The latest algorithms to train models are mainly open-sourced and available for everyone. It is data that differentiates models.

Although this insight might seem like a stroke of luck, a closer look at Alexandr's past reveals a history of pursuing opportunities where he could gain these insights. Before enrolling at MIT in 2015, he already had top-tier Silicon Valley experience. He started his career right out of high school as a software engineer at Addepar, a wealth management platform based in Mountain View. At the same time, he was couch surfing across San Francisco, pitching various tech CEOs on why they should take a chance on him. He managed to convince Adam D’Angelo, CEO & co-founder of Quora, to take a chance on him with a classic Silicon Valley pitch — he was passionate about coding and would work through anything. After two years, Alexandr returned to MIT with the experience of improving the infrastructure performance of one of the world’s most popular sites. Many of his university peers then were still learning how to submit their first Github pull requests.

Even after returning to MIT, Alexandr continued working. During an internship at Hudson River Trading, he worked on building trading algorithms. At the same time, he continued to work on personal projects. During his first (and only) spring break at MIT, he flew to San Francisco right after finals to join Y Combinator. He eventually dropped out of MIT to start Scale.

The core idea that data is the code was born at MIT, but turning that into a product required more than just a good idea. It required the practical experience Alexandr had gained at Quora and the insights of co-founder Lucy Guo from her time at Snapchat. Both Snapchat and Quora relied heavily on outsourcing manual content moderation processes to handle images and posts flagged by users. The process of finding, hiring, and managing outsourced teams was cumbersome.

Based on these experiences, Scale was built to streamline the data labeling process with just a line of code. This innovation resonated with users, and the product quickly gained popularity. It topped Product Hunt. At that time, topping Product Hunt was more a sign of genuine user excitement for the product and less the result of a concerted marketing effort.

2. History

Phase 1: Building the Data Engine (2016-2019)

In its formative years, Scale dedicated its efforts to creating a straightforward API for training data. The company quickly became a preferred provider for autonomous vehicle (AV) companies such as Cruise, Nuro, Lyft, Uber, and Waymo, all of which have substantial data requirements. By successfully meeting these needs, the company established a strong foothold in the AV sector. This success allowed the company to broaden its services to encompass a variety of use cases, including natural language processing, e-commerce, and augmented/virtual reality. During this period, Scale earned the trust of leading AI teams, becoming their go-to provider for data.

Phase 2: Building the AI Engine (2020-2022)

Having established a reputation as a reliable provider of machine learning data, Scale turned its attention to the next challenge in AI development: managing the entire life cycle of AI development across teams. To address this, the company launched several new products, including Rapid, a self-serve data labeling tool. They also began offering fully managed models-as-a-service, partnering with customers to ensure they had the necessary infrastructure to deliver high-performing models. This expansion allowed Scale AI to grow beyond merely providing data to also managing the model, thereby broadening its market opportunity.

Phase 3: Generative AI and End-User Applications (2022-present)

Scale's close collaboration with OpenAI from the early days of GPT development gave them an insider's perspective on the foundational models that have driven the current wave of generative AI. In response to this trend, the company introduced new products tailored to Generative AI, such as Spellbook, a tool for comparing models and prompts. They also launched end-user applications like Donovan, designed to assist defense and intelligence professionals in decision-making. Another key initiative is helping enterprises build custom models. As an ex-Scale employee noted —

“One of the benefits of having started off in annotation space and having built a bit of a reputation for being a thought leader in AI is that you have customers coming to you with very specific business problems.” (Tegus)

3. Pain point

In the age of deep learning, data labeling has become an even more crucial aspect of training machine learning models to perform tasks with high precision. Data labeling, also known as data annotation, is the process of assigning labels or tags to raw data, such as images, text, or audio, to create a dataset that can be used to train and evaluate machine learning models. By providing models with labeled data, developers can help algorithms recognize patterns, learn from these patterns, and eventually make predictions or decisions based on new, unlabeled data.

In computer vision applications, images must be labeled with relevant information to help train models to recognize objects or features. For example, in a self-driving car project, photos of traffic scenes may be annotated with bounding boxes around vehicles, pedestrians, and traffic signs. The labeled data is then used to train a model to recognize and respond to these objects in real-time.

To achieve passable performance, tens of thousands of labeled images are often required for low-stakes use cases. At the same time, millions of examples are necessary for more nuanced applications, such as autonomous vehicles.

A good rule of thumb for dataset sizes is as follows:

• 10,000 labeled examples make a great dataset.

• 100,000 to 1 million labeled examples make an excellent dataset.

• Over 1 million labeled examples create a world-class dataset.

Moreover, the need for labeling continues after initial training. Companies continuously strive to develop better models, and the ever-changing world requires models to adapt to new scenarios, such as learning new road signs in different countries.

While volume is essential, the quality of labeled data is more critical. In practical use cases like medical imaging, expert data labeling is required. Most people cannot read an X-ray image, requiring expert medical knowledge to label the data accurately. This can be expensive, with some companies paying $250/hour to a radiologist to label medical images. Quality assurance is still crucial even in trivial use cases like labeling cats in an image. A high-quality dataset is what distinguishes a model that appears to be randomly guessing and one that intelligently predicts outcomes.

Even for generative models, data annotation is required. As discussed in my note, Data Moats in Generative AI, most foundation models we interact with today go through a fine-tuning phase to make them follow human instructions — which is also data annotation.

Training foundation models like GPT-4 go through two main stages:

• Pre-training: In this stage, the model is exposed to a broad array of text from various sources. This exposure helps the model to learn language patterns, context, and an understanding of different subjects. The pre-training phase is crucial as it forms the backbone of the model's knowledge. It's like teaching a child the basics of language and world knowledge before they can understand complex instructions.

• Supervised Fine-Tuning (including RLHF): After pre-training, the model undergoes supervised fine-tuning, which includes techniques like Reinforcement Learning from Human Feedback (RLHF). This stage is akin to giving the model a specialized education. Here, it learns to understand and follow specific instructions, respond appropriately to queries, and refine its responses based on feedback. This stage ensures that the model is not just knowledgeable but also useful and safe in practical applications.

4. Products

Scale’s products can be segmented by layer of the AI stack (app/model/data) and by type (services/software).

Data

Scale’s data engine is a feat of software and operations. The company relies on a global workforce of around 240,000 people across Kenya, the Philippines, and Venezuela, managed through its subsidiary, Remotasks. These individuals provide the ground truth data essential to the company's success.

Training this diverse workforce, including many non-native English speakers, is challenging. Extensive training is necessary for understanding US road signs and mastering complex labeling tasks. This process is further complicated by high workforce churn. For example, the annual attrition rate in the Philippines is 50%.

Scale has developed software to automate the labeling process and review human work, resulting in a unique human-machine collaboration. Initially built with human annotations, machine learning models perform a first pass of labeling. This is then handed off to humans for review. If there's a significant difference between human and machine labels, the task is sent to more humans for further review. The company's ability to provide high-quality labels at scale and competitive pricing has drawn comparisons to Amazon.

Services

• Rapid is a self-service data annotation platform designed to expedite the production of high-quality labels. It allows users to upload their data, select or create an annotation use case, and utilize Scale's workforce to receive labeled data quickly.

• Pro is a robust data platform designed for businesses leveraging AI. It allows users to initiate labeling via an API, work with dedicated Engagement Managers for customized project setup, label large volumes of data, including complex 3D and Sensor Fusion data formats, and guarantee the highest quality labeled data.

• Test & Evaluation services involve continuous testing of LLMs to identify and mitigate risks. Red teaming, a key component, simulates adversarial attacks to uncover and address system vulnerabilities.

Software

• Studio is a comprehensive labeling platform that enhances the efficiency of a customer’s labeling team. It supports customers who prefer to label data in-house, offering tools to manage data, define annotation use cases, oversee project progress, and track labeler performance.

• Nucleus is a data management tool for machine learning that helps improve model performance by visualizing datasets, ground truth, and model predictions. It also allows for curating interesting slices within datasets for active learning and identifying critical edge cases.

Models

Services

• Custom models offering is designed to build, manage, and deploy large language model applications. This product focuses on fine-tuning large language models for improved performance on specific use cases. They allow customers to customize models to their particular use cases.

Software

• Spellbook enables teams to deploy production-ready large language model-based applications in minutes. It allows users to create and compare prompts and provides features for evaluation and comparison. Spellbook is a prompting IDE built by Scale that will enable users to go through the entire pipeline of creating and comparing prompts.

• Generative AI Platform is a full-stack solution that allows businesses to customize, build, test, and deploy enterprise-ready Generative AI applications. It enables the comparison, testing, and fine-tuning of pre-trained base models from various providers, such as OpenAI, Anthropic, and Google. It can be deployed in a company's Virtual Private Cloud (VPC) or hosted by Scale.

Applications

• Forge allows marketers and brands to create AI-generated images of their products. This tool helps create visual content for advertising campaigns, social media, and other promotional materials. With Scale Forge, users can generate various images, such as products in different scenes or with other products.

• Donovan supports decision-making processes within the defense and intelligence sectors. It can analyze structured and unstructured data, quickly identifying trends, insights, and anomalies. Donovan also offers advanced summary and translation capabilities, reducing the time needed for manual translation and auditing.

There is a slew of partially launched or sunsetted products like Synthetic, Document AI, E-commerce AI, Chat, etc. Scale likes to experiment with new products and see which ones get traction.

5. Leadership & Team

Alexandr Wang (Founder and CEO) - founded the company as an MIT dropout and became the youngest self-made billionaire. He started Scale with Lucy Guo, who left the company in 2018.

Dennis Cinelli (CFO) - Before joining Scale, Cinelli held several senior roles at Uber, including Vice President and Head of Mobility for the U.S. and Canada. His tenure at Uber also included serving as the Global Head of Strategic Finance, where he supported Uber's 2019 IPO. Cinelli's extensive experience in finance and technology includes positions at General Electric, GE Healthcare, Wabtec, and Aflac,

Arun Murthy (CPTO) - He joined the company with a rich data and product management background, having co-founded Hortonworks and served as its CPO before its merger with Cloudera, where he also held the CPO role. Murthy's experience extends to being one of the original members of the Hadoop team at Yahoo.

I don’t have an accurate count of Scale’s employee base. They laid off 20% of their ~700 employees in Jan 2023. According to LinkedIn, they have grown 70% since then. This puts their employee count at over 1,000. But Scale’s website still says they have 600 employees.

6. Market

Scale’s market opportunity can be divided into its core AI services market and select generative AI markets as its new growth vector. During its early days, Scale focused on data annotation but grew to be a fuller AI IT service provider, helping companies build production models over time. In 2023, this is a $27B market growing 20+%. This isn't particularly impressive given the competitive nature of the services market, the absence of a winner-takes-all dynamic, and the low margins.

But with generative AI, Scale’s market opportunity expanded and accelerated. They are the preferred data annotation vendor for top foundation model labs. This positions them to help enterprises build custom generative AI models. Releasing their Generative AI Platform and select apps like Donovan gives Scale another growth vector — a market that will almost double yearly to $55B by 2027.

7. Competitors

Scale’s core data annotation business is a competitive category with a long tail of vendors. Below is a select list of venture-backed competitors. This list excludes pure software providers, like Hasty (acquired by Cloud Factory), Dataloop, and Snorkel, and those focusing on particular modalities like Encord and v7 Labs for computer vision. While there are a lot of competitors, an ex-Scale senior employee commented that there hasn’t been a flagship customer that they’ve lost to customers —

“I’ve looked at all those companies at some point, and I’ve never really seen on the data annotation side anything that would tell me that they’re going to do better for the enterprise side. And I mean, I spent 14 months in that space, I haven’t really seen any one of them take off or really grow dramatically…And I don’t think I recall any impressive Scale customer leaving to go to any of them.” (Tegus)

Based on customer conversations, Scale remains the trusted resource by top AI teams because of its ability to process data at high quality and throughput. This is what convinced one of Scale’s flagship customers to move away from competitors —

“[We moved] most because of two things. One is the quality of the labeling. And the second one is the throughput…We also looked at the pricing of other solutions. Scale AI is still quite competitive in this market.” (Tegus)

Another flagship customer noted Scale’s breadth of product offering as something competitors can’t match —

“Do you know why I love Scale? Because it allows me to reduce the time of working with 15 other companies and can just work with one company. And I know that I have many features within this company…Other companies are more focused on labels or pipeline deployment of labels or managing the datasets or creating some synthetic data or providing the taskers to label your images or programmatically provide all of it. So it's more features-oriented versus Scale is across all of the different features. .” (Tegus)

As Scale becomes more of a software company, it will directly compete with C3 and Palantir. Scale’s recent focus has been on servicing the US government, having won a $250M contract with the US Department of Defense.

C3 (NYSE: AI, Mkt. Cap: $4.4B) is rooted in its IoT origins of building custom solutions for industrial and energy companies like Shell. C3 AI went public in December 2020 and has been pursuing its strategy of building a growing library of industry solutions, forging deep industry partnerships, running in every cloud, and facilitating reuse through standard data models. It now has solutions for industrial processes, supply chains, sustainability, financial services, and the public sector.

Palantir (NYSE: PLTR, Mkt. Cap: $55.4B) is rooted in building applications for complex, high-value government and commercial use cases. The company went public in September 2020 and is broadening its market appeal across industries beyond the government. The company's product platform includes Gotham, Apollo, and Foundry. Gotham is designed for the intelligence community to identify patterns within datasets. Foundry is the commercial counterpart of Gotham. Apollo is a platform that allows customers to deploy their software in any environment.

8. Financials

Scale’s ARR (annualized run rate, not recurring revenue) has been growing 114% every year since 2018. Last year, ARR grew 162% to $760M. This was driven by the foundation model labs’ need for human alignment and expertise.

Scale’s gross margin is around 50-60%, lower than 75% for the average software company. This is because of the heavy service component of data labeling.

9. Valuation

Scale was last valued at $7.3 billion when it raised a $325M series D in 2021. ARR quadrupled since then, along with a generative AI tailwind.

The exercise of valuing Scale here is to sketch how public investors might view them and not calculate an exact target price typical of equity research reports. There is not enough data to do so. There are only unconfirmed ARR and gross margin figures. If you are considering investing in Scale as a private company, factor in premiums and discounts for preference stack, major shareholder control, illiquidity, etc.

The most direct public comparables are C3 and Palantir, which are service-driven AI software companies. The comparison is imperfect because Scale engagements do not necessarily lead to recurring revenue - once the data annotation project is done, it is done. C3 and Palantir are building customized software for their customers, which becomes recurring revenue by licensing it for continued use. Data labeling company Appen would have been another comparable, but the market severely discounted their stock price after losing a key contract with Google, representing a third of their revenue. Another comparable group would be IT services companies like Accenture, IBM, Infosys, and Cognizant, whose project-based business models more reflect Scale’s. However, these companies have low single-digit growth compared to Scale’s >100% CAGR over the past few years. If Scale decides to go IPO, what valuation it’ll get depends on what narrative the investors buy into.

• Narrative 1: Scale is a high-growth service-driven AI software company. Comparable group: C3 and Palantir (11-18x NTM ARR)

• Narrative 2: Scale is an IT services company with limited long-term growth. Comparable group: IT services (2-4x NTM ARR)

• Narrative 3: Scale is a high-growth services company with the potential of having better margins. Comparable group: Median software company as a proxy for something in between narratives 1 and 2 (6-7x NTM ARR)

Public market investors put a high premium on growth and AI. Despite having a positive 15-20% rule of 40/R40 metric (NTM growth + FCF margin), IT services companies are valued at 2-4x NTM ARR. This is substantially lower than C3’s 11x NTM ARR despite having a negative 9% R40 metric. With Scale’s growth rate and market position, public investors will likely assign higher multiples to Scale around 11-18x NTM ARR. In a downside scenario, Scale might be valued as a median software company at 6-7x NTM ARR.

10. Key Debates

1. How much of their revenue will be recurring? The more recurring it is, the higher the margins and the more it will be valued like a software company. Scale only recently started selling end-user applications, its GenAI platform, and building custom LLM models for customers. This coincides with a competitive environment where startups, IT consultancies, and big cloud providers are all heavily investing in generative AI.

2. What is the longevity of Scale’s core data business? Using AI to annotate data, partially or entirely, is a growing trend. Using GPT-4 as the final evaluator of other models is common practice because GPT-4 beats the average human across many tasks. A study by the University of Zurich reveals that zero-shot ChatGPT outperforms crowd-workers and even trained individuals in annotation tasks. While human-annotated ground truth data is still considered the gold standard, it is feasible that a multimodal GPT-5 or the next-gen Mistral model could replace humans across many annotation tasks in 2024.

   A counterpoint to this is that foundation models require regular fine-tuning because data distributions will change. ChatGPT already showed performance degradation, with many users noticing it has become lazier. With change comes fine-tuning to maintain its usefulness for users.

Gamma is the canvas for modern work documents. Users can create visually engaging, interactive documents easily by themselves or with the help of an AI assistant.

Why Gamma is a generational company

• Efficient momentum: Gamma is the most popular product in its category while having raised the least amount of capital and having the leanest team

• Seasoned venture-backed operators: the founders led teams at Optimizely. Grant was the COO of ClearBrain, another venture-backed startup, before starting Gamma with Jon and James

• Rethinking the medium: Gamma is forging their own path by owning the UX, not just being an add-on to an existing platform or an AI wrapper

• Top investors: Accel led their seed round

Briefing

Slide presentations have become the lingua franca of the modern workplace. Whether it's an internal team meeting, a key client pitch, or a conference presentation, PowerPoint reigns supreme. But for many, creating slide decks can be an exercise in frustration. Templates look stale and inflexible. Formatting text and images feels like a slog. Trying to craft a compelling narrative with bullet points and stock photos is tough. Ultimately, most people just wish the deck would make itself.

That's the insight Grant Lee, James Fox, and Jon Noronha had when they founded Gamma in 2020. Their goal was to reinvent presentations by combining the simplicity of documents with the visual appeal of slides. And thanks to rapid advances in AI, Gamma may have cracked the code.

The Anti-PowerPoint

According to Lee, a significant portion of time spent on slide deck creation is not on the content itself but rather on its formatting. He observed that the emphasis on aesthetics often overshadowed the message's clarity and effectiveness.

Instead of adhering to this norm, Gamma's vision was to introduce a tool that prioritizes communication over appearance. Lee described their unique approach: "Our primitive is this concept of a card. A card can be any length and any height." This flexibility means cards could house varied content forms - text, images, video, or web embeds (Side note: I wish Substack would support more embed types). For Gamma, the objective was clear: to build an alternative to PowerPoint, a platform responsive across devices and formats.

Lee's perspective on the flaws of traditional presentations is rooted in the old "print and project" paradigm. In days gone by, slides had fixed dimensions due to the need for physical printing or projection. But in today's hybrid work environment, presentations have evolved, with many viewing them individually on their devices.

"COVID rapidly accelerated our shift away from this era of work," Lee commented. As offices started blending remote and in-person workflows, the restrictions of slide dimensions and the necessity for paper printouts became increasingly redundant.

Drawing from his personal experience, Lee reminisced, "Back in my banking days, I was the one at midnight binding these books so that the managing directors could flip through these pieces of paper, mark all their edits on the piece of paper, and hand them back to me." In the current hybrid work landscape, the potential to go beyond static slides is immense.

Closing his thoughts, Lee stated, "We're not trying to craft an incrementally better slideware tool. We're challenging the status quo by asking: what if slide decks were truly optimized for communication and content, minimizing the exhaustive work of formatting?"

Writing Like a Doc, Presenting Like a Deck - Powered by AI

By late 2022, Gamma had launched a robust editor to rearrange content into shareable presentations. "Our slogan was 'write like a doc, present like a deck,'" said Lee. "The insight there was that making slides is a painstaking visual process that’s intimidating to a lot of people." This resonated with a lot of users who wanted a simple yet beautiful way to present their work.

But teaching users to build presentations from scratch proved challenging. Gamma used to push users to a blank page and hope they figure it out. Well, 2% would and 98% wouldn't.

At the same time, AI was advancing rapidly. ChatGPT showed the creative potential of large language models. Availability of models like DALL-E made generating images simple. In response, Gamma pivoted to an AI-first approach. Users could describe their ideal presentation and Gamma would generate a custom first draft using OpenAI's API.

"It never occurred to us that AI could really transform the creation experience, nor that that was where its real potential was. That's something that I think dawned on us more over time as we built out the product," Lee admitted.

In March 2023, Gamma relaunched their product with integrated AI features. The difference was "like night and day," said Lee. Signups jumped from hundreds per day to over 10,000 after launch. The viral growth continued for weeks, accumulating millions of users globally. Lee attributes the viral growth to AI's ability to showcase Gamma's value instantly. Users sign up, describe the name of a presentation they want to make, and see it being created in real time.

But it's not just a parlor trick. Users can tweak the AI's initial suggestions using Gamma's real-time editing tools. As Lee explained, “AI lets them jump between those things very quickly. It fast forwards you right to the end, and while it won’t make something perfect, it's so much easier to edit it and tweak it." For many, AI eliminated the activation barrier that previously deterred them from trying new presentation software.

AI provided the "magic wand" to shortcut users past the learning curve of new software. This has worked well for the company with millions of users generating millions of decks each month. Gamma is now the most popular product in its category while also having the leanest team and raising the least amount of funding. This efficiency reflects Lee’s experience as a seasoned startup operator. He emphasized how much a small team with complimentary skills can deliver.

Forging Their Own Path

Gamma sits at the intersection of two red-hot trends - generative AI and future of work software. But taking on entrenched incumbents like Google, Microsoft and Canva won't be easy.

Gamma can't compete slide-for-slide in core features — yet. Instead, they're betting that AI-powered automation and focus on responsive content will open up new use cases beyond old-school slide decks. Already, many Gamma users publish their presentations as websites or share interactive embedded previews.

What’s not obvious to skeptics is that they are forging their own path by creating their own medium similar to how Notion created its unique experience. In the early days of Notion, it was often compared to Google Docs and Microsoft Word. But Notion charged ahead and today its one of the most loved and used product. It is still early days for Gamma but what makes them different from AI-wrapper companies is that AI is not the core of the product. Creating a new UX for users to better present ideas is.

If they succeed, Gamma will fulfill the promise of making presentations as easy as writing a document. For now, they've already realized the dream of many office workers - a presentation that designs itself.

Product Notes

Instead of writing up a note for Gamma, I thought the product is best experienced directly. Head on over to Gamma to see how it ingested this article to produce a doc-like presentation: Gamma on Gamma (Its awesome)

Briefings highlight generational AI scaleups, startups, and projects. I was fortunate to have chatted with co-founder & CEO Brian Raymond. Read on to learn more about his unconventional path from constitutional design expert to AI, how Unstructured came to be, and why it is a generational company.

Subscribe now

Unstructured simplifies the process of converting unstructured data into a format usable for AI, specifically focusing on Large Language Models (LLMs). Users simply upload raw files containing natural language to Unstructured's API and receive back clean data, bypassing the need for custom Python scripts, regular expressions, or open-source OCR packages.

Why Unstructured is a generational company:

• Traction: Unstructured is already generating several millions of revenue within a year of founding

• Large existing market: Extracting data from documents is an existing market and a large problem in search of a much better solution

• Seasoned, venture-backed operators: The founders helped scale the NLP startup Primer.ai

• Backed by top investors: Madrona and Bain Capital Ventures led their Series A and Seed rounds

An Unconventional path to AI

Brian has one of the most interesting founder backgrounds. Most AI founders today have technical backgrounds, many have PhDs in computer science. Brian was also pursuing a PhD. But instead of studying how computer transistors work, he was studying how countries transition from communism to capitalism. His constitutional expertise got him a fellowship at the CIA, which eventually convinced him to join the agency as an intelligence analyst before completing his degree. A CIA intelligence analyst is responsible for gathering, interpreting, and assessing information related to national security. Each focus on specific subjects like political movements, foreign governments, or emerging technologies. His focus was Iraq. I was curious about what a day-in-the-life looked like for him.

Brian described his routine as: 'I was subscribed to thousands of RSS feeds. Every morning I would come in at 6AM, then read intensively until 8-9AM. I would have hundreds of tabs open. This was what I did. Day in day out.' He maintained this regimen for five years from 2009-14. As ISIS was rising, he was tasked with becoming the Iraq Briefer for the White House and State Department. A Briefer translates the analyst's detailed reports into a digestible format, focusing on key takeaways, implications, and recommendations. Faced with the demanding role of synthesizing multiple analysts' reports, Briefers must be prepared to report to policymakers first thing in the morning. Reflecting on his transition to this role, Brian had mixed feelings. 'I was not thrilled waking up at midnight and going and prepping briefing books for the next six hours. Every morning.'

On his 2nd day as a Briefer, ISIS took over Iraq’s second largest city Mosul. This was a turning point for the international community since it demonstrated that ISIS was a force capable of capturing a large city. The following week, he was asked to serve as Country Director for Iraq at the National Security Council (NSC), which is the US President’s principal forum for national security. Within a week, his role shifted from being an information junkie analyst to policy advisor to the US president.

How is this all relevant to being a startup founder? By age 27, Brian had already honed the skills crucial for a CEO: synthesizing vast amounts of information, managing diverse stakeholders, and making high-stakes, time-sensitive decisions.

From Primer to starting Unstructured

Fast forward to 2018, Brian joined Primer, a startup helping organizations analyze massive amounts of text. There he led the company’s global public sector practice, which at one point made up 75% of the business. As a General Manager, he had to wear many hats from product manager to salesperson to customer success. It was an exciting time according to Brian. Large language models (LLMs) had just started emerging then with the publications of the Transformer (2017) and BERT (2018) papers. And Primer was at the cutting edge of it. With LLMs, Brian was helping intelligence analysts deal with increasing amounts of data. According to Primer, in 1995 an intelligence analyst reads 20,000 words a day. That grew 10x to 200,000 by 2015. In comparison, someone would only be able to go through ~100,000 words if they read at the average speed for 8 hours straight. Knowledge workers face similar information challenges from spending three hours each day searching for information, flipping through 20 browser tabs, and reading five documents to find the one that matters.

The rising amount of information also reflects the diversity of sources and content of files. Ingesting new sources into a machine learning pipeline is a significant engineering effort. Brian faced this challenge constantly while building solutions for Primer’s customers. He looked for a solution for years. First, he canvassed data integration vendors but they were focused on structured data. He then spoke to intelligent document processing companies who dealt with unstructured documents. But each iteration of filetype, layout, language, and medium requires a new data engineering pipeline. The ROI for either group of vendors was low.

Brian left Primer in early 2022 to better understand the problem. Over the next few months, he interviewed almost 100 data scientists and found a common theme: the repetitive use of regex, OCR, and Python scripts for data preprocessing. Data scientists are not rewarded for building preprocessing pipelines but rather for creating models and improving metrics. Convinced of the opportunity in streamlining unstructured data preprocessing—an issue no one seemed keen to solve—he brought on ex-Primer colleagues Crag Wolfe and Matt Robinson to start Unstructured.

What is next for Unstructured

The founders were inspired by the movement they saw in Hugging Face in which they saw thousands of data scientists & developers constantly building and experimenting on the platform. But they know that in spite of having a buffet of models to choose from, data scientists are still hampered by the preprocessing step. Unstructured helps bridge that gap. In addition to what the project already has, a significant investment they are making is in a to-be-released state-of-the-art model called Chipper. It is an OCR-free model that will look at the document, understand the content, and return the content in a nice clean format. Not only will they make Chipper freely available via open-source, they are also optimizing it for CPUs so it can be deployed at scale.

I wondered how are they going to make money — the classic open-source company conundrum. Do they sell services, offer an upgraded proprietary version, or a managed software service. According to Brian, it will be the third. They are going to offer an enterprise-oriented SaaS with user management, nice interface, admin controls, and reliable provisioning. While that does not sound sexy, that is the bedrock of the most successful open source companies today — Databricks, Confluent, Elastic.

Granted that it is convenient to list successful examples with the benefit of hindsight. The cynic would point out that most open-source companies fail to make money. By that measure, Unstructured stands out because it is already making millions in revenue within a year of founding.

The coming months will drive that even higher:

• October: Chipper model launch

• October: Azure & AWS marketplace listing

• November: SaaS offering

If you are a data scientist, check out the project and their free (for now) API service.

Unstructured Links: Website | Github | Blog | Documentation

Product Notes

To make generative AI useful for enterprises, models have to incorporate proprietary data whether via finetuning or retrieval augmented generation (RAG). Often, the data is stuck inside documents from reports to presentations to chat logs. Extracting data into a usable format is painful. Let us take digital documents as an example. While most of the world’s documents are standardized into a handful of file formats, there is an infinite variation of formatting, content, and layout inside each one. There is only so much that can be templatized. The image below shows screenshots of documents that are stored as PDFs. But building pipelines to extract data for each would drive data scientists mad, let alone a generalizable pipeline. This is why intelligence document processing companies in the past years can only specialize in one document type or use case (e.g. invoices, NDAs, contracts).

Unstructured helps solve this by offering APIs that can ingest documents regardless of file type, layout, or location and render standardized, clean machine learning-ready data. Data engineers & scientists can build generalizable pipelines for their applications. They break down the process into the following steps:

1. Ingest: Unstructured can currently ingest over 20 file types. File type determines what kind of data can be extracted. A regular .txt file does not have pages while a .pdf would.

   

2. Partitioning: Break down a document down into a primitive set of elements across document types such as title, body text, list-type items, etc.

3. Cleaning: These elements can then be cleaned for extra white spaces, annoying ASCII characters that break some libraries, and other little annoyances that typically require a bunch of regex rules.

4. Extraction: Some text or elements that have relevant information such as email addresses, phone numbers, etc. These are important metadata to filter searches.

5. Staging & chunking: These are “second” mile steps in an application where text is processed further for specific applications. Users can convert the extracted data into a CSV or chunk them into sections. One of the problems with today’s LLM applications is naive chunking, where in a text is split into sections that ignores a document’s information architecture (e.g. section, chapter). Unstructured can intelligently understand the structure to chunk text.

More than just AI applications

Unstructured is valuable even in non-AI use cases. The extracted data can improve information retrieval better by providing additional keywords and metadata to search against. Making an S3 bucket or Google drive full of PDFs searchable is extremely valuable. It is the core value propositions of many vertical startups today.

Chipper model / OCR-free document extraction

Not much yet is known about Chipper today except that is is based on Donut (Document Understanding Transformer) So we go through Donut below instead.

Imagine your typical scanner or a scanning app on your phone. You take a picture of a document—say, an invoice. Traditional systems would first read the text (a process called OCR) and then another piece of software would try to figure out what this text means. Is it a date? An item name? The software needs to piece these together like a puzzle.

Donut does it differently. When you scan the invoice, it does not only read the text; it also 'sees' how the document is laid out. It understands that the bold text at the top is probably a title, the table in the middle lists the items you bought, and the numbers at the bottom are the total cost. It is like glancing at a page in a book and understanding the whole story, not just the words. So, it does not just see words and images; it understands how they relate to each other in the document. The most practically innovative part is that it does all of this without needing the initial text-reading step (OCR). That means fewer chances for errors and a faster understanding of what the document is all about.

Subscribe now

Subscribe now

BabyAGI is an AI-powered task management agent. It automates brainstorming and task management by generating tasks according to the outcomes of previous tasks and a set objective. The system leverages off-the-shelf models, APIs, and components to create, prioritize, and execute tasks​.

Briefing

Most advances that AI Twitter folks prattle about are technical advances resulting in better, simpler, faster models. These advances are the result of smart PhDs in universities or engineers in technology companies keeping GPUs whirring all day. But last March, we saw an atypical non-technical innovation, generative agents, stir not only Twitter but also the general public media. The popular generative agent projects simply stitch together off-the-shelf models and databases. Some engineers on Twitter bemoan some of the projects because of how sloppily for-loops are put together.

But what made these projects so successful is giving the general public glimpses of AGI. Not the singularity kind but a practical kind - AI that autonomously figures out how to achieve an objective. Another interesting fact about this development is that most of the pivotal open source projects were not developed by the PhDs & developers in AI labs. Instead, they were built by hobbyists. One in particular, BabyAGI, was created by the unlikeliest of suspects: a venture capitalist.

The tinkerer

Yohei Nakajima started his career bringing the nascent Los Angeles tech community together with coworking spaces and events. One part of the job he loved was meeting and learning from people. This curiosity made him a good match for the venture industry where he eventually spent a decade bridging global corporations & startups to work together and scouring the world for startups to join Techstars. So it was foreseeable, if not expected, that he eventually started his own venture firm Untapped Capital. His profile was perfect for it. But what’s not obvious is his shadow resume - his build-in-public log. Each entry is a project, or a presentation of one, that Yohei built himself. Since starting Untapped in 2020, he has logged over 100 entries. Curiosity to learn is a common trait among investors. But what is rare is an investor who builds. And even rarer is one that builds so prolifically.

Going through his build log, I noted down three themes: low/no-code, web3, and AI. Some of the more notable projects are:

• Low/no-code: Dealflow digest, a tool to connect founders and investors

• Web3: PixelBeasts, an NFT project with 10,000 pixelated profile pics. It was the first integration of Pixels.xyz, a web3 game

• AI: Unofficial Zapier x OpenAI integration, which became the official integration

A cynic might draw a thread that he’s just following the hype cycle of what is hot in tech. But looking closer, the projects all are oriented towards helping him become a better venture investor. Tinkering helps him get a concrete grasp of the technologies he is investing in. But he is also building tools to automate the repeatable parts of his job: automating intros, drafting investment memos, answering FAQs, and many more. One of his AI projects is Mini Yohei, a chatbot that can complete regular Yohei in addressing questions from his portfolio companies.

Beginnings of BabyAGI

HustleGPT trended in early March as a creative challenge to use ChatGPT as an entrepreneurial companion - an AI cofounder - to turn $100 into your financial goals (e.g., $100,000). As a venture investor, this naturally intrigued Yohei. He is always on the look out for founders. So when he read about the HustleGPT, he wondered if it was possible to take it a step further: building an AI founder. This extra step is a subtle but is the crux to creating autonomous AI agents. In HustleGPT, a human manned the ChatGPT terminal at every turn. A fully AI founder meant autonomy. It had to think, plan, and execute on its own.

The prompt above generated a program that Yohei calls a “Task-Driven Autonomous Agent”, the predecessor to BabyAGI 1.0. For simplicity, let’s call the former BabyAGI version 0.0. The program creates agents that leverage GPT-4, Pinecone, and LangChain to autonomously plan & perform tasks based on the objective a user types in. It was built to mirror Yohei’s day to day work flow - tackling first thing on his task list, then throughout the day add new tasks, and then at night review & reprioritize tasks for the next day. BabyAGI 0.0 was then fed ChatGPT to write a pretty convincing scientific paper, which was in turn used to create a Twitter thread that garnered attention.

How long did it take him to do all of these? 2-3 hours.

Spurring the agentic AI movement

A few days later, he open sourced BabyAGI 1.0, a pared down version zero. Version 1.0 simplified the code base by removing some of the tools as LangChain and Zapier. Yohei wanted the core BabyAGI to be more of a template so that other users can easily pick it up and swap in their preferred tools. Intentional simplicity worked. BabyAGI quickly grabbed the Twitter mindhive, inspiring a wave of projects and startups. I’ve listed a sampling of them below.

• Cognosys - AI agent designed to revolutionize productivity and simplify complex tasks.

• Embra - A fast, ChatGPT-like assistant for your mac. Personalized to you — and your work.

• aiVA - The industrial AI virtual advisor your team will love. Reliable expert guidance at your finger tips, grounded in domain-specific knowledge and best practices.

• Aomni - Aomni can break down a high level research question into a step-by-step plan, and execute it for you while you enjoy your coffee.

• Nexus - a freelancer marketplace composed of AI agents

• GPTRPG - A simple RPG-like environment for an LLM-enabled AI Agent to exist in

Agents have as many applications as there are human tasks. There are over a hundred additional examples if you follow the Twitter threads here and here. With BabyAGI’s popularity came pull requests, questions, and comments, overwhelming Yohei with the amount of outreach. He publicly asked for help. He was not a professional software developer, nor an experienced open source contributor. He was a full-time venture investor.

And help did come.

One remarkable aspect of open source is that once a community is built, help comes organically. Fraser Kelton, the previous Head of Product at OpenAI and a venture investor at Spark Capital, volunteered to foster the budding community. BabyAGI now has an official website along with a Discord channel to bring the contributor community together. Of course, when a VC comes in to help an open source project, I can’t help but wonder if it’ll become a startup. But Yohei confirmed there is no company behind BabyAGI. What Fraser gave was elbow grease backed by the experience of founding a startup and leading product management at Airbnb and OpenAI. In the 10 weeks since Baby AGI launched, it has garnered 15.5K Github stars, inspired thousands of offshoot projects, and published three new mods of the original BabyAGI The latest mod, called BabyDeerAGI, improves the original program by making it smarter (stops when it finishes work) and faster (parallel task execution). To experience BabyAGI, here’s a free app that one BabyAGI contributor generously built.

I asked Yohei what is next for him. He said he’ll continue to spend time on BabyAGI (see his June 18 update below) but declined to share specifics. Not because he fears someone will steal his ideas - he did open source BabyAGI. Rather, because for him, it takes the fun out of building. Discovering, iterating, pivoting is all part of the process. A rule that Yohei follows when building is let his curiosity guide him, unrestricted by a preset commitment. What this means for the Yohei-curious, like me, is we just have to follow his Twitter. Or better yet, contribute to the BabyAGI repo.

BabyAGI Links: Github | Twitter | Discord

Yohei Nakajima: LinkedIn | Twitter

Project Notes

BabyAGI aims to be the simple framework that developers can use as the basis for building any AI assistant. Given the diverse set of tasks that assistants will be built for in the future, there is no plan to have pre-built solutions for every possible use-case within the main codebase. Instead, the plan is to provide a simple foundation which builders can start from and pair that with easy-to-follow recipes.

Using BabyDeerAGI codebase as the ‘latest’ version of BabyAGI, the key concepts are (program functions & variables in quote marks):

1. Task Abstraction: In this system, a task is a fundamental unit of work. It is represented as an object with properties that describe the work to be done (‘task’), how it should be done (‘tool’), what other tasks it depends on (‘dependent_task_ids’), its current status (‘status’), and what results it produces (‘output’). This abstraction allows any type of work to be modeled into a task, given that it can be performed by one of the available tools.

2. Tool Abstraction: Tools are methods or procedures used to perform tasks. They are abstracted in a way that allows them to be interchangeably used depending on the nature of the task. For instance, if a task requires information generation, the ‘text-completion’ tool (using OpenAI's GPT-3.5 model) is utilized. If a task requires information gathering from the web, ‘web-search’ and ‘web-scrape’ tools are used. If a task requires human input, the ‘user-input’ tool is invoked.

3. Task Dependencies: Tasks can depend on one another, creating a relationship where some tasks can only be performed after certain others are completed. This relationship is modeled through the ‘dependent_task_ids’ property of a task. This concept allows the system to handle complex objectives that require multiple, dependent steps to achieve.

4. Task Management: The system manages tasks through a task list. The task list ensures that tasks are executed in the correct order, respecting their dependencies. Tasks are continually checked and executed if their dependencies are met, updating their status and storing their output when complete.

5. Automated Task Generation: Using an AI model, the system can generate its own tasks based on a given objective. This allows it to break down complex objectives into manageable tasks that can be performed by the available tools.

6. Session Summary: The system maintains a running summary of the work it has done. This summary includes the objective and the output of all completed tasks. It provides a consolidated view of the results produced by the system.

Here’s how BabyDeerAGI works step-by-step:

1. The system starts with a high-level ‘OBJECTIVE’.

2. The ‘task creation agent’ breaks down this objective into a list of manageable ‘tasks’, each associated with a specific ‘tool’ and potentially depending on other tasks.

3. These tasks are managed within a ‘task list’.

4. The system enters a loop, continuously checking the ‘task list’ and executing ‘tasks’ whose dependencies are met.

5. Each ‘tool’ performs its specific function to complete its assigned task.

6. Upon task completion, the ‘output’ is stored, the ‘status’ is updated, and the ‘session summary’ is updated.

7. This loop continues until all tasks are complete, at which point the session summary is saved, providing a consolidated view of the accomplished work.

Subscribe now

Numbers Station is building an intelligent data stack tool powered by foundation models so data workers spend less time on mundane data tasks, and more time generating insights. They are taking a layer approach to tackling the problem space by focusing first on the data transformation stage.

Why Numbers Station is a generational startup:

• Founded by top-tier researchers and experienced operators

  • Chris Aberger was one of the first engineers at SambaNova Systems, an integrated hardware & software unicorn, where he led the machine learning team

  • Ines Chami co-authored the seminal paper on using foundation models on structured data

  • Sen Wu co-authored some of the most widely cited AI papers, including the research that led to now-AI unicorn Snorkel

  • Chris Re leads the Hazy Research group at Stanford which is, in my opinion, producing some of the most productizable AI research. Re is also a serial venture entrepreneur having started two companies sold to Apple (Lattice and Inductiv), two AI unicorns (SambaNova, Snorkel), and advising many more

• Differentiated approach to the data lifecycle journey

  • They are starting with the first step of any data process: data preparation. A large portion of generative-AI-for-data startups focus on text-to-SQL using off-the-shelf models

  • They build customized models for customers that are as accurate yet are 700x smaller and costs >2000x cheaper than OpenAI’s models

• Large growing market

  • Estimates show that current spend on data preparation & data analytics tools is $25-30 billion, growing 10%. If the trend continues, the incremental spend of $2-3 billion a year is a large opportunity on its own.

Briefing

Telling the CEO of your first employer that you’d consider yourself a failure if you didn’t resign & start your own company in four years is very unusual. But that is what Chris Aberger told Rodrigo Liang, co-founder of SambaNova Systems, when he joined the company as one of the first engineers. Four years later, Chris co-founded Numbers Station along with Ines Chami, Sen Wu, and Chris Re.

For Ines, she decided to be a founder to learn. She could have joined the AI teams of big tech companies to continue researching. But she thought that nothing beats learning from building a company. “Zero regrets. I’m learning more than I expected” as she reflects on her founder journey so far. “My biggest learning is learning how to execute a research vision from ideation to production. In research, we have to prove concepts and develop prototypes. But in startups, it is solving last mile problems and delivering a working product.”

Their vision is that foundation models can automate complex data-intensive workflows. In the seminal paper on applying foundation models to data tasks co-authored by Ines, Chris Re, & others found that large foundation models generalize and achieve state-of-the-art performance on data cleaning and integration tasks, even though they are not trained for these data tasks. While the paper was first published in May 2022, the team has already been building foundation models for years. They all saw the potential of the new technology well in advance of the ChatGPT-induced generative AI hype.

Post ChatGPT

If you tried to explaining foundation models to business executives last year, they’d be annoyed at you for wasting their time. Today, they’d be annoyed that you’re assuming they don’t know what foundation models are. Which is great according to Ines. “The tune has changed so much since we started Numbers Station. Now we can focus on talking about the value of using these models to their data.”

The top three use cases of foundation models were helping writers write, coders code, and data analysts analyze. Writing assistants were quickly commoditized. Github Copilot seems to have won the mindshare for coding assistants. But the story for the data space is still being written. The data stack is complicated, with multiple vendors for each layer. With a large market, this space is hot with startups finding a wedge and incumbents defending their positions. Dozens of “generative-AI-for-data” startups were launched in the past six months. Incumbents Microsoft, Google, Tableau, and Thoughtspot quickly launched their own products as well.

Amidst this frenzied pace of new product launches, Numbers Station is instead focusing on how to execute better internally. Ines’ view is that “The external competition is there but our pace remains [very fast]. If anything, the market is being educated faster so we can now tell users how we’re differentiated.” Aberger adds that they’re not dismissing competition. “There are a lot of really smart people and companies looking at related problems and we definitely study them to see what is going on, if there are things to learn, and how to get better ourselves. In general, I like to worry about things we can control, so [that’s] focusing internally on our execution and how to build a generational product.”

What differentiates Numbers Station

What makes the team standout is their complementary experiences at the forefront of large data systems & cutting-edge AI. Aberger led machine learning at SambaNova, which has been training LLMs for enterprises since 2020. While the other co-founders have been pioneering AI research at Stanford. Their experience is reflected in how they’re uniquely approaching the problem and their product’s technical edge.

The process to go from raw data to BI dashboards is complex. Enterprises hire a dedicated team and purchase several specialized tools to manage the process. A large proportion of generative-AI-for-data startups use foundation models to generate SQL statements to run on data warehouses. Numbers Station instead started from first principles: where does the problem start? It starts with data preparation. So their first product is focused on data preparation. In this step, SQL is just one of many tools. Cleaning typos, matching records, and un-SQL-able data transformations require different tools. So they’ve built a suite of tools for data preparation. One of which is AI transformation, a single tool to freely transform data. This can be used to judge the sentiment of a text, summarize other data entries, and correct typos. Users don’t have to train separate machine learning models, learn regex, or drudge manually transforming data.

That may sound like a simple application of foundation models. Anyone can sign up for ChatGPT and instruct it to transform each row in a data table. This is where the team’s technical prowess shines. Applying off-the-shelf general foundation models to large data sets in an enterprise setting would be prohibitively expensive and slow. In an experiment evaluating Numbers Station’s customized models, their team showed that they can build a model 700x smaller than OpenAI’s models but with similar accuracy. A smaller model runs much faster and cheaper. To illustrate the affordability and scalability of their customized models, they compared the cost of running sentiment analysis on 1M rows: Numbers Station’s inference costs $1.7 compared to OpenAI’s $3.7K for GPT-4, a 2275x cost reduction.

What’s next for Numbers Station

Data prep is a large market on its own, ~$5 billion by some estimates. But Numbers Station’s goal is much bigger: automating the entire data stack. Data prep is step one. Automating the semantic layer is next, according to Aberger. Data prep creates clean datasets. But understanding clean data is also a problem.

In a large organization, different teams will have different definitions of what an “active user” is and even how to calculate “revenue”. Should active user be one that has logged in the past 7 days or one that has done a set of activities in the past 30 days? Should the foreign exchange rate at the end of the month or on the date of billing or on the date of wire transfer be used to aggregate global revenue into a single currency? These may seem trivial, but teams do spend days & weeks to resolve inconsistencies. The semantic layer provides a common understanding of organization's data, ensures that the data is consistent and trustworthy, and helps avoid duplicative work.

The product is still in private beta but they told me to expect generally availability in a couple of months. This is exciting news if you’re someone who had to spend days correcting typos and standardizing zip codes (like me).

Subscribe now

Product Problem Space Notes

Their product is still in private preview so instead of writing about it, this section will instead describe the problem space based on the primer Generative AI for Modern Business Intelligence.

Modern BI problems & generative AI opportunities

The modularity of the modern data stack (MDS) and modern business intelligence (MBI), while beneficial, has introduced new problems: disconnected tools and unmanageable data swamps. MDS often consist of disconnected tools that need specialized knowledge to integrate. The scalability of storage & compute also leads to a store-everything mentality creating unmanageable data swamps in SQL-centric data stores. With data being ingested from different sources, understanding the context becomes difficult. Tracing back entities and tables become increasingly perplexing with each step losing context. Sometimes even requiring another tool to decipher. Consequently, teams struggle to identify the source of truth, leading to ad-hoc, bespoke tables for answering specific questions. This creates "data debt" as these one-off solutions accumulate over time.

These problems present opportunities for generative AI to unlock the value of MBI.

1. Data munging / transformation - where Numbers Station is currently today

Pain point: Data preparation tasks such as classification, transformation, and cleaning are time-consuming and tedious.

Status quo: Data analysts and engineers spend a significant amount of time on manual data wrangling, which slows down the analysis process.

Use case solution: Generative AI automates data preparation tasks. For example, suppose an organization has inconsistent date formats across multiple data sources. The AI identifies the discrepancies, standardizes the date formats, and cleans the data.

2. Data documentation (table-to-text and SQL-to-text) - where Numbers Station is going next

Pain point: Understanding and navigating complex data structures is challenging, especially when documentation is lacking or outdated.

Status quo: Data documentation is often created manually, which is time-consuming, error-prone, and difficult to maintain.

Use case solution: Generative AI generates dataset documentation, including descriptions of fields, data types, and relationships between tables. For example, when given a database schema for an e-commerce platform, the AI can create a document explaining each table (e.g., orders, customers, products) and their relationships.

3. Natural language querying (usually but not limited to text-to-SQL)

Pain point: Non-technical users struggle to extract insights from data stored in databases due to the learning curve associated with SQL or other query languages.

Status quo: Business users often rely on data analysts or engineers to write SQL queries, which is time-consuming and create bottlenecks in decision-making.

Use case solution: Generative AI converts natural language queries into SQL code. For example, a product manager asks, "What is the average revenue per user for the past three months?" The AI generates the SQL query, retrieves the information, and presents the result to the user.

Activeloop is the company behind Deep Lake, a multi-modal vector database for AI. Deep Lake enables teams to store, manage, and analyze complex data like text, video, image, and point cloud data, providing a simple API for connecting data to machine learning models. Their product offers:

• A tensor data format built for deep learning

• An in-browser data visualization engine to view unstructured data

• A Tensor Query Language (TQL) to query unstructured data

• Integrations with the most popular MLOps tooling

Why Activeloop is a generational company

• Solving a real pain point in a fast growing market

  • AI teams need tools to manage large datasets from visualizing them to making sure GPUs are maximized when training. There’s also the trend towards large models, OpenAI’s GPT-3 was trained on 45TB worth of text data. Facebook’s Segment Anything Model was trained on 1 billion images. While most AI teams won’t build their own foundation models from scratch, they’ll still need to regularly handle massive data for fine tuning

• Fast growing open-source project

  • Deep Lake is gaining open source traction with >5.7K Github stars. Over the past month they have been getting 7-8x more downloads as developers are figuring out that it can also be used as a vector store for LLM applications

• Founder-problem fit

  • Davit Buniatyan (CEO, founder) has experienced dealing with petabyte-scale data problems multiple times from his time as a PhD student at Princeton to building one of the first LLMs, years before GPT-3

Briefing

Davit's journey began while pursuing his PhD at Princeton. He originally wanted to do computer vision research but instead found himself poring through high-resolution images of mice brains for neuroscience research. While it wasn’t the path he expected, looking at brains led him back to computer science. He was studying real neural networks (dendrites and axons) to learn how to create better artificial ones. After all, deep learning models were inspired by our brains.

While he enjoyed this line of research, he also wanted to give entrepreneurship a try. This led him to collaborate with two fellow doctoral students, Sergey and Jason, to work through product ideas. The trio eventually took a leave of absence from their PhD programs to join Y Combinator in 2018. To which Davit shared an amusing anecdote about their YC interview experience:

YC: So, what did you figure out that no else figured out?

The trio were caught off-guard. PhD students like them are indoctrinated into thinking that truly novel research, one that no else but them figured out, only happens with dissertations. For the most part, they are just reading and citing other people’s work. But Jason was quick on his feet and blurted out one of the best venture-backable one-liners.

Jason: We figured out how to run cryptomining and neural network inference at the same time on a GPU better than either being done separately.

Davit and Sergey quickly glanced at Jason, surprised & impressed by his answer. Jason surprised himself too. But the reason why they were all surprised was because they hadn’t actually figured it out.

YC: That sounds awesome. Show us the benchmarks.

They had to deliver. After sheepishly giving a middle school-like excuse of leaving behind their testing laptops at their Airbnb, they rushed back and frantically ran tests to validate what Jason said. Jason, on his part, blurted out a hunch. An intuition based on his research. Fortunately, that intuition proved to be true. They eventually compiled convincing results to share with the interviewers.

YC usually tells founders if they’re accepted on the same day as their interview. They’ll get a call if they’re accepted. An email if they're rejected.

They got an email.

Fortunately it was just YC asking for a callback. They still had a chance. So they called back, eager to share the numbers.

Activeloop: We got the benchmarks!

YC: Oh. Nobody cares about the benchmarks. We just wanted to see how you guys [react]. You guys are in. Congrats.

Activeloop: ¯\(ツ)/¯

After getting into YC, the trio eventually built a secure computing infrastructure distributed across cryptomining GPUs. Cryptominers were rewarded with tokens for providing compute. This was back in 2018, making them an OG web3 company. So how did Activeloop go from web3 to AI? By going back to the original inspiration and by learning from customers. The motivation for Activeloop goes back to Davit’s experience at Sebastian Seung’s Neuroscience Lab at Princeton. He helped manage the lab’s respository of 20,000 brain images, each 100,000 by 100,000 pixels large. One of Activeloop’s earliest customers wanted a model to search through 18 million text files. Not only did they manage to slash training time from two months to just one week, they’ve also inadvertently built one of the first large language models. Even before OpenAI. Another early customer asked Activeloop to build a data pipeline to extract insights from high-resolution aerial imagery accompanied with sensor data. They were constantly dealing with large, sometimes petabyte-scale, unstructured data.

There are two ways to manage large scale data sets for machine learning. First is by having a large enough compute power, the other is by compressing data into a streamlined format for faster processing. Their original idea of coordinating a network of incentivized GPU rigs is one way to get large compute. But they learned that passing large data across a distributed, heterogenous network is too slow. They eventually abandoned the crypto angle and went down the path of compressing data. This eventually became one of their key innovations: a tensorial data format optimized for deep learning.

At some point, Jason and Sergey decided to leave Activeloop to continue their PhD studies. The Y Combinator stint was just to scratch their entrepreneurial bug. Davit had to make a choice too: go back to complete his PhD or continue building the company. This is a common dilemma for PhD founders in Silicon Valley. The Databricks co-founders also found themselves in this situation. But they had the benefit of being able to shuffle workload across seven co-founders who were each at varying points of their academic professions. Davit & team didn’t. As Sergey and Jason went their own ways, Davit soldiered on. He gave up his PhD candidacy and stayed true to what he told his PhD advisor even before starting at Princeton: he wanted to build a company.

That was in 2021. Back then, he wasn’t sure if he made the right decision. But looking at the inflection point below, it seems like he did.

Foundation models have led to the popularity of vector databases by generating embeddings that capture complex patterns in data. These embeddings require specialized storage and search capabilities, which vector databases are designed to handle. Embeddings are represented by vectors (think a list of numbers) and is just a simpler form of tensors (think multi-dimensional array). Since Activeloop natively stores data in tensor format, it is also a vector datastore by default. The open-source community noticed this and started using Deep Lake with other popular AI projects like LangChain and LlamaIndex. But being a datastore isn’t the same as being a database for real-time inferencing. There are additional indexing and querying considerations. I wasn’t able to get into specifics of what Activeloop’s roadmap is, but Davit did share that there will be a massive upgrade on this front soon. The graphic below is a bit of a teaser. If you’re building AI apps and want a datastore, you should check out Activeloop.

Links

Activeloop | Deep Lake Github | Davit’s LinkedIn | Davit’s Twitter

Product notes: Pain point and solution

Activeloop is a multi-modal data lake for deep learning on unstructured data. It is not competing against the likes of Snowflake and other analytics BI tools. It competes with traditional data lake setups like vanilla AWS S3 and with vector databases like Chroma, Pinecone, and Weaviate. Since not much yet is known about Activeloop’s vector/tensor database plans, this section will focus on the data lake side. The status quo for storing unstructured data is a data lake or a distributed file system. There are two problems with this set up:

It is slow because the data structure is not optimized for deep learning. If engineers just want to build an app that pulls a user’s JPG profile picture from the data lake every time the profile page is loaded, a standard data lake set up works. But for deep learning, images are much larger and have relevant metadata attached to it. In the case of visual deep learning, labels, bounding boxes, point clouds, dimensions, etc. are all relevant data related to each image. Transferring all of these data into the CPU/GPU is slow. Activeloop built a way to store of all of these data into a (tensorial) format that streams it directly to the chips for fast loading, training, and inferencing.

It is cumbersome because there’s no robust data tooling for handling data tooling. Its not unheard of that a data scientist would have one window with a sample image open and another with the CSV of its label. Also, off-the-shelf datasets like those from Hugging Face are already clean with instructions on how to process it. That is valuable. But there are many real-world problems where AI teams have to figure out how to turn raw data into a clean data set. That is an iterative process. Activeloop provides a way to visualize large datasets, manage dataset versions like Git, and query them with a SQL-like Tensor Query Language (TQL). Instead of having to match image IDs from a CSV with the labels, users can instead query the Activeloop interface to visualize all the relevant data.

In short, Activeloop maintains the benefits of a vanilla data lake with one key difference: it stores complex data, such as images, videos, annotations, as well as tabular data, in the form of tensors and rapidly streams the data over the network to (a) Tensor Query Language, (b) in-browser visualization engine, or (c) deep learning frameworks without sacrificing GPU utilization.

Data lakes vs vector databases

Data lakes and vector databases serve different purposes and have different features, although both can be used in the context of deep learning and querying. Here's an overview of their differences and features:

Data Lake:

1. Purpose: A data lake is a centralized storage repository designed to hold raw, unprocessed data in its native format. It can store structured, semi-structured, and unstructured data, and is used to support big data analytics, machine learning, and deep learning tasks.

2. Storage: Data lakes can store vast amounts of data from diverse sources, including text, images, videos, audio, and sensor data. They are typically built on top of distributed storage systems like Hadoop Distributed File System (HDFS) or cloud-based storage services like Amazon S3.

3. Processing: Data lakes support data preprocessing, transformation, and analysis using various tools and frameworks such as Apache Spark, Hadoop, or TensorFlow. They can accommodate complex data pipelines and workflows to prepare data for machine learning and deep learning tasks.

Vector Database:

1. Purpose: A vector database is a specialized database designed to store and efficiently search high-dimensional vectors, typically generated as a result of feature extraction from data using deep learning or other machine learning techniques. Its primary purpose is to enable fast and accurate similarity search and retrieval based on vector representations.

2. Storage: Vector databases store feature vectors, which are compact representations of the original data, instead of the raw data itself. This enables efficient storage and search but requires preprocessing and feature extraction to generate the vectors.

3. Processing: Vector databases focus on indexing and searching vectors rather than data preprocessing or transformation. They employ specialized data structures and algorithms like k-d trees or approximate nearest neighbor (ANN) search to enable fast and accurate similarity search.

In summary, a data lake is designed for storing and processing diverse raw data and supports data preprocessing and transformation for deep learning tasks, while a vector database is specialized in storing and efficiently searching high-dimensional vectors generated. Activeloop’s Deep Lake is designed to combine the best of both, enabling companies to build their own data flywheels while also powering their AI apps in production.

Special thanks to Specter for supporting Generational’s startup series. When I was still a venture investor, I have tried different systems and even attempted to build my own to help me find the best founders and companies to partner with. I’d say that Specter’s the best. Check them out.

Subscribe now

LlamaIndex (previously known as GPT Index) provides a central interface to connect your LLM’s with external data. LLMs are pre-trained on large amounts of publicly available data but to make it more practical for companies, it has to be augmented with private data. To perform data augmentation in a performant, efficient, and cheap manner, companies have to solve two components: data ingestion and data indexing. That’s where the LlamaIndex comes in. It offers:

• Offers data connectors to your existing data sources and data formats (API’s, PDF’s, docs, SQL, etc.)

• Provides indices over your unstructured and structured data for use with LLM’s. These indices help to abstract away common boilerplate and pain points for in-context learning:

• Provides users an interface to query the index (feed in an input prompt) and obtain a knowledge-augmented output.

• Offers you a comprehensive toolset trading off cost and performance.

Why LlamaIndex is a generational company

• Solving a real pain point in a fast growing market

  • One weakness of foundation models is it hallucinates and needs to be pointed to actual data. LlamaIndex provides a solution to integrate, index, and query external data sources

• Entrepreneurial and technical founders

  • Jerry Liu (CEO, creator of the project) graduated from Princeton as the co-president of the Entrepreneurship Club, published research at top AI conferences, and worked at venture-backed startups (Uber, Robust Intelligence)

  • Simon Suo (co-founder) graduated from University of Toronto and Waterloo, published research at top AI conference, and worked at venture-backed startups (Uber, Waabi)

• One of the fastest growing open source projects

  • The projected had fewer than 700 Github stars at the beginning of January 2023. It grew to 12.5K by the second week of April

Briefing

As foundation models gained popularity in the industry in 2022, Jerry Liu was keen to acquire hands-on experience with generative large language models. He wanted to build a sales bot but had trouble feeding data to GPT-3. “I fed information to GPT-3 but it had limitations on context window. I don’t want to do a fine tuning of the model. I wanted to solve this pain point. How do I store the data separately, learn how to ingest data, and do retrieval.” That experience is what led to the creation of LlamaIndex according to Jerry, creator of the project and co-founder of the company productizing it.

Jerry's interest in generative AI can be traced back several years to his time as an undergraduate at Princeton University, where he first explored generative adversarial networks (GANs) during the initial wave of generative AI in the late 2010s. GANs, which differ in architecture from today's foundation models, are primarily used for visual tasks like image synthesis, data augmentation, and style transfer. This early interest led him to AI research at Uber and a year-long stint as a machine learning engineer at Quora. His research on sensor compression and motion planning was published at top machine learning conferences, including NeurIPS and CVPR.

After 2.5 years at Uber, Jerry decided to shift from AI research to its application in industry. In February 2021, he joined Robust Intelligence, a Sequoia-backed startup that manages AI risk by monitoring and testing machine learning models. There, he led engineering for an AI firewall product designed to vet data before it enters a model.

On November 8, 2022, Jerry launched LlamaIndex via a tweet. Initially starting as a simple tree index for organizing information, LlamaIndex initially saw slow growth because Jerry found “the tree index did not work well in practice”. However, he kept building and evolved the project into a versatile LLM toolkit that supports multiple data modalities and index types. By January 2023, LlamaIndex reached an inflection point and began to see accelerated growth.

When I caught up with Jerry in March, he was just three weeks into building the project full time. I’m always intrigued to hear why founders decided to start a company, especially when its still fresh in their minds. “Seeing the community adoption by March, I knew I had something going on. And I’ve always wanted to build a company. It goes back to my Entrepreneurship Club days at Princeton.” He was the co-president of Princeton’s entrepreneurship club and led HackPrinceton, one of the largest hackathons with students from dozens of other schools participating. In hindsight, his path from big tech, to startup employee, to founder only seemed natural.

The market has seen a surge of new open-source projects involving foundation models, with SOTA alternatives emerging bi-weekly. Notable among these are the Llama-family of models: Meta's GPT-3 alternative Llama, Stanford's ChatGPT alternative Alpaca, and a cross-university ChatGPT alternative Vicuna. Most projects, however, are alternatives to OpenAI's models. Building an LLM-powered software requires more than just the base model trained on public data. Fetching and transforming private data will make LLM much more useful to companies. That is where LlamaIndex comes in. Jerry envisions it to be “the data management interface for your LLM. The tool to interface with any database, workflows, ETL jobs.” The project is currently focused on textual data, but Jerry has outlined a near-term roadmap for LlamaIndex:

1. Expanding robust libraries to support other unstructured data like images and structured SQL data.

2. Improving the ease of use and modularity of indexes to enable more complex indexing.

3. Optimizing pre-processing of data (e.g., text chunking) and querying (e.g., minimizing token usage).

Over the past few weeks, I have been personally experimenting with LlamaIndex (see personal example below) and participating in the community as a Discord member. Throughout this time, I have increasingly the practicality and versatility that it brings to the table. Developers who want to build LLM-apps and contribute to open source should check out LlamaIndex.

Links

Github page | Documentation | Discord | LlamaIndex Twitter | Jerry’s Twitter

Product notes: Pain point and solution

Indices for unstructured and structured data: Handling large amounts of unstructured and structured data can be challenging when working with LLMs due to context and prompt limitations. Finding the relevant text in arbitrary chunks can be difficult and inefficient.

The status quo today for analyzing documents means chunking the text into pieces, vectorizing each piece, feeding each to a database, vectorizing the user query to find the closest text pieces, and then feeding these as context to the LLM. This works well for short documents and simple questions but struggles with knowledge-heavy queries on long documents, like books, that go beyond the LLM's context window. Non-fiction books generally follow a logical structure, organized into sections, chapters, and paragraphs. To get a clear understanding, readers need to go through the text linearly. Naive chunking ignores this important structure, so developers have to create indexes that fit usage patterns.

Here’s my experience building with LlamaIndex. A friend suggested I check out the book The Power Law by Sebastian Mallaby. It caught my interest, but I didn't want to spend the time reading it from cover to cover. So, I bought a digital copy and fed it into LlamaIndex. The project supports multiple index structures (list, vector, tree, graph, etc.) which can be stacked on top of each other so it supports multiple Q&A and conversational situations. To support chapter-specific questions and summaries, I created a list + vector index for each chapter, allowing the LLM agent to guide questions to retrieve from particular chapter indices. The retrieved text is then fed sequentially to the LLM, making sure the final answer makes sense. In contrast, basic chunking and summarization may appear to be intelligently retrieving text but may end up randomly selecting text from various parts of the book.

Integration with external data sources: Integrating LLMs with different data types can be challenging because it requires users to preprocess and convert data into a format that LLM systems can understand.

For instance, let's say a user wants to create an AI chatbot that answers questions based on their company's internal knowledge base, which includes data in various formats. LlamaIndex simplifies this process with data connectors that handle the integration, allowing users to focus on leveraging the LLM's capabilities for their chatbot instead of dealing with data ingestion issues.

User-friendly querying interface: Querying LLMs with external data can be cumbersome, as it often requires manual crafting of input prompts and managing the output.

Take, for example, a user who wants to use an LLM to generate summaries of news articles based on their organization's collection of news sources. Without LlamaIndex, the user would need to manually craft a context-aware prompt and ensure the generated summary is accurate. With LlamaIndex's querying interface, users can simply input their request, and the system automatically creates the context-aware prompt, manages the LLM interaction, and returns a knowledge-augmented output.

Balancing cost and performance: Finding the right balance between LLM performance and the cost of deployment can be difficult, especially when choosing between advanced models like GPT-4, which are powerful but expensive and slow, and faster, less capable models like GPT-3.5.

Consider a user who wants to build a content generation tool that creates both simple and complex articles on various topics. LlamaIndex provides tools to help users optimize their LLM usage, allowing them to balance performance and cost by selecting the appropriate model for each task. Users can configure their LLM deployment to use the faster GPT-3.5 for simpler content generation and the more powerful GPT-4 for tasks that require advanced understanding and synthesis, ensuring they get the best value for their investment.

Generational is hosting a small group dinner in San Francisco on April 26 at 7PM. This will be a mix of folks building and investing at the forefront of AI. One seat is reserved for (you) readers. Interested in joining this month’s or future dinners? Reach out on LinkedIn or leave a comment.

Special thanks to Specter for supporting Generational’s startup series. When I was still a venture investor, I have tried different systems and even attempted to build my own to help me find the best founders and companies to partner with. I’d say that Specter’s the best. Check them out.

Harvey is a platform that uses natural language processing, machine learning and data analytics to automate and enhance various aspects of legal work, such as contract analysis, due diligence, litigation and regulatory compliance. Whilst the output needs careful review by a lawyer, Harvey can help generate insights, recommendations and predictions based on large volumes of data, enabling lawyers to deliver faster, smarter and more cost-effective solutions to their clients. receive new posts and support my work.

Founders: Gabriel Pereyra, Winston Weinberg

Signals:

1. Repeat founder

   • Gabriel started an undisclosed company (2018-2020)

2. Top company alumni

   • Gabriel worked at Google/Deepmind (2 yrs) and Meta (<1 yr)

   • Winston worked at O’ Melveny & Myers (1 yr)

3. Top university alumni

   • Gabriel graduated from USC (BS)

   • Winston graduated from USC (JD)

The Future Unicorn series is powered by Specter, a data intelligence provider for the world's leading investors like Accel and Bessemer. I have been working with data-driven tools for venture capital for a long time, and Specter's is the best one.

No Product Notes

No product notes for Harvey because they’re still in stealth. But its worth calling out that they’ve already partnered with Allen & Overy, the 11th largest law firm by revenue globally, to power 3,500 lawyers across 43 offices. That’s exciting.

Lexion offers a cloud-based contract management system that helps businesses of all sizes manage their contracts more effectively. Its platform is designed to automatically extract key data from contracts, such as expiration dates and renewal terms, eliminating the need for manual review. Lexion's platform also includes analytics tools that provide businesses with insights into their contracts, allowing them to identify trends and patterns.

Founding leadership: Gaurav (CEO), Emad (CTO)

Signals:

1. Repeat founder

   • Gaurav founded BillMonk (acquired by Obopay) and Precision Polling (acquired by Precision Polling)

2. Top company alumni

   • Gaurav worked at Amazon (1+ yrs) and Survey Monkey (6 yrs)

   • Emad worked at Microsoft (7+ yrs)

3. Top university alumni

   • Gaurav graduated from Rice University (BS)

   • Emad graduated from Stanford University (MS)

4. Top investors

   • Khosla led series A

   • Madrona led seed

They’re hiring across GTM and engineering

The Future Unicorn series is powered by Specter, a data intelligence provider for the world's leading investors like Accel and Bessemer. I have been working with data-driven tools for venture capital for a long time, and Specter's is the best one.

Product Notes

Pain point and person

According to World Commerce & Contracting, a professional association on contracts, 60-80% of business transactions are governed by written contractual agreements and a typical Fortune 1000 company maintains 20,000-40,000 active contracts at any given point of time. While the entire organization handles contracts in some manner, like employees negotiating their offer letter and CEOs negotiating M&A agreements, the legal team is responsible for reviewing, negotiating, and managing all contracts.

In-house legal teams are often small and nimble. With that many contracts to manage, they are often overwhelmed with contract work. A survey commissioned by legal talent provider Axiom showed that ~80% of in-house lawyers are stressed out and ~50% of those respondents are burned out. So what more specifically are the problems?

• Hunting down the final executed version of contracts can be painful because contracts are stored across different employees’ systems

• Knowing all obligations at a glance because sometimes new features or terms might be violating existing agreements

• Managing and tracking legal work because often job requests come in via email which isn’t a work management tool

Product

Lexion has five capabilities that can be grouped into Contract Management and Contract Creation capabilities:

• Contract Management: Repository, Workflow, No-code automation

• Contract Creation: Sign, AI contract assistant

Repository - a single source of truth for contracts, AI-powered automatic filing and search, automated alerts for key dates, and quick insights into contract content

Workflows - a central dashboard for managing tasks, approvals, drafts, discussions, and KPI reports related to contracts

No-code automation - a no-code automation designer that allows users to create end-to-end automated workflows. Users can automate tasks such as contract creation, approval, signature, and more based on deal details or task status changes.

Sign - like Docusign. A legally-binding e-signature with tracking.

AI contract assistant - is a new capability that is in beta. This offers inline suggestions for new languages, helps with redlining, allows users to look up clauses from existing contracts or a clause library, and generates simplified summaries of clauses and redlines to aid in responding to questions or negotiating with counterparties.

LangChain is an open-source project that helps developers create applications on top of large language models (LLMs). LLMs require a unique set of tooling to maximize its usefulness and to put guardrails around it. I used LangChain to build GPT-VC 3.0 (Streamlit cloud is having stability issues so it may not load).

LangChain is the open source project. The company is still in stealth.

Founder: Harrison Chase (and maybe others, to be confirmed)

Signals:

1. Venture-backed experience

   • Harrison was a machine learning engineer at Robust Intelligence (3+ yrs)

2. Traction

   • Grew to 7.2K Github stars in 4 months (started with a Tweet in October)

   • Packed community events (see picture below)

3. Top investor

   • Benchmark invested (to be confirmed)

4. Top university alumni

   • Harrison graduated from Harvard University

The Future Unicorn series is powered by Specter, a data intelligence provider for the world's leading investors like Accel and Bessemer. I have been working with data-driven tools for venture capital for a long time, and Specter's is the best one.

Project Notes

Since the company is in stealth, there is no commercial product yet. This section will be about the LangChain project.

Pain point

A key challenge of working with large language models is coaxing it to produce the desired outputs given a limited context window - or commonly referred to as a prompt. OpenAI’s latest GPT-3 model (text-davinci-003) can process up to ~3,000 words (4,000 tokens) for each prompt.

For everyday use cases, like writing a blog, a story, a tabloid, etc., <100 words is enough. But for more complex use cases, like a customer service chatbot, ~3,000 words is not enough to fit a company’s customer service policies and processes (and probably not a good idea to try to). LLMs don’t have stock knowledge of any company’s policies and databases. Developers have to figure out how to bring external knowledge into the model by chaining together different tools or software modules. For example above, a document fetching tool that extracts the most relevant customer support policies for each interaction would help LLMs understand the context without going the context window limit.

Product

LangChain helps developers chain together different primitives. Primitives can be a prompt template, a tool, a LLM, or even other chains. Its like LEGO. We’ll discuss the main primitives.

1. Prompt template

   An LLM takes a prompt as input to produce an output. A prompt template is a reproducible way to creating prompts with input variables that can be determined by the user or other tools. Here’s a prompt template that generates a company name based on a product, which is the input variable.

   from langchain import PromptTemplate
   
   
   template = """
   I want you to act as a naming consultant for new companies.
   
   Here are some examples of good company names:
   
   - search engine, Google
   - social media, Facebook
   - video sharing, YouTube
   
   The name should be short, catchy and easy to remember.
   
   What is a good name for a company that makes {product}?
   """
   
   prompt = PromptTemplate(
       input_variables=["product"],
       template=template,
   )

2. Large language model

   LangChain provides a common API/syntax to use the LLMs from different providers like OpenAI, Cohere, Hugging Face, and AI21 Labs. This makes it easy for developers to use and test different LLMs.

3. Document loaders

   LangChain has a lot of pre-built integrations to load documents of different types (e.g. PDF, HTML) from different sources (e.g. Notion, Google Drive, s3, websites). Loading documents make it easily accessible either for accessing directly or by processing it to store inside a vector database.

4. Utilities

   Utilities is a catch all set of integrations to access 3rd party services, aside from document leaders. One of the most common utility is accessing the Google Search API so developers can fetch Google search results to feed into the prompt.

From these primitives, developers can create a chain that takes an end user input to search for relevant documents, feed those into the prompt template, and generate an output from the selected LLM. LangChain has some complex utility chains available off the shelf like a Moderation chain that detects inappropriate text and a Math chain that translates a query into a Python calculator (LLMs are not good at math).

Claypot unifies streaming and batch systems to make it easier and cheaper for companies to do online prediction and continuous evaluation. They provide the infrastructure for other companies to build, train, and serve their models.

Founders: Chip Huyen, Zhenzhong Xu

Signals:

1. Venture-backed startup experience

   • Chip was a machine learning engineer at Snorkel (1+ yrs)

2. Top company alumni

   • Chip was a deep learning engineer at NVIDIA (1+ yrs)

   • Zhenzhong was an engineer at Microsoft (7+ yrs) and Netfllix (6+ yrs)

3. Top university alumni

   • Chip graduated from Stanford University (BS, MS)

4. Top investors

   • Lightspeed invested

   • Quiet Capital invested

They’re looking for founding machine learning and infrastructure engineers to join their team!

The Future Unicorn series is powered by Specter, a data intelligence provider for the world's leading investors like Accel and Bessemer. I have been working with data-driven tools for venture for a long time, and Specter's is the best one.

Product Notes

Claypot is still in stealth so there’s no product to talk about. But we know the problem they’re trying to solve from their white paper.

Problem and persona

Claypot is tackling the data scientists & machine learning engineers’ problems of batch predictions and manual, stateless model retraining.

Batch prediction refers to precomputed machine learning predictions calculated at regular intervals. In a scenario of you browsing an e-commerce store, batch predictions implies that the item recommendations you see are somewhat relevant in general, but irrelevant to what you are looking at the moment. If you were browsing for books yesterday and today you are looking for same-day delivery toilet paper, the store will be recommending the wrong type of paper for your restroom relief. This is not the optimal user experience because you will have to do a few more clicks (aren’t we spoiled). In some contexts though, batch prediction will not work at all. Collision detection for cars needs real-time (sometimes called online) predictions.

Manual, stateless model retraining refers to the process of manually retraining a machine learning model and the trigger to retrain the model is not because of data distribution shifts or model degradation. This was how I practiced data science years ago. I built models for banks that were manually retrained every year only because the regulator requires the banks to. Stateless retraining means training the model from scratch across the whole data set. The problem with this set up is that retraining is time-consuming (manual) and costly (runs through entire data set). Another problem is that the models might already be useless before retraining. A simple example of this is a real estate price estimation model with sqft as a key variable. In San Francisco, the average $/sqft was over $1,200 in 2021. In 2022, it rapidly dropped to $1,000. If the model only retrains once a year, the price estimates would have been off by 20% in 2022.

Unlearn creates digital twins of patients to enable biopharma partners to run faster, more successful clinical trials. Unlike a traditional clinical trial, a digital twin is created for every patient using a machine learning model trained on historical data.

Founders: Charles Fisher (CEO), Aaron Smith (Head of Machine Learning), and Jonathan Walsh (Head of Data Science)

Signals:

1. Venture-backed experience

   • Charles worked as a machine learning engineer at Leap Motion (<1 yr)

   • Aaron worked as an algorithm engineer at Leap Motion (3+ yrs)

   • Jonathan Walsh was a data scientist at Leap Motion (1+ yrs)

2. Fast team growth

   • Grew 85% YoY to ~70 employees

   • And still actively hiring

3. Top investors

   • Insight Partners led series B

   • 8VC led series A

4. Top company alumni

   • Charles was a scientist at Pfizer (1 yr)

5. Top university alumni

   • Charles graduated from University of Michigan (BS) and Harvard University (PhD)

   • Aaron graduated from Penn State (BS, MS) and University of Pennsylvania (PhD)

   • Jonathan graduated from University of Chicago (BS) and University of Washington (PhD)

This series is powered by Specter, a data intelligence provider for the world's leading investors like Accel and Bessemer. I have been working with data-driven tools for venture for a long time, and Specter's is the best one.

Product Notes

Problem and persona

Clinical trials play a crucial role in the advancement of new medical treatments and therapies. To evaluate the safety and effectiveness of these new treatments, randomized controlled trials (RCTs) are widely accepted as the standard method.

Placebo vs active: In RCTs, a placebo is a treatment or substance that has no active therapeutic effect, but is given to one group of participants (the "placebo group") in order to compare the results to another group of participants who receive the active treatment (the "active group"). The use of a placebo allows researchers to determine whether the active treatment is having a specific effect on the participants, or if any changes seen are due to other factors, such as the natural course of the disease or a psychological response to receiving treatment.

Sponsor vs investigator: In clinical trials, the sponsor and the investigator have distinct roles. The sponsor, typically a pharmaceutical or biotech company or a government agency funds and manages the trial, develops the protocol, obtains funding, and ensures compliance with regulations and ethical guidelines. They also handle the trial's overall management, data analysis, and final report. On the other hand, the investigator, a healthcare professional, conducts the trial at a specific site such as a hospital, clinic or research institution, and is responsible for participant recruitment, obtaining informed consent, and ensuring the trial is conducted according to protocol and regulations.

A problem that RCTs face: However, these trials almost always face the challenge of participant recruitment and retention. This delays the study's timeline which takes between 10-15 years and costing up to $2 billion for a single treatment. These difficulties in recruitment can stem from a lack of patient engagement or difficulty in identifying eligible participants. Delays in the study can not only result in financial losses for the sponsors but also damage the reputation of the investigators.

Product

Unlearn generates Digital Twins, which are synthetic humans that can replace control participants. Investigators can reduce the number of human patients required to run the study, accelerating the timeline.

The hurdle that Unlearn overcame is generating digital synthetic humans reliable enough for medical scrutiny. Their model starts with historical clinical trial data, which they carefully curate from multiple sources. A clinical trial dataset is a panel data composed of patients’ medical data over a long period of time. For example, in Unlearn’s initial landmark study they had data 44 variables for ~1,900 patient data over 18-months in 3-month intervals. Every 3 months, each patient underwent multiple tests to measure variables from cognitive ability to cholesterol count. The Unlearn team then used a generative AI model (a conditional restricted Boltzmann machine) to simulate the patients’ variables over time. They found that the synthetic data is hardly distinguishable from the real data.

Unlearn productized the study process as a service to pharmaceutical and biotech companies by following the steps:

1. Curate datasets

2. Build a digital twin model

3. Compare synthetic data to actual trial data

This series is powered by Specter, a data intelligence provider for the world's leading investors like Accel and Bessemer. I have been working with data-driven tools for venture for a long time, and Specter's is the best one.

Also, I’ve moved back to Substack with a new domain quild.io (in case you find something’s out of place).

Fixie is using AI to rethink how the world builds software systems. The building blocks of computing are shifting from complex, brittle, hand-coded services to flexible, general-purpose AI models that, given appropriate instructions, can perform a wide range of tasks.

They envision a future in which systems, composed of AI-powered agents, interact with people as well as each other using natural language and rich media. From this vision, they are building a new computing platform that aims to usher in a new era of software.

Since they’re still in stealth, there are no in-depth product notes this time.

Founders: Matt Welsh (CEO), Hessam Bagherinezhad (Chief AI Officer), Zach Koch (CPO), Justin Uberti (CTO)

Signals:

1. Venture-backed experience

   • Matt was the SVP of engineering at OctoML (2+ yrs) and a principal engineer at Xnor.ai (1 yr)

   • Hessam was the head of machine learning for Xnor.ai (<3 yrs)

   • Zach has worked in multiple startups the latest being a PM at Verbling (<2 yrs), an online language learning platform

   • Justin was a distinguished engineer at Clubhouse (1+ yrs)

2. Top company alumni

   • Matt was an engineer at Google (8+ yrs) and Apple (<1 yr)

   • Zach was a product lead at Google (6+ yrs) and Shopify (1+ yrs)

   • Justin was a distinguished engineer at Google (15 yrs) and was the chief architect at AOL (9+ yrs)

3. Top university alumni

   • Matt graduated from UC Berkeley (PhD) and Cornell University (BS)

   • Hessam graduated from University of Washington (PhD)

4. Top investor

   • Signal Fire invested

They’re hiring founding AI, backend, and frontend engineers. Check them out.

The updated Foundation Model Primer for Investors and Builders is out. Since its first publication last August 2022, the report has expanded: >2x in length, 3x in the number of companies include in the market map, and an outlook for next year. Check it out here:

Quild

Foundation Model Primer (Dec 2022 Edition)

Hey all, I missed this week's Future Unicorn post to write this 30+ page report on Foundation Models for you. Probably not the typical Christmas read but, hey, why not…

Read more

3 years ago · Kenn

Subscribe now

Perplexity.ai is developing a search engine using large language models. Today, it offers two services: Perplexity Ask, which allows users to ask questions and receive answers, powered by OpenAI GPT and Microsoft Bing. (2) BirdSQL, a structured search engine for Twitter that uses OpenAI Codex to translate natural language into SQL.

With news of Microsoft rolling out a ChatGPT + Bing search experience by March 2023, Perplexity would have to find a new angle or wedge. But the caliber of the founding team is still one to bet on.  

Founders: Aravind Srinivas, Dennis Yarats, Andy Konwinksi  

Signals:

1. Repeat founders

   • Andy co-founded Databricks

2. Top company alumni

   • Denis worked as an engineer at Microsoft (2 yrs), Quora (3 yrs), Facebook (6 yrs)

   • Aravind worked as researcher at OpenAI (1 yr) and Google (1 yr)

3. Top university alumni

   • Denis graduated from NYU (PhD)

   • Aravind graduated from UC Berkeley (PhD)

   • Andy graduated from UC Berkeley (PhD)

Product notes

Pain point and persona

Google is everyone's information gateway. If you have a question, whether medical, financial, or random, you Google it (or Bing it). Google does a great job surfacing the top 3 relevant links at the top of their search results. The challenge is that there is still a lot of information and clicks to go through. You would read the previews of the links, click into them, read through the website, take notes, and then synthesize all of the information.

It doesn't sound as bad as I made it to be. I grew up in the 90s where going to the library was the norm as a kid. Research back then meant going to the library, decoding the shelf indexes, curating a stack of books, looking through the page indexes, and hopefully finding the information you're looking for.

That aside, wouldn't it be nice if someone (or something) just answered the question for you concisely? OpenAI's ChatGPT opened up the possibility of a new way of searching. You can ask it any question and it'll answer concisely based on its stock knowledge of the world up until 2021.

But wouldn't it even be better if someone combined both ChatGPT and Google/Bing so the answers are up to date and attributable with website links? Its like an analyst or a librarian in a digital box. That is what Perplexity is building.

Product

Perplexity's home page is just like any search engine's. A search bar with a section on trending searches.

If you ask ChatGPT what the latest news is, it'll say that it can't browse the internet.

But if you ask Perplexity, it'll give an answer with links.

The concise answer wasn't particularly impressive for that one. But Perplexity also has a neat feature to ask for a Detailed answer. I was able to verify that Perplexity's answers are the latest news since it cites its sources.

And I love their answer to what makes a unicorn. I didn't know it also meant a third-party in a relationship. Ha.

The team at Perplexity is able to do this by combining both OpenAI and Bing APIs. Each search query is sent to the Bing API. The top Bing result snippets are included in the prompt to OpenAI's GPT, which then produces a summary that includes citations. Its a brilliant combination of two powerful APIs into a single user interface that changes how we look for information on the web.

OneSchema is an embeddable spreadsheet importer and validator. Product and engineering teams use OneSchema to avoid the costly and complicated process of building and maintaining spreadsheet import. Designed for businesses of all sizes, OneSchema empowers product and engineering teams to launch beautiful, performant, fully customized spreadsheet importers in hours, not months.

Founders: Christina Gilbert, Andrew Luo

Signals:

1. Venture-backed experience

   • Andrew was an engineer at Affinity (3+ yrs)

2. Top company alumni

   • Christina was a product manager at Google (4 yrs)

3. Top investors

   • General Catalyst led seed

   • Sequoia Capital participated in seed

   • Box Group participated in seed

4. Top university alumni

   • Christina graduated from Stanford University (BS)

   • Andrew graduated from Stanford University (BS)

Embeddable spreadsheet importer

Product notes

Pain point and persona

Underlying every SaaS product are data models to support basic functions from user registration to do the doings it was built for (a Digitally Baffled reference). For example, a CRM needs the data fields account name, creation and close dates, assigned account manager, and many more. An HRIS needs employee name, date of birth, SSN, bank account number, any many more. Developers build the data models and its relationship to each other - like the chart below for Salesforce's Sales Cloud.

As organizations adopt new products, they need to migrate current data to fit the new software's data model. This has been traditionally been done manually. But as software becomes more self-serve, software companies have increasingly built self-serve data importers for users. Spreadsheet data in the form of CSVs (comma separated values) and XLSXs (Excel) are the most common formats that business users deal with. The challenge then becomes validating that the imported data is formatted and mapped correctly. Building a scalable and easy-to-maintain impoter (data models change as a product matures) takes months of developer time.

Product

OneSchema is an embeddable CSV (for now) importer that developers can plug into their product easily. There are four main components to OneSchema's product: parser, data validation, error handling, and mapping.

Parser: processes structured data (e.g. CSV), interpreting its structure, and extracting the individual data elements that it contains into a spreadsheet. Basically, it "reads" the CSV or Excel file.

Data validation: checks each cell in the spreadsheet using rules. It determines how data should be cleaned. Validations are applied to data to ensure the data meets specific requirements (e.g. business logic) - imagine interchanging European and US date formats. OneSchema has 30+ pre-built validators from dates to country codes to SSN. Developers can also build custom validators.

Error handling: After error detection comes error fixing. Some errors are easy to fix, like capitalization. Some are more difficult, like interpolating missing values - should you use mean, median, mode, percentiles? The user experience is important as well. OneSchema can apply hot fixes in bulk and navigate & filter errors. Sometimes, select data points just don't make sense and the end user needs to corroborate with colleagues for troubleshooting. OneSchema has a neat quality-of-life feature that allows exporting the data to an Excel file with errors highlighted and annotated. In large data files, this is a life saver. Imagine having to annotate hundreds to thousands of cells.

Mapping: Mapping the data is linking the spreadsheet columns to the data model fields. It is a relatively straightforward feature. OneSchema has another neat quality-of-life feature that intelligently suggests mappings.  

If you have more time, here's a 3-min product demo.

What I'm trying to understand this coming holiday break: Implications of Geoffrey Hinton's forward-forward algorithm presented at NEURIPS. If you've thought about it or read it, would love to pick your brain over coffee/meal/eggnog.

Level AI is an omnichannel customer intelligence platform to automate quality assurance programs and provides advanced contact center analytics, personalized agent coaching, and real-time agent assistance. Level AI's semantic engine monitors and analyzes all of your contact center conversations across all channels.

Founders: Ashish Nagar (CEO)

Signals:

1. Venture-backed experience

   • Ashish was employee #3 at glass company Kinestral/Halio and search startup Relcy (backed by Sequoia and Khosla)

2. Fast team growth

   • Grew 96% YoY to ~100 employees

   • And hiring a lot more

3. Top company alumni

   • Ashish was a product manager for Amazon building Alexa AI (2 yrs)

4. Top investors

   • Battery Ventures led series A and B

5. Top university alumni

   • Ashish graduated from Stanford (MS/MBA) and IIT Delhi (BS)

Cutting-edge NLU for contact centers

Product notes

Pain point and persona

Digitization of customer interactions

The voice of a company is no longer the local store associate that customers used to see in person, but rather the contact center agent we may never talk to again. Younger generations prefer to not call companies, instead preferring to communicate via live chat, texting, email, and social media. With are a half-trillion words exchanged between companies and customers in contact centers every day, the latter has become the new digital storefront. While transcribing all these interactions is an opportunity, making it valuable and actionable is a challenge.

Its hard being a contact center agent

Contact center agents are among the most stressed and lowest paid professionals. Everything about their work is monitored and evaluated. They're expected to know everything about a product and problem - when even the people who built those product themselves don't. This has led to high turnover rates of up to 44% in large contact centers.  

Difficult to train and retain contact center talent

For managers, the high turnover rate of contact center agents and the shift to remote work during the pandemic have made training these agents more difficult. According to Gartner, 80% of contact center leaders expect an increase in remote work post-pandemic, which will further complicate the training and monitoring of agents.

Product

Level plays in the contact center quality assurance, performance management, and - to some extent - knowledge base categories.

Ingesting, transcribing, and organizing data

The first part is getting three types of data organized into Level. The first is customer data: CRMs/CDPs for basic customer info and ERPs for their purchase/order data (think Salesforce and NetSuite). The second is interaction data: this spans several channels from email to transcribed phone calls (think Twilio, Five9, Gmail, Zendesk). The last data are customer and product policies: knowledge base/wikis (think Guru, Zendesk).

Quality assurance (for contact center managers)

The QA module analyzes each interaction, including key moments, intent, and verbal behaviors. These are fed into a scoring model that scores each interaction which then rolls up to an agent score as well. By scoring all interactions, QA managers can quickly identify and resolve issues.

Real-time assistant (for contact center agents)

Agents have to triage and resolve customer requests and issues in real-time. That is what happens when agents put customers (you) on hold. Level listens to conversations and surfaces suggestions to the agents. There's a lot happening here in real-time: ingest voice data, transfer it to a transcription model, transcription happens, then another model (or set of models) figures out customer intent, then another model searches through knowledge bases to surface the most relevant information.  

Customer experience analytics (for CX leaders)

And of course, reporting. CX leaders need to know how the team is doing, how the customers are feeling, and so on. For example, if the number of unpleasant conversations have increased, managers can drill down to understand is it because a product has defects or some other issue.

OpenAI released ChatGPT last week. As a nerd, its super cool. But thinking about its broader impact gives me pause - It'll become notoriously hard to identify fake news and kids can use ChatGPT to do 80% of their homework. More testing and thinking in the coming weeks.