6 VCs explain how startups can capture and defend market share in the AI era

You cannot escape conversations about AI no matter how far or fast you run. Hyperbole abounds around what current AI tech will be able to do (revolutionize every industry!) and what current AI tech will be able to do (take over the world!). Closer to the ground, TechCrunch+ is working to understand where startups might find footholds in the market by levering large language models (LLMs), a recent and impactful new method of creating artificially intelligent software.

How AI will play in startup land is not a new topic of conversation. A few years back, one venture firm asked how AI-focused startups would monetize and whether they would suffer from impaired margins due to costs relating to running models on behalf of customers. That conversation died down, only to come roaring back in recent quarters as it became clear that while LLM technology is quickly advancing, it’s hardly cheap to run in its present form.

But costs are only one area where we have unanswered questions. We are also incredibly curious about how startups should approach building tools for AI technologies, how defensible startup-focused AI work will prove, and how upstart tech companies should charge for AI-powered tooling.

10 investors talk about the future of AI and what lies beyond the ChatGPT hype

With the amount of capital flowing to startups working with and building AI today, it’s critical that we understand the market as best we can. So we asked a number of venture capitalists who are active in the AI investing space to walk us through what they are seeing in the market today.

What we learned from the investing side of the house was useful. Rick Grinnell, founder and managing partner at Glasswing Ventures, said that within the new AI tech stack, “most of the opportunity lies in the application layer,” where “the best applications will harness their in-house expertise to build specialized middle-layer tooling and blend them with the appropriate foundational models.” Startups, he added, can use speed to their advantage as they work to “innovate, iterate and deploy solutions” to customers.

Will that work prove defensible in the long run? Edward Tsai, a managing partner at Alumni Ventures, told us that he had a potentially “controversial opinion that VCs and startups may want to temporarily reduce their focus on defensibility and increase their focus on products that deliver compelling value and focusing on speed to market.” Presuming massive TAM, that could work!

Read on for answers to all our questions from:

Rick Grinnell, founder and managing partner, Glasswing Ventures
Lisa Calhoun, founding managing partner, Valor VC
Edward Tsai, managing partner, Alumni Ventures
Wei Lien Dang, general partner, Unusual Ventures
Rak Garg, principal, Bain Capital Ventures
Sandeep Bakshi, head of Europe investments, Prosus Ventures

Rick Grinnell, founder and managing partner, Glasswing Ventures

There are several layers to the emerging LLM stack, including models, pre-training solutions and fine-tuning tools. Do you expect startups to build striated solutions for individual layers of the LLM stack, or pursue a more vertical approach?

In our proprietary view of the GenAI tech stack, we categorize the landscape into four distinct layers: foundation model providers, middle-tier companies, end-market or top-layer applications, and full stack or end-to-end vertical companies.

We think that most of the opportunity lies in the application layer, and within that layer, we believe that in the near future, the best applications will harness their in-house expertise to build specialized middle-layer tooling and blend them with the appropriate foundational models. These are “vertically integrated” or “full-stack” applications. For startups, this approach means a shorter time-to-market. Without negotiating or integrating with external entities, startups can innovate, iterate and deploy solutions at an accelerated pace. This speed and agility can often be the differentiating factor in capturing market share or meeting a critical market need before competitors.

On the other hand, we view the middle layer as a conduit, connecting the foundational aspects of AI with the refined specialized application layer. This part of the stack includes cutting-edge capabilities, encompassing model fine-tuning, prompt engineering and agile model orchestration. It’s here that we anticipate the rise of entities akin to Databricks. Yet, the competitive dynamics of this layer present a unique challenge. Primarily, the emergence of foundation model providers expanding into middle-layer tools heightens commoditization risks. Additionally, established market leaders venturing into this space further intensify the competition. Consequently, despite a surge in startups within this domain, clear winners still need to be discovered.

Companies like Datadog are building products to support the expanding AI market, including releasing an LLM observability tool. Will efforts like what Datadog has built (and similar output from large/incumbent tech powers) curtail the market area where startups can build and compete?

LLM observability falls within the “middle layer” category, acting as a catalyst for specialized business applications to use foundational models. Incumbents like Datadog, New Relic and Splunk have all produced LLM observability tools and do appear to be putting a lot of R&D dollars behind this, which may curtail the market area in the short-term.

However, as we have seen before with the inceptions of the internet and cloud computing, incumbents tend to innovate until innovation becomes stagnant. With AI becoming a household name that finds use cases in every vertical, startups have the chance to come in with innovative solutions that disrupt and reimagine the work of incumbents. It’s still too early to say with certainty who the winners will be, as every day reveals new gaps in existing AI frameworks. Therein lie major opportunities for startups.

How much room in the market do the largest tech companies’ services leave for smaller companies and startups tooling for LLM deployment?

When considering the landscape of foundational layer model providers like Alphabet/Google’s Bard, Microsoft/OpenAI’s GPT-4, and Anthropic’s Claude, it’s evident that the more significant players possess inherent advantages regarding data accessibility, talent pool and computational resources. We expect this layer to settle into an oligopolistic structure like the cloud provider market, albeit with the addition of a strong open source contingency that will drive considerable third-party adoption.

As we look at the generative AI tech stack, the largest market opportunity lies above the model itself. Companies that introduce AI-powered APIs and operational layers for specific industries will create brand-new use cases and transform workflows. By embracing this technology to revolutionize workflows, these companies stand to unlock substantial value.

However, it’s essential to recognize that the market is still far from being crystallized. LLMs are still in their infancy, with adoption at large corporations and startups lacking full maturity and refinement. We need robust tools and platforms to enable broader utilization among businesses and individuals. Startups have the opportunity here to act quickly, find novel solutions to emerging problems, and define new categories.

Interestingly, even large tech companies recognize the gaps in their services and have begun investing heavily in startups alongside VCs. These companies apply AI to their internal processes and thus see the value startups bring to LLM deployment and integration. Consider the recent investments from Microsoft, Nvidia, and Salesforce into companies like Inflection AI and Cohere.

What can be done to ensure industry-specific startups that tune generative AI models for a specific niche will prove defensible?

To ensure industry-specific startups will prove defensible in the rising climate of AI integration, startups must prioritize collecting proprietary data, integrating a sophisticated application layer and assuring output accuracy.

We have established a framework to assess the defensibility of application layers of AI companies. First, the application must address a real enterprise pain point prioritized by executives. Second, to provide tangible benefits and long-term differentiation, the application should be composed of cutting-edge models that fit the specific and unique needs of the software. It’s not enough to simply plug into OpenAI; rather, applications should choose their models intentionally while balancing cost, compute, and performance.

Third, the application is only as sophisticated as the data that it is fed. Proprietary data is necessary for specific and relevant insights and to ensure others cannot replicate the final product. To this end, in-house middle-layer capabilities provide a competitive edge while harnessing the power of foundational models. Finally, due to the inevitable margin of error of generative AI, the niche market must tolerate imprecision, which is inherently found in subjective and ambiguous content, like sales or marketing.

How much technical competence can startups presume that their future enterprise AI customers will have in-house, and how much does that presumed expertise guide startup product selection and go-to-market motion?

Within the enterprise sector, there’s a clear recognition of the value of AI. However, many lack the internal capabilities to develop AI solutions. This gap presents a significant opportunity for startups specializing in AI to engage with enterprise clients. As the business landscape matures, proficiency in leveraging AI is becoming a strategic imperative.

McKinsey reports that generative AI alone can add up to $4.4 trillion in value across industries through writing code, analyzing consumer trends, personalizing customer service, improving operating efficiencies, and more. Ninety-four percent of business leaders agree AI will be critical to all businesses’ success over the next five years, and total global spending on AI is expected to reach $154 billion by the end of this year, a 27% increase from 2022. The next three years are also expected to see a compound annual growth rate of 27% — the annual AI spending in 2026 will be over $300 billion. Despite cloud computing remaining critical, AI budgets are now more than double that of cloud computing. Eighty-two percent of business leaders believe the integration of AI solutions will increase their employee performance and job satisfaction, and startups should expect a high level of desire for and experience with AI solutions in their future customers.

Finally, we’ve seen consumption, or usage-based priced tech products’ growth slow in recent quarters. Will that fact lead startups building modern AI tools to pursue more traditional SaaS pricing? (The OpenAI pricing schema based on tokens and usage led us to this question.)

The trajectory of usage-based pricing has organically aligned with the needs of large language models, given that there is significant variation in prompt/output sizes and resource utilization per user. OpenAI itself racks upward of $700,000 per day on compute, so to achieve profitability, these operation costs need to be allocated effectively.

Nevertheless, we’ve seen the sentiment that tying all costs to volume is generally unpopular with end users, who prefer predictable systems that allow them to budget more effectively. Furthermore, it’s important to note that many applications of AI don’t rely on LLMs as a backbone and can provide conventional periodic SaaS pricing. Without direct token calls to the model provider, companies engaged in establishing infrastructural or value-added layers for AI are likely to gravitate toward such pricing strategies.

The technology is still nascent, and many companies will likely find success with both kinds of pricing models. Another possibility as LLM adoption becomes widespread is the adoption of hybrid structures, with tiered periodic payments and usage limits for SMBs and uncapped usage-based tiers tailored to larger enterprises. However, as long as large language technology remains heavily dependent on the inflow of data, usage-based pricing will unlikely go away completely. The interdependence between data flow and cost structure will maintain the relevance of usage-based pricing in the foreseeable future.

Lisa Calhoun, founding managing partner, Valor VC

There are several layers to the emerging LLM stack, including models, pre-training solutions, and fine-tuning tools. Do you expect startups to build striated solutions for individual layers of the LLM stack, or pursue a more vertical approach?

While there are startups specializing in parts of the stack (like Pinecone), Valor’s focus is on applied AI, which we define as AI that is solving a customer problem. Saile.ai is a good example — it uses AI to generate closeable leads for the Fortune 500. Or Funding U using its own trained dataset to create a more useful credit risk score. Or Allelica, using AI on treatment solutions applied to individual DNA to find the best medical treatment for you personally in a given situation.

Tools like Datadog can only help the acceptance of AI tools if they succeed in monitoring AI performance bottlenecks. That in and of itself is probably still largely unexplored territory that will see a lot of change and maturing in the next few years. One key aspect there might be cost monitoring as well since companies like OpenAI charge largely “by the token,” which is a very different metric than most cloud computing.

What can be done to ensure industry-specific startups that tune generative AI models for a specific niche will prove defensible?

Focus on what you’re solving for the client. It’s almost 100% certain AI is only solving a part of any problem. How you tie all the parts together is what creates a great solution.

Over the next cycle, we expect the average applied AI customer to have about as much “AI technical competence” in house as they have current technical competence, which in general is not a lot. It remains to be seen if things like “prompting” and “fine-tuning” will become part of the expected in-house experience, but that takes time. Startups that solve for a natural interface are going to be ahead of the curve.

So far we see most of our AI startups sticking with a traditional SaaS pricing model. On the back end, usage-based tiering was already the norm (AWS, Google), so it’s not so different and venture-backed companies already have a vast knowledge base on how to segment price.

Edward Tsai, managing partner, Alumni Ventures

I think there will be plenty of startups on both sides. However, I think the bar for good infrastructure is higher given the risk of competition from OpenAI, Anthropic, Cohere, and other foundational model companies expanding their LLM stack-adjacent tooling and solutions.

“Vertical” (end to end) startups that provide “vertical” (sector) solutions that make it easy to solve business solutions and need minimal extra tooling I think are worth meaningful consideration. In the SaaS world, many vertical specific end-to-end companies have done well like Toast and Procore. In the same way, AI companies that provide vertical solutions with easy-to-use workflow and industry-specific data enrichment can be very useful to their end enterprise customers.

Incumbents like Datadog certainly take some market share when they deploy a good product to their users. That said, customers have many different needs. As there are many other data and APM observability startups competing with Datadog, there are a number of AI-specific ML and LLM observability players like Arize and Aporia. Similar to the cybersecurity world, there are startups focused on cybersecurity for AI like HiddenLayer and Protect AI. The U.S. enterprise market is used to a blend of choosing large platforms and best-of-breed solutions.

How much room in the market do the largest tech companies’ services leave for smaller companies and startups tooling for LLM deployment?

A tangential view is that there continues to be a healthy set of open source tooling related to LLMs and AI. A number of projects have successfully raised capital, be it LangChain, Weaviate, or Together. Open source startups benefit from the contributions of the community, which long-term I think is a competitive advantage.

What can be done to ensure industry-specific startups that tune generative AI models for a specific niche will prove defensible?

Good industry-specific product workflow that requires some industry-specific know-how and lift (niche industry integrations, finding data sources that aren’t immediately available) raises the bar for startup competition, making it harder for incumbents and new entrants with less industry-specific knowledge or willingness to go deep to compete.

That said, I’d like to suggest a controversial opinion that VCs and startups may want to temporarily reduce their focus on defensibility and increase their focus on products that deliver compelling value and focus on speed to market. The dramatic improvement of “human-like” LLM performance over the last year has enabled startups to deliver significant value to end customers very quickly. Hence I believe in many markets “someone will do it.” Some companies are going to jump to do products with significant value but not immediately clear how defensible the market is. The early customer growth and revenue growth will help the startup get above the increasing noise in the market and give time, resources, and customer eyeballs to deploy new products in more defensible niches.

There are markets for a whole gamut of solutions, from open source solutions that need a tech team to deploy to easy-to-use, drag-and-drop solutions for business users. Startups should consider what range and entry point they want to use to deploy their product. Simplicity and beauty are hard. Hence I think designing AI and LLM related products for enterprise AI customers with a lens of simplicity and usability may create a slight edge for startups versus large tech companies. Some large tech companies may not be willing to make the product trade-offs for simplicity and usability in order to service its largest customers who require significant ability for customization.

As underlying foundational model performance goes up and pricing comes down, we will likely continue to see more traditional SaaS-based pricing. In some use cases, where based on deployment, significant ongoing API calls to foundational models is required, there may be a hybrid model where usage is capped or throttled for abnormally high usage or a custom “enterprise” model.

Wei Lien Dang, general partner, Unusual Ventures

Both, but most will start with individual layers. For example, a company like Lamini is focused on fine-tuning. LlamaIndex is working on data management. LangChain started with handling sequences of LLM calls before becoming more of a toolkit. Even companies that provide an end-to-end platform like MosaicML started with a particular focus (training). Over time, more will pursue a vertical approach because no one wants to use a dozen different tools to build an LLM app, but each of them will still have a core specialization.

I’ve personally seen more than a dozen new startups trying to apply LLMs to observability, all with the same story. It means startup founders have to think extra hard about the “Why now?” for their companies beyond just being able to use LLMs. LLMs by themselves aren’t necessarily a reason for a new stand-alone company to exist. Datadog and other incumbents aren’t reducing the overall market size but they are certainly putting more pressure on startups to be able to differentiate their products and compete from a GTM perspective.

How much room in the market do the largest tech companies’ services leave for smaller companies and startups tooling for LLM deployment?

There’s still a lot of room because if nearly every company is going to be using LLMs/AI, only a small fraction of them are going to have the desire and budget for a substantial professional services engagement. We’ll see more startups providing tooling for LLM deployment that doesn’t require a dedicated AI infrastructure team, too. This is what’s driving the idea of an “AI engineer” — AI will have its largest impact when the 30 million+ developers out there can easily use LLMs to build applications. OpenAI, Anthropic, Cohere, and others have set a certain expectation among users around making it easier to get started with LLMs.

A lot, or at least it should guide it a lot. I see many founders, especially those coming from AI/ML backgrounds believing that their end users will have the same level of expertise as them. But it depends on what market segment and user persona they’re targeting. As an example, many F500 companies have in-house ML teams and a high level of technical sophistication from using ML for fraud detection, recommendations, and other use cases. As you go downmarket, there is much less firsthand experience working with AI and yet there is a groundswell of interest in getting started in a self-service way. The level of existing AI expertise someone has completely impacts how they describe their problem, the product they need to solve it, and how they want to adopt.

Yes. Both infrastructure and application companies are going to move away from usage-based pricing over time as they look to capture more value from their customers through features and better developer/user experience. Investors are also going to pay more attention to it because it directly impacts the margin profiles of these businesses.

Rak Garg, principal, Bain Capital Ventures

There are a couple of layers that I think stay horizontally striated, namely the chipset and pre-training/fine-tuning layers toward the bottom of the stack. It’s hard for me to imagine that a startup would build a vertical solution that goes that far deep into the stack. Above those layers, I think we’ll see both striated solutions and vertical approaches, depending on the specific customer profile. For the technology-forward, build over buy type of company, I expect them to consume specialized, striated providers of each layer of the stack in a best-of-breed way. This gets them ultimate flexibility, customizability, and domain-specificity to their own context and circumstances. Regulated industries are another example of this — they can’t consume most commercial products because of compliance reasons and will consume various components to build in-house. For more functional areas within non-regulated companies, like legal, finance, support, operations, etc., I expect them to buy verticalized apps that bundle everything that specific group needs, similar to how the sales team might bring in HubSpot instead of asking a central data team to build a CRM.

The macro environment is putting pressure on many enterprise budgets, and it’s easier to expand, spend or win favorable pricing with an existing vendor than try to bring a new one in. This dynamic is what causes incumbent LLM products to pose such a threat to startups. The market opportunity is still there, especially in areas like security and observability, which historically have favored best-of-breed products to incumbent suites, but budget noise created by incumbent solutions that are 80% as good will make GTM challenging for a lot of startups. I encourage companies to focus on new opportunities: the areas that prior solutions could never automate well, and the areas that can be reimagined to be 100x better now. Anything less will not be sufficient.

How much room in the market do the largest tech companies’ services leave for smaller companies and startups tooling for LLM deployment?

There’s still quite a bit of room for infrastructure and for vertical applications of software. In infrastructure, the largest tech companies are the few places in industry that actually have LLMs running in production, at scale, for a variety of use cases. The people working in and around these technologies could leave to start companies that handle various jobs to be done in productionizing LLMs, including inference, data curation, unstructured data ETL, RLHF playgrounds, and more. In vertical applications, especially in regulated industries and in manual roles, there is a lot of opportunity due to the lack of technical sophistication in many incumbents.

What can be done to ensure industry-specific startups that tune generative AI models for a specific niche will prove defensible?

Recent papers like LIMA and Chinchilla are increasingly pointing toward data quality, curricula learning, and data curation as key contributors to LLM quality. We’ve learned from our own portfolio that access to the highest quality, and most differentiated data for training or fine-tuning makes a real difference to end users. Aside from model quality, products must be good enough to win users and prevent them from churning — the same as it always was.

In my experience speaking with dozens of F500 buyers, large companies have a wealth of technical competence across the organization but can be a bit lethargic in harnessing it across the organization. Startups that are building infrastructure will have to convince buyers that the timing is right, that the increased overhead in technical operations will be worth the payoff, and that the infrastructure solves a real problem that the company is struggling with right now. Startups building applications need to appeal to a functional user that will champion them within the organization, and that process has less to do with technical competence and more to do with understanding the jobs to be done that the champion has on their plate.

The economics of training and running inference, and the pricing models of hosted API providers like OpenAI, essentially require the companies using those providers to pursue a usage-based component to their pricing. We’re seeing a mixture of platform fees (e.g., $100k per year for your organization to access the platform) and usage-based fees (e.g. $10k for 100 compute credits which are expended according to a specific schedule) to make it worthwhile to serve LLMs-enabled features to end-users.

Sandeep Bakshi, Head of Europe Investments, Prosus Ventures

Building solutions across the stack continues to be extremely interesting, and we have spent most of our time thus far in the vertical specific and tooling space (i.e., picks and shovels). The vertical approach, as exhibited by our portfolio company Corti, an AI co-pilot for healthcare, allows companies to build for a specific enterprise with a specific buyer persona in mind. It also enables the startup to focus training their models on vertical specific datasets, which allow for more relevant answers as opposed to results from broader datasets.

What can be done to ensure industry-specific startups that tune generative AI models for a specific niche will prove defensible?

This will come down to their ability to have models that are trained on data that is specific to their niche and for said data to be somewhat proprietary. Ultimately, it also depends on the buyer persona again. As expected, selling into large enterprises is more time-consuming with a larger barrier to entry, as there are more complex and multifaceted approval processes. Thus, once a company sells into an enterprise and continues to deliver an exceptional product and compelling customer experience, it is by nature defensible.

This is dependent on the enterprise AI customer. Large tech incumbents (which are enterprise customers) will have their own technical talent and will have the ability to also hire cutting-edge technical talent given the nature of their work. The way I like to think about this question is “What would attract a technically competent employee to a company?” We believe this answer lies in whether that enterprise is doing technically challenging work. For enterprises focused on verticals, the technical nature of the AI work will likely be more about assessing solutions and implementing them in their specific vertical as opposed to building, and thus we’d find that these enterprises have more demand for out-of-the-box solutions.