The massive explosion of generative AI models for text and image has been unavoidable lately. As these models become increasingly capable, “foundation model” is a relatively new term being tossed around. So what is a foundation model?
The term remains somewhat vague. Some define it by the number of parameters, and therefore, how large a neural network is, and others by the number of unique and hard tasks that the model can perform. Is making AI models larger and larger and the model’s ability to tackle multiple tasks really that exciting? If you take away all the hype and marketing language, what is truly exciting about these new generations of AI models is this: They fundamentally changed the way we interface with computers and data. Think about companies like Cohere, Covariant, Hebbia and You.com.
We’ve now entered a critical phase of AI where who gets to build and serve these powerful models has become an important discussion point, particularly as ethical issues begin to swirl, like who has a right to what data, whether models violate reasonable assumptions of privacy, whether consent for data usage is a factor, what constitutes “inappropriate behavior” and much more. With questions such as these on the table, it is reasonable to assume that those in control of AI models will perhaps be the most important decision-makers of our time.
Is there a play for open source foundation models?
Because of the ethical issues associated with AI, the call to open source foundation models is gaining momentum. But building foundation models isn’t cheap. They require tens of thousands of state-of-the-art GPUs and many machine learning engineers and scientists. The realm of building foundation models to date has only been accessible by the cloud giants and extremely well-funded startups sitting on a war chest of hundreds of millions of dollars.
Almost all the models and services built by these few self-chosen companies have been closed source. Still, closed source entrusts an awful lot of power and decisions to a limited number of companies that will define our future, which can be quite unsettling.
We’ve entered a critical phase of AI where who gets to build and serve these powerful models has become an important discussion point.
The open sourcing of Stable Diffusion by Stability AI, however, posed a serious threat to the foundation model builders determined to keep all the secret sauce to themselves. Cheering from developer communities around the world has been heard regarding Stability’s open sourcing because it liberates systems, putting control in the hands of the masses vs. select companies that could be more interested in profit than what’s good for humanity. This now affects the way insiders think about the current paradigm of closed source AI systems.
Potential hurdles
The biggest obstacle to open sourcing foundation models continues to be money. For open source AI systems to be profitable and sustainable, they still require tens of millions of dollars to be properly run and managed. Though this is a fraction of what the large companies are investing in their efforts, it’s still quite significant to a startup.
We can see Stability AI’s attempt at open sourcing Neo-GPT and turning it into a real business fell flat, as it was outclassed by companies like Open AI and Cohere. Tthe company now has to ideal with the Getty Images lawsuit, which threatens to distract the company and further drain resources — both financial and human. Meta’s counter to closed source systems through LLaMA has poured gas in the open source movement, but it’s still too early to tell if they’ll continue to live up to their commitment.
The good news is that models are becoming smaller with very recent techniques like low-rank adaption (LoRa) and chain of thoughts (CoT) prompting. But they still require many iterations to make them commercially viable, which naturally entails millions of dollars and lots of compute power.
Right now the majority of open source generative AI companies fail, and we have no data points on how “big” successful open source AI projects could become. That makes it difficult for VCs to write the kind of checks such projects need — particularly at a time when the banking system is so fragile. While this could improve in the future, it currently translates to strategic investments, which reinserts the Big Tech companies.
Regardless of where funds potentially come from, we must face the reality that after the hype goes away and marketing messages wane, challenges remain on the fronts of sustainability, risk-to-success ratio and profitability in addition to maintaining quality and accuracy to ensure value. In short, open source AI companies must figure out how to become real businesses, which has long been the biggest impediment.
Implications for startups
As we’ve seen with the mass proliferation of ChatGPT, foundation models are the future, but how they will influence it remains to be seen. A new breed of startups is rising up to do all sorts of incredible things — whether they are built as closed or open source. In an effort to decide their best path forward, founders may ask themselves difficult questions related to all aspects of their business. For example:
- Funding: Where is your money coming from? Will you have deep enough pockets to go it alone as a closed source company? If not, what kind of partner will you seek? Can you maintain control of your project? If you go the open source route, how will you fund it?
- Team: Does your project lend itself to community input and development, or do you require strict quality controls? How will you be able to attract the best talent to build out your project?
- Experimentation: How will people be able to experience your project to learn about it and test it out? Will you open trials and do extensive marketing? Will you open a playground for developers? Do you plan to generate excitement through open source community networks?
- Loyalty: How will you create your own community of users? What happens if you change approaches, i.e. how much will you lose reputationally and in terms of users if you start as an open source project and then must go the opposite direction if a partner with resources demands it?
- Ethics and education: What are your responsibilities when it comes to how your project is used? What will you do if it is co-opted for unintended, harmful purposes? Will you step in if you see “inappropriate behavior,” and if so, to what degree? How will you monitor your project and convey your policies to customers/users? In short, where are the guardrails?
This is only a start. There are so many more questions than answers right now.
Determining future success
As startups move forward in building foundation models for a particular niche, it is important that they recognize new milestones will be required to evaluate their relevance and ultimate value. As Radical Ventures’ Ryan Shannon recently noted:
Unlike a traditional startup, which can simply write code, ship a product and iterate on customer feedback, Foundation Model companies need to spend more time building and training their models in order to get a product to a position in which it’s viable and ready to be used. Often, this can take multiple years, millions of dollars (or … hundreds of millions of dollars) and several iterations before products are good enough for companies to charge customers to use them.
This is a tall order — and one that may require a leap of faith from investors or members of a larger community. The upfront investment in foundation models is substantially higher than that which other startups need, but the adoption on the back end can be unprecedented. These are transformational technologies unlike anything we’ve ever seen.
With the right amount of time, money and talent, foundational models — be they open or closed — will not only usher in the future, but control it to some extent. Foundation models will guide how we consume the information that shapes our perspectives and decisions, making a profound impact on how society communicates, learns, understands and creates.
The stakes are incredibly high. Open source models need to figure out a business model that works over the long term while closed models must address ethical concerns head-on, with behavioral guardrails and oversight in place. No perfect solution has emerged in this messy, rapidly evolving landscape, but confronting the big questions and examining our responsibilities is essential to innovation. When we consider all that is possible — both for good and for bad — protections are uncovered and real progress happens.