Yahoo, otherwise known as the company that pays my salary (full disclosure: Yahoo owns TC), today announced that it’s spinning off Vespa, the big data serving engine, into an independent venture.
Jon Bratseth, previously a VP architect in the big data and AI group at Yahoo and one of the main contributors to Vespa, has been appointed CEO.
Yahoo says it’ll continue to invest in Vespa (with cash) and remain its largest customer following the spinout. Yahoo will also own a stake in Vespa and hold a seat on the spun-out firm’s board of directors.
Yahoo created Vespa back in 2005 after acquiring paid search service provider Overture, and, through it, a Norwegian search engine called AlltheWeb.com. Working with the e-commerce division within Yahoo, the AllTheWeb team of roughly 30 people retooled their search tech into a more general-purpose tool that Yahoo developers could use internally to compute over large-scale datasets in real time.
Over the next decade or so, Yahoo expanded Vespa along several directions, enabling the tool to handle input beyond text strings, personalize content based on users’ click-through histories and take direction from machine learning algorithms. Then, in 2017, Yahoo open sourced Vespa, hoping to rally developer support behind the software — and foster something of an ecosystem both internally and externally.
It was arguably Yahoo’s biggest open source software release since Hadoop in 2006, involving a substantial rewrite of the Vespa codebase by the Vespa group, still based in Norway. And it evidently paid off. Today, Vespa drives searches and related-article recommendations on Yahoo-owned sites and performs ad targeting on Yahoo-branded web properties such as Yahoo Sports, Yahoo Finance, Yahoo News and Yahoo’s advertising network.
Yahoo claims that it’s using around 150 apps created with Vespa and that these apps collectively serve a user base of a billion people — processing 800,000 queries per second.
Outside of Yahoo, thousands of brands, including Spotify, OkCupid and Wix, now use either the open source release of Vespa or the cloud-hosted, fully managed version sold by Yahoo, Vespa Cloud. And Vespa’s open source package has been downloaded over 10 million times.
While there’s a number of open source alternatives to Vespa available, including Solr and Elasticsearch, Yahoo makes the case that Vespa goes “several steps further” than what’s on the market.
“The biggest differentiator stems from our vision of providing for the complete needs of any application that combines data and AI at scale online,” Bratseth told me via email. “We have a very experienced team who are all working on improving Vespa to this vision, and a culture of continuously shipping improvements, and — from what I can see — we’re still moving much faster than any others.”
So why spin Vespa out into its own company? Capacity, Bratseth says.
“A growing number of companies are asking to deploy their applications to our cloud service, and we’ll have more capacity to build out this functionality as a stand-alone,” he told me.
I wonder whether Yahoo’s plan to return to the public markets had something to do with the move, too. But neither Bratseth nor Lara Davis, Yahoo’s chief strategy officer, would confirm or deny this. (I tried to reach Yahoo CEO Jim Lanzone for comment through a Yahoo spokesperson, but was told he was traveling.)
“Since 2017, Vespa has been helping customers perform a variety of AI-powered tasks: searching millions of documents within a global organization, serving better data-driven online ads or allowing AI-based language apps the ability to scale,” Davis said. “We know Vespa is the market-leading technology for doing that at scale, so the opportunity for Yahoo to continue as both an investor and customer, continuing to root for their success, is a phenomenal outcome for everyone. As part of the deal, we are migrating key proprietary and battle-tested platform technology to Vespa and will continue to support many of their systems as they transition to an independent company.”
Lanzone did provide a prepared statement more or less echoing what Davis said:
“Vespa has been a critical component to Yahoo’s AI and machine learning capabilities across all of our properties for many years,” he said in a press release. “While remaining Vespa’s biggest customer and a key investor, we’ll continue to leverage all that Vespa has to offer while simultaneously creating a new business opportunity that allows other companies to harness its technology as an independent entity.”