Regular search engines such as Google and Yahoo use statistics to make sense of the Web. They count links, keywords, and other items on a page to determine its rank in search results. Semantic search engines try to actually understand the meaning of the words found on the Web and other documents to bring back the most relevant results to a query. Microsoft bought Powerset for $100 million to gain semantic search expertise, but so far all it can search is Wikipedia.. Hakia, Textwise, and other startups are also working on semantic search. Now comes NetBase, which brings a slightly different approach that its says can scale to the entire Web.
NetBase has been around for a while. Originally called Accelovation, it has raised $9 million in two rounds of venture funding over the past four years, has 30 employees, and counts among its current customers P&G, Caterpillar, 3M, BP, Kraft, BASF, and Goodyear. It is now changing its name and offering its core semantic indexing technology as a platform for other companies to build their own products. Already, scientific publisher Elsevier uses NetBase to power its Illumin8 research tool for searching scientific articles, patents, and Websites.
NetBase takes a sophisticated linguistic approach, actually diagramming sentences to determine the relationship between words and phrases. It does particularly well with causal relationships, allowing it to tease out cause and effect from raw text. → Read More
Microsoft promises that this is just the beginning of the integration with the recently acquired Powerset, but incorporating better Wikipedia clips into Live Search is a far cry from the original promise of the next generation search startup: true natural language search.
Instead we have more Live Search results with dedicated answers (not sure why Powerset was needed for this), and better Wikipedia results (“Since Wikipedia articles show up in a large percentage of Live Search queries, it’s important that the captions are top notch.”). → Read More
There is one glaring detail that everyone who reported on the Microsoft-Powerset acquisition has failed to mention: Powerset is a Unix based company. All of its core components – its applications, its packaging system, and its continuous integration system – were designed and written to run on a Unix platform. As a large software company, Microsoft is a big supporter of its own craftsmanship. Unsurprisingly, all of Microsoft’s products, including Live search, run on Windows servers. Thus we come to the source of the Microsoft-Powerset repugnance. While not impossible to port software from Unix to Windows, but there is no denying that it is a hellish task. Powerset has developed a large repository of code over the last three years and naturally it has opted to utilize many open source projects in its development stack which all would have to be ported as well. A large chunk of the software that it uses is already cross platform, but those that are not present a real problem. When Powerset launched, developer Kevin Clark gave a nice shout-out to the open source technologies that Powerset relies on. The state of each of those technology’s compatibility with Windows is given below. Windows Compatibility of the open source technologies Powerset uses Hadoop/Hbase Requires Cgywin to run on Windows. Ruby Runs slow on Windows, good opportunity for Microsoft to put IronRuby to use? Ruby on Rails Will run anywhere Ruby will run. IronRuby reached the Rails singularity recently. Merb Same situation as Rails. No word yet on IronRuby capability. God Does not run on Windows period. Nor is Windows support planned. However, the good samaritan who wrote God happens to work for Powerset. Maybe they could beg him to make a Windows version. Mongrel Fully supported on Windows. Mootools Javascript- their battle is in the browser not the OS. Erlang Good Windows support! YAWS Erlang based, also works on Windows. Memcached There is an unofficial port for Windows that should not be used in production environments. To sum it up, anything requiring Cgywin is not going to be capable of operating in a high traffic production environment like Live.com. The official Ruby interpreter on Windows is too slow and IronRuby is too new and not sufficiently field tested to be placed behind Microsoft’s search in any capacity. Using God is not an option, so they are going to have to find some other way to → Read More
I spoke with Powerset cofounder/CEO Barney Pell and Microsoft’s Live Search General Program Manager Ramez Naam shortly after Microsoft’s announcement of their acquisition of Powerset earlier today. Microsoft intends to use Powerset’s natural language search technology as a major differentiating factor v. no. 1 search player Google (see our recent coverage of Live Search Cashback, a another Microsoft search effort aimed at getting more market share). TechCrunchIT goes into detail on how effective Powerset may be as a weapon. But a few things are clear – the resource limitations (cash and computing resources) that slowed Powerset’s development are now history. The relevance problem is less important since Microsoft core search relevance is quite good. And users really seem to like the beta launch of Powerset even with the limited dataset. Naam says 5% of searches contain elements of natural language that keyword based search algorithms don’t handle well, and there’s an assumption that as better results are returned, more people may start to simply type a normal sentence instead of a couple of keywords. Microsoft will integrate at least parts of Powerset technology into Microsoft Live Search by the end of the year, Naam says. I expect we’ll be hearing a lot more about natural language search coming out of Microsoft shortly. The full interview transcript is below, and you can listen to the MP3 over at TalkCrunch. → Read More
Techcrunch confirms that Microsoft has announced that they have acquired natural language search engine Powerset for a sum rumored to be around $100M. The acquisition is another key step in Microsofts efforts to gain market share in the online search market – currently dominated by Google and Yahoo. The search team at Microsoft said today in a blog post that they were interested in Powerset because of the strong team of engineers at the company, a team lead by CTO and co-founder Barney Pell that will remain intact in San Francisco post-acquisition. Microsoft also say that Powerset has solved two key search problems: The difference in phrasing and context between search queries and the context on websites. For example – related words such as ‘tree’ and ‘shrub’ or in context such as ‘what is’, ‘who is’ etc. Powerset, they claim, also offers more relevant and clearer search results. Users might find what they believe is a relevant result in a results listing on standard search, only to click through and find that the page is not relevant to their query. Current search engines work based on keyword search – they find pages based on the density of keywords that the user has searched for and then rank the results based on a number of factors ranging from the credibility of the page (based on inbound links) through to simple keyword density. Powerset uses its technology to actually analyze natural language elements on each webpage. A query such as “who wrote catcher in the rye” using normal keyword search will attempt to find those very words, while with Powerset and natural language search it will analyze what the user is actually looking for, even though there may not be a website that directly spells out “who wrote catcher in the rye”. The technology is all based on analyzing the users intention when searching based on the words they use. These advantages come at a cost – as natural language search requires more computational power to index web pages than normal keyword search. For this reason Powerset has been restricted to indexing only Wikipedia. The resources that Microsoft has in terms of server infrastructure provide Powerset a shortcut in achieving their goal of indexing the entire web – a task that would likely have cost tens or hundrends of millions of dollars in infrastructure investment had they attempted it alone. The rumors → Read More
Microsoft will announce today that they have acquired San Francisco based semantic search engine Powerset. The acquisition price is not being disclosed, but our understanding from sources close to the deal is that the previously rumored $100 million is “roughly accurate.” In May we reported that Powerset was in acquisition discussions with Microsoft and was hoping to bring another bidder to the table. Google was the likely candidate, but they have publicly dismissed the notion of contextual search as a revolutionary step forward. Microsoft, which is clearly interested in improving its search market share, turned out to be the best fit. Rumors resurfaced last week about the imminent deal. Powerset recently launched a showcase for its semantic search product, although they lacked the funds to do a full web index to prove out the product. As part of Microsoft, they won’t have that problem any longer. Now they just have to fight the bureaucracy to make sure the project continues to move forward. The company had raised $12.5 million in venture financing, plus another $8 million or so in convertible debt as bridge financing. That means investors will get a decent return (but not a home run), and the founders and employees will also take some real money off the table. We first covered Powerset in October 2006, and they were a TechCrunch40 company. Update: Microsoft announcement is here, Powerset is here. CrunchBase Information Powerset Microsoft Information provided by CrunchBase → Read More
VentureBeat is reporting that Microsoft has agreed to buy semantic search engine Powerset for somewhere around $100 million, which is the price we previously reported was being offered to the company. Our sources have been saying this deal is highly likely since May, but hasn’t actually been signed yet and could still be disrupted by the ongoing Microsoft-Yahoo negotiations. Dave Wehner, a Managing Director at investment bank Allen & Co. (he’s the guy who sold Bebo for $850 million to AOL), is representing Powerset in the deal. Powerset debuted at TechCrunch40 last fall and opened a showcase of its technology to the public just last month. Powerset has raised around $12.5 million in venture capital, and is rumored to have taken another $8 million or so in convertible debt as bridge financing. CrunchBase Information Powerset Microsoft Information provided by CrunchBase → Read More
Powerset, the natural language search engine that partially launched in May, has released a mobile version of their site that allows users to quickly search Wikipedia from their iPhone. Since the release of the iPhone a number of sites including iPodia and Wapedia have released optimized versions of Wikipedia (though none actually made by the online encyclopedia). These sites reformat Wikipedia articles to better fit the iPhones screen while shrinking (or removing) images to conserve bandwidth. What sets Powerset apart, and may make it the premier way to look up information from the iPhone, is the search engine’s ability to find both the relevant article and the exact passage that pertains to the search query. Even through the iPhone sports a relatively large screen, browsing through large amounts of text can still be a pain, which makes this feature even more valuable. Powerset has lofty goals, aiming to use their natural search technology to overtake traditional search giants like Google. So far the company is only using Wikipedia for search results, so it’s hard to tell how well the technology will work once Powerset finally indexes the web, but for the time being it may well be the best reference tool on the iPhone. You can watch a brief demo in the video below: Powerset iPhone Web App Demo from officialpowerset on Vimeo. CrunchBase Information Powerset Information provided by CrunchBase → Read More
2008 is the year of the search engine startup. Hot on the heels of Powerset’s partial launch earlier this week, stealth search engine Blekko (no logo, no website, just this and, apparently, some technology) raised a second round of financing. The company raised $3 million in equity at a $23 million post-money valuation. All previous investors participated, and new investors Marc Andreessen, SoftTech VC and Western Technology Investment also invested. They simultaneously closed a $1 million lease line with Western Technology Investment for server leases. We don’t know much yet about Blekko, which was founded by former Topix founder/CEO Rich Skrenta. The company says they won’t be launching anything to the public until 2009. See our original post on Blekko for more background information. See our coverage of Cuill as well, another hot stealth search startup we’re tracking. CrunchBase Information Blekko Rich Skrenta Information provided by CrunchBase → Read More
Today marks another milestone for San Francisco based contextual search engine Powerset. They’ve launched a showcase for their user search experience – effectively the search engine minus the web crawl. For now, Powerset queries only Wikipedia and augments results with data from Freebase. The product launch comes just a day after reports that the company is being shopped to potential buyers by investment bank Allen & Co. I have been able to test Powerset via their labs site for the last few weeks. I wrote about it last month, and the version that just launched is very similar. There is no way to look at Powerset today and determine if it can be as disruptive to search as Google was when it launched almost a decade ago. That’s because it only queries Wikipedia, and so there is little need for proper ranking algorithms to sort the good from the bad results. But what user can see is how effective a way it is to gather information quickly. For someone doing research, Powerset effectively removes a number of steps towards getting to the final information. It is particularly effective when the information needed is on many different web pages. For example, a query on Powerset of “when did earthquakes hit tokyo” yields stunning results. Try this query at Google or even wikipedia to compare – instead of just picking out keywords that are in your query and on a web page, Powerset is actually making some sense of the content included in the wikipedia pages: The way that Powerset returns queries means that answers are often found in the result snips, as above. They are also structuring a lot of the Wikipedia and (and already structured Freebase) data and inserting it into results. So a search for “Bill Clinton” shows results, but also shows Freebase structured data along with additional query refinements to get to more information. The important thing below isn’t the structured data in the results, its the fact that you can click on the action words and drill down into very specific queries (to find, for example, what bills he signed, or which Supreme Court justices he nominated, or who he slept with). Powerset is indexing web pages much differently than normal search engines, which generally just record content to match against keyword queries. Instead, Powerset is trying to understand the content on the page so that it → Read More
San Francisco based search startup Powerset will be launching shortly. For now, Powerset will query only Wikipedia and Freebase. But as I said when the product was demo’d to me a few weeks ago, it is compelling nonetheless: “When I tested the service I had something very similar to the “Aha!” feeling that ran through me the first time I ever used Google. In short, it is an evolutionary, and possibly revolutionary, step forward in search.” But now the company may have to make a hard decision: sell now to one of the big Internet players looking for a point of differentiation in search, or take the risk of going it alone and possibly getting a huge, multi-billion dollar payoff down the road. According to our sources, Powerset is exploring both options. They hired Dave Wehner, a Managing Director at investment bank Allen & Co. (he’s the guy who sold Bebo for $850 million to AOL, and is working on LinkedIn’s huge financing), to represent them in a possible sale or financing. CNET is reporting today that Microsoft may be bidding for the company. According to our sources, those discussions have been going on for well over a month, and their most recent bid is “around $100 million.” That probably won’t be enough to convince Powerset and their investors to sell. The big question is whether Google will step in to try and keep Powerset out of Microsoft’s hands, and start a real bidding war. That could drive the price significantly higher. Google, however, has publicly dismissed the notion of contextual search as a revolutionary step forward. Whether that’s true or not is yet to be seen. But Powerset may find itself as a valuable chess piece in the emerging search war between Google and Microsoft. And if Google bets wrong, they could find their commanding lead in search eroded over time. A relatively small acquisition to keep Powerset out of Microsoft’s hands, even if just a hedging move, may suddenly be attractive to them. CrunchBase Information Powerset Information provided by CrunchBase → Read More
San Francisco based Powerset will be publicly launching a long-awaited beta version of the service in the coming weeks, the company told me yesterday. They are working on a new kind of search engine that will understand natural language searches and compete with keyword matching engines that dominate search today. An early version of the search engine, which was demo’d to me yesterday at their offices, has been available to some users of their Powerlabs site. But for the most part, it’s been kept very quiet. The early version of the service will serve as a showcase for the user interface and engine itself, but it will not have a full web index behind it. For now, Powerset will query only Wikipedia and Freebase. But when I tested the service I had something very similar to the “Aha!” feeling that ran through me the first time I ever used Google. In short, it is an evolutionary, and possibly revolutionary, step forward in search. I’ll temper that statement since the company is not putting anything more than a tiny index of two sites behind the service for now. In particular, the fact that Powerset doesn’t have to bother with spam control and other relevance issues (which is what made Google so great when it launched), means it can’t yet be considered any kind of challenger in the search space. But anyone who uses it will be able to see the potential value of the engine when it is placed in front of a full web index. For now the company is keeping specific features of the engine confidential, but I can say it has evolved significantly since a screen shot was released in mid-2007. In preparation for the launch, some of the Powerset team have vowed not to shave until the product is released. They are chronicling their facial hair adventure on a site called Powerstache, which has been covered by Jessica Guynn at the LA Times. Rumors have also been swirling around the company in general. A number of sources have said that Powerset is pitching for additional capital. And the company also appears to have put plans to hire a new CEO on hold – founder Barney Pell is still firmly in charge at the company. Powerset is one of three new search engines that we’re keeping a close eye on. The other two, Cuill (pronounced “cool”) and Blekko, → Read More
Ok, so that isn’t an actual picture of the new Microsoft Blews news aggregator that was announced by Microsoft Research today, but tell me that the screen shot (see below) doesn’t bring back memories of eating Rocket Pops on the beach as a child (or wherever you ate them). But back to Blews. It’s a news aggregator (see Techmeme and about 45 others, including this gem), but it goes beyond mere clustering of stories to show what’s important right now based on who’s linking to what in near real time. Blews, which is only looking at political news, also tells you the bias of the links in to a story: BLEWS uses political blogs to categorize news stories according to their reception in the conservative and liberal blogospheres. It visualizes information about which stories are linked to from conservative and liberal blogs, and it indicates the level of emotional charge in the discussion of the news story or topic at hand in both political camps. BLEWS also offers a “see the view from the other side” functionality, enabling a reader to compare different views on the same story from different sides of the political spectrum. BLEWS achieves this goal by digesting and analyzing a real-time feed of political-blog posts provided by the Live Labs Social Media platform, adding both link analysis and text analysis of the blog posts. Here’s what all that looks like: Liberal links are blue (rasberry) and on the left, conservative links are red (cherry) and on the right. The middle is the story itself in white (lemon). The dots around the edges suggest the emotional charge of the commentary, which can drip off of the Rocket Pop in very hot weather. I note that no one on the team (Michael Gamon, Sumit Basu, Dmitriy Blenko, Danyel Fisher, Matthew Hurst and Christian Konig) is a user interface specialist or web designer. Putting aside the UI, which is hard to do, the artificial intelligence behind Blews could be interesting. It is very hard to get a machine to decipher emotion and meaning from raw text unless they are doing mere keyword searches (see, for example, Powerset). Microsoft is calling this hard bit “detecting emotional charge.” If they’ve got it right, or are close, there are an unlimited number of potential applications for the technology. As an aside, this somewhat reminds me of ScoutLabs, a startup we wrote about → Read More
Keyword search gets you pretty far when looking for pure information, but doesn’t help much on more qualitative searches like trying to find the hippest restaurant in SOHO. Searches like the latter rely on the opinions of people, not webmasters, which is one of the reasons Circo’s has launched their new qualitative search engine. The engine currently lets users search for hotels and restaurants by qualities like size, ambiance, or other qualities pulled from reviews from around the web. They have plans to expand to other categories in the future. Circos is categorized under the ever expanding umbrella of semantic search engines, which currently includes the likes of Hakia, PowerSet, Kosmix, SemantiNet, Quintura, and TrueKnowledge. However, the engine is most like Kango, which has also taken on the task of categorizing hotels based on user reviews. VibeAgent also has a search engine for its own site that will search hotels based on qualities. While Kango auto-generates tags after pouring through user reviews, Circo lets users search for any qualities they’re interested in. The engine then grades and ranks the results by each quality on an “A” through “F” scale based on how well the description fits for reviewers. For example, a hotel reviewers feel is spacious would rate highly if searching for openness, but poorly if you’re looking for a tiny room. As with most search engines, Circos’ real test will be whether its application draws users away from other hotel and restaurant sites with less sophisticated search engines. Currently there are a bunch competing in the space. However, Circos says their technology can easily be extended to other categories since their algorithm does all the tough work of pulling the most relevant qualities from reviews. If hotels and restaurants don’t appeal, another category may hold their home run. Circos is angel funded, based in San Mateo, and has eight employees (4 in Singapore). http://www.theworldisbeautiful.com/circos/circosintro2.swf → Read More
Rich Skrenta, who created the first computer virus (Elk Cloner), co-founded the Open Directory Project, and co-founded online news site Topix, may have bitten off the biggest challenge of his career – taking on Google. In search. Skrenta left Topix last June. He started his new company, Blekko, almost immediately, along with five others from the Topix core team. They raised $2 million in seed funding in September from Baseline Ventures, two early Googlers (David DesJardins and Jeremy Wenokur), and the founding team. The company is still deep in stealth and, apparently, working out of a garage in true startup style (see image below). The Blekko website, which today has nothing on it except a picture of a puppet created by Skrenta’s daughter, isn’t even close to having a landing page up, let alone the final product. But eventually Skrenta says they’ll launch a full scale search engine to compete with the big guys. Skrenta, who’s very media savvy, won’t say much about how he’s going to tackle search (he’s not a fan of PageRank though:“PageRank wrecked the web. Google is the cause of all of this. and Google is going down with it.”). He says they are looking at improvements on the back end (indexing and query serving) as well as the user search experience itself. Beyond that, he says we have to wait. And it might be a long wait at that. The company, Skrenta says, may not have a public prototype available until 2009. Normally an entrepreneur announcing they’re taking on Google with a six person team and just $2 million in funding would either be laughed at or ignored. In Skrenta’s case, he has proven himself more than once as capable of taking on big challenges and winning. This will be a company to watch, and speculate on, in 2008. There are other promising search startups out there. Powerset, Cuill (we’ll be hearing more about them soon) and the upcoming Wikia Search Engine are all yet to launch. Mahalo is growing fast (but still tiny). Can anyone unseat Google? Perhaps not any time soon. But you don’t have to get much market share to be a huge winner in this space – every 1%, they say, is worth a cool billion dollars. CrunchBase Information Blekko Rich Skrenta Information provided by CrunchBase → Read More
Don’t expect to see natural-language search at Google anytime soon. Despite the buzz of startups like Powerset and, to a lesser degree, true knowledge, Google’s head of research Peter Norvig pooh-poohs the notion that people are clamoring to write full sentences in search boxes. In a Q&A with Technology Review, he says: We don’t think it’s a big advance to be able to type something as a question as opposed to keywords. Typing “What is the capital of France?” won’t get you better results than typing “capital of France.” True, true. But he does acknowledge that there is some value in the technology: We think what’s important about natural language is the mapping of words onto the concepts that users are looking for. . . . To give some examples, “New York” is different from “York,” but “Vegas” is the same as “Las Vegas,” and “Jersey” may or may not be the same as “New Jersey.” That’s a natural-language aspect that we’re focusing on. Most of what we do is at the word and phrase level; we’re not concentrating on the sentence. We think it’s important to get the right results rather than change the interface. In other words, a natural-language approach is useful on the back-end to create better results, but it does not present a better user experience. Most people are too lazy to type in more than one or two words into a search box anyway. The folks at both Google and Yahoo know that is true for the majority of searchers. The natural-language search startups are going to find out about that the hard way. If Google doesn’t trounce them first. CrunchBase Information Powerset true knowledge Information provided by CrunchBase → Read More
Natural-language search startup Powerset is going through some growing pains. Barney Pell is stepping down from the CEO spot. He will now become the CTO, and he and Powerset’s board will conduct a search for a new CEO. Powerset’s other founder and COO, Steve Newcomb, is not in the running for the top job. He has left the company. At the Web 2.0 conference, Pell gave an impressive demonstration of Powerset’s search technology, although it was restricted to a limited data set. How the search engine will do against the entire Web, which is a much bigger technical challenge, has yet to be seen. But this shakeup does raise a big question. Why step down as CEO and leave a huge leadership gap (with no COO either) before you find a new CEO to take things over? Perhaps this was done more for internal reasons. Announcing everything all at once sends a signal to employees about the direction of the company, and minimizes future surprises. The CEO search also indicates that Powerset may finally be ready to open up its search engine to the general public sometime next year. Or perhaps Powerset’s board has become impatient with the company’s progress and wants new leadership. You can read Pell’s explanation about the transition here. (You can read our previous coverage here). CrunchBase Information Powerset Information provided by CrunchBase → Read More
A reader noticed that stealth search engine Powerset is using Amazon’s Mechanical Turk service to gauge user reactions to search results. See the screen shot (click for larger view) – users are shown a query and a number of results and are asked to evaluate the relevancy of each result from five choices. In this case, the query is “revealing bikinis.” Users are asked to evaluate four sets of results within ten minutes, and are paid $0.02 for the effort. The current batch of Powerset projects have run their course, and there are currently no other projects available on Mechanical Turk. I spoke with Powerset CEO Barney Pell this evening who confirmed that they are using Mechanical Turk to get human feedback on search results. He says the results are not all Powerset generated – rather, they show results from Powerset, Google and others to see which users prefer for a given query. He also says this is an ongoing project, and new ones will be added soon. Pell also said that Powerset plans to use Mechanical Turk over the long haul, even after launch. They’ll put actual user queries into Mechanical Turk in real time, add Powerset and competitor results and see which results people find more relevant. If results suggest Powerset isn’t more relevant, they’ll adjust their engine. Powerset also uses the EC2 computing service, another web service offered by Amazon. They recently released some of their internal growth models that allow people to compare the relative costs of EC2 to building out a real data center. → Read More
