Search engine Blekko has raised $30 million in new funding Yandex, MLC Private Equity, and existing investors U.S. Venture Partners, CMEA Capital and PivotNorth Capital (Ashton Kutcher is also a previous investor). This brings Blekko’s total funding to $55 million. Yandex’s CEO Arkady Volozh will join Blekko’s board.

Blekko launched last year as the shiny new search engine that wanted to take on Google. The search engine differentiates itself from Google by giving users tools to do new types of searches that they can’t do elsewhere. Blekko offers unique query refinement tools to human editors called Slashtags (i.e. /news or /date or /amazon or /blogs) to filter results to what you are looking for (you can read our full review of the platform here).

Earlier this year the search company expanded its efforts to eliminate spam from search results and banned pages from well-known content farms as well as blocking pages from 1.1 million domains that were generating spam. The company has also signed partnerships with Topix, fellow search engine DuckDuckGo and recipe search engine Foodily.

Blekko has also integrated social data into results, as well as Facebook comments. The company also considering integrating Twitter data into results as well.

Volozh says of Blekko: “We love blekko and think it’s a great product – a quality search engine that organically combines search algorithms with expert opinions. We believe the outlook for this approach is strong and that the blekko team is poised to make it a success.”

“Yandex is a partner and investor that shares our mission of making search the best experience it can be,” said Rich Skrenta, CEO of blekko. “Having access to one of the world’s top pools of search talent and the fantastic products they have built will help us grow blekko in the U.S.”

The search engine currently indexes approximately 3.5 billion URLs, which pales in comparison to Bing and Google, which are both over 15 billion. But one of the key advantages to using Blekko is that it is such a thorough spam blocker.

After announcing its Zorro visual revamp and its Three Search Engine Monte challenge, Blekko is announcing today that it’s partnered with recipe search engine Foodily in order to curate its recipe search results.

As of today. users who search on Blekko for things like Kale Chips or Grilled Fennel will tap into the Foodily community’s curation for foodie-related slashtags like /recipe /nocarbs or /glutenfree, providing people with a curated result as well as insight into what recipes their friends like using Blekko’s slashtag technology and Facebook integration.

“Foodily’s mission is to help everyone find the food they want and love, and share it with their social circle. We’re excited to play a role in helping Blekko searchers find great resources on food, cooking and recipes,” said Foodily CEO Andrea Cutright, in a release.

Funded to the tune of $24 million by Ron Conway, Mike Maples, Marc Andreesen and others, Blekko currently has over a 100K slashtags, aiming to provide users with a more relevant search experience. Narrowly targeting vertical search by partnering up with interest-based communities is an interesting strategy, as indeed there are sometimes riches in niches.

Search engine Blekko is pretty excited about their new Zorro launch that I just wrote about. All new design, a move from red to blue links and a general declutterfication is just the shiny stuff on top. Underneath there’s a new web index and they’re really putting those slashtags to work – about 1,000 of them are now auto-added to appropriate results pages.

But enough tech jargon. The proof is in the pudding, they say, and Blekko just served some pudding. Try out their new tool called 3 Engine Monte (derived from the popular 3 Card Monte confidence game.

You can get to it from the link above, or via the Blekko home page. Or just add /monte to the end of any query. You’ll see three sets of results in three columns, all formatted identically. Your job is to pick the one you think is Blekko and click on it. You’ll then see if you’re right or wrong.

I’ve played it a few times and can beat it just about every time. The key is to look for certain signals, like the inclusion of content farm stuff. It still appears high in Google and Bing results, but you won’t see it at all in Blekko. Give it a try, and good luck.

Ok, I got a little over excited in the title. But the new version of search engine Blekko, called Zorro and launching right now, is pretty cool. There have been big improvements visually. Gone are the red links of previous the version along with most of the left sidebar clutter. In fact, that left sidebar is gone and has been replaced with small icons next to search results to tell you what site they’re from at a glance.

The company has also increased search relevance substantially by auto-including some 1,000 slash tags, up from just a handful previously. That means that for many results you are looking at hand picked sites that are known to have high quality content. Content farms just can’t get through slashtags.

Search for “pregnancy tips” and you’ll see to slash tags, for /pregnancy and /health, and quite good results compared to Google. But on Blekko you’re not done. Click on one of those slash tags to drill down into results relevant to that tag. Answer relevance goes even higher. On Google you’d have to visit the next page of results, or rephrase your query. Both are time consuming.

What’s most amazing about Blekko, though, is it’s still around. So many ambitious search startups like Cuil and SearchMe went for it and flailed. But Blekko, six months in, is growing nicely and has around 1 m unique IPs visit the site monthly. Early contextual ad testing, they say, is producing good results. When they finally turn that on for good, a real revenue stream will be flowing to Blekko. In other words, it may be time to start thinking of Blekko as a long term “we’re still here” startup. They have no appearance of fail anywhere near them yet.

One more thing Blekko is launching today is a fun tool called 3 Engine Monte. More on that in my next post.

For comparison, competitors Google and Yahoo are currently at 18 months of user data retention and Bing is at six months, which is the European standard. In fact, Yahoo recently extended its data retention policy from 90 days to 18 months because it needed it to “compete” with Google in offering personalized recommendations. With this move Blekko is essentially saying, “Unlike Yahoo, we don’t need to compete.” Search engines like DuckDuckGo and Startpage do not collect any user info.

Granted, with $24 million from US Ventures, CMEA Capital and Marc Andreessen, Blekko can keep pulling stunts until the cows come home. In addition to this move Blekko will now be introducing its HTTPS Preferred offering, which will automatically point searchers to HTTPs secure websites when available.

Blekko has also amped up its ad opt up services with Super Privacy and No Ads privacy opt out settings which allow users to opt out of advertising while searching. ”Search engines know too much about their users. Our goal at Blekko is to find a balance between retaining information to improve our search engine, and not retaining information that a user prefers to keep private,” said CTO Greg Lindhal.

Blekko is basically pulling out all the bells and whistles, performing parlor tricks like adding Facebook comments to search because it has to, as the search market is currently comprised of Google at 65.7%, Yahoo at  15.9% and Microsoft at  14.1% with Blekko not even ranking in the top five.

Right now the company, which completed 50 million searches last month from 750K uniques (up 33% since March), indexes 3.5 billion URLs, compared to Google and Bing which are both over 15 billion.

And a search engine’s gotta do what a search engine’s gotta do. But is garnering press attention with its weekly reactive maneuvers necessarily worth the effort? We’ll soon find out.

Topix has become a respectable web property, which is why today’s announcement that it will be partnering with young search engine, Blekko, seems like an interesting move. Blekko only launched publicly back in November of last year, so the human-curated, slash-tagging search engine is still very much an unestablished entity.

I think many would agree that search is need of some de-spamming and a fresh take, and Blekko is certainly that, but the road before them is uphill to say the least. Blekko has made a laudable effort to crackdown on spam, eliminating 1 million spammy links from its search results in March.

What’s more, Blekko does not accept paid traffic, it uses humans on top of algorithmic search functions, and it’s getting more social — all great things. I applaud how ferociously Blekko is attacking the current problems in search, hey, it attracted 575K unique IPs in March, so people are checking it out.

I just worry that the search engine’s crusade against spam may result in the elimination of legitimate sites as well, and it can be slow to crawl smaller sites, and update name changes, and so on. Right now it indexes 3.5 billion URLs, which is just no match for Bing and Google, which are both way over 15 billion.

That being said, I am very intrigued to see how Blekko integrates with Topix, which averages over 1 million searches a month. With a more focused set of data to index, I think Blekko could start turning out some great search results. From testing it out a little bit today, it’s worked great, but I’ll have to dive deeper to know for sure.

But obviously what really keeps this from being a shocking development, or a surprising risk for a sizable platform, will not come as a huge shock to those familiar with the two companies’ histories. Blekko CEO Rich Skrenta and Blekko Marketing VP Mike Markson are both co-founders of Topix.

So, while Topix was presumably predestined to add Blekko search at some point, no surprise there, it is interesting to see that the Blekko team now finally feels that its search engine has been tuned (and tested) enough to make this implementation launchable.

“It is rather unprecedented for a search start up to be able to power search on a site of scope and scale as Topix,” Skrenta said. “We have a long history of collaboration and camaraderie with the Topix team so it’s a thrill to be working together again in bringing a new search experience to Topix users.”

Is it working? Chime in with your own experiences.

Ever since its launch in November 2010, Blekko has been on a mission to eliminate spam and content farms from search results. The human-curated search engine, which is also known both for using actual mammals to edit search results and for its employ of slashtags for easy categorization, announced in March that it had banned over 1 million spammy domain names from its site. Using a new algorithm it calls “AdSpam”, Blekko investigates the quality of a doman’s content, as well as the type of ads it includes, to identify those of the lowest quality. Those that don’t pass muster get the boot — which should be music to any searcher’s ears.

Now, whether or not Blekko can compete with the Googles of the world in the long-term remains to be seen, but I hope so. You might say that Google has rested on its laurels for a bit too long, and, in the meantime, Blekko seems to have been taking the necessary steps to make search a more pleasurable and less spam-loaded experience. Search is desperately in need of a fresh and holistic approach. And, today, Blekko is further rounding-out its competitive engine by going social, announcing that it will be integrating Facebook comments into its results pages.

Considering that the Facebook News Feed has become an extremely popular source for link-sharing, updates, and social commentary — and Facebook Connect now practically blankets the Web — the social network is a logical partner for Blekko. And it gives it that much-envied social flair it had been lacking.

But, how does it work? Using Blekko’s Facebook integration is easy: You simply connect with your Facebook account on the Blekko homepage, and go about your normal searching. The major difference, though, is that when you type “TechCrunch” into the search bar, you’ll still the same search results you would otherwise; yet, now, all mentions of TechCrunch in your Facebook news feed (and thereby mentioned in your friends’ feeds) populate the right column. Look out!

You can also use a hashtag in the Blekko search bar (“/facebook”), and the engine will serve you with results from your Facebook comments and those of your friends. There’s also a box above your Facebook comment results in the right sidebar that allows you to post directly to your wall.

Blekko search was already pretty fantastic, if just for low spam counts, so why the integration? “The Web is increasingly a social experience and search has got to get more social too”, said Blekko Mastermind and CEO Rich Skrenta. “This brings the social graph and social commentary right to the results page. Because what your friend says about information is as important as any expert’s advice could ever be.”

I haven’t yet been totally convinced by the social recommendation evangelists that I’m better off hearing suggestions from my friends than a group of experts, but the social wave has crested, and it’s impossible to avoid. Get on the bandwagon, or get out of the way.

After all, my friends do know what kind of pizza I like, so if I happen to be searching for pizza places on Blekko, this will likely enhance Blekko’s search results. And, on the flip side, though my gut reaction would be to think that search results for something more obscure like, say, “translational research in neuroscience” might not exactly be augmented by the Facebook peanut gallery. But maybe one of my friends happens to be studying that very thing in med school.

Regardless, this is a smart (and logical) move for Blekko, especially considering that it began leveraging Facebook Likes in its search parameters earlier this year. Users who login to Blekko with Facebook can see whether or not their Facebook friends “like” particular search results. Users can also refine their results by opting to search only those sites that have been “liked” by their friends. At first glance, this idea seems fantastic, but it’s really only useful if your friends (in the former example) are actually using Blekko and (in the latter) if they are frequent users of the like button.

The search engine currently indexes approximately 3.5 billion URLs, which pales in comparison to Bing and Google, which are both over 15 billion. Obviously, this is a bit of a problem, because, on top of this, Blekko is such a thorough spam blocker, it can also leave out sites that just seem spammy but aren’t.

On the bright side, Skrenta says that Blekko has been growing traffic every month since launch in November. The site saw roughly 575K unique IPs last month. And that is all organic usage adoption, he says, Blekko does not entertain paid traffic, SEO, facebook viral loops, and so on.

In the end, I applaud Blekko’s efforts to become more social, and look forward to (what I hope) is its inevitable integration with Twitter. Right now you can use the Twitter slashtag to search the site using Twitter’s API, but they haven’t made any Twitter social search integrations, though Skrenta tells me he’s considering it.

Combining actual human editors with intelligent anti-spamming algorithms is a dynamic combo for search. So, here’s to hoping that Blekko gets the kind of user adoption it needs to reach the tipping point and give Google (and Bing) a run for their respective monies.

They’re calling the new algorithm “AdSpam.” “One of the strongest signals that a page is spam is aggressive participation in self-service
online advertising networks,” says the company. When they compare low quality sites (based on existing signals) and see lots of keyword based ads alongside that content, it’s very likely to be blocked.

“Domains with low quality content plus keyword ads are “machines that print money,” says Blekko CEO Rich Skrenta. “Machines that print money will be exploited.”

Blekko is processing around 1 million search queries per day and has 500,000 unique monthly visitors, he says.

Instead of coming up with its own version of Google Analytics or AdSense, Blekko has gone the widget route, riding the wave of publicity surrounding the JC Penney paid links controversy by emphasizing search transparency. Perhaps the most useful tool in the suite, Blekko is offering publishers a Link Roll Widget, an easily embeddable way to “show off” the organic links to their sites (customizable by number and size) much like they show off retweets or Facebook “Likes,” among other things.

Along with a free t-shirt and Blekko trading cards (yeah) for anyone who asks, here’s a sampling of what you can find on Blekkogear.

Blekko Link Roll Widget

The Blekko Link Roll Widget allows you to customize a widget showing recent inbound links to your site as they happen in “realtime,” as organic linking is a the web’s foremost method of validation.

Embeddable Search Box

Blekko now offers publishers the ability to embed a custom Blekko search box on different verticals, using the Blekko curated slashtags like /techblogs or /food or even something more personalized like my slashtag that only searches TechCrunch. This might be useful for sites that don’t have their own powerful search functionalities.

Blekko Toolbar
In the same vein as the Google Toolbar, the Blekko Toolbar allows you to search from Blekko, add your Blekko slashtags from your browser, view a site’s SEO statistics (including visible stats for Inlinks and Site Rank) as well as mark sites as spam while you browse.

Lack of Chrome support is heartbreaking, though not surprising.

Blekko RSS Feeds

Blekkogear now provides you with a one stop shop to create RSS feeds for slashtag searches, like this one for techblogs.

Blekko Search Engine Default

Allows you to easily make Blekko your default search engine, another one that’s a no-go for Chrome and inexplicably Safari.

Site Badge

Blekko is also offering a badge with it own Site Rank score and Inbound links numbers, for websites who want to take pride in where they stand on Blekko and look super nerdy in the process.

While I’m suspect of the actual utility of some of these (specifically the Site Badge) Blekkogear is an interesting move by the search underdog as a way to get more distribution and eyeballs on its unique approach to search. And with its $24 million in funding from Conway, Andreessen and others, I wouldn’t be surprised if Blekko, who most recently paired up with hacker darling StackOverflow, pulls another similar stunt next week.

Editor’s note: This is a guest post submitted by Mahendra Palsule, who has worked as an Editor at Techmeme since 2009. Apart from curating tech news, he likes analyzing trends in startups and the social web. He is based in Pune, India, and you can follow him on Twitter.

What’s the Next Big Thing after social networking?

This has been a favorite topic of much speculation among tech enthusiasts for many years. I think we are already witnessing a paradigm shift – a move away from simple social sharing towards personalized, relevant content.

The key element of the next big thing is the increasing significance of the Interest Graph to complement the Social Graph. While Facebook, Twitter, and Google are already working on delivering relevant content, a slew of startups are focusing exclusively on it.

Relevance is the only solution to the problem of information overload.

The above matrix is a representation of how the process of online information discovery has evolved over time.

Phase I: The Search Dominated Web

This is how Google began its dominance over the web two decades ago, using PageRank to surface the most popular web pages as identified by other web pages that linked to them.

Phase II: Web 2.0 With Social Bookmarking

In the Web 2.0 era, social bookmarking services gained significant traction, surfacing popular content. Sites like Reddit and StumbleUpon are hugely popular even today, driving millions of page views.

Phase III: Personalized Recommendations

Services like Hunch, GetGlue, etc. have focused on building an Interest Graph for users, to deliver personalized recommendations using a ‘taste engine’.

Phase IV: Personalized Serendipity

The latest crop of startups is focusing on personalization using a combination of Interest and Social Graphs. Personalized Serendipity is what Jeff Jarvis calls ‘Unexpected Relevance’. Examples include Gravity, my6sense, Genieo, and TrapIt.

What Exactly Is Relevance?

The battle against information overload is sometimes presented as a choice between Relevance and Popularity, where ‘relevant’ is equated to ‘personalized’ as against popular.

However, Relevance does not always mean Personalized. Relevance is very dynamic – it depends on the needs of a person at a specific point in time. There are times when users want to know about the most popular stories, and other times when they seek personalized content.

There are multiple approaches to filtering information for Relevant Content. Google, Paper.li, and PostRank are examples of algorithmic filtering, while Reddit, Hacker News use a crowdsourcing approach. Klout can be used to filter Twitter streams by influence, while Facebook uses social affinity as a filter for its newsfeed and social signals for its new Comments Plugin. Location is another high-impact signal for delivering relevant content, gaining importance in a mobile world.

In other words, Relevance spans across all the quadrants of the Discovery Matrix above, and none of the above approaches to filtering for relevance is the ‘best approach’. There is no killer approach to Relevance. Henry Nothhaft, Jr., CMO of TrapIt, described it as “the myth of the sweet spot”. The competitive edge will be with services that support multiple discovery methods, multiple filtering approaches, have flexibility, and support multiple mobile platforms.

Quora: A Showcase Of The Interest Graph

Quora has pioneered the use of the Interest Graph as a dominant signal for its newsfeed. Quora asks new users to select Topics to follow, as part of its onboarding process, which is the first revelation that Topics are as important as Users to follow.

Quora’s newsfeed is an interesting showcase of what happens when you mix an Interest Graph with a Social Graph – and the result is the mysterious addictiveness so many have experienced, but found difficult to explain. An item pops up in your newsfeed not because you were following a user, but because you were following a related topic.

This often leads to Personalized Serendipity – or Unexpected Relevance – which is why Quora gets many people hooked.

The war over the Interest Graph began between Twitter and Facebook last year, as Erick described so eloquently. So how did Quora beat them to this game?

For starters, Quora is built from the ground-up with the Interest Graph being a backbone of the framework. Twitter’s ‘Browse Interests’ is too broad and primitive to be of use, even at present. And while Facebook has a mechanism for allowing publishers to push new items to your feed, most publishers have been unaware of this functionality.

This is also the reason why Facebook’s Like Button now publishes a full news feed story. The future clearly belongs to who best captures the Interest Graph as Max Levchin and Bill Gurley put it.

The implications of a Relevance-driven web are wide-ranging and broad in scope. Better utilization of the Interest Graph by services will lead to better ad targeting, and a potential decrease in reliance on CPM/CPC-based advertising. Monetization focus will be on higher yields through transactions and subscriptions as Dave McClure once described. Online media publishers will focus on Relevance Metrics revealing engagement and time-spent on site, than primitive metrics like page views and traffic.

Social media may lose its obsession with follower numbers and traffic, evolving to context-driven reputation systems and algorithms.

Interest Graphs will be used to build Better Social Graphs. Today’s monolithic Interest Graph will get further specialized into Taste Graphs, Financial Graphs, Local Network Graphs, etc., yielding higher relevance for different needs.

The Age of Relevance beckons!

Sixteen years has passed. I receive about 400 e-mails a day now from people all over the world. E-mail has become part of my life and has changed the way I communicate and the way I work. I don’t know anyone anywhere who doesn’t have an e-mail address. When I went to Sikkim, India, last year, a Buddhist monk in a remote Himalayan monastery even gave me his e-mail address. The web has also evolved in a similar fashion—it seems to be everywhere and connects everyone, for everything. Internet technologies are now toppling dictatorships in the Middle East.

But what has really changed in search? We still go to the same text boxes, enter expressions that we hope the computer will understand, get back lists of web pages that reference those words, and click on links to find the information we are looking for. The only real difference is that now the top links take you to spam sites—which want you to click on other links that make them money and that make Google money. Creating low-quality, low-cost information pages has become such big business that the leading content farm, Demand Media, just went public and is valued at $1.9 billion. According to Blekko’s spam clock, over 1 million spam pages are created every hour. So the web is becoming one giant heap of trash.

I had hoped to put Google and Bing on the spot at the recent BigThink conference by challenging them to fix the spam problem; perhaps to follow Blekko’s bold lead in blacklisting the leading polluters. But they instead got into a pissing match about who was copying whom.  Google simply changed the subject. And when I asked the panelists about their long-term vision on search, I was really disappointed at the shallowness of the response. They weren’t talking about changing the world—just about fine-tuning what they’ve been doing forever. You can watch the video below to see what I mean.

In the hope that I don’t have to wait another 16 years to see advancements in search, I’ll share my views on where it needs to go. Perhaps you can also share your views and we can inspire a new generation of startups to do to the current search leaders what Google did to Lycos and AltaVista: antiquate them.

In a nutshell, what I want is for my computer—or a new Google—to serve me. I don’t want to be serving it words that make sense to it, and then have to wade through pages of spam it delivers to me, to find the information I am seeking. I want it to learn what I like and what my friends like and tell me what I want to know or do what I need it to.

If I am visiting New York City tomorrow and want to eat dinner at a moderately priced North Indian restaurant near where I am staying, I want my computer to suggest the two or three places that I will like and that have space available. I book my flights and hotel reservations on line, have my calendar on Google, tweet my likes and dislikes, and talk to my friends on different social media sites, after all. So why can’t my new Google simply take my information and my friends’ information and give me what I want?

Is this so hard? I don’t think so. This week, we witnessed a computer, Watson, beating the top Jeopardy contestants. It was able to parse human speech patterns, make sense of complex questions, do very sophisticated searches, and come back very quickly with the right answer. It didn’t respond with a series of links—it computed the probability that its answer was correct and responded accordingly.

Watson’s technology is a great start, but I want much more. I also want it to analyze my social graph and get recommendations from friends who matter. For example, when it comes to Indian food, I don’t care what my academic peers say or what my South Indian friends say; I want input from fellow Punjabis—they know their tikka masalas and saag paneers better than anyone else does. If I am looking for health-related advice, I want to know what doctors say. If I am shopping for a gift for my wife, I want input from women who share her tastes. This isn’t rocket science.

And then I want more. I want my new Google to automatically make a dinner reservation for me, buy me a ticket to a movie that I may want to watch, or place an order on the cheapest and most reliable shopping site. Yes, I know there are already many applications/sites that do this. Why aren’t my preferred sites integrated into the search function—so I never have to see the 90s-era text links?

What I really don’t ever want to see is the spam that Google delivers. The present page-ranking system is easy to bait—just add the right key words to some garbage content or pay Google for an ad, and your listing appears at the top of everyone’s search results.

Google took some good steps forward this week with its announcement that it will let users tag sites as spam via a Chrome extension and rank the websites of people they know higher than others. But this is still more of the same—the spammers still get top billing. Just do a search on a term like “digital camera under $200” to see what I mean. The results are practically useless. (Bing just added a nice option to let you search by price, but that is buried in its spammy results when you do a regular search).

We need some out-of-the-box thinking here. I doubt we will see this from Google, because it makes billions by serving up ads.  So here is an excellent opportunity for entrepreneurs.

Editor’s note: Vivek Wadhwa is an entrepreneur turned academic. He is a Visiting Scholar at UC-Berkeley, Senior Research Associate at Harvard Law School, Director of Research at the Center for Entrepreneurship and Research Commercialization at Duke University, and Distinguished Visiting Scholar at The Halle Institute for Global Learning at Emory University. You can follow him on Twitter at @wadhwa and find his research at www.wadhwa.com.

A decade ago I tried Google for the first time. Like everyone said, it was magic – the result I wanted was right there at the top. For someone who’d been using AltaVista for years before that it was a very pleasant experience. Anyone who was on the Internet before Google came along knows exactly what I’m talking about. Google just felt right. It got the job done.

It’s been a creeping feeling, growing over the years, but it sort of feels like pre-Google again. Search is a really bad overall experience. Travel searches, for example, are a joke, and startups like Gogobot are popping up to try to fix that. When I’m trying to figure out the best hotel for me when I travel I bail on Google entirely and head to Tripadvisor (shudder), and Gogobot.

Same for gadget product reviews. GDGT, Amazon and occasionally Consumer Reports seem to have the best collections of data, so I just go there directly and bypass Google. In fact, I use Google mostly for navigation, not discovery these days. Meaning I know the document I’m trying to find and figure out the best search query to locate it. But pure discovery? It’s a shit show of layer upon layer of SEO madness vying for my click.

Is there actual evidence of Google failing at search? Probably somewhere, but certainly not in the search share numbers. They maintain a healthy, almost monopolistic, lead in search despite huge efforts by Microsoft to compete. But then again, AltaVista had huge search share too, right before they suddenly didn’t any more.

And while I watch search startups like Blekko make serious attempts to fix search by thinking about the problem a little differently, it’s just too early to know if they’ll succeed.

So what is the evidence that search still sucks? Well, you know it’s true, just like me. And the fact that the mighty Google is suddenly taking every opportunity to toot their own search horn shows they know it, too. They tore into Microsoft for stealing data with just a little too much vehemence. In the end it felt like less of a gotcha moment, and more like entrapment.

And then today, with this JC Penney nonsense. For months the company gamed Google to get the top result in dozens of lucrative product searches. Google recently discovered it and shut it down. And then, as best I can tell, fed the story to the NY TImes as a sort of victory lap.

I say it should be an embarrassing moment for Google, not one to celebrate. In fact I did say it, here. Google’s Matt Cutts responded by lightly trashing Bing: “@arrington the newer/most recent spate of links happened in the last 3-4 months; not over a year. JCP still ranking on [dresses] on eg Bing.”

Which is fine. It’s always fun to slap Bing around a little, I guess.

Vanessa Fox, who used to fight spam at Google, weighed in as well, saying “@arrington – spam fighting will always be an ongoing battle at Google. Have to balance being aggressive in algorithms w/ collateral damage.” Earlier today she also reported on the JC Penney story.

When companies start to flail they nearly always do a couple of things. First, they trash the competitors. Then they talk about how hard the problem is and that the solution is a long term one.

Altavista did a lot of that in the late nineties. Right before a competitor came in and fixed the AltaVista problem permanently.

Yes, search is very hard. But Silicon Valley is really good at doing hard things. The real problem right now is that there’s a perception that Google is untouchable in search. When a venture capitalist sees a pitch from a new search startup all they can think about is the Cuil debacle. And since venture capitalists are just about the most risk averse people in Silicon Valley, the funds just don’t flow.

But all the evidence suggests otherwise. Demand Media is worth $1.6 billion, and their entire business is based on pushing cheap, useless content into Google to get a few stray links. If Google was good at search, Demand Media wouldn’t exist. And Bing wouldn’t be making solid gains in search market share. And JC Penney wouldn’t be able to massively game search results for a few months, during the holiday season, without getting caught until months later.

We need to see a real competitor emerge in search. If only because it will make Google up its game, and make all of us a lot happier.

While Microsoft VP Harry Shum and Google’s Matt Cutts squabbled on a panel together at the Farsight 2011 conference, Blekko CEO Rich Skrenta looked on, more a human symbol of the call for greater market diversity than anything else.

Earlier in the week Blekko made the decision that it would block content farms like Demand Media’s eHow and AnswerBag and is now announcing another milestone, over 30 million search queries in January and over 110K slashtags (curated topics) created since its launch in November.

This breaks down to about 10-15 search queries a second and over 1 million searches a day, which is at levels over its original traffic spike at launch. In contrast, Google was serving over 88 billion searches a month at last count (the 2010 Comscore numbers have yet to come out as far as I can see).

Even though the concept of slashtags is hard to get used to,  here’s why I think Blekko has a fighting chance:

The company, helmed by Skrenta and Greg Lindahl is funded to the nines with $24.4 million from investors like Marc Andreessen, Ron Conway, Jeff Clavier and wildcard Ashton Kutcher. Skrenta told me himself that Blekko isn’t going away any time soon, especially if they keep adding streamlined utilities like the ability to search by your own customized slashtags, date, by site type, both date and site type and most recently the ability to search by Facebook “Likes.” Unveiling a mobile version three months in is also a savvy move.

Blekko keeps getting referenced in articles like this one simply because it keeps reinforcing its place at the search table terms of innovation and press stunts, like its Spam Clock, which is a running meter of all the spam created in the US (1 million pages every hour) or the bold move to ban specific sites when Google only broadly referenced spam filtering algorithms in the wake of critical articles like “Why We Desperately Need A Better Google.”

Once you get past the intimidation of creating a funny sounding slashtag, the site becomes extremely useful on a micro-personal level. Individuals will use it in a different ways, depending on what frustrates you about search. But I have a feeling Blekko doesn’t care if you use it now, it’s got a bunch more tricks in the works to keep you coming back.

In previous posts, I wrote about the epic battles that are brewing between spammers and content farms—which are turning the web into a massive garbage dump—and search providers, which have to choose between profit and customer satisfaction. This is a serious problem. The content farms are “dumbing down” the web by churning out thousands of mostly low-quality articles, every day, on topics that Google tells them they can make money from. All of these players are raking in billions of dollars at our expense.

I had the opportunity to moderate a panel discussion this week between Google, Microsoft, and Blekko. The event, which I emceed, was called Farsight 2011: Beyond the Search Box, and was organized by BigThink and Microsoft.  As I joked, it seemed odd that Google was playing the role of “evil” monopolist; Microsoft, the “good” contender, whilst Blekko was a fly on the wall.

I had originally invited Google SEO development head, Amit Singhal, and Microsoft Research GM, Ashok Chandra. But Ashok dropped out in favor of Corporate VP of Bing, Harry Shum; and Amit dropped out in favor of Webspam team head, Matt Cutts. I was very disappointed, because Matt has the reputation of being a really nice guy—a “teddy bear”. Harry has the reputation of being feisty. I was afraid that Harry and Rich Skrenta (Blekko CEO) would devour Matt. And it would seem as if I had set up an ambush for Google.

Little did I know that Google had its own ambush cooking: that Matt was more like a tiger than like a teddy bear.

At the conference, I gave my spiel on my vision of search: how I want my computer to serve me and tell me what I want to know, rather than my having to cater to its whims by entering specific keywords in a text box and reading through text links—which are often baited by spammers. I challenged Matt to tell everyone what Google was doing about the spam. Matt, instead, went on the warpath and accused Bing of stealing Google’s information. He disclosed a sting operation that his team had run. He expressed outrage at Microsoft’s ethics. Harry Shum fired back, defended Bing, and accused Google of playing games.

There has been extensive media coverage of this. Harry Shum and Yusuf Mehdi of Microsoft both posted blogs to respond to Google’s allegations. So I don’t need to visit the same territory. You can watch the video of the event and form your own opinions. There was a lot more discussed in 40 minutes than was covered by the media, so this is worth watching.

Both sides have strong views and believe they are right. In opening the debate, I said that, as a professor, I can’t condone any kind of plagiarism or cheating—and that is what Microsoft’s usage of Google data seems to amount to. But in the tech world, such information exchange is the norm. Everyone cheats and this may be a good thing for innovation. So there is no black and white here. Both sides are right and they are wrong.

The one thing that is clear is that Google pulled off a huge PR coup. It changed the topic. Media coverage isn’t about spam and how Google profits from this any more; we are debating how valuable Google’s search results are.

Here are the real issues we should be discussing:

1. Who really owns the data that Google and Bing are tussling over? Is it the search providers—which “cheat” and copy from all over the web? Or is it the content creators—us—who they “steal” from? Why do Google and Microsoft believe that they own our information? And why aren’t they paying us for using this?
2. Facebook rivals Google in web traffic and will get way ahead. And Google can’t search within Facebook’s walls. Doesn’t this give a huge, long-term, advantage to Bing, which can (within limits)?
3. Blekko announced a bold decision to block content farms—sites such as eHow and Answerbag. Will Google and Microsoft take similar steps? Will they be able to forsake the revenue? Can the volumes of spam we are dealing with even be screened algorithmically or do we need curated search solutions?
4. We need a standard measure of web quality. Google says that it has not noticed any reduction in web quality. Yet most experts agree that this has declined significantly over the past two or three years. Why doesn’t Google, as the market leader, work with its competitors to create an open measure that can be used by everyone? Let Google prove to us that it is, indeed, better than the rest.
5. Why not allow web users to designate what sites are spam and make this information publicly available? Google lets you filter your own results, but why not share these data with everyone? Sites that believe they are unfairly labelled can lodge an appeal. Why the secrecy?

So let’s get back on topic. Harry Shum and Matt Cutts can duke it out in a bar somewhere. What I want is for them to clean up the web and give me the best search results.

Editor’s note: Vivek Wadhwa is an entrepreneur turned academic. He is a Visiting Scholar at UC-Berkeley, Senior Research Associate at Harvard Law School , Director of Research at the Center for Entrepreneurship and Research Commercialization at Duke University, andDistinguished Visiting Scholar at The Halle Institute for Global Learning at Emory University. You can follow him on Twitter at @vwadhwa and find his research at www.wadhwa.com.

Blekko, the search engine that is fighting the good fight against web spam with human editors, is joining biggies Google and Bing in the mobile search arena today with an Android and iPhone application double whammy. Says Blekko CEO Rich Skrenta, “In a world where people want the most relevant answers on the go, mobile search is becoming increasingly more significant.”

The app has a simple interface which allows you to view search results whether or not you are logged in with your Blekko account. With the exception of Facebook integration, the app pretty much provides the gamut of features found on Blekko itself, most notably the ability to search by /slashtag or curated topic. Results are sorted by most relevant and by date.

The app also offers suggested slashtags for each search at the top when you scroll down on a search. For Blekko power users, an interface with the buried treasure features of “Mark as Spam,”"View SEO info,”"Add to Slashtags” and “Open in Safari” can be accessed by clicking on the arrow next to each individual result in your search and then clicking on the box/arrow icon in the bottom right corner to reveal further options (see the image on the right, above).

Blekko, which boasts more than 100,000 slashtags created after its launch in November, has raised $24 million from VC superstars like Ron Conway, Mike Maples, Jeff Clavier and Marc Andreessen and most recently actual superstars like Ashton Kutcher.

As of today the app is free in the App Store and Android Market.

I had a feeling that this would get Google’s attention. And I had the same concern as when I challenged the Russian government, once, in a Bloomberg BusinessWeek article about Skolkovo (a new tech park). I feared that Google would either blacklist me or do its equivalent of putting me in a Gulag—deliver even more spam when I search websites.

But I was delighted to get an e-mail from Amit Singhal, the head of Google’s SEO team. His message was exemplary for those wanting to learn how to handle a PR crisis. Here is part of what he wrote:

I read your post on TechCrunch yesterday and was quite disappointed by the fact that Google search failed your students at their task. My team and I treat every such failure as an inspiration to improve Google. Would it at all be possible for me to get a few queries from your students for which our algorithm failed? We will debug every aspect of our system for those queries.

He went on to invite me to visit Google to show me how they run the search system and listen to any other criticism I had to offer.

I took Amit up on this and spent hours with him; with Matt Cutts, who heads Google’s webspam team; and with their lead developers. They were incredibly open and honest. They acknowledged the deficiencies of Google search, shared ideas on how they plan to fix them, and asked for feedback.

I raised concerns that “content farms” are turning the web into a massive garbage dump, that many sites are simply replicating the content of others like TechCrunch, and that Google has no incentive to stop this because it gains advertising revenue from the spammers.

The Google developers assured me that there is a very high Chinese Wall between them and the business side of the company; that they have been instructed by Google’s executives to do only what is in the interest of users—to keep improving quality of search results and the user experience. They said they understood the issues and had many solutions to the technical problems. I questioned whether the spam problems could even be solved algorithmically; whether the only solution was a curated web-search model like that of Blekko and DuckDuckGo. They convinced me that they could, and would, win the battle.

Matt said he would post a blog, which he did, on Jan 21. In it, he explained that Google had already made improvements to make it harder for “spammy on-page content to rank highly”; had radically improved its ability to detect hacked sites, which were a major source of spam in 2010; and was about to implement a change that would directly address the issue of sites copying others’ content. Most importantly, he acknowledged that something had to be done about the “content farms,” and said that Google would.

Not surprisingly, Matt’s blog led to another avalanche of media coverage. As it turns out, the biggest content farm of them all, Demand Media, was set for an IPO this week (on Jan 26).   Savvy bloggers and journalists began to question whether it could sustain its profits without Google’s support. The Wall Street Journal asked Did Google Just Make Demand Media Less in Demand?, and GigaOm wondered Did Google Just Declare War on Demand Media?

Nonetheless, Demand Media had a spectacular IPO. Its investors reaped huge bounties, with the company achieving a market cap of $1.7 billion—valuing it higher than the New York Times. So the public markets rewarded junk over quality. And they called Google’s bluff.

Where does that put us? Do we have to watch the web become one big toxic waste dump—as the spammers rake in billions of dollars? Or will Google indeed save the day?

There is an event on Tuesday, Feb 1, called Farsight 2011: Beyond the Search Box, to discuss these questions. It will be live-streamed on TechCrunch (watch for a post by Jon Orlin on that day) and is being organized by BigThink, a public online forum for intellectuals (people like Gary Kasparov, Jimmy Carter, Malcolm Gladwell, Salman Rushdie, Nouriel Roubini, and Paul Krugman). BigThink has thousands of videos on its site, which 1.5 million people watch every month.

I am emceeing the BigThink event and moderating a panel with three big players: Matt Cutts from Google; Harry Shum, Microsoft Corporate Vice President who heads Bing development, and Rich Skrenta, founder and CEO of Blekko.

Here are some questions that I plan to ask the panelists. Please share your comments below and suggest additional questions. I can’t promise I’ll cover all the topics you raise, but I will bring up as many as I can.

1. How will they save the web? Is it possible for search engines to separate the wheat from the chaff—tell the difference between content produced by regular people and large-scale junk produced by the spammers?

2. How are the engines really different? Most people can’t tell the difference between Google and Bing. Where is the magic?

3. What lies ahead? What is the future of search?

I have no doubt that this will be a very lively and informative event.  There are also other great presentations such as:

• Jaron Lanier, named by Time Magazine in 2010 as one of the 100 most influential people in the world, speaking about the need for a new sustainable revenue model for search.
• Esther Dyson, in an address entitled “The Future of Search is Verbs,” speaking about how we want search to help us do something—a set of many verbs.
• Demos from Blaise Agüera y Arcas, Architect of BING maps at Microsoft; and Luc Barthelet, Executive Director of Wolfram|Alpha.

I hope you’ll tune in.

Editor’s note: Vivek Wadhwa is an entrepreneur turned academic. He is a Visiting Scholar at UC-Berkeley, Senior Research Associate at Harvard Law School , Director of Research at the Center for Entrepreneurship and Research Commercialization at Duke University, and Distinguished Visiting Scholar at The Halle Institute for Global Learning at Emory University. You can follow him on Twitter at @vwadhwa and find his research at www.wadhwa.com.

The recent victory of IBM’s Watson computer against human competitors in an exhibition round of Jeopardy got computer scientist Stephen Wolfram thinking about how regular search engines might fare in such a match-up. So he took 200,000 known Jeapardy clues and ran them through six search engines (Google, Bing, Ask, Blekko, Wikipedia Search, and Yandex). He excluded known Jeopardy sites from the results, and didn’t test his own Wolfram Alpha because it is not designed for those kinds of queries.

What he found is that the search engines did fairly well, depending on how you measure success. Google did slightly better than the rest, but Bing and Ask were close behind. On average, Google got the correct answer somewhere on its first results page 69 percent of the time, versus 68 percent for Ask and 63 percent for Bing. Google got the right answer somewhere in the title or snippet of text of the very top result 66 percent of the time, versus 65 percent for Bing (and Ask dropped to 51 percent).

In comparison, most humans answer 60 percent of Jeopardy clues correctly, while the top player of all time, Ken Jennings, answered 79 percent correctly. So it is conceivable that a system could be created using regular search engines that could beat most humans. But Wolfram cautions:

Of course, the approach here isn’t really solving the complete Jeopardy problem: it’s only giving pages on which the answer should appear, not giving specific actual answers. One can try various simple strategies for going further. Like getting the answer from the title of the first hit—which with the top search engines actually does succeed about 20% of the time.

Answering Jeopardy clues correctly and consistently is a hard problem for computers to solve because of all the variations and nuances of human language. Yet “just using a plain old search engine gets surprisingly far,” concludes Wolfram.

Blekko, a search engine, isn’t exactly a flashy new startup. There’s the name for starters – it was something they came up with as a placeholder while they were stealth and decided that was a good long term name, too.

And then there was the whole Cuil fiasco. Which would make any new search startup blanch. So Blekko hasn’t turned on the hype machine at all. They’re just quietly doing their thing, and growing nicely.

Now, though, they’ve gone Hollywood. Ashton Kutcher (image is from TechCrunch50 2008), whose movie is currently no. 1 in box offices, is making a somewhat more geeky splash today. He’s invested $200,000 in Blekko, as an extension to the company’s most recent round of funding.

You can follow Ashton’s activities on Blekko here. The company has raised around $25 million in debt and equity capital to date.

This semester, my students at the School of Information at UC-Berkeley researched the VC system from the perspective of company founders. We prepared a detailed survey; randomly selected 500 companies from a venture database; and set out to contact the founders. Thanks to Reid Hoffman, we were able to get premium access to LinkedIn—which was very helpful and provided a wealth of information.  But some of the founders didn’t have LinkedIn accounts, and others didn’t respond to our LinkedIn “inmails”. So I instructed my students to use Google searches to research each founder’s work history, by year, and to track him or her down in that way.

But it turns out that you can’t easily do such searches in Google any more. Google has become a jungle: a tropical paradise for spammers and marketers. Almost every search takes you to websites that want you to click on links that make them money, or to sponsored sites that make Google money. There’s no way to do a meaningful chronological search.

We ended up using instead a web-search tool called Blekko. It’s a new technology and is far from perfect; but it is innovative and fills the vacuum of competition with Google (and Bing).

Blekko was founded in 2007 by Rich Skrenta, Tom Annau, Mike Markson, and a bunch of former Google and Yahoo engineers. Previously, Skrenta had built Topix and what has become Netscape’s Open Directory Project. For Blekko, his team has created a new distributed computing platform to crawl the web and create search indices. Blekko is backed by notable angels, including Ron Conway, Marc Andreessen, Jeff Clavier, and Mike Maples. It has received a total of $24 million in venture funding, including $14M from U.S. Venture Partners and CMEA capital.

In addition to providing regular search capabilities like Google’s, Blekko allows you to define what it calls “slashtags” and filter the information you retrieve according to your own criteria. Slashtags are mostly human-curated sets of websites built around a specific topic, such as health, finance, sports, tech, and colleges.  So if you are looking for information about swine flu, you can add “/health” to your query and search only the top 70 or so relevant health sites rather than tens of thousands spam sites.  Blekko crowdsources the editorial judgment for what should and should not be in a slashtag, as Wikipedia does.  One Blekko user created a slashtag for 2100 college websites.  So anyone can do a targeted search for all the schools offering courses in molecular biology, for example. Most searches are like this—they can be restricted to a few thousand relevant sites. The results become much more relevant and trustworthy when you can filter out all the garbage.

The feature that I’ve found most useful is the ability to order search results.  If you are doing searches by date, as my students were, Blekko allows you to add the slashtag “/date” to the end of your query and retrieve information in a chronological fashion. Google does provide an option to search within a date range, but these are the dates when website was indexed rather than created; which means the results are practically useless. Blekko makes an effort to index the page by the date on which it was actually created (by analyzing other information embedded in its HTML).  So if I want to search for articles that mention my name, I can do a regular search; sort the results chronologically; limit them to tech blog sites or to any blog sites for a particular year; and perhaps find any references related to the subject of economics. Try doing any of this in Google or Bing

The problem is that content on the internet is growing exponentially and the vast majority of this content is spam. This is created by unscrupulous companies that know how to manipulate Google’s page-ranking systems to get their websites listed at the top of your search results. When you visit these sites, they take you to the websites of other companies that want to sell you their goods. (The spammers get paid for every click.) This is exactly what blogger Paul Kedrosky found when trying to buy a dishwasher. He wrote about how he began Googleing for information…and Googleing…and Googleing. He couldn’t make head or tail of the results. Paul concluded that the “the entire web is spam when it comes to major appliance reviews”.

Unfortunately, it isn’t just appliance reviews that are the problem. Almost any popular search term will take you into seedy neighborhoods.

Content creation is big business, and there are big players involved. For example, Associated Content, which produces 10,000 new articles per month, was purchased by Yahoo! for $100 million, in 2010. Demand Media has 8,000 writers who produce 180,000 new articles each month. It generated more than $200 million in revenue in 2009 and planning an initial public offering valued at about $1.5 billion. This content is what ends up as the landfill in the garbage websites that you find all over the web. And these are the first links that show up in your Google search results.

The bottom line is that we’re fighting a losing battle for the web and need alternative ways of finding the information that we need. I hope that Blekko and a new breed of startups fill this void: that they do to Google what Google did to the web in the late 90’s—clean up the spam and clutter.

Editor’s note: Vivek Wadhwa is an entrepreneur turned academic. He is a Visiting Scholar at UC-Berkeley, Senior Research Associate at Harvard Law School and Director of Research at the Center for Entrepreneurship and Research Commercialization at Duke University. You can follow him on Twitter at @wadhwa and find his research at www.wadhwa.com.