Have you ever tried to search for reviews of web hosting services? Results typically fall into two categories: A) Anecdotal experiences and B) Spammers trying to cash in on affiliate programs. It’s nearly impossible to get an idea of what hosts actually have a good reputation.
Kevin Ohashi, a former moderate of the popular Web Hosting Talk forum, wants to change that. Ohashi did he master’s thesis on sentiment analysis, which gave him the idea of using Twitter to determine which hosts have the most satisfied customers. Thus Review Signal was born.
Ohashi admits that sentiment analysis is an imprecise science. He says that a 60-70% accuracy rate is really good in sentiment analysis. But if a user goes to the front page of Review Signal and sees three or four obviously miscategorized tweets, that discredits the whole site. So Ohashi is focusing on confidence over completeness. If Review Signal isn’t extremely confident that it has the correct sentiment of a particular tweet, or whether the tweet is referring to a specific web hosting service, that tweet will be thrown out. That means settling for a smaller sample size, but better credibility. It also means that Amazon Web Services can’t yet be ranked as a web host because there are too many inaccurate results, but Ohashi says AWS is a bit of an outlier of a web host anyway.
He also admits that keeping spammers from gaming the results won’t be easy. “It’s going to be a constant battle,” he says. “Google fights that fight every day with SEO.” He spends a lot of time looking at the tweets that are analyzed and tweaking the algorithms so that bad results are thrown out and he’ll continue to do so. Also, users can vote on whether particular reviews are helpful.
And he points out that he has been collecting tweets for a year and a half so he’ll be able to spot it if a company has a big spike in positive reviews. He’s already seen one non-malicious case in which a host had a sudden spike in positive sentiment. It was the hosts anniversary and many users were posting “happy birthday” messages. He says that he did adjust the algorithms a little bit to throw out reviews that were just something along the lines of “happy birthday,” but he did keep tweets that said something along the lines of “happy birthday, been a happy customer for years.”
I asked him about whether saw any trends regarding GoDaddy’s outage or any of the bad publicity the company has had over the years. He said he has data from several incidences: the Super Bowl ads, the former CEO’s elephant hunting fiasco, the company’s position on SOPA and the recent DNS outage. Only the recent downtime has had any noticeable impact.
By posting the tweets that are being analyzed on the site he hopes to bring more transparency to the process of ranking hosts. Ohashi does have affiliate links on the site, but he says he doesn’t let it influence the rankings. For example, Linode, which doesn’t have an affiliate program, is one of the top ranked host on the site. For now this might be OK, but if the site grows both the distrust and the temptation (even subconscious) to tweak the algorithms in favor of affiliates will likely grow — especially since he has to hand pick the hosts that are ranked. So I’m guessing he’s going to have to find another business model eventually.
However, he dismisses the idea of selling brand monitoring services — there are plenty of companies in that market already. He wants to be a consumer business. He says he’d also like to get into other verticals. He says he’ll start by adding review of related services, like domain registrars and e-mail providers. But he’d like to get into other areas, such as movie and entertainment reviews as well. The key, he says, will be in getting representative data. Twitter works really well for hosting reviews, but it won’t work well for all types of products and services.
Another challenge he will face is access to tweets. For now he’s just using the Twitter search API, but that’s shutting down. As the site grows he’ll probably need to get access to the firehose, or get access to search results from someone who does. That will increase his costs.
Challenges aside, it already looks like a huge improvement over the status quo. Let’s hope it stays that way.