Topix has become a respectable web property, which is why today’s announcement that it will be partnering with young search engine, Blekko, seems like an interesting move. Blekko only launched publicly back in November of last year, so the human-curated, slash-tagging search engine is still very much an unestablished entity.

I think many would agree that search is need of some de-spamming and a fresh take, and Blekko is certainly that, but the road before them is uphill to say the least. Blekko has made a laudable effort to crackdown on spam, eliminating 1 million spammy links from its search results in March.

What’s more, Blekko does not accept paid traffic, it uses humans on top of algorithmic search functions, and it’s getting more social — all great things. I applaud how ferociously Blekko is attacking the current problems in search, hey, it attracted 575K unique IPs in March, so people are checking it out.

I just worry that the search engine’s crusade against spam may result in the elimination of legitimate sites as well, and it can be slow to crawl smaller sites, and update name changes, and so on. Right now it indexes 3.5 billion URLs, which is just no match for Bing and Google, which are both way over 15 billion.

That being said, I am very intrigued to see how Blekko integrates with Topix, which averages over 1 million searches a month. With a more focused set of data to index, I think Blekko could start turning out some great search results. From testing it out a little bit today, it’s worked great, but I’ll have to dive deeper to know for sure.

But obviously what really keeps this from being a shocking development, or a surprising risk for a sizable platform, will not come as a huge shock to those familiar with the two companies’ histories. Blekko CEO Rich Skrenta and Blekko Marketing VP Mike Markson are both co-founders of Topix.

So, while Topix was presumably predestined to add Blekko search at some point, no surprise there, it is interesting to see that the Blekko team now finally feels that its search engine has been tuned (and tested) enough to make this implementation launchable.

“It is rather unprecedented for a search start up to be able to power search on a site of scope and scale as Topix,” Skrenta said. “We have a long history of collaboration and camaraderie with the Topix team so it’s a thrill to be working together again in bringing a new search experience to Topix users.”

Is it working? Chime in with your own experiences.

“Would you like to comment on the accusations being made about your site by the Attorney General?” Excuse me? What is it about the week after my birthday? Two years ago, Google launched a local news product the day after my birthday and this year, it was Cheryl Truman of the Lexington KY Herald-Leader calling me with this little gem.

Evidently, this is how a California company finds out that the Attorneys General of Kentucky and Connecticut have “significant concerns” about how your company does business. They have a press conference. When I told Cheryl I had no idea what she was talking about, she kindly forwarded me the press release from Jack Conway, the Attorney General of Kentucky.

Through this press release, which accused us of requiring payment to review abusive posts, I discovered that the Kentucky Attorney General had allegedly sent a letter asking me to provide information regarding our terms of service and policies around payment for expediting reviews. (The letter to which the press release referred was put in the US Mail and post marked five days after this incident.)

Topix is the largest local forum site in the US and deal with 3-4 million comments a month, and I was going to find out a little more than I bargained for with regard to the politics involved in focusing on local audiences.

Dr. Jekyll and Mr. Press Release

This initial dynamic – being communicated at via excoriatory press release proved to be a consistent theme going forward. I suppose I should take it as a testament to our success that taking on Topix would be perceived as an important public policy goal for two candidates for the US Senate. After waiting the 10 or so days to actually get the letter, we set up a call with our lawyers and a bevy of assistant attorneys general. After my lawyers reconfirmed that we weren’t being accused of breaking any specific laws, I decided to take a pretty open stance with these guys and give them the background on what we did, and how we did it, figuring that if they knew what we were doing, and in particular, if they knew that the paid expedition of reviews was only about 1% of all of our feedback, that we would be able to clear this up pretty easily. (I was wrong about this).

The call with these guys was actually pretty cordial. We walked them through how we ran feedback at Topix, that how in January 2010, we posted 3.6M comments, had our Artificial Intelligence systems remove 390k worth before they were ever even put up, and how we had over 28k feedback emails and 210k user flags, resulting in over 45k posts being removed from the system. When we went through the various issues with them, we ended up coming to what I thought was a set of offers to resolve the issues at hand. The folks on the phone indicated that these were good steps, and that they would circle back with their respective Attorneys’ General and get back to us.

No good deed goes unpunished

So, after opening the kimono and giving these guys a whole lot of info on how we ran things, how big we were and that we dedicated 20% of our staff on these issues, what was the response. (You could probably see this one coming.)

That’s right. Another press release. This time from 23 states’ Attorney’s General.

This pile-on took much of what we had told them, and turned it against us. We had mentioned that we required three separate people to flag something before we would take action (mainly to prevent individuals from easily spiking things that they didn’t like). That was called out as a particular sin to be cleansed from our site. They also asked us to drop the priority review program in its entirety, drop the time it takes us to review posts from 7 days to 3 and “immediately revamp our AI technology to block more violative posts” amongst other things.

So, more calls from local newspaper reporters who had been sent the press release, this time from a few more states. And the AP.

Somewhere around this time, I reached out to my friend Chris Kelly, who was the former chief privacy officer for Facebook, and whose wife used to work in the office of the Kentucky Attorney General. He gave me some good advice on the issue, and since he was pretty busy for running for the California Attorney General position at the time, referred me to talk to a former Attorney General from Indiana, Jeff Modisett who is now in private practice. While our law firm, PietraGallo, out of Pittsburgh, had been a great job with coordination and laying out our options, I wanted to make sure that if things got much worse, that I had an escalation path with someone who knew these guys personally. Perhaps the most useful outcome of these conversations was that I discovered the form factor of resolution for issues like this was a so called “joint statement”, such as those between MySpace and Craigslist and a myriad of Attorneys’ General.

You can’t fight city hall

After the second smackdown by press release, I concluded that it wasn’t worth trying to stand up for the $20k a month in prioritized reviews we were doing, and that it simply wasn’t worth the cost of getting in deeper. So, armed with the understanding that we should be seeking out a “joint statement” we pretty much walked down the list with the Assistant AG from Kentucky, Todd Leatherman, and tried to find a way to say “yes” in as painless a way as possible. There were also some points they made which were probably good for us to implement (like giving law enforcement a more direct way of reaching us), and this time I made it clear that I wouldn’t be officially answering anything unless they agreed to communicate with us first before holding a press conference to highlight any remaining disagreements.

Our joint statement was issued on August 9th, and reported in 238 different articles including a post on TechCrunch. All hail our government in action.

I hope you learned your lesson

Going through this was a tremendous pain in the ass, and I would recommend avoiding it if you can. However, if you run a site that people actually use, chances are something like this might happen to you. So, here are my thoughts and recommendations.

This is going to happen more – The States’ Attorneys General are the place that complaints about your company will probably end up. This is especially true if you host a social or community based site where people can post things that others may dislike. And, there’s no downside to attacking a company based in California for these guys (MyScape, Facebook, Craigslist have all been targets in the past couple of years). Taking complaints from your citizenry and turning them into political capital is simply too good an opportunity for these guys to pass up.

Nobody expects the Spanish Inquisition – Unlike most other people in business who will attempt to reach out to you to get what they want, and use the threat of going public as a tool, our experience is that the offices of the Attorneys’ General seem to be most happy communicating via press conference, without any sort of preliminaries. This is primarily a political exercise, and you’re dealing with people who are very empowered to make life difficult for you.

Pissed off people, not illegality, is the issue to watch – At no time during this process were we accused of breaking any laws. The Attorneys General have interpreted their mandate of consumer protections very broadly, and if a lot of people *think* you are doing something wrong, you are likely to be headed for a problem. The biggest issue for us is that we put a button that had non deterministic outcomes (three people needed to flag a post) next to an option that wanted you to pay $19.99. The fact that we told people in big bold letters that this was paying for making your free review faster did not discourage people from concluding that we had disabled the free feedback and that we were charging to remove posts, rather than letting people jump the queue.

Take this seriously and lawyer up– especially If you find yourself in a place where there is significant downside (instead a small feature, you need to shut down, or there is a massive monetary amount at play), you really want to find some folks who have worked with these guys, and potentially have been one of these guys before. We have great lawyers, but you really need to keep in mind that an AG essentially is a state run law firm employing hundreds of people. I essentially caved and it still took six months and more money than I care to think about.

You’re not a wimp, you’re a pragmatist – I never thought I would be putting my name on a joint statement with 34 states attorneys’ general. It turns out that they’ve organized themselves into a sort of zone offense strategy and the Richard Blumenfeld AG from Connecticut, essentially is running point for other AG’s who want to take on technology companies. Things quickly escalate, and it’s unlikely to be good for business to be in a protracted fight with the government. I will point out that Microsoft and Intel were both accused of all sorts of monopolistic business practices in the 90’s and Intel mostly worked with the government and Microsoft mostly didn’t. Seems to me that Intel’s approach worked better.

It takes longer than you think – Especially when these guys start to pile on, it takes them a while to discuss what they’re going to do with you. For us, he first press release came out on Feb 11, and the joint statement on August 9th. The ball is often in their court, and there’s no much you can do in the meantime.

Sometimes they’re right – Despite the rather venal and self serving communications strategies, the issues that the Attorneys’ General take on usually have some set of consumers who have brought it up in the first place. While I think I have every right to offer an expedited review system, it pissed off enough people to enter the issue intot he realm of the political, and it’s probably better for us to take it off the site. Likewise, it was probably good for us to give law enforcement a more straightforward way to communicate with us (people like to post about crimes they commit or see). (And I’m glad Cuomo, the NY AG took on the banks last year and made them cash out auction rate securities for businesses like ours.)

I think I’m going to take the week after my birthday off next year.

This is content that individuals or entities find to be offensive or damaging. Topix takes up to seven days, they say, to review flagged content, but payment of the priority fee pushes the complaint to the top of the list. From the joint statement, embedded in full below:

Topix.com is a popular website that, inter alia, allows consumers to post comments, polls and surveys in local forums for the purpose of facilitating discussion about news and other matters of local community interest, and thus provides consumers with an open platform to become more informed regarding local matters. As with all communication tools, it can be misused by consumers. Specifically, a number of posts submitted by consumers contain false, obscene or derogatory information. Some consumers have also complained to the Attorneys General about the impact of inappropriate posts on their personal and family relationships, their reputations in their communities and the impact upon their children. Consumers have also raised concerns about the amount of time it takes Topix to review reports of abusive posts and Topix’s policy of charging $19.99 for a “Priority Review” of such posts.

Topix has agreed to eliminate the priority review product entirely, and “will continue to refine and improve the technical tools and human resources necessary to allow it to review Feedback requests for reported posts.” The company has agreed to aim for a response time of three working days.

I spoke with Topix CEO Chris Tolles by phone today about the agreement. This is a small percentage of total Topix revenue, he says. He added: “Dealing with elected officials from all fifty states and, evidently, other U.S. territories, is part of doing business on the Internet in 2010.”

In fact, Topix is the number one result for a ton of small towns across the country. That’s what happens when you’ve been aggregating local news for six years. But the site has never garnered much attention in the startup scene because back in 2005 (before TechCrunch even existed — our initial coverage was in 2007), it took a majority equity investment from three of the largest media firms in the U.S., Knight Ridder, Tribune and Gannett. Since then, the focus has been all about getting the business to work.

And while Topix has been through some permutations throughout the years, the focus has always been on local and yes, topics. Aggregation was the key to all of this for a while, but in recent years, the community that uses the site has been largely responsible for a lot of the content. And that’s more clear than ever as the site is about to cross 100 million posts (comments).

The milestone should be hit at some point today, as the site is getting about 125,000 comments a day now. That’s on top of 120 million average monthly pageviews from 8.8 million unique visitors. All huge numbers for a site that hardly anyone talks about.

But that’s the thing, people are talking about it, you just may not realize it unless you’re in one of these smaller towns. For these people, Topix is a gathering spot to discuss the local news. Many of these cities (like Pepper Pike) may not have a daily local newspaper or website that makes it easy to discuss issues. That’s exactly what Topix does. And perhaps even a little too easy as the site has to remove some 45,000 comments a month due to things like hate speech, CEO and co-founder Chris Tolles tells us.

Not just that, Topix gets about 10 subpoenas a week due to comments, Tolles says. That’s a pain, but it’s the price you pay for having such an active community.

Why not ban anonymous commenting? Tolles has thought about this a lot, and done internal studies. But all conclude that it wouldn’t be good for the site. For example, while you’re 50% more likely to get a crappy comment from anonymous commenter, you’re also going to get five times the number of comments when you allow for anonymous ones. And if there’s a situation where a person feels the need to call out a local official, doing so anonymously allows them to feel protected, Tolles says.

So, Topix has all these comments, and all this content — but the challenge has been how to monetize it. While Topix has deals with seveal big players, including CNN, the real money remains in advertising. So Tolles has spent much of the past year building up an ad sales team. And following the ad collapse last year, it’s finally starting to pay off. Topix is seeing $4 eCPMs, Tolles says. If that keeps up, he thinks the site can turn a profit relatively soon.

Something else interesting about Topix: mobile usage is huge. While the service has had (what Tolles considers to be currently a sub-par) iPhone app for a while, it’s the mobile web that people seem to be taking to to use the site. And something like 75% of the people who are actively commenting on a mobile device are visiting from an iPhone. That may seem somewhat surprising given Topix’s focus on small towns and communities, but it seems to speak to the iPhone’s appeal to the masses. The big time commenting numbers also apparently speak to Topix’s appeal to the masses.

Topix, the local news community that allows readers to submit and edit stories, has secured deals with six content providers as it moves to establish itself as a one-stop-shop for local information.

Topix will now use Eventful as an events database, and will offer current TV and movie listings through Zap2It. InfoUSA will supply business directories, and the site will begin using LiveDeal for pet classifieds. Topix will be using Apartments.com to flesh out their apartment listings, and will use Informa Research Services to provide mortgage rate information.

The company originally launched Topix.net (now Topix.com) in 2004 as an automated aggregator of local news. In 2007 the site relaunched to allow users to add and modify news stories.

Topix is clearly moving towards becoming a central hub for localized information rather than simply a news site. While the new features are logical extensions to what they’ve offered in the past, Topix may be trying to differentiate itself from Google News, which introduced local news earlier this year and has a much higher readership.

The localization option is now available on news.google.com, but does not seem to have been rolled out to country specific news portals as yet. Using the service is as simple as entering your post/ zip code or location into the Local News option box that now appears automatically when you visit Google News.

The feature pulls news stories based on your location, and are presented as a sub-section on the main Google News page, and have a sidebar menu entry and dedicated page as well.

According to Google:

As always, results will be clustered with multiple sources on a story. The top stories for a given area will be at the top of your results. Our article rankings will also take into account a publication’s location so we can promote all the local sources for each story.

Google News attracts a far larger audience than Topix – 47 million/month v. just 6.2 million for Topix (Comscore, December 2007 worldwide audience), but until now they have not provided good local news coverage. Topix now has some serious competition.

The official explanation is below. The real explanation will come out soon, I’m sure:

Even though we were seeing eye-to-eye on what needed to be done, after the press tour and some initial conversation about where we needed to go, Rich told me that he wasn’t having as much fun right now, and looking at what we needed to do, didn’t see much opportunity for what he really loves to do — architect from the metal on up and put out the 1.0 that is so far ahead of everything else.

See our coverage of the company here. They’ve raised around $20 million in funding.

The news was first written about by USA Today this afternoon. Topix is partly funded by USA Today’s parent company, Gannett, McClatchy and Tribune.

Topix already has local news pages for cities and towns across the United States and Canada (here’s the page for Atherton, CA). The news is auto-generated for those pages currently. Tonight, Topix will let users of the site become local editors, and write or expand on articles that will be of interest to that community.

Topix started allowing readers to leave comments on the site just over a year ago. The first editors will be selected from heavy commenters on the site.

Screen shots below. See the Topix corporate blog, and CEO Rich Skrenta’s personal thoughts as well.

Topix brings a long list of features to news search and focuses on local information.

The company was started by several founders of the DMOZ directory. The company says that it is currently experiencing substantial growth, something that traffic analysts Hitwise confirm. Hitwise reported in September that Topix’s market share in online news had grown 24% in one month since an August relaunch, that the site gets more traffic than the LA Times and the NY Post; Topix visitors were unusually weighted towards states like Kentucky, Alabama, Mississippi, Tennessee, and North Carolina and its users were 29% more likely to be over the age of 55 than the average Internet user.

What does Topix offer that Yahoo! News and Google News don’t? A few things. When Topix relaunched in August, the company said its index had grown to include 50,000 news sources – 10 times more than the Google News and 7 times as large as Yahoo! News. Topix has topical pages, has strong search by zip code, integrates relatively active forums and visualization. The year long time line displayed with every search is a fast way to skip to a particular date in a news search and shows the context of the point in time you are skipping to. Try using the timeline on Topix and then try to accomplish the same result with Yahoo or Google news search. Topix also has the most extensive support for RSS.

Though traffic stats have to be taken with a giant grain of salt, look at those demographics intersecting with the feature set. Topix is proof that you can load a site up with features and still win over mainstream users.

The company provides local and national news to online publications owned by their investors like the LA Times, the Chicago Tribune and the Baltimore Sun. Topix also powers Ask.com’s local search and says it has other distribution deals with Yahoo!, Bloglines, Newsgator, CitySearch and several other large sites.

Topix has always been a forward looking company; before Google starting moving to integrate blog search and news search and offer search results more than a few weeks old, Topix had done both of those well for more than a year. Why they need almost $20 million in investment over the last two years is unclear, but it can’t be denied that they are doing a good job of providing a valuable service to many online properties and are creating a rich experience for users once they are drawn in.

I look forward to seeing more exciting things come from this company. Local search is becoming increasingly competitive, but when it comes to news and features, Topix is a key site to watch.

Founded in 2002 by some of the creators of the DMOZ open directory project, majority stake in Topix is now owned by media companies Gannett, McClatchy/Knight Ridder and the Tribune Company.

The best new feature in this week’s relaunch is the visual timeline for the number of results from each day in the last year. It’s like Technorati’s blog search timeline, but on Topix you can click any point in the visual and see results from that day and prior. Google and Yahoo News results only go back 30 days or less.

Searches on Topix are now case sensitive, so you can search for “IT” for example, without getting a page full of results for the word “it.”

Search results from blogs are highlighted with a little blue “B” next to the story titles. Integrating blog and news search results by default is a nice touch. Words in the result excerpts that have their own dedicated topic pages are highlighted and display a drop down box of their own recent results.

With forums and classified ads and loads of news categories, Topix really flirts with feature overload. In the past, pages been too messy to use. This week’s relaunch is focused on some very helpful changes though and makes this powerful site easier to use. In fact, I’d say it’s gone from being almost unusable to being a uniquely powerful news search service.

Unlike Yahoo, which took the step of integrating blog search results into the news area but pushed them off to the side, Topix has integrated blog posts into results in the same manner as their other 12,000 news feeds.

The Topix blog post announcing the addition gives great information on how they chose the 15,000 feeds, and how blog coverage differs from traditional news coverage. Good stuff. I’m really happy to see this.