TechMeme founder Gabe Rivera makes an interesting observation on the Google News story all over the blogosphere today.
One thing that bugs me: they’re now hosting original news content, yet they prohibit other aggregators from crawling it (per robots.txt restrictions and TOS). Of course Google News relies on the openness of other organizations with original news content.
Google crawls news sites and grabs their content for republishing on Google News. They rely on the willingness of those news sites to get distribution on Google. But Google restricts others from crawling Google News itself via their robots.txt file and terms of use, which state that “you may not…use any robot, spider, other device or manual process to monitor or copy any content from the [Google News] Service.”
That policy wasn’t a problem when Google was simply aggregating news from around the web. But now they are hosting original news content, written by people that are involved in the story. And they are telling the world that no one else can crawl that content and display it. Yahoo News, TechMeme and every other non-Google owned news service on the web is restricted from using that content.
The restrictive policy hasn’t changed with the new feature launch, and this may just be an oversight. We’ll find out soon enough if Google intends to build a wall around this news content, or share it with the rest of the web.