Twitter is living up to its promise to open up its data stream as much as possible to developers. While I was negotiating with Twitter cofounder Evan Williams to sit down and do a video interview at Foo Camp last weekend, Gnip founder Eric Marcoullier was hitting him up to give Gnip, and therefore everyone, Twitter’s XMPP “firehose.” Williams was obviously in a good mood, because I got my interview and, as I just found out today, Eric got his data feed.
What does this mean for the average Twitter user? It means that more third party services will start to work better. Today, other than a handful of services like Summize (which was just acquired by Twitter) and Friendfeed, third party apps must talk to Twitter via their normal APIs. Those APIs require applications to send Twitter a request and then get a response. The two way communication creates a big load on Twitter in the aggregate.
With XMPP Twitter just sends out all of their data in a constant stream, whether you ask for it or not. The third party, in this case Gnip, takes the data and parses it for further use.
Gnip acts as an intermediary between applications that create social content and those that consume it. They take the Twitter feed, which is a list of usernames, Twitter status URLs and time stamps, and make it available to any third party that requests it. Both Plaxo and MyBlogLog are already using the new feed, and more partners will add it immediately. And every third party that takes data from Gnip doesn’t have to take it from Twitter, easing the overall load on Twitter’s servers.
For now Gnip is only sending updates for requested users, not the richer data that some applications like Twhirl need to build a Twitter-like desktop environment. Twitter may give Gnip permission to send additional data, like @replies and direct messages, over time (if that last sentence doesn’t mean anything to you, it means you aren’t a crazy-heavy Twitter user, just disregard it).
What this means is that Twitter is taking yet another step towards openness and leaning on outside parties to help them with scaling issues.