Speeding Up RSS

I’m sorry, but RSS feeds are way too slow. I know this first-hand. As part of my job here at TechCrunch, I monitor a lot of RSS feeds for breaking news. We also produce our own feed and I can see how quickly it propagates to various feed readers and feed-powered news aggregation services. The lag time between posting a story and seeing it pop up in the RSS feed is usually a few minutes, and then it can take another 10 to 15 minutes or so for it to appear in something like Google Reader. And the TechCrunch feed is probably checked more frequently for updates than most other feeds. In our business, every second counts and RSS just isn’t cutting it.

While there is an argument to be made that RSS is dying, being replaced by more instantaneous forms of content delivery such as Twitter and other real time streams, many people aren’t quite yet ready to give up on it. Instead, they want to save it by speeding it up. Tomorrow, at our Real Time Stream CrunchUp, we will see three demos of projects that do just that in slightly different ways.

Google engineers Brad Fitzpatrick and Brett Slatkin will show a demo of a new push protocol called pubsubhubbub, Netvibes CEO Freddy Mini will demo his similar RSS Instant Update Hub, and WordPress engineer Andy Skelton will show off a Jabber client which uses the XMPP protocol to push blog headlines into an IM-like environment faster than RSS.

The pubsubhubbub and Netvibes technologies create RSS hubs, which push out feeds as soon as they are available. This approach is in contrast to the polling method which is the foundation of RSS. The polling method sucks because it requires the server acting on behalf of the RSS subscriber to constantly ping the server where the RSS feed is published to ask if there is anything new yet. Depending on how often this happens, you end up with a lag. As communications become more real-time, this lag is becoming more noticeable.

The way pubsubhubbub fixes this is by putting an RSS Hub in the middle which more efficiently pushes out the feeds to the servers subscribing to them. It is an open protocol and can be applied to any existing RSS or Atom feed, as well as other real time streams. If you think about the Twitter firehose that everyone wants access to, this approach lets anyone create their own firehose for different types of data streams. It is more of a federated approach. You can think of these RSS Hubs as a content delivery network of sorts for RSS feeds, similar in concept to what Akamai does for video streams.

Netvibes is creating its own proprietary version of this for its own service, which it is developing independently. It is called the RSS Instant Update Hub. All of those widgets on your Netvibes page today take forever to load because they each have to fetch the underlying feeds of data. The Instant Update Hub will cache and push these feeds automatically so that the widgets load faster and they update continuously without requiring a refresh. Any data stream that is supported by a Netvibes widget today, which goes well beyond RSS, will be pushed through the Instant Update Hub. It will also form the basis for a new stream reader which Netvibes will introduce later this year as an alternative to its current widget grid and magazine-style layouts (see screenshot below).

The WordPress Jabber client uses a different push technology, XMPP, to speed up RSS. The effect is that headlines pop up like instant messages. Jabber is mostly used for IM clients such as Gtalk, but WordPress is using it as a feed reader and micro-blog publisher. The great thing about it is that it is two-way. In the demo, Skelton will show how the Jabber client can be used as an interface to to post directly to your blog. Feed reading and blog posting can all be done from the same place in a more real-time fashion. The Jabber client can also be used as a blog commenting system and embedded as a widget directly into a Web page, turning comments into more of a chat room.

What we are seeing is the world’s of publishing and IM colliding. The faster we can close the loop between publish and response, the more we are going to see real-time data streams take on the look and feel of public IM systems. Twitter is asynchronous, but it often feels immediate with back and forth conversations sometimes happening almost fast as a private IM chats. That is just a taste of things to come, as all publishing platforms get up to speed.

(Photo credit: Flickr/joiseyshowwa)


Here’s the video from PubSubHub’s demo at the Real-Time event: