The New Datastream Aggregators, FriendFeed and Standards

Nik Cubrilovic

Nik Cubrilovic (koo-bree-low-vick) is an Australian-born entrepreneur, technologist, software developer and blogger. Nik has been a writer and advisor to Techcrunch since 2005, is a founding editor of TechcrunchIT, and is currently working at Techcrunch and on the Crunchpad project. Nik is the founder and CEO of Omnidrive, a web content and storage platform. Nik was also the founder... → Learn More

Thursday, June 26th, 2008

FriendFeed is an example of a new type of aggregator where multiple data types, or streams, for a user can be bought together. The difference between this new type of aggregator and the standard RSS feed-reader type of aggregator is that the later does a good job of supporting a set of formats that are almost standard, while the new aggregators such as FriendFeed are building in app-specific support.

The disadvantages of this model are that only the selected applications can participate in the aggregator and network, a disadvantage to smaller providers and a position that would only cement the place of the more popular services. In addition, service-specific API support would not scale very well, as every API update and nuance would have to be tracked, maintained and debugged.

A very important issue is how feed and data types should be standardized to allow all application providers to participate in aggregator ecosystems such as FriendFeed, rather than just the 41 selected and supported.

The 41 services provided are just feeds, be it plain descriptive XML or RSS or Atom or some variation thereof, they are effectively feeds of varying data types. They can be easily broken down into the different categories and hence data types – such as link feeds, content feeds, image feeds, event feeds etc. etc. Currently each application API developer implements what they feel is a good format for an API (and often poorly) and it is often very different to what similar applications provide.

With generic feed standards, either through the use of Microformats or the use of defined namespace extensions, it would level the field so that any application or service can participate rather than those that are popular amongst the digerati. Imagine if the earliest news feed readers only supported a set of 41 sites? So you wouldn’t even be able to subscribe to your own blog, let alone the blog of a friend or colleague.

For some reason, outside of content feeds (blog RSS feeds), the other formats have developed in a sporadic and non-standard manner. The standardization of feeds around data types has almost reached a point of becoming an urgent priority, as the mechanisms that previously worked well in defining standard formats broke down with the new types of shared media.

FriendFeed recognize the problems and have commited to supporting the Media RSS standards, but Bret from FriendFeed also said that opening up the aggregator could result in more noise and spam which would drive away users. I believe that the problem may be overstated, and a subscription-based model effectively results in pure, chosen content.

Either way, the next messaging or photo sharing application being built should be able to build an API based on a standard spec for each data type. While there are plenty of new standard format efforts (and new efforts seem to launch every other day), there hasn’t been a best practice decision made by the key influencers (eg. FriendFeed – a bigger stream aggregator) and developers to force the issue.

This could be solved by a large-scale web services proxy, but it would be better if format proxying was only an interim solution.

Tags:
  • http://www.techcrunch.com michael arrington

    reminds me of my post on the Centralized Me.

    http://www.techcrunch.com/2008/03/30/friendfeed-the-centralized-me-and-data-portability/

    Somehow FriendFeed is now at the very center of the data standards and portability debate.

  • http://www.techcrunch.com michael arrington

    reminds me of my post on the Centralized Me.

    http://www.techcrunch.com/2008/03/30/friendfeed-the-centralized-me-and-data-portability/

    Somehow FriendFeed is now at the very center of the data standards and portability debate.

  • http://www.heychinaski.com Tom Martin

    The issue with requiring FriendFeed to adopt and support an open format is that a standardization of this service would completely remove their value add. The development time and infrastructure that FriendFeed have contributed would become worthless when the entire problem could be solved by a simpler Google Reader type app that only needs to parse one format.

  • http://www.heychinaski.com Tom Martin

    The issue with requiring FriendFeed to adopt and support an open format is that a standardization of this service would completely remove their value add. The development time and infrastructure that FriendFeed have contributed would become worthless when the entire problem could be solved by a simpler Google Reader type app that only needs to parse one format.

  • Peter

    seems odd to cover this on TCIT.

  • Peter

    seems odd to cover this on TCIT.

  • http://www.techcrunch.com/2008/06/26/myspace-opens-up-the-data-pipe-with-launch-of-data-availability/ MySpace Opens Up The Data Pipe With Full Launch Of Data Availability

    [...] data portability requires constant syncing of data so that the users remain in control. But until real standards emerge on just how to do that (and there are some big hurdles), MySpace’s approach seems more than reasonable. This is a [...]

  • http://www.techcrunch.com/2008/06/26/myspace-opens-up-the-data-pipe-with-launch-of-data-availability/ MySpace Opens Up The Data Pipe With Full Launch Of Data Availability

    [...] data portability requires constant syncing of data so that the users remain in control. But until real standards emerge on just how to do that (and there are some big hurdles), MySpace’s approach seems more than reasonable. This is a [...]

  • http://www.techcrunchit.com/ Nik Cubrilovic

    Peter – we will bring it all together over time

  • http://www.techcrunchit.com/ Nik Cubrilovic

    Peter – we will bring it all together over time

  • John

    Peter,

    I think this is a great article for techcrunchit. It goes more into the technical nuances that techcrunch does not cover.

    Great coverage guys. I just wish the article had a little more depth. You cant kick off a conversation like this and leave it in its current state.

    I imagine you read a few articles, papers or books for this story. Linking to a few of those could help spawn a more complete, interesting and educated discussion!

    Just my two cents.

    Keep up the great work guys.

  • John

    Peter,

    I think this is a great article for techcrunchit. It goes more into the technical nuances that techcrunch does not cover.

    Great coverage guys. I just wish the article had a little more depth. You cant kick off a conversation like this and leave it in its current state.

    I imagine you read a few articles, papers or books for this story. Linking to a few of those could help spawn a more complete, interesting and educated discussion!

    Just my two cents.

    Keep up the great work guys.

  • http://www.techcrunchit.com/ Nik Cubrilovic

    “Great coverage guys. I just wish the article had a little more depth. You cant kick off a conversation like this and leave it in its current state.”

    Definitely not, just kicking off an ongoing conversation..

    I will put more related links into posts, this came up in a real-world conversation. Thanks for the feedback!

  • http://www.techcrunchit.com/ Nik Cubrilovic

    “Great coverage guys. I just wish the article had a little more depth. You cant kick off a conversation like this and leave it in its current state.”

    Definitely not, just kicking off an ongoing conversation..

    I will put more related links into posts, this came up in a real-world conversation. Thanks for the feedback!

  • http://www.krisvandenbergh.be Kris

    Good article! This is going to be a real challenge for the semantic web. We must really structure data in order to make it relevant, but where do you draw the line? We have Atom feeds, which are very widely used, but what about microformats? We have a lot of them! There is hCalendar, XFN, hCard Dublin core, to name some. Besides, they are usually used within a website and not really in external data feeds. How far can you standardize?
    What I also believe, is that we need intelligent systems who are able to recognize the structure of a whole document, and not justs parts of it that have some MediaRSS in it. Can we standardize a format that allows you to specify a type of web service? How to distinguish a blog from a status service or bookmarking service?

    Interesting discussion I think.

  • http://www.krisvandenbergh.be Kris

    Good article! This is going to be a real challenge for the semantic web. We must really structure data in order to make it relevant, but where do you draw the line? We have Atom feeds, which are very widely used, but what about microformats? We have a lot of them! There is hCalendar, XFN, hCard Dublin core, to name some. Besides, they are usually used within a website and not really in external data feeds. How far can you standardize?
    What I also believe, is that we need intelligent systems who are able to recognize the structure of a whole document, and not justs parts of it that have some MediaRSS in it. Can we standardize a format that allows you to specify a type of web service? How to distinguish a blog from a status service or bookmarking service?

    Interesting discussion I think.

  • http://blog.gnipcentral.com/2008/07/01/the-what-of-gnip-changing-apis-from-pull-to-push/ Gnip: We got $h*t to pop » The WHAT of Gnip: Changing APIs from pull to push

    [...] For a great overview of the situation Gnip is plunging into, checkout Nik Cubrilovic’s post on techcrunchIT; “The New Datastream Aggregators, FriendFeed and Standards.”. [...]

  • http://blog.gnipcentral.com/2008/07/01/the-what-of-gnip-changing-apis-from-pull-to-push/ Gnip: We got $h*t to pop » The WHAT of Gnip: Changing APIs from pull to push

    [...] For a great overview of the situation Gnip is plunging into, checkout Nik Cubrilovic’s post on techcrunchIT; “The New Datastream Aggregators, FriendFeed and Standards.”. [...]

  • http://www.techcrunch.com/2008/07/01/gnip-launches-to-ease-the-strain-on-web-services/ Gnip Launches To Ease The Strain On Web Services

    [...] a fuller description of how Gnip works, see the full overview at TechCrunchIT and this discussion on datastream [...]

  • http://www.techcrunch.com/2008/07/01/gnip-launches-to-ease-the-strain-on-web-services/ Gnip Launches To Ease The Strain On Web Services

    [...] a fuller description of how Gnip works, see the full overview at TechCrunchIT and this discussion on datastream [...]

  • http://jp.techcrunch.com/archives/20080701gnip-launches-to-ease-the-strain-on-web-services/ TechCrunch Japanese アーカイブ » ウェブサービスの負担を軽減するGnip

    [...] Gnipの動作に関するより詳しい情報はTechCrunchITの概観記事および、データストリームアグリゲーションに関するこちらの議論を参照して欲しい。 [...]

  • http://jp.techcrunch.com/archives/20080701gnip-launches-to-ease-the-strain-on-web-services/ TechCrunch Japanese アーカイブ » ウェブサービスの負担を軽減するGnip

    [...] Gnipの動作に関するより詳しい情報はTechCrunchITの概観記事および、データストリームアグリゲーションに関するこちらの議論を参照して欲しい。 [...]

  • http://www.DesireMesh.com/2008/09/gnip-making-data-portability-suck-less-or-more-compelling/ Desire Mesh » Blog Archive » GNIP – making data portability suck less (or more compelling?)

    [...] to easily access user data from a variety of sources. Currently an application consuming user data must write-in direct support for each API for every service it requires, which usually requires a lot of development effort in terms of [...]

  • http://www.DesireMesh.com/2008/09/gnip-making-data-portability-suck-less-or-more-compelling/ Desire Mesh » Blog Archive » GNIP – making data portability suck less (or more compelling?)

    [...] to easily access user data from a variety of sources. Currently an application consuming user data must write-in direct support for each API for every service it requires, which usually requires a lot of development effort in terms of [...]

  • http://clicaaqui.com/contextualizando-tendencias-em-aplicacoes-sociais-para-2009/ Clica Aqui » Contextualizando : Tendências em Aplicações Sociais para 2009

    [...] em informações demográficas imprecisas ou falsas nas redes sociais; d) a consagração dos agregadores de conteúdo, da inteligência coletiva e das conversações on-line como uma mídia; e) a popularização das [...]

blog comments powered by Disqus