Gnip Brings Data Portability To Web Services

Next Story

Gnip Launches To Ease The Strain On Web Services

Gnip is a new web services platform being launched today by the former founders of MyBlogLog (see previous Techcrunch announcement of Gnip here). The goal of Gnip is to serve as a web services proxy to enable consuming services to easily access user data from a variety of sources. Currently an application consuming user data must write-in direct support for each API for every service it requires, which usually requires a lot of development effort in terms of protocol and format support, maintenance of each API integration and other issues such as uptime and availability. With Gnip, developers can write once for the Gnip API and gain access to user data for a number of supported publishing services.

Starting this morning, Gnip has support for publishers such as Digg, Flickr, del.icio.us, MyBlogLog, Six Apart and many more. Gnip have established a formal relationship with some of these providers, such as Digg, so that they are able to retrieve all data and make it available to consuming applications using Gnip. In the case of Digg, this involved allowing Gnip unlimited access to their data API. Current consumers of the service include Plaxo, Lijit, Spokeo and MyBlogLog. Gnip is launching with either publisher or consumer partnerships or support for 15 different products.

A publisher can either push data to Gnip using their API’s, or Gnip can poll the latest user data. For consumers, Gnip offers a standards-based API to access all the data across the different publishers. A key advantage of Gnip is that new events are pushed to the consumer, rather than relying on the consuming application to poll the publishers multiple times as a way of finding new events. For example, instead of polling Digg every few seconds for a new event for a particular user, Gnip can ping the consuming service – saving multiple round-trip API requests and resolving a large-scale problem that exists with current web services infrastructure. With a ping-based notification mechanism for new events via Gnip the publisher can be spared the load of multiple polling requests from multiple consuming applications.

On the publishing side Gnip supports formats such as standard RSS, Atom or XMPP. A consuming application can pick up a number of the Gnip-supplied client library toolkits (written for Php, RubyOnRails, Java etc.) and hook into the Gnip API and receive notifications of new events or request data. With Gnip as a proxy, data formats can be normalized and standardized, with load being taken off of the publishing source and instead placed on Gnip and its event notification service.

What Gnip are doing for web services is very similar to ping services for weblogs, except that multiple formats and types of services are supported. A service such as Twitter can easily offload almost all of their API traffic to Gnip and consuming applications can make use of a notification-based web service rather than a polling-based service. With a notification based service, clients can be informed of user updates within seconds, rather than minutes or hours – meaning more up-to-date information and all via a standards-based interface.

For output, Gnip will support Atom initially, with plans to extend support to XMPP and other formats in the near future. Also in the pipeline is support for private data feeds using oAuth as the authentication mechanism – a problem that is hard to solve across multiple data streams but one that once solved will be implemented within Gnip. Future services via Gnip include transforming data types and providing an identity services, both features which are due within the next few months.

This off course depends on Gnip being able to handle the load and traffic for all such services, and because it is notification based rather than polling-based, they already cut a large number of the API calls being used today. Gnip aims to act as a proxy for web services traffic, allowing the publisher to push data once to Gnip, from where it is either pushed or polled by the various consumers. While today Gnip only supports public data streams, the service already has a good level of support both amongst publishers as well as consumers and offers a compelling value proposition for both (eg. imagine Gnip taking care of API traffic for twitter). The initial version of Gnip was developed in co-operation with Pivotal Labs, who are well known for their app development experise (and who have recently been bought in at Twitter)

In an ideal world, all services would support standard formats and protocols such as XMPP, and there would be no requirement for proxies. Gnip not only offers the standard interface, but also provides other services to publishers and consumers which make it an almost invaluable part of the new web services infrastructure.

More coverage at Techcrunch

  • http://blog.programmableweb.com/2008/07/01/data-portability-and-pushability-with-gnip/ Data Portability and Pushability with Gnip

    […] can be thought of as an ‘activity bridge’ for the social web, or as Nik Cubrilovic of TechCrunchIT put it, doing for social activity what pinging services have done for blogs. Except that the activities […]

  • bong

    Dosnt this break ssl as now the user will not have end to end security?

  • bong

    Doesn’t this break ssl? There is no end to end security

  • http://www.gnipcentral.com Eric Marcoullier

    Bong, it’s only public data. What do you need ssl for?

  • http://www.rss-blogger.de/blog/2008/07/gnip-zentralbahnhof-oder-universal-ubersetzer-furs-web20/ Gnip - Zentralbahnhof oder Universal-Übersetzer fürs Web2.0? :: RSS Blogger

    […] techCrunchIT, Ankündigung im März auf techCrunch, Gnip und Twitter, Gnipper im Interview, direkter Link zum Interview (MP3) […]

  • http://fr.techcrunch.com/2008/07/02/le-fondateur-de-mybloglog-lance-gnip-pour-faciliter-la-portabilite-des-donnees/ TechCrunch en français » Le fondateur de MyBlogLog lance Gnip pour faciliter la portabilité des données

    […] Vous pouvez découvrir une explication plus complète sur TechCrunchIT […]

  • http://jp.techcrunch.com/archives/20080701gnip-launches-to-ease-the-strain-on-web-services/ TechCrunch Japanese アーカイブ » ウェブサービスの負担を軽減するGnip

    […] Gnipの動作に関するより詳しい情報はTechCrunchITの概観記事および、データストリームアグリゲーションに関するこちらの議論を参照して欲しい。 […]

  • http://bbook.com Dan

    A step up from Mashery I would think, though what will be done about more complex APIs, or private ones? Support?

  • http://www.techcrunchit.com/ Nik Cubrilovic

    Dan: checkout the video we did with them in the next post, we talk about private data and a lot more. I think Mashery is more focused on providing an out-of-the-box dev community platform, whereas Gnip are a proxy for these services.

  • http://www.strategie-influence.com/le-fondateur-de-mybloglog-lance-gnip-pour-faciliter-la-portabilite-des-donnees-961/ Le fondateur de MyBlogLog lance Gnip pour faciliter la portabilité des données

    […] Vous pouvez découvrir une explication plus complète sur TechCrunchIT […]

  • http://blog.broadbandmechanics.com/2008/07/blogging-on-july-11-2008 Marc’s Voice » Blog Archive » Blogging on July 11, 2008

    […] Gnip seems really coolio […]

  • http://www.DesireMesh.com/2008/09/gnip-making-data-portability-suck-less-or-more-compelling/ Desire Mesh » Blog Archive » GNIP - making data portability suck less (or more compelling?)

    […] the TechCrunchIT excerpt puts it, the BEFORE GNIP and AFTER GNIP might as well be the difference between NIGHT and DAY: The […]

  • http://www.insidefacebook.com/2010/06/30/gnip-facebook-apis/ Gnip Standardizes and Reduces Strain on Facebook APIs

    […] Gnip acts as a proxy between content publishers like Facebook and Flickr, and services that utilize API data, like FriendFeed and Plaxo. Normally, these data consumers would have to expend resources writing and maintaining code to interface with each publisher’s APIs. Then, to keep their own sites up-to-date, they’d have to incessantly query each API to occasionally find a change, such as a new photo being uploaded. Gnip simplifies this process by standardizing many of the most popular APIs so developers only have to support the one overarching Gnip API. Then whenever data changes on the publisher’s site, it notifies Gnip, which then pushes the data to the consuming service, eliminating needless API calls. For a more technical walkthrough of API aggregators, see TechCrunch IT’s in-depth guide to Gnip. […]

  • http://www.tjoozey.com/2010/06/30/gnip-standardizes-and-reduces-strain-on-facebook-apis/ Gnip Standardizes and Reduces Strain on Facebook APIs » Gnip, Facebook, APIs, Then, Pages, Gnips » App Developer Tyler Johnson Blog - tjoozey.com

    […] Gnip acts as a proxy between content publishers like Facebook and Flickr, and services that utilize API data, like FriendFeed and Plaxo. Normally, these data consumers would have to expend resources writing and maintaining code to interface with each publisher’s APIs. Then, to keep their own sites up-to-date, they’d have to incessantly query each API to occasionally find a change, such as a new photo being uploaded. Gnip simplifies this process by standardizing many of the most popular APIs so developers only have to support the one overarching Gnip API. Then whenever data changes on the publisher’s site, it notifies Gnip, which then pushes the data to the consuming service, eliminating needless API calls. For a more technical walkthrough of API aggregators, see TechCrunch IT’s in-depth guide to Gnip. […]

blog comments powered by Disqus