Gnip, a provider of social media data to enterprise applications, is now delivering over 30 billion paid social activities per month to its customers. This is the largest number of paid social media activities that have ever been distributed in a 30 day period in Gnip’s history.

For background, Gnip serves as an API hub for social streams, collecting data from services like Twitter, Facebook and other social sources, and pushing it out to other data-consuming applications and Websites. Applications using Gnip’s platform can get public data streams for over 100 feeds and sites, including Twitter, Facebook, YouTube, WordPress, Flickr, and now Google+ without ever visiting those sites or accessing their individual APIs. And Gnip also has premium access to the Twitter firehose of data.

For basis of comparison, at the start of 2011, Gnip was delivering 300 million activities per month. By May, that number was up to 3 billion activities per month. And in October, Gnip delivered 30 billion activities. At this rate, Gnip says it will be be delivering 300 billion
activities per month by March of next year.

So what’s causing this increase in demand for social data? Gnip says that more customers are signing up to tap into and mine the social data stream. Second, Gnip says that there is major interest in the premium-access to the Twitter firehose. Hedge funds are using the data stream to drive trading strategies, business intelligence companies layering social data onto their existing structured data sources and more. And more customers are using multiples sources to enrich their product capabilities.

Gnip’s CEO Jud Valeski explains that Gnip will be adding Twitter-like premium data source relationships with other social services in the near future. The company also plans to increase the ability for more complex filtration of social data streams.

Editor’s note: In this guest post, serial entrepreneur Nova Spivack gives Twitter some suggestions for how to make money. Spivack’s latest startup, Bottlenose, is looking at new ways to mine the Tweet stream.

I’ve been puzzling over Twitter’s recent tactical moves around their API, Ubermedia and Tweetdeck, for a few months now, and it just doesn’t add up. In fact I think Twitter’s current strategy may take them in a direction where they end up missing out on their biggest potential win.

If Twitter continues to go down the media company path, without incorporating their API into the plan, that could not only force a large part of their ecosystem to go elsewhere, but it could deprive them of a much larger potential infrastructure revenue opportunity, and could even end up costing them the company.

After all, Silicon Valley is littered with the  burned out wreckage of once-great media companies that failed create and keep third-party app ecosystems: AOL, Friendster, MySpace, Yahoo – to name a few. It’s very hard to maintain leadership as an online media company without an ecosystem of outside apps increasing reach, innovation, and stickiness.

In light of this, I’ve been exploring an alternate path for Twitter that leverages their API in a much bigger way, and this path appears to be a better strategy. According to my own experimental revenue  projections for Twitter, this alternative path is not only a good tactical move, but it’s a good business move because it increases Twitter’s reach, number of active users, and revenues massively.

This path fulfills the promise of Twitter as an infrastructure, without sacrificing the media company play. A media company + an infrastructure is a much stronger strategic position to be in than either on their own.

Another side-effect of this proposal is that it eliminates the need for Twitter to buy Tweetdeck, or Ubermedia. It makes the wholediscussion about the risk of Tweetdeck and Ubermedia to Twitter completely irrelevant, a non-issue, and will save Twitter $50 million in unnecessary acquisition costs.

It also eliminates the tension between Twitter and their ecosystem of third-party client apps. And it returns more revenues to everyone, especially Twitter. In the end, this could make Twitter a much bigger and more important company, and would certainly lock in their dominance of global realtime messaging and advertising.

To understand my proposal, first, what is Twitter really? Well, if history is any indication, it’s a messaging infrastructure for the Web. Let’s shelve the question of whether it’s the optimal messaging infrasture (it’s not, by a long-shot), but it works well enough for the moment.

Twitter’s APIs are a big piece of how Twitter grew so quickly: Twitter surged because of third-party developers pumping data in and out of Twitter via these APIs in all manner of apps and services, which massively extend the reach, innovation, and impact of Twitter.

Instead of abandoning their DNA and clamping down on API use and competing with their own ecosystem, my analysis shows that Twitter would do far better through a combined strategy.

In the combined strategy Twitter would continue to have a destination portal and their own official apps, but would also actively encourage – and monetize – an ecosystem of third-party apps on their APIs, including client apps that effectively competed with their destination. This competition would however not harm Twitter, it would make the ecosystem even bigger, and would deliver very significant incremental revenues to Twitter as well.

The key to the combined strategy is a new way for Twitter to monetize their API’s. Let’s call this the “freemium API” option. Here’s how it works conceptually:

Twitter would change their API terms to give third-party apps two choices: Either use the API for free but accept in-stream ads from Twitter, or pay a very nominal fee per tweet (around $0.1 per thousand tweets in or out of the API, a 10 cent CPM). Apps that opt to pay for the premium API could easily monetize with their own ads or subscriptions to more than compensate for the 10 cent CPM to Twitter, and would make money on the delta. Even if you think that’s too high, Twitter could cut that in half and still make money.

Here’s the model in a little more detail:

1. Third-party apps that don’t mind carrying Twitter’s ads could use the free API. They would be able to run their own ads outside the stream, but not inside the stream in their apps – only Twitter’s ads could appear inside the stream for the free API. These ads would come from Twitter and could even be personalized or targeted per user or topic.
2. Third-party apps that either don’t want ads at all, or don’t want Twitter’s ads, could use the premium API, pay the fee, monetize, and make money on the spread. They could monetize with their own ads or through subscription models or commerce or whatever they want. In this option, third-party apps would not be allowed to inject their own ads into the outgoing stream; instead they could display their own ads interleaved within the stream, in their user-interfaces, but these ads would not be pushed out to Twitter, they would only appear for their own users. This way Twitter would not be flooded with ads.

By launching this freemium API model, in addition to their existing portal business and their official client apps, Twitter would be able to monetize their entire ecosystem, including every third-party app. The beauty is: Twitter gets paid no matter where a user enters their network or views content; Twitter makes money from 100% of all tweets and views. It’s a vastly more scalable business model than just being a destination or media company and trying to own 100% of the user-experience.

As an experiment, I’ve run the numbers and they look good; See for yourself. Of course these numbers are based on anecdotal data, such as a rumor I heard from a credible source that Twitter’s actual revenues this year are closer to $75mm. It could add another $20 million off the bat with a freemium API strategy.

My projections simplify matters in several dimensions for the sake of convenience in sketching out the scenarios, and perhaps have growth rate and audience share assumptions that are debatable – but regardless, even if we were to tweak the model a bit, the conclusion is the same: Twitter would have a bigger audience and greater revenue growth if they included the Freemium API model and made it a priority.

Interestingly, Twitter currently licenses all of its bulk data through a third-party company, Gnip. Gnip prices their data at $0.0001/tweet or $0.1/1K tweets – exactly what I proposed in my model. Instead of that money going to Twitter, some or all of it is presently going to Gnip. This makes very little sense to me.

Why would Twitter give away their API – their platform – to an outside company, especially when at its root Twitter is an API? I think it would ultimately make more sense to take that in-house, and if I were Gnip I would be worried about that. Perhaps Gnip is an acquisition target by Twitter in the future? In fact, Gnip is a much bigger potential threat for a company like Twitter than Tweetdeck or Ubermedia are, in my opinion.

So far we’ve analyzed what happens if Twitter DOES take the strategy of offering a freemium API. But what happens if they DON’T? Either of two sub-optimal outcomes:

• If Twitter allows 3^rd party clients but does not monetize the API in any way – then eventually 3^rd party clients will take significant market share away from them. This is the problem they are facing with Tweetdeck and Ubermedia currently.
• If Twitter tries to stop (A) by blocking or clamping down on 3^rd party clients – it won’t work. First of all this will cause existing 3^rd party client apps to leave the Twitter network, taking large portions of high-value power-users with them. There are numerous stealth projects now underway to create alternative networks to Twitter, and sooner or later one of these will succeed. More importantly, if Twitter blocks use of their API they will cut themselves off from being a platform and infrastructure, making them vulnerable to attack by competing services (like Facebook or Google) that might be more developer friendly and that take more of a platform approach.

The conclusion of this is that it is clear, to me at least, that if Twitter turns its back on their platform and API DNA, they are missing out on what may be their most important tactical opportunity.

Being a platform and having thousands of 3^rd party apps will increase their reach, massively increase adoption and engagement, and create a much more powerful and sticky network-effect. In short, killing their own ecosystem to save their portal business would be cutting off their nose to save their face.

So what should Twitter do? Simple. They should not buy Tweetdeck or Ubermedia. There is no need to worry about Ubermedia or anyone else. They should not clamp down on their API or try to block third-party client apps.

Twitter could solve all these problems, and double the value of their business, in an instant by simply launching a freemium API, along the lines of what I’ve proposed here.

If Twitter simply embraced their API roots instead of turning against them, all their “frenemies” would become friends again, and Twitter could focus on building the best realtime ad network and messaging infrastructure in the world, instead of competing with their own channel partners.

If Twitter doesn’t do this, then mark my words, they will eventually lose their dominant role, as well as all the goodwill they currently have. And they will force the market to come up with competing solutions.

At the end of the day, without an ecosystem, Twitter’s network effect will fall apart pretty quickly. If Twitter loses their ecosystem by competing with it, they will end up in the graveyard of once-great Internet companies. I personally would not like to see that happen.

I would like to see Twitter function as an infrastructure, not merely a media company. It’s better for Twitter, it’s better for their ecosystem, and it’s better for the world. But if they fail to do that, I’ll happily embrace better solutions when they emerge.

Let me be clear, going open source is a great idea, I’m not sure if it will help Tr.im all that much, but on paper it sounds great. That’s what they should have done originally. But in a post today on Tr.im’s blog the service first apologizes for this whole fiasco, and then attempts to place blame elsewhere. As I read it, it’s either Bit.ly’s fault for making a low-ball offer to buy the Tr.im, Twitter’s fault for picking Bit.ly over Tr.im as its URL shortener of choice, 301works.org‘s fault for being a “public relations stunt”, and yes, even TechCrunch’s fault because we “simply repeat vertbatim what twitter/bit.ly feeds [us]“.

Let me again be clear: We received no shortage of tips from very good sources last week about what Tr.im was doing behind the scenes while all of this played out. Not one of those tips was from either Bit.ly or Twitter or anyone directly related to them. Instead, they came from third-parties who were actively or passively pursuing a Tr.im acquisition. For example, we heard the $80,000 to $100,000 figure Nambu wanted for Tr.im from no less than three sources. And even more sources came forward to say they had heard that general price range as well and thought it was unreasonable, especially considering how Nambu handled the Tr.im situation, losing user trust in the process. So is it really any surprise today that they announce they’re going open source?

Does it suck that Twitter’s choice of Bit.ly made it hard for services like Tr.im to operate? Sure. But plenty of others are still out there doing it rather than descending into conspiracy theories. And it’s just poor form to drag other services into this mess, like Gnip, which is trying to do a good thing with 301works.org. Yes, it was Bit.ly’s idea, but Gnip is now handling it as a independant third-party, and no shortage of other URL shortening services have joined on. Obviously, some of those services probably don’t like that Twitter chose Bit.ly, but they deal with it and realize that a movement like this is worth teaming up with rivals for the good of all of them.

“It saddens me that this is how Tr.im is portraying 301works,” Gnip’s Eric Marcoullier told us today. He went on to reiterate that Gnip is simply serving as an independant third party for the project, attempting to do something good and useful. And while a ReadWriteWeb article on the matter today seemed to suggest that Tr.im was working actively with Gnip for this new crowd-sourced Tr.im, Marcoullier notes that anyone is free to push data through Gnip, but that he hasn’t specifically talked to Nambu about how Tr.im will use it. He also notes that since Tr.im will now be open source, he hopes the community behind it makes its data a part of 301works.

We have reached out to Tr.im to try and clear up their misplaced anger about this, but have yet to hear back. If we do, we will update.

Perhaps you heard about 301works in one of our recent pieces about how Bit.ly was attempting to salvage the Tr.im wreckage. The idea was the 301works would be a centralized hub for all shortened URLs, not run by any one URL-shortener. Tr.im balked at the idea of joining, but plenty of others are, including Bit.ly, Awe.sm, Adjix, betaworks, Cligs, and URLizer. All of them are teaming up with Gnip to launch this project.

One of the holdups in Tr.im’s participation was that it didn’t want one company ruling all of this data. And while Gnip will be handling it at first, to get the project off the ground, the plan is still to find a non-profit group to manage 301works. All the members are clear that they want this to be an open-source project that sets users’ minds at ease about using URL-shorteners.

The service will launch sometime in the next few weeks, after the participating companies have a chance to tell their users that they will be backing up their links on 301works. While most are unlikely to have a problem with that, some might, so they’re giving them some time to opt-out.

So how will 301works actually work? Well here are the key points for how companies will be able to back up their links:

• URL shortening services decide the frequency that they will make updates.
• URL shortening services decide how their updates can be made available to the public. Some services will provide regular uploads and downloads (hourly, daily, weekly, etc) and some will opt for a pure archival approach.
• Gnip is providing the infrastructure service to support aggregating data from URL shortening services. Gnip will provide the infrastructure service to compress the data into pre-defined download options for end users.
• Companies will be able to submit data via a REST API using HTTP POST over SSL. In addition, Gnip can provide other approaches on request.

All of this sounds great on paper, but the question of just how well this system works remains to be seen. Still, it’s promising that we’re seeing a bunch of companies take action on this so quickly after an incident that left a lot of people concerned about the future of URL-shorteners.

The Web is speeding up and Gnip wants to help push it along. Today, the API aggregation platform is releasing its own Push API which lets any site patch together its own version of Friendfeed or Twitter-like data stream. Gnip will be speaking at TechCrunch’s Real-Time Stream CrunchUp tomorrow on the Real-Time Business panel.

Gnip lets data-consuming services like Plaxo that take data from other services (like Twitter, Facebook Friendfeed, Digg, Delicious, etc.) collect data from requested users pushed to them. Data consumers using Gnip’s platform can get public data streams for over 30 social media networks and sites, including Twitter, Digg, Delicious, YouTube, WordPress, Flickr, Six Apart and others without ever visiting those sites or accessing their individual APIs.

The new push service lets companies filter and white-label the stream so the technology is fully integrated into the business’ infrastructure. Companies list out the most common data requests that are made on their APIs and websites and Gnip will collect the relevant data and deliver it in real-time to any approved third-party. For example, a travel website like Expedia or Kayak may use Gnip’s service to track and deliver real-time information on how customers are interacting with airline deals to the vendors that are listing flights on their site, like American Airlines or Delta. The real-time capabilities would let a travel site analyze real-time data and syndicate changes in fare sales immediately.

Gnip is also launching a early-stage startup partner program that will let startups access to all of Gnip’s service features and data services. The program is aimed towards software development startups that have been in business for less than 3 years and generating less than $200,000 in revenue. Of course, Gnip requires that partners pay a fee of $1000 but says the services that they will receive are valued at $10,000 per month. Dipity, Toobla and Klout are all pilot partners of the new program.

Gnip, a platform that helps move data around from one social network to the next, is now integrated with Facebook so that the platform can access data via Facebook’s recently launched open API stream.

Gnip lets data-consuming services like Plaxo that take data from other services (like Twitter, Friendfeed, Digg, Delicious, etc.) collect data from requested users pushed to them. Data consuming services are no longer required to build pollers for any of the publishers pushing data into Gnip, they just give Gnip an endpoint and they push the data to them in real time. With Gnip’s Facebook integration, developers and data collectors can choose the specific Facebook users from among those that have authorized their applications and then Gnip will immediately begin collecting the relevant data, normalize it and deliver it in real-time to the developer’s separate applications.

Data consumers using Gnip’s platform can also get public data streams for over 30 social media networks and sites, including Twitter, Digg, Delicious, YouTube, WordPress, Flickr, Six Apart and others without ever visiting those sites or accessing their individual APIs, subject only to the terms of service of those networks. Gnip also offers a number of filter options to allow data consumers the ability to create rules based queries based on tags, keywords, etc.

There is something about great sales people or deal makers that is entirely social. They are connectors, as Malcolm Gladwell calls them—people who know the interests, skills, and needs of everyone in their social or business circle and connects them together. If you are really good at this, like Sidney Weinberg (a legend who helped build Goldman Sachs), you are a super-connector.

Zentact has the modest goal to help you become a super-connector. It has a long way to go before it can do that. But it is starting with the kernel of something that is intriguing. At its core, Zentact is a browser add-on (for Firefox only right now) that helps you read the Web with the interests of your social network in mind. If you want to try it out, we have 500 invites for the private beta (but once you are in, you can invite as many people as you want by sending them a message through Zentact).

Here is what is supposed to happen once you have Zentact all set up. Reading an article about black labs? A box pops up to remind you that your co-editor loves black labs and lets you email him the article with a note right from that page. Run across a blog post that mentions a contact’s company? Same thing happens. You can forward that article or link and make yourself look thoughtful in the same way that setting up automatic birthday reminders in your calendar or Amazon makes you look like you went the extra effort to remember someone’s birthday.

“This is basically a karma points system,” says investor and adviser Eric Marcoullier, the CEO of Gnip. Marcoullier and Zentact co-founders John Sampson and Jared Brandt are all MyBloglog refugees. They founded Zentact with a few hundred thousand dollars. (Howard Lindzon is their biggest angel investor).

Before you can start using Zentact, first it needs to ingest all of your contacts from Gmail, Hotmail, or Yahoo Mail. Then—and this is its weak point—you have to tag each contact with their interests, company affiliations, and whatnot. That right there almost makes it a non-starter for me. Some people, I realize, will do this obsessive tagging (sales people, recruiters, biz dev types), but not most people. I have 2,318 contacts in Gmail. There is no way I am going to go through and tag those. It would be much better if Zentact could simply get all of this information from one of my existing social networks like Facebook or LinkedIn, which already know all of my contacts’ interests and affiliations.

The software, however, does have a saving grace. Whenever you email someone in Gmail, a Zentact box opens up right in the email below the address field where you can add tags and other data in a piecemeal fashion. That makes the process a lot less daunting, and lets you focus on the people you actually are in contact with the most frequently.

Once that is set up, then the Zentact box is supposed to pop up whenever it reads a tag contextually on a page and tell you which contacts you might want to email the link with a thoughtful note. It acts a a surrogate super-connector brain in that way. Explains Marcoullier:

The vast majority of people get information, they process it for internal use, and then move on. Why not build a plugin that reads what you are reading and alerts you to the fact that it might be relevant to someone in your network?

That could be helpful or annoying depending on how often it pops up. You can moderate that based on the tags you use. The actual mechanics of Zentact are still a little bit buggy in my experience (although that could just be my dying computer or the fact that I imported way too many contacts). But this is a rough beta and there is a lot of promise here.

Once you’ve tagged a bunch of contacts, then you can click on each tag at Zentact’s Website and see all the contacts who share a certain interest. This could be great for organizing soccer games or biking trips or a startup, or that matter. It all depends on how good is your personal tag database. How well do you know your contacts and what they are good at?

Zentact will soon add the ability to actually connect like-minded contacts to each other, which is really what super-connectors do. When it does that and can automatically tag your contacts for you, then I’ll really be excited about it. But what I really want is for my existing social networks to do this for me.

The company acts as a clearing house for social content, easing the load on content distributors like Digg, Twitter, Delicious and Six Apart. Content consumers like Plaxo and MyBloglog benefit from a single endpoint and a standardized way of accessing data. In short, it unclogs the plumbing.

TechCrunchIT spoke with the Gnip founders on video immediately after launch. In September they launched version 2.0 of the service, and discussed their business model.

Gnip, the guys that are helping move data around from one social network to the next, launched v 2.0 of the service tonight.

The new version of the service allows data consumers (services like Plaxo that take data from other services, like Twitter, Friendfeed, Digg, Delicious, etc.) to have data from requested users pushed to them. It’s no longer “Hey, TechCrunch just tweeted. Go query the API to get the data.” Now it’s “TechCrunch just tweeted – here’s the data.” Data consumers are no longer required to build pollers for any of the publishers pushing data into Gnip, they just give Gnip an endpoint and they push the data to them in real time.

Data consumers can get complete public data streams for Twitter, Digg, Delicious, Six Apart and others without ever visiting those sites or accessing their individual APIs, subject only to the terms of service of those services. And this data can be gathered via a REST-based API or the newly launched XMPP support.

Gnip also added a number of filter options to allow data consumers the ability to create rules based queries based on tags, keywords, etc.

Gnip’s business model is freemium – lots of data for free and commercial data consumers pay when they go over certain thresholds (non commercial use is free). The model is based on the number of users and the number of filters tracked. Basically, any time a service is tracking more than 10,000 people and/or rules for a certain data provider, they’ll start paying at a rate of $0.01 per user or rule per month, with a maximum payment of $1,000 per month for each data provider tracked. For now billing is turned off and the service remains completely free. Thirty to sixty days from now, people will start to pay.

Since launching TechcrunchIT we have been pounding away about open standards and data availability. One of the biggest victims of this focus recently has been Twitter, who went from being a leading light in the field, to closing everything up and now finally today have gone back to being open again. Gnip has announced this morning that they now have access to the Twitter XMPP feed, and that they will be re-publishing it and making it available to everybody.

We first wrote about Gnip when they launched. At the time they supported a large number of data providers but the obvious missing provider was Twitter, who had at the time shut off access to their XMPP ‘firehose’.

This is an important move for a number of reasons. First, it allows users to access their data from any other application including IM – its no longer up to Twitter in terms of what can be done with the data, which should result in some interesting applications and mashups. Second it better establishes XMPP as the standard for communications messages on the web. Previously applications integrating with Twitter were polling its API and checking for changes every x minutes, much like RSS. You only have to look at the widget in the sidebar of this very blog to see the problems with that – the requests are throttled, data is dropped and it results in 98% of traffic to a server not actually carrying any data. With XMPP and the push model, all that is redundant. XMPP is the HTTP for communications on the web, and soon we will look back at the days of polling a web service and wonder what the hell we were smoking.

Last of all, this is a perfect case study for the next applicaiton built in terms of what to do and what not to do. One of the biggest drivers of user adoption at Twitter was the large number of applications that were available to give users access to the data. This opens up user choice and a market around a platform, and the only way a firmly established platform can be displaced is either that a better solution comes alone, or more likely, the platform owners theselves screw it up by demanding too much from users.

What does this mean for the average Twitter user? It means that more third party services will start to work better. Today, other than a handful of services like Summize (which was just acquired by Twitter) and Friendfeed, third party apps must talk to Twitter via their normal APIs. Those APIs require applications to send Twitter a request and then get a response. The two way communication creates a big load on Twitter in the aggregate.

With XMPP Twitter just sends out all of their data in a constant stream, whether you ask for it or not. The third party, in this case Gnip, takes the data and parses it for further use.

Gnip acts as an intermediary between applications that create social content and those that consume it. They take the Twitter feed, which is a list of usernames, Twitter status URLs and time stamps, and make it available to any third party that requests it. Both Plaxo and MyBlogLog are already using the new feed, and more partners will add it immediately. And every third party that takes data from Gnip doesn’t have to take it from Twitter, easing the overall load on Twitter’s servers.

For now Gnip is only sending updates for requested users, not the richer data that some applications like Twhirl need to build a Twitter-like desktop environment. Twitter may give Gnip permission to send additional data, like @replies and direct messages, over time (if that last sentence doesn’t mean anything to you, it means you aren’t a crazy-heavy Twitter user, just disregard it).

What this means is that Twitter is taking yet another step towards openness and leaning on outside parties to help them with scaling issues.

Battle Over: Twitter Open Up To Gnip. Read more at TechcrunchIT >>

Gnip is a new web services platform being launched today by the former founders of MyBlogLog (see previous Techcrunch announcement of Gnip here). The goal of Gnip is to serve as a web services proxy to enable consuming services to easily access user data from a variety of sources. Currently an application consuming user data must write-in direct support for each API for every service it requires, which usually requires a lot of development effort in terms of protocol and format support, maintenance of each API integration and other issues such as uptime and availability. With Gnip, developers can write once for the Gnip API and gain access to user data for a number of supported publishing services.

Starting this morning, Gnip has support for publishers such as Digg, Flickr, del.icio.us, MyBlogLog, Six Apart and many more. Gnip have established a formal relationship with some of these providers, such as Digg, so that they are able to retrieve all data and make it available to consuming applications using Gnip. In the case of Digg, this involved allowing Gnip unlimited access to their data API. Current consumers of the service include Plaxo, Lijit, Spokeo and MyBlogLog. Gnip is launching with either publisher or consumer partnerships or support for 15 different products.

A publisher can either push data to Gnip using their API’s, or Gnip can poll the latest user data. For consumers, Gnip offers a standards-based API to access all the data across the different publishers. A key advantage of Gnip is that new events are pushed to the consumer, rather than relying on the consuming application to poll the publishers multiple times as a way of finding new events. For example, instead of polling Digg every few seconds for a new event for a particular user, Gnip can ping the consuming service – saving multiple round-trip API requests and resolving a large-scale problem that exists with current web services infrastructure. With a ping-based notification mechanism for new events via Gnip the publisher can be spared the load of multiple polling requests from multiple consuming applications.

On the publishing side Gnip supports formats such as standard RSS, Atom or XMPP. A consuming application can pick up a number of the Gnip-supplied client library toolkits (written for Php, RubyOnRails, Java etc.) and hook into the Gnip API and receive notifications of new events or request data. With Gnip as a proxy, data formats can be normalized and standardized, with load being taken off of the publishing source and instead placed on Gnip and its event notification service.

What Gnip are doing for web services is very similar to ping services for weblogs, except that multiple formats and types of services are supported. A service such as Twitter can easily offload almost all of their API traffic to Gnip and consuming applications can make use of a notification-based web service rather than a polling-based service. With a notification based service, clients can be informed of user updates within seconds, rather than minutes or hours – meaning more up-to-date information and all via a standards-based interface.

For output, Gnip will support Atom initially, with plans to extend support to XMPP and other formats in the near future. Also in the pipeline is support for private data feeds using oAuth as the authentication mechanism – a problem that is hard to solve across multiple data streams but one that once solved will be implemented within Gnip. Future services via Gnip include transforming data types and providing an identity services, both features which are due within the next few months.

This off course depends on Gnip being able to handle the load and traffic for all such services, and because it is notification based rather than polling-based, they already cut a large number of the API calls being used today. Gnip aims to act as a proxy for web services traffic, allowing the publisher to push data once to Gnip, from where it is either pushed or polled by the various consumers. While today Gnip only supports public data streams, the service already has a good level of support both amongst publishers as well as consumers and offers a compelling value proposition for both (eg. imagine Gnip taking care of API traffic for twitter). The initial version of Gnip was developed in co-operation with Pivotal Labs, who are well known for their app development experise (and who have recently been bought in at Twitter)

In an ideal world, all services would support standard formats and protocols such as XMPP, and there would be no requirement for proxies. Gnip not only offers the standard interface, but also provides other services to publishers and consumers which make it an almost invaluable part of the new web services infrastructure.

More coverage at Techcrunch

MyBlogLog founder Eric Marcoullier sold his company to Yahoo in January 2007 for an estimated $10 million. He left Yahoo in July 2007 with the seed of a new idea germinating in his head – “Make data portability suck less.”

The result of that thinking is Gnip, a new service we first mentioned in March 2008 when they announced seed funding.

Today the details are being revealed and the service is launching. Gnip isn’t a consumer service. Rather, it’s designed to sit in between social networks and other web services that produce a lot of user content and data (like Digg, Delicious, Flickr, etc.) and data consumers (like Plaxo, SocialThing, MyBlogLog, etc.) with the express goal of reducing API load and making the services more efficient.

A close analogy is a blog ping server (see our overview here). Ping servers tell blog search engines like Technorati and Google Blog Search when a blog has been updated, so the search engines don’t have to constantly re-index sites just to see if new content has been posted. Instead, the blog tells the ping server when it updates, which tells the search engines to drop by and re-index. The creation of the first ping server, Weblogs.com, by Dave Winer resulted in orders of magnitude better efficiency for blog search engines.

The same thinking basically applies to Gnip. The idea is to gather simple information from social networks – just a username and the fact that they created new content (like writing a Twitter message, for example). Gnip then distributes that data to whoever wants it, and those downstream services can then access the core service’s API, with proper user authentication, and access the actual data (in our example, the actual Twitter message).

From a user’s perspective, the result is faster data updates across services and less downtime for services since their APIs won’t be hit as hard.

For a fuller description of how Gnip works, see the full overview at TechCrunchIT and this discussion on datastream aggregators.

Digg, a launch partner of Gnip, clearly sees the benefit – they are giving unfettered access to Gnip to their API in the hope that some third party services will stop using it altogether and move to Gnip instead. Other launch partners include Plaxo, Delicious, Discus, Flickr, Get Satisfaction, MyBlogLog, Six Apart, Iminta, Lijit, Social Thing and Spokeo. Notably absent from the list of partners is Twitter, which may be the one service that needs something like Gnip the most.

Gnip worked with Pivotal Labs to develop the service.

Eric is now preparing to launch a new startup, Gnip. Details are scarce for now – Marcoullier isn’t saying what the new startup will do other than a hint on the site itelf – “Web 2.0 Infrastructure,” and a message that the service will launch in May.

The startup is already funded, he says, with a $1 million round from Foundry Group and SoftTech VC. Foundry Group’s Brad Feld and SoftTech VC’s Jeff Clavier are joining the board of directors of the company.