Twitter comes clean

Next Story

Robotic stepping stones, so you never have to touch the pavement again, m'lord

Twitter developer manager Alex Payne has updated the Twitter FAQ with the actual, real, honest story on the return of Track to its users. First, the relevant text:

When will the firehose be ready?

By late January, early February 2009. For at least Q1 2009, the “firehose” (the near-realtime stream of all public status updates on Twitter) will only be available to a small group of trusted partners. The firehose is a stream HTTP solution; a client connects to it and the stream begins, ceasing only when the client disconnects. Once we’re confident in the stability of the service, we’ll add partners on a case-by-case basis. We may allow a wider selection of clients to consume subsets of the public stream (that is, updates from a collection of user IDs or matching specific search terms). We do not intend to allow anonymous, unregulated public access to this stream for any number of legal, financial, and technical reasons.

Now, the translation:

Real soon now, especially now that FriendFeed has a quarter of our page views with a stunningly familar hockeystick of growth, we will release the firehose to trusted partners. Trusted means those vendors who will agree not to allow access to… see below. The firehose is the full stream of our data that has been blocked from its contributors since May, 2008. Once we’re sure it is stable, we’ll continue to make it available while adding what must be semi-trusted cases. It’s also possible we’ll deliver a subset of the firehose (analogous to somewhat pregnant) defined as Track on identities and keywords. The keyword here is “may”. Finally, we won’t allow anonymous unregulated access, period. That is, even though we have numerous partners and untrusted startups currently recording Twitter notices and storing them for unregulated anonymous access since Twitter began.

FriendFeed co-founder Bret Taylor appeared on NewsGang Live Friday, and told me relationships with Twitter continue to be good. The two companies are working through some problems with the rate limiting curbs introduced by Twitter several weeks ago, but Taylor anticipates a resolution shortly. Several third party Track projects, most notably including Dustin Sallings’ TwitterSpy, have been disabled due to the 20,000 API call limit imposed. Sallings is blunt in this FriendFeed thread:

They’re going to offer a friendfeed-style HTTP firehose to a limited group. My suspicion is that that group will be limited more by how threatening a business is than even by how much twitter’s traffic may be reduced by such a partnership. I might be wrong, but the only ideas they seem to have for making money from their business involve removing value their customers want.

Meanwhile, Taylor says FriendFeed is moving forward with enhanced realtime tools to help model Twitter and other data. Rooms will gain new controls for aggregating multiple streams, a major search-related announcement is coming later this week, broader filtering and track functionality awaits a several-month rewrite of some parts of the core architecture, and most importantly, FriendFeed will continue to employ an open, inward and outward-facing data strategy. This is in sharp contrast to both Twitter and Facebook, who allow ingress but limit outbound flow.

There are several efforts underway to work around or via the back channel with Twitter to reengage track services. Services such as Twhirl that have released betas with “track” support may fall into both categories, but eventually Twitter will find a happy medium where monetization will begin to flow. In the meantime, FriendFeed continues to offer a more conversational and flexible model, making it a significant competitor for user contributions. Even now, it’s trivial to maintain a Twitter presence via FriendFeed that would require a fundamental change in developer relations to undermine.

Now that Twitter has achieved a certain stability and clarity in its rate-limiting strategy, the next phase will focus on identifying and rationalizing its trusted partners. The fundamental value proposition of track – the filtering of micromessages based on a combination of identity and conversational context – can now be achieved in FriendFeed with greater fidelity and, soon, realtime alert mechanisms that allow more personalized and affinity-powered flow regulation. The result: time-efficient information at the center of the user experience.

Over time, Twitter’s huge audience size and mainstream media acceptance will become less significant, forcing Twitter to name its price for its unique value even as it is watered down by more flexible tools and micro-community adoption of its competitors. Regardless of the anger in the community, which clearly has been discounted as a small minority in Twitter’s game plan, the clarity of Twitter’s rate limiting and brute force approach in managing its developer community now stand in sharp contrast to FriendFeed’s approach.

  • gregorylent

    they could come clean about a few more things …

    why do people i have unfollowed previously, show up again?

    why do the numbers suddenly fall sometimes?

    what percentage of tweets actually get sent to me from people i follow?

    weird company, in some ways …

    enjoy, gregory lent

    • Jenny DiDonato

      Good point, about the un-followers popping up again. Weird.

  • Robin Barooah

    I asked Alex Payne in a public forum why they were switching away from XMPP to their own HTTP Push solution. His answer was that they were having problems with the java XMPP libraries that they were using, none of which were built to handle the kind of load they have, and that XMPP itself was designed to handle instant messaging, which isn’t exactly what they are doing, and so they had found themselves having to make proprietary changes anyway.

    As a result of this experience, they decided to develop their own HTTP-push based solution. This was only a couple of weeks ago, and any prudent engineer would want to open up a new interface like this in a controlled manner.

    Mr Payne seemed to be speaking plainly and gave no reason to suspect that there was a sinister political motivation behind this step, rather that it was a pragmatic technical decision.

    Of course that doesn’t mean they don’t have the fears about competition that the commentators suggest, but I don’t yet see any evidence for them acting on such fears.

    • Steve Gillmor

      I impugn no sinister motive whatsoever. The latest FAQ update makes clear exactly what is going to happen, including timing, scope, business relationships, who’s in, who’s out, and what we can expect to monitor in terms of hewing to this version of the roadmap.

      It is not unfair to suggest that these statements have been all over the map for some time, and need to be taken with a grain of salt. I don’t think they are acting out of fear, but out of what they see as self-interest. If you listen to Bret Taylor in his conversation with me, you’ll see that others have different notions of how to use openness as part of a strategy.

      To summarize, Alex is indeed speaking plainly. The XMPP issue seems largely moot to this positioning. Twitter has come clean in its plans to provide what kind of access to whom of its firehose.

  • Twitter comes clean « Timothy |Tim| Peters [.ca]

    […] 2, 2009 · No Comments Link: Possibly related posts: (automatically generated)No TitleRhett and Link Sing for Social […]

  • Janes

    what percentage of tweets actually get sent to me from people i follow?

  • Crazyglues

    I love the Picture for this story, that’s just Priceless… :)

  • Bengo

    Yeah, I mean this in a friendly way, but did you rush this? Readability and comprehensibility are pretty low. And I know what it’s about!

    • laura

      I agree, I am not an ubergeek but the story is really hard to follow.
      Can anyone tranlate please into normal speak……..
      thank you

      • laura

        translate, sorry :)

  • Twitter comes clean at

    […] TechCruch reporting that Twitter are finally talking about their firehose, that stream of tweets that was XMPP is now […]

  • Twitter Me This » Help Desk Screeds

    […] An article about the internal politics of the Twitter community. […]

  • EDITing in the Dark » Rise of Micro…commenting??

    […] whatever reason, I saw a story somewhere that said that FriendFeed was dead or dying (really or not ??). Then over about the same time, I’ve had quite the number of requests to subscribe to my […]

  • Ben Barren - Confessions of a Mad Man » win/LOSE superbowl medical monday

    […] the following type of work mentioned on TC_IT about friendfeed : “Meanwhile, Taylor says FriendFeed is moving forward with enhanced realtime tools to help model Twitter and other data. Room… awaits a several-month rewrite of some parts of the core architecture, and most importantly, […]

  • Mike

    Twitter may have cut down on the “hard failures” but the number of “soft failures” keeps growing. In addition to all of the problems mentioned by others so far, there are others; like a little problem of tweets not showing up in the correct timelines (a MAJOR problem that most people don’t notice because they don’t see it, but it happens A LOT, which is a problem for a messaging service); a horrible mechanism for dm’s where if I delete a dm I sent you it also is removed from you inbox and vise-versa, even if you haven’t read it (if you don’t have email or sms notification turned on, you might never know it was there); and a help desk nightmare (they use GetSatisfaction and and it is still like howling into the abyss). I’ve cut way back on the number of people I’m following, the number of people I follow, and my overall usage because at least when Twitter was down I knew it was down, with these soft failures I can’t trust it at all.

  • Peter Urban

    Twitter’s strength for the masses is it’s simplicity compared to friendfeed. However I do believe that twitter should push hard to establish themselves as a backbone supporting a large amount of front end communication services.

  • Jay Cuthrell


    As you saying there is a greater implication for XMPP being dropped (for now?) in favor of, well, anything else? i.e. analogous to XML being dropped in favor of EDI?

    Are you of the opinion that a move of this type takes Twitter into an island or silo or pick your own metaphor of proprietary format?

    I only read this adoption of the HTTP stream as a concern over stability in the wake of greatly publicized failures. As the FAQ has changed, I’d fully expect some leeway in the API team to have more than one fire hydrant if you’ll allow that analogy to extend to your fire hose.

    Of course, big pipes cost money in all analogies do they not?


    • Steve Gillmor

      I don’t think HTTP Post or some variant is a bad idea. Whatever it takes to provide the firehose is a good thing. How that firehose is provided and to whom is much more relevant to the growth of micromessaging.

      Though there are many who feel this Twitter roadmap is a disappointment, it may well accelerate the time when filtered track becomes available. Whether its Twitter or FriendFeed is more the question.

      • Jay Cuthrell

        Agreed. The Twitter or FF is going to be very interesting.

        Thanks for covering this btw!

      • Andy

        I’m not sure what HTTP Post has to do with twitter providing a stream of data via HTTP. It would be most efficient as a GET request, send some headers, client gets a confirmation back that it has connected also via HTTP headers, and the stream just starts being sent.

      • Steve Gillmor

        Not sure either, but several third parties have described how they use what they call HTTP Post to replace XMPP on the Identica/Laconica platform. @kshep on Twitter and/or for more.

      • Ken Sheppardson

        Just came across this… better late than never, I suppose. What we’re doing with Identispy–a service built to provide “track” for Identica and any other Laconica site that wants to participate–is providing an HTTP API to post messages to the service. It’s not that hard to simply configure a Laconica instance to push your site’s full public XMPP feed to Identispy, but for those admins who don’t want to deal with creating an XMPP account or rely on someone else to administer an XMPP server there’s a module distributed with Laconica to use HTTP. You ask us for an API key, add two lines to your config file, fire up the HTTP module, and all messages on your system are posted to Identispy as an Atom payload in an HTTP POST request.

        I know the terms “GET” and “POST” have gotten pretty muddled over the years, but submitting an message/document via POST (in this case an Atom message) is pretty common usage.

  • Daily Diigo Discoveries 02/02/2009

    […] Twitter comes clean […]

  • pradeep

    nice post

  • Darren

    this post implies that twitter have friendfeed envy in some way.

    You also failed to mention twitter is not the only startup to with XMPP, gnip has the same problems and now post xml to 3rd party sites.

    Its very effective and can deal with a heavy load and also most webservers can handle high loads.

    Friendfeed is a great service but its not twitter.

    I would say friendfeed should invest some time in their UI over pandering to tech bloggers feature wish lists.

    twitter is easy to understand for the mainstream friendfeed is not in its current form.

  • Twitter comes clean « Power of Thought
  • Bob Wyman

    No argument that XMPP is inherently less efficient than an HTTP Post implementation is sustainable. In both cases, you have an initial chunk of data that establishes the connection followed by a stream of data.

    It is certainly the case that open source XMPP servers are built to handle the specific case of using XMPP for instant messaging and thus are not tuned well for the “firehose” application, however, building a trivial XMPP server, suitable for “firehose” publishing, is something that shouldn’t take a good engineer with TCP/IP network experience more than a day or two. The resulting server would be more efficient than using any of the existing HTTP servers since they are designed for a different use pattern.

    My comments above are based on experience, not speculation. Five years ago, at, when building a system virtually identical to what Twitter is attempting to build today, we did as suggested above and built our own simple XMPP server for “firehose” feeds that published streams containing millions of blog entries each day.

    There is nothing in XMPP that makes it unsuitable for this application.

    bob wyman

    • Chris Heath

      yeah bob … you tell ’em

      and steve: so much for the bearhug eh?

      • Steve Gillmor

        The bearhug is alive and well. And thanks to Bob for the expertise. I doubt the XMPP/HTTP issue is in the least relevant or important versus the overriding business issues.

  • Jeremiah

    I think that this shows all of us that there will be a large competitor to Twitter within the next year or two and just like there was Live Journal and then MySpace and most recently Facebook evolution in this space will be telling

  • Tweet, Tweet… when does the Twitter-ing cease? | Strategic Blend

    […] * * * UPDATE * * * It Gets WORSE! […]

  • Easy money | Finance and Money Articles

    […]  Twitter comes cleanPosted by stevegillmor via TechCrunchIT   […]

  • Programmer Helper

    I’ve cut way back on the number of people I’m following, the number of people I follow, and my overall usage because at least when Twitter was down I knew it was down, with these soft failures I can’t trust it at all.

  • greg

    Makes me think of In Front of Your Noses’ post last Friday: “Should I kill myself or read another Twitter message?”: Camus’ question for the social media generation (

    I think I’m going for the former this time.

  • Super Bowl Ads: The Good, The Bad and The Ugly | Seo Services, LLC - Indiana based search engine optimization consultant

    […] the recent inauguration. Twitter which had doubled its capacities for the Obama Inauguration day – well supposedly – it was relatively quiet throughout with occasional comments on the […]

  • 090203 Techie Links | Recruiting News and Views

    […] Twitter comes cleanRough translation: As Twitter becomes a commodity, its current policies that limit developer access to the full flow of Tweets will be its achilles heel.   I’m on Twitter, Facebook, LinkedIn and Friendfeed. Catch up with me. […]

blog comments powered by Disqus