Deconstructing Twitter

Next Story

Pioneer Elite Kuro PRO-111FD plasma HDTV reviewed

Today we get the latest full ongoing iterative back-filling explanation of Twitter and XMPP. Dave Winer’s intuition that FriendFeed is one of several (4) receipients of the XMPP data stream turns out to be correct. In a post this afternoon, Biz Stone confirms Summize and adds Twittervision and Zappos as the others. FriendFeed makes only small demands on its current subscriber base, but its rapid growth in recent weeks suggests that will become more of a problem as time goes by.

Stone suggests Summize has “greatly improved on Twitter’s Track feature” with its extended filtering capability. More accurately, it has improved on a search function that Twitter does not have, but Track’s real-time display of keyword “hits” enables conversations that Summize only simulates. Track, when combined with XMPP over IM (Gtalk/Gchat in my case) allows instant replies as hits present themselves, where Summize has no direct input to Twitter but rather calls the Web application for replies to search results.

A work-around for that limitation would be to allow the Summize stream to be syndicated to other services, something that appears to be working in some fashion via Twhirl’s implementation. Clicking on the Spylass icon produces an input window to query Summize’s server. But you have to manually enter terms, and the results are not made available or updated in real time while the possibility of a conversation ephemerally exists.

It’s difficult to tell in this query by blog post communication with the mother ship just what the deals are with the 4 favored sippers from the firehose. It may just be a design decision on Summize’s part as to how to “greatly improve” on Track, but the other possibility suggested by earlier statements on the subject is that Summize is at least somewhat constrained in how it can pass along the stream to other parties.

Specifically, is the constraint (if it exists) that the more ongoing requests for standing searches (Track) the more stress on Twitter’s core? We know the average interval between a message being posted on Twitter to surfacing in Summize is somewhere around 30 to 60 seconds. It’s hard to say without testing a service no longer available what the interval between posting and appearing on Twitter is, but a followed post is sometimes as fast as 10 seconds.

A reasonable assumption can be made that Summize is recording the XMPP stream and then searching it, separating any additional load from Twitter’s servers. Therefore, there is no reason other than business considerations why Summize could not provide access to its search services to other parties. But Twitter continues to offer language that can only be read as an evasion of the fundamental issue.

While the XMPP feed of the full Twitter Public Timeline is an amazing resource, drinking from the fire hose is not the best way to quench a thirst. With continued updates and refinement, our API will support most scenarios in a way that preserves overall system performance.


Yes, Track rocks. But doing what Track does is not the best way to do what Track does. We will continue to try and figure out how to provide fixes and improvements to Twitter without crashing our system. Even though we have no additional demands on our system if Track (or super-fast search) is offloaded to another party, we’re not going to do that because we want to reduce the Track capability from its current amazing status to one that we control through our API where we can limit the business case to “most” scenarios, in other words, scenarios that are less than amazing.

Despite Twitter’s best efforts to restrict access to their data, a third-party developer has created a service called which implements Track using Summize data. Regardless of the effort Twitter makes in controlling this data, there is so much user demand for these features that developers will always find a way around it. Twitter would be better served by opening this data up rather than attempting to control it and fighting both the developer and user communities.

  • Dan MacTough

    So glad you mentioned So many of us are jonesing for track’s return.

  • David Damore

    Exciting changes just on the horizon. We can start to ‘see’ them now.
    Where all this leads in the next three to six months will be very interesting.

  • » Blog Archive » TechCrunchIT » Blog Archive » Deconstructing Twitter

    […] XMPP was designed to handle. Why not use it? What are they protecting by pissing people off? [From TechCrunchIT » Blog Archive » Deconstructing Twitter] You can skip to the end and leave a response. Pinging is currently not […]

  • dMix

    Yay, TechcrunchIT and Twitter again. Plus some FriendFeed mentions.

    I love getting my business technology news here.

  • Nik Cubrilovic

    would like to hear what developers of twitter apps think of this.

  • Nik Cubrilovic

    dMix: might not seem relevant to you right now, but we wont be getting very far in building out web services if even Twitter, once a leading light in openess, is protecting user data in this way. How are we supposed to ask the big guys to co-operate?

  • Luke S.

    The reasonable assumption that you make regarding Summize recording and parsing through the feed sounds quite plausible. Now, why can’t Twitter just set up several mirror sites that do something similar and then transition to a new architecture later?

    That way, they might just be able to also open up the XMPP stream to everyone else.

  • Scott Kingery

    So glad to find TwitterSpy for IM! Wrote up a quick how-to on setting it up in GTalk/Chat:
    Now lets hope that TwitterSpy can scale.

  • Mindaugas Dagys


    In the light of this maybe idea of decentralized Twitter starts looking more appealing, even though this is promoted by Dave Winer.
    We all would not be at the mercy of what the service owners decide (what is best in their business interests).

    To Twitter: do you really think you are going to win fighting your most dedicated users?

  • Matt Terenzio


    I was planning on using Summize in my client to imitate Track anyway, so it doesn’t change my plans as a developer of an app. . .but. . .

    There is no way that Twitter can claim witha straight face that this is in response to technical challenges. It’s completely a business decision.

    Hell, the Twitter developer community is so ga-ga over Twitter they’d be lining up to mirror the firehose stream in the way that SourceForge and Linux-distros get mirrored by the community.

    In fact, that’s what would still happen if they hadn’t put conditions on the way Summize and the others can use the data. But I’m sure they have and that kinda proves it’s a business move, not a technical one.

  • Paris

    At last a Steve Gillmor that does NOT drone on about Twitter… on wait sorry my mistake.

  • come on...

    excuse me, why is Twitter / Summize / FriendFeed considered IT news? It’s not as if we’re not getting enough of them in TechCrunch…
    Can we get to more interesting & relevant news? I suggest a VMWAre update, seems pretty dynamic over there today, and surprisingly (or not?) neither TechCrunch IT nor TechCrunch have written about it. Sure, it’s way more fun to write about Twitter, right.

  • Technology Trends » Blog Archive » TechCrunchIT » Blog Archive » Deconstructing Twitter

    […] TechCrunchIT » Blog Archive » Deconstructing Twitter Blogged with the Flock Browser […]

  • Notebook For the Week of July 7 « nina scaletti

    […] Deconstructing Twitter. I’m still not 100% behind the philosophy of Twitter, but this breaks it down a bit. [TechCrunchIT] […]

  • Interview With Evan William: Summize Acquisition, API Issues And Their Revenue Model

    […] has been criticized for severely limiting API functionality and discriminating against most developers by only allowing […]

  • Interview With Evan Williams: Summize Acquisition, API Issues And Their Revenue Model

    […] has been criticized for severely limiting API functionality and discriminating against most developers by only allowing […]

  • TechCrunch Japanese アーカイブ » Evan Williamsに直撃インタビュー!Summize買収、API問題、収益モデルなどなど色々聞いてみました。

    […] TwitterはAPIの機能性を厳しく制限したり、パートナーの4社(しかもその中の1社をつい最近買収)にしか”firehose” XMPPフィードへのアクセスを認めないなど、多くの開発者達を差別していると批判されてきた。インタビューの中で、Williamsはこの件に関して大変率直な姿勢であり、同社はこの問題に対応する為に、データポリシー問題を解決中だとしている。例えばFriendfeedには完全なXMPPフィードがあり、なおかつそれを利用してTwitterの失策から急速にマーケットを拡大してきた。 Michael Arrington (以下MA): (XMPP firehoseを公開しないのは) 他社が情報を効率的に取得出来過ぎるからですか? 彼等も競争相手だからですか?全ての情報が公開されて開かれているというのは素晴らしい事だと思いますが、相手はあなた方の競争相手でもありますし、あなたは今相手の全ての情報を握っていますよね。 […]

  • lieben

    Interessante Informationen.

blog comments powered by Disqus