The Crown Jewels

Next Story

Plasma torch garbage processing produces usable gas, electricity

It turns out the battle for control of Twitter rests almost exclusively in the unique value proposition of XMPP-served track. As Twitter strips away various features of its service to rebuild a scalable fail-whale -proof version, the one remaining hurdle is restoration of a fully-functional Track over IM.

For the last two weeks, a one-way IM service via Gchat inside Gmail or Gtalk standalone has provided a stream of tweets but not the previously enabled ability to post back to Twitter via the IM window. In addition, there is no support for the Track function, which interweaves Tweets from any endpoint on the Twitter network that correspond to the keywords you “track” on. Track was briefly available over SMS several weeks ago, but was then withdrawn.

Track and realtime IM users have made do with a cornucopia of third party add-ons, most significantly the Summize service that lets you create compound OR searches to simulate Track’s multiple keywords. Instead of seeing these “hits” in the Gchat stream, you create a separate Firefox tab and watch for the title bar to change when new entries appear. You then click the “refresh” link that appears and scroll down through the latest entries.

As for the realtime flow aspects of the IM integration, I run Twhirl, an Adobe AIR application that pushes new entry notifications up from the bottom of the screen as they appear; clicking on one brings the Twhirl window forward with the latest results. It’s not quite realtime, as Twhirl uses API calls that have frequently been gated by Twitter as demand overwhelms the gasping Twitter servers. You can add FriendFeed and its search capability to Twhirl in another window, and even click on a button to access Summize searches.

Ever since Twitter shut down its XMPP stream to third party developers, it’s been a Twitter parlor game to figure out how Summize continues to provide realtime service. In conversations with TechcrunchIT over the past 24 hours, the mystery has been cleared up. In fact, Summize is using the XMPP service provided by Twitter as it has all along. While Summize is not specifically named, Twitter acknowledged in a post to the Twitter development talk Google group on June 19th by Alex Payne that “the public timeline PubSub feed” is being delivered to several subscribers. Those subscribers were chosen because they provided a great deal of value to the widest audience possible at least cost to the overburdened Twitter infrastructure.

Allowing Summize to replace some aspects of the Track functionality is keeping power users from bolting to other services, not just because the promise of restoring overall service is implicit in the continued bootstrap but also that Twitter signals that they recognize the unique and perhaps critical intellectual property that Track in realtime represents. The dominant market share that Twitter still enjoys is leveraged by the ability of anyone in the cloud to signal to any other user (and most significantly, group of users) in realtime as a response to or the trigger of a swarming event on the network.

Many users have made much of migrating to other services, most aggressively FriendFeed, as a more reliable and perhaps more focused cloud of users. But Twitter’s closely-held XMPP subscribers do not include FriendFeed or other close competitors, so most Twitter users are keeping their accounts open in anticipation of the revival of access to realtime conversations. The Alex Payne post makes clear that revamping Twitter internals to support additional XMPP subscribers is not high on the priority list, but neither is Twitter willing to allow Summize or other current subscribers to syndicate the feed out to the third party cloud of developers.

There have been internal discussions of the business implications of handing out the stream of all public updates. Enabling the stream for existing and future subscribers will likely entail signing a terms of use document that specifies, amongst other things, that we won’t allow any resyndication of content (without a further agreement). This won’t effect some projects, but clearly will effect others. Such is business.

And such is the high-wire act that Twitter executives have to perform. Much of Twitter’s momentum has derived from its harnessing of a viral developer community and the resulting innovation and value creation akin to the Facebook explosion when they opened up API access. But keeping developers in play requires an even-handed and transparent approach to the key assets: rapid realtime access to the core user population in a way that allows Twitter’s vanilla services to be extended to head off better-funded or fast followers who don’t have to rework the underlying architecture in place, but rather can build a system to address the market opportunity Twitter discovered.

In some ways the dilemma resembles Facebook’s stance in relationship to Google’s Friend Connect initiative, which forced Facebook to retreat to legalisms to deny Google access they already allow to other API users. When questioned closely about this apparent disconnect at Supernova, Facebook’s Dave Morin was compelled to acknowledge that, like Twitter, such is business – and is best left not to the technologists but the lawyers.

Twitter has bought enough time with its users, a hardy bunch who have proven they can endure unending pain with the faint promise of better days ahead. But the third party developer crowd will need to know sooner than later whether it makes sense to hitch their star to the Twitter cloud, or move laterally to other platforms or even a federated alliance where users will aggregate in hopes of improving their clout and leverage to gain new features and authority.

  • Dewald Pretorius

    As a Twitter third-party developer ( I must say that I’m appalled to learn that there’s not even-handed treatment of all developers. I will certainly not develop further Twitter add-ons until I know that I will get a fair shake of the deal.

    As far as Twitter saying to users to look for replies on Summize, someone wrote to me the other day, saying it’s like Twitter saying, “We’re a little busy, go see if you can crash the Summize servers.”

  • Matt Terenzio

    In the wake of there is a flood of skepticism laced with a dash of hope.
    It is reasonable that Twitter might not lose its loyal core, but then again it is also reasonable that a smaller community could develop on and maintain itself.

    When Jabber was formed it did little to dent the IM silos that continue to thrive. But XMPP did not die either, and continues to be significant and grows in it’s multi-colored usage.

    Similarly, if a small, seemingly insignificant federation could form based upon the protocols that proposes it could be years from now that a major vendor in need of some market share decides to adopt those protocols in the way that Gtalk adopted XMPP.

    One could argue that GTalk did little to make waves in the existing IM silos. Yes, but I’d say it’s too early to tell. Is XMPP still gowing? Yes. Will OpenMicroblogging find its place. Maybe. Just not today.

  • Dave Winer

    Are you sure that FriendFeed isn’t hooked into Twitter through XMPP? I find my Twitter status messages show up on FF almost instantly. If they aren’t hooked in, they’re polling like mad (which I don’t think is the way they work).

    It’s good to see you guys picking up on this. It was the subject of my podcast with the Gnip guys yesterday. Very important that people understand this, though it’s exactly the kind of thing most users don’t want to know about.

  • Dave Winer

    For example, it took less than a minute for this twit to show up on FF.

    It’s *always* that fast. I don’t see how that could be polling, I think they’re getting the XMPP flow.

  • A Sad Developer

    I’m really happy you guys are covering this.

    I spent a LOT of time developing Twitter web apps and the XMPP public feed delivery was central to their function. But now unless you are on the special users list there is no way to get all public tweets (minus the 60/minute they give you through the API).

    It’s frustrating to just get locked out after spending so much time making stuff for Twitter users. For the community in general it sucks because, let’s be honest, half the reason Twitter is cool and interesting is because of all the random little Twitter sites that popped up. Now they are all foo bar.

    It makes me more sad than angry.

  • http://blog/ francine hardaway

    Whoa, that’s interesting! If I were a developer, I’d feel the same way the other developers have already said they do. As a user, I’d like to see it work. I went over to, found my friends, spent a nice evening, but it ain’t Twitter. And I don’t care what Robert says about FF, it ain’t Twitter either.

    Go Steve for revealing all of this! Thank God I was an English major and all I have to do is use these products.

  • Rick

    Great post.

    The only thing that pisses me off is that we have to guess and do research about the issues. No one from Twitter bothers to update except via a crappy blog or occasional google groups. (btw, i setup tweetstatus that just retweets their blog rss feed).

    One of the founders tweets about his caribbean vacation but doesn’t bother to address the situation to the community via twitter.

    I’d have more faith if I knew more.

  • sull

    nice post.
    i’ve built one twitter app that does not rely on public_timline (tweetshots) but knowing about the architecture issues, the new/forthcoming API constraints and business is business… I have held back on new features/services using twitter until i see how this all plays out… which is quite interesting regardless of where you stand.

    Laconica has me excited since its an approach i support and will work on setting up a “federated” node like to interop. i’ll do that on should be interesting as a parallel evolution to twitter/friendfeed etc.

    real-time data… that’s the cake. and we all want the cake. once you know it and can get it, there’s no going back. human nature. especially in geek land. fastest chips, fastest cars, biggest tv, coolest phone etc. real-time ubiquitous internet communications…. a holy grail of sorts. and to compete, you can’t be happy with “old data” and random data (missing out on every post) which is what all 3rd party developers are able to get… the scraps. prob should be good enough for many, but of course it’s not ;)

    good stuff, steve.

  • Michael Krigsman

    Thanks for covering what appears to be an arcane issue and explaining how in fact it’s of central importance.

  • scott

    I was using track for a while before it was taken away. I did not see a lot of value relevant to my needs. I don’t really miss not having it. Why should I care? What do you want to track Steve?

  • Richard Emory

    Give developers an inch, and they want a mile. I think opening up any APIs was a big mistake. It brings a lot of crappy apps, complaints and headaches.

    Would a retail store give you the keys to their warehouse?

    Twitter has taken many millions of dollars in venture capital. How are they ever going to make money by giving away the store?

  • till

    @scott: You can use “track” for many things – let’s say, someone references you in a tweet like, “Check out this link, http://foobar via @scott” – you won’t get that in replies, because to show up in replies, it would have to be start with “@scott”. I personally find it useful/interesting to see what people talk about if the mention “me”.

    Or say, I have a product and want to know if people talk about it on Twitter – another reason to use track. Yeah, I can do all of that via tweetscan etc., but why should I use four tools when I can use just one?

    Btw, I found this article to be very interesing. Let’s critize authors when there is a reason for it, and not just because it was hip last week. :)

  • till

    Apologies for poor grammar, it’s late here. ;)

  • Steve Gillmor


    The point of this particular post was that Track is a unique and valuable property that is being managed carefully to preserve both user and developer allegiance. For me, Track represents a way of enabling conversations with the entire Twitter cloud, extending the relatively small Follow view into one where people can contact and be contacted by anyone based on common interests and issues as they arise. This in turn leads to opportunities for creating micro-communities that, in this next social buildout of the network, can have substantial economic and political power.

    In turn, developers can help manage and build out services for these communities that extend the power and stickiness of the originating platform. This is why the difficult task of remaining open enough to retain credibility is testing the skill of Twitter and its investors.

  • Steve Gillmor


    Interesting question. The little Twitter has said on this subject suggests the other subscribers are high-volume users of the XMPP stream, which certainly might include FriendFeed by process of elimination. It could also be explained by multiplexing multiple API requests that are staggered. Perhaps FriendFeed might be willing to answer the question, though they might be reluctant to rock an important boat right now. Let’s ask.

  • Richard Taylor

    Weird. TC has separate comments threads on this post in TC and TC IT. Maybe I do need to use FriendFeed to aggregate all my feeds and comments to one place.

    I just posted a comment in TC thread saying that Twitter does use polling to implement their XMPP feed. At least that is what their API specs say.

  • Einar Vollset

    FF has to be sucking on XMPP in order to get 100% coverage. Damn it, I’m getting sick of this..

  • Darren

    this is why development is on hold.

    maybe they should charge for the pubsub access and seperate it out from the twitter system I.E. have the pubsub broadcast to outside twitter own servers and have them broadcast to the accounts that want it.

  • Fake Reggie Perrin

    What does this have to do with the crown jewels?

  • exapted

    SMS will die. I guess we are talking about this because we want to understand who will control the standards for the software services that run on twitter-like platforms. That will be the company that will have more influence on everything else and have a higher market value.

    After SMS dies (because everyone has smart-phones with always-on instant messaging softwares), will Twitter still have a lock on it’s users? That depends. Maybe they will continue to be the pipes for such systems for SMS input. Whether they will control what happens with the data after it gets onto such a system – that is the real question.

  • exapted

    Or you could conclude that Twitter just wishes to be an SMS gateway for the social web, until SMS is no longer used by anyone, in 2015 for example, at which point all of the services riding on it will each be independent, and Twitter itself will be like ICQ is today.

  • Aronski

    Now we’re getting to the meat of the tomato. As much as I have been one to spot the black helicopters in the distance, it usually comes down to money and control of the Mcguffin that makes it possible. I too has been wondering how Summize could be chugging along without some other pipe to feed them.

    In discussions about Track with others I have thought about the fact that if you gave any mook access to the realtime stream that you would not only be giving away the keys to the kingdom but a smart person could use the XMPP stream for less than social reasons. Here is a river of information, already public, already prone, that not only could be re-purposed but used to siphon value away from Twitter’s information currency. For large users of the API or XMPP, they can spot that right away. What about an individual who sets up a focused Track trap through something as simple as the XMPP/Jabber IM client, creating a huge document full of data stored for free in his Gmail box that he can sift through for whatever reason?

    I begin to wonder if when Track returns there will be controls not only for the services using the stream (business arrangements indeed!) but if there will be a limit on how many Track words an individual can use at one time without a premium or a throttle to prevent the Twitter servers from glowing Cherry Red…

    As rare as it is, the genie escaped the bottle for a time giving some of us a taste of what this torrent of information energy could do and then they closed the flume, coaxing the genie back behind their wall. Something as innocuous as a messaging system that people use to share what they’re stuffing in their mouths or how much they hate traffic has the potential to be many things for many people. Twitter has not handled the invasive growth of their service, the subsequent collapses and hobbled current operation with much grace or even initial acknowledgment of the effect on their addicted core users.They, silent, push the annoying junkies from their doorway as they head to the health food store for a lactose free beverage or off to the beach, knowing full well the hooks are set, the demand is there and all they need now is to shore of the banks of the river and start selling water rights.

    We as users are unfortunately getting used to being treated like criminals. We all love a deal and especially anything for free. We tend to hack and workaround things when they don’t do what we need them to do or when the provider holds them hostage. But we have a limit and tend to vote with our feet, even if it’s just to the other side of the street for a 2 year contract and then wander back, muttering about how things were over there.

    They have an opportunity here to build a relationship with their users to be the evangelists they need to go to the next level when they’ve restructured their system. I was discussing what I felt was so important about Twitter with a non-user yesterday and went to try to show him and of course, Twitter was completely pear shaped. Not a good chance for growth there…

    Man, they need to get their features squared away and this thing solid because if they don’t, someone else will. The current alternatives are not much to look at but someone else is coming. Enterprise will not stand for the flaky, just-out-of-rehab operation we’ve seen in the last 6 weeks. If they change their access to the stream for big and power users, so be it. It has to work. It wouldn’t be the first time a great idea was done better by someone other than the inventor.

  • A Sad Developer


    About the staggered API requests: Twitter caches the public_timeline api so the most you could get is 60 tweets / minute even if you checked it every second.

  • Gregg

    track is nice, but is it that important to the other 98% of twitter users that don’t care to see @myname in 10 seconds? An rss feed from summize with @myname gives me plenty of time to see and respond to someone who’s reply to me wasn’t formatted for reply. With google reader, i have an archive of those as well.

    Maybe it’s because I’m not pimping my name/latestcriticalopinion/website/company/idea/adsense and following 25,000 twitters that I don’t get it. How does anyone really have time to follow a gtalk track stream in realtime anyway? money I guess.

  • luke

    Time to introduce a new arconym.
    FTFT – fix the fucking twitter

blog comments powered by Disqus