Spy vs. Spy

Next Story

Mortal Kombat versus DC Universe trailer

On demand computing has seen some significant challenges over the past few weeks. From Amazon to Rackspace at the provisioning level to Gnip and the social news stream to Apple’s iPhone software and hardware rollout, some new strategies are emerging to survive the reworking of the technology landscape.

The ongoing Twitter saga provides an efficient microcosm for examining the delivery of services over the network and how the consumption of those services affects the provider’s game plan. As Twitter struggles to regain its footing, the company has stripped down to its shorts and then slowly added back features and users while dodging events and memes that might overwhelm their repair work.

In the rebuilding process, things reached a crescendo when Twitter’s @reply functionality vanished for almost a week. Normally, the reply architecture (which co-founder Biz Stone admitted in an interview was not in the original plan and is being reengineered) can be easily routed around with the Track keyword feature. But although @replies are back, Track and the IM XMPP stream it leverages have still not returned.

Not only that, but Twitter’s public statements on the subject have been fragmentary, misleading in some cases, and disquieting not only to users but more importantly the developer community that has sprung up around the service and indeed contributed to its rapid growth. Remindful of the lessons learned from Watergate, and that it’s the coverup that will kill you, some developers are looking for ways to challenge Twitter’s apparent bet that they can survive by stonewalling their potential partners.

A case in point is the Twitterspy service, created by two developers as a labor of love to return Track functionality to the firehouse stream of data Twitter is allowing 4 vendors to sip from but not pass along. The main player is Summize, whose search site and API have kept most Track fans from completely abandoning the service. During the recent Apple WWDC conference, Twitter openly suggested its users utilize Summize, prompting some to speculate there was some special relationship. This was confirmed last week when Twitter acknowledged Summize, FriendFeed, and two others were allowed access.

Though Twitter specifically ruled out any syndication of the stream by these partners, somehow SUmmize allowed the Twitterspy developers access as long as the number of stream requests did not exceed every 15 minutes. The developers set up two servers to collect and index the stream, one via Jabber.org and the other patched into Yahoo’s IM platform. Since Twitter’s XMPP stream was made available over Jabber, that allowed Gtalk and other Jabber-compliant IM systems to stream Twitter status posts including Track. Ironically, a patchwork restoration of that stream minus Track failed at the end of this week.

Meanwhile, Twitterspy added the ability to post directly from within its IM window back to Twitter, as well as follow and unfollow users on the parent service. The service, which had 2 users when we wrote about this last Monday, now has 82 users and 341 topics or keywords as of 11:30am Pacific. Earlier this week, the lead developer was contacted by Jabber.org and told that if the number of users reached 1000 they would be shut down. At the current rate of growth that might occur by next weekend if not sooner.

Twitterspy is therefore vulnerable on several levels: first, the Jabber.org server could go offline in about a week, but replacing it by setting up the open source Jabber server on a separate machine could take anywhere from one to two weeks. Secondly, should Summize be pressured or on their own decide to block access to Twitterspy, a distributed network of servers would need to be setup to make it difficult for Summize to distinguish between a human and an automated request. That might take from 2 to 4 weeks, though existing Jabber distributed and clustering technology would make that available without severe coding costs.

Though Summize could at that point completely shut down its XMPP stream, or get bought by Twitter and turned into a proprietary hub, the resultant outcry not only by developers but users could be sufficiently onerous to make that draconian outcome unlikely. In addition, dynamic redistribution of keyword requests to maximize highly-trafficked keywords over less requested ones could allow Twitterspy servers to get more timely updates with the same allocation of Summize resources currently consumed by the 15-minute throttle.

Of course, Twitter could just acknowledge that this is is not a technology issue and enter into an operating agreement with an intermediary such as Gnip until such time as they could take it in house. Or they could continue to utilize a “don’t ask don’t tell” policy where they tell the Twitterspy (as they have) and others that they will not authorize syndication from the 4 favored partners, while looking the other way when small ad hoc efforts such as Twitterspy operate with Summize data.

The danger Twitter runs is in allowing these grass roots movements to build out a distributed network that not only uses Twitter Track as a carrot to encourage adoption but also builds a contiguous message bus that could align with other services to get the kind of scale Twitter enjoys. You can certainly conceive of an automated service that idetifies frequent Track sginallers and then automatically registers them as followers on a separate system that could go operational once the alternate cloud of users reaches a critical mass or Twitter decides to lock out access to their API for Follow requests altogether.

Taking that step might be survivable for Twitter, but more likely the company would have to figure out what the value of Track is and provide some sort of rationale for a business model that would prove acceptabe as a contract with users or developers who would in turn pass along the costs via some sort of model. Carving away a large portion of Track effectiveness to a real-time message bus that sits alongside Twitter would likely force the company to the table, and that’s exactly what this loose coalition of engineers is in the process of building.

At a larger scale, cloud computing business models require a large degree of transparency on the part of the service vendor, not just of how they are serving the user but what vulnerabilities the user may encounter once data has been indexed and stored. In Track’s brief moment in the sun before Twitter unraveled, already the number of auto-bots, marketers, spammers, and disruptors of these new real time channels has accelerated dramatically. The very vehemance of these strategies and Twitter’s contortions in locking down the XMPP streams underscore both the power and economic stakes of Track and its extensions through filtering and dynamic provisioning.

  • http://blog.broadpool.com/ Glen

    I think you guys have no idea what Information Technology means IRL.

  • Michael Markman

    @1. If we limit Information Technology only to what it has meant, then we’re closing off many paths to new value creation. There’s something new and powerful brewing here. Why argue about what bucket it fits in?

  • http://dmix.ca dMix

    Steve Gilmours Posts:
    Spy vs Spy – Mentions Twitter
    The Enterprise iPhone Twitter Connection – About Twitter
    Send in the Clowns – No Twitter (best post as well)
    Deconstructing Twitter – About Twitter
    Plus ça change – Mentions Twitter
    Independence Day – Mentions Twitter
    The Crown Jewels – Mentions Twitter

    — I stopped on page 3.

    Thanks for your insightful information technology posts.

    I’ve unsubscribed, so I’ll stop complaining about this.

  • Steve Gillmor


    Welcome to the Ostrich Pool.

  • http://www.smallbizpod.co.uk Alex Bellinger

    If Twitter really starts shutting down services like twitterspy, this surely demonstrates that Twitter knows how important track is. That’s a good thing, albeit that it gives track addicts short term withdrawal symptoms.

    If the withdrawal ends up being long term, then whichever developer/alternate service can concoct a methadone-like track substitute could walk off with the Twitter’s stash of users.

  • http://www.masonlee.org Mason Lee

    Glen, dMix– This hypothetical world-wide dynamic pub-sub architecture that Steve is always going on about (and using Twitter as an example for) has pretty radical implications for information technology.

    Steve, have you talked much to folks like Tibco and webMethods (now Software AG) to see what their message brokers might be able to do for a track-like system? Seems like these companies will have a lot more experience in this realm than Twitter. I’d be interested to read some enterprise perspectives on the problem, if you have links. Enjoying your posts and radio show every time I get the chance.


  • http://blog.echovar.com/?p=449 echovar » Blog Archive » A Fine Madness: Spy vs. Spy

    […] not certain how these things are connected. But watch the whole thing and I’m sure something will […]

  • http://blog.echovar.com/?p=450 echovar » Blog Archive » Boundaries of the Real-Time Conversational Flow through the Network

    […] that right, they’ll need to focus on delivering high volume, high quality data liquidity. The key question is under what terms that data will be available. The economics of real time stock exchange data is […]

  • http://datacenterjunkie.com/ Michael T. Halligan

    I’m blown away. Who’d have thought that TechCrunch had run out of space for posts about Twitter that you guys would have to create a new web property, solely for the purpose of talking about Twitter (and some other fluff).

    TechCrunchIT = TechCrunch Instatiable for Twitter?

    Don’t get me wrong, Mike’s a nice guy, and I’ve had fun at the two bbqs I attended at his pad, but surely there must be something new in the world of IT to talk about besides Twitter? Maybe the hyperactive REIT and Datacenter market? Utility computing? Something besides Advertising and Twitter-addiction?

blog comments powered by Disqus