On demand computing has seen some significant challenges over the past few weeks. From Amazon to Rackspace at the provisioning level to Gnip and the social news stream to Apple’s iPhone software and hardware rollout, some new strategies are emerging to survive the reworking of the technology landscape.
The ongoing Twitter saga provides an efficient microcosm for examining the delivery of services over the network and how the consumption of those services affects the provider’s game plan. As Twitter struggles to regain its footing, the company has stripped down to its shorts and then slowly added back features and users while dodging events and memes that might overwhelm their repair work.
In the rebuilding process, things reached a crescendo when Twitter’s @reply functionality vanished for almost a week. Normally, the reply architecture (which co-founder Biz Stone admitted in an interview was not in the original plan and is being reengineered) can be easily routed around with the Track keyword feature. But although @replies are back, Track and the IM XMPP stream it leverages have still not returned.
Not only that, but Twitter’s public statements on the subject have been fragmentary, misleading in some cases, and disquieting not only to users but more importantly the developer community that has sprung up around the service and indeed contributed to its rapid growth. Remindful of the lessons learned from Watergate, and that it’s the coverup that will kill you, some developers are looking for ways to challenge Twitter’s apparent bet that they can survive by stonewalling their potential partners.
A case in point is the Twitterspy service, created by two developers as a labor of love to return Track functionality to the firehouse stream of data Twitter is allowing 4 vendors to sip from but not pass along. The main player is Summize, whose search site and API have kept most Track fans from completely abandoning the service. During the recent Apple WWDC conference, Twitter openly suggested its users utilize Summize, prompting some to speculate there was some special relationship. This was confirmed last week when Twitter acknowledged Summize, FriendFeed, and two others were allowed access.
Though Twitter specifically ruled out any syndication of the stream by these partners, somehow SUmmize allowed the Twitterspy developers access as long as the number of stream requests did not exceed every 15 minutes. The developers set up two servers to collect and index the stream, one via Jabber.org and the other patched into Yahoo’s IM platform. Since Twitter’s XMPP stream was made available over Jabber, that allowed Gtalk and other Jabber-compliant IM systems to stream Twitter status posts including Track. Ironically, a patchwork restoration of that stream minus Track failed at the end of this week.
Meanwhile, Twitterspy added the ability to post directly from within its IM window back to Twitter, as well as follow and unfollow users on the parent service. The service, which had 2 users when we wrote about this last Monday, now has 82 users and 341 topics or keywords as of 11:30am Pacific. Earlier this week, the lead developer was contacted by Jabber.org and told that if the number of users reached 1000 they would be shut down. At the current rate of growth that might occur by next weekend if not sooner.
Twitterspy is therefore vulnerable on several levels: first, the Jabber.org server could go offline in about a week, but replacing it by setting up the open source Jabber server on a separate machine could take anywhere from one to two weeks. Secondly, should Summize be pressured or on their own decide to block access to Twitterspy, a distributed network of servers would need to be setup to make it difficult for Summize to distinguish between a human and an automated request. That might take from 2 to 4 weeks, though existing Jabber distributed and clustering technology would make that available without severe coding costs.
Though Summize could at that point completely shut down its XMPP stream, or get bought by Twitter and turned into a proprietary hub, the resultant outcry not only by developers but users could be sufficiently onerous to make that draconian outcome unlikely. In addition, dynamic redistribution of keyword requests to maximize highly-trafficked keywords over less requested ones could allow Twitterspy servers to get more timely updates with the same allocation of Summize resources currently consumed by the 15-minute throttle.
Of course, Twitter could just acknowledge that this is is not a technology issue and enter into an operating agreement with an intermediary such as Gnip until such time as they could take it in house. Or they could continue to utilize a “don’t ask don’t tell” policy where they tell the Twitterspy (as they have) and others that they will not authorize syndication from the 4 favored partners, while looking the other way when small ad hoc efforts such as Twitterspy operate with Summize data.
The danger Twitter runs is in allowing these grass roots movements to build out a distributed network that not only uses Twitter Track as a carrot to encourage adoption but also builds a contiguous message bus that could align with other services to get the kind of scale Twitter enjoys. You can certainly conceive of an automated service that idetifies frequent Track sginallers and then automatically registers them as followers on a separate system that could go operational once the alternate cloud of users reaches a critical mass or Twitter decides to lock out access to their API for Follow requests altogether.
Taking that step might be survivable for Twitter, but more likely the company would have to figure out what the value of Track is and provide some sort of rationale for a business model that would prove acceptabe as a contract with users or developers who would in turn pass along the costs via some sort of model. Carving away a large portion of Track effectiveness to a real-time message bus that sits alongside Twitter would likely force the company to the table, and that’s exactly what this loose coalition of engineers is in the process of building.
At a larger scale, cloud computing business models require a large degree of transparency on the part of the service vendor, not just of how they are serving the user but what vulnerabilities the user may encounter once data has been indexed and stored. In Track’s brief moment in the sun before Twitter unraveled, already the number of auto-bots, marketers, spammers, and disruptors of these new real time channels has accelerated dramatically. The very vehemance of these strategies and Twitter’s contortions in locking down the XMPP streams underscore both the power and economic stakes of Track and its extensions through filtering and dynamic provisioning.