Twitter CEO Dick Costolo has once again promised that Twitter users will, indeed, be able to download a full archive of their tweets in just a matter of weeks. This latest announcement was delivered in response to an audience question during a talk sponsored by the Ford School of Public Policy and School of Information at Costolo’s alma mater, the University of Michigan, where the CEO discussed Twitter’s role in the future of global communication and democratized access to information.
This isn’t the first time Costolo has made this promise, mind you. He said the same thing in September, according to reports from his talk at the Online News Association conference which Twitter later confirmed with TechCrunch. At the time, Costolo had said that he would like to see the feature before the end of the year, given engineers’ capacity.
From the sounds of this latest report, the hard deadline Costolo keeps promising isn’t earning him any friends among Twitter’s engineering team working on the project.
“By the end of the year I’ve already promised this, so the engineers – when I promised it publicly they’re already mad at me so they can keep being mad at me,” said Costolo at UM. “Now, again, once again, I caveat this with the engineers who are actually doing the work don’t necessarily agree that they’ll be done by the end of the year, but we’ll just keep having that argument and we’ll see where we end up year-end.”
OK then.
You can imagine why those engineers would be mad – the undertaking is no small matter. At peak times during the election, Costolo noted that Twitter was serving 15,000 tweets per second to 1.3 million timelines (the timeline being all the tweets from the Twitter users you’re following). He says that Twitter’s system is engineered for real-time search and distribution, not archive search and distribution. If you wanted to search against an entire user’s database of tweets, he explained, it would be so slow that it would actually slow down the rest of Twitter’s real-time distribution. To address this problem, Twitter is in the process of building a different type of archival system for these historical tweets.
To put this engineering challenge on an even grander scale, consider this: Twitter now sends a billion tweets every two and half days, Costolo remarked during the same talk. “It took three years, two months and one day for the first billion tweets to be sent. From the time the company started in 2006 to mid-2009 three years, two months and one day for the first billion tweets to be sent. It has a nice rhythm that it was three, two, one. It probably wasn’t exactly one day, but we like to say that; so it’s three, two, one,” he said. “We now send a billion tweets every two and a half days. So the volume, the noise level has increased dramatically.”
Note that Twitter reported a half a billion tweets per day in October, up from 140 million per day in early 2011. It’s also a different way to measure its data than Twitter had in earlier years, when it would often tout the number of tweets, but really mean “items,” in terms of shares and re-shares which have to be shown on other peoples’ timelines.
The rest of the talk has other interesting factoids too, including the below:
- Since Costolo took over the CEO role, Twitter has grown to 500 million users sending over 350 million tweets per day.
- During the World Cup in summer 2010, Twitter was seeing 3,000 tweets per second, and was sweating the system crashes. Recently, it saw 12,000-15,000+ tweets on Election Day in the U.S.
- When the TV show The X-Factor (U.K.) put a hashtag on the screen, it generated 27,000 tweets in 90 seconds.
- Twitter is working with the Japanese government on a “lifeline project” which includes things like making lists of government agencies on Twitter that people can follow in the wake of a disaster or emergency. The project will be expanded to all of Twitter’s markets, with the U.K. and Spain indicating interest already.
- Twitter hands over every government request for the take-down of tweets to the website Chilling Effects.
- Twitter is looking into China. (“Once the new government gets settled there, and once Xi Jinping gets settled in we could go to that government and talk to them. But we won’t compromise the way Twitter works in order to operate within a country,” said Costolo.)
- The service will “never change” its 140-character limit.
- Twitter sees surfacing both the “roar of the crowd” and the “best signal” in Twitter search a “big engineering and design challenge” it still faces.
- Twitter is ramping up on big data visualization. The in-house data analytics team which worked with designers to provide visualizations of Twitter data internally, is now showing more of that data externally – like with the Japanese earthquake, for example.
Hat tip to TechCrunch contributor and CrunchFund VC MG Siegler, who posted a link to this talk on his Tumblr blog, parislemon.com.
Comment