Twitter CEO Dick Costolo has once again promised that Twitter users will, indeed, be able to download a full archive of their tweets in just a matter of weeks. This latest announcement was delivered in response to an audience question during a talk sponsored by the Ford School of Public Policy and School of Information at Costolo’s alma mater, the University of Michigan, where the CEO discussed Twitter’s role in the future of global communication and democratized access to information.
This isn’t the first time Costolo has made this promise, mind you. He said the same thing in September, according to reports from his talk at the Online News Association conference which Twitter later confirmed with TechCrunch. At the time, Costolo had said that he would like to see the feature before the end of the year, given engineers’ capacity.
From the sounds of this latest report, the hard deadline Costolo keeps promising isn’t earning him any friends among Twitter’s engineering team working on the project.
“By the end of the year I’ve already promised this, so the engineers – when I promised it publicly they’re already mad at me so they can keep being mad at me,” said Costolo at UM. “Now, again, once again, I caveat this with the engineers who are actually doing the work don’t necessarily agree that they’ll be done by the end of the year, but we’ll just keep having that argument and we’ll see where we end up year-end.”
You can imagine why those engineers would be mad – the undertaking is no small matter. At peak times during the election, Costolo noted that Twitter was serving 15,000 tweets per second to 1.3 million timelines (the timeline being all the tweets from the Twitter users you’re following). He says that Twitter’s system is engineered for real-time search and distribution, not archive search and distribution. If you wanted to search against an entire user’s database of tweets, he explained, it would be so slow that it would actually slow down the rest of Twitter’s real-time distribution. To address this problem, Twitter is in the process of building a different type of archival system for these historical tweets.
To put this engineering challenge on an even grander scale, consider this: Twitter now sends a billion tweets every two and half days, Costolo remarked during the same talk. “It took three years, two months and one day for the first billion tweets to be sent. From the time the company started in 2006 to mid-2009 three years, two months and one day for the first billion tweets to be sent. It has a nice rhythm that it was three, two, one. It probably wasn’t exactly one day, but we like to say that; so it’s three, two, one,” he said. “We now send a billion tweets every two and a half days. So the volume, the noise level has increased dramatically.”
Note that Twitter reported a half a billion tweets per day in October, up from 140 million per day in early 2011. It’s also a different way to measure its data than Twitter had in earlier years, when it would often tout the number of tweets, but really mean “items,” in terms of shares and re-shares which have to be shown on other peoples’ timelines.
The rest of the talk has other interesting factoids too, including the below:
Hat tip to TechCrunch contributor and CrunchFund VC MG Siegler, who posted a link to this talk on his Tumblr blog, parislemon.com.