Twitter users, the Twitpocalypse is upon us.
Twitpocalypse is the name given to a bug that’s about to be exposed. Apparently, it’s similar to the Y2K bug in its nature, and stems from the fact that every tweet sent out has a unique numeric identifier. This identifier is about to hit 2,147,483,647. This number is the signed integer limit and apparently when some third-party Twitter clients start hitting it, the identifiers will start turning negative, and those apps are likely to crash as a result.
This crash was supposed to happen sometime tomorrow, according to the countdown, but it looks like Twitter has just moved up the Twitpocalypse time to 21:00 GMT, which is 2 PM Pacific/5PM Eastern time today. Yes, in a couple hours.
They’re forcing the failure now so that all hands are on deck working on the issue, rather than having it go down in the middle of the night.
This Google Group thread run by some developers working at Twitter explains more. Engineer Matt Sanford originally explained:
The overflow of the 32-bit signed integer value for status ids (a.k.a “The Twitpocalypse” [1]) is fast approaching. The current estimate is around tomorrow at around 11am GMT, or 3:00am Pacific time in the case of Twitter. There is some discussion internally about accelerating things so we’ll be in the office and able to cope. Nobody is their freshest at 3:00am, not to mention it would be nice to not have apps broken throughout the weekend if one-person developer teams don’t notice. No decision has been made yet but I wanted to get something out to you all so you know what’s going on in the event we decide to do this.
But now is saying:
The responses to @twitterapi and all discussions internally show a preference to not waiting until the middle of the night. The current plan is to force this issue at 21:00 GMT (2:00pm Pacific/5:00pm Eastern for those in the US). This will let us make sure we have all staff available in the unlikely event something goes wrong on our end. We’ll also be available when people who don’t follow the twitter-dev-talk list start reporting errors. While we did warn developers about the Twitpocalypse I’m sorry we didn’t think about setting a drop-dead date and scheduling this previously. We’ll keep trying to improve on warnings like this. Good night, and good luck.
So yeah, be prepared for some Twitter apps to fail in about 2 hours. Hopefully Twitter will be able to resolve this quickly.
Update: It’s possibly a coincidence, but Twitter has just welcomed two new members to its API team today. Is Twitter manning up for the battle?
Update 2: It’s now past 2 PM and no reports of massive failures yet. Perhaps this really is just like Y2K.
Update 3: Here’s an update from Twitter developer Doug Williams:
Just an update, there is a lot of coordination that it takes to pull something like this off. We need the operations team to watch the servers and application. The services team to work closely with the ops folks to ensure that any problems on our end are properly tracked and fixed. And Matt is running around coordinating the entire effort.
That said, the deadline may slip a bit as we work to ensure that we’ve covered our bases, and that the engineering team is ready to react to unforeseen problems.
Doing what we can to keep the tweets flowing.
Update 4: The crisis looks mostly averted. More here.
Holy Sh**
…ah I don’t worry at all about this…sounds to me like April’1st joke….come on NASA was hacked and twitter can’t be?
Btw: about those twitter apps, hm…bad coding?
Not bad coding…just not thought out on Twitter’s part. They had no clue they’d have two billion tweets.
How do you figure it’s Twitter’s fault?
they created the API
Because any Business Analyst will tell you, you need to think things out throughly. Which they did not. Nor did they make an attempt to deal with this PR earlier, which would have mitigated some of the disgust felt around by some people at companies who do this sort of thing for a living.
They wrote it, they own the problem.
‘Fault’ might be an un-kind way to express it, however.
Nevermind my earlier reply – clearly I can’t read and missed the part about ’some third party clients’.
Duh on me.
Damn. I was hoping this post would be about Twitter exploding. I’m so tired of hearing about it. It really is the poor man’s email and pointless blog service rolled into one. Oh well, here’s hoping it gets abandoned soon.
It’s their fault coz they designed it.
BTW: why would anyone use a SIGNED variable for an ID field? The least would be using UNSIGNED which would defer this “Twitpocalypse” long enough for them to start using 64-bit unsigned.
They don’t really have 2 billion tweets!!
The update of tweet numbers is +1 and +10 alternatively.
The dictionary definition of twit is A foolish or annoying person. I think that describes the userbase very well.
Guids/UUIDs people. Why is that so hard?
@BB Totally Agree!
Kind of makes you wonder about the architects that designed the system. Why would anyone use an integer is beyond me. The only advantage is the data size in the database, but the disadvantages are numerous.
They likely used integer because ruby on rails migrations assume a primary key of type integer by default. Twitter was originally built on rails and the developers probably didn’t anticipate the insane demand. If they had fail whale sightings wouldn’t be so frequent.
Because statuses are incremental by nature. They are on a timeline and the ability to sort by status_id is very useful. GUIDs only make sense when the data is unrelated individually or among groups. Prime example would be RSS feed items. Within each feed there’s a time relation (and the GUID can reflect this), but each feed doesn’t need an ID relative to the rest of the RSS feeds available on the net.
Yeah. Being on a timline means you can sort by the tweet’s time. Why would you need status_id for that?
And yeah, time can be represented as an integer, if you’re worried about indices
You mean tweets don’t have a timestamp to sort by? If that’s true, this is a double fail.
Don’t even really need GUIDS; sender + timestamp is enough to uniquely identify any tweet.
You should never derive meaning for a primary key or sequential ID. In this case the time stamp of when the tweet was made should be used for sorting.
I think history shows that people who think a lot about scaling issues early on end up designing things that never need to scale. It is generally easier to redesign a popular application for scale, than to redesign a scaleable application for popularity.
Good point!
I can think of 3 ways twitter could avoide this but it seems to be they want it to happen.
indeed. maybe they ‘ll even release a ‘fail shark’ this time!. And Techcrunch will host 25 consecutive articles on it.
awesome smack in the face for the “hindsight architects” commenting here!
it’s not hindsight. Using GUIDs is just as easy as integers… most people just do it by default.
GUIDs are bad, because they’re difficult to index.
Try looking at a list of GUIDs and selecting the last ten thousand of them, efficiently, without comparing dates.
Use the timestamps!
OK, GUID people…what kind of toy websites use GUIDs for primary keys? Do you understand the performance implications of checking for a duplicate key on every insert? Of course you can assume that the GUID will always be unique and skip the unique constraint (assuming your db allows that), but why play Russian Roulette with your data?
Twitter made an architectural decision to use an int for a primary key. They are very fortunate to see the day that design decision becomes unsupportable.
This isn’t the end of the world. All they have to do is change the type to long in their db and their code. Deploying that change will immediately break most (if not all) apps consuming their api. Of course the 3rd party developers are going to have a long weekend, and their users will probably have to download updates. Unexpected things happen in life. Deal with it. Move on!
There are 3.4 × 10^38 possible GUIDs. Unless the algorithms for generating them are screwed up, you’re never in a trillion years going to get a duplicate.
MusicBrainz uses GUIDs.
There must be 5x that many tweets by now.
Zoinks! It’s time to stock up the underground bomb shelter. Anyone know how long canned tuna fish lasts? There’s no expiration date.
that is because the tuna fish in the can have already expired. i thought everyone knew that.
So this is only a problem with 3rd party apps using their API? Any news or place to check for which apps have been updated?
Excellent Post!
How stupid is that, a company with $55 million, can’t even manage it’s own application.
I feel sorry for the investors whose situations are even worse than the rest of us
You clearly do NOT understand the issue. This is not a problem with Twitter itself, it’s a problem with the Twitter clients not being prepared for a very large integer. It’s similar to the problem of Unixtime bug coming up in 2036.
You’re not prepared for a very large integer, yet you indeed showed something promising to pull off some 55 million, this is clearly not understandable and to me so stupid.
Do you think users care about Unixtime? You can’t tell your people, we’re sorry but we couldn’t expect 100 million of you. Come on!
Plus why Facebook doesn’t have that kind of problems even if they have more users and data than Twitter.
You really have no clue at all about software, do you?
The problem is solvable, but…shock and awe, computers and programming are not infinite in their capacity to handle data, and sometimes you have to change data types in midstream. App developers will have to update their software to use data types larger than a signed word. If they don’t, their app breaks, because of the very nature of integral data types and binary computing. Get a clue.
ignore the bottom feeders. they failed math back in college.
They had no idea they’d have 2 billion tweets and that’s not your fault? Who created the API to allow 3rd parties to access their software but they didn’t think it will be that much right?
Sean’s response was so good that it’s a perfect response to your response. Reposted here for convenience:
You clearly do NOT understand the issue. This is not a problem with Twitter itself, it’s a problem with the Twitter clients not being prepared for a very large integer. It’s similar to the problem of Unixtime bug coming up in 2036.
lol
uhhhh, clearly you don’t understand the problem in the least.
Yes, users will feel the pain (if they are using a bad Twitter client), but this was a programming gaffe on the part of the *client programmers*, not on the *twitter service programmers*.
This is not a problem in Twitter, it’s a problem in 3rd party Twitter apps that are using standard 32-bit signed integers which maxes out at a little over 2,000,000,000.
Poorly designed 3rd party apps will experience this problem. All that 3rd party apps will need to do is update to 64 or 128 bit integers and they’ll be fine.
This is not a problem created by Twitter, and there’s almost nothing Twitter could have done to prevent this problem. Eventually, you’re going to go over 2 billion if you’re this popular.
Kevin, try reading what I wrote.
The problem is 100% with 3rd party applications, NOT Twitter. This is NOT Twitter’s fault.
Don’t make like you know what you’re talking about when you clearly know nothing about software.
This seems like a great way to surprise everyone with a new feature. Stir up the expectation for “something big” to happen under the ‘guise that it’s negative and then BAM! awesome new feature instead.
Classic under promise over deliver. I would be impressed and squealing like a school girl if this is the case.
remember, you are talking about the people who raised millions of $$ to disguise their inability to keep up a simple text-delivery service under colourful whale clipart.
Dude, you just accused Twitter of having a clever plan. Get a hold of yourself man! You’re not making any sense!
Love the headline. Sensationalistic journalism at it’s best. I’m sure this bug will really go down as hard as even the Y2K bug.
The story that wasn’t.
It’s more like the IPv4 problem…not enough bits to count all of the individual tweets. Simply a case of not thinking your product could ever be as popular as it eventually got.
At least we are all migrated over to IPv6 now, so the Twitter thing should be cleared up pretty quickly.
Oh, wait…
YouTube is already down it’s the beginning of the end!
Can’t they just force a major fail whale and delay until monday.
The problem isn’t on the Twitter end, but in 3rd-party developers storing tweet ids as integers in their database, I gather.
Once you get to a certain point, a straight integer will fall down go boom.
Again: Not twitter’s fault, but poor planning on the part of some 3rd-party developers.
Should we stock up on canned food and water?
that’s “twood” and “twatter” for you
ha. exactly.
Tell us about it…Youtube is already happening! lol
Now, twitter next…gosh!
ALL HELL BREAK LOOSE!
Before everyone jumps on the twitter-is-lame bandwagon, understand this is not a problem with Twitter itself. This will only potentially be a problem for third party apps who are NOT using unsigned integers for twitter status IDs. Any app that crashes because of this is not worth using (the programmer clearly doesn’t know what they’re doing), so if your app crashes, it’s time to find a new one.
Another problem is when they hit the 4.2 billion ID or so. If Twitter is using an unsigned 32-bit integer, they can get up to 4.2 billion. Hopefully they’ve already prepared for that and have switched to 64 bit already, which will basically give them room for growth until the year 20 billion or so. Otherwise, shit will hit the fan internally as well, not just with third party apps.
Doh, serious sentence malfunction. Redo:
If Twitter is using an unsigned 32-bit integer, they can *ONLY* get up to 4.2 billion, *AND NO MORE*
hey thr Sean — if you twitter, it would be great to know your account! visit me at @shinas, thx
They’ll just use BigInteger (and probably already are) internally. The client apps will have to do the same. Simple refactor.
Ok people the solution to this problem is SIMPLE either change the data type to LONG or BIG INTEGER.
If the apps that rely on the api are coded correctly then it should be simple to accomodate this change.
Chirp and let loose the birds of HELL.
And if Twitter clients crash, Where will I bitch about not getting my custom URL?
Excellent point.
This might be interesting…
Or they are just doing the April Fools joke ahead of time…
alter table updates change id id bigint not null primary key auto_increment;
Clearly you have never done that on a large table. You need a much more creative solution than that.
shouldn’t take more than an hour on an innodb table with some million rows. i ‘ve done it
If their database only had a million rows, I doubt they’d be currently overflowing a signed 32 bit int.
We are not talking about some million rows though…we are talking about approximately 2 Billion rows (thus the twitpocalypse). And surely we are not talking about a single table…. rather several tables with forign keys into the status tables. And I’d also bet we are not talking about a single DB server… rather copies of the database spread over tables. Not at quick as a simple alter table statement.
That said… repeating what everyone else has said. Despite the design of Twitter having been called in to question in the past, this is not a Twitter issues. It is a 3rd party app issue where I can bet that several are using signed int data types.
Have no fear though… the sun will rise in the east tomorrow, and the IRS will still be in business. Life goes on.
create table twitter2()?
It’ll only take a few hours for tweets to fill up the new tables. And for low-volume users, viewing their profiles will need to read from both databases (or from the cache), but in a month or so you’ll be exclusively using the new db structure anyway.
The title of this article sounds like this whole world is going stop working for sometime
.
Hope I can sleep well when our twitter guys will fix this bug so no issues for me.
Hope this means all of their servers will explode and we don’t have to read about Twitter stories anymore. MG might be out of a job..
So ham sandwich #2,147,483,647 is the tipping point. Who knew…
@meetingwave I have mine stocked with Vodka. The stuff never goes bad… Who needs food when you have Vodka…
I guess the party is over
http://twitter.com/urbanhack/status/2135672132
which is larger than
http://twitter.com/twitterapi/status/2134819653
I don’t know what the “Twitpocalypse” will bring in a little over an hour, but if it’s bad, let me say that it’s been an honor serving with you
Someone should sponsor a prize for whoever gets tweet # 2,147,483,647.
Since they are forcing it, that wouldn’t as interesting I would guess. That picture of the dead tweetie bird is hilarious though.
Anyone ever heard of a Unsigned Big Int?
18,446,744,073,709,551,615 Thats like 8 billions times more tweets than they already have.
to all the guys out there developing apps…
this is what happens when you get guys who think that really developing apps is easy/simple…
half these guys wouldn’t know a big endian from a lsb if it bit their ass…
much less how to properly allocate memory, when dealing with a shared mem allocation…
brings chips and coke and watches the event live
Congrats! This post is leading in today’s “stuff I could care less about” contest!
Stuff you could not care less about, actually.
http://incompetech.com/gallimaufry/care_less.html
Unless, of course, you actually had the capacity to care just a little bit less about this issue. Perhaps it is more important to you than we thought?
2^32 = 4,294,967,296 – but that includes the range of -2,147,483,647 to 2,147,483,647
why don’t they just switch to BIGINT unsigned? That will give them up to the number 18,446,744,073,709,551,615 which is surely plenty of time to figure things out