Toodledo, a basic but nifty free online to-do list management tool, is having a very bad day. According to a message posted on its website, a storm hit the city where their datacenter is located, prompting the engineers to switch to power generators.
This caused Toodledo to wind up with a corrupted database, which it is currently frantically trying to restore.
So, here’s the story. A big storm went through the city where our datacenter is located. The datacenter decided to proactively switch to generators. During the switch, something got screwed up, and the power went off for a few minutes. As (bad) luck would have it, this caused our database to get corrupted. We are currently working to bring it back online and restored from the live backup. The crack team at Rackspace is on the job. Thanks Rackspace! Unfortunately, the database is so large, that it will take some time to transfer and verify all the data. Hopefuly not more than a few hours. We know that this is very bad, and we apologize for any inconvience that this will cause. Please check the forums when we are back online for a full report.
Update: Its obviously taking longer than we expected and we are really sorry for that.
Update #2: I know that our downtime is totally unacceptable. I am literally sick to my stomach about this whole thing. We have successfully restored all of the data up until June 10, 4am CST and we are working very hard to restore the remaining 12 hours of updates that happened before the servers crashed. Obviously, this has been a major event. We had multiple simultaneous errors happen that have made recovery slower than anticipated. Because our database is so large (many many many gigabytes), it just takes a long time to transfer and import the data. I know that many of you are upset, and all I can say is that I am deeply sorry. We are already starting to rethink our backup strategy so that recovery doesn’t take so long in the future.
Update #3: Its coming along. I know it is taking forever. We have restored up to June 10, 11:30am CST. At this rate, it should take about *crossing fingers* 2 more hours. We’ll then run some sanity tests and hopefully go live. Again, I can’t tell you how sorry I am about this. The good news is that we shouldn’t have lost any data.
I feel genuinely sorry for Jake Olefsky, the sole independent developer behind the Toodledo service. If this is any comfort to him: an Amazon datacenter got hit by a lightning strike this morning, causing its EC2 service to suffer a fairly major outage.
Or how cloudy weather can have a devastating effect on cloud-based services.
(Thanks for the tip Gavin)