Amazon chief technology officer Werner Vogels just released a very technical paper about a Web storage system being tested internally at Amazon called Dynamo. As applications move to massive grids of computers on the Web (like the ones that power Amazon’s e-commerce site or Google’s search engine and online apps), a new type of Web operating system is developing that treats all of those connected servers as one big computer. (Engineering types can download a PDF of the paper or read it online at the Dynamo link above).
Dynamo is not an alternative to S3, Amazon’s publicly-available data storage service, or competing Web hosting/storage services like Nirvanix and Flexiscale (see previous post). There are no plans at this point to offer Dynamo as a Web Service. It is an internal-only project that sounds like a rethinking of what a relational database should be when computing scales to massive Web proportions (i.e., systems running on tens of thousands of computers).
As Nick Carr puts it:
At the start of the last century, the great engineering project was the creation of an electric grid that could deliver power to millions of users with a reliability and an efficiency that were previously unthinkable. Today’s great engineering project, of which Amazon’s Dynamo is but one manifestation, is to build a computing grid that can achieve similar breakthroughs in the processing and delivery of information.
The race to create a Web operating system is heating up. It is such a huge undertaking that there are only a few companies—Amazon, Google, Microsoft, Yahoo, IBM— that can tackle it. But this time around, it is unlikely that any one of them is going to own it outright.