For Microsoft, Google’s RoboHornet Benchmark Isn’t Real-World Enough, Launches Its Own Version

Yesterday, Google launched the alpha version of RoboHornet, a new open source browser benchmark that, as Google engineer and project lead Alex Komoroske put it, is “a living, dynamic benchmark that aims to use the collective efforts of the web development community and ultimately get browser vendors to fix real-world performance pain points.” Microsoft’s IE 10 scores extremely well in this test. Nevertheless, the company’s engineers clearly believed that the test wasn’t “real-world” enough and today, Microsoft launched a new version of RoboHornet that expands the test to simulate typical browser scenarios better.

According to Microsoft’s director of Internet Explorer marketing Roger Capriotti, Microsoft’s own RoboHornet Pro, which takes its visual cues from the Matrix movies, takes RoboHornet’s micro-benchmark approach and runs it “in the context of a real-world scenario.”

Most of today’s browser benchmarks like Sunspider, Kraken and Google’s recently launched Octane focus on JavaScript, but in Google’s view (and that of many other browser developers), JavaScript is rarely a bottleneck for modern web apps.

RoboHornet Pro uses modern web technologies like CSS3 Animations, CSS3 Transforms, CSS3 Text Shadows, custom WOFF fonts, Unicode, and Touch. Running Microsoft’s tests, Chrome – as expected – performs significantly worse than Internet Explorer (and Firefox, too, beats Chrome in our strictly non-scientific tests). Chrome, says Microsoft, “wasn’t designed to handle a benchmark load in the context of a real-world scenario.”

Most vendor benchmarks, of course, favor their own developers’ products, so the fact that Microsoft does much better here doesn’t come as a big surprise. Microsoft has consistently said that real-world performance matters more than benchmarks. If anything, though, the rivalry between the top browser vendors and this focus on speed has pushed browser performance forward considerably over the last few years.