Opera, Safari Beat Chrome On Google’s Own JavaScript Conformance Test
MG Siegler
Mar 19, 2010

Back in June, Google launched Sputnik, a suite of tools that runs over 5,000 tests to check a web browser’s JavaScript conformance. Last week, they made the tool a lot easier for anyone to use, with a version that works in the web browser. The results are interesting.

Notably, both the Opera and Safari web browsers beat Google’s own Chrome browser in the test. As you can see in the picture above, Opera is the clear leader, with only 78 failures (the closer to the center, the less errors). Safari came in second with 159 errors, with Chrome in third with 218 errors. Firefox is close behind with 259 errors, while Internet Explorer is the outlier with 463 errors.

These tests were run on Windows machines, with the latest released version of each browser. Using the web tool on my Mac, though, shows similar results (at least for Opera, Chrome, Safari, and Firefox — there is no IE for Mac anymore).

While much of the focus on JavaScript is about speed (that’s what the SunSpider test measures, for example), Sputnik is interesting because it focuses on conformity, making it more like the Acid3 test, which tests web standards compliance. Chrome, Safari, and Opera have all passed Acid3, with Firefox getting very close (94/100 for Firefox 3.6). IE, meanwhile, again lags behind with just 20/100 for IE8. And even the new IE9 preview only scores 55/100.

Speaking of IE9, I tried to run the Sputnik tool in the preview build of the new browser on Windows 7. Unfortunately, it completely shut down several times after getting up to about 50 failures after only a few hundred of the 5,000+ tests — not a good sign. But again, it’s just a very early preview release of the browser, and early SunSpider results for the browser have been good.

Advertisement
Advertisement
  • C.S.

    This is because Safari and Opera are actual browsers, not tinker toys.

  • http://www.facebook.com/profile.php?id=6205900 Andrew Mager

    For some reason I will never use Opera though.

  • http://www.facebook.com/profile.php?id=835705461 Mike Dubnik

    Agreed… there’s just something about Opera I really don’t like. Also on PCs Safari works like poo (in my personal experience).

  • bob e

    Meh I stick with IE8 thanks :)

  • http://allusis.net Tony

    Interesting for sure. Internet Explorers stats always make me smile :)

    OSX 10.6 / Chrome 5.0

  • http://www.facebook.com/profile.php?id=701561804 Nicolas Charbonnier

    Check out my video interview with a representative of Opera Software where he says that they are faster than Chrome and the Android Webkit based browser: http://armdevices.net/2010/02/16/opera-10-for-android-nexus-one-and-the-iphone/

  • Cata

    What about the Internet Explorer 9 platform preview? I did not get the test to run on it. Anyone else?

  • http://www.facebook.com/profile.php?id=661360706 Gary Wilkinson

    I don’t think Operas GUI fits in nicely with any operating system it always seems so foreign. If they sorted that out I’d be more interested as they’ve always been very innovative.

    Really can’t see how people still use IE8 though, it’s not that it’s bad it’s just Chrome and Firefox are just better.

  • Jacob

    You should check out Opera 10.50. It looks a lot better now (not as nice as Chrome though, imo).

  • http://meliondesign.com Damien

    That’s the case of Opera 10.50 which brings native GUIs in each operating system… instead of Qt only like it used to be.

  • Gleb Esman

    I always loved Opera – this was the only browser + tools that allowed be to properly debug AJAX/javascript.

    Kudos to the whole team behind it.

    Without much buzz they delivered exceptional product!

    Gleb,
    http://www.memberwing.com/

  • Elliot

    what about IE 6 what the hell is wrong with you people leaving out IE 6

  • http://www.pleft.com/ Sander

    MG, from the title it seems you miss the point of the 2-D graph. Quoting the explanation on Sputnik’s homepage:

    “The distance between browsers shows how different they are; the more similar two browsers are – the more test failures they have in common – the closer they will be.”

    So in this test, Chrome does a fine job by being compatible with the other three modern browsers, while Safari seems to act too different from Firefox.

    So a browser does not win by getting nearer the center (unless all other browsers have the same goal), but by being the center.

  • Rankhar

    Maybe they did include it! But the result is located somewhere outside of your screen…

  • Damien

    That’s right we should not follow standards… but happily do the same mistakes as everyone!

  • toddq138

    Also an interesting browser to use is Lynx
    http://en.wikipedia.org/wiki/Lynx_(web_browser)

    Despite its text-only nature and age, it can still be used to effectively browse much of the modern web, including performing interactive tasks such as editing Wikipedia. The speed benefits of text-only browsing are most apparent when using low bandwidth internet connections, or older computer hardware that may be slow to render image-heavy content.

  • toddq138

    I also want to mention that the text-based browser Lynx released their recent version in the middle of 2009…For comparison sake, Internet explorer 8 came out at the beginning of 2009.

  • alma

    Safari Windows != Safari MacOSX
    Safari Windows is so buggy, you can crash it with 4 lines js

  • Harish

    Are there any tests to compare the CPU and memory utilization for flash content on these browsers?…that IMHO will be very interesting.

  • http://tweetprivate.com Shan

    From which year you started working for Microsoft?

  • http://librarianchat.com/forum/ librarianchat

    I’ll stick with Chrome.

  • http://www.xilnex.com bsooi

    Totally Agree with Rankhar.
    IE6 is far too behind :D

  • http://www.xilnex.com bsooi

    Both IE and Firefox will hit very high memory usage after two days of not restarting it.
    Anyone has the same experience?

  • gobo

    Opera rocks, it’s fast and sharp.
    BTW ver 10.1 gets acid3 100/100 on my linux machine

  • http://hellologic.com Nathan J. Brauer

    Because it’s straight up ugly.

  • http://www.techbusy.org/ abhi

    Lol. Shame for Google.

  • Municipal Hare

    I LOL’d.

  • Jon

    Firebug is leagues better than Dragonfly or the one for Chrome/Safari (whatever it’s called).

  • http://youdidntdidyou.com/ YouDidntDidYou

    this is funny because I’ve been using Opera for quite awhile and even when you tell it’s faster/better than the rest they never believe you, same with Four Thirds cameras from Panasonic and Olympus…some people are so entrenched.

  • http://talential.com Manuel

    the newer versions are not so ugly anymore. i use chrome and opera. although i like chrome for its simplicity, design and speed, i use opera too as it is considerably faster on some pages. at least it feels that way…

  • http://www.optimizedir.com Farooq Azam

    There are many other factors too, one should not judge only on the basis of javascript conformance.

  • len

    Opera is currently the fastest browser…. http://bit.ly/opera-fastest-browser-doh

    Any violent reactions?

  • Goofball Jones

    It’s interesting that this “article” seems to uncover something when in fact the techcrunch’s Siegler just grabbed all his data from Google’s own Chromium Blog.

    The Chromium Blog wasn’t even trying to cover anything up. They ran the tests and posted the results. The little image above is straight from Google. So I think they’re being very open about it all, not trying to cover up anything and showing they still have a bit of work to do.

  • Richard

    Do you feel better now?

  • http://www.marketingtactics.com Dave Barnes

    @MG,
    “the less errors”
    I am sure you meant to write “fewer errors” as an error is a discrete object.

  • Bard

    I’m sure you have not seen the new Opera UI yet. Stunning.

  • http://www.mattcutts.com/blog/ Matt Cutts

    My favorite comment of the day. :)

  • http://xboxmansion.com David

    This test makes me wonder how soon Internet Explorer will be dropped by the internet community.

  • Robbe Block

    Has anyone bothered to look at the kind of tests that fail using Chrome? I’m using Chrome 5.0.342.5 dev on Ubuntu and I just ran the tests and 220 of them failed, but a lot of them look like this:

    /**
    * @name: S7.5.3_A1.13;
    * @section: 7.5.3;
    * @assertion: The “float” token can not be used as identifier;
    * @description: Checking if execution of “float=1″ fails;
    * @negative
    */
    float = 1;
    testCompleted();

    My guess is that in this case, the test checks whether the call to testCompleted gets executed or not.
    As you can read from the comments: “float” can’t be used as an identifier and I think whether that the evaluation should stop there, and thus not call ‘testCompleted()’; but as browsers need to be robust, I don’t think it’s so bad that Chrome just skips the statement and still calls testCompleted.
    It can be that I’m wrong about this test, imho you cannot look at the quantity of tests that failed to see if it is a good JavaScript engine or not, the severity of the tests also matters. (I’ld rather use a browser that fails to render all of the fancy effects implemented by jQuery than a browser that has only one major security related issue.)
    Apart from that, as Farooq Azam states above me: there are so many factors that determine whether a browser is good or not, don’t look only at the javascript conformance.

  • http://www.facebook.com/profile.php?id=508107036 Justin Hubbard

    The thing is… chrome shouldn’t be beating anyone, its much younger… pretty sad that it has become the standard to beat when safari ie, Firefox and opera all been out years… chrome is doing great for its age

  • http://www.gildus.com.ar/gildus/?p=1281 Gildus» Blog Archive » La prueba de Sputnik de Google.

    [...] TechCrunch. [...]

  • http://www.facebook.com/profile.php?id=100000903121998 John Bill

    What ever it is but I am with Chrome

  • Adnaan Badr

    oprah…oww d talkshow lady…she is into browsers too..hmm..good for her

  • http://newmoviex.net/?p=15 The Final | FULL HD

    [...] Opera, Safari Beat Chrome On Google’s Own JavaScript Conformance Test [...]

  • lucideer

    You may be right that weighting tests by perceived significance would give a better indication, but there are over 5000 tests – it would be a huge (and also contentious) task just to classify them. Right now, it gives an excellent indication, as is – certainly better than many of the binary “support/doesn’t support” conformance comparison pages for other web standards such as CSS and DOM.

  • http://www.chromechina.com/thread-1831-1-1.html Chrome的JavaScript符合性测试不敌Opera和Safari – Chrome论坛

    Chrome的JavaScript符合性测试不敌Opera和Safari – Chrome论坛…

    Chrome的JavaScript符合性测试不敌Opera和Safari – Chrome论坛…

  • http://nealabq.com/ nealabq

    Browser Zen: Don’t strive for the center. Be the center.

  • Sam

    They fail because they do not allow code execution

  • Anjali

    I found it beyond amusing that Google was actually forced to mention Opera in this conformance test.

    Their normal strategy for years has been to be silent on ever mentioning Opera and always never reveal Opera’s equal or superior standards-compliance, hoping Google will strangle it…

    Notice, even now, after Opera 10.5′s superiority on a myriand of others’ tests (all subjective, artifical and HIGHLY fallible), Google fails to list Opera as a supported browser for their web apps, as tested. They list Opera v6.03 and say they don’t test Opera v9.5+. Nice. (Any reply, Matt? I thought so.)

    As Steve Jobs said: “‘Do No Evil, my a–.”

    Btw, check out how they have to deal with instrasigent or unhelpful or lazy web site designers via Opera’s innovative browser.js tweaks.

    If you check out Opera v10.5, you will be shocked how far ahead — it still is. Remember, every negative, astro-turfing comment you read against Opera is written because, if you actually use it for a few days, you can’t believe how innovative it is…and then, you discover 100 newer, hidden features that others don’t have or will copy later:

    o paste & go
    o integrated Rewind
    o Fast-forward
    o Opera Turbo
    o on-demand plug-ins
    o fit-to-width
    o scroll markers
    o find headers
    o sync typed history & search shortcuts
    o single-char bookmark nicknames
    o OTA-sync with Opera Mini/Mobile
    o server-name-completion –never typing in .COM or .NET or any gTLD
    o plus 900 more user speed-ups

    And then it knocks you over again…you get the idea.

    And then it dawns on you that Opera has been doing this for their impressed users…for FIFTEEN years.

    @MG: Would be cool, if you or TC, in general, had a follow-up about why Google (and others) choose to not do the trivial testing for Opera, given its very accommodating standards-compliance. (And, no, anyone saying, “…having *only* 110 million desktop and mobile users” is a cop-out.)

  • http://www.facebook.com/profile.php?id=546326314 Alex Graham

    Maybe help the environment and shut down that computer, unless you’re awake for 48hrs straight ;) While it is interesting, it’s redundant in the real world…

  • http://appleinforma.com/2010/03/21/safari-segundo-en-el-test-de-google/ Safari, segundo en el test de Google | Apple-Informa

    [...] TechCrunch Etiquetas: safari ¡Comparte este [...]

  • operakat

    Yes, good point. Google only has $25 billion total cash, with net income of ~$110 million per WEEK.

    Their resources (and motivations) are so limited to build a browser that makes their web apps more usable (as opposed to the behemoth, Opera and its pokey 10-week Opera v10.5 dev cycle with full rewrite of JS Carakan, SVG Vega and Presto rendering engines — along with Win7 UI integration and their continuing full-Cocoa rewrite for OSX).

  • http://www.robbennet.com Rob Bennet

    I never comment…but that was funny!

  • Confused

    my guess is that M.Ghandi photo shopped the IE icon on that chart being an Apple/Google whore an all.

  • Confused

    in fact just remove the Opera logo and that MG’s life in a single diagram.
    pretty sad really.

  • http://themacfeed.com/2010/03/the-weekend-update-march-21-2010/ The Weekend Update (March 21, 2010) | TheMacFeed

    [...] downloaded the 10.6.3 update. Steve Jobs is now pushing for organ donor legislation. Chrome didn’t do so well when it came to Google’s JavaScript test. Apple 2.0 is now reporting that 190,000 pre-orders [...]

  • http://scbsradiolombok.wordpress.com SCBS Lombok

    The best part of Opera, at least to me is the feature of Opera Turbo. Incredibly fast for text browsing. I love it so much.

  • http://www.ArticlePlayground.com/ Article Playground

    The javascript in my browser for flash shockwave is not working, and i really need to upload some videos to a real cool and fantastic online community. Anybody know how I can fix my browser’s shockwave flash plug in, besides uninstalling it?

  • http://scbsradiolombok.wordpress.com Hana

    Im glad to hear my pav browser Opera beat it. Truly ashame of google

  • Craig

    Why is it a shame for Google? I like the fact that they have transparency. They have open-sourced most/?all of the Chrome browser because they want to raise the bar on the performance and standards compliance on ALL browsers. It’s in their interests to do so, as by making the web browsing experience better, people will spend more time in their browsers. Anybody who thinks that they are in it just to make Chrome rank #1 in browser share hasn’t thought about it very much.

  • http://dan.cx/ Daniel15

    I definitely agree with you, I’ve been using Opera since version 6 was released. I’d never switch to any other primary browser :)

  • http://myguidetogolf.com/?p=113 Garmin Mobile PC with GPS 10x Bluetooth GPS Receiver for Laptop Computers | MyGuideToGolf.com

    [...] Opera, Safari Beat Chrome On Google's Own JavaScript Conformance Test [...]

  • http://jsridhar.blogspot.com SJ

    When fox came out I liked it a lot better than IE. I have been using it 5 or 6 years now. Each new release or even beta I would install. IE just looked to be really really really outdated. Then Chrome came into existence ans slowly I find that I am moving to Chrome.

    FF slow startup and memory hogging was attributed to the fact it supports extensions and it was my extensions not the browser itself that was slowing things down. Guess what Chrome got extensions and still starts up almost as soon as I click on the icon. Chrome is getting better all the time.

    P.S: I used Netscape at one time at that time it was better then IE. And had a neat HTML editor – Composer – I don’t miss teh browser but the composer I wish for at times :)

  • noob

    or, you could use a rock.

  • http://www.etondigital.com/internet-explorer-1-browser-for-innovation-compatibility-and-reliability-and-no-im-not-being-bribed-to-write-this-piece/ Internet Explorer: #1 browser for innovation, compatibility, and reliability! (And, no – I’m not being bribed to write this piece) | etonDIGITAL

    [...] is that IE continues to lag behind its competitors at the moment in technical performance (check for example this recent test of Java competence of rival browsers on Techcrunch: no surprise I…). However, amidst all the anti-IE fervour that sometimes fills the blogosphere, it’s worth [...]

  • IE 8 fails

    How much did microsoft pay you?

  • http://disconnectedneuronactivity.wordpress.com/2010/03/23/sputnik/ Sputnik « disconnected neuron activity

    [...] Today I found Sputnik by Google, which measures JavaScript conformance. Interestingly, Chrome is no longer the top on this test. [...]

  • http://11tech.wordpress.com/2010/03/27/browser-benchmark-sputnik-mag-opern/ Browser-Benchmark Sputnik mag Opern « 11tech

    [...] (via techcrunch) [...]

  • SteveT

    Love Opera myself & 10.51 is the fastest browser discounting beta’s but why should Google be ashamed it’s their test & hopefully it will encourage Browsers to adhere more closely to the standards.

  • http://intensedebate.com/profiles/nusret1 yuregininsesi

    this is funny because I've been using Opera for quite awhile and even when you tell it's faster/better than the rest they never believe you, same with Four Thirds cameras from Panasonic and Olympus…some people are so entrenched

  • http://www.diamondphotography.org.uk Diamond Photography

    Over the years I've used many browsers, but keep going back to Firefox. IE is probably the best for comparability but i get far fewer attacks using FF. I did use Opera for a little while but one of my favorite programs (RoboForm) was not compatible so went back to FF.

  • http://www.sesligoruntulu.com sesli goruntulu chat

    hobaa sapii sapii kops kops iste hepsi bu opera falan ise yaramaz explorer en iyisi

  • http://www.nimalasvideoreview.com/ Nimala

    I like using firefox and chrome.

blog comments powered by Disqus
Advertisement
Got a tip? Building a startup? Tell us