The graph above is hilarious. It represents the way in which people rate videos on YouTube. As you can see, there are some 1-stars and a huge amount of 5-stars, and then basically no 2, 3, or 4 stars. As such, YouTube has a blog post today admitting that maybe its star system isn’t the best way to vote on videos.
Of course, anyone who has used YouTube for an extended period of time will already know this. And really, the same seems to be true of basically all 1 to 5 star crowd rating systems. It’s easy to know if a video (or anything) is good or bad, but how on Earth do you determine if it’s 2 star, 3 star, or 4 star-worthy? Everyone likely has their own opinions about what would constitute those ratings, and naturally, they’re all completely subjective.
Apple’s App Store faces a similar problem. And it also shares another issue that YouTube faces, most people are probably only going to vote if they absolutely love or hate an app (or a video). Hence, the 1 and 5 star usage.
YouTube asks in its post, “Would a thumbs up/thumbs down be more effective, or does favoriting do the trick of declaring your love for a video?” Yes, the two vote option (thumbs up/thumbs down) or the one vote option (favoriting) are both better methods because they’re more defined.
Zagat’s subscriber-based numerical rating system works pretty well in major markets like NYC. A NYC 25 is usually much better than a NYC 20.
Zagat voters, however, don’t vote on a 30 point scale – I think it’s only a 0-3 point scale in the submission forms. (They then average all votes and multiply by 10.)
I’ve always been a bit against the rating system, but it’s also set the standard for a lot of other sites out there which now have the exact same kind of thing. The only site that comes to mind in which the five-star system actually works somewhat well is Amazon. Other than that it’s pretty worthless.
I would go for the two-vote option. I’m not going to favorite videos that don’t pertain to me just because I think they’re funny. Once I absorb someone’s content, unless it’s viral or it is mentioning me in it, I usually don’t need to return. So why add that video to my favorites if it really isn’t?
Yeah, I feel like people put time aside for thinking about ratings in proportion to how much time they spend with what they’re rating. For example, you buy something on Amazon for $20–that’s a huge chunk of change, and you’re going to want to talk about the product. If you don’t like it, you might also feel obligated to prevent other people from making the same mistake of purchasing that product. On the other hand, people go through tens of Youtube videos at a time–unless you’re trying to flag the video as a MUST see or a MUST NOT see (after all, all that’s at stake is ~30 seconds on average for the next guy), it’s not going to be worth it for you to do a middle-of-the-road critique and actually decide between 2, 3, and 4 stars. I think that’s what explains the ambiguity of the middle ratings.
finally MG! there’s a fundamental difference between ratings and rankings. why has it taken so long to grasp such a simple concept?
ratings are extremely subjective (without defined definitions) and very limiting. On the other hand, rankings actually differentiate between different options telling you which one is better than the other. for one, i know http://www.flickchart.com asks this very question.
i hate to promote on TC boards, but it’s hard not to when that’s EXACTLY what we’ve created with Rank ‘em (www.gorankem.com). it’s a crowdsourced ranking platform that allows music fans to rank their favorite songs from all their favorite artists.
When you combine the collective intelligence opinion for each artist, it creates a pretty powerful resource for all of us to use whether we are the most casual music observer or the most diehard fanatic.
we’re in private alpha at the moment, but feel free to request an invitation.
Urbanspoon is easier to use than Zagat. They use Thumbs up/down and it’s free.
Agreed. What’s more, if you think the 5 star system is unsuited to YouTube videos (which don’t change over time), they’re even more unsuited to restaurant/bar reviews which are a lot more volatile.
If you’re interested, I discussed this very topic a few months ago: http://www.thebuzz.at/blog/2009/05/why-yelp-has-it-all-wrong/
They should show the distribution on each video (i.e. 1 stars: 5, 2 stars: 10, 3 stars: 4, 4 stars: 10, 5 stars: 30) except do it with a LIKE, NEUTRAL, DISLIKE (3 star system) since most videos will get either a good rating or a bad rating or no rating at all.
Won’t lie to you guys, the rating systems that are being used are pretty silly. Starring in my opinion is dumb. I think the options should be based on language people actually use to signify their level of interest in something. Simple as that. Thumbs up and down is too narrow. They had it ok with the range of the starring. But starring doesn’t matter to anything else except for hotels and restaurant. And those have actually taken up the context of starring.
Everything else trying to attach itself to starring mechanisms is an unwanted distraction.
I completely agree. Rating systems should use the natural language people use when they tell a friend about a video they just watched. That was “awesome” “stupid” “interesting” “hilarious” whatever. YouTube could just capture these in a separate text box when you’re posting a comment and then show these below the titles when you’re scrolling down a list of videos. “Cat falls asleep on fan” might have “Top Impressions: Hilarious (32) Cute (19) Funny (8).” “New Musician lights up the stage in LA” might have “Top Impressions: Awesome (41) Sick (19) Talented (7)”.
This seems like a much better way to let people know what to expect when they watch a video than just saying it has 3.5 stars.
Like Buzzfeed does with their site (OMG, WTF, Cute, etc.)
For sites like youtube, where people just watch content for fun a two star/ one start system suits more. Either I like the content or I dont.
But for topics that I am very critical/passionate about I would want to rate it on a more granular scale. If I am a food enthusiast or a fashion guru and if I am asked to rate an item/design I would prefer a wider scale than just a yes or a no.
But for youtube I just dont care. Either the video is awesome or just not worth my time,
It would seem to be obvious that the percentage of people that do not bother to vote is itself a useful metric that can be figured into the total score. You would just have to ascertain what a no-vote implies in aggregate. Thumbs up, thumbs down, and no-vote should be a sufficient basis for a simple rating.
I’m a fan of the two-vote option, or even a three-vote option… dislike, neutral, and like. I don’t think I’ve ever given anything a 2 or a 4 rating.
5 star rating doesn’t work for a site like youtube because the stars are good for the person who made the video rather than the person who is viewing the video.
On a site like Netflix you wouldn’t 5 star a bad movie because you don’t want it recommending crappy movies to you. If the star system put more weight on the person rating it rather than the person who posted it, it may be more valuable.
Netflix does very well with a 1-5 system, and I see movies across the spectrum, with the top of the bell curve around 2.5-3, just as you’d expect. It’s not an inherent problem with a 1-5 rating system.
Netflix would do much better with just a Lat.fm type “Love/Skip” feature. Or Thumbs Up/Down.
@Charbax: Please do not change the Netflix staring system.
I think Netflix rating works because it uses clearer terms…
Hated it.
Didn’t like it.
Liked it.
Really liked it.
Loved it.
While YouTube is…
Poor.
Nothing Special.
Worth Watching.
Pretty Cool.
Awesome!
Netflix guides you towards the middle with “Liked it”… Youtube guides you towards the end with “Awesome!”
And who is to say that awesome is better than pretty cool?
“That was pretty cool dude!”
“That was awesome dude!”
Both phrases could said using the same tone and inflection and mean the same thing. There’s not enough difference.
But take the phrases…
“I really liked that movie!”
“I loved that movie!”
That’s a difference I can easily comprehend, and therefore, my ratings on Netflix are easier to make, which is why the rating systems works so well on Netflix, and not so much on YouTube.
I propose that YouTube change their “descriptors” to what Netflix uses and see if people rate things differently.
Good point. Never thought about it that way but it’s true.
You and @Balpreet up above have points that work well together. The rating system is really dependent on the content, and clarity of the rating system. I’d also add in the use of the rating in relation to the user.
Site’s like YouTube are generally:
* short form content therefore “hit/miss” – harder to get more granular ratings
Where as a site like Netflix has:
* long form content with nuances – easier to get granular ratings
* clear rating descriptors
* clear understanding that my more specific ratings == better recommendations for me
In my experience, the Netflix star rating system also seems to work reasonably well.
The only 5 star system that comes to mind that works is the Netflix. Having it average out user ratings works well for me and is (usually) pretty acurate.
5=Loved, this is one the best i’ve ever seen
4=Yeah, this is a good movie, I’d watch it again
3=So-so, wouldn’t watch it again
2=I only finished the movie cuz I thought it might get better
1=Couldn’t even finish
In the App Store it just doesn’t work because half the amount of 1/5 voters are idiots. When they uninstall something they just downloaded in some promotion or just because it was free or cheap and later notice that they don’t need it, they give it 1/5 when the rating dialog pops-up. So the ratings system totally fails there. Maybe a more in depth review would be the answer with some basic questions which calculate a rating. In this way people who don’t care enough to give a decent rating just don’t rate at all because they are too lazy.
Of course this doesn’t apply to Youtube.
since the yahoo post reminded me of how yahoo music did it, with their 5 star system that worked, i think youtube just erred going the 5 route instead of a 2 or 3 star system. i actually think their up or down votes are pretty irrelevant but since they are moving on to feature video’s they could have a system where they feature the most voted video of the day on their front page. i think it might be too easy to game that incentive though.
I give this article 2 stars
5 Stars for me.
Just rate it by how long the user watched the video – if it sucks, they’ll stop watching
This is a very good point. I think that five start systems are supposed to work on the basic idea that the one and five votes will average out to a value somewhere in the middle. In this way the two, three, and four star ratings are averages based on the ratio of one star votes to five star votes.
However, in general it seems that most people just vote the same. They go along with the crowd so you don’t get that averaging out.
One place where you got a consistent difference in the ratings on YouTube was the last election. The ratings for the Obama videos were uniformly between 4 and 5. The ratings for the McCain videos were very close to 2.5. They all liked the Obama videos. Only half of the viewers liked the McCain videos.
The 5 star system should be scrapped in favour of something like the Zune rating system. You either love it, hate it, or you don’t rate it.
They should use the Zune system, you either love it, hate it, or don’t rate it.
Yup, that’s the only way!
If you think something is so-so, you simply are just not going to rate it.
This is a very good point. I think that five start systems are supposed to work on the basic idea that the one and five votes will average out to a value somewhere in the middle. In this way the two, three, and four star ratings are averages based on the ratio of one star votes to five star votes.
However, in general it seems that most people just vote the same. They go along with the crowd so you don’t get that averaging out.
How about simply checking how long a video is watched before quitting?
Google obviously does that already.
The “Love It” “Hate It” rating is much better.
Yup, like Last.fm has Love/Skip. That would be very good for watching playlists on Youtube. A shortcut on the keyboard like the Space bar or Enter button should make you rate “Love” and the ctrl button could be used to skip the video which also basically tells Google “Don’t show me this type of stuff again”. After a while Google will be able to generate a personal TV for you with one awesome video after the other and it will replace TV.
One place where you got a consistent difference in the ratings on YouTube was the last election. The ratings for the Obama videos were uniformly between 4 and 5. The ratings for the McCain videos were very close to 2.5. They all liked the Obama videos. Only half of the viewers liked the McCain videos.
This is a very good point. I think that five start systems are supposed to work on the basic idea that the one and five votes will average out to a value somewhere in the middle. In this way the two, three, and four star ratings are averages based on the ratio of one star votes to five star votes.
However, in general it seems that most people just vote the same. They go along with the crowd so you don’t get that averaging out.
Even — | – | ± | + | ++ is much better.
Don’t know, why anyone should rate a bad video with a star (improve it)! This is ridiculous!
two points:
the high number of 5-star ratings on the site is actually a symptom of the viral nature of the site. although some people surf youtube looking for videos, the majority of the views come from people following links to the latest must-watch video. naturally, these videos are highly rated.
secondly, there is little incentive in rating a video in the first place. if users received a benefit for doing so, they would likely put more thought behind the value given. in that case, there would more likely be more 2-, 3-, and 4-star ratings.
two points:
the high number of 5-star ratings on the site is actually a symptom of the viral nature of the site. although some people surf youtube looking for videos, the majority of the views come from people following links to the latest must-watch video. naturally, these videos are highly rated.
secondly, there is little incentive in rating a video in the first place. if users received a benefit for doing so, they would likely put more thought behind the value given. in that case, there would more likely be more 2-, 3-, and 4-star ratings.
Thumbs are definitely the way to go… here is a great example of the Best TechCrunch Videos.
Yesterday, I finished writing a fantastic, modular AJAX / Scriptaculous / Prototype class for my site. Now I’m likely scrapping it all and going to thumbs up / down system. Thanks TechCrunch…..