Ratings

The internet is moving to a very democratic place where for all sorts of content there is the possibility for comments and ratings. This is generally a good thing, but transparency in the whole system is pretty critical because frankly most people don't have a clue how to participate in a rating system.

I was poking around on Netflix and came across a user review that started with this:

I gave it a 5 star review because I love it. I may have given it an 8 on a 10 star scale, but since I only get 5 stars thats what its getting.

And then later in the same comment:

I can see where people would rate this lower but I rate things based on if I loved it and could watch it again. So all in all, okay 3D effects, obvious plot, and some of the acting is sub-par but I still loved it.

Um, yeah. I'm glad that Netflix tells me that this person is only 21% similar to me. But I'm still kind of bitter that their rating gets averaged in with everyone else's and in the end result is something that I really can't trust.

It's extremely important to establish a consistent description of what different ratings mean. Netflix does indeed include a description in the tooltip:

1 - Hated It
2 - Didn't Like It
3 - Liked It
4 - Really Liked It
5 - Loved It

Unfortunately these definitions leave a bit too much to the imagination. I mean, I love my wife and I love ice cream, yet these are not the same kinds of love. But the problem is that both are totally valid uses of the word. There's the love that's about deep meaning, and there's the infatuation that is just wanting to experience something over and over again. When I rate my Netflix movies it's based on some combination of these two aspects: meaning and repetition.

1 - I feel like less of a person for experiencing this
2 - That was kind of a waste of my time
3 - Not life altering, but entertaining
4 - That made me feel something
5 - I am a better person for having experienced this

1 - Actively upset that I spent any time watching that
2 - Wish I would have watched something else
3 - A fine use of my time to view it once
4 - Want to see it again
5 - Could watch it over and over; will watch it anytime it's on

This is how movies like Back to the Future and Phoebe in Wonderland can both end up being five stars. It's unfortunate that the different meaning behind those two ratings get lost, but it's the best I can do with a one dimensional scale.

Tangent: It's interesting browsing someone's movie collection. Movie purchases operate primarily under the repetition scale, not the meaning scale. You can totally love a movie, yet not need feel the need to own it. You can also have total fluff that makes you feel good that you want to have around so you can watch whenever. As tempting as it can be to judge someone's tastes by their displayed collection, it's only part of the picture.

There's less of a distinction with games (either of the video or board form) where repetition is more integral to the experience. Compare to Board Game Geek's rating definitions:

10 - Outstanding. Always want to play and expect this will never change
9 - Excellent game. Always want to play it.
8 - Very good game. I like to play. Probably I'll suggest it and will never turn down a game.
7 - Good game, usually willing to play.
6 - Ok game, some fun or challenge at least, will play sporadically if in the right mood.
5 - Average game, slightly boring, take it or leave it.
4 - Not so good, it doesn't get me but could be talked into it on occasion.
3 - Likely won't play this again although could be convinced. Bad.
2 - Extremely annoying game, won't play this ever again.
1 - Defies description of a game. You won't catch me dead playing this. Clearly broken.

People wouldn't know what to do with a board game that changed their life but wasn't worth playing a second time. Although that concept doesn't exist in the board game world, it is something we regularly see in film. But I'd love to see the board game that delivered that experience.

I often see reviews that knock video games for being too short. Because video games are quite a bit more expensive than seeing a movie there's an expectation that the consumer should get their money's worth. My time is not as plentiful as it used to be, so a game that delivers a quality experience with no multiplayer or other replay value is just fine. I love the short and sweet game, and so their assumed negative criticism is actually a positive to me.

It's kind of because of this information loss that I don’t try to give numeric ratings on my blog. But I do put a good deal of thought into it whenever I give one elsewhere. But it's a problem that not everyone puts the same level of care into their ratings. I think my first exposure to this was a decade ago back in the days of Am I Hot or Not, which is probably the first real large scale use of internet driven ratings. One person might be sitting there agonizing over the fine difference between a seven and an eight; another person might be treating it as binary with "hot" being ten and "not" being one. And even if you have people putting real thought into it, they many not use the scale uniformly (for example I know that I underuse the one and two star ratings for Netflix).

It's weird, because I think this whole rating thing is hugely important to helping people sift through ever growing heap of content available to us. But I also think that we as humans pretty much suck at it. I think that identity is key to making the experience more accurate, like when Netflix tells me that the rating I'm looking at is from someone who is a bad match for me, or when you choose a particular editorial source whose opinions align with our own. But how much will we really grow and be challenged if we're only exposed to stuff that is like what we already like?

It's a hard problem space, and I hope there are smart people out there thinking really hard about it. In the meantime, I rate this post a "Q".