Rating systems overrated
By Erik van Kempen | March 31, 2007
Web developers are constantly looking for new techniques to interact with readers/users. Now, the upcoming Web 2.0 bubble involves an increase of the need for interaction techniques, such as rating systems, which are widely used to initiate interaction between the content supplier and the users.
But... Rating systems, as they are implemented these days, are not as great as they seem to be. There are two main disadvantages, of the use of rating systems:
- Ratings are strongly time-related
- They cover only extreme opinions
Extreme opinions
Most opinions covered by rating systems are extremely positive or extremely negative. Users are quite lazy and impatient, so they won't do anything that doesn't have an instant result. People with an extreme opinion are eager to share their feelings with others. People with a less extreme opinion won't take the time to write comments or submit ratings.
Example:
There's a web site where developers can share their extensions for Mozilla Firefox. People can submit their comments and rate the extension on a 0-to-5 star scale.
I've submitted my extensions and found that people will only reward the extension with 5 stars or 0 stars. The 5-star rating is caused by people who are extremely happy that I developed the extension and the 0-star ratings were mostly caused by things like missing features, bugs etc., especially when other functionality was broken by my extension. The number of 'normal' ratings are so small, that they don't even count.
I think this example shows a completely natural response. People like to express their extreme opinions to the world. So rating systems are not showing the right statistics. From a mathematical point of view, rating systems should show the mean of a normal distribution. But instead of this normal distribution, the bell shape is transformed into a double bell shape where the mean values are at the extreme values.
Time
Another issue is that ratings are strongly related to time in a few ways. First not one user will thoroughly examine his thought about the subject and then base their rating on those examinations. It's an opinion that was created in less than a minute, depending on the subject. Secondly the subject can change in time. This depends on the subject too, but if you reconsider the example, you can imagine that extensions are constantly updated etcetera, but the ratings and comments aren't.
Maybe it's time to think about rating systems and their value again in order to improve the systems, but most of the problems will never be solved probably.