Collaborative Music Ranking

February 10, 2005

After browsing through the results of this year's Pazz & Jop Poll, I started thinking about the best way to do collaborative ranking. What system will provide you with the most accurate results?

The Voice chose a system that gives the participating critics a fair amount of freedom. Each entry divides 100 points among ten albums, with the maximum for any one album being thirty points and the minimum being five.

In this system, I can choose to give a lot of points to a couple artists and stiff the rest with five points. Craig Marks (Blender) is an example of someone who did this. Another option is giving every album equal weight, like Sia Michel, (Spin). Personally, I think this method is a cop-out, but that could be a whole other conversation. Stephen Thompson (The Onion) decided to give sequential values for each album (as you go down in the list, a point is dropped). Finally, you could defy a specific convention, like Greg Kot (Chicago Tribune), and carefully doll out points among your ten albums.

Without a doubt, this freedom makes each lists unique, but does it give the most accurate tallies? If you were to switch to a standard ranking system (#1 album gets 10 points, #10 album gets 1, or something), would the results change much? I would love to grab all of this data and try it out.

What's more interesting is what it would mean if the rankings are almost the same. Does that mean that all of the different ranking techniques cancel each other out? I would love to hear what James Surowiecki, the author of The Wisdom of Crowds, has to say about this.

There is likely one system that is more telling than any other, but I haven't a clue what that is. Also, each system would tell a different side of the story (which album was the most talked about, which was the favorite, etc.). If we were to use a number of different systems on the same set of data, looking at the results of each side by side would likely provide a pretty good answer.

That being said, if anyone's willing to figure out how to get the data out then I'll run the numbers and see how the lists compare. I might try to email the Voice to see if they'd let me give it a shot. Let me know if you have any pull.

Advertisement