The problem with collaborative filtering

The problem with collaborative filtering

The problem is that it is hard to find anything new. For example, my great new Yahoo Unlimited music service is a great way to listen to pretty much everything. I searched for a random song I saw on Indian MTV and it was there (Shoniyhe by Juggy D). It was there and I sort of like it.

But Yahoo won't recommend that to me. It keeps recommending Morrissey Live at Earl's Court, Bjork Medulla, Yo La Tengo, Pavement, Stone Roses, Cure, Moby, Pixies, Black Rebel Motorcycle, Belle and Sebastien... some of which I certainly love, some of which I actually hate because they are similar but crucially different (e.g., Cure is not Smiths) and some of which are totally predictable easy to find new albums by bands I already like.

Rarely if ever though does collaborative filtering help me find *different* stuff that I will like because the algorithm is a somewhat naive interpretation of what I already like. It is good at finding more stuff that is similar but similar is not the full set of what I like. In fact, I only like *some* (maybe most) similar stuff (e.g., new Moz album) but I really *want* something *different*. So how to find it? This system doesn't recommend Nouvelle Vague or Scissor Sister or Natascha Atlas or whatever. I can only stumble on these things in normal ways (by following what is "new").

Need a deeper mechanism. One thing that used to work for me back when such things existed was surfing users on Napster. You could see someone's library, and judging from that collection, you could determine that album in their collection (which you didn't recognize) would be good. But from some other guy's library, album Y would probably suck. Even though both guys had Morrissey and Pixies in there - the richer information let you judge one guy as a good predictor and the other as bad. If the algorithms still suck at doing this, why not make the anonymized favorites lists available to let us make those judgments?