Award-Winning Paper Reveals Key to Netflix Prize

When the organizers of the Netflix Prize contest announced late last week that one team had met the requirement for the $1 million Grand Prize, Yehuda Koren, a member of the seven-person multinational team, was in Paris to present a paper at KDD-09, the 15th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. The ideas he laid out won the conference’s Best Paper Award — and, not coincidentally, had much to do with reaching the contest’s target of improving the accuracy of Netflix movie recommendations by 10 percent.

In the paper, Koren, a senior research scientist at Yahoo! Research Israel in Haifa, showed a way to improve Netflix’s recommender algorithm by using information about changes in the ratings over time. His role on the winning team, BellKor’s Pragmatic Chaos, involved refining the model that deals with temporal dynamics.

While movies themselves stay the same, the humans who rate them are anything but static. As Koren puts it, "The way I rate movies today can be very different from how I rate them even tomorrow." To the frustration of Netflix Prize contenders, a four-star rating can mean "great" or merely "so-so" depending on the user’s current mood or comparisons with other recently seen movies. Besides such erratic shifts in the rating scale, people’s actual tastes tend to change over time — as when someone tires of action films, for example, and develops a yen for screwball comedies. There’s an overall rise in ratings over the years, as well: for various reasons, a typical movie’s ratings become more favorable as the DVD ages.

Other investigators had noticed such time-based dynamics in the Netflix dataset, but, unlike Koren, they had trouble putting them to successful use. "This is the first paper that does a good job of giving a mathematical model of these phenomena," says Charles Elkan, a Netflix Prize judge and Professor of Computer Science and Engineering at the University of California, San Diego.

Past attempts at modeling temporal dynamics in collaborative filtering or recommender systems weren’t sensitive enough, Koren explains: although recent data may reveal more about a user’s current preferences than older data, simply underweighting older ratings loses too much valuable information for that approach to work. The trick to not tossing the baby with the bathwater is to retain everything that predicts the user’s long-term behavior while filtering out temporary noise. To that end, Koren’s model has one component for absorbing the lasting signal and another for detecting signs of transient noise, such as certain abrupt changes in the ratings.

Though BellKor’s Pragmatic Chaos surpassed current Netflix accuracy by 10.05 percent, the Grand Prize technically isn’t theirs just yet: other participants have until July 26 to match or beat BellKor’s feat. But the time-based methods are already a winner. According to Koren, they’ve helped Yahoo! improve the relevance of the ever-changing News stories displayed on the site’s home page.

Marina Krakovsky (marinakrakovsky.com) is a writer in the San Francisco Bay Area.