Why $1m Netflix algorithm never went to production

Posted to Statistics  |  Tags: , ,  |  Nathan Yau

Five and a half years ago, Netflix offered data and a $1 million prize to improve their recommendation system by at least ten percent. In 2009, a statistics team at AT&T Labs, BellKor, did that. Unfortunately, Netflix never integrated the algorithm into production.

If you followed the Prize competition, you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later. This is a truly impressive compilation and culmination of years of work, blending hundreds of predictive models to finally cross the finish line. We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then.

That’s too bad. Netflix knows their business better than anyone, but I sure wish Keeping Up with the Kardashians wasn’t listed in my top 10 right now.

[via Techdirt]


  • Did this improved algorithm get incorporated into Jinni.com? Jinni is a beta site (by their own label), and you can link your Jinni recommendations to Netflix, so they may be affiliated. The recommendation engine at Jinni is reviewed much more favorably (and my own personal experience would corroborate this) than the recommendation engine on Netflix.

  • If you don’t want Keeping Up with the Kardashians in your top 10, maybe you shouldn’t watch it so often…. :-p

  • The blog post points out that the algorithms from the first year of the competition were indeed used, and are still a key part of their recommender. The last year or so of the competition was focused on combining models and teams. The Grand Prize winning solution contained over 800 models from 4 teams – it was engineered to win a contest, and was not surprisingly suitable for a production system.

    Similarly, Netflix learned that the best recommender algorithm will be a blend of many individual models, even if the specific Grand Prize solution was not useful to them, the learnings from the competition were extremely valuable.


How You Will Die

So far we’ve seen when you will die and how other people tend to die. Now let’s put the two together to see how and when you will die, given your sex, race, and age.

The Best Data Visualization Projects of 2011

I almost didn’t make a best-of list this year, but as I clicked through the year’s post, it was hard …

Real Chart Rules to Follow

There are rules—usually for specific chart types meant to be read in a specific way—that you shouldn’t break. When they are, everyone loses. This is that small handful.

Pizza Place Geography

Most of the major pizza chains are within a 5-mile radius of where I live, so I have my pick, …