Why $1m Netflix algorithm never went to production

Posted to Statistics  |  Tags: , ,  |  Nathan Yau

Five and a half years ago, Netflix offered data and a $1 million prize to improve their recommendation system by at least ten percent. In 2009, a statistics team at AT&T Labs, BellKor, did that. Unfortunately, Netflix never integrated the algorithm into production.

If you followed the Prize competition, you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later. This is a truly impressive compilation and culmination of years of work, blending hundreds of predictive models to finally cross the finish line. We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then.

That’s too bad. Netflix knows their business better than anyone, but I sure wish Keeping Up with the Kardashians wasn’t listed in my top 10 right now.

[via Techdirt]


  • Did this improved algorithm get incorporated into Jinni.com? Jinni is a beta site (by their own label), and you can link your Jinni recommendations to Netflix, so they may be affiliated. The recommendation engine at Jinni is reviewed much more favorably (and my own personal experience would corroborate this) than the recommendation engine on Netflix.

  • If you don’t want Keeping Up with the Kardashians in your top 10, maybe you shouldn’t watch it so often…. :-p

  • The blog post points out that the algorithms from the first year of the competition were indeed used, and are still a key part of their recommender. The last year or so of the competition was focused on combining models and teams. The Grand Prize winning solution contained over 800 models from 4 teams – it was engineered to win a contest, and was not surprisingly suitable for a production system.

    Similarly, Netflix learned that the best recommender algorithm will be a blend of many individual models, even if the specific Grand Prize solution was not useful to them, the learnings from the competition were extremely valuable.


Most popular porn searches, by state

We’ve seen that we can learn from what people search for, through the eyes of Google suggestions: state stereotypes, national …

10 Best Data Visualization Projects of 2015

These are my picks for the best of 2015. As usual, they could easily appear in a different order on a different day, and there are projects not on the list that were also excellent.

Famous Movie Quotes as Charts

In celebration of their 100-year anniversary, the American Film Institute selected the 100 most memorable quotes from American cinema, and …

Shifting Incomes for American Jobs

For various occupations, the difference between the person who makes the most and the one who makes the least can be significant.