Why $1m Netflix algorithm never went to production

Posted to Statistics  |  Tags: , ,  |  Nathan Yau

Five and a half years ago, Netflix offered data and a $1 million prize to improve their recommendation system by at least ten percent. In 2009, a statistics team at AT&T Labs, BellKor, did that. Unfortunately, Netflix never integrated the algorithm into production.

If you followed the Prize competition, you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later. This is a truly impressive compilation and culmination of years of work, blending hundreds of predictive models to finally cross the finish line. We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then.

That’s too bad. Netflix knows their business better than anyone, but I sure wish Keeping Up with the Kardashians wasn’t listed in my top 10 right now.

[via Techdirt]


  • Did this improved algorithm get incorporated into Jinni.com? Jinni is a beta site (by their own label), and you can link your Jinni recommendations to Netflix, so they may be affiliated. The recommendation engine at Jinni is reviewed much more favorably (and my own personal experience would corroborate this) than the recommendation engine on Netflix.

  • If you don’t want Keeping Up with the Kardashians in your top 10, maybe you shouldn’t watch it so often…. :-p

  • The blog post points out that the algorithms from the first year of the competition were indeed used, and are still a key part of their recommender. The last year or so of the competition was focused on combining models and teams. The Grand Prize winning solution contained over 800 models from 4 teams – it was engineered to win a contest, and was not surprisingly suitable for a production system.

    Similarly, Netflix learned that the best recommender algorithm will be a blend of many individual models, even if the specific Grand Prize solution was not useful to them, the learnings from the competition were extremely valuable.


A Day in the Life of Americans

I wanted to see how daily patterns emerge at the individual level and how a person’s entire day plays out. So I simulated 1,000 of them.

Famous Movie Quotes as Charts

In celebration of their 100-year anniversary, the American Film Institute selected the 100 most memorable quotes from American cinema, and …

Jobs Charted by State and Salary

Jobs and pay can vary a lot depending on where you live, based on 2013 data from the Bureau of Labor Statistics. Here’s an interactive to look.

Unemployment in America, Mapped Over Time

Watch the regional changes across the country from 1990 to 2016.