Why $1m Netflix algorithm never went to production

Posted to Statistics  |  Tags: , ,  |  Nathan Yau

Five and a half years ago, Netflix offered data and a $1 million prize to improve their recommendation system by at least ten percent. In 2009, a statistics team at AT&T Labs, BellKor, did that. Unfortunately, Netflix never integrated the algorithm into production.

If you followed the Prize competition, you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later. This is a truly impressive compilation and culmination of years of work, blending hundreds of predictive models to finally cross the finish line. We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then.

That's too bad. Netflix knows their business better than anyone, but I sure wish Keeping Up with the Kardashians wasn't listed in my top 10 right now.

[via Techdirt]

4 Comments

  • Did this improved algorithm get incorporated into Jinni.com? Jinni is a beta site (by their own label), and you can link your Jinni recommendations to Netflix, so they may be affiliated. The recommendation engine at Jinni is reviewed much more favorably (and my own personal experience would corroborate this) than the recommendation engine on Netflix.

  • If you don’t want Keeping Up with the Kardashians in your top 10, maybe you shouldn’t watch it so often…. :-p

  • The blog post points out that the algorithms from the first year of the competition were indeed used, and are still a key part of their recommender. The last year or so of the competition was focused on combining models and teams. The Grand Prize winning solution contained over 800 models from 4 teams – it was engineered to win a contest, and was not surprisingly suitable for a production system.

    Similarly, Netflix learned that the best recommender algorithm will be a blend of many individual models, even if the specific Grand Prize solution was not useful to them, the learnings from the competition were extremely valuable.

Favorites

How We Spend Our Money, a Breakdown

We know spending changes when you have more money. Here’s by how much.

Graphical perception – learn the fundamentals first

Before you dive into the advanced stuff – like just about everything in your life – you have to learn the fundamentals before you know when you can break the rules.

Jobs Charted by State and Salary

Jobs and pay can vary a lot depending on where you live, based on 2013 data from the Bureau of Labor Statistics. Here’s an interactive to look.

Watching the growth of Walmart – now with 100% more Sam’s Club

The ever so popular Walmart growth map gets an update, and yes, it still looks like a wildfire. Sam’s Club follows soon after, although not nearly as vigorously.