Whoa. What did I just read?
I think most of you know of Freakonomics, but in case you don’t, it started as a book in 2005, by economist Steven Levitt and journalist Stephen Dubner. The book examines corners of life (like cheating in sumo) through data. It’s a good read. SuperFreakonomics was the follow-up in 2009. Freakonomics has since grown up into a media company, complete with documentary, radio show, and blog. Needless to say, it’s had a lot of success.
In the latest issue of American Scientist, statisticians Kaiser Fung and Andrew Gelman wrote a strong critique of Levitt and Dubner’s work.
In our analysis of the Freakonomics approach, we encountered a range of avoidable mistakes, from back-of-the-envelope analyses gone wrong to unexamined assumptions to an uncritical reliance on the work of Levitt’s friends and colleagues. This turns accessibility on its head: Readers must work to discern which conclusions are fully quantitative, which are somewhat data driven and which are purely speculative.
Fung and Gelman then cite examples that they believe erroneous.
It’s not mean-spirited, but Gelman has a way of offending even if he doesn’t mean to, so I knew a third of the way through that this could not end well.
Dubner replied. (Skip part II, which addresses a different issue that shouldn’t have been an issue in the first place.) He assesses — after explaining why almost everything that Fung and Gelman wrote is wrong — that they were blinded by their want to disprove.
[O]nce they’d picked up a hammer, did everything look like a nail?
Dubner continues:
I can certainly understand why Freakonomics is an appealing target for someone like Gelman-Fung. As I noted earlier, there are strong incentives to attack, particularly in the public sphere, where one can get a ton of attention in a blink by assailing the reputation of someone who’s been plugging away for years. Whether in the academy, the media, the political arena, or elsewhere, public discourse these days often seems little more than a tit-for-tat game in which you wait for someone or something to achieve a certain momentum and then shout as loudly as you can that it’s “wrong!” Or, in written form: Epic fail.
I’ve only read the first book, which like I said was really good, so I can’t really go with either side, but Dubner provides some compelling arguments, and I have a feeling most people will believe him more.
Update: Gelman replies to the reply and Fung adds to that.