Whoa. What did I just read?
I think most of you know of Freakonomics, but in case you don’t, it started as a book in 2005, by economist Steven Levitt and journalist Stephen Dubner. The book examines corners of life (like cheating in sumo) through data. It’s a good read. SuperFreakonomics was the follow-up in 2009. Freakonomics has since grown up into a media company, complete with documentary, radio show, and blog. Needless to say, it’s had a lot of success.
In the latest issue of American Scientist, statisticians Kaiser Fung and Andrew Gelman wrote a strong critique of Levitt and Dubner’s work.
In our analysis of the Freakonomics approach, we encountered a range of avoidable mistakes, from back-of-the-envelope analyses gone wrong to unexamined assumptions to an uncritical reliance on the work of Levitt’s friends and colleagues. This turns accessibility on its head: Readers must work to discern which conclusions are fully quantitative, which are somewhat data driven and which are purely speculative.
Fung and Gelman then cite examples that they believe erroneous.
It’s not mean-spirited, but Gelman has a way of offending even if he doesn’t mean to, so I knew a third of the way through that this could not end well.
Dubner replied. (Skip part II, which addresses a different issue that shouldn’t have been an issue in the first place.) He assesses — after explaining why almost everything that Fung and Gelman wrote is wrong — that they were blinded by their want to disprove.
[O]nce they’d picked up a hammer, did everything look like a nail?
Dubner continues:
I can certainly understand why Freakonomics is an appealing target for someone like Gelman-Fung. As I noted earlier, there are strong incentives to attack, particularly in the public sphere, where one can get a ton of attention in a blink by assailing the reputation of someone who’s been plugging away for years. Whether in the academy, the media, the political arena, or elsewhere, public discourse these days often seems little more than a tit-for-tat game in which you wait for someone or something to achieve a certain momentum and then shout as loudly as you can that it’s “wrong!” Or, in written form: Epic fail.
I’ve only read the first book, which like I said was really good, so I can’t really go with either side, but Dubner provides some compelling arguments, and I have a feeling most people will believe him more.
Update: Gelman replies to the reply and Fung adds to that.
It’s worth reading Gelman’s response: http://andrewgelman.com/2012/03/a-kaleidoscope-of-responses-to-dubners-criticisms-of-our-criticisms-of-freaknomics/
Hi Nathan,
Well, Gelman has already replied to Dubner’ reply. Having read myself the 2nd book (their climate change chapter is *really* bad), I’ll side with Gelman:
http://andrewgelman.com/2012/03/a-kaleidoscope-of-responses-to-dubners-criticisms-of-our-criticisms-of-freaknomics/
My dad always said that you could tell how reliable a journalist was by how they reported on something you actually knew something about. By that test, the Freakonomics guys fail miserably: their coverage of climate change in their second book is embarrassingly bad. (See http://www.bloomberg.com/news/2009-10-19/freakonomics-guys-flunk-science-of-climate-change-eric-pooley.html for one review — there are lots of others.) Unfortunately, I think Levitt’s contrarian nature, and the fact that you can’t sell books by telling people, “Yup, everyone you’ve already heard is actually right,” got the better of the data.
Althought I have yet to read Super Freakonomics, I find it interesting that climate change (and issue on which there is no agreement on the basic postulates between the concerned and skeptical) is the issue on which to claim the Freakonomics crew is weak.
One of the folks at Crooked Timber unpacks this a little, as sort of a rise and fall story: http://crookedtimber.org/2012/03/21/stephen-j-dubner-my-part-in-his-upfall/ He cites Krugman’s take as well: http://krugman.blogs.nytimes.com/2009/10/16/a-counterintuitive-train-wreck/
Bottom line according the them: Started off great, then they got too big for their britches, as it were.
I think Dubner is right in this is tit for tat.
Are the conclusions embellished to sell books? Probably.
Is the actual research behind it performed by peer reviewed internationally acknowledged economists? Yes.
Like most science (social or otherwise), by the time its converted into digestable public media, conclusions are exaggerated and numbers are ignored or misrepresented. It’s sad that it happens but I guess sometimes to get the general public to think and care about issues we need to give them exaggerated examples. I think the blame falls more on Dubner rather than Levitt here. That said, I love the books and they have me thinking about economics in new and different ways. Surely that’s a good thing too. If Gelman has a problem with the actual research, he should write a comment in a peer reviewed economics journal, not on a blog. Why should the criticism not be subject to the same rigour and scrutiny and it’s target
Tom:
I think we are basically in agreement.
I respect Steven Levitt’s peer-reviewed research very much. Many of the Freaknomics examples that Kaiser and I discussed in our article were not published in peer-reviewed journals, economics or otherwise. Actually, it was the opposite: in their climate-change chapter, Levitt and Dubner dismiss the peer-reviewed literature in favor of a conversation with a retired software executive who has a Ph.D. in physics. In another case, we criticize not the peer-reviewed literature but rather the conclusions that Levitt and Dubner draw from it in their book. Their claims about drunk driving and drunk walking did not appear in a peer-reviewed journal. Further examples we discuss in our reply to Dubner (http://andrewgelman.com/2012/03/a-kaleidoscope-of-responses-to-dubners-criticisms-of-our-criticisms-of-freaknomics/) include Casey Mulligan’s claim in October 2008 that the economy is going fine and a dubious claim about the economic effects of the Tiger Woods scandal. Neither of these appeared in a peer-reviewed journal. On his blog, Dubner presented without skepticism an outlandish claim that beautiful parents are 36% more likely to have girl babies. That claim was in fact published in a peer-reviewed journal (in biology, not economics), and I indeed published a criticism of that claim in the same journal, following the practice that you recommend.
In short, I am a big fan of much of Freakonomics and also have a strong (although not absolute) respect for peer-reviewed publications. The criticisms that Kaiser and I expressed were of the weaker aspects of Freakonomics, places where Levitt and Dubner pushed dubious claims with weak reasoning. As Kaiser and I wrote, we are happy about many aspects of Freaknomics but recommend that readers read it with skepticism rather than too quickly trusting its claims based on extrapolation from Levitt’s deservedly strong reputation as a researcher.
We are in agreement Andrew.
A point that Freakonomics tries to drive home is that everything is driven by incentives. Whereas some of the great work was previously done under Levitt’s success and drive as an academic, I have a feeling that some of the (more recent?) less rigorous claims and reporting have been more incentivized by the “Freakonomics” brand to be a successful business.
They were right. At the end of the day, incentives are the king.
Steven Levitt won the John Bates Clark Medal, which among economists carries almost as much weight as the Nobel. A medal does not make a man and it’s always possible that an individual can fool a group of people. That said, there are always problems with methodology. The real question is how material it is. I think @Tom Lau is right, if it’s truly a methodological problem and you think you know better, send it to peer review. But, currently the trend is for economists to use their blogging platform for anything from macroeconomics to criticizing 7 year old books and getting plenty of attention for it.
That it is 7 years old is irrelevant. If something is influential it can be read, and criticized, at any time, there’s no statute of limitations.
So is the general consensus here that Dubner and Levitt can say on their blog all they want, but to criticize them you have to go through peer review? That makes no sense. The original book likely wasn’t peer reviewed, it was published by a trade press. It might not have been reviewed at all before publication except by the editor.
It’s odd to find that on a blog people are criticizing a critique of a blog based not on the substance of the critique but because it was on a blog.
After I drove the kids to school, I realized I mangled my point pretty badly. A serious conversation can occur on a blog, bus, or anywhere. Your point on using a blog to criticize a blog because it was on a blog is valid and does seems odd. Most blogs are blather. Paradoxically, my post disqualifies itself as well. In my opinion, blogs don’t currently have impact like trade press and academic journals whether we like it or not. I think the low barrier to entry will forever allow noise to discredit the platform. With eBooks, the barrier to entry is fundamentally zero. The trade press unfortunately has a barrier to entry which creates a perceived authority. I personally put trade press in the same category as blogs. Whether academic journal or trade press or blog, the reader is the ultimate decider of validity. In a perfectly clear headed world, we wouldn’t judge a book by it’s cover, but I sure start there.