In a previous life, I thought anything published in an academic journal was legit, but as a stat student, the story is quite the opposite. Whenever I hear results or see data from some study, I become an instant skeptic.
Were there really that many deaths from 1998 to 2007? Did housing prices really increase that much over the past decade? Do that many people really support that presidential candidate?
Whether my skepticism is a good thing, that’s still up for debate. However, the article, Most Science Studies Appear to Be Tainted By Sloppy Analysis, in the Wall Street Journal says I should question.
We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong.
…
Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. “People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual,” Dr. Ioannidis said.
Not That Surprised
One of the assignments on my qualifying exam was to look at data from an article that had been published in Science (a very prominent academic journal). The final results of the authors’ “analysis” were that wide-ranging animals should not be placed in captivity, because it is poor for their health. The recommendation was to either provide more space in zoos or to only house animals that are not wide-ranging. The authors made a bunch of assumptions about the data, like independence and causality, that weren’t warranted, and carried out a very poor analysis leading to biased conclusions.
The article on wide-ranging animals was clearly chosen by my professor because the results blatantly sucked, but I can only imagine how many other pseudo-results are out there that aren’t as obvious. Is it fair to say that most science studies are sketchy? That might be a slight exaggeration, but probably not too far off.
[via Statistical Modeling, Causal Inference, and Social Science]
