Published Data and Results Not Always Legit

Posted to Statistics  |  Nathan Yau

In a previous life, I thought anything published in an academic journal was legit, but as a stat student, the story is quite the opposite. Whenever I hear results or see data from some study, I become an instant skeptic.

Were there really that many deaths from 1998 to 2007? Did housing prices really increase that much over the past decade? Do that many people really support that presidential candidate?

Whether my skepticism is a good thing, that's still up for debate. However, the article, Most Science Studies Appear to Be Tainted By Sloppy Analysis, in the Wall Street Journal says I should question.

We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong.

...

Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.

Not That Surprised

One of the assignments on my qualifying exam was to look at data from an article that had been published in Science (a very prominent academic journal). The final results of the authors' "analysis" were that wide-ranging animals should not be placed in captivity, because it is poor for their health. The recommendation was to either provide more space in zoos or to only house animals that are not wide-ranging. The authors made a bunch of assumptions about the data, like independence and causality, that weren't warranted, and carried out a very poor analysis leading to biased conclusions.

The article on wide-ranging animals was clearly chosen by my professor because the results blatantly sucked, but I can only imagine how many other pseudo-results are out there that aren't as obvious. Is it fair to say that most science studies are sketchy? That might be a slight exaggeration, but probably not too far off.

[via Statistical Modeling, Causal Inference, and Social Science]

Favorites

The Most Unisex Names in US History

Moving on from the most trendy names in US history, let’s look at the most unisex ones. Some names have …

19 Maps That Will Blow Your Mind and Change the Way You See the World. Top All-time. You Won’t Believe Your Eyes. Watch.

Many lists of maps promise to change the way you see the world, but this one actually does.

Reviving the Statistical Atlas of the United States with New Data

Due to budget cuts, there is no plan for an updated atlas. So I recreated the original 1870 Atlas using today’s publicly available data.

Think Like a Statistician – Without the Math

I call myself a statistician, because, well, I’m a statistics graduate student. However, the most important things I’ve learned are less formal, but have proven extremely useful when working/playing with data.