Published Data and Results Not Always Legit

Posted to Statistics  |  Nathan Yau

In a previous life, I thought anything published in an academic journal was legit, but as a stat student, the story is quite the opposite. Whenever I hear results or see data from some study, I become an instant skeptic.

Were there really that many deaths from 1998 to 2007? Did housing prices really increase that much over the past decade? Do that many people really support that presidential candidate?

Whether my skepticism is a good thing, that’s still up for debate. However, the article, Most Science Studies Appear to Be Tainted By Sloppy Analysis, in the Wall Street Journal says I should question.

We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong.

Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. “People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual,” Dr. Ioannidis said.

Not That Surprised

One of the assignments on my qualifying exam was to look at data from an article that had been published in Science (a very prominent academic journal). The final results of the authors’ “analysis” were that wide-ranging animals should not be placed in captivity, because it is poor for their health. The recommendation was to either provide more space in zoos or to only house animals that are not wide-ranging. The authors made a bunch of assumptions about the data, like independence and causality, that weren’t warranted, and carried out a very poor analysis leading to biased conclusions.

The article on wide-ranging animals was clearly chosen by my professor because the results blatantly sucked, but I can only imagine how many other pseudo-results are out there that aren’t as obvious. Is it fair to say that most science studies are sketchy? That might be a slight exaggeration, but probably not too far off.

[via Statistical Modeling, Causal Inference, and Social Science]


Divorce and Occupation

Some jobs tend towards higher divorce rates. Some towards lower. Salary also probably plays a role.

The Best Data Visualization Projects of 2011

I almost didn’t make a best-of list this year, but as I clicked through the year’s post, it was hard …

Graphical perception – learn the fundamentals first

Before you dive into the advanced stuff – like just about everything in your life – you have to learn the fundamentals before you know when you can break the rules.

Jobs Charted by State and Salary

Jobs and pay can vary a lot depending on where you live, based on 2013 data from the Bureau of Labor Statistics. Here’s an interactive to look.