Published Data and Results Not Always Legit

Posted to Statistics  |  Nathan Yau

In a previous life, I thought anything published in an academic journal was legit, but as a stat student, the story is quite the opposite. Whenever I hear results or see data from some study, I become an instant skeptic.

Were there really that many deaths from 1998 to 2007? Did housing prices really increase that much over the past decade? Do that many people really support that presidential candidate?

Whether my skepticism is a good thing, that’s still up for debate. However, the article, Most Science Studies Appear to Be Tainted By Sloppy Analysis, in the Wall Street Journal says I should question.

We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong.

Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. “People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual,” Dr. Ioannidis said.

Not That Surprised

One of the assignments on my qualifying exam was to look at data from an article that had been published in Science (a very prominent academic journal). The final results of the authors’ “analysis” were that wide-ranging animals should not be placed in captivity, because it is poor for their health. The recommendation was to either provide more space in zoos or to only house animals that are not wide-ranging. The authors made a bunch of assumptions about the data, like independence and causality, that weren’t warranted, and carried out a very poor analysis leading to biased conclusions.

The article on wide-ranging animals was clearly chosen by my professor because the results blatantly sucked, but I can only imagine how many other pseudo-results are out there that aren’t as obvious. Is it fair to say that most science studies are sketchy? That might be a slight exaggeration, but probably not too far off.

[via Statistical Modeling, Causal Inference, and Social Science]

Favorites

This is an American Workday, By Occupation

I simulated a day for employed Americans to see when and where they work.

Where People Run in Major Cities

There are many exercise apps that allow you to keep track of your running, riding, and other activities. Record speed, …

10 Best Data Visualization Projects of 2015

These are my picks for the best of 2015. As usual, they could easily appear in a different order on a different day, and there are projects not on the list that were also excellent.

A Day in the Life of Americans

I wanted to see how daily patterns emerge at the individual level and how a person’s entire day plays out. So I simulated 1,000 of them.