Published Data and Results Not Always Legit

Posted to Statistics  |  Nathan Yau

In a previous life, I thought anything published in an academic journal was legit, but as a stat student, the story is quite the opposite. Whenever I hear results or see data from some study, I become an instant skeptic.

Were there really that many deaths from 1998 to 2007? Did housing prices really increase that much over the past decade? Do that many people really support that presidential candidate?

Whether my skepticism is a good thing, that’s still up for debate. However, the article, Most Science Studies Appear to Be Tainted By Sloppy Analysis, in the Wall Street Journal says I should question.

We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong.

Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. “People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual,” Dr. Ioannidis said.

Not That Surprised

One of the assignments on my qualifying exam was to look at data from an article that had been published in Science (a very prominent academic journal). The final results of the authors’ “analysis” were that wide-ranging animals should not be placed in captivity, because it is poor for their health. The recommendation was to either provide more space in zoos or to only house animals that are not wide-ranging. The authors made a bunch of assumptions about the data, like independence and causality, that weren’t warranted, and carried out a very poor analysis leading to biased conclusions.

The article on wide-ranging animals was clearly chosen by my professor because the results blatantly sucked, but I can only imagine how many other pseudo-results are out there that aren’t as obvious. Is it fair to say that most science studies are sketchy? That might be a slight exaggeration, but probably not too far off.

[via Statistical Modeling, Causal Inference, and Social Science]


How You Will Die

So far we’ve seen when you will die and how other people tend to die. Now let’s put the two together to see how and when you will die, given your sex, race, and age.

Divorce Rates for Different Groups

We know when people usually get married. We know who never marries. Finally, it’s time to look at the other side: divorce and remarriage.

Think Like a Statistician – Without the Math

I call myself a statistician, because, well, I’m a statistics graduate student. However, the most important things I’ve learned are less formal, but have proven extremely useful when working/playing with data.

Top Brewery Road Trip, Routed Algorithmically

There are a lot of great craft breweries in the United States, but there is only so much time. This is the computed best way to get to the top rated breweries and how to maximize the beer tasting experience. Every journey begins with a single sip.