Automatic versus manual data analysis

Posted to Statistics  |  Tags:  |  Nathan Yau

Hilary and Roger touch on some interesting topics in the most recent Not So Standard Deviations, specifically on scalable and automated data analysis.

At the surface, it can seem like computers should be able to do the bulk of any analysis. Plug in the data, crunch the numbers in an algorithmic black box, and presto change-o you get a list of actionable insights. From that point-of-view, you should be able to build software that does almost everything for you. That’s almost never the case, and you realize it quickly once you dig into the data yourself.

It’s the same deal with visualization.

You see the end result, and it’s easy to imagine applying the same chart to another dataset. Geometry and color are easy to make with a couple lines of code. The chart should be generalizable, right? Sure, but the challenge is getting to that final point. There are various paths you can take when you start with a dataset — what it means, the questions you want to ask — along with various decisions along the way.

Automating the process. That’s the hard part.

Favorites

Top Brewery Road Trip, Routed Algorithmically

There are a lot of great craft breweries in the United States, but there is only so much time. This is the computed best way to get to the top rated breweries and how to maximize the beer tasting experience. Every journey begins with a single sip.

The Most Unisex Names in US History

Moving on from the most trendy names in US history, let’s look at the most unisex ones. Some names have …

Think Like a Statistician – Without the Math

I call myself a statistician, because, well, I’m a statistics graduate student. However, the most important things I’ve learned are less formal, but have proven extremely useful when working/playing with data.

Causes of Death

There are many ways to die. Cancer. Infection. Mental. External. This is how different groups of people died over the past 10 years, visualized by age.