How can we now cope with a large amount of data and still do a thorough job of analysis so that we don't miss the Nobel Prize?

For the past year, I've been slowly drifting off my statistical roots - more interested in design and aesthetics than in whether or not a particular graphic works or the more numeric tools at my disposal. I've always had more fun experimenting on a bunch different things rather than really knuckling down on a particular problem. This works for a lot of things - like online musings - but you miss a lot of the important technical points in the process, so I've been (slowly) working my way back to the analytical side of the river.

If you really want to learn about a large dataset, visualization is only part of the answer. It's an exploratory process. You create a graph. You create a whole bunch of graphs. Notice anything interesting? Okay, let's look over there. This process is called exploratory data analysis, coined by famed statistician John Tukey back in the 1970s. Too often we settle on a particular graphic because it looks pretty, or worse, it helps prove our point. We get blinded by outside motivations, that we forget to listen and look at what else the data have to say. On the flip side, we often like to visualize everything at once and leave it at that. This works to an extent, but we miss out on a lot of details.

Basically, what I'm trying to say is that design can do wonders for visualization, yes, but so can analysis. Put the two together, and you're going to gain a much better understanding of a dataset than if you were to have just one or the other. In my experience, designers are afraid of statistical methods and statisticians are oblivious to design. I say - put the two together. Learn both, and we'll all be that much better at understanding the even bigger data to come.

To fully appreciate the revolutionary nature of PRIM-9 one has to view it against the backdrop of its time. When Statistics was widely taken to be synonymous with inference and hypotheses testing, PRIM-9 was a purely descriptive instrument designed for data exploration. When statistics research meant research in statistical theory, employing the tools of mathematics, the research content of PRIM-9 was in the area of computer-human interfaces, drawing on tools from computer science. When the product of statistical research was theorems published in journals, PRIM-9 was a program documented in a movie.

John W. Tukey's Work on Interactive Graphics. The Annals of Statistics, Vol. 30 No. 6. 2002.

Luckily, you can appreciate Tukey's work here at the ASA video library. It's even more amazing when you consider where computers and technology were at back then. Who knows where Statistics would be if it weren't for Tukey and his brilliance and creativity. I can't imagine, or maybe I just don't want to.

Tukey was someone who truly understood data -- structure, patterns, and what to look for -- and because of that, he was able to create something amazing.

Most are familiar with the Netflix Prize. If you're not, Netflix has offered a one million dollar prize to whoever improves their movie recommendation by a certain amount. It's been going on for a little over a year with still no grand prize winner. The dataset is 100 million ratings.

The above is a visualization of the Netflix dataset. Each dot represents a movie, and the closer two dots are the more similar the two corresponding movies are based on Netflix ratings. I'm guessing the orientation of the dots was decided by some variant of multidimensional scaling.

It's kind of fun to scroll over the clusters. Like in the bottom right we see Babylon 5, Buffy the Vampire Slayer, Alias, and Battlestar Galactica clumped together. The giant blob in the middle, however, is pretty useless; it'd probably benefit from some zoom functionality.

The Need to Explore

I'm kind of surprised that I haven't seen more Netflix visualizations like this (or ones better than this), because I'm pretty sure it would help see some relationships that typical analysis won't provide. I was browsing the forum and saw someone ask if others had had success loading the 100 million observation dataset into R. Silly undergrad.

A computer scientist, designer, and statistician walk into a bar; they discuss how they would boost the Netflix recommendation system. The punchline is that they win a million dollars, but I'm not sure what happens in between.

The New York Times recently put up a cool data exploration tool to sift through the transcript of the most recent Republican debate. They call it the transcript analyzer. There are three key features:

1. View where candidates put in their two cents indicated by the blue, highlighted rectangles
2. Read the actual chunks of transcript for each block
3. Search the transcript to see when specific words and phrases were used indicated by the smaller gray highlighted rectangles

My particular favorite is the search feature because it really allows readers to dig into the transcript or a reader can find out which candidate is (or isn't) talking about his or her point of interest and when in the debate the topic was discussed. The intuitive text scrolling is pretty awesome too. Good job, New York Times!

[via Jon Udell]

On their new exploration section, Twitter blocks is available for viewing and use. The viz is in Flash and is supposed to allow you to explore your neighbors as well as your neighbors' neighbors. I think the higher up the blocks are, the more recent. It's kind of hard to say. Other than that, I'm actually not really sure what I'm looking at. I thought it might be because I'm not following that many people, but I viewed the blocks for the public timeline and still had trouble deciphering. Maybe others will have better luck.

Update: Michal posted on the feedback they've been getting on Twitter Blocks that's certainly worth reading:

So we get this a lot: "Beautiful! But useless!". We've heard it in response to most projects we've done over the past few years (one exception has been Oakland Crimespotting, whose stock yokel response is: "no way am I moving to Oakland!").

This kinda surprises me. I think their other projects are pretty useful and informative.