This is fun. Tyler Vigen wrote a program that attempts to automatically find things that correlate. As of writing this, 4,000 correlations were found so far (and actually over 100 more when I finished). Some of the gems include: the divorce rate in Maine versus per capita consumption of margarine, marriage rate in Alabama versus whole milk consumption per capita, and honey produced in bee colonies versus labor political action committees. Many things correlate with cheese consumption.

“Type I” and “Type II” errors, names first given by Jerzy Neyman and Egon Pearson to describe rejecting a null hypothesis when it’s true and accepting one when it’s not, are too vague for stat newcomers (and in general). This is better. [via]

Using baby name data from the Social Security Administration, Brian Rowe made this straightforward interactive that lets you search a name to see how its regional popularity changed over over time.

Hadley Wickham offers a detailed, practical guide to finding and removing the major bottlenecks in your R code.

It’s easy to get caught up in trying to remove all bottlenecks. Don’t! Your time is valuable and is better spent analysing your data, not eliminating possible inefficiencies in your code. Be pragmatic: don’t spend hours of your time to save seconds of computer time. To enforce this advice, you should set a goal time for your code and only optimise only up to that goal. This means you will not eliminate all bottlenecks. Some you will not get to because you’ve met your goal. Others you may need to pass over and accept either because there is no quick and easy solution or because the code is already well-optimized and no significant improvement is possible. Accept these possibilities and move on to the next candidate.

This is how I approach it. Some people spend a lot of time optimizing, but I’m usually better off writing code without speed in mind initially. Then I deal with it if it’s actually a problem. I can’t remember the last time that happened though. Obviously, this approach won’t work in all settings. So just use common sense. If it takes you longer to optimize than it does to run your “slow” code, you’ve got your answer.

Naked Statistics by Charles Wheelan promises a fun, non-boring introduction to statistics that doesn’t leave you drifting off into space, thinking about anything that is not statistics. From the book description:

For those who slept through Stats 101, this book is a lifesaver. Wheelan strips away the arcane and technical details and focuses on the underlying intuition that drives statistical analysis. He clarifies key concepts such as inference, correlation, and regression analysis, reveals how biased or careless parties can manipulate or misrepresent data, and shows us how brilliant and creative researchers are exploiting the valuable data from natural experiments to tackle thorny questions.

The first statistics course I took—not counting the dreadful high school stat class taught by the water polo coach—actually drew me in from the start. Plus, I needed to finish my dissertation, so I didn’t pick it up when it came out last year.

I saw it in the library the other day though, so I checked it out. If anything, I could use a few more anecdotes to better describe statistics to people before they tell me how much they hated it.

Naked Statistics is pretty much what the description says. It’s like your stat introduction course with much less math, which is good for those interested in poking at data but well, slept through Stat 101 and have an irrational fear of numbers. You get important concepts and plenty of reasons why they’re worth knowing. Most importantly, it gives you a statistical way to think about data, flaws and all. Wheelan also has a fun writing style that makes this an entertaining read.

For those who are familiar with inference, correlation, and regression, the book will be too basic. It’s not enough just for the anecdotes. However, for anyone with less than a bachelor’s degree (or equivalent) in statistics who wants to know more about analyzing data, this book should be right up your alley.

Keep in mind though that this only gets you part way to understanding your data. Naked Statistics is beginning concepts. Putting statistics into practice is the next step.

Personally, I skimmed through a good portion of the book, as I’m familiar with the material. I did however read a chapter out loud while taking care of my son. He might not be able to crawl yet, but I’m hoping to ooze some knowledge in through osmosis.

Artist Scott Kildall generates what he calls World Data Crystals by mapping data on a globe with cubes and clustering them algorithmically. He then produces the result in physical form for something like the piece below, which represents world population.

Ben Moore was curious about overrated and underrated films.

“Overrated” and “underrated” are slippery terms to try to quantify. An interesting way of looking at this, I thought, would be to compare the reviews of film critics with those of Joe Public, reasoning that a film which is roundly-lauded by the Hollywood press but proved disappointing for the real audience would be “overrated” and vice versa.

Through the Rotten Tomatoes API, he found data to make such a comparison. Then he plotted one against the other, along with a quick calculation of the differences between the percentage of official critics who liked and that of the Rotten Tomatoes audience. The most underrated: Facing the Giants, Diary of a Mad Black Woman, and Grandma’s Boy. The most overrated: Spy Kids, 3 Backyards, and Stuart Little 2.

The plot would be better without the rainbow color scheme and a simple reference line through the even-rating diagonal. But this gets bonus points for sharing the code snippet to access the Rotten Tomatoes API in R, which you can generalize.

While we’re on an R kick, Hilary Parker described how to create an R package from scratch, not just to share code with others but to save yourself some time on future projects. It’s not as hard as it seems.

This tutorial is not about making a beautiful, perfect R package. This tutorial is about creating a bare-minimum R package so that you don’t have to keep thinking to yourself, “I really should just make an R package with these functions so I don’t have to keep copy/pasting them like a goddamn luddite.” Seriously, it doesn’t have to be about sharing your code (although that is an added benefit!). It is about saving yourself time. (n.b. this is my attitude about all reproducibility.)

I need to do this. I’ve been meaning to wrap everything up for a while now, but it seemed like such a chore. Sometimes I even go back to my own tutorials for copy and paste action. Now I know better. And that’s half the battle.

Following the lead of JavaScript for Cats by Maxwell Ogden, Scott Chamberlain and Carson Sievert wrote R for Cats. It’s a playful introduction to R intended for those who have little to no programming experience.

The bulk of it so far is a primer on data structures, and there’s a little bit on functions and some dos and don’ts. It’s stuff you should know before you get into more advanced tutorials.

Mainly though: ooo look, kitty.

Once you’re done with that (It only takes about 30 minutes.), there are lots of other resources for getting started with R.

Matt Daniels compared rappers’ vocabularies to find out who knows the most words.

Literary elites love to rep Shakespeare’s vocabulary: across his entire corpus, he uses 28,829 words, suggesting he knew over 100,000 words and arguably had the largest vocabulary, ever.

I decided to compare this data point against the most famous artists in hip hop. I used each artist’s first 35,000 lyrics. That way, prolific artists, such as Jay-Z, could be compared to newer artists, such as Drake.

As two points of reference, Daniels also counted the number of unique words in the first 5,000 used words from seven of Shakespeare’s works and the number of uniques from the first 35,000 words of Herman Melville’s Moby-Dick.

I’m not sure how much stock I would put into these literary comparisons though, because this is purely a keyword count. So “pimps”, “pimp”, “pimping”, and “pimpin” count as four words in a vocabulary and I have a hunch that variants of a single word is more common in rap lyrics than in Shakespeare and Melville. Again, I’m guessing here.

That said, although there could be similar issues within the rapper comparisons, I bet the counts are more comparable.

Isoscope, a class project by Flavio Gortana, Sebastian Kaim and Martin von Lupin, is an interactive that lets you explore mobility around the world.

We drive to the closest supermarket, take the bike to the gym or walk to the cafe next door for a nice chat among friends. Getting around — thus mobility — is an essential part of our being. We were especially intrigued by those situations when our mobility is compromised such as in traffic jams or during tough driving conditions. How do those restrictions impact our journeys through the city and who is affected most? Obviously, a car can hardly bypass a traffic jam, whereas a bike is more flexible to continue its journey. Let alone the pedestrian who can stroll wherever he wants to. Isoscope tries to answer the questions above by comparing different means of transport and their sensitivity for disturbances.

Similar in flavor to the commute maps before it, Isoscope is a bit different in that it focuses on specific time frames, such as Fridays at 8am. Using data from the HERE API, a travel polygon is estimated for each hour of the day selected. Your initial result is an abstract blot overlaid on a map, but then use the menu to change days and highlight hours.

Because Game of Thrones. Max Fleishman and Fernando Alfonso III for The Daily Dot compared the size of dragons on various shows and movies, from Mushu to Toothless to Smaug to Balerion. The tiny black dot on the left bottom corner is a person.

See also the size comparison of science fiction starships, Pixar characters, and everything else.

In light of the Donald Sterling brouhaha, Amanda Cox for The Upshot put up some charts for why you shouldn’t be surprised that people still say racist things, based on data from the General Social Survey dating back to 1972. Mmhm.

You probably remember how Target used purchase histories to predict pregnancies among their customer base (although, don’t forget the false positives). Janet Vertesi, an assistant professor of sociology at Princeton University, made sure that sort of data didn’t exist during her nine months.

First, Vertesi made sure there were absolutely no mentions of her pregnancy on social media, which is one of the biggest ways marketers collect information. She called and emailed family directly to tell them the good news, while also asking them not to put anything on Facebook. She even unfriended her uncle after he sent a congratulatory Facebook message.

She also made sure to only use cash when buying anything related to her pregnancy, so no information could be shared through her credit cards or store-loyalty cards. For items she did want to buy online, Vertesi created an Amazon account linked to an email address on a personal server, had all packages delivered to a local locker and made sure only to use Amazon gift cards she bought with cash.

The best part was that her modified activity—like purchasing $500 worth of Amazon gift cards in cash from the local Rite Aid—set off other (in real life) triggers.

George & Jonathan used an interactive audio visualization for their recent album George & Jonathan III. This is a fun one. You can rotate the camera as you like, as the full album plays and notes are represented with dashes and dots.

Michal Migurski thinks about finding the right job for the tool rather than the other way around:

Near the second half of most nerd debates, your likelihood of hearing the phrase “pick the right tool for the job” approaches 100% (cf. frameworks, rails, more rails, node, drupal, jquery, rails again). “Right tool for the job” is a conversation killer, because no shit. You shouldn’t be using the wrong tool. And yet, working in code is working in language (naming things is the second hard problem) so it’s equally in-bounds to debate the choice of job for the tool. “Right tool” assumes that the Job is a constant and the Tool is a variable, but this is an arbitrary choice and notably contradicted by our own research into the motivations of idealistic geeks.

Along the same lines, Frank Chimero on not trying any new tools for the year and how each represents someone’s perspective:

Everything that’s made has a bias, but simple implements—a hammer, a lever, a text editor—assume little and ask less. The tool doesn’t force the hand. But digital tools for information work are spookier. The tools can force the mind, since they have an ideological perspective baked into them. To best use the tool, you must think like the people who made it. This situation, at its best, is called learning. But more often than not, with my tools, it feels like the tail wagging the dog.

These approaches apply well to analysis and visualization. In the early goings especially, there tends to be an obsession with what tools to use. Which is best? Which is fastest? Which can handle the most data? Which makes everything beautiful? And yeah, it’s good to give these some thought in the beginning, but don’t get stuck asking so many questions or pondering so many scenarios that you never settle down and do actual work.

There’s always going to be a new application that promises to help you do something with your data. Work on this stuff long enough and you’ll find that you probably won’t need that new thing.

Learning regular expressions tends to involve a lot of trial and error and can be confusing for newcomers. RegExr is an online tool that lets you learn more interactively. Add a body of text in one area and type various regular expressions in another. Matches are highlighted and errors are noted on the fly, which is kind of perfect. Even if you aren’t new to regular expressions, this is worth bookmarking for later.

Researchers at the University of Washington’s Center for American Politics and Public Policy built the Legislative Explorer to show the lawmaking process in action. The visualization allows you to watch over 250,000 bills and resolutions introduced from 1973 to present.

The left half represents the U.S. Senate, with senators sorted by party (blue=Democrat) and a proxy for ideology (top=liberal). The House is displayed on the right. Moving in from the borders, the standing committees of the Senate and House are represented, followed by the Senate and House floors. A bill approved by both chambers then moves upward to the President’s desk and into law, while an adopted resolutions (that does not require the president’s signature) moves downward.

Each dot represents a bill, so you can see them move through the levels. Use the drop-down menus at the top to focus on a Congress, a person, party, topic, and several other categorizations. Or use the search to focus on specific bills. Finally, when you do press play, be sure to keep an eye on the counters on the bottom.