What the heck’s a data guy? According to Gerard, who studied computer science and economics in college

It means that I’m the type of person who, instead of planning for a vacation like a normal person, will write a script to pull down airline data for all possible destinations and routes, load the data into R and perform a regression analysis to find the best time to buy.

Oh, so that’s what a data guy is. I guess that makes me a data guy.

This should be good for Swivel, who has seemed to be missing the “data guy” piece of the puzzle. Will Swivel’s visualization tools improve? Will data become more reliable on Swivel? I don’t know. It’s possible. There’s definitely a lot of work to be done, so one person won’t be enough, but hey, it’s a start. It’s not often that I see a computer science / economics person. I’m an electrical engineering and computer science / statistics person myself, and I like to see people with dual backgrounds (even if they did go to the other school across the bay).

That being said, applications like Swivel, Many Eyes, and Data360 make me wonder where all the statisticians are. I see mathematicians, designers, economists, and businessmen. Come on statisticians. Show yourselves. The world needs you.

Anheuser-Busch (Budweiser), Miller, and Coors lead the way in beer. Albeit, this is shipment data, not sales data, so take the numbers with a grain of salt.

The extreme dominance of the top three American beers was somewhat surprising to me, because I never see people order any of those three at restaurants. However, I gave it a few more seconds of thought. I’m thinking parties, sporting events, and drunken nights. The American beers go down easier (because they’re like water), so it’s easier to get drunk. To get drunk, people drink more. So I guess the watery dominance isn’t that surprising. I guess when people buy beer for taste at restaurants, they look to different brands.

Anyhow, I’m really starting to become a fan of these bubble charts. They’re really easy to read and can quickly spruce up a hard-to-read table of numbers. They also seem to scale decently. By well, I don’t mean in like the thousands, but in the tens, I think the bubbles can hold their own.

What kind of beer do you prefer?

UCLA Statistics has a pretty extensive list of resources on how to use R and GRASS. For those unfamiliar, R is a programming language and environment for statistical computing and graphics. GRASS is an open source geographic information system (GIS). And of course, both are completely free and completely useful.

The StatGrad discussion board is now online — a place where stat students can hang out.

One of the things I miss most about going to school is hanging out with my cohort. I work from home in Buffalo, and I get bored and restless pretty easily. When I was at school and feeling restless, I could just go down to the stat lounge, sit on the ridiculous-looking Ikea couch, and relax with some classmates. We never sat around and talked about probability theory or the law of large numbers (ok, maybe we did sometimes), but because we were all stat students, we all had this data-ish way of thinking. Know what I mean?

That’s what I’m hoping for StatGrad. I’m not interested in finding help for specific stat problems or trying to answer R questions. There are plenty of books and online resources for that type of stuff. I’m just hoping that StatGrad can become a place where stat grad students can hang out when they’re bored. Complain about undergrads, discuss anything interesting happening in our field, look for job opportunities, and stay up to date on calls for papers.

Join StatGrad now. I know you want to. Please? Come on, I’m bored.

This venn diagram showing results from tests for Autism really seems to be making its rounds lately. It began with Igor Carron asking on his blog if there was a better way to display the data. Then Andrew Gelman put something of a redesign challenge up on his blog, and after Andrew, the challenge headed on over to Junk Charts. Redesigns are flying off the wall! From bar, to mosaic, to tornado charts, there’s clearly many ways to represent data.

Which one is the best? It’s hard to say, because they all have advantages and disadvantages and the answer really depends on what point you’re trying to drive home.

However, I can find one advantage that the original venn diagram has over its redesigns — it’s intuitive for many people. John Venn introduced his diagram in 1881, over a century ago. That’s a long time for people to adjust. People understand it. It makes sense. Yes, this particular venn is really ugly and probably didn’t belong in a Powerpoint presentation, but doesn’t it say something that re-designers were able to read it and use the data it provided? I think so.

So in the spirit of Indexed, here’s to you Mr. Venn.

Some time last month, Many Eyes introduced their text visualization, the word tree. The user starts from a word or phrase, which is the root (or the trunk?) of the tree and then the branches are the continuation of the sentence in which the word appeared. The advantage over the word tree is that the order of words stays the same, as opposed to a jumbled tag cloud:

Hence, the word tree allows the user to gain a better understanding of text flow and writing patterns than she would with a cloud.

I found that it was very easy to create a word tree with some text that I had uploaded, but while starting exploration, I was unsure about what words to begin with. The word tree interface is similar to Martin Wattenberg’s earlier Baby Name Wizard. The user naturally has some ideas on what to start with since it’s an exploration of names. However, with the word tree, it’s not as obvious, because the user might be exploring a body of text she’s unfamiliar with.

So instead I began sifting with a word cloud, which gave me an idea of some important words and phrases used in the text. Then it was simple to move from the word cloud to the word tree. The two viz tools — cloud and tree — go together quite nicely as the cloud kind of works as a suggestion box for the tree. As a standalone, the word tree is off to a good start.

I’ve never really been interested in baseball. I’ve always been more of a basketball and football fan. However, my summer roommate was a die hard baseball fan, and I’m convinced that he brainwashed me into rooting for the New York Mets. Just a couple of weeks ago, someone told me he was a Phillies fan, and I let out a blech of disgust without even thinking about it.

So with the Mets’ most recent loss, I’m a bit disgruntled, and I’m sure my old roommate is pissed as can be. The Mets are no longer leading the Phillies for the number one spot in the NL east.

What better way to see how poorly the Mets are playing than with a graphic? I decided to compare this year’s Met season with the 1986 Met World Series winning season, because that should probably be what they’re shooting for. As my roommate would angrily exclaim, “If they can’t get their #%&$ act together, they don’t serve to go to the playoffs!”

I saw this map of the average snow levels in Buffalo. I think I just glanced at it and that was about it. When you first look at the map, what do you make of the colors? When I see green for snow levels, I think no snow. Am I crazy? What do you think?

So the image was kind of in my head all this summer while I was in NYC. When I told people that I was going back to Buffalo after my internship, they always gave this look that said, “Ha, have fun during the winter,” and then they would actually say it and then go into how they measure the snow level by comparing it against a giant pole.
Read More

Mint was released last week. It’s an online application that brings financial data from all of your credit card and bank accounts into one place. Think Quicken online and free.

It’s super easy (only takes a few seconds) to add your financial accounts, and you only have to do it once. After you’ve added your accounts, Mint will update your data every night and compile them into useful reports. You’ll get an overview of spending trends, transactions, and even ways you can save money based on your current credit cards’ interest rates.

So far I’ve found it useful simply because all of my data is one place. As I’ve made my way into adulthood, I’ve slowly accumulated more and more credit cards to the point where it’s kind of annoying to login to every account to see how much debt I have.

One Small Annoyance

My one gripe about Mint is that the spending trends and savings features haven’t been that informative, but I imagine will get better once more data comes in and Mint continues to tweak the system. My highest hope is that they do something about the dreaded 3-d pie chart…

Overall though, I’m looking forward to seeing Mint grow and develop into an extremely useful tool that brings all of your data into one place and represents it in a way that’s understandable and interesting.

Friday was my last day at The Times, and this past Sunday, my last graphic ran in the paper. The story discussed Judge Michael Mukasey’s past rulings and experience. Who’s Michael Mukasey? He’s up for the spot of the new attorney general of course.

Anyways, I got to look through a lot of cool data on past rulings and busted out R for some statistical fun. This brought me to my last graphic. It compared Mukasey to his New York Southern District peers. You can see he’s been more strict overall but less strict in immigration cases. Unfortunately some spacing between each bar was lost in the web version; it looks much better in the actual paper.

That’s not the most exciting part though.

Read More

When I think airplanes and data visualization, I think of Aaron Koblin’s Flight Patterns. Aaron uses data from the Federal Aviation Administration to show flights all across the United States, including Hawaii and Alaska. Even without the presence of an actual map, you can see a basic geography and where lots of flights are going and coming from. Flight Patterns is an oldie, but still a goodie. Here’s a video:

Speaking of flights, I’m currently waiting for my twice-delayed flight back to Buffalo. Thank goodness for free WiFi. Although it still doesn’t make up for the delays. I hereby shake my mental fist of rage at you, Jet Blue.

John Maeda, a professor in the MIT Media Lab, gives his talk on simplicity and how it plays a role in his position between technology and art. I read John’s book, The Laws of Simplicity, a few months ago, and yes, as many will tell you, it’s a pretty simple book. There are ten laws of simplicity that boil down to the main point — get rid of everything that’s unnecessary and nothing more. Although nothing earth-shattering, John’s book makes some good points and has some interesting anecdotes from his many trips to Japan and family life; it’s a nice read for some lazy Sunday. He’s also a pretty entertaining speaker, so sit back, relax, and enjoy yet another TED talk.

Today is my last day at The New York Times. Ten weeks and twenty something graphics later, I’m leaving NYC much more knowledgeable about data visualization and journalism and how they can make a powerful pair. It’s a bittersweet ending today.

On the one hand, it’s been amazing working for such a prominent newspaper, but on the other, I’m also looking forward to taking a few days off doing nothing and then moving forward towards finishing, err, starting my dissertation. Do I know my topic? No, not really, but one thing’s for sure. Data visualization is what I want to do and I’ve been extremely fortunate to have learned from some of the best this summer.

Onward ho.

My roommate pointed out a couple of weeks ago that I always get Chinese takeout for dinner; however, we never get home at the same time, and most days, she’s not even in the apartment when I arrive. How could she, a very bright and educated individual, come to such a conclusion after seeing so little data?

In fact, by my count, she only saw me bring home Chinese takeout twice before she decided that yes, I do in fact eat Chinese every single day of the week. In reality I rotate through four choices — sandwiches, Japanese, pizza, or Chinese with a few ventures out every now and then. This week I’ve had Japanese, hot dogs, Mediterranean twice, sandwich, burger, and Chinese.

This is one of the reasons we need Statistics. What we perceive isn’t always the truth. I might have had Chinese takeout on Monday and Friday, but do you know what I had on the days in between? If no, can you make an educated guess?

I’m thinking it might be time to revive my step count data collection with a nifty USB pedometer from Brando.

This Pedometer can store 3 days of step data and upload the data to your PC via USB! Through your data, the software can chart your outcome, view the calories burned and details on your daily activies. You can get easily to control your weight by this Pedometer and no over weight anymore!

If I had this, it wouldn’t be such a big deal if I forgot to record a couple of days. As I noted in a previous post, one of the difficulties of getting good step data was simply getting it into the spreadsheet. This bad boy records 3 days worth of data. Plus the USB and software, I imagine, could make record-keeping a lot smoother. Plus no over weight anymore!

Worth investigating, I think? The release date is somewhere November 2007. I’m about 1 percent positive that this could very well be as popular as the iPhone.

[via Gizmodo]

For those interested in or who already use Stata, the first Stata users group on the west coast is coming up on October 25-26. It’s $150 for both days and of course students get a good discount at only $50. I’m an R user myself, but to each his own.

Stata Users Group meetings started in Britain in 1995 and have spread to Italy, Sweden, Germany, The Netherlands, Spain, Australia, and the East Coast. Talks are intended to be accessible to a general audience with mixed levels of expertise in Stata and statistics. Stata developers will also attend, both to present new Stata features and to take notes during the popular “Wishes and grumbles” session. We hope you will consider joining the meeting as a presenter or an attendee.

When I tell people that I’m a graduate student in Statistics, there are two responses that I get more than any others. The most popular of the two usually goes something like this.

Oh man, I hated statistics in college. The professor totally sucked and I never knew what was going on. All I remember is mean and some… curve thing? I don’t know. What’s standard deviation anyways?

I threw that standard deviation bit in for effect. No one actually asks about it, and I’m pretty sure most people don’t even remember ever hearing about it. It’s that whole selective memory thing — blocking out the bad and remembering only the happies.

So anyways, every time someone tells me they absolutely hated statistics in college, I die a little inside and start bawling like a two-year-old whose lost her bottle. No, no, I’m kidding, but the first thing I think is, “Gee, thanks for letting me know that! Like I really wanted to know that you hate what I study. You know what? I think I hate you a little bit now.” I’m exaggerating a tad, but it’s slightly frustrating after hearing it so many times.

But why do so many people hate statistics?
Read More

With a stricter border patrol, more Mexican illegal immigrants are taking dangerous routes to get into the United States. As a result, treks through the dehydrating Arizona desert have caused a significant number of deaths. Most likely there are more deaths than this graph indicates because the data was only for deaths reported by the Border Patrol. There could very well be cases the Border Patrol did not handle or knew about.

This graph was straightforward, mainly a waiting game for data from first, the Government Accountability Office and then the Border Safety Initiative. Take a look at the GAO report done last year, reporting a double in border-crossing deaths from 1995 to 2005. It’s a little odd though that they use numbers from two different sources, so take it with a grain of salt.

I actually did this graphic some time last month for the Week in Review. Slipped through the cracks somehow. It was a slightly different experience doing a graphic for this desk, because, well, I guess they don’t request graphics very often.

As an aside, I just realized that old Times links are behind that silly TimesSelect thing, which kind of sucks. I hear TimesSelect is going to be free sometime in the near future though. Good.

In a previous life, I thought anything published in an academic journal was legit, but as a stat student, the story is quite the opposite. Whenever I hear results or see data from some study, I become an instant skeptic.

Were there really that many deaths from 1998 to 2007? Did housing prices really increase that much over the past decade? Do that many people really support that presidential candidate?
Read More