Time Magazine’s multimedia section has a fun, little piece showing some statistics for a day in the life of the average American. There’s some mapping for average commute time, annual traffic delays, and city population shifts. I’m not a huge fan of the map on the third dimension, but oh well.
There’s also a simple grid ranking jobs by level of happiness. Priests are apparently the happiest with gas station attendants at the very bottom. Poor gas station attendants. I guess I might classify myself as a computer programmer which is somewhere in between waiters and dress makers. Maybe I should consider a change in focus. Although, I could also consider myself an engineer which is towards the top of the rankings. Alright, I’m an engineer. The title of “computer programmer” has a weird stigma attached to it anyways.
For our humanflows visualization, we used data from the United Nations Common Database and the Migration Information Source. The great thing about these types of sources is that they are publicly available so that everyone gets to have fun with the data. The downside is that the data is accessible via a user interface that often makes it a chore to get all of the data.
Hence, to save you some time, you can now download the migration database that we used. I don’t see any reason why you have to go through the whole data importing process when we already did it. Enjoy!
Disclaimer: Keep in mind that the data is from the United Nations and Migration Information Source, so you should refer to the two sites for any documentation. In a nutshell, the inflows table is from MIS and the rest is from United Nations. If you’re looking for more, you might also want to check out OECD. I really wanted to use their data at the time, but was having trouble accessing it from Spain.
You used to only be able to get a small thumbnail to “share” the visualization you found or created on Many Eyes (well, outside of taking screenshots and emailing), but Many Eyes just announced the embed feature. In the same way you can embed YouTube videos, you can embed Many Eyes visualizations. This is a really big step forward, because users can share what they’ve found or seen more easily and as a result, it’s more likely others will become drawn in. You know, it’s that whole viral marketing thing.
Just one weird thing. I had to change the single quotes in the copy and paste snippet to double quotes for the embedding to work, because my version (or all versions?) of WordPress escapes the single quotes.
We like our names. And that preference can have negative repercussions, according to research published last month. Major leaguers with “K” initials tend to strike out more, perhaps reflecting the batters’ unconscious pull to appear next to the strikeout symbol “K” on scorecards. Students with initials C and D have worse grades than the A’s and B’s and everyone else, gravitating toward the grades their initials represent.
Of course, I’m a little skeptical about all of these studies, and with tiny effects like 0.02, these studies probably deserve it. In any case, they’re still interesting to read about. I wonder how one could get his hands on such data. The data’s probably just an email away, but in my current half-asleep stooper, I’ll leave that for another time. I’m sure it’d be really interesting to play with though.
Have you read Freakonomics? If you have, all of these name studies remind me of that chapter about the two brothers named Winner and Loser. If you haven’t read the book, uh, there’s a chapter on two brothers named Winner and Loser.
Most are familiar with the Netflix Prize. If you’re not, Netflix has offered a one million dollar prize to whoever improves their movie recommendation by a certain amount. It’s been going on for a little over a year with still no grand prize winner. The dataset is 100 million ratings.
The above is a visualization of the Netflix dataset. Each dot represents a movie, and the closer two dots are the more similar the two corresponding movies are based on Netflix ratings. I’m guessing the orientation of the dots was decided by some variant of multidimensional scaling.
It’s kind of fun to scroll over the clusters. Like in the bottom right we see Babylon 5, Buffy the Vampire Slayer, Alias, and Battlestar Galactica clumped together. The giant blob in the middle, however, is pretty useless; it’d probably benefit from some zoom functionality.
The Need to Explore
I’m kind of surprised that I haven’t seen more Netflix visualizations like this (or ones better than this), because I’m pretty sure it would help see some relationships that typical analysis won’t provide. I was browsing the forum and saw someone ask if others had had success loading the 100 million observation dataset into R. Silly undergrad.
A computer scientist, designer, and statistician walk into a bar; they discuss how they would boost the Netflix recommendation system. The punchline is that they win a million dollars, but I’m not sure what happens in between.
The Visualizar Showcase is officially open and ready for public viewing, so if you’re in Madrid (and I’m about 80% sure you will be) from now until January 5, 2008 check out the projects spawned from two weeks of hard work. You can find a complete list of the projects at the Visualizar website, but here are a few of my favorites in no particular order.
Mail Garden
Mail Garden, from Kjen Wilkens, explores emails under a garden metaphor with the implication that our email is in someway living (like all data). In the visualization, emails exist as plants and as you scroll over you can read each email. The best part of of Mail Garden though is probably when you’re not using it. When the system is idle, you can watch your plants (your emails) gently sway back and forth in the wind.
TweetPad
As if Twitter weren’t playful enough, TweetPad, by Elie Zananiri, is a visualization that lets you playfully explore the live Twitter feed. Elie’s main interest was in word interaction, and you can see that clearly in the TweetPad. Move the cursor clockwise for synonyms, back and forth to shuffle words, and counter clockwise to revert back to the original tweet all the while the Twitter feed is coming at you live.
Spamology
This visualization, as you might have guessed, explores one of the most popular canned meats in the world. No, just kidding. Spamology, by Irad Lee, explores email spam. The visualization is nice as you explore the small and giant buildings of spam, but it’s the sound accompaniment that really makes it. Sound corresponds to the height of each spam building. Usually, pieces like this end up sounding like noise, but this was more like beautiful music.
Now before I cover every work, which I’m a little tempted to do, I’ll stop here. If you happen to be in Madrid, Spain, go check it out. If you read this blog, you’re more than likely to enjoy the projects on display at the Medialab… or you can watch it on the news. Visualizar was also featured on some news show in Madrid. Be patient. The segment on the workshop comes some time around ten and a half minutes.
Yahoo: Look Google, I’ve got a Flash charts API now. I make it easier for people to plot their data, and look, pretty colors.
Google: So what. Look what I’ve got. I have URL-based chart creation with fun, cartoon-ish Google colors. My API is way easier, and plus, since I’m Google, everyone will use my API and not yours.
Y: Why are you so mean to me? We both have two O’s in our name. Can’t we be friends?
G: No. That’s right, you heard me. I’m better. Now kiss my feet.
Sigh, poor Yahoo. Right after Yahoo released their flash-based charting API, Google proudly announces a super simple charting API of their own. The idea is very straightforward. It all starts with the URL http://chart.apis.google.com/chart and from there
Josh Tynjala of the Yahoo! Flash Platform team contributes the new YUI Charts Control, a hybrid JavaScript/Flash component that supports bar, line, and pie charts. The Charts Control draws data from the same DataSource Utility that underpins the YUI DataTable Control, making it possible to do combined chart/table visualizations. The Charts Control accepts CSS style information, allowing you to skin the chart itself without touching the underlying .swf file. But if you do want to dig into the Flash side of this project, you can get full access to those assets on the ASTRA site.
What does this mean? It means that we’re probably going to see a lot more hack-ish looking charts online (example above); but we might also see some nice-looking charts since it seems like they’re potentially customizable. In any case, it’s good to see this. There’s some cruddy Flash-based chart libraries that people are actually charging money for. This free and open library should have some positive effects.
I made a few tweaks and our humanflows visualization prototypes are now online. There’s a bit of information on how humanflows came about, who was involved, and a day-by-day recap of the design process. Once you get to the prototypes section, give the applets a few seconds to load and hopefully you’re not disappointed. The interaction is pretty intuitive. All you have to do is click and hold to browse the flow lines and the map. Also, if you can, go full screen on your browser. It looks much better that way (and how it was intended to be shown).
Again, I’d like to thank Miguel, Iman, and Monica for making my trip to Spain and the Visualizar workshop a memorable experience. Thank you!
The New York Times recently put up a cool data exploration tool to sift through the transcript of the most recent Republican debate. They call it the transcript analyzer. There are three key features:
View where candidates put in their two cents indicated by the blue, highlighted rectangles
Read the actual chunks of transcript for each block
Search the transcript to see when specific words and phrases were used indicated by the smaller gray highlighted rectangles
My particular favorite is the search feature because it really allows readers to dig into the transcript or a reader can find out which candidate is (or isn’t) talking about his or her point of interest and when in the debate the topic was discussed. The intuitive text scrolling is pretty awesome too. Good job, New York Times!
After two weeks at Visualizar, I’m back in the United States. It’s good to be back. I don’t know how many people know this (because I certainly didn’t), but the people in Madrid (or all of Spain?) eat a ridiculous number of sandwiches. I spoke to a couple of locals who said it’s pretty common to eat two sandwiches a day every day. I’m all sandwiched out.
Anyways, the Visualizar symposium / workshop was a lot of fun, really interesting, and I ended up learning a lot more than I expected from some incredibly talented people. During my two weeks, I had the opportunity to work with designers Miguel Cabanzo, Iman Moradi, and Monica Sanchez and we managed to build a visualization framework that shows migration data with economic indicators. We call the piece humanflows.
Human Flows, the Piece
I just tried putting humanflows online, but of course it’s not working on my server right now (because all computers are against me), so I settled for a couple of screencasts. You’ll just have to take my word for it that the whole thing came together really nicely with a kiosk-looking type setup and a designer’s touch (three of them, actually). The visualization itself was done in Processing.
Here’s the first one that just shows the flows. Right off bat, you can see the huge rush to the United States (especially immigrants from Mexico).
This one shows the flows with unemployment rate.
We also did one with GDP, but you get the idea.
Of course, now that we have a framework, there’s so many other things that I can think of adding. Functionality like specific country selection and the ability to browse through other indicators would really allow some serious data exploration and since we were working with data form the United Nations Common Database, which has a hundreds of publicly available datasets, there’s a lot to work with.
So there it is. Humanflows.
Through the development process, I learned a lot about what I can do with Processing as well as gained an entirely different perspective on data visualization — a designer’s perspective. Simple concepts like color and more complex ideas like how to approach a large dataset are some of the things that I learned that I think are important for statisticians and the more technically-involved data people to know. I’ll cover that stuff in later posts though.
For now, I’d appreciate any comments on our visualization and any ideas on how to improve it. How would you visualize migration data?
I feel like it’s been forever since my last post, so I just wanted to let everyone know that I am not dead.
It’s the last few days here at Visualizar so I’ve got a couple of late nights ahead to make sure we get our project done, and on Wednesday, we set things up for the one-month exhibition. That should be fun. It’ll be especially nice to see everyone else’s work out on display.
The most interesting part about this workshop has probably been working and talking to designers about data visualization. I’m a statistician. Everyone else is a designer of some sort. With a statistics background and just coming off my New York Times internship, it felt really strange for the first week to go from the very literal and straightforward representation of data to the artsy, metaphorical data visualizations.
The defining moment — when I saw a huge difference between designers and statisticians’ views on visualization — was what followed after a talk from someone from the GapMinder foundation.
I’ll get into all of this stuff I’ve learned once I return to the lovely United States of America. In the meantime though, there was short blurb about the Visualizar workshop on We Make Money Not Art. There’s a picture of my back. I’m famous.
Oh, and if you’re really bored, the MediaLab has a Flickr stream. They’ve been taking tons of photos.
I’m staying in a hostel here in Madrid and am currently in the “Internet Room.” I’m on my laptop, but there are six desktop computers in front of me, all of which are occupied. Three of the six people have Facebook open plus myself. It’s come to the point that Facebook has so many ways to share information, that almost everyone can find some use for it. Is there some way to share data in some similar social way?
I know there’s some data blogging available and a few social data sites, but they don’t have the same feel as Facebook. I think the main reason people like Facebook (other than an entertaining way to waste a few hours) is because they personally relate to the information displayed and there’s some kind of connection between friends and strangers. Read More
It’s been a couple of days here in Madrid. It’s about 6:00am in the morning and I really should get back to bed. I’m incredibly jet lagged though, so that’s not really an option. The past couple of nights I’ve woken up at 4:30am and have had trouble falling back asleep. Anyone who knows me, knows that I’m very much a night person and always wake up late, so obviously, I’ve been feeling a little off the past couple of days.
Anyways, the past couple of days have been interesting. I flew in on Wednesday, and was extremely tired. I only slept maybe an hour on the plane. Once I came in, I got lost for several hours looking for the hostel and then the Medialab. That was fun.
I’ve joined this group of three graphic designers / media artists. We’re dealing with a good bit of migration data in a project now known as Humanflows, and a good bit of data means a lot of Statistics fun.
…
Um…
Loopdee loo…
OK, I’m finding myself in a bit of a daze at the moment, so I think I’ll pause it here, and resume a coherent thought later…
Have a good weekend :)
On a completely unrelated note, I just had real hot chocolate for the first time. I mean, it was like melted chocolate with cream. Delicious.
I was about to click away, but then I saw movement on the map. In addition to recent incidents, the map also has police unit tracking. You can see where certain units are at any given time as well as a video feed. That’s pretty cool. However, it doesn’t seem live, because every car is Officer Heinz, every car shows the same video, and the timestamp on the video shows November 2004. I guess it’s just a demo or prototype right now.
How cool would it be if that were live though? I can imagine plasma screens on the walls of every gang’s central control station. Crime could be transformed forever.
We all know fast food is incredibly bad for us and yet we still eat it. Why? Because it has tons of fat and tastes delicious. Nevermind that we will die a few days earlier for every French fry we eat.
Over at Calorie Counter, they try to make us feel guilty with numbers. Check out the Carl’s Jr. Double Six Double Dollar Burger with 1,520 Calories and a delicious 111 grams of fat. I’m a little surprised that it beat out the Burger King Triple Whopper with cheese. I shudder just thinking about eating one of those.
Anyways, there’s a whole lot of numbers here but not an incredible amount of meaning. How bad is bad? How much fat should I consume per day? Is 111 grams of fat bad? If yes, how will it directly affect me? Yes, 111 grams of fat is bad for you. You will directly feel the effects as you sit on the toilet in the morning wondering why it is taking you so long to take a dump. Now that’s context.
Also, with all the numbers, I bet all the tables would benefit from some kind of chart or, at the least, a simple infographic. Any takers? We should have a contest for who can make fast food the least appealing using nutritional data and without bending the truth.
Every day during the summer I walked past “Moveable Type” in The New York Times lobby. Since my adviser was one of the people working on it, I had the privilege to see it up close before the actual opening.
The picture is nice, but it’s nothing like standing there and experiencing the news. It’s especially nice to be in the middle of the two walls of panels (there’s a panel behind the photographer) and you get bits and pieces of the day’s paper and archive coming at you visually and um, auditorily. These bits and pieces are coming parsed from the paper in an intelligent (statistical) way. Listen to the NPR clip below to find out more. There’s also a video on The Times page.
Really, really great. Or as my adviser would say, “so sexy.” If you’re ever in the area, you should definitely take a look.
Many Eyes now has more detailed mapping functionality with the help of ESRI data. It was really only a matter of time before this happened. It’s come to the point where I almost instantly think ESRI when I think maps–that and The Times maps department (who frequently uses ESRI data :). Anyways, this is pretty nice looking stuff. They’ve got bubbles, color coding, and multiple maps in matrix form (to compare).
I didn’t get a chance to look at the maps in depth, but one thing that I noticed is that the region bubbles are only labeled if they’re at least a certain size. If they’re smaller than that threshold, then it’s just the bubble. I’m not sure what the threshold is, but I feel that it could be a bit lower so that more labeling can happen.
There’s also (of course) zoom-in, zoom-out, and panning– features we have come to expect from online mapping applications. Zoom and pan gets a little sluggish when there are multiple maps, but the feature still feels pretty useful.
When people I know can’t decide whether or not go to graduate school, I always encourage them to do so, because cool stuff like this happens. First I get to intern at The New York Times and now I’m headed to Madrid, Spain for two weeks to attend the Visualizar workshop. As you might have guessed, it’s a visualization workshop, and it’s headed by Benjamin Fry, Bestiario and Adrian Holovaty. I’m not sure who Bestiario and Adrian (although I will soon), but Ben is most recently known, or I guess most widely known for his work on Processing with Casey Reas.
There are ten projects, of which one I think I will be collaborating on. I’m not really sure how it’s going to work yet. Unfortunately I’m going to miss the conference part of Visualizar, because I couldn’t get to Spain soon enough on such short notice. I’m headed back to Buffalo on Monday (I’m in Los Angeles for the week) and then my flight to Spain is on Tuesday.
Sorry in Advance
Sorry in advance as my posts on FlowingData become a little sporadic during these two weeks, but I’ll be sure to write about the goings on in Spain while I’m there. I’m pretty sure it’s going to be really interesting and extremely educational.
Graphwise launched a few weeks ago, but I’m just hearing about it now, so I guess there hasn’t been a whole lot of buzz about this new application.
The Graphwise group has got a spider crawling the Web for data in HTML data tables and as a result, has accumulated a pretty big data warehouse. There’s currently 2,766,560 extracted tables in the Graphwise database. That’s pretty good, and I think they’re building on a pretty good idea. However, Graphwise advertises itself as three pieces of a three-piece puzzle — get data, visualize, and share.
To say the least, the visualize and share portions need work. Here’s a visualization from the front page:
I…I…don’t know what to say. Why the 3-d bars with the gradient background and the giant, semi-transparent Earth in the foreground blocking everything? It makes me want to throw up. It seriously looks like someone threw up data on the screen — data vomit. The javascript-enabled graphs seem to be making the browsing experience pretty sluggish too.
Am I being too harsh? My conscious is yelling at me for calling the graphs regurgitated food.
OK, OK. So to sum things up — the data warehousing and Web crawling are great. The spiders are clearly doing their job, so thumbs up for that. As for the visualizations, I, well, uh, it needs work (along with all the other junk that comes with running these types of data-centric applications).
Time Magazine’s multimedia section has a fun, little piece showing some statistics for a day in the life of the average American. There’s some mapping for average commute time, annual traffic delays, and city population shifts. I’m not a huge fan of the map on the third dimension, but oh well. 
For our 
Visualize This: The FlowingData Guide to Design, Visualization, and Statistics (2nd Edition)
