Oftentimes, you'll want to fit a line to a bunch of data points to make it easier to spot patterns or relationships. It might be observations over time or it might be two variables that are possibly related. In either case, a scatter plot just might not be enough to see anything useful. This tutorial will show you how to graph a fitted line, or loess curve, to such a scatter plot.

As an example, we'll take a look at monthly unemployment data, from 1948 to February this year, according to the Bureau of Labor Statistics.
Read more →

Back in 1990, Ben Shneiderman, of the University of Maryland, wanted to visualize what was going on in his always-full hard drive. He wanted to know what was taking up so much space. Given the hierarchical structure of directories and files, he first tried a tree diagram. It got too big too fast to be useful though. Too many nodes. Too many branches.

The treemap was his solution. It's an area-based visualization where the size of each rectangle represents a metric since made popular by Martin Wattenberg's Map of the Market and Marcos Weskamp's newsmap.

Here's a really easy way to make your own treemap in just a couple lines of code. We're looking to make something like the above.
Read more →

How do you make a heatmap? This came from kerimcan in the FlowingData forums, and krees followed up with a couple of good links on how to do them in R. It really is super easy. Here's how to make a heatmap, with just a few lines of code, like this.
Read more →

You've seen the NameExplorer from the Baby Name Wizard by Martin Wattenberg. It's an interactive area chart that lets you explore the popularity of names over time. Search by clicking on names or typing in a name in the prompt. It's simple. It's sexy. Everybody loves it.

This is a step-by-step guide on how to make a similar visualization in Actionscript/Flash with your own data and how to customize the design for whatever you need. We're after last week's graphic on consumer spending (above).
Read more →

There are about a million ways to make a choropleth map. You know, the maps that color regions by some metric. The problem is that a lot of solutions require expensive software or have a high learning curve...or both. What if you just want a simple map without all the GIS stuff? In this post, I'll show you how to make a county-specific choropleth map using only free tools.

Here's what we're after. It's the most recent unemployment map from last week.
Read more →

Most of us create graphs with actual graphing software. Maybe it's Microsoft Excel. Maybe it's R. Whatever it is though it's usually specialized for analysis. What if you want to make a graphic for a publication or a presentation that's polished and fully customized? Adobe Illustrator gives you the control you need to do this. It's not graphing software. It's illustration software, but once you get the hang of things, Adobe Illustrator can be a valuable tool in your visualization arsenal.

In the rest of this post, I'll go over the steps to make the above graph in Illustrator. I used the U.S. immigration data from our recent contest.
Read more →

Following up on my post last week about using Twitter to track eating and weight, some of you voiced some interest in creating your own Twitter bot. This post covers how you can do that.

The Gist of It

Creating my own Twitter bot was pretty straightforward (much more than I thought it'd be), mostly because Twitter provides an API and the resources to make it that way.

I wanted something really simple that I could play around with. I just wanted to be able to send a direct message to my Twitter bot, and from there, it would store my data. OK, so here are the basic steps I took:

1. Create Twitter account for bot
2. Turn on email notification for direct messages only
3. Check email periodically for new direct messages
4. Parse direct messages and store in database

Read more →

Weather Underground is a useful site and a fun place for weather enthusiasts. WU has a bunch of weather data (current and historical) from established weather stations, like at airports, and home stations setup by hobbyists. One problem: most of the data is in HTML tables instead of the CSV format that we like. I say most because you can download hourly data from a single day in CSV, but if you want say, temperature data over the past 5 years, you're kind of at a loss.

But wait, there's a solution. That solution is BeautifulSoup, an XML/HTML parser written in Python. Um, parse... what does that mean? Basically, the Python script will go through, or look at, a document extracting certain information from that document.

Back to WU. Like I said, there's historical data in HTML tables like this. I just want the actual mean temperature in Fahrenheit for the past five years or so. I could go to every single page manually and record the temperature in Excel, but why do that when I can make the computer do it for me? Read more →