Here's the technical definition of great circles on Wikipedia:

A great circle, also known as a Riemannian circle, of a sphere is the intersection of the sphere and a plane which passes through the center point of the sphere, as distinct from a small circle. Any diameter of any great circle coincides with a diameter of the sphere, and therefore all great circles have the same circumference as each other, and have the same center as the sphere. A great circle is the largest circle that can be drawn on any given sphere. Every circle in Euclidean space is a great circle of exactly one sphere.

The important bit is that the shortest distance between two points on a sphere is the minor arc of a great circle. When currents and wind don't interfere, ships and aircraft use great circle routes, which makes it perfect to show air carrier coverage. This is what you'll do in this example.

It turns out these maps are relatively easy to make in R once you know how to put the pieces together. The maps that I posted on flight connections for each airline are what we're after. With only about 30 lines of code, you can produce a series of maps that show flights for every major airline, so you get a lot of bang for the amount of effort.

Step 0. Setup

1. I wish they'd update the site; it totally looks like something out of the 1990s, but nevermind that. It's useful software.You're going to use R in this example, so download the free and open-source software if you haven't already. It's a straightforward one-click install¹.

Step 1. Load packages

Open R. You need two packages to do the heavy-lifting: maps and geosphere. If they're not installed, you can do that via the main menu Packages & Data > Package Installer. Once installed, load the two packages as follows:

library(maps)
library(geosphere)

The first package maps, is used to draw the base maps, and the second, geosphere is used to draw the great circle arcs.

Step 2. Draw base maps

The maps package makes it easy to draw geographic areas in R with the map() function. Pass it a database name, and you you get a map in one line of code. For example, to map the United States, type the following in the R console.

map("state")

Here's the map that you get:

Blank state map created in R. Nothing else.

2. Map projections are finicky in R and can be a pain sometimes, but if you want to map with a different projection like say, Albers, look into mapproject() in the maps package.The projection isn't the prettiest thing in the world, but it'll do for now². The bigger problem is that the state database doesn't include Alaska or Hawaii. To include the two often left out states, you use the world database.

map("world")

This gives you a full black and white map of the world.

A blank world map is just as easy to make in R.

Step 3. Limiting boundaries

The data at hand, which you'll get to soon, is only domestic flights for the United States, so you should focus on that area. Use the xlim and ylim variables to limit the map to a rectangle that only covers a range of latitude and longitude.

You can also play with the color of the base map at this point. By default, maps() doesn't fill regions, but it will if you set fill to TRUE. For some reason though, if you set the fill color, you can't change the border color. So instead (if you don't want to edit in Illustrator later) you can set the line width (lwd) to something really skinny. For the purpose of this example, we want the the border lines to get out of the way.

xlim <- c(-171.738281, -56.601563)
ylim <- c(12.039321, 71.856229)
map("world", col="#f2f2f2", fill=TRUE, bg="white", lwd=0.05, xlim=xlim, ylim=ylim)

And here's what you get with the above code. A simple and clean map of the United States that includes Alaska and Hawaii.

Focusing on all states. That includes Hawaii and Alaska.

Step 4. Draw connecting lines

Now that you have a map, you can draw connecting lines. This is really easy with gcIntermediate() from the geosphere package. Pass it the latitude and longitude of the two connecting points, and gcIntermediate() spits out the coordinates of points on the circle.

The n argument indicates how many points you want the function to return. The more points you indicate, the smoother the resulting line will be, but up to a certain point, you won't see much difference. The addStartEnd argument indicates that you want to include the start and end points in the great circle coordinates. Lastly, use lines() to actually draw the line.

lat_ca <- 39.164141
lon_ca <- -121.640625
lat_me <- 45.213004
lon_me <- -68.906250
inter <- gcIntermediate(c(lon_ca, lat_ca), c(lon_me, lat_me), n=50, addStartEnd=TRUE)
lines(inter)

This draws a great circle arc from California to Maine.

A single connection.

Similarly, you can add another line simply by doing the same as above with different latitude and longitude coordinates. For example, you can draw a line from California to Texas, and you can use the col argument to set the line color to red.

lat_tx <- 29.954935
lon_tx <- -98.701172
inter2 <- gcIntermediate(c(lon_ca, lat_ca), c(lon_tx, lat_tx), n=50, addStartEnd=TRUE)
lines(inter2, col="red")

Okay, now let's do two connections.

Step 5. Load flight data

So now you know how to do the hard part. You just have to iterate over latitude and longitude pairs to get the full map for a specific carrier. Start by loading the data with read.csv(), as shown below:

airports <- read.csv("http://datasets.flowingdata.com/tuts/maparcs/airports.csv", header=TRUE) flights <- read.csv("http://datasets.flowingdata.com/tuts/maparcs/flights.csv", header=TRUE, as.is=TRUE)

3. I used a five-line Python script to aggregate data that I downloaded from the Bureau of Transportation Statistics. The original file was about 50mb. You can aggregate in R, but it's usually a better idea to not load large-ish files in R. Luckily airport latitude and longitude coordinates were available on the page for Data Expo 2009. Otherwise, I would've geocoded them myself, which I started to do and then got stalled by API limits.This is processed data that I cleaned up for this tutorial. It's flight counts between each airport, categorized by airline³.

Step 6. Draw multiple connections

Data in. To map all connections for say, American Airlines, you filter as shown in line 3. Then you loop over each row of data, which has latitude/longitude for two two airports and the number of flights between them.

map("world", col="#f2f2f2", fill=TRUE, bg="white", lwd=0.05, xlim=xlim, ylim=ylim)

fsub <- flights[flights$airline == "AA",]
for (j in 1:length(fsub$airline)) {
	air1 <- airports[airports$iata == fsub[j,]$airport1,]
	air2 <- airports[airports$iata == fsub[j,]$airport2,]

	inter <- gcIntermediate(c(air1[1,]$long, air1[1,]$lat), c(air2[1,]$long, air2[1,]$lat), n=100, addStartEnd=TRUE)

	lines(inter, col="black", lwd=0.8)
}

Here's the mess of black lines that you get from the above code.

Rough image of American Airlines flights.

Not bad, but you can do better than that with some rearranging and coloring appropriately.

Step 7. Color for clarity

You learned how to change the color of lines back in step 4. You change the col argument in lines(). You hard-coded the color though. You could instead create a vector of colors that scaled from, say, light gray to black, and then pick a shade from farther down the vector for connections with more flights. That way connections with more flights would be more prominent.

If only there were a way to create a color scale automagically. Oh wait, there is. It's called colorRampPalette(). Pass it the base colors you want to use, and it'll fill in everything in between. More specifically, it creates a function that you can pass a number two, indicating how many shades you want to use. In the code below, we use 100 (lines 1 and 2).

Then you do the same as you did the previous step, but instead of setting all lines to black, you calculate the color based on how many fewer flights the current connection has compared to the maximum flight count.

pal <- colorRampPalette(c("#f2f2f2", "black"))
colors <- pal(100)

map("world", col="#f2f2f2", fill=TRUE, bg="white", lwd=0.05, xlim=xlim, ylim=ylim)

fsub <- flights[flights$airline == "AA",]
maxcnt <- max(fsub$cnt)
for (j in 1:length(fsub$airline)) {
	air1 <- airports[airports$iata == fsub[j,]$airport1,]
	air2 <- airports[airports$iata == fsub[j,]$airport2,]

	inter <- gcIntermediate(c(air1[1,]$long, air1[1,]$lat), c(air2[1,]$long, air2[1,]$lat), n=100, addStartEnd=TRUE)
	colindex <- round( (fsub[j,]$cnt / maxcnt) * length(colors) )

	lines(inter, col=colors[colindex], lwd=0.8)
}

Below is the map that you get. The problem is the longer, less prominent flights are obscuring the more popular connections, because they're being drawn on top. The above code just draws lines in the order that the data comes.

Use color to emphasize more prominent flights.

To fix this we use the same method that Paul Butler used for his Facebook map. We just order connection from least to greatest flight counts. That way less popular connections are drawn first and therefore, will be on the bottom, while the darker connections will be drawn on top.

pal <- colorRampPalette(c("#f2f2f2", "black"))
pal <- colorRampPalette(c("#f2f2f2", "red"))
colors <- pal(100)

map("world", col="#f2f2f2", fill=TRUE, bg="white", lwd=0.05, xlim=xlim, ylim=ylim)

fsub <- flights[flights$airline == "AA",]
fsub <- fsub[order(fsub$cnt),]
maxcnt <- max(fsub$cnt)
for (j in 1:length(fsub$airline)) {
	air1 <- airports[airports$iata == fsub[j,]$airport1,]
	air2 <- airports[airports$iata == fsub[j,]$airport2,]

	inter <- gcIntermediate(c(air1[1,]$long, air1[1,]$lat), c(air2[1,]$long, air2[1,]$lat), n=100, addStartEnd=TRUE)
	colindex <- round( (fsub[j,]$cnt / maxcnt) * length(colors) )

	lines(inter, col=colors[colindex], lwd=0.8)
}

Layer dark on top of light so that it's easier to read.

That's much better. At this point, you can play around with color, by changing the shades in colorRampPalette(). Below uses a light gray (#f2f2f2) to red, but you can do whatever you like. You can even use more than two colors.

pal <- colorRampPalette(c("#f2f2f2", "red"))

Red? Sure, you can do that, too.

Step 8. Map every carrier

The only thing left to do is make a map for each carrier. You can do it manually by changing the airline code and the rerunning the script, but there's an easier way. Find all unique carriers with the unique() function, and iterate over each one. The place that you put "AA" you replace with carriers[i] to indicate the current carrier in the loop. The code below will create a PDF for each carrier and save to your current working directory.

# Unique carriers
carriers <- unique(flights$airline)

# Color
pal <- colorRampPalette(c("#333333", "white", "#1292db"))
colors <- pal(100)

for (i in 1:length(carriers)) {

	pdf(paste("carrier", carriers[i], ".pdf", sep=""), width=11, height=7)
	map("world", col="#191919", fill=TRUE, bg="#000000", lwd=0.05, xlim=xlim, ylim=ylim)
	fsub <- flights[flights$airline == carriers[i],]
	fsub <- fsub[order(fsub$cnt),]
	maxcnt <- max(fsub$cnt)
	for (j in 1:length(fsub$airline)) {
		air1 <- airports[airports$iata == fsub[j,]$airport1,]
		air2 <- airports[airports$iata == fsub[j,]$airport2,]

		inter <- gcIntermediate(c(air1[1,]$long, air1[1,]$lat), c(air2[1,]$long, air2[1,]$lat), n=100, addStartEnd=TRUE)
		colindex <- round( (fsub[j,]$cnt / maxcnt) * length(colors) )

		lines(inter, col=colors[colindex], lwd=0.6)
	}

	dev.off()
}

Here is the map for American Airlines again, produced by the code above. I fiddled with color some to match the maps I created for the original flight post.

A dark map background with gray to white to blue paths.

That's all there is to it. Can you think of other datasets this method could be applied to? Give this tutorial a whirl and post your results in the comments.

For more examples, guidance, and all-around data goodness like this, sign up for FlowingData membership.

A bubble chart can also just be straight up proportionally sized bubbles, but here we're going to cover how to create the variety that is like a scatterplot with a third, bubbly dimension.

The advantage of this chart type is that it lets you compare three variables at once. One is on the x-axis, one is on the y-axis, and the third is represented by area size of bubbles. Have a look at the final chart to see what we're making.

Step 0. Download R

We're going to use R to do this, so download that before moving on. It's free and open-source, so you have nothing to lose. Plus it's a need-to-know-name of 2011, so you might as well get to know it now. You can thank me later.

Step 1. Load the data

Assuming you already have R open, the first thing we'll do is load the data. We're examining the same crime data the we did for our last tutorial. I've added state population this time around. One note about the data. The crime numbers are actually for 2005, while the populations are for 2008. This isn't a huge deal since we're more interested in relative populations than we are the raw values, but keep that in mind.

Okay, moving on. You can download the tab-delimited file here and keep it local, but the easiest way is to load it directly into R with the below line of code:

crime <- read.csv("http://datasets.flowingdata.com/crimeRatesByState2008.csv", header=TRUE, sep="\t)

You're telling R to download the data and read it as a comma-delimited file with a header. This loads it as a data frame in the crime variable.

Step 2. Draw some circles

Now we can get right to drawing circles with the symbols() command. Pass it values for the x-axis, y-axis, and circles, and it'll spit out a bubble chart for you.

symbols(crime$murder, crime$burglary, circles=crime$population)

Run the line of code above, and you'll get this:

Circles incorrectly sized by radius instead of area. Large values appear much bigger.

All done, right? Wrong. That was a test. The above sizes the radius of the circles by population. We want to size them by area. The relative proportions are all out of wack if you size by radius.

Step 3. Size the circles correctly

To size radiuses correctly, we look to the equation for area of a circle:

Area of circle = πr²

In this case area of the circle is population. We want to know r. Move some things around and we get this:

r = √(Area of circle / π)

Substitute population for the area of the circle, and translate to R, and we get this:

radius <- sqrt( crime$population/ pi )
symbols(crime$murder, crime$burglary, circles=radius)

Circles correctly sized by area, but the range of sizes is too much. The chart is cluttered and unreadable.

Yay. Properly scaled circles. They're way too big though for this chart to be useful. By default, symbols() sizes the largest bubble to one inch, and then scales the rest accordingly. We can change that by using the inches argument. Whatever value you put will take the place of the one-inch default. While we're at it, let's add color and change the x- and y-axis labels.

symbols(crime$murder, crime$burglary, circles=radius, inches=0.35, fg="white", bg="red", xlab="Murder Rate", ylab="Burglary Rate")

Notice we use fg to change border color, bg to change fill color. Here's what we get:

Scale the circles to make the the chart more readable, and use the fg and bg arguments to change colors.

Now we're getting somewhere.

By the way, you can make a chart with other shapes too with symbols(). You can make squares, rectangles, thermometers, boxplots, and stars. They take different arguments than the circle. The squares, for example, are sized by the length of a side. Again, make sure you size them appropriately.

Here's what squares look like, using the below line of code.

symbols(crime$murder, crime$burglary, squares=sqrt(crime$population), inches=0.5)

You can use squares sized by area instead of circles, too.

Let's stick with circles for now.

Step 4. Add labels

As it is, the chart shows some sense of distribution, but we don't know which circle represents each state. So let's add labels. We do this with text(), whose arguments are x-coordinates, y-coordinates, and the actual text to print. We have all of these. Like the bubbles, the x is murders and the y is burglaries. The actual labels are state names, which is the first column in our data frame.

With that in mind, we do this:

text(crime$murder, crime$burglary, crime$state, cex=0.5)

The cex argument controls text size. It is 1 by default. Values greater than one will make the labels bigger and the opposite for less than one. The labels will center on the x- and y-coordinates.

Here's what it looks like.

Add labels so you know what each circle represents.

Step 5. Clean up

Finally, as per usual, I clean up in Adobe Illustrator. You can mess around with this in R, if you like, but I've found it's way easier to save my file as a PDF and do what I want with Illustrator. I uncluttered the state labels to make them more readable, rotated the y-axis labels, so that they're horizontal, added a legend for population, and removed the outside border. I also brought Georgia to the front, because most of it was hidden by Texas.

Here's the final version. Click the image to see it in full.

Cleanup and a key make the chart more informative.

And there you go. Type in ?symbols in R for more plotting options. Go wild.

For more examples, guidance, and all-around data goodness like this, buy Visualize This, the new FlowingData book.

FlowingData reader Chris asks:

I was wondering, have you ever considered doing a Chernoff faces tutorial for R? I think Chernoff faces are pretty interesting and I haven't seen much about them on the web.

This wasn't the first time someone's asked how to make Chernoff faces, so I did a quick search. Guess what. There's an R package for that. This tutorial describes how to apply Chernoff faces to your own data.

Chernoff Faces

The point of Chernoff faces is to display multiple variables at once by positioning parts of the human face, such as ears, hair, eyes, and nose, based on numbers in a dataset. The assumption is that we can read people's faces easily in real life, so we should be able to recognize small differences when they represent data. Now that's a pretty big assumption, but debate aside, they're fun to make.

1. Because these are faces rather than abstract geometric shapes, be careful what you show with this method and who you show it to. As was the case in this tutorial, those who aren't familiar with the method might take the faces literally and take offense.We've seen them applied to baseball players and judge ratings. In this tutorial, we'll look at US crime rate by state.¹

Download R

Like in previous tutorials, we'll be using R (surprise, surprise), the software environment for statistical computing and graphics, to make our Chernoff faces, so if you haven't already, download and install R first before moving on. It's free, open-source, and a one-click install. Go on, I'll wait for you.

Step 1. Install package

Once you've opened up R, the first thing we need to do is install the aplpack (Another Plot Package) package by Peter Wolf. Go to the the "Packages & Data" menu in R, and select the "Package Installer." Select "CRAN (binaries)" in the dropdown menu if it's not already on that, and then click on "Get List." Scroll down to "aplpack" and click on the "Install Selected" button and installation should begin.

The Another Plot Package will do most of the grunt work.

Alternatively, you can also just type this in the R console:

install.packages("aplpack")

Step 2. Load the data

Next we need to load the data into the R environment. Like I said, we'll be looking at crime rates by state. I got the data from Infochimps, which is actually from Table 301 of the 2008 US Statistical Abstract, but it's typically a headache going through dot gov navigation, so I avoid it when I can.

I cleaned the datafile I got from Infochimps a little bit more so it only includes the numbers we're interested in. You can find it here, but you don't need to download it. We'll load it directly into R via the URL using the read.csv() command.

crime <- read.csv("http://datasets.flowingdata.com/crimeRatesByState-formatted.csv")

To view the data, type the following:

crime[1:6,]

This shows you the first six lines of our dataset. Note that there are eight columns. The first column is state name, with the exception of the row for US average and District of Columbia later on. The rest of the columns are seven categories of crime.

Step 3. Make some faces

Once the data is in, it's actually really easy to make some faces using the faces() function from the aplpack package. So far we've only installed the package, so now we'll load it:

library(aplpack)

If you get errors when you try to load, you might want to check to see if you installed the package correctly.

Okay, let's make some faces:

faces(crime[,2:8])

Here we're telling R to use the faces() function, using columns 2 through 8 of our crime data. Remember, the first column is state name. You get something that looks like this:

Default Chernoff Faces using faces()

Step 4. Change Features

This is pretty much what we want except for two things. The first is that the faces are labeled with numbers. That isn't of much use without a key. The second is that some of the faces are smiling. For more positive datasets like quality of life or baseball stats, that would make sense. The higher the value, the better. This is crime data though. The higher the value, the worse. Smiles for rate of larceny theft doesn't seem quite right.

Unfortunately, the faces() function doesn't let us choose what face parts to associate with each metric, so we need to find a workaround. According to the documentation (view by typing ?faces), the curve of the smile is applied to the sixth column in the input matrix, which is crime in this case.

Ah. Here's what we'll do. We make the sixth column in our data all the same value. That way all smile curves will be neutral. Here's how we can do that:

crime_filled <- cbind(crime[,1:6], rep(0, length(crime$state)), crime[,7:8])

The cbind() function combines multiple columns to form a matrix. In the above, we combine the first six columns of crime, stick a column of zeros whose length matches the number of rows in our crime data, and then we end with the last two columns in crime. We save the new matrix into a variable called crime_filled. Similar to in Step 2, you can type the following to see the first rows of crime_filled.

crime_filled[1:6,]

Notice the new column of zeros?

Now use faces() with crime_filled:

faces(crime_filled[,2:8])

We get similar faces, but with no more smiles:

Using different features to indicate variables

Step 5. Add labels

Instead of numbers, it'd be much more useful to include state names. Easy.

faces(crime_filled[,2:8], labels=crime_filled$state)

It's the same as previous, but we use the labels argument to use the state column in crime_filled to label with state names.

Add state name labels so it's not so ambiguous.

Much more useful now. We can easily associate the faces with a state. It's a little cluttered, but we can fix that up easy in Illustrator.

Step 6. Fix up in Illustrator (optional)

You can pretty much stop here if you like, but as most of you know, I like to save the image as a PDF, bring it into Adobe Illustrator (aff), and clean things up to make it more readable. You can also try Inkscape, the open-source alternative, although I've never tried it.

After some label cleanup and some annotation, here's our final result. What's going on there Washington, D.C.?

Uncluttered labels, unambiguous features, and cited data source

Not too bad, right?

Read the R documentation on faces() for more details on what else you can do with the function. Remember, documentation is your friend when it comes to making full use of R.

Now go on. Have some fun with your new Chernoff toy.

For more examples, guidance, and all-around data goodness like this, order Visualize This, the FlowingData book on visualization, statistics, and design.

Maybe you have observations over time or it might be two variables that are possibly related. In either case, a scatter plot just might not be enough to see something useful. A fitted line can let you see a trend or relationship more easily.

As an example, we'll take a look at monthly unemployment data, from 1948 to February this year, according to the Bureau of Labor Statistics.

What LOESS is

First, let's briefly go over what we're actually doing with this loess thing. LOESS stands for locally weighted scatterplot smoothing. It was developed [pdf] in 1988 by William Cleveland and Susan Devlin, and it's a way to fit a curve to a dataset.

If we plot unemployment without any lines or anything fancy, it looks like this:

Dot plot showing unemployment over time

Most of us are familiar with fitting just a plain old straight line. The end result is a slope and an intercept. You know the whole y=mx + b equation back from middle school?

Scatterplot with a linear fit, y = mx + b

So without going into the nitty-gritty, the above fit looks at all the data and then fits a line. Loess however, moves along the dataset, and looks at chunks at a time, fitting a bunch of smaller lines that connect to make one smooth line.

Alright, enough background. On to the how-to.

Step 0. Download R

You've already done this, right? If not, you can download it for Windows, Mac, or Linux. Don't let the out-dated site full you. You can get a lot done with the free software, and it'll be a simple one-click install for most.

Step 1. Load the data

Like I said, I got the data from the Bureau of Labor Statistics. You can download it here in CSV format if you like, but we'll load it directly into R with the following:

unemployment <- read.csv("http://datasets.flowingdata.com/unemployment-rate-1948-2010.csv", sep=",")

You're basically telling R to load data in the unemployment variable from the given URL, and columns are separated by commas.

Once it's loaded, take a brief look by typing unemployment[1:10,]. Your screen will look something like this:

As usual, you load your data in R before you start anything else

There are four columns, but we're actually just going to use that last one: Value.

Step 2. Time to plot

Yup, it's already time to make the scatterplot with fitted curve:

scatter.smooth(x=1:length(unemployment$Value), y=unemployment$Value)

Since we're only looking at unemployment, the x-axis is just a sequence from 1 to the total number of observations. Here's what the above line will give you.

Fit a LOESS curve to the dots

Not bad, right? Two lines of code, and you've already got your plot. We can do a little better though. Let's fix it up a bit.

Step 3. Modify axis limits

It's usually a good idea to start your values axis at zero if you can. The above graph doesn't start at zero, so let's fix that using the ylim argument to make it go from 0 to 11.

scatter.smooth(x=1:length(unemployment$Value), y=unemployment$Value, ylim=c(0,11))

Update the axes to start at zero

That's a little better. Now let's do something about the color.

Step 4. Modify colors

I want the curve to stand out some more. Everything blends together as it is now. We'll use the col argument to change the dots to light gray:

scatter.smooth(x=1:length(unemployment$Value), y=unemployment$Value, ylim=c(0,11), col="#CCCCCC")

Make the fitted the line the point of interest and put dots in the background

Step 5. Save as PDF and do whatever

So at this point, you can fuss around with arguments to tweak. Just type ?scatter.smooth to read documentation on the function. As many of you know though, I like to take it into Adobe Illustrator at this point. This just happens to be what works for me. There are lots of ways to edit PDF files.

Anyways, after some color changes, and label cleanup, we're done.

Title, color, cite, and fonts

Tada. And it only took two lines of code. How about that? Give it a try for yourself, and happy graphing.

For more examples, guidance, and all-around data goodness like this, pre-order Visualize This, the upcoming FlowingData book.

Back in 1990, Ben Shneiderman, of the University of Maryland, wanted to visualize what was going on in his always-full hard drive. He wanted to know what was taking up so much space. Given the hierarchical structure of directories and files, he first tried a tree diagram. It got too big too fast to be useful though. Too many nodes. Too many branches.

The treemap was his solution. It's an area-based visualization where the size of each rectangle represents a metric since made popular by Martin Wattenberg's Map of the Market and Marcos Weskamp's newsmap.

Here's a really easy way to make your own treemap in just a couple lines of code. We're looking to make something like the above.

Step 0. Download R

Like before, we're going to use R, so you'll want to get it before going any further. Download it for Windows, Mac, or Linux. Don't let the out-dated site full you. You can get a lot done with the free software.

Step 1. Load the Data

We'll use data covering a hundred popular posts on FlowingData. Here it is in CSV format. You don't have to download it though. We'll just load it directly into R. The main thing to take note of is what is there. There's post id, number of views, number of comments, and category.

Okay, let's load it into R using read.csv():

data <- read.csv("http://datasets.flowingdata.com/post-data.txt")

Loading data in CSV format into R.

Easy enough. We just used the read.csv() function to load data from a URL. If your data is on your computer, you could also do something like data <- read.csv("post-data.txt"). Just make sure the data file is in your current working directory, which you can change via the "Miscellaneous" menu.

Step 2. Load the Portfolio Library

Only a few more lines of code, and you've got a treemap. It's so easy, because we're going to use the portfolio library in R. First, you have to install it. You can either install the library via the "Package Installer" or you can do it through the command line. Let's do the latter. Type this in the console to install portfolio:

install.packages("portfolio")

Once installed, load it into R:

library(portfolio)

Step 3. Make the Treemap

It's time to make the treemap with map.market(). Type this in the console:

map.market(id=data$id, area=data$views, group=data$category, color=data$comments, main="FlowingData Map")

Tada. You should get something like this:

The default treemap uses a red-green color scale.

To sum up, we did this with four lines of code:

data <- read.csv("http://datasets.flowingdata.com/post-data.txt")
install.packages("portfolio")
library(portfolio)
map.market(id=data$id, area=data$views, group=data$category, color=data$comments, main="FlowingData Map")

Step 4. Customize

Now maybe you want to modify something like color. The cool thing about R is that you can see the code for all the functions, edit it, and then use your customized version. If the green and red scheme isn't for you or you don't care about the positive/negative cutoff, then you can change the code to do that. I won't go into detail, but if you type map.market in the console, you'll see the function. You can change color or cutoff around lines 36-46.

For example, you can do a black and white color scheme:

You don't have to stick to the default color scale though.

I was alright with the green for this, so I saved it as a PDF and then loaded it into Illustrator as usual. I numbed the green some, cleaned up the labels with a new font and layout, and updated the legend.

Touched up version of treemap with black-green color scale.

And there you go - a treemap with just a few lines of code in our all-trusty R. Rinse and repeat with your own data.

For more examples, guidance, and all-around data goodness like this, order Visualize This, the FlowingData book on visualization, design, and statistics.

The Heatmap

In case you don't know what a heatmap is, it's basically a table that has colors in place of numbers. Colors correspond to the level of the measurement. Each column can be a different metric like above, or it can be all the same like this one. It's useful for finding highs and lows and sometimes, patterns.

On to the tutorial.

Step 0. Download R

We're going to use R for this. It's a statistical computing language and environment, and it's free. Get it for Windows, Mac, or Linux. It's a simple one-click install for Windows and Mac. I've never tried Linux.

Did you download and install R? Okay, let's move on.

Step 1. Load the data

Like all visualization, you should start with the data. No data? No visualization for you.

For this tutorial, we'll use NBA basketball statistics from last season that I downloaded from databaseBasketball. I've made it available here as a CSV file. You don't have to download it though. R can do it for you.

I'm assuming you started R already. You should see a blank window.

Initial R window when you open it. Exciting, I know.

Now we'll load the data using read.csv().

nba <- read.csv("http://datasets.flowingdata.com/ppg2008.csv", sep=",")


We've read a CSV file from a URL and specified the field separator as a comma. The data is stored in nba.

Type nba in the window, and you can see the data.

What the data looks like when you load it into R

Step 2. Sort data

The data is sorted by points per game, greatest to least. Let's make it the other way around so that it's least to greatest.

nba <- nba[order(nba$PTS),]


We could just as easily chosen to order by assists, blocks, etc.

Step 3. Prepare data

As is, the column names match the CSV file's header. That's what we want.

But we also want to name the rows by player name instead of row number, so type this in the window:

row.names(nba) <- nba$Name


Now the rows are named by player, and we don't need the first column anymore so we'll get rid of it:

nba <- nba[,2:20]


Step 4. Prepare data, again

Are you noticing something here? It's important to note that a lot of visualization involves gathering and preparing data. Rarely, do you get data exactly how you need it, so you should expect to do some data munging before the visuals. Anyways, moving on.

The data was loaded into a data frame, but it has to be a data matrix to make your heatmap. The difference between a frame and a matrix is not important for this tutorial. You just need to know how to change it.

nba_matrix <- data.matrix(nba)


Step 5. Make a heatmap

It's time for the finale. In just one line of code, build the heatmap (remove the line break):

nba_heatmap <- heatmap(nba_matrix, Rowv=NA, Colv=NA,
col = cm.colors(256), scale="column", margins=c(5,10))


You should get a heatmap that looks something like this:

Default cyan to purple heatmap

Step 6. Color selection

Maybe you want a different color scheme. Just change the argument to col, which is cm.colors(256) in the line of code we just executed. Type ?cm.colors for help on what colors R offers. For example, you could use more heat-looking colors:

nba_heatmap <- heatmap(nba_matrix, Rowv=NA, Colv=NA,
col = heat.colors(256), scale="column", margins=c(5,10))


Changing to heat colors with the col argument

For the heatmap at the beginning of this post, I used the RColorBrewer library. Really, you can choose any color scheme you want. The col argument accepts any vector of hexidecimal-coded colors.

Step 7. Clean it up - optional

If you're using the heatmap to simply see what your data looks like, you can probably stop. But if it's for a report or presentation, you'll probably want to clean it up. You can fuss around with the options in R or you can save the graphic as a PDF and then import it into your favorite illustration software.

I personally use Adobe Illustrator, but you might prefer Inkscape, the open source (free) solution. Illustrator is kind of expensive, but you can probably find an old version on the cheap. I still use CS2. Adobe's up to CS4 already.

For the final basketball graphic, I used a blue color scheme from RColorBrewer and then lightened the blue shades, added white border, changed the font, and organized the labels in Illustrator. Voila.

Updated heatmap in Illustrator with clearer labels and a blue-white color scale

Rinse and repeat to use with your own data. Have fun heatmapping.

For more examples, guidance, and all-around data goodness like this, buy Visualize This, the new FlowingData book.

You've seen the NameExplorer from the Baby Name Wizard by Martin Wattenberg. It's an interactive area chart that lets you explore the popularity of names over time. Search by clicking on names or typing in a name in the prompt. It's simple. It's sexy. Everybody loves it.

This is a step-by-step guide on how to make a similar visualization in Actionscript/Flash with your own data and how to customize the design for whatever you need. We're after last week's graphic on consumer spending (above).

Audience

This tutorial is for people with at least a little bit of programming experience. I'll try to make it as straightforward as possible, but the concepts might be a little hard to grasp if you've never written a line of code. Just a heads up. Of course it never hurts to try.

If you don't care about customization or integration into an application and don't mind putting your data in the public domain, you could also just dump your data into Many Eyes, and use the Stack Graph.

Get Adobe Flex Builder

Like I said, this is all in Actionscript, so before we start anything, I strongly recommend you get Adobe Flex Builder if you don't already have it. You can buy it, get a trial version from the Adobe site, or if you're in education, you can get it for free.

There are ways to compile Actionscript without Flex Builder, but they are more complicated.

Working With Flare

Luckily you don't have to start from scratch. In fact, most of the work has already been done for you using Jeffrey Heer's Flare visualization toolkit. It's an Actionscript library. We're going to work off one of the sample applications: JobVoyager. Once you get your development environment setup, it's just a matter of switching in your data, and then customizing the look and feel.

Okay, let's get started (finally).

Step 0. Download and import Flare

Download the most recent version of Flare, and then unpack the contents into your working directory.

Open Flex Builder. You should see something like this:

Right click on the Flex Navigator (left-hand side) and click on "Import..." You'll get a popup that looks like this:

Select "Existing Projects into Workspace" and click "Next." Browse to where you put the Flare files. Select the "flare" directory, and then make sure "flare" is checked in the project window.

Do the same thing with the "flare.apps" folder. Your Flex Builder window should look like this once you've expanded the flare.apps/flare/apps/ and click on JobVoyager.as.

If you clicked the run button right now (the green button with the white play triangle top left), you should see the working JobVoyager. Get that working, and you're done with the hardest part.

Step 1. Update the data source

Let's dive into the code now so you can adapt the visualization to your own data and customize the aesthetics. Again, we're going to adapt this code to consumer spending data to make last week's graphic.

First we need to change the data source. This is specified on line 57.

private var _url:String = "http://flare.prefuse.org/data/jobs.txt";
 

Change the _url to point at the spending data, which like jobs.txt, is also a tab-delimited file. The first column is year, the second category, and the last column is expenditure:

private var _url:String = "http://datasets.flowingdata.com/expenditures.txt"
 

Now the file will read in our spending data instead of for jobs. Easy stuff so far.

Step 2. Change the years

The next two lines, line 58 and 59 are the column names, or in this case, the distinct years that job data was available. It's by decade from 1850 to 2000. We could make things more robust by finding the years in the loaded data, but since the data isn't changing we can same some time and explicitly specify the years.

Our expenditures data is annual from 1984 to 2008. We'll change lines 58-59 accordingly.

private var _cols:Array = [1984,1985,1986,1987,1988,1989,1990,1991,
		1992,1993,1994,1995,1996,1997,1998,1999,
		2000,2001,2002,2003,2004,2005,2006,2007,2008];
 

Step 3. Data headers

Next we need to change references to the data headers. The original data file (jobs.txt) has four columns: year, occupation, people and sex. Our spending data only has three columns: year, category, and expenditure. We have to adapt the code to this new data structure.

Luckily, it's pretty easy. The year column is the same, so we just need to change any people references to expenditure (vertical axis) and any occupation references to category (the layers). Finally, we'll remove all uses of gender.

At line 74 the data is reshaped and prepared for the stacked area chart. It specifies by occupation and sex as the categories (i.e. layers), uses year on the x-axis, and people on the y-axis.

var dr:Array = reshape(ds.nodes.data, ["occupation","sex"],
     "year", "people", _cols);
 

Change it to this:

var dr:Array = reshape(ds.nodes.data, ["category"],
     "year", "expenditure", _cols);
 

We only have one category (sans sex), and that's uh, category. The x-axis is still year, and the y-axis is expenditure.

Step 4. Sorting

Line 84 sorts the data by occupation (alphabetically) and then sex (numerically). We'll just sort by category:

data.nodes.sortBy("data.category");
 

Are you starting to get the idea here?

Step 5. Proper categories

Line 92 colors layers by sex, but we don't have that split in our data, so we don't need to do that. Remove the entire row:

data.nodes.setProperty("fillHue", iff(eq("data.sex",1), 0.7, 0));
 

We'll come back to customizing the colors of the stacks a little later.

Step 6. Labeling areas

Line 103 adds labels based occupation:

_vis.operators.add(new StackedAreaLabeler("data.occupation"));
 

We want to label based on spending category, so we'll change the line accordingly:

_vis.operators.add(new StackedAreaLabeler("data.category"));
 

Step 7. Interactive filters

Lines 213-231 handle filtering in JobVoyager. First there's the male/female filter and then there's the filter by occupation. We don't need the former, so we can get rid of lines 215-218 and then make line 219 a plain if statement.

Similarly, lines 264-293 at buttons to trigger the male/female filter. We can get rid of that too.

Step 8. Search categories

We're getting really close to fully customizing the voyager to our spending data. Go back to the filter() function at line 213. Again, we need to update the function so that we can filter by spending category instead of occupation.

Here's line 222 as-is:

var s:String = String(d.data["occupation"]).toLowerCase();
 

Change occupation to category:

var s:String = String(d.data["category"]).toLowerCase();
 

Step 9. Color scheme

If you ran the code right now, everything should compile correctly, and it'll look something like this:

Color is specified in two places. First lines 86-89 specify stroke color and colors everything red:

shape: Shapes.POLYGON,
lineColor: 0,
fillValue: 1,
fillSaturation: 0.5
 

Then line 105 updates saturation (the level of red), by count. The code for the SaturationEncoder() is in lines 360-383. We're not going to use saturation though. Instead, we'll explicitly specify the color scheme.

First update lines 86-89 to this:

shape: Shapes.POLYGON,
lineColor: 0xFFFFFFFF
 

We're going to make stroke color white with lineColor. If there were more spending categories, we probably wouldn't do this because it'd be cluttered. We don't have that many though, so it'll make reading a little easier.

Next, make an array of the colors we want to use ordered by levels. Put it towards the top around line 50:

private var _reds:Array = [0xFFFEF0D9, 0xFFFDD49E, 0xFFFDBB84,
    0xFFFC8D59, 0xFFE34A33, 0xFFB30000];
 

I used the ColorBrewer for these colors.

Then we'll add a new ColorEncoder around line 110:

var colorPalette:ColorPalette = new ColorPalette(_reds);
_vis.operators.add(new ColorEncoder("data.max", "nodes", "fillColor", null, colorPalette));
 

Tada, you now have something that looks like what we're after:

Here's the finished product.

Step 10. Download the code and see for yourself

To play with this yourself, just complete step 0 and then replace JobVoyager.as with this file. It has all the updates we've just covered.

Step 11. Where to Go From Here

There's a lot of things you can do with this. You can apply this to your own data, use a different color scheme, and further customize to fit your needs. Maybe change the font or the tooltip format. Then you can get fancier and integrate it with other tools or add more Actionscript, so on and so forth. If anything, you should at least check out what else you can do with Jeffrey's Flare visualization toolkit.

Have fun!

For more examples, guidance, and all-around data goodness like this, pre-order Visualize This, the upcoming FlowingData book.