Resources to Find the Data You Need, 2016 Edition

By Nathan Yau  /  Posted to Guides  /  Tags: ,

Before you get started on any data-related project, you need data. I know. It sounds crazy, but it’s the truth. It can be frustrating to sleuth for the data you need, so here are some tips on finding it (the openly available variety) and some topic-specific resources to begin your travels.

This is an update to the guide I wrote in 2009, which as it turns out, is now mostly outdated. So, 2016. Here we go.

Demographics

I’ve been spending more time than usual in this area. The most reliable data comes from government organizations such as the Census Bureau and the Bureau of Labor Statistics. At least, this is the case in the United States. I can’t speak for other countries’ organizations.

The challenge is to sift through outdated government websites and portals with meaningless acronyms. Data.gov was an effort to relieve some of the pain, but I’ve found it much more helpful to go through Google.

DATA SOURCES

  • Census Bureau — They provide general population data and breakdowns, along with much more information via the American Community Survey. Their downloading portal, American FactFinder, does take some getting used to, but there’s a lot of data available there at your fingertips. Unless you want more historical data pre-2000. In this case, Google search is your friend, because even if it’s on the Census site somewhere, you probably won’t find it through the web navigation.
  • IPUMS — Maintained by the Minnesota Population Center, this makes Census microdata much easier to download.
  • Bureau of Labor Statistics
  • Organisation for Economic Co-operation and Development — For global indicators.
  • UNdata — Also for global indicators.

Health

Health data was supposed to be easy to access years ago with various government initiatives, but that never quite caught on. There are sites, but they’re not especially usable. Instead, it’s typically easier to go to the source.

DATA SOURCES

Geographic

There’s been a growing number of places to download geographic data it seems, but there are a handful that have been around for a while and remain reliable.

DATA SOURCES

News

If you’re interested in exploring the data that you see in a news graphic, reputable news sources provide their source either in the graphic or in the accompanying text. Usually it doesn’t lead you directly to the source, but it tells you where to look.

DATA SOURCES

Sports

You’d think there would be a ton of sports data available with Sabermetrics taking off, but most of that data is private and cost a lot. You can still download historical box score type data though pretty easily.

DATA SOURCES

General Purpose

There was a time when a whole bunch of data sites existed to share and upload data, but those are gone or no longer exist in their original format. Still, there are some.

DATA SOURCES

Scraping and APIs

Then there are the less formatted routes to getting data. You can scrape data from websites or make use of openly available APIs. I use the Google APIs sometimes and still use the Python library BeautifulSoup every now and then.

This route requires some programming, but it can be worth a look.

General Tips in Your Search

This isn’t a comprehensive list. Not even close. But it should be enough to get you started and give you an idea for where to search. My searches typically start at Google, trying to be as specific as possible. If I don’t find anything at first, then I go more and more general.

Sometimes I’ll look for related data graphics that people made and then see if they provide their source. It’s customary these days. Side note: If there isn’t a source listed, you probably shouldn’t trust the graphic. But yeah, there is so much visualization out there that you can usually find something.

Finally, if you’re just looking for inspiration or for some data to poke at, there is always the Data Sources category here on FlowingData.

Learn to visualize your data. Become a member.

Join Today

Membership

This is for people who want to learn to make and design data graphics. Your support goes directly to FlowingData, an independently run site.

What You Get

  • Instant access to tutorials on how to make and design data graphics
  • Source code and files to use with your own data
  • Four-week course on visualization in R
  • Hand-picked links and resources from around the web

Favorites

19 Maps That Will Blow Your Mind and Change the Way You See the World. Top All-time. You Won’t Believe Your Eyes. Watch.

Many lists of maps promise to change the way you see the world, but this one actually does.

Famous Movie Quotes as Charts

In celebration of their 100-year anniversary, the American Film Institute selected the 100 most memorable quotes from American cinema, and …

Interactive: When Do Americans Leave For Work?

We don’t all start our work days at the same time, despite what morning rush hour might have you think.

How You Will Die

So far we’ve seen when you will die and how other people tend to die. Now let’s put the two together to see how and when you will die, given your sex, race, and age.