Resources to Find the Data You Need, 2016 Edition

By Nathan Yau  /  Posted to Guides  /  Tags: ,

Before you get started on any data-related project, you need data. I know. It sounds crazy, but it’s the truth. It can be frustrating to sleuth for the data you need, so here are some tips on finding it (the openly available variety) and some topic-specific resources to begin your travels.

This is an update to the guide I wrote in 2009, which as it turns out, is now mostly outdated. So, 2016. Here we go.

Demographics

I’ve been spending more time than usual in this area. The most reliable data comes from government organizations such as the Census Bureau and the Bureau of Labor Statistics. At least, this is the case in the United States. I can’t speak for other countries’ organizations.

The challenge is to sift through outdated government websites and portals with meaningless acronyms. Data.gov was an effort to relieve some of the pain, but I’ve found it much more helpful to go through Google.

DATA SOURCES

  • Census Bureau — They provide general population data and breakdowns, along with much more information via the American Community Survey. Their downloading portal, American FactFinder, does take some getting used to, but there’s a lot of data available there at your fingertips. Unless you want more historical data pre-2000. In this case, Google search is your friend, because even if it’s on the Census site somewhere, you probably won’t find it through the web navigation.
  • IPUMS — Maintained by the Minnesota Population Center, this makes Census microdata much easier to download.
  • Bureau of Labor Statistics
  • Organisation for Economic Co-operation and Development — For global indicators.
  • UNdata — Also for global indicators.

Health

Health data was supposed to be easy to access years ago with various government initiatives, but that never quite caught on. There are sites, but they’re not especially usable. Instead, it’s typically easier to go to the source.

DATA SOURCES

Geographic

There’s been a growing number of places to download geographic data it seems, but there are a handful that have been around for a while and remain reliable.

DATA SOURCES

News

If you’re interested in exploring the data that you see in a news graphic, reputable news sources provide their source either in the graphic or in the accompanying text. Usually it doesn’t lead you directly to the source, but it tells you where to look.

DATA SOURCES

Sports

You’d think there would be a ton of sports data available with Sabermetrics taking off, but most of that data is private and cost a lot. You can still download historical box score type data though pretty easily.

DATA SOURCES

General Purpose

There was a time when a whole bunch of data sites existed to share and upload data, but those are gone or no longer exist in their original format. Still, there are some.

DATA SOURCES

Scraping and APIs

Then there are the less formatted routes to getting data. You can scrape data from websites or make use of openly available APIs. I use the Google APIs sometimes and still use the Python library BeautifulSoup every now and then.

This route requires some programming, but it can be worth a look.

General Tips in Your Search

This isn’t a comprehensive list. Not even close. But it should be enough to get you started and give you an idea for where to search. My searches typically start at Google, trying to be as specific as possible. If I don’t find anything at first, then I go more and more general.

Sometimes I’ll look for related data graphics that people made and then see if they provide their source. It’s customary these days. Side note: If there isn’t a source listed, you probably shouldn’t trust the graphic. But yeah, there is so much visualization out there that you can usually find something.

Finally, if you’re just looking for inspiration or for some data to poke at, there is always the Data Sources category here on FlowingData.

Learn to visualize your data. Become a member.

Join Today

Membership

This is for people who want to learn to make and design data graphics. Your support goes directly to FlowingData, an independently run site.

What You Get

  • Instant access to tutorials on how to make and design data graphics
  • Source code and files to use with your own data
  • Four-week course on visualization in R
  • Hand-picked links and resources from around the web

Favorites

Reviving the Statistical Atlas of the United States with New Data

Due to budget cuts, there is no plan for an updated atlas. So I recreated the original 1870 Atlas using today’s publicly available data.

Where Bars Outnumber Grocery Stores

A closer look at the age old question of where there are more bars than grocery stores, and vice versa.

Years You Have Left to Live, Probably

The individual data points of life are much less predictable than the average. Here’s a simulation that shows you how much time is left on the clock.

Shifting Incomes for American Jobs

For various occupations, the difference between the person who makes the most and the one who makes the least can be significant.