Resources to Find the Data You Need, 2016 Edition

Before you get started on any data-related project, you need data. I know. It sounds crazy, but it’s the truth. It can be frustrating to sleuth for the data you need, so here are some tips on finding it (the openly available variety) and some topic-specific resources to begin your travels.

This is an update to the guide I wrote in 2009, which as it turns out, is now mostly outdated. So, 2016. Here we go.

Demographics

I’ve been spending more time than usual in this area. The most reliable data comes from government organizations such as the Census Bureau and the Bureau of Labor Statistics. At least, this is the case in the United States. I can’t speak for other countries’ organizations.

The challenge is to sift through outdated government websites and portals with meaningless acronyms. Data.gov was an effort to relieve some of the pain, but I’ve found it much more helpful to go through Google.

DATA SOURCES

  • Census Bureau — They provide general population data and breakdowns, along with much more information via the American Community Survey. Their downloading portal, American FactFinder, does take some getting used to, but there’s a lot of data available there at your fingertips. Unless you want more historical data pre-2000. In this case, Google search is your friend, because even if it’s on the Census site somewhere, you probably won’t find it through the web navigation.
  • IPUMS — Maintained by the Minnesota Population Center, this makes Census microdata much easier to download.
  • Bureau of Labor Statistics
  • Organisation for Economic Co-operation and Development — For global indicators.
  • UNdata — Also for global indicators.

Health

Health data was supposed to be easy to access years ago with various government initiatives, but that never quite caught on. There are sites, but they’re not especially usable. Instead, it’s typically easier to go to the source.

DATA SOURCES

Geographic

There’s been a growing number of places to download geographic data it seems, but there are a handful that have been around for a while and remain reliable.

DATA SOURCES

News

If you’re interested in exploring the data that you see in a news graphic, reputable news sources provide their source either in the graphic or in the accompanying text. Usually it doesn’t lead you directly to the source, but it tells you where to look.

DATA SOURCES

Sports

You’d think there would be a ton of sports data available with Sabermetrics taking off, but most of that data is private and cost a lot. You can still download historical box score type data though pretty easily.

DATA SOURCES

General Purpose

There was a time when a whole bunch of data sites existed to share and upload data, but those are gone or no longer exist in their original format. Still, there are some.

DATA SOURCES

Scraping and APIs

Then there are the less formatted routes to getting data. You can scrape data from websites or make use of openly available APIs. I use the Google APIs sometimes and still use the Python library BeautifulSoup every now and then.

This route requires some programming, but it can be worth a look.

General Tips in Your Search

This isn’t a comprehensive list. Not even close. But it should be enough to get you started and give you an idea for where to search. My searches typically start at Google, trying to be as specific as possible. If I don’t find anything at first, then I go more and more general.

Sometimes I’ll look for related data graphics that people made and then see if they provide their source. It’s customary these days. Side note: If there isn’t a source listed, you probably shouldn’t trust the graphic. But yeah, there is so much visualization out there that you can usually find something.

Finally, if you’re just looking for inspiration or for some data to poke at, there is always the Data Sources category here on FlowingData.

Become a member. Support an independent site. Make great charts.

See What You Get

Favorites

Life expectancy changes

The data goes back to 1960 and up to the most current estimates for 2009. Each line represents a country.

Toilet Paper Calculator

Maybe you’re starting to run low. Here’s how much you’ll need when you go to restock.

Pizza Place Geography

Most of the major pizza chains are within a 5-mile …

The Most Unisex Names in US History

Moving on from the most trendy names in US history, …