Resources to Find the Data You Need, 2016 Edition

Before you get started on any data-related project, you need data. I know. It sounds crazy, but it’s the truth. It can be frustrating to sleuth for the data you need, so here are some tips on finding it (the openly available variety) and some topic-specific resources to begin your travels.

This is an update to the guide I wrote in 2009, which as it turns out, is now mostly outdated. So, 2016. Here we go.

Demographics

I’ve been spending more time than usual in this area. The most reliable data comes from government organizations such as the Census Bureau and the Bureau of Labor Statistics. At least, this is the case in the United States. I can’t speak for other countries’ organizations.

The challenge is to sift through outdated government websites and portals with meaningless acronyms. Data.gov was an effort to relieve some of the pain, but I’ve found it much more helpful to go through Google.

DATA SOURCES

• Census Bureau — They provide general population data and breakdowns, along with much more information via the American Community Survey. Their downloading portal, American FactFinder, does take some getting used to, but there’s a lot of data available there at your fingertips. Unless you want more historical data pre-2000. In this case, Google search is your friend, because even if it’s on the Census site somewhere, you probably won’t find it through the web navigation.
• IPUMS — Maintained by the Minnesota Population Center, this makes Census microdata much easier to download.
• Bureau of Labor Statistics
• Organisation for Economic Co-operation and Development — For global indicators.
• UNdata — Also for global indicators.

Health

Health data was supposed to be easy to access years ago with various government initiatives, but that never quite caught on. There are sites, but they’re not especially usable. Instead, it’s typically easier to go to the source.

DATA SOURCES

• Centers for Disease Control and Prevention — There are various tables in PDFs and Excel spreadsheets (ehhh), but there are also things like WONDER which let you download larger sets, albeit with a clunky interface.
• World Health Organization

Geographic

There’s been a growing number of places to download geographic data it seems, but there are a handful that have been around for a while and remain reliable.

DATA SOURCES

• Natural Earth — Shapefiles for various features at various levels.
• TIGER — High-granularity boundary and feature files from the Census Bureau.
• OpenStreetMap
• ArcGIS Open Data

News

If you’re interested in exploring the data that you see in a news graphic, reputable news sources provide their source either in the graphic or in the accompanying text. Usually it doesn’t lead you directly to the source, but it tells you where to look.

DATA SOURCES

• ProPublica Data Store — Some for a fee and others that are free to download.
• New York Times — APIs to access NYT-related data.

Sports

You’d think there would be a ton of sports data available with Sabermetrics taking off, but most of that data is private and cost a lot. You can still download historical box score type data though pretty easily.

DATA SOURCES

• Sports Reference — Covers the major sports in the US.
• Betfair — Not sports directly, but gambling.

General Purpose

There was a time when a whole bunch of data sites existed to share and upload data, but those are gone or no longer exist in their original format. Still, there are some.

DATA SOURCES

• Wikipedia — Not all data of course, but there are a lot of HTML tables in there.
• Amazon Public Datasets — A handful of larger datasets.
• Awesome Public Datasets — A big hodgepodge catalog of open data.
• Corpora — “Potentially useful in the creation of weird internet stuff.”
• Berkeley Data Lab — Links to interesting data sources.

Scraping and APIs

Then there are the less formatted routes to getting data. You can scrape data from websites or make use of openly available APIs. I use the Google APIs sometimes and still use the Python library BeautifulSoup every now and then.

This route requires some programming, but it can be worth a look.

General Tips in Your Search

This isn’t a comprehensive list. Not even close. But it should be enough to get you started and give you an idea for where to search. My searches typically start at Google, trying to be as specific as possible. If I don’t find anything at first, then I go more and more general.

Sometimes I’ll look for related data graphics that people made and then see if they provide their source. It’s customary these days. Side note: If there isn’t a source listed, you probably shouldn’t trust the graphic. But yeah, there is so much visualization out there that you can usually find something.

Finally, if you’re just looking for inspiration or for some data to poke at, there is always the Data Sources category here on FlowingData.