Weather in the private sector is over a $1.5 billion industry, and it's largely because of the government's open weather data. You can find what the weather is just about anywhere with just a few clicks of the mouse. It wasn't always like that though. Clay Johnson, former director of Sunlight Labs, describes the history of open weather data, starting with Thomas Jefferson in the late 1700s.
Continue Reading
Data Sources
-
How weather data became open data
-
Afghanistan war logs revealed and mapped
This past Sunday, well-known whistle-blower site Wikileaks released over 91,000 secret US military reports, covering the war in Afghanistan. Each report contains the time, geographic location, and details of an event the US military thought was important enough to put on paper.
Continue Reading -
Data and its impact on journalism
In regards to the UK's recent boom in open data, Simon Rogers of the Guardian, ponders data's role in journalism, and the opportunities this new found information could bring:
The impact on journalism is expected to be great. The Chicago-based web developer and founder of the neighbourhood news site EveryBlock, Adrian Holovaty, says it's going to be challenging but exciting for journalists. "As more governments open their data, journalists lose privileged status as gatekeepers of information – but the need for their work as curators and explainers increases. The more data that's available in the world, the more essential it is for somebody to make sense of it."
This need not only creates a fresh brand of news, but also a new type of journalist:
I once prided myself on my lack of maths knowledge. Now I find myself editing a datajournalism site, the Guardian's datablog: a site where we use Google Spreadsheets to post key datasets. We make the data properly accessible, then encourage our users to take the numbers, produce graphics and applications and help us look for stories.
Priding yourself on a lack of know-how on how to deal with data is a little weird, but okay.
In any case, people always ask me how to get into information design, infographics, visualization etc. Journalism is one of those choices, and there's a lot of opportunity there if you've got the skills.
-
Egregious Citations Issued to BP
BP processes about 1.5 million barrels of crude oil per day, across six refineries in the United States. In total, 150 refineries in the United States process just under 18 million barrels per day, so BP processes about 8.5 percent of it. However, as reported by the Center for Public Integrity, 97 percent of the most dangerous violations found by OSHA were on BP properties.
Continue Reading -
Live webcast: Community Health Data Initiative
Health and Human Services (HHS) is about to announce the launch of their Community Health Data Initiative over in DC right now. The point is to make health data more usable for consumers and communities.
Today groups will be presenting how they've made use of the data in the past few weeks from about 9:30 to 10:30 - as in right now. I've embedded the live webcast below.
They're just going through the formalities of thank yous and intros right now, but the good stuff should start soon.
Continue Reading -
Twitter data buffet is back in business
Almost a year and a half ago, Infochimps, the data repository slash marketplace, released a giant scrape of Twitter data representing 2.7 million users, 10 million tweets, and 58 million connections. Twitter soon requested that they take it down while they figured out how they wanted to handle licensing, privacy, etc.
That was in 2008, before Twitter really started booming. Fast forward to now. Twitter and Infochimps have figured out what they want to do, and the Twitter census data is back up. It's no longer a measly 2.7 million users anymore though. The population has grown to 35 million.
Continue Reading -
World data released ‘is a dream come true’
In another step towards open data and all that jazz, the World Bank released World Development Indicators 2010 today, which is meant to serve as a progress report of the world.
The WDI provides a valuable statistical picture of the world and how far we've come in advancing development," said Justin Yifu Lin, the World Bank’s Chief Economist and the Senior Vice President for Development Economics. “Making this comprehensive data free for all is a dream come true.
More importantly though, this comes with the launch of the freely available online database and public API to 1,000+ indicators. There used to be a big fee for this data. I can't speak for the API, but the website is well-designed. It has profile pages for each country, links to download the indicators in Excel and XML, and hey, are those graphs implemented in HTML5? I spy
<canvas>tags.
Continue Reading -
TransparencyData makes campaign finance data easier to access
Anyone who's looked at campaign finance data knows it can get messy really quick (especially if you're getting it directly from the FEC). Sunlight Labs' newly launched TransparencyData aims to make the process a lot easier.
They've merged state data from FollowTheMoney and federal data from OpenSecrets and made it easy to search with a clickable interface. Select from a number of filters such as amount, recipient, or contributor, and then download data in bulk or make use of the API.
Continue Reading -
Buy and sell data at Data Marketplace
Add another site to the list of places to find the data you need. Data Marketplace connects people who want data to people who can find, scrape, and cull data.
Here's how it works. If you want data, you put in a request and optionally, a deadline and budget. A provider can then go find that data for you, maybe through scraping a difficult-to-parse website, and then post it online. You then have the option to purchase the tabular data.
There are three big humps to get over though for Data Marketplace to work.
Continue Reading -
Data.gov.uk versus Data.gov – Which wins?
Back in May last year, the US government launched Data.gov as a statement of transparency, and the Internet rejoiced. Just a couple of weeks ago, Data.gov.uk launched. How do the two compare? Here's my take. -
Data.gov.uk Gearing Up For Launch, er, Does Launch
Update: I had scheduled this post for next week, but apparently, Data.gov.uk launched today. The site isn't loading for me right now though. I guess they weren't prepared for traffic.
Data.gov, a catalog of US data, launched last year. Now it's the UK's turn. Well, not yet. But soon. Data.gov.uk is still under lock and key, but it has granted access to some developers. Ito Labs, the group behind mapping a year of OpenStreetMap edits posted screenshots of their maps that show vehicle counts (above).
Here are some comparison maps between 2001 and 2008, by vehicle type.
Continue Reading -
Unemployment, 2004 to Present – The Country is Bleeding
The Bureau of Labor Statistics released the most recent unemployment numbers last week. Things aren't looking good for the unemployed, I'm afraid.
I showed my younger sister the maps. Her response: "It looks like the country is bleeding."
Continue Reading -
Target Store Openings Since the First in 1962 – Data Now Available
FlowingData readers who have been around for a while will remember I made a map early this year that showed the growth of Target stores across America. It starts with the first one in 1962 and then goes from there. It was a follow-up to the Walmart map, which I shared the code and data for.
Continue Reading -
NYC BigApps Competition – $20k In Prize Money
It's exciting times for data heads. The launch of Data.gov back in May got things jump started; San Francisco recently announced DataSF; and now New York is getting in on the party with the announcement of their own Data Mine (live at 1pm EST today) and the NYC Big Apps competition.
Continue Reading -
30 Resources to Find the Data You Need
Let's say you have this idea for a visualization or application, or you're just curious about some trend. But you have a problem. You can't find the data, and without the data, you can't even start. This is a guide and a list of sources for where you can find that data you're looking for. There's a lot out there.
Universities
Being a graduate student, I always look to the library for books and resources. Many libraries are amping up their technology and have some expansive data archives. Many statistics departments also tend to keep a list of data somewhere. Continue Reading
-
Share and Sell Data with Infochimps (100 Invites)
There's a lot of data on the Web, but it's all very scattered. At the same time, there's a lot of data sitting on people's hard drives that we don't have access to. There are various reasons why people don't share, but mainly, they just don't see the point.Infochimps tries to solve both of these problems with an open data marketplace.
Continue Reading -
IT Dashboard and Data from USAspending.gov
Taking another step towards data transparency, the US government provides the IT dashboard via USAspending.gov:
The IT Dashboard provides the public with an online window into the details of Federal information technology investments and provides users with the ability to track the progress of investments over time. The IT Dashboard displays data received from agency reports to the Office of Management and Budget (OMB), including general information on over 7,000 Federal IT investments and detailed data for nearly 800 of those investments that agencies classify as "major." The performance data used to track the 800 major IT investments is based on milestone information displayed in agency reports to OMB called "Exhibit 300s." Agency CIOs are responsible for evaluating and updating select data on a monthly basis, which is accomplished through interfaces provided on the website.
Along with a page to filter and download spending data, there's a variety of views into the IT spending data that all provide a pretty good level of interaction.
Continue Reading -
Taking a Closer Look at Airplane-Bird Collisions
While we're on the subject of flight, ever since that plane landed in the Hudson River a few months ago, the thought of bird-airplane collisions haven't strayed too far from the media (or my mind each time I fly). In light of all the hoopla, the Federal Aviation Administration (FAA) finally gave in and opened up their bird strike database to the public.
Below is an interactive exploring this data breaking things down by bird type, location, phase of flight, and time of day. Click through to this post to view.
Continue Reading -
Explore World Data with Factbook eXplorer from OECD
The Organization for Economic Co-operation and Development (OECD) makes a lot of world indicators available (e.g. world population and birth rate). Much of it goes unnoticed, because most people just see a bunch of numbers. However, the Factbook eXplorer from the OECD, in collaboration with the National Center for Visual Analytics, is a visualization tool that helps you see and explore the data.
Those who have seen Hans Rosling's Gapminder presentations - and I imagine most of us have - will recognize the style with a play button and a motion graph in sync with parallel coordinates and a map. Choose an indicator, or several of them, press play, and watch the visualization move through time.
Also, if you've got your own data, you can load that too, which is certainly a nice touch.
-
Data.gov is Live – Get Your Data While it’s Hot
Big news. Data.gov is now live. Government data is at your fingertips.
The purpose of Data.gov is to increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government. Although the initial launch of Data.gov provides a limited portion of the rich variety of Federal datasets presently available, we invite you to actively participate in shaping the future of Data.gov by suggesting additional datasets and site enhancements to provide seamless access and use of your Federal data. Visit today with us, but come back often. With your help, Data.gov will continue to grow and change in the weeks, months, and years ahead.
I was actually expecting an API of some sort, but it's a searchable catalog that makes it easier to find the datasets scattered across all the U.S. agency sites. I still need to explore more to figure out what exactly is there, but this is big news for data fans. What do you think of the new site? Discuss in the comments below.
[via infosthetics]
Where are the biggest box office movies (not) streaming?
Find out what percent you are in
High-resolution maps of science
Why I want to quit cable
Cinemetrics creates a visual fingerprint for movies
Vehicles involved in fatal crashes
World subway paths at scale
Visualize This