Back in May last year, the US government launched Data.gov as a statement of transparency, and the Internet rejoiced. After the launch, excitement kind of fizzled with the actual Data.gov site, but big cities like San Francisco, New York, and Toronto got in on the open data party.
Then just a couple of weeks ago, Data.gov.uk launched, which brought me back to the US counterpart. How do the two compare? Here's my take. Continue Reading
Update: I had scheduled this post for next week, but apparently, Data.gov.uk launched today. The site isn't loading for me right now though. I guess they weren't prepared for traffic.
Data.gov, a catalog of US data, launched last year. Now it's the UK's turn. Well, not yet. But soon. Data.gov.uk is still under lock and key, but it has granted access to some developers. Ito Labs, the group behind mapping a year of OpenStreetMap edits posted screenshots of their maps that show vehicle counts (above).
Here are some comparison maps between 2001 and 2008, by vehicle type.
Once Data.gov.uk is up, it'll be interesting to see how it compares to its US counterpart. Even more interesting will be the projects that come out of it.
Despite all the broohaha over Data.gov, not many useful projects (or datasets) come to mind. Can you think of any? There's still a long way to go from both sides of government and developer.
[Thanks, Oliver]
How has consumer spending changed over the past 25 years? Do we spend more on some things and spend less on other than we did in the early 80s? In this interactive, based on data from the Bureau of Labor Statistics, you can explore just that.
The Bureau of Labor Statistics released the most recent unemployment numbers last week. Things aren't looking good for the unemployed, I'm afraid.
I showed my younger sister the maps. Her response: "It looks like the country is bleeding."
While the recession is "over" the unemployment rate rose to 9.8% in September from 9.7% in August. That's 214,000 more people who are jobless in the United States. The last time unemployment was this high was back in June 1983 when it was 10.1%.
Check out the more detailed view here:
From 2004 to 2007, unemployment was actually decreasing, but things went sour in 2008, and we've been trying to bounce back ever since.
Update: See the step-by-step tutorial on how you can make a map like this with your data.
FlowingData readers who have been around for a while will remember I made a map early this year that showed the growth of Target stores across America. It starts with the first one in 1962 and then goes from there. It was a follow-up to the Walmart map, which I shared the code and data for.
Anyways, I often still get emails about the Target data. I finally got around to asking if I could release it, and lucky for your the answer was yes. So here you are. Go wild.
By the way, if anyone has similar data for Starbucks, let me know. There's gotta be at least one Starbucks analyst who reads this blog. Maybe?
[Thanks, Cole]
It's exciting times for data heads. The launch of Data.gov back in May got things jump started; San Francisco recently announced DataSF; and now New York is getting in on the party with the announcement of their own Data Mine (live at 1pm EST today) and the NYC Big Apps competition.
Here's the idea. NYC releases 170 datasets. Whoever can best make use of the data will win part of the $20,000 in prize money. The individual or startup with the best Web or mobile application gets an invite to a dinner with NYC mayor, MIchael R. Bloomberg.
Some of the data in the list of 170 sets include:
Not too shabby, right? And that's just a small subset.
No doubt this is going to be an interesting competition. I don't know about you, but I'm going to be keeping an eye on NYC Big Apps from December to January. If the competition is a hit, other big cities will follow.
At the very least, we're going to see some cool stuff coming out of The New York Times graphics department :).
Let's say you have this idea for a visualization or application, or you're just curious about some trend. But you have a problem. You can't find the data, and without the data, you can't even start. This is a guide and a list of sources for where you can find that data you're looking for. There's a lot out there.
Being a graduate student, I always look to the library for books and resources. Many libraries are amping up their technology and have some expansive data archives. Many statistics departments also tend to keep a list of data somewhere. Continue Reading
There's a lot of data on the Web, but it's all very scattered. At the same time, there's a lot of data sitting on people's hard drives that we don't have access to. There are various reasons why people don't share, but mainly, they just don't see the point.
Infochimps tries to solve both of these problems with an open data marketplace.
If you're looking for data, search the Infochimps catalog, and you might find what you're looking for. The system is loosely structured and meant to be a publicly curated data place with a mix of open data and for-sale data. Some data sources are simply outgoing links while others are stored in Infochimps infrastructure.
If you're on the other side, and you have data to offer, you can put your dataset up for sale. Fill out some forms, specify your price, and let Infochimps handle the rest like storage and cataloging. Infochimps takes a 20% commission on each sale for their service.
Selling data is of course nothing new. Search for databases for sale, and you'll get plenty of results, but this makes it easier for individuals and small groups to make their data available. Oh, and you can make your data open also.
The main challenge I see here is making sure the cataloged data are of good quality. It's one thing when the data are open and free, but when you're paying money, you want to make sure you're buying a product that's worth the price.
Currently, there's a star rating system, but it's unclear who decides how many stars go on a dataset. There's also no way to get a data sample, so all you get is a description pre-purchase.
Clearly, there's still a lot of work to be done with the application, but there's plenty of potential.
Infochimps is currently beta testing. If you'd like try it out, there's invites for the first 100 FlowingData readers who sign up. Use the code 'dataflowing' when you register.
UPDATE: Infochimps has kindly provided 100 more invites in case you missed first. Use this code when you sign up: flowswithdata.
Taking another step towards data transparency, the US government provides the IT dashboard via USAspending.gov:
The IT Dashboard provides the public with an online window into the details of Federal information technology investments and provides users with the ability to track the progress of investments over time. The IT Dashboard displays data received from agency reports to the Office of Management and Budget (OMB), including general information on over 7,000 Federal IT investments and detailed data for nearly 800 of those investments that agencies classify as "major." The performance data used to track the 800 major IT investments is based on milestone information displayed in agency reports to OMB called "Exhibit 300s." Agency CIOs are responsible for evaluating and updating select data on a monthly basis, which is accomplished through interfaces provided on the website.
Along with a page to filter and download spending data, there's a variety of views into the IT spending data that all provide a pretty good level of interaction.
One thing I can't really figure out is if "IT investments" means investments in the traditional sense like stocks, or if it's something else. I was a little surprised that the government is making investments at all but I guess I didn't have any good reason to think that. I don't know. Maybe someone can explain it to me.
[Thanks Justin & Preston]
While we're on the subject of flight, ever since that plane landed in the Hudson River a few months ago, the thought of bird-airplane collisions haven't strayed too far from the media (or my mind each time I fly). In light of all the hoopla, the Federal Aviation Administration (FAA) finally gave in and opened up their bird strike database to the public.
Below is an interactive exploring this data breaking things down by bird type, location, phase of flight, and time of day. Click through to this post to view.
Beware of the Canada Goose and gulls, rats of the sea. The sparrow, Mourning Dove, and European Starling seem to get in the way plenty also, but don't cause nearly as much damage.
On the flip side - poor birds. What a way to go.
Do you see anything interesting?
The Organization for Economic Co-operation and Development (OECD) makes a lot of world indicators available (e.g. world population and birth rate). Much of it goes unnoticed, because most people just see a bunch of numbers. However, the Factbook eXplorer from the OECD, in collaboration with the National Center for Visual Analytics, is a visualization tool that helps you see and explore the data.
Those who have seen Hans Rosling's Gapminder presentations - and I imagine most of us have - will recognize the style with a play button and a motion graph in sync with parallel coordinates and a map. Choose an indicator, or several of them, press play, and watch the visualization move through time.
Also, if you've got your own data, you can load that too, which is certainly a nice touch.
Big news. Data.gov is now live. Government data is at your fingertips.
The purpose of Data.gov is to increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government. Although the initial launch of Data.gov provides a limited portion of the rich variety of Federal datasets presently available, we invite you to actively participate in shaping the future of Data.gov by suggesting additional datasets and site enhancements to provide seamless access and use of your Federal data. Visit today with us, but come back often. With your help, Data.gov will continue to grow and change in the weeks, months, and years ahead.
I was actually expecting an API of some sort, but it's a searchable catalog that makes it easier to find the datasets scattered across all the U.S. agency sites. I still need to explore more to figure out what exactly is there, but this is big news for data fans. What do you think of the new site? Discuss in the comments below.
[via infosthetics]
Google announced today that they have made a small subset of public datasets searchable. Search for unemployment rate and you'll see a thumbnail at the top of the results. Click on it, and you get a the very Google-y chart like the one above, so instead of searching for unemployment rates for multiple years, you can get it all at once.
This is an obvious move for Google as it continues in its efforts to make the world and all the data in your life searchable. It is still a very limited number of datasets at this point from the U.S. Bureau of Labor Statistics and U.S. Census Bureau's Population Division, but I'm sure you can imagine the growth in the coming months. Maybe Google will make some real use out of Gapminder's Trendalyzer that they purchased a couple years back. Whatever happened to the Palimpsest Project?
Check it out for yourself, or if you're lazy, watch the video:
The most exciting part about this isn't the graphs or even the new searchability. It's this growing availability of data. I think most of the data that Google will index is stuff that's already available. You just have to know where to look. The main point here is that there's so much data out there on the Web that Google (and Wolfram?) has found that indexing is now worth their while, and with data.gov on the way, we, as data scientists are in for some exciting times.
[via ReadWriteWeb]
Just about everywhere you go there's something in the news about swine flu, and so naturally, when I first heard about it, I waited for The New York Times to put up a graphic. That was the first one. Here's the second (above).
Nicolas Rapp from AP also provides this explanation of how swine flu spreads:
Finally, if you'd like to take a look at the data yourself, the Guardian Data Store provides a full list of swine flu cases by country and by date.
[Thanks, @datavis]
The Center for Responsive Politics (CRP), a research group well-known for its tracking of monetary influence on United States politics, announced some great news. Their expansive dataset is now available to the public via OpenSecrets.
Politicians, prepare yourselves. Lobbyists, look out. Today the nonpartisan Center for Responsive Politics is putting 200 million data records from the watchdog group's archive directly into the hands of citizens, activists, journalists and anyone else interested in following the money in U.S. politics.
Yeah, 200 million data records. Correction. 200 million cleaned, formatted, and documented data records. Awesome. They've got data on campaign finances, lobbying, personal finances, and 527 organizations, which can be downloaded as CSV files or via the RESTful API. Let the mashups begin.
[via Ben Fry | Thanks, Gegtik]
