• Data.gov is Live – Get Your Data While it’s Hot

    May 21, 2009  |  Data Sources

    Big news. Data.gov is now live. Government data is at your fingertips.

    The purpose of Data.gov is to increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government. Although the initial launch of Data.gov provides a limited portion of the rich variety of Federal datasets presently available, we invite you to actively participate in shaping the future of Data.gov by suggesting additional datasets and site enhancements to provide seamless access and use of your Federal data. Visit today with us, but come back often. With your help, Data.gov will continue to grow and change in the weeks, months, and years ahead.

    I was actually expecting an API of some sort, but it's a searchable catalog that makes it easier to find the datasets scattered across all the U.S. agency sites. I still need to explore more to figure out what exactly is there, but this is big news for data fans. What do you think of the new site? Discuss in the comments below.

    [via infosthetics]

  • 37 Data-ish Blogs You Should Know About

    May 6, 2009  |  Statistics, Visualization

    You might not know it, but there are actually a ton of data and visualization blogs out there. I'm a bit of a feed addict subscribing to just about anything with a chart or a mention of statistics on it (and naturally have to do some feed-cleaning every now and then). In a follow up to my short list last year, here are the data-ish blogs, some old and some new, that continue to post interesting stuff. Continue Reading

  • Google Adds Search to Public Data

    April 28, 2009  |  Data Sources, Online Applications

    Google announced today that they have made a small subset of public datasets searchable. Search for unemployment rate and you'll see a thumbnail at the top of the results. Click on it, and you get a the very Google-y chart like the one above, so instead of searching for unemployment rates for multiple years, you can get it all at once.
    Continue Reading

  • Tracking Swine Flu Worldwide – Where and How, Plus Data

    April 28, 2009  |  Data Sources, Infographics

    Just about everywhere you go there's something in the news about swine flu, and so naturally, when I first heard about it, I waited for The New York Times to put up a graphic. That was the first one. Here's the second (above).
    Continue Reading

  • Narrow-minded Data Visualization

    April 22, 2009  |  Statistics

    I was going to let this one slide, but people kept commenting, essentially trashing FlowingData, and that's just not cool. As you might recall, I put in my picks for the best data visualization projects of 2008 a while back. They were the fine work of statisticians, designers, and computer scientists, all of them beautiful, and all of them built to tell an interesting story with the dataset at hand. None of them were traditional graphs or charts.
    Continue Reading

  • Millions of Money-in-Politics Data Records Now Available

    April 15, 2009  |  Data Sources

    The Center for Responsive Politics (CRP), a research group well-known for its tracking of monetary influence on United States politics, announced some great news. Their expansive dataset is now available to the public via OpenSecrets.

    Politicians, prepare yourselves. Lobbyists, look out. Today the nonpartisan Center for Responsive Politics is putting 200 million data records from the watchdog group's archive directly into the hands of citizens, activists, journalists and anyone else interested in following the money in U.S. politics.

    Yeah, 200 million data records. Correction. 200 million cleaned, formatted, and documented data records. Awesome. They've got data on campaign finances, lobbying, personal finances, and 527 organizations, which can be downloaded as CSV files or via the RESTful API. Let the mashups begin.

    [via Ben Fry | Thanks, Gegtik]

  • Taking a Look at Facebook Statistics from All Facebook

    March 24, 2009  |  Data Sources

    facebook

    Facebook started as a spinoff of Hot or Not in 2003. Now Facebook is the world's biggest online social network. It's certainly come a long way with millions of users around the world, the opening of the Facebook Platform, and quite possibly a personal data gold mine. All Facebook, the unofficial Facebook resource, provides news, and more importantly, data on growth, demographics, pages, and applications. A lot of it is locked behind a not so pretty widget, but interesting nevertheless. The above graphic is a look at some of that data.

    [Thanks, @mobiletek]

  • Data Visualization is Only Part of the Answer to Big Data

    March 20, 2009  |  Design, Exploratory Data Analysis

    How can we now cope with a large amount of data and still do a thorough job of analysis so that we don't miss the Nobel Prize?

    — Bill Cleveland, Getting Past the Pie Chart, SEED Magazine, 2.18.2009

    For the past year, I've been slowly drifting off my statistical roots - more interested in design and aesthetics than in whether or not a particular graphic works or the more numeric tools at my disposal. I've always had more fun experimenting on a bunch different things rather than really knuckling down on a particular problem. This works for a lot of things - like online musings - but you miss a lot of the important technical points in the process, so I've been (slowly) working my way back to the analytical side of the river.
    Continue Reading

  • What’s Wrong With this Financial Bubble Chart?

    February 26, 2009  |  Mistaken Data

    Average US Consumer Spending Bubble Infographic

    If there's anything good that has come out of America's financial crisis, it's the interesting and high-quality infographics. This isn't one of them. Below is an ill-conceived bubble chart from BillShrink that "shows" average U.S. consumer spending. Notice anything wrong with it?

    Bar versus bubble debate aside, there is a ton of room for improvement as well as huge need for some fact-checking and common sense. For a blog on a site for personal finance, the graphic is, well, not something to be proud of. FlowingData readers know that I like to stay away from heavy-handed critique on what works and what doesn't (I leave that to you guys), but this BillShrink graphic is just so clearly confusing that it's worth pointing out what doesn't work so we can learn from others' mistakes. Can you find the flaws?

    [Thanks, Jess]

  • Google’s Chief Economist Hal Varian on Statistics and Data

    February 25, 2009  |  Quotes, Statistics

    I keep saying the sexy job in the next ten years will be statisticians. People think I'm joking, but who would've guessed that computer engineers would've been the sexy job of the 1990s?

    Hal Varian, The McKinsey Quarterly, January 2009

    Varian then goes on to say:

    The ability to take data - to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it's going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it.

    I think statisticians are part of it, but it's just a part. You also want to be able to visualize the data, communicate the data, and utilize it effectively. But I do think those skills - of being able to access, understand, and communicate the insights you get from data analysis - are going to be extremely important. Managers need to be able to access and understand the data themselves.

    Wait a minute. Is this a pitch for FlowingData? I think so :). Check out the full article for more (or listen to the podcast). It's an interesting read.

  • Obama Launches Recovery.gov – Your $787 billion at Work

    February 17, 2009  |  Statistics

    The Barack Obama administration is clearly making an effort to get information out to the public. We saw the plan to distribute billions of dollars to help our economy. Obama has now launched Recovery.gov as a way for you to keep track and understand where $787 billion from the American Recovery and Reinvestment Act is used.
    Continue Reading

  • Fail: Area Circles on Wall Street

    February 16, 2009  |  Mistaken Data

    I know next to nothing about the economy, stocks, and investments, but I do know a little bit about charts and graphs. The above area circles were prepared by someone at JP Morgan. I don't know, you might have heard of 'em. The circles are based on data from Bloomberg and meant to show the change in market value from 2007 to 2009. The problem here is that the creator sized circles by diameter instead of area, so the difference looks ginormous. I mean, the value change is significant but not that big.
    Continue Reading

  • Open Thread: Is Google Latitude Dangerous?

    February 12, 2009  |  Data Sharing, Discussion, Online Applications

    Google recently released Google Latitude, which is an online application that lets you share your location with online friends:

    Of course when any application shares where you are at any given time, people start to feel like Big Brother is looming in the background ready to sneak up on us from behind a giant bush. Some call it a real danger, but is it really? I put this question out to all of you:

    Is Google Latitude a danger to anyone who uses it?

    My take on things is that people are already doing it anyways, so why not make it easier for those who are interested? Sure, if some stalker got a hold of your location, that could be bad, but that's true for a lot of data... credit card statements, cell phone logs, Twitter... As long as the proper security are put in place, I don't see what all the fuss is about.

  • Sensors in Footballs – Was the Pass Good?

    December 30, 2008  |  Statistics

    Graduate student researchers are pretty much putting sensors in everything these days. There's always more data to collect and more information to gather. Computer engineering students from Carnegie Mellon University experiment with sensors in footballs and gloves to measure grip, trajectory, speed and position.

    "You'd never want to replace the human referees because they make these calls based on years of experience, and no technology can replace that," she said. "But in addition to the instant replay, if you had a supplementary system that said this is exactly where the ball landed and where the player stopped with it, you could make these kinds of calls accurately."

    So far, she and her squad of undergraduate and graduate students have focused on two things: gloves with touch sensors that can transmit that information wirelessly to a computer, and a football equipped with a global positioning receiver and accelerometer that can track the location, speed and trajectory of the ball.

    Eventually, the same kind of sensors used in the gloves could be adapted to shoes, to measure stride and running patterns, or even shoulder pads, to calculate blocking positions and force.

    Yes, it's the end of the post-game show as we know it.

  • All You Can Eat at the Twitter Data Buffet

    December 24, 2008  |  Data Sources

    Philip from infochimps posts the results of some heavy Twitter scraping. Data for 2.7 million users, 10 million tweets, and 58 million edges (i.e. connections between users) to satisfy your data hunger are available for download. I know a lot of you social network researchers will especially appreciate the big dataset, and best of all, Twitter gave Philip permssion to release. Yes, you could use the Twitter API, but isn't it better when someone does it for you?

    Download the data here. The password is the Ramanujan taxicab number followed by the word
    'kennedy' - all one word. Google is your friend, if that doesn't make sense.

    [Thanks, Tim]

  • Do You Hate Statistics as Much as Everyone Else?

    December 15, 2008  |  Statistics

    Photo by Darwin Bell

    It happened again. I told someone I study statistics. He told me that he hated statistics in college. It doesn't annoy me like it used to - I've come to expect it - but why do so many people have this beef with stat? Is it really that boring? Confusing? What is it about statistics that turns people off? So I reach out to all of you:

    What is it that makes statistics so uninteresting?

    I'm going to assume that the icky factor is less for FlowingData readers (obviously), but still, I implore you - tell me why statistics sucks. I must know.

  • Scientists Can Now Map Your Dreams to an Image

    December 12, 2008  |  Statistics

    I thought this was a joke when I first read it, but scientists from Japan’s ATR Computational Neuroscience Laboratories have developed software that can map brain activity to an image. Subjects were shown letters from the word neuron and images were reconstructed and displayed on a computer screen.

    A spokesman at ATR Computational Neuroscience Laboratories said: "It was the first time in the world that it was possible to visualise what people see directly from the brain activity.

    "By applying this technology, it may become possible to record and replay subjective images that people perceive like dreams." The scientists, lead by chief researcher Yukiyaso Kamitani, focused on the image recognition procedures in the retina of the human eye.

    It is while looking at an object that the eye's retina is able to recognise an image, which is subsequently converted into electrical signals sent into the brain's visual cortex.

    The research investigated how electrical signals are captured and reconstructed into images, according to the study, which will be published in the US journal Neuron.

    I'm not sure how much brain activity from the retina has to do with activity during dreams, but it's interesting nevertheless (although I am sure - like all interesting science - it is slightly hyped by the media).

    [via Telegraph & Pink Tentacle & Chunici]

  • Amazon Gets In On the Public Data Arena

    December 5, 2008  |  Data Sources

    It was really only a matter of time, but Amazon now hosts public data sets. Not small data sets though - more like the ones in between 1 gigabyte and 1 terabyte:

    Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications. AWS is hosting the public data sets at no charge for the community, and like all AWS services, users pay only for the compute and storage they use for their own applications. An initial list of data sets is already available, and more will be added soon.

    Previously, large data sets such as the mapping of the Human Genome and the US Census data required hours or days to locate, download, customize, and analyze. Now, anyone can access these data sets from their Amazon Elastic Compute Cloud (Amazon EC2) instances and start computing on the data within minutes. Users can also leverage the entire AWS ecosystem and easily collaborate with other AWS users. For example, users can produce or use prebuilt server images with tools and applications to analyze the data sets. By hosting this important and useful data with cost-efficient services such as Amazon EC2, AWS hopes to provide researchers across a variety of disciplines and industries with tools to enable more innovation, more quickly.

    There's the human genome data set, US Census data from the past 3 decades, labor statistics, and some others. Still waiting on Google to follow through with their data hosting plans.

    [via TechCrunch | Thanks, David]

  • Guess What State Searches for ‘Poo’ the Most – StateStats

    December 5, 2008  |  Mapping, Statistics

    StateStats is like Google Insights but on a state level. Type in a search term and get Google search levels with correlations to certain "metrics" like obesity or support for Obama. Any Web application that uses correlation tends to make me feel a bit iffy, but it's just for fun, so I guess it's okay.

    Being the immature man-child that I am, the first thing I type in the search field is poo. I thought it was hilarious interesting that Louisiana's relative search rate was so much higher than all the other states. Apparently, obesity correlates moderately.

    I'm sure all of you will search for more sophisticated terms.

    [Thanks, @Chimp711]

  • Neighborhood Boundaries with Flickr Shapefiles

    November 28, 2008  |  Data Sources, Mapping

    Neighborhood Boundaries by Tom Taylor uses Flickr Shapefiles and Yahoo! Geoplanet "to show you where the world thinks its neighbors are." Yahoo! provides access to the Where on Earth (WOE) database, which attempts to describe locations as a hierarchy. For example - a town belongs to a city, a city to a county, a county to a state. The Flickr API stores shape files identified by the WOE ID. Here's the punchline. The shapefiles are built using only the latitude and longitude from geotagged photos on Flickr. There's no GIS involved here.

    Why this matters, I can't really say. I think it's mostly to show how much data is stored in geotagged Flickr photos. I'm no GIS expert though. Anyone care to comment on the significance?

    [Thanks, @couch]

Copyright © 2007-2014 FlowingData. All rights reserved. Hosted by Linode.