Texting on the toilet

January 30, 2012 to Data Sources • Share on Twitter • Comments (11)

I thought this riveting post on the New York Times Bits blog about the rise of the toilet texter deserved a graphic. Since their graphics department is no doubt busy with elections, I took the liberty. I am — the 91 percent.

I got the numbers straight from the Bits post, but you can download the full report from 11mark for all the demographics. You have to register though, and I didn't want to be the guy who creates an online account to just read a report on what people do while they make dooty. I have standards.

What Facebook knows about you

December 14, 2011 to Data Sources • Share on Twitter • Comments (14)

Facebook logs and saves a lot of data about you and what you do on their site. This shouldn't be surprising given the more time people spend on Facebook, the greater the cash flow, but just how much data do they store? Austrian law student Max Schrems, because European law states that citizens can do this, requested all the data Facebook had about him. He got back a CD with 1,222 PDF files.
Continue Reading

Geo API from Infochimps brings you closer to mapping fun

August 31, 2011 to Data Sources • Share on Twitter • Add Comment

Mostly because of the popularity of smartphones, location data is all the rage nowadays. You're almost always connected no matter where you are. Rich location data can help provide you a new sense of place, and at the same time, this sort of data can paint an interesting picture of what's going on in your country or around the world. Hence, Infochimps, the one-stop shop for data folk and developers, just announced their new Geo API.
Continue Reading

Reporters make it easier to access Census data

August 29, 2011 to Data Sources • Share on Twitter • Add Comment

Census data can provide valuable information, but the datasets are not always the easiest to access. So you often end up spending a lot of time getting your data in order before you actually get to do anything with it. Investigative Reporters and Editors has released the next phase in their Census project to make Census 2010 more accessible via a simple interface. Easily download data in bulk as CSV or shapefiles or build it into your applications with the API.

[census.ire.org via @bryanboyer]

Get a coffee, give a coffee API

August 7, 2011 to Data Sources • Share on Twitter • Add Comment

Jonathan Stark, a mobile application consultant, is running an interesting social experiment with his Starbucks card:

Jonathan's Card is an experiment in social sharing of physical goods using digital currency on mobile phones. I stumbled on the idea while doing research for a blog post about Broadcasting Mobile Currency.

Based on the similarity to the "take a penny, leave a penny" trays at convenience stores in the US, I've adopted a similar "get a coffee, give a coffee" terminology for Jonathan's Card.

Simply save the picture of Jonathan's Starbucks card onto your smartphone and use it to buy your coffee. If you like, add money to the card so that someone else can buy a coffee.

The best part is that Stark provides a simple API that returns the balance on the card every minute. When do people buy coffee? How do people give and take? Are people more likely to give when there's a large balance or when there's nothing left? Lots of fun things to look at.

[Jonathan's Card via @kn0thing]

Pew Research raw survey data now available

May 25, 2011 to Data Sources • Share on Twitter • Add Comment

The Pew Research churns out a lot of interesting results from a number of surveys about online and American culture, but they usually only shared aggregated results, pre-made charts and graphs. This is well and good for the information-consuming public; however, these results can spawn curiosities that are fun to dig into. Luckily, the Pew Research Center launched a Data Sets section that provides raw survey responses and the questions in a variety of easy-to-use data formats.

Our raw data, previously posted only as SPSS files, is now available in comma-delimited (.csv) format for all reports going back to 2003. We hope that making our data available in this open-source format will make analysis easier for researchers who don’t own a copy of SPSS to analyze our data.

This should be fun. Recent datasets include the social side of the Internet, health tracking habits, and reputation management.

[Pew Research via @kzickhur]

Map your location – that your iPhone secretly records

April 20, 2011 to Data Sources, Mapping • Share on Twitter • Comments (7)

Researchers Alasdair Allan and Pete Warden have found that the iPhone records cell tower access, and hence your location, in an easy-to-read file that is transferred as you switch devices. And they do this whether you like it or not.

The more fundamental problem is that Apple are collecting this information at all. Cell-phone providers collect similar data almost inevitably as part of their operations, but it’s kept behind their firewall. It normally requires a court order to gain access to it, whereas this is available to anyone who can get their hands on your phone or computer.

Allan and Warden provide an open-source application, iPhone Tracker, that maps that data. The good news is that the data doesn't seem go to be anywhere other than your own backups and devices. Privacy concerns aside, this kind of makes me wish I had an iPhone; although I suspect my map would be painfully boring.

[iPhone Tracker via Marco]

Data.gov and other transparency sites to be shut down due to budget cuts

March 31, 2011 to Data Sources • Share on Twitter • Comments (11)

Last week, there were rumblings over the end of the Statistical Abstract, and I suggested that it was just a sign of changing technologies. I thought that Data.gov and similar sites were the natural progression. Here's the problem with that argument. Congress is planning on shutting down Data.gov and other transparency sites in the next few months.
Continue Reading

Tell-all telephone reveals politician’s life

March 30, 2011 to Data Sources, Mapping • Share on Twitter • Comments (9)

Not many people understand the importance of data privacy. They don't get out how little bits of information sent from your phone every now and then can show a lot about your day-to-day life.

As the German government tries to come to a consensus about its data retention rules, Green party politician Malte Spitz retrieved six months of phone data from Deutsche Telekom (by suing them), to show what you can get from a little bit of private mobile data. He handed the data to Zeit Online, and they in turn mapped and animated practically every one of Spitz' moves over half a year and combined it with publicly available information from sources such as his appointment website, blog, and Twitter feed for more context.
Continue Reading

Lots of health data released via Health Indicators Warehouse

March 1, 2011 to Data Sources • Share on Twitter • Comments (1)

The government has been making a big push for more open health-related data, and a couple of weeks ago, they released a whole bunch of it with the launch of HealthData.gov. It's the same interface as Data.gov, but for health. Additionally, the Health Indicators Warehouse launched with different data and a slightly more useable interface.

A quick scan of the data available, however, does seem to indicate that a lot of it is spotty or outdated (like on data.gov), which doesn't make it especially useful. For example, some data sets are only one data point, while others are only a single year. At least it's a start.

[Health Indicators Warehouse via @periscopic]

Million song dataset available for download

February 24, 2011 to Data Sources • Share on Twitter • Comments (2)

Need music data? Get all the data you want and more from the freely available million song dataset, offered by LabROSA at Columbia University and Echo Nest. There's lots of metadata on song features and your standard stuff like year and artist. There are also several code wrappers and samples to help researchers make use of the data right away.

[Million Song Dataset via @MacDivaONA]

Sunlight Labs opens up Real Time Congress API

February 17, 2011 to Data Sources • Share on Twitter • Comments (4)

Sunlight Labs continues its work for a more open government with its recent release of the Real Time Congress API.

Today we're making available the Real Time Congress API, a service we've been working on for several months, and will be continuing to expand.

The Real Time Congress API (RTC) is a RESTful API over the artifacts of Congress, kept up to date in as close to real time as possible. It consists of several live feeds of data, available in JSON or XML. These feeds are filterable and sortable and sliceable in all sorts of different ways, and you can read the docs to see how.

There are seven data types the API will report:

Bills
Votes
Amendments
Videos
Floor Updates
Committee Hearings
Documents

Now someone has to do something with all of this data coming in. Can you think of a useful application for what is essentially an automated government Twitter feed?

[Real Time Congress API]

Find more of the data you need with DataMarket

January 31, 2011 to Data Sources, Online Applications • Share on Twitter • Comments (5)

Add another online destination to find the data that you need. DataMarket launched back in May with Icelandic data, but just a few days ago relaunched with data of the international variety. They tout 100 million time series datasets and 600 million facts. I'm not totally sure what that means (100 million lines, sets of lines?), but I take it that means a lot.

Just over 2 years and countless cups of coffee after we started coding, DataMarket.com launches with international data. You can now find, visualize and download data from many of the world’s most important data providers on our site.

At first glance DataMarket feels a lot like now defunct Swivel. Search for the data you want and you get back a list of datasets. The focus on only time series though is actually a plus in that they can provide more specific tools to visualize and explore. The current toolset isn't going to blow you away, but it's not bad.
Continue Reading

A guide for scraping data

January 17, 2011 to Data Sources • Share on Twitter • Add Comment

Data is rarely in the format you want it. Dan Nguyen, for ProPublica, provides a thorough guide on how to scrape data from Flash, HTML, and PDF. [via @JanWillemTulp]

Search how phrases have been used via Google Ngram Viewer

December 20, 2010 to Data Sources, Online Applications • Share on Twitter • Comments (20)

Language changes. Culture changes. And we can see some of these changes via what authors write about in books over the years. Google's Book Ngram Viewer lets you search through this data, and shows a graph similar similar to the output of Google Trends. The above is the trends for nursery school, kindergarten, and child care:

This shows trends in three ngrams from 1950 to 2000: "nursery school" (a 2-gram or bigram), "kindergarten" (a 1-gram or unigram), and "child care" (another bigram). What the y-axis shows is this: of all the bigrams contained in our sample of books written in English and published in the United States, what percentage of them are "nursery school" or "child care"? Of all the unigrams, what percentage of them are "kindergarten"? Here, you can see that use of the phrase "child care" started to rise in the late 1960s, overtaking "nursery school" around 1970 and then "kindergarten" around 1973. It peaked shortly after 1990 and has been falling steadily since.

Find anything interesting?
Continue Reading

Jon Stewart explains Wikileaks’ Cablegate

December 2, 2010 to Data Sources, News • Share on Twitter • Comments (14)

You've probably already heard and read about Wikileaks' Cablegate. If not, Andy Baio has a fine roundup with significant coverage and events to get you caught up quick. Alternatively, you can watch Jon Stewart and The Daily Show explain in the clip below (slightly NSFW, because it mentions a body part).
Continue Reading

How do people use Firefox?

November 30, 2010 to Data Sources, News • Share on Twitter • Comments (3)

Mozilla Labs just released a bunch of anonymized browsing data for their open data visualization competition:

This competition is based on Mozilla's own open data program, Test Pilot. Test Pilot is a user research platform that collects structured user data through Firefox. All data is gathered through pre-defined Test Pilot studies, which aim to explore how people use their web browser and the Internet.

There are two datasets in various formats. The first is browsing behavior from 27,000 users, including on/off private browsing that we saw a few months ago. The second dataset is from 160,000 users and is on how they actually use the Firefox interface.

Additionally, both sets have survey answers to questions like "How long have you used Firefox?" which could make for some fun and interesting breakdowns.

The deadline is December 17.

[Mozilla Labs]

Opportunities in Government 2.0

October 27, 2010 to Data Sources • Share on Twitter • Add Comment

Vivek Wadhwa talks government data and the (financial) opportunities ripe for the picking:

What is happening with the opening up of government data is nothing less than a silent revolution. There are literally thousands of new opportunities to improve government and to improve society—and to make a fortune while doing it. Unlike the Web 2.0 space, which is overcrowded, Gov 2.0 is uncharted territory: a new frontier to explore, grow things on, and settle on. It’s fresh soil for unlikely seedling ideas that, if they take root, could lead to very successful ventures. So I encourage entrepreneurs to stake their claims as soon as they can.

Wait a minute. Hold up. You can do more with government data than awkward dashboards? Bring it.

[TechCrunch via @ucdatalab]

How people use private browsing

August 25, 2010 to Data Sources, Statistics • Share on Twitter • Comments (16)

Private browsing. All the modern browsers have it. Turn it on, and the browser won't keep your history during the session. Sometimes it's used to pay bank bills on a public computer. Sometimes it's used for other stuff. In an opt-in study looking at a week in the life of a browser, Mozilla looked at how people use private browsing.

Again, it's worth noting that people opted in to this study (about 4,000 of them), and Mozilla only recorded when users started and stopped private browsing. Nothing in between.

That said, they came up with two basic findings. The first is when people typically use private browsing (above).

They saw usage spikes during the lunch hours as well as just before the work day ended. The other spike is after the dinner hours and then finally, in the late hours of the night.
Continue Reading

Data Sources