Don’t want to share our data / OK, what’re you hiding?

Posted to Data Sharing  |  Nathan Yau

I don’t want my credit card numbers floating around, because then I’d be screwed. That kind of data needs to be locked up tight behind a billion firewalls, a lock safe, five armed guards, and another locked safe and then one more guard plus another safe. However, there are lots of other kinds of data that should be online and publicly available or at least accessible via a phone call.

As a student, I’ve always received data from the prof or from some magical place called data land. It’s not that easy in the real world, and as an intern, I’m beginning to see a trend — if you’re not willing to give me your data or some tiny subset of your data, then you’re probably hiding something.

I recently did a whole lot of back and forth for two weeks trying to get some data from a group that will go unnamed. Without getting into too many details, I wanted data that showed the group’s progress — what they’ve accomplished over X number of months. You should probably also know that this group has taken a lot of heat lately for their slow pace and shotty labor.

Here’s how it went.

Day 1-3

“Nathan, can you contact so and so and ask them for this and this data or see what they have?” Sure, no problem. I emailed the reporter’s contact, who happened to be a contractor for the big group I was trying to get data from. We exchanged some emails, and it turned out that the contractor was working with the data that was exactly what we wanted. Um, gimme.

Day 4-8

The contractor had to get approval from the “chief of staff.” Unfortunately the chief of staff was out for the week, so he had to go through some other people. Contractor gets distracted, and I get forwarded to public affairs. “Oh great, this will be fun,” I thought. Of course, this is when it got especially painful. After some misunderstandings and 11 emails later, it was back to the contractor. Same old story. Need approval, yada yada. Keep in mind that during all of this, my co-worker is putting together a graphic.

Day 9-13

It was just all waiting now. They had the data and were waiting to get the sign off. I called one or two times a day and sent an email to both the contractor and public affairs guy once a day. There was lots of fluffy, meaningless talk during this phase.

Day 14

At the end of Day 14, I got the phone call. “Nathan, we have some data that we’re ready to send your way. Your patience has been rewarded.” I can’t believe he actually said that. My patience had been rewarded with nothing. Too bad the graphic was already entering its final editing stages without their data.

The data wasn’t really worth the effort.

Hence, the Difficulty

So here we stand with this great idea of sharing data. So wonderful and marvelous, we can’t even fathom how we can benefit. However, data can be very revealing, and there are many groups, people, and organizations who aren’t ready to show what they have. Either they’re afraid of sharing data for security reasons (which is understandable), or they’re afraid because they’re worried about what they’re handing over. In both cases, it’s a huge blockade that I don’t see us getting through any time soon.


Life expectancy changes

The data goes back to 1960 and up to the most current estimates for 2009. Each line represents a country.

10 Best Data Visualization Projects of 2015

These are my picks for the best of 2015. As usual, they could easily appear in a different order on a different day, and there are projects not on the list that were also excellent.

Famous Movie Quotes as Charts

In celebration of their 100-year anniversary, the American Film Institute selected the 100 most memorable quotes from American cinema, and …

Watching the growth of Walmart – now with 100% more Sam’s Club

The ever so popular Walmart growth map gets an update, and yes, it still looks like a wildfire. Sam’s Club follows soon after, although not nearly as vigorously.