I don't want my credit card numbers floating around, because then I'd be screwed. That kind of data needs to be locked up tight behind a billion firewalls, a lock safe, five armed guards, and another locked safe and then one more guard plus another safe. However, there are lots of other kinds of data that should be online and publicly available or at least accessible via a phone call.
As a student, I've always received data from the prof or from some magical place called data land. It's not that easy in the real world, and as an intern, I'm beginning to see a trend -- if you're not willing to give me your data or some tiny subset of your data, then you're probably hiding something.
I recently did a whole lot of back and forth for two weeks trying to get some data from a group that will go unnamed. Without getting into too many details, I wanted data that showed the group's progress -- what they've accomplished over X number of months. You should probably also know that this group has taken a lot of heat lately for their slow pace and shotty labor.
Here's how it went.
Day 1-3
"Nathan, can you contact so and so and ask them for this and this data or see what they have?" Sure, no problem. I emailed the reporter's contact, who happened to be a contractor for the big group I was trying to get data from. We exchanged some emails, and it turned out that the contractor was working with the data that was exactly what we wanted. Um, gimme.
Day 4-8
The contractor had to get approval from the "chief of staff." Unfortunately the chief of staff was out for the week, so he had to go through some other people. Contractor gets distracted, and I get forwarded to public affairs. "Oh great, this will be fun," I thought. Of course, this is when it got especially painful. After some misunderstandings and 11 emails later, it was back to the contractor. Same old story. Need approval, yada yada. Keep in mind that during all of this, my co-worker is putting together a graphic.
Day 9-13
It was just all waiting now. They had the data and were waiting to get the sign off. I called one or two times a day and sent an email to both the contractor and public affairs guy once a day. There was lots of fluffy, meaningless talk during this phase.
Day 14
At the end of Day 14, I got the phone call. "Nathan, we have some data that we're ready to send your way. Your patience has been rewarded." I can't believe he actually said that. My patience had been rewarded with nothing. Too bad the graphic was already entering its final editing stages without their data.
The data wasn't really worth the effort.
Hence, the Difficulty
So here we stand with this great idea of sharing data. So wonderful and marvelous, we can't even fathom how we can benefit. However, data can be very revealing, and there are many groups, people, and organizations who aren't ready to show what they have. Either they're afraid of sharing data for security reasons (which is understandable), or they're afraid because they're worried about what they're handing over. In both cases, it's a huge blockade that I don't see us getting through any time soon.
I began my path of higher education at Berkeley as an Electrical Engineering and Computer Science student. As a stat graduate student, it's hard to remember sitting in all of those (boring) engineering classes.
