Add another site to the list of places to find the data you need. Data Marketplace connects people who want data to people who can find, scrape, and cull data.
Here’s how it works. If you want data, you put in a request and optionally, a deadline and budget. A provider can then go find that data for you, maybe through scraping a difficult-to-parse website, and then post it online. You then have the option to purchase the tabular data.
There are three big humps to get over though for Data Marketplace to work.
The first is that there has to be people who are willing to buy data. There’s so much emphasis nowadays on open government, transparency, and free APIs that the thought of actually paying money for data, to the average consumer, could feel a little weird.
The second thing is that the data has to be publicly available somewhere in some form. It seems that in most cases, you’re not paying for the data so much as you are for someone to scrape a site and get the data in a machine-readable format. If data isn’t openly available somewhere through an application, web forms, or whatever, then most likely it’s behind a wall for a reason (e.g. privacy).
Finally, there has to be analysts, programmers, etc. who are able to get the data people want. Now there’s no shortage of people who can do this. It’s more about people who are willing to do this. Looking through some of the data requests that are already up, you’ll see that a lot of them are very specific and aren’t just a matter of scraping one site, but many sites, along with some analytical savvy. Not a lot of people are going to put in that effort unless the price is right. It then goes back to that first thing about what consumers are willing to pay.
This of course is in direct competition to Infochimps, who opened up their marketplace a few months ago, and sites like AggData, less so. It’ll be interesting to see where they go with this in the next six months.
I think in the end, for these marketplaces to really thrive, there’s going to have to be some kind of community that develops around the data, sort of like that of Freebase. There’s only so much demand for Walmart, Target, and McDonald’s locations. The first two by the way, are freely available here on FlowingData.
I like the idea from a data gathering point of view. What I don’t like is “screen scarping”. Most websites are copyrighted (if they don’t mention a specific license) and as such it’s not allowed to screen scrape them, let alone sell the data afterwards. I do advocate open data and sharing but scraping a website for data and selling it is plain wrong.
i kind of have that same feeling. it’s definitely a gray area that I’m sure the DM and infochimps guys have had to deal with. i wonder what their take on it is.
So scraping a site for store locations is illegal, even if they list the store locations or allow you to search store locations by zip code?
I share Nathan’s skepticism, since datamarketplace relies a lot on a pool of talented data wrangler’s the serve each request, with no certainty of being paid if the requester decides not to purchase the data or to purchase from someone else.
Infochimps makes more sense, where people are investing their time in cleaning and organizing data and betting that others will want it. There is 1 dataset on Infochimps that I have my eye on, but in my case it is time and not price that holds me back from buying it.
Perhaps the datamarketplace model would work if providers could sell their services based on expertise (e.g., they specialize in extracting social network data or in connecting a given dataset to geographic locations), to make it easier to find the person who can deliver the data you want.
That said, I am glad datamarketplace trying it a different way.
Pingback: Emerging Marketplaces For Data - PSFK