|
Over the past four years there was a 43 percent increase in prescriptions for antidepressants. Some news outlets attribute this rise to the recession. People more depressed equals more drugs. Ben Goldacre of Bad Science explains why said outlets need to be more careful with their analyses.
From what I can tell, all the reports took an aggregate (the 43 percent) and then made a big assumption to explain it. I'm all for data journalism, but statistics is rarely that straightforward.
In a guest post for the guardian.co.uk Datablog, I thought out loud about the possible end of Data.gov and what it means for open government data. Let me know what you think.
Update: Funding might not be cut completely (for now).
Last week, there were rumblings over the end of the Statistical Abstract, and I suggested that it was just a sign of changing technologies. I thought that Data.gov and similar sites were the natural progression. Here's the problem with that argument. Congress is planning on shutting down Data.gov and other transparency sites in the next few months.
Continue Reading
Not many people understand the importance of data privacy. They don't get out how little bits of information sent from your phone every now and then can show a lot about your day-to-day life.
As the German government tries to come to a consensus about its data retention rules, Green party politician Malte Spitz retrieved six months of phone data from Deutsche Telekom (by suing them), to show what you can get from a little bit of private mobile data. He handed the data to Zeit Online, and they in turn mapped and animated practically every one of Spitz' moves over half a year and combined it with publicly available information from sources such as his appointment website, blog, and Twitter feed for more context.
Continue Reading
“The problem isn’t that specialised companies lack the data they need, it’s that they don’t go and look for it, they don’t understand how to handle it.”
—Hans Rosling, A Data State of Mind, March 2011
Google UK produced a short book called Think Quarterly to distribute to partners and advertisers, but it's actually pretty interesting for a more general audience. Articles feature Hans Rosling, Hal Varian, and others. Also a hat tip to FlowingData in Simon Rogers' list of sexy resources.
The Web is a game of pageviews, and outlets such as Twitter and Facebook are a way to rack up the counts. The more people who share your posts and articles, the more new people that visit your site. So what kind of articles are shared more often? How do people with interact with these articles? Yahoo! research scientist Yury Lifshits digs into Facebook likes for some ideas, using data collected from 45 sites, 100k+ articles, and 40 million reactions, between October 2010 and January 2011.
Continue Reading
Hot off the MIT Sloan Sports Analytics Conference, Sean Gregory argues for more "stats geeks" on the sidelines and in the huddle during the game.
[S]itting next to your team's manager, a scruffy baseball lifer, in the dugout is not just another scruffy baseball lifer, spitting tobacco. Instead, by his side is a guy with a Ph.D. in theoretical physics, a beautiful mind who can calculate complex probabilities, in real time, in his head. He can tell you the odds of so-and-so throwing such-and-such a pitch to so-and-so on such-and-such a count.
It's a fluffy article with not much on what the stat person would actually do, so you'll have to imagine. Honestly, I hope sports statistics doesn't come to that though. Unpredictability is what makes games so fun to watch.
[Time via @amstatnews]
We learned the strategy to win Rock-Paper-Scissors every time, but does it really work? For the New York Times, Gabriel Dance and Tom Jackson give you your chance:
Computers mimic human reasoning by building on simple rules and statistical averages. Test your strategy against the computer in this rock-paper-scissors game illustrating basic artificial intelligence. Choose from two different modes: novice, where the computer learns to play from scratch, and veteran, where the computer pits over 200,000 rounds of previous experience against you.
Be sure to play at least five rounds, and then click on the button to see what the computer is thinking. In veteran mode, the computer searches its database for sequences that match your last five moves and its last five moves and then tries to predict what you'll throw next.
Are you good enough to beat the basic artificial intelligence?
The government has been making a big push for more open health-related data, and a couple of weeks ago, they released a whole bunch of it with the launch of HealthData.gov. It's the same interface as Data.gov, but for health. Additionally, the Health Indicators Warehouse launched with different data and a slightly more useable interface.
A quick scan of the data available, however, does seem to indicate that a lot of it is spotty or outdated (like on data.gov), which doesn't make it especially useful. For example, some data sets are only one data point, while others are only a single year. At least it's a start.
William Briggs and John Briggs examine the differences between movies that have won Best Picture and those that were top at the Box Office, based on money, gender, age, and genre. "There was only one Oscar winning movie with a leading actress older than 50: Jessica Tandy in Driving Miss Daisy. Eight women were at least 40 in Oscar winning movies, e.g. Myrna Loy, Bette Davis, Sandra Bullock. However, half of these were just 40 or 41."
[via]
Need music data? Get all the data you want and more from the freely available million song dataset, offered by LabROSA at Columbia University and Echo Nest. There's lots of metadata on song features and your standard stuff like year and artist. There are also several code wrappers and samples to help researchers make use of the data right away.
[Million Song Dataset via @MacDivaONA]
Sunlight Labs continues its work for a more open government with its recent release of the Real Time Congress API.
Today we're making available the Real Time Congress API, a service we've been working on for several months, and will be continuing to expand.
The Real Time Congress API (RTC) is a RESTful API over the artifacts of Congress, kept up to date in as close to real time as possible. It consists of several live feeds of data, available in JSON or XML. These feeds are filterable and sortable and sliceable in all sorts of different ways, and you can read the docs to see how.
There are seven data types the API will report:
Now someone has to do something with all of this data coming in. Can you think of a useful application for what is essentially an automated government Twitter feed?
There's something very strange about the anonymity of the Web that brings out the worst in some people. I don't get it, but it's something we have to deal with for now. Courtney Stanton, who wrote a couple of posts that drew the ire of a bunch of trolls, had a look at the troll comments versus the sincere ones.
I should warn you that the following content does contain adult themes, but it's the contrast between the groups that's most interesting (and the good use of Many Eyes).
Continue Reading
Sudhir Venkatesh, a professor of sociology at Columbia University, along with his students, has been studying the sex work industry since the 1990s. In a recent article for Wired, Venkatesh describes how the business has changed over the past couple of decades.
Continue Reading
OkCupid continues their analysis on the mysteries of the dating world, this time on the best questions to ask on a first date, or rather, the best questions to ask when you actually want to find out something else. Will your date have sex on the first date? Ask your date if he or she likes the taste of beer, because:
Among all our casual topics, whether someone likes the taste of beer is the single best predictor of if he or she has sex on the first date.
Well, okay, not entirely correct. The question is if they would consider sex, not if they'd actually do it. I've considered buying just about every Apple product, but all I have is the one Macbook. Still interesting though.
I'm still waiting for LinkedIn to start doing this sort of analysis. I mean it's more or less the same thing, except you're trying to find a company to work for rather than a partner to, uh, have beer with. Who's with me?
Statisticians everywhere are squealing in delight over this story on fellow statistician Mohan Srivastava, who used his know-how to crack the code of a tic-tac-toe scatcher lottery game. After winning three dollars on a scratcher ticket that was given to him as a gag gift, Srivastava got to wondering about the process of how tickets were made. As a geological consultant who figures out if areas are worth mining for gold, he wondered if he could do the same with this scatcher.
Continue Reading
Apparently moods on Twitter can be used to predict the ups and downs of the stock market, according to work from Johan Bollen and Huina Mao of Indiana University-Bloomington: "Measuring how calm the Twitterverse is on a given day can foretell the direction of changes to the Dow Jones Industrial Average three days later with an accuracy of 86.7 percent."
I can't wait until Twitter is used to predict when I want to eat and sleep, and my robot can cook me gourmet meals and provide turn down service accordingly. And it better be accurate to the minute. Anything less is failure.
Christopher Beam for Slate explains research being done at UCLA in collaboration with the LAPD on predictive policing:
Predictive policing is based on the idea that some crime is random—but a lot isn't. For example, home burglaries are relatively predictable. When a house gets robbed, the likelihood of that house or houses near it getting robbed again spikes in the following days. Most people expect the exact opposite, figuring that if lightning strike once, it won't strike again. "This type of lightning does strike more than once," says Brantingham. Other crimes, like murder or rape, are harder to predict. They're more rare, for one thing, and the crime scene isn't always stationary, like a house. But they do tend to follow the same general pattern. If one gang member shoots another, for example, the likelihood of reprisal goes up.
This happened in my neighborhood when I was in fifth grade. We lived in a pretty quiet neighborhood, but one morning a window was open. Someone had come into our house while we were sleeping and stole whatever was in immediate reach. They also stole my dad's brand new bicycle from the garage. Same thing happened to my neighbor two days later.
[Slate via @amstatnews]
Add another online destination to find the data that you need. DataMarket launched back in May with Icelandic data, but just a few days ago relaunched with data of the international variety. They tout 100 million time series datasets and 600 million facts. I'm not totally sure what that means (100 million lines, sets of lines?), but I take it that means a lot.
Just over 2 years and countless cups of coffee after we started coding, DataMarket.com launches with international data. You can now find, visualize and download data from many of the world’s most important data providers on our site.
At first glance DataMarket feels a lot like now defunct Swivel. Search for the data you want and you get back a list of datasets. The focus on only time series though is actually a plus in that they can provide more specific tools to visualize and explore. The current toolset isn't going to blow you away, but it's not bad.
Continue Reading
President Barack Obama delivered his State of the Union address yesterday, and this year it was "enhanced" by charts and graphs. Basically, as Obama spoke, graphics that you could equate to Powerpoint slides showed up on the side. What'd you think of the enhancement? Did it add or detract from the message? Were the graphics used honestly and effectively?
One thing's for sure: there's something wrong with that bubble chart. Uh oh.
Continue Reading
FlowingData explores how designers, statisticians, and computer scientists are using data to understand ourselves better — mainly through data visualization. More...
Where are the biggest box office movies (not) streaming?
Find out what percent you are in
High-resolution maps of science
Why I want to quit cable
Cinemetrics creates a visual fingerprint for movies
Vehicles involved in fatal crashes
World subway paths at scale
Visualize This