A global mood ring called Twitter

October 7, 2011 to Statistics • Share on Twitter • Comments (3)

In a follow-up to their mood maps, Scott Golder and Michael Macy of Cornell University look at mood cycles during the hours of the day:

They found that, on average, people wake up in a good mood, which falls away over the course of the day. Positive feelings peak early in the morning and again nearer midnight, while negative feelings peak between 9pm and 3am. Unsurprisingly, people get happier as the week goes on. They’re most positive on Saturdays and Sundays and they tend to lie in for an extra two hours, as shown by the delayed peak in their positive feelings. The United Arab Emirates provide an interesting exception. There, people work from Sunday to Thursday, and their tweets are most positive on Friday and Saturday.

It's strange that good mood peaks around midnight. Maybe the people who are in a bad mood slowly go to sleep, leaving only those in a good mood to tweet. Then again, negative mood also seems to peak around midnight. Peculiar. I don't have access to the full article, so if anyone does, I'd be interested to hear Golder and Macy's interpretations.

[Discover Magazine via @albertocairo]

PDF data woes

September 14, 2011 to Data Sharing • Share on Twitter • Comments (34)

We do not provide these tables in Excel or CSV format. You will have to cut and paste from the pdf.

— A government group that provides a lot of data

If you're going to provide a dataset to the public, or anyone for that matter, please don't use PDF as your one and only format. At the very least, provide it in Excel. You can easily export spreadsheets to PDF. I don't hold anything against the person who sent me this message. She was just doing her job. But organizations need to get with the times and provide data in a way that is actually usable.

Teaching math with context and applications

September 1, 2011 to Statistics • Share on Twitter • Comments (4)

Most of us have gone through the paces of algebra through calculus in high school. I remember lots of problems and fact sheets. Sol Garfunkel and David Mumford imagine a math education system that teaches skills for the real world and increases quantitative literacy:

Imagine replacing the sequence of algebra, geometry and calculus with a sequence of finance, data and basic engineering. In the finance course, students would learn the exponential function, use formulas in spreadsheets and study the budgets of people, companies and governments. In the data course, students would gather their own data sets and learn how, in fields as diverse as sports and medicine, larger samples give better estimates of averages. In the basic engineering course, students would learn the workings of engines, sound waves, TV signals and computers. Science and math were originally discovered together, and they are best learned together now.

[New York Times]

Geo API from Infochimps brings you closer to mapping fun

August 31, 2011 to Data Sources • Share on Twitter • Add Comment

Mostly because of the popularity of smartphones, location data is all the rage nowadays. You're almost always connected no matter where you are. Rich location data can help provide you a new sense of place, and at the same time, this sort of data can paint an interesting picture of what's going on in your country or around the world. Hence, Infochimps, the one-stop shop for data folk and developers, just announced their new Geo API.
Continue Reading

Why one death is more moving than a million

August 31, 2011 to Statistics • Share on Twitter • Comments (6)

We read the story about the suffering of an individual, and we're moved. We read in the paper that millions have died over the years due to hunger, and we're not quite as moved. This is due in part to our inability to imagine big numbers, but as David Ropeik for Psychology Today explains, the way we perceive risk also is a factor:

Paul Slovic, one of the pioneers of research into the way we perceive risk, calls this greater concern for the one than the many "a fundamental deficiency in our humanity." As the world watches but, insufficiently moved, fails to act to prevent mass starvation or stop genocides in Congo or Kosovo or Cambodia or so many more, who would not agree with such a lament. But as heartless as it seems to care more about the one than the many, it makes perfect sense in terms of human psychology. You are a person, not a number. You don't see digits in the mirror, you see a face. And you don't see a crowd. You see an individual. So you and I relate more powerfully to the reality of a single person than to the numbing faceless nameless lifeless abstraction of numbers. "Statistics," as Slovic put it in a paper titled "Psychic Numbing and Genocide", "are human beings with the tears dried off." This tendency to relate more emotionally to the reality of a single person than to two or more people, or to the abstraction of statistics, is especially powerful when it comes to the way we perceive risk and danger, because what might happen to a single real person, might happen to you. As the familiar adage puts it, "There but for the grace of God go I."

[Psychology Today via @alexlundry]

Reporters make it easier to access Census data

August 29, 2011 to Data Sources • Share on Twitter • Add Comment

Census data can provide valuable information, but the datasets are not always the easiest to access. So you often end up spending a lot of time getting your data in order before you actually get to do anything with it. Investigative Reporters and Editors has released the next phase in their Census project to make Census 2010 more accessible via a simple interface. Easily download data in bulk as CSV or shapefiles or build it into your applications with the API.

[census.ire.org via @bryanboyer]

Statisticians as a tribe

August 23, 2011 to Statistics • Share on Twitter • Comments (2)

Peter Curran for BBC Radio 4 puts the tribe of statisticians under the anthropological microscope. At the Royal Statistical Society Awards and Summer Reception, Curran interviews a number of statisticians on what they do and what statistics is really about. I mainly post this though for the part where he whispers about what he is seeing as if he were in a jungle studying a tribe of monkeys. Cracked me up.

What's statistics to you?

[BBC via @TimHarford]

Statistics for gambling

August 18, 2011 to Statistics • Share on Twitter • Comments (1)

Statistics isn't just for finding out how our world works and how companies can improve their business. No. It's also for useful stuff, like, you know, gambling. Great interview with Edward Thorp [pdf], who's best known for bringing card counting in blackjack to the masses.

I received my PhD in mathematics and then went out into the university world to teach. As it happened, I'd always had an interest in applications from all of my science play in my high school years. One idea I'd had during those days was the physical predicting of roulette. That idea had stuck with me, so as I was getting my PhD, I was working on that problem, just on the side for fun. That gave me an outlook toward gambling games that later paid off in the market. Although conventional wisdom held that you couldn't beat these games, the outlook was that that wisdom was not necessarily true and, in fact, was probably wrong. Gambling games, which were perceived to be efficient — in the financial-world sense of the word — might not be. In fact, I was convinced that wasn't the case in roulette. So I came to this orientation that the conventional wisdom wasn't right. That led me not only to build a wearable computer for roulette in conjunction with Claude Shannon of the Massachusetts Institute of Technology, but also to investigate card-counting in blackjack. I happened to see an article on blackjack strategy published in a statistical journal that was fairly close to even. After I used it just for fun, I came back and figured out a way to construct a winning strategy for the game.

[Edward Thorpe via @pkedrosky]

Get a coffee, give a coffee API

August 7, 2011 to Data Sources • Share on Twitter • Add Comment

Jonathan Stark, a mobile application consultant, is running an interesting social experiment with his Starbucks card:

Jonathan's Card is an experiment in social sharing of physical goods using digital currency on mobile phones. I stumbled on the idea while doing research for a blog post about Broadcasting Mobile Currency.

Based on the similarity to the "take a penny, leave a penny" trays at convenience stores in the US, I've adopted a similar "get a coffee, give a coffee" terminology for Jonathan's Card.

Simply save the picture of Jonathan's Starbucks card onto your smartphone and use it to buy your coffee. If you like, add money to the card so that someone else can buy a coffee.

The best part is that Stark provides a simple API that returns the balance on the card every minute. When do people buy coffee? How do people give and take? Are people more likely to give when there's a large balance or when there's nothing left? Lots of fun things to look at.

[Jonathan's Card via @kn0thing]

How algorithms shape our world

July 29, 2011 to Statistics • Share on Twitter • Comments (4)

Kevin Slavin, chairman and co-founder of Area/Code, describes how algorithms (processing tons and tons of data) are intertwined with our day-to-day and how they will change our future.

And the thing is is that this isn't Google. This isn't information. These aren't financial stats; this is culture. And what you see here, or what you don't really see normally, is that these are the physics of culture. And if these algorithms, like the algorithms on Wall Street, just crashed one day and went awry, how would we know, what would it look like?

Probably gloriously frightening. Watch the TED talk below.

[How algorithms shape our world | Thanks, Liam]

Statistics has a new name

July 29, 2011 to Statistics • Share on Twitter • Comments (4)

Simon Rogers, for The Guardian, outlines the new, hot trend on the block data journalism. It's a good, quick bullet list for what it's all about. Rogers thinks back:

Two years ago, when we launched the Datablog, all this was new. People still asked if getting stories from data was really journalism and not everyone had seen Adrian Holovaty's riposte. But once you've had MPs expenses and Wikileaks, the startling thing is that no-one asks those questions anymore. Instead, they want to know, "how do we do it?"

Further down, he notes:

You can become a top coder if you want. But the bigger task is to think about the data like a journalist, rather than an analyst. What's interesting about these numbers? What's new? What would happen if I mashed it up with something else? Answering those questions is more important than anything else.

That is what an analyst does though. A good one at least. If you're an analyst (or a statistician) and you're not asking what's interesting about the numbers, then you're in the wrong profession. So really, if you're a statistician, you very well could take up data journalism. Or another job with data in the title.

A natural reaction to statistics, even among some statisticians, is that once you graduate you either go into research or you work as a number-crunching monkey. If that's your thing, go for it with gusto, but if not, there's a lot of opportunity out there and on the way (in a variety of fields) for stat people — data journalists, information designers, data scientists, analysts, data artists, or whatever you want to call it. At the core, it's working with data, and that's what statisticians do best.

Undergraduate grade inflation

July 19, 2011 to Statistics • Share on Twitter • Comments (38)

It's a given that some colleges and programs give more A's than others, but according to data collected by Stuart Rojstaczer from about 230 schools, it seems that average GPAs have been increasing overall.
Continue Reading

Data Without Borders

June 21, 2011 to Statistics • Share on Twitter • Comments (6)

Data is everywhere, but use of data is not. So many of our efforts are centered around making money or getting people to buy more things, and this is understandable; however, there are neglected areas that could actually have a huge impact on the way we live. Jake Porway, a data scientist at The New York Times, has a proposition for you, tentatively called Data Without Borders.

[T]here are lots of NGOs and non-profits out there doing wonderful things for the world, from rehabilitating criminals, to battling hunger, to providing clean drinking water. However, they’re increasingly finding themselves with more and more data about their practices, their clients, and their missions that they don’t have the resources or budgets to analyze. At the same time, the data/dev communities love hacking together weekend projects where we play with new datasets or build helpful scripts, but they usually just culminate in a blog post or some Twitter buzz. Wouldn’t it be rad if we could get these two sides together?

Yes. It would be rad. If you're an NGO looking for help or a data hacker with a desire to provide some, sign up to the mailing list, and help Jake get the ideas rolling.

[Doing Good With Data via @ireneros]

Most common iPhone passcodes

June 20, 2011 to Statistics • Share on Twitter • Comments (8)

Some people use a passcode on their iPhones simply to prevent their kids from mucking around with it or accidentally calling the police. Others use it for actual security reasons — because there's private information on your phone that you wouldn't want a stranger to have access to. If you're in the latter group, hopefully you use a passcode that isn't easy to guess.

Daniel Amitay, developer of the Big Brother Security Camera app (now removed from the App Store), added some code to the app to record user passcodes anonymously. Here are his findings.
Continue Reading

Analysis of passwords in Sony security breach

June 13, 2011 to Statistics • Share on Twitter • Comments (11)

A little over a week ago, Sony was hit yet again with another security breach — this time over one million passwords, that were stored in plain text, were released into the wild. Software architect Troy Hunt took a closer look at the dataset and found just how predictable people's passwords are.
Continue Reading

Pew Research raw survey data now available

May 25, 2011 to Data Sources • Share on Twitter • Add Comment

The Pew Research churns out a lot of interesting results from a number of surveys about online and American culture, but they usually only shared aggregated results, pre-made charts and graphs. This is well and good for the information-consuming public; however, these results can spawn curiosities that are fun to dig into. Luckily, the Pew Research Center launched a Data Sets section that provides raw survey responses and the questions in a variety of easy-to-use data formats.

Our raw data, previously posted only as SPSS files, is now available in comma-delimited (.csv) format for all reports going back to 2003. We hope that making our data available in this open-source format will make analysis easier for researchers who don’t own a copy of SPSS to analyze our data.

This should be fun. Recent datasets include the social side of the Internet, health tracking habits, and reputation management.

[Pew Research via @kzickhur]

Open thread: Can you spot the wrongness in this tax graph?

May 17, 2011 to Mistaken Data • Share on Twitter • Comments (30)

The argument behind this graph in The Wall Street Journal is that the middle class has most of the money and ties into a larger argument about who should be taxed what. There is after all a spike in the middle. Is that really the case though? Sound off in the comments.

(Cheat sheet: Jonathan Chait explains what's going on and Kevin Drum improves the graph to show more truth, although his graph can be improved, too. Grab the data here [Excel spreadsheet] from the IRS, and give it a go.)

[Wall Street Journal via @joandimicco]

Plush statistical distribution pillows

May 13, 2011 to Statistics • Share on Twitter • Comments (3)

For the statistical nerd in you or for the child you are raising as one, Nausicaa Distribution on Etsy sells handmade gifts inspired by statistical distributions. Above shows the dastardly gang of five evil distribution plushies: Weibull, Cauchy, Poisson, Gumbel, and Erlang. Judging by their moustaches, you better watch out when they're around.
Continue Reading

Charts about sex

April 21, 2011 to Statistics • Share on Twitter • Comments (8)

OkCupid adds another report to their growing list of analyses on relationships. This time around, they look at sex and how ideas vary by demographic. The above graph shows per capita GDP versus portion of people looking for casual sex.

We were amazed at this result—money seems to be a more powerful influence on sex drive than culture or even religion.

You have, for example, Portugal, Oman, Slovenia, and Taiwan within a few pixels of each other on the right side of the graph, and Syria, Sri Lanka, and Guatemala almost stacked on the left, and all of them sit along the trend line.

Interesting as usual. What amazes me more is that so many people answer such private questions. Have any of you tried OkCupid? Are these questions part of the matching process?

See OkCupid for more findings on sex such as drive and body type and Twitter usage and commitment.

[OkCupid]

Map your location – that your iPhone secretly records

April 20, 2011 to Data Sources, Mapping • Share on Twitter • Comments (7)

Researchers Alasdair Allan and Pete Warden have found that the iPhone records cell tower access, and hence your location, in an easy-to-read file that is transferred as you switch devices. And they do this whether you like it or not.

The more fundamental problem is that Apple are collecting this information at all. Cell-phone providers collect similar data almost inevitably as part of their operations, but it’s kept behind their firewall. It normally requires a court order to gain access to it, whereas this is available to anyone who can get their hands on your phone or computer.

Allan and Warden provide an open-source application, iPhone Tracker, that maps that data. The good news is that the data doesn't seem go to be anywhere other than your own backups and devices. Privacy concerns aside, this kind of makes me wish I had an iPhone; although I suspect my map would be painfully boring.

[iPhone Tracker via Marco]

Statistics