• Who’s Going to Win Super Bowl XLII?

    February 3, 2008  |  Statistics

    I just put down $20 on today's game for the New York Giants to cover the 12-point spread. Of course, knowing me, I got to thinking how that betting line is decided. Is there one person who calculates the spread? Do Las Vegas casinos just put up numbers based on past experiences? I did a little bit of research, and here's what I found.
    Continue Reading

  • Weekend Minis – Government, Environment & Angry Employee

    February 2, 2008  |  Data Sources

    FedStats - Provides access to the full range of official statistical information produced by the Federal Government, including population, eduction, crime, and health care.

    MAPLight - A detailed database that brings together information on campaign contributions and votes in the California legislature. Check out the video tour.

    EarthTrends - A collection of information regarding the environmental, social, and economic trends that shape our world.

    Angry Employee Deletes All of Company's Data - A woman about to "lose" her job goes to the office at night and deletes 7 years' worth of data. Can we say backup, please?

  • Bad Statistics Leads to Poor Results and a Questionable Trial Verdict

    February 1, 2008  |  Mistaken Data

    Peter Donnelly talks about the misuse of statistics in his TED talk a couple of years back. The first 2/3 of the talk is an introduction to probability and its role in genetics, which admittedly, didn't get much of my interest. The last third, however, gets a lot more interesting.

    Donnelly talks about a British woman who was wrongly convicted largely in part because of a misuse of statistics. A so-called expert cited how improbable it would be for two children to die of sudden infant death syndrome, but it turns out that "expert" was making incorrect assumptions about the data. This doesn't surprise me since it happens all the time.

    Lesson Learned

    People misuse statistics every day (intentionally and unintentionally), and oftentimes it doesn't hurt much (which doesn't make it any better), but in this case improper use directly affected someone's life in a very big way. One of the most common assumptions I see is that every observation is independent, which often is not the case. As a simple example, if it's raining today, does that change the probability that it will rain tomorrow? What it didn't rain today?

    In other words, the next time you're thinking of making up or tweaking data, don't; and the next time you need to analyze some data but aren't sure how, ask for some help. Statisticians are nice and oh so awesome.

    Here's Donnelly's talk:

  • Journal of Quantitative Analysis in Sports is Live

    January 30, 2008  |  Statistics

    basketball-rounded

    Whenever I tell people that I study Statistics, they almost always respond, "So what do you do with that?" After they get over their initial shock, I often get, "If I were in Statistics, I'd study sports statistics." I usually respond by telling them that while it would probably be a lot of fun, I don't think there is much money in it (because I gotta eat, right?) and that statisticians usually take that as a part time gig. I'm thinking I might have to change that response though, as the game of sports statistics is showing signs of life with the recent Journal of Quantitative Analysis in Sports.

    Articles in the Journal of Quantitative Analysis in Sports (JQAS) come from a wide variety of sports and perspectives and deal with such subjects as tournament structure, frequency and occurrence of records and the optimal focus of training for decathlons. Additionally, the journal serves as an outlet for professionals in the sports world to raise issues and ask questions that relate to quantitative sports analysis. Edited by economist Benjamin Alamar, articles come from a diverse set of disciplines including statistics, operations research, economics, psychology, sports management and business.

    Maybe I'll read regularly and take up sports betting as my new hobby.

  • Walker Tracker – A Community Site for Pedometer Fans

    January 23, 2008  |  Data Sharing

    Those of you who have been around since the beginning know that I am just obsessed with my pedometer. Albeit, lately, I haven't felt inclined to go for a winter stroll in the below freezing weather. When I was keeping track of my steps though, one of the difficulties was staying consistent. Sometimes I would forget to wear my pedometer, while other times I would forget to record my steps.

    I imagine Walker Tracker could help a bit in solving that second problem. I know it was always easier to make it to the gym when I knew one of my friends was going to meet me there. Walker Tracker is like that friend at the gym. The site lets you keep track of your steps as well as see how others are doing.

    We're trying to change the world. We're trying to get you and us and everyone we know off the elevator and out of the car and onto the sidewalks and trails. We're doing it one step at a time.
    Get up, stand up and walk.

    OK, maybe it's a little hoorah, but if you feel like actually accomplishing a new year's resolution this year, Walker Tracker could be a good place to start.

    [via Web Worker Daily]

  • Google Decides to Host a Whole Lot of Scientific Data – Palimpsest Project

    January 21, 2008  |  Data Sources

    Google ResearchIn its continued efforts for absolute power over all information ever created in the world, Google will be hosting open-source scientific datasets at its research section. Here are the presentation slides from Google's Jon Trowbridge:

    In the next few weeks, terabytes of data will be made available to the public. For example, all 120 terabytes of Hubble Space Telescope data is going to be online. That's kind of cool but kind of scary too. Such a large amount of data is bound to affect lots of people on many different levels.

    For scientists, data will be available for deeper research. For the scientists who generated the data, their research could be placed under more critical scrutiny. Existing data applications might be eclipsed by the data giant, or it could go the other way such that the general public grows more aware of data-type things. Mashups will in turn spring up as well as more visualization, I am sure.

    All of this Doesn't Matter If...

    Of course, all of this depends on what data end up on the Google servers and how easily accessible the data are. Knowing Google, I don't think accessibility will be a problem. Getting data will be the super hard part. Who will be willing to contribute their data? What type of data will get contributed? Will it be the good, raw data or more cleaned and processed data? Do researchers even want to share their data with the rest of the world?

    It's going to be interesting to see what goes up on Google Research in these coming weeks.

    [via Wired and Pimm]

  • Iraq Body Count: A Human Security Project

    January 17, 2008  |  Data Sources

    Iraq Body CountIraq Body Count keeps track of civilian deaths by cross checking media reports and hospital, morgue, and NGO figures. Along with a widget counter that you can post on your blog or site, IBC also makes their database available for download.

    Systematically extracted details about deadly incidents and the individuals killed in them are stored with every entry in the database. The minimum details always extracted are the number killed, where, and when.

    The data comes in two sets -- incident reports and individuals who have lost their lives -- in the form of CSV files.

    Albeit, the data is a little depressing, but still very necessary.

  • Going Beyond Collaborative Visual Analytics with Statistics

    January 15, 2008  |  Social Data Analysis

    Jeffrey Heer et al. writes in Design Considerations for Collaborative Visual Analytics about a couple of models for social visualization -- information visualization reference model and the sensemaking model. The former is a simpler, more straightforward model starting with raw data -> processed data -> visual structures -> actual visualization; while the latter is a bit more complicated with similar stages but with feedback loops. My main reflections weren't so much with the ideas proposed by the paper. Rather, I'm more interested in what was not mentioned -- not only in this paper but in other social data analysis papers.

    Continue Reading

  • 25 Highest Grossing Films of All Time (Wallpaper)

    January 2, 2008  |  Data Sources

    I love to look at how the current week's movies are doing at the box office. I'm not really sure what it is. I think it's kind of like a gauge for what good movies are out; or maybe I'm just constantly amazed by the millions of dollars that movies make; or I think it could be my addiction to numbers?

    Something that always strikes me as interesting is how movies are always breaking records at the box office. So and so movie just broke the record for most money made over a single weekend or a month or a long holiday weekend or for a Thursday when there was at least 2 inches of rain and a dog skateboarded two miles.

    I took a look at the 25 highest grossing American films, adjusted for inflation. I'm so tired of hearing statistics for money comparisons over time that don't adjust for inflation. Wow, gasoline prices are at an all time high. Well guess what -- so are milk, bread, burgers, televisions, light bulbs, paper, cars, and everything else on the planet. Sorry, slight tangent.

    Download the Wallpaper

    As an early birthday gift to you, here are my results in wallpaper form:

    Grossing Films Wallpaper 1024 x 7681024 x 768

    1280 x 1024

    1440 x 900

    The movie titles are color coded for genre and the higher grossing films are in a larger font. Drama and action/adventure clearly dominate -- The hills are alive. Luke, I am your father. Phone home. I'll never go hungry again.

    Surprisingly (at least to me), only 7 of the top 25 films won the Oscar for best picture and of the top 50, only 9 won best picture.

  • John Tukey and the Beginning of Interactive Graphics

    January 1, 2008  |  Exploratory Data Analysis

    John TukeyWith the start of a new year, it only seems right to open with John Tukey and his work with interactive graphics. In 1972, when computers were giant and screens were green, John Tukey came up with PRIM-9, the first program to use interactive dynamic graphics to explore multivariate data. PRIM-9 allowed picturing, rotation, isolation, and masking. In other words, PRIM-9 allowed users to see multivariate data from different angles and identify structures in a dataset that might otherwise have gone undiscovered (kind of like the more recent GGobi).

    To fully appreciate the revolutionary nature of PRIM-9 one has to view it against the backdrop of its time. When Statistics was widely taken to be synonymous with inference and hypotheses testing, PRIM-9 was a purely descriptive instrument designed for data exploration. When statistics research meant research in statistical theory, employing the tools of mathematics, the research content of PRIM-9 was in the area of computer-human interfaces, drawing on tools from computer science. When the product of statistical research was theorems published in journals, PRIM-9 was a program documented in a movie.

    John W. Tukey's Work on Interactive Graphics. The Annals of Statistics, Vol. 30 No. 6. 2002.

    Luckily, you can appreciate Tukey's work here at the ASA video library. It's even more amazing when you consider where computers and technology were at back then. Who knows where Statistics would be if it weren't for Tukey and his brilliance and creativity. I can't imagine, or maybe I just don't want to.

    Tukey was someone who truly understood data -- structure, patterns, and what to look for -- and because of that, he was able to create something amazing.

  • Using Data to Find Likely Crime Spots

    December 27, 2007  |  Statistics

    I stumbled across this article about Aili Malm, a GIS specialist (I think) who uses social network analysis to find the most probably locations of organized crime.

    "I look at where organized crime groups are located and I study how these groups are linked to one another," she explained. "I can chart their cell phone use or e-mail communication or with whom they co-offend. Based on these connections, I try to isolate the important players. Then I take the social and make it spatial. I look at individuals important to the criminal network and map where they live and where they commit their crimes."

    It's just like that show Numb3rs on CBS. Albeit, math and statistics is a bit glorified on the show, but hey, at least it's loosely based on reality.

  • Download Detailed Baseball Statistics from the DataBank

    December 21, 2007  |  Data Sources

    1220-baseball

    Baseball (or all sports for that matter) statistics are all over the place. You can easily find data for pretty much whatever sport and for whichever player you want at any given time. The problem is that if you want to download all of the data at once, you usually have to write a script and do some parsing. Who wants to do that? I don't.
    Continue Reading

  • Migration/Demographics Database Available for Download

    December 14, 2007  |  Data Sources

    United Nations and Migration InformationFor our humanflows visualization, we used data from the United Nations Common Database and the Migration Information Source. The great thing about these types of sources is that they are publicly available so that everyone gets to have fun with the data. The downside is that the data is accessible via a user interface that often makes it a chore to get all of the data.

    Hence, to save you some time, you can now download the migration database that we used. I don't see any reason why you have to go through the whole data importing process when we already did it. Enjoy!

    Disclaimer: Keep in mind that the data is from the United Nations and Migration Information Source, so you should refer to the two sites for any documentation. In a nutshell, the inflows table is from MIS and the rest is from United Nations. If you're looking for more, you might also want to check out OECD. I really wanted to use their data at the time, but was having trouble accessing it from Spain.

  • Does Your Name Affect How You Perform in Life?

    December 12, 2007  |  Statistics

    Studies on names and performance seem to be all the rage right now:

    We like our names. And that preference can have negative repercussions, according to research published last month. Major leaguers with "K" initials tend to strike out more, perhaps reflecting the batters' unconscious pull to appear next to the strikeout symbol "K" on scorecards. Students with initials C and D have worse grades than the A's and B's and everyone else, gravitating toward the grades their initials represent.

    Of course, I'm a little skeptical about all of these studies, and with tiny effects like 0.02, these studies probably deserve it. In any case, they're still interesting to read about. I wonder how one could get his hands on such data. The data's probably just an email away, but in my current half-asleep stooper, I'll leave that for another time. I'm sure it'd be really interesting to play with though.

    Have you read Freakonomics? If you have, all of these name studies remind me of that chapter about the two brothers named Winner and Loser. If you haven't read the book, uh, there's a chapter on two brothers named Winner and Loser.

  • Netflix Prize Dataset Visualization

    December 11, 2007  |  Exploratory Data Analysis

    netflix-prize (1)

    Most are familiar with the Netflix Prize. If you're not, Netflix has offered a one million dollar prize to whoever improves their movie recommendation by a certain amount. It's been going on for a little over a year with still no grand prize winner. The dataset is 100 million ratings.

    The above is a visualization of the Netflix dataset. Each dot represents a movie, and the closer two dots are the more similar the two corresponding movies are based on Netflix ratings. I'm guessing the orientation of the dots was decided by some variant of multidimensional scaling.

    It's kind of fun to scroll over the clusters. Like in the bottom right we see Babylon 5, Buffy the Vampire Slayer, Alias, and Battlestar Galactica clumped together. The giant blob in the middle, however, is pretty useless; it'd probably benefit from some zoom functionality.

    The Need to Explore

    I'm kind of surprised that I haven't seen more Netflix visualizations like this (or ones better than this), because I'm pretty sure it would help see some relationships that typical analysis won't provide. I was browsing the forum and saw someone ask if others had had success loading the 100 million observation dataset into R. Silly undergrad.

    A computer scientist, designer, and statistician walk into a bar; they discuss how they would boost the Netflix recommendation system. The punchline is that they win a million dollars, but I'm not sure what happens in between.

  • Transcript Analyzer for Republican Debate

    December 4, 2007  |  Exploratory Data Analysis

    New York Times Transcript Analyzer

    The New York Times recently put up a cool data exploration tool to sift through the transcript of the most recent Republican debate. They call it the transcript analyzer. There are three key features:

    1. View where candidates put in their two cents indicated by the blue, highlighted rectangles
    2. Read the actual chunks of transcript for each block
    3. Search the transcript to see when specific words and phrases were used indicated by the smaller gray highlighted rectangles

    My particular favorite is the search feature because it really allows readers to dig into the transcript or a reader can find out which candidate is (or isn't) talking about his or her point of interest and when in the debate the topic was discussed. The intuitive text scrolling is pretty awesome too. Good job, New York Times!

    [via Jon Udell]

  • Sharing Personal Data to Push Social Data Analysis

    November 19, 2007  |  Social Data Analysis

    I'm staying in a hostel here in Madrid and am currently in the "Internet Room." I'm on my laptop, but there are six desktop computers in front of me, all of which are occupied. Three of the six people have Facebook open plus myself. It's come to the point that Facebook has so many ways to share information, that almost everyone can find some use for it. Is there some way to share data in some similar social way?

    I know there's some data blogging available and a few social data sites, but they don't have the same feel as Facebook. I think the main reason people like Facebook (other than an entertaining way to waste a few hours) is because they personally relate to the information displayed and there's some kind of connection between friends and strangers.
    Continue Reading

  • Fast Food Restaurant Menu Items Compared

    November 14, 2007  |  Data Sources

    McDonald’s Big MacWe all know fast food is incredibly bad for us and yet we still eat it. Why? Because it has tons of fat and tastes delicious. Nevermind that we will die a few days earlier for every French fry we eat.

    Over at Calorie Counter, they try to make us feel guilty with numbers. Check out the Carl's Jr. Double Six Double Dollar Burger with 1,520 Calories and a delicious 111 grams of fat. I'm a little surprised that it beat out the Burger King Triple Whopper with cheese. I shudder just thinking about eating one of those.

    Anyways, there's a whole lot of numbers here but not an incredible amount of meaning. How bad is bad? How much fat should I consume per day? Is 111 grams of fat bad? If yes, how will it directly affect me? Yes, 111 grams of fat is bad for you. You will directly feel the effects as you sit on the toilet in the morning wondering why it is taking you so long to take a dump. Now that's context.

    Also, with all the numbers, I bet all the tables would benefit from some kind of chart or, at the least, a simple infographic. Any takers? We should have a contest for who can make fast food the least appealing using nutritional data and without bending the truth.

  • 100 Reasons You Should Be Interested in, Want to Share, and Get Excited About Data

    November 7, 2007  |  Data Sharing

    When I talk about data, people often zone out or don't really see the interest. Why does this happen? People just don't understand the wonder that is data and how much of their life is led by data. With that in mind, why would people share their data? You can't share something you don't know exists. Off the top of my head, here's 100 reasons to be interested in, want to share, and get excited about data.
    Continue Reading

  • All Linkin Park Songs Look the Same? Maybe Not.

    October 29, 2007  |  Mistaken Data

    Linkin Park Songs on Last.fm

    On Last.fm, someone took snapshots of some Linkin Park songs, compared them, and concluded that all Linkin Park songs look are the same. I guess at a glance, the songs might appear the same because of the dark chunk towards middle left, but it kind of stops there. Sure, there's some loud to soft and soft to loud alternation, but who likes songs who are loud (or soft) throughout?

    The beginning of the post:

    Each image above shows the audio level in (roughly) the first 90 seconds of a Linkin Park song. The tempo has been adjusted for a few tracks for better visual alignment.

    Wait a minute. The tempo was adjusted for better visual alignment? If you're adjusting the tempo, then really, all songs can be made to look the same. On top of that, we don't know the x-axis or y-axis units. Finally, there's a lot more to a song other than dynamics -- such as key, tempo, rhythm, and lyrics.

    Continue Reading

Copyright © 2007-2014 FlowingData. All rights reserved. Hosted by Linode.