• New Lessons Every Day

    July 20, 2007  |  Miscellaneous, The Times

    steps1

    Every day I learn a lot, and every day I get better. For most of the day today, I worked on a single graphic (that hopefully runs in the paper). I gave it to the person in charged, and oh man, there was a lot to change. Fonts, labels, fill colors, bar widths, spacing, layer orientation, size... on and on and on. I think it might have been faster for him to make the graphic himself than it was for him to fix mine.

    Sigh. Gotta practice.

    The graphic above is the number of daily steps I've taken since I started wearing a pedometer. Can you tell when I moved to the city and was forced to walk to the subway and work?

  • My First Graphic, Getting There

    July 18, 2007  |  The Times

    So I started on a graphic today, and it might actually be in the paper. I'm pretty excited to see my first graphic published. I won't say what it is or when until I actually find out if it gets published to save myself from any embarrassment, but nevertheless, cool to think about.

    In other news, I got to see a coworker do his stuff with some mapping and what not, whizzing through Adobe Illustrator like it was part of him. His attention to detail and his ability to do it quickly were what impressed me the most. I have a lot to learn and a lot to do.

    It's also comforting to know that previous interns didn't know any Adobe Illustrator either, so I don't feel as dumb anymore.

  • Second Day at The Times

    July 17, 2007  |  The Times

    Ok, so I said I would update after my first day, but I got home around 8p, and was just too tired to do anything. I forgot what it's like to continuously work for 8 hours a day. It's 9:30p right now, and honestly, I'm ready to pass out.

    My first two days have been interesting, and it's very clear that I have a lot to learn, in terms of visualizing graphics. I have yet to create an actual graphic. Rather I've been sifting through Iraq data, looking for stuff that's interesting, and then putting it in a spreadsheet. I was also given an article to read that could possibly benefit from a graphic, but it turns out that the writer already got some amazing photos and a small map, so that was a no-go.

    I think my strength is R, so I'm going to try to improve (um, learn) mapping in R. I think if I can do that, I will be of much more help. I was asked today if I could do maps, but unfortunately, I've only done very basic things, and the task was time-sensitive. Sigh.

    Am I rambling? I feel like I'm rambling. I need to sleep. But I need to learn R. Alright then.

    On a completely random note, I was able to see Matthew Carter, a master font designer, who has designed fonts like Georgia and Verdana, speak today. I never really put much thought into type faces, but wow, there's a whole lot that goes into it. A lot of subtleties that involve making more text fit on a page without cluttering or techniques to make text easier to read on a newspaper or from a magazine.

    Oh yeah, I also saw my idols -- the IBM Visual Communications lab. I didn't get a chance to talk to them though, but they looked like a friendly bunch.

  • The Times: Wealthiest Americans Ever

    July 15, 2007  |  Statistical Visualization

    Wealthiest Americans ever

    In honor of my New York Times induction day, a visualization of The Wealthiest Americans Ever. You think good ol' Billy would be there at the top of the list, worth $82 billion, there have been a few who have preceded the software giant e.g. John D. Rockefeller worth a crazy $192 billion. Just think how many Jack in the Box tacos you could buy with that kind of money.

  • Internship at the New York Times Tomorrow

    July 15, 2007  |  The Times

    Two months flew by in a hurry in between Spring quarter ending and my New York Times internship beginning. I arrived in New York City today and I'm starting at the times tomorrow. For the next 10 weeks, I'm going to be working in the graphics department, and I anticipate it's going to have to be a pretty steep learning curve.

    I don't really know what I'm going to be doing yet, so I've familiarized myself with Adobe Illustrator and Flash and tried to pick up a few more skills in R. Originally, I was thought I was going to become some kind of expert in Illustrator and/or Flash, but after some time with the books, I've got a long a road ahead. Not being so hot in either Illustrator or Flash, I'm a little nervous, but I guess I'll just wait and see.

    Updates on my first day tomorrow.

  • More Mapping from amMap Offering Flexibility

    July 13, 2007  |  Mapping, Online Applications

    amMap

    Yes, more mapping. Map, map, map. amMap offers a Flash-based mapping tool that you can download and customize to your liking.

    Ammap is an interactive flash map creation software. Use this tool to show locations of your offices, routes of your journeys, create your distributor map. Photos or illustrations can be used instead of maps, so you can make different presentations, e-learning tools and more.

    There's some smooth browsing and zooming, and it's pretty sleek. Those who appreciate simplicity will appreciate amMap. Plus, it's free :) Continue Reading

  • TED Talk: What do we really know about the spread of AIDS?

    July 13, 2007  |  Statistics

    In her TED talk, Emily Oster challenges our conception of AIDS and suggests other covariates that we need to look at (e.g. export volumes of coffee). Until we get out of the mindset that poverty and health care are the only causes/predictors of AIDS, we won't be able to find the best way to fight the disease. Another great use of data.

    I do have one small itch to scratch though. Emily had a line plot that shows export volumes and another line, on the same grid, of HIV infections, both over time. It reminds me of the plots that Al Gore uses with carbon dioxide levels and temperature. Anyways, using the plot, Emily suggests a very tight relationship between export volumes and HIV infections. Isn't export volume pretty tightly knit to poverty? I don't know. She's the economist, so she would know (A LOT) better than me. I guess I just wish she talked a little bit about the new and different data she has that compels us to change our conceptions.

  • Gas Prices Over Time, 2000-2006

    July 12, 2007  |  Statistical Visualization

    Gas Prices over TimeWhile on the subject of gas prices, Foreign Policy has a graph of the prices per gallon of gasoline from 2000 to 2006. With the US at the lower tier, I feel like a bit of a whiner ("Waa waa waa, it costs 30 dollars to fill my tank"). At the lower end, it seems Venezuela seems the place to be, with some major government subsidizing going on.


  • America Wins in Petrol Consumption per Day

    July 12, 2007  |  Statistical Visualization

    petrol

    A very simple graph from The Economist (spiced up a bit with a picture of a delicious gasoline droplet) that quickly gets its point across. The United States uses a lot of petrol compared to other countries, while at the same time, it costs less to fill up a Honda Civic in the US than most other places.

    However, the left graph is based on 2003 data. I wonder what the graph looks like now? Similar, I'm sure, but still something to look at.

    Anyways, something really interesting here -- even though Venezuela has crazy low gas prices, the average petrol consumption per day over there is still quite low. Whether this is a cultural thing or just some weird supply and demand thing (that I have no clue about) might be worth some investigating.

    In any case, just because we have lower gas prices (that we still complain about) than a lot of the world, we're still consuming a lot. What's our excuse?

  • Making Public Data Public

    July 11, 2007  |  Data Sources

    As Jon Udell has mentioned, there's a ton of data online, but it's not often we can find it, often hidden in the deep, dark basement of some website. He has proposed that people book mark public datasets on del.icio.us under the tag "publicdata". I think this is a great idea. In turn, you can subscribe to the feed with the url http://del.icio.us/tag/publicdata.

    I've been doing this already for a while, but I had been just tagging with "data". So I'm going to join in on the party and start tagging with publicdata, and I hope others will too. Until sites like Many Eyes and Swivel get more wind beneath their wings, I think it's necessary.

  • Visualization of Taste Explosions from Ratatouille

    July 11, 2007  |  Data Art

    Ratatouille Visualization

    By now, I'm sure everyone has heard of Pixar's most recent movie, Ratatouille. If you haven't seen it, I HIGHLY recommend it. Not only is it beautiful animation and a nice story, but it's about food. I love Pixar. There are a few scenes in the movie when the main character, Remy, and his brother, Emile, are eating and experiencing the taste of some exquisite cheese.

    There was pretty taste visualization going on done by Michel Gagne.

    Around 1400 drawings were created for the animation. Each one was scanned, painted and composited using two softwares: Animo and Photoshop.

    That's a lot of hand drawings, but quite nice results. Good job, Michel.

  • 10,000 (Literal) Steps to Healthier Living

    July 10, 2007  |  Self-surveillance

    PedometerIt's really easy to be lazy when you work from home. I can tell you this first-hand.

    Twenty-six steps from my bedroom to the kitchen; 6 steps from bedroom to study room; 29 steps from study room to kitchen; 24 steps from kitchen to bathroom. Do some back and forth, go through the rotation a few times, and that's my day. I can easily go a whole day walking (or dragging my feet) only 300 steps. That's sad.

    Just how sad is it? The Walking Site (um, yes, there really is a walking site :) recommends 10,000 steps per day. Wow, only 9,700 steps away! I'm pretty sure I'm slowly getting fatter due to my sloth-like behavior.

    In efforts to avoid the gut, I'll be wearing my trusty pedometer to shoot for 10,000 steps per day. Of course I'll be logging this data online, and we can all see how un-lazy I can become. Who knows?

    I can tell you this though. I used to wear this nifty step counter a few months back, and it certainly made me more aware of my laziness. I started walking more and took the long route, around campus, from my office to the car. Sometimes, we just need to see proof to change. As if a pot belly and excessive sweating wasn't enough.

  • Browser Statistics Firefox Add-on

    July 9, 2007  |  Software

    mozilla-browser-stats

    I just added the Browser Statistics add-on to my Firefox browser. On the bottom left corner, it shows the number of kilobytes downloaded for the current page, total number of kilobytes downloaded since the last start of the browser, and number of pages loaded. I'm going to try to log these numbers each day and try to make use of the data (uh, if laziness doesn't get the best of me). If only there were some automated data logging.

  • Finding Weirdness in Temperature Data

    July 9, 2007  |  Mistaken Data

    wunderplot500

    After parsing Weather Underground pages to grab temperature data, it's time to look at the data. Can't download all that data and not do anything with it!

    First off, in my initial pass of my parsing script, I accidentally cut the month range short, so I didn't get any data for December from 1980 to 2005. It should be noted that these plots don't show this missing data. Um, there's no axes or labels either. Sorry, I got a little lazy, but that's not the point now anyways.
    Continue Reading

  • Grabbing Weather Underground Data with BeautifulSoup

    July 9, 2007  |  Tutorials

    Weather Underground is a useful site and a fun place for weather enthusiasts. WU has a bunch of weather data (current and historical) from established weather stations, like at airports, and home stations setup by hobbyists. One problem: most of the data is in HTML tables instead of the CSV format that we like. I say most because you can download hourly data from a single day in CSV, but if you want say, temperature data over the past 5 years, you're kind of at a loss.

    But wait, there's a solution. That solution is BeautifulSoup, an XML/HTML parser written in Python. Um, parse... what does that mean? Basically, the Python script will go through, or look at, a document extracting certain information from that document.

    Back to WU. Like I said, there's historical data in HTML tables like this. I just want the actual mean temperature in Fahrenheit for the past five years or so. I could go to every single page manually and record the temperature in Excel, but why do that when I can make the computer do it for me?
    Continue Reading

  • Juice TESTING in Competitive Sports

    July 8, 2007  |  Mistaken Data

    Juice testing

    It's easy to see how Statistics got this bad wrap because it's so easy to lie with data, charts, and graphs. Sometimes it's on purpose -- someone might try to present "good" results that actually suck. Sometimes it's accidental -- someone might have misread or didn't read the documentation that came with the data. In the case of Swivel's most recently featured graph, it was the latter. A case of mistaken identity so to speak.

    The data about doping tests in sports came from here. Now the graph on Swivel would have you believe that the data represent the number of doping cases found in each sports; however, according to the USADA report, the data is actually the number of tests the association conducted inside and outside competition during the first quarter of this year. The report contains no data on the USADA's findings.

    What We Learn

    What can we learn from this? It's great to visualize data, but you have to be careful. Read the documentation. Find out what the data is about, because without context, the visualization or any findings are practically useless. Statistics isn't to lie. In fact, it's the exact opposite. Statistics came about and exists today to reveal the truth.

  • Xtimeline to Explore and Create Timelines

    July 7, 2007  |  Online Applications

    xtimeline

    Xtimeline allows you to explore all sorts of user-created timelines from the US war in Iraq to the life of Angelina Jolie to the history of pornography. I think the site is still pretty new since the most viewed timelines for the month, past 3 months, and year are still all the same, but nevertheless, from the looks of things, a nice community seems to be developing over there.

    The timelines are (I think) in javascript and what you see is a timeline of user-entered events. As you click and drag through time, events are displayed on the right. You can click on the events for more details where events can be anything from text, a picture, or a Flash-embedded video.

    One suggestion -- it looks like timelines can only be ended by a single user. It would be cool if multiple users could contribute to a single timeline, because I think it's hard to remember all the dates (especially the months) for certain events. We can't all be like Victor, who seems to know an awful lot about Britney Spears.

    *UPDATE* I just read the xtimeline blog. Yup, xtimeline did in fact, just open up to the public July 1.

  • Grab Data with templatemaker

    July 7, 2007  |  Software

    Adrian Holovaty released templatemaker yesterday. Adrian is probably best known as the guy, featured on YouTube, who played the MacGyver theme song. So clearly, he a man a many talents.

    Anyways, templatemaker is a Python script to extract data from text, um, HTML. For example, you could pass a review page from a site like Yelp, or several pages, and the script will "learn" the template. Once a template is established, you can extract the stuff that changes (e.g. ratings, restaurant name). Here, in Adrian's words:

    You can give templatemaker an arbitrary number of HTML files, and it will create the "template" that was used to create those files. ("Template," in this case, means a string with a number of "holes" in it, where the holes represent the parts of the page that change.) Once you've got the template, you can then give it any HTML file that uses that same template, and it will give you the raw data: "The value for hole 1 is 'July 6, 2007', the value for hole 2 is 'blue'," etc.

    It's under the BSD license, so all the more reason to use it. I haven't used it yet, but looking forward to it.

  • History Over Space and Time

    July 6, 2007  |  Mapping

    Maps of War

    As a representation of history over time and space, Maps of War does a pretty good job of displaying the information in the form Flash animations. It's quite simple really. The animation starts centuries back (e.g. 2000BC) and moves to geographic regions. In the above map, I watched who has controlled the middle east, beginning 3000BC up through 2006.

  • Hans Rosling: Providing Data, Inspiring Change

    July 6, 2007  |  Online Applications

    Okay, so this video has been posted probably on thousands of blogs already, but you know what, I don't care. Hans Rosling gives an amazing talk on poverty and life around the world, and he uses his interactive exploratory tool, Trendalyzer (acquired by Google), to show the different levels of health, education, and money around the world. Trendalzyer: useful, yes, but not the main point of the talk. Watch Rosling's talk all the way through. You won't be disappointed.

Copyright © 2007-2014 FlowingData. All rights reserved. Hosted by Linode.