• How to Make Your Own Twitter Bot – Python Implementation

    November 5, 2008  |  Coding, Self-surveillance, Tutorials

    Following up on my post last week about using Twitter to track eating and weight, some of you voiced some interest in creating your own Twitter bot. This post covers how you can do that.

    The Gist of It

    Creating my own Twitter bot was pretty straightforward (much more than I thought it'd be), mostly because Twitter provides an API and the resources to make it that way.

    I wanted something really simple that I could play around with. I just wanted to be able to send a direct message to my Twitter bot, and from there, it would store my data. OK, so here are the basic steps I took:

    1. Create Twitter account for bot
    2. Turn on email notification for direct messages only
    3. Check email periodically for new direct messages
    4. Parse direct messages and store in database

    Continue Reading

  • Google Visualization API Opens Up

    When Google first launched their visualization API, you could only use data that was in Google spreadsheets, which was pretty limiting. Yesterday, Google opened this up, and you can now hook in data from wherever you want. What does that mean? It means that developers now have access to all the visualization API offerings like before, but it's now a lot easier to hook visualization into data applications.

    Headed for Googley Waters

    It also means we're about to see a boom in web applications that look very Googley. Motion charts (above) are going to spread like wildfire and ugly gauges will grace us with their presence. It'll be similar to the Google Maps craze, but not quite as rampant. In a couple months from now, I will have a long list of online places that use the Google visualization API. It's going to be interesting where online visualization goes from here.

    Going back to my original question, to what extent do you think the now-open Google Visualization API will affect visualization on the Web?

    [via ReadWriteWeb]

  • 40 Essential Tools and Resources to Visualize Data

    October 20, 2008  |  Software

    One of the most frequent questions I get is, "What software do you use to visualize data?" A lot of people are excited to play with their data, but don't know how to go about doing it or even start. Here are the tools I use or have used and resources that I own or found helpful for data visualization – starting with organizing the data, to graphs and charts, and lastly, animation and interaction.

    Organizing the Data


    by sleepy sparrow

    Data are hardly ever in the format that you need them to be in. Maybe you got a comma-delimited file and you need it to be in XML; or you got an Excel spreadsheet that needs to go into a MySQL database; or the data are stuck on hundreds of HTML pages and you need to get it all together in one place. Data organization isn't incredibly fun, but it's worth getting to know these tools/languages. The last thing you want is to be restricted by data format.

    PHP

    PHP was the first scripting language I learned that was well-suited for the Web, so I'm pretty comfortable with it. I oftentimes use PHP to get CSV files into some XML format. The function fgetcsv() does just fine. It's also a good hook into a MySQL database or calling API methods.

    RESOURCES:

    Python

    Most computer science types - at least the ones I've worked with - scoff at PHP and opt for Python mostly because Python code is often better structured (as a requirement) and has cooler server-side functions. My favorite Python toy is Beautiful Soup, which is an HTML/XML parser. What does that mean? Beautiful Soup is excellent for screen scraping.

    RESOURCES:

    MySQL

    When I have a lot of data - like on the magnitude of the tends to hundreds of thousands - I use PHP or Python to stick it in a MySQL database. MySQL lets me subset on the data on pretty much any way I please.

    RESOURCES:

    R

    Ah, good old R. It's what statisticians use, and pretty much nobody else. Everyone else has it installed on their computer, but haven't gotten around to learning it. I use R for analysis. Sometimes though, I use it to extract useful subsets from a dataset if the conditions are more complex than those I'd use with MySQL and then export them as CSV files.

    RESOURCES:

    Microsoft Excel

    We all know this one. I use Excel from time to time when my dataset is small or if I'm in a point-and-click mood. Continue Reading

  • Compare Media Coverage of Presidential Candiates with Everymoment Now

    September 15, 2008  |  Online Applications

    I keep stumbling on rants about how media coverage of presidential candidates is uneven, biased, etc. Everymoment Now provides a way to see what's going on with the election from the coverage (and sort of statistical) standpoint. From Craig, the developer of Everymoment:

    In order to limit the scope I've decided to keep the focus (for now) on the 2008 US general election. It's a timely, pertinent and, I believe, quite fascinating topic to study under this sort of data visualization lens. When all is said and done, you'll be able to use this site to look back over the last 100 days leading up to the election and see how the shifts between candidates played out in the media. I think we all have a sense that things may get pretty nasty in the coming weeks. I feel that having a bird's eye, hindsight view of how things went down, which stories the media focused on and how that ultimately influenced the final outcome will be an invaluable resource.

    Check out spikes in coverage of the candidates or even events and locations. Lots of sparklines and lots of bar graphs very nicely organized.
    Continue Reading

  • See the World Through SimCity’s Eyes – One Up On OnionMap

    September 10, 2008  |  Mapping, Online Applications

    Michael comments, "Onionmap is nothing when compared to this Chinese site...They've practically mapped out the entire Shanghai (and quite a few other China cities) in a SimCity-like fashion! Amazing stuff!" He's completely right. Edushi maps Shanghai with great detail. While OnionMap looks like Google Maps with SimCity sprinkles, Edushi is just straight up SimCity.

    Unfortunately my three years of Chinese classes in high school did me no good, and I don't understand a thing on the site. Maybe someone can translate and let us know what Edushi is all about. Chinese CitySearch?

    [Thanks, Michael]

  • Mozilla Labs Ubiquity Plugin Makes Mashups Easy

    September 3, 2008  |  Software

    Mashups have been around for a while now, but for the most part have required at least a little bit of web development. Maybe it's a line of javascript or thousands. Mozilla Labs, with the Ubiquity plugin, aims to make mashup-making accessible so that everyone can view data how they want everywhere on the Web. Use natural language like "map this" to stick a map into your email or get Craigslist offerings out of the list and onto a map.

    For version 0.1, the application looks interesting. Check out the demo (or even install the plugin yourself):

    [Thanks Colin and Jodi]

  • Keep Track of Presidential Race from Many Perspectives – perspctv

    August 28, 2008  |  Online Applications

    Keep track of what's getting reported about the presidential race in somewhat realtime with perspctv. It's a nicely done news dasboard that updates on its own showing updates from CNN, Twitter, and the Blogosphere. It also shows poll results, predictions, daily reach, and search volume.

    They've got charts (above); they've got maps:

    they've got timelines:

    and they've got widgets:

    In essence, it's a news aggregater, but it's a really good one and a great dashboard for you election junkies.

    [Thanks, Iman]

  • Tell Stories With Interactive Timelines from Dipity

    August 18, 2008  |  Online Applications, Visualization

    Timelines, much like calendars, can be used to show changes over time in a straightforward way. When you have a bunch of events that occurred at certain times, mark them on a timeline, and you quickly get a sense of what's going on. Take the timeline of 10 largest data breaches for example. You see breaches get more dense as time goes by.

    Wrap this idea into web application form, and you get Dippity. There have been similar timeline applications, but Dippity does it a bit better with a primary focus on telling stories with timelines and a good interface. Zoom in, zoom out, drag, and get alternative views as flipbook, list, and map.

    Below is a little bit of context to my gas price chart. Check out the full version for a better idea of what Dippity offers. Continue Reading

  • Many Eyes Adds Wordle to its Extensive Visualization Toolbox

    August 13, 2008  |  Data Art, Online Applications

    I'm sure you've seen Wordle by now, which puts an artistic spin on the traditional tag cloud. An application by Jonathan Feinberg, Wordle lets you put any text or RSS/atom feed in as input and get a cloud of words sized by frequency and arranged every which way. Above is a Wordle cloud of the current FlowingData feed.

    Many Eyes recently added Feinberg's visualization to their slew of other visualization tools.

    Wordle marks a departure from the more analytical visualizations on Many Eyes. Why bring a self-described “toy” to a site for social data analysis? People have reported finding value beyond entertainment in creating these word clouds. Teachers have used Wordles in classrooms as conversation catalysts; others have created them to express their identities, and scholars have used them to visualize the output of statistical explorations of texts.

    No doubt Many Eyes, with Martin Wattenberg and Fernanda Viégas (who know a thing or two about design) at the helm, recognizes that data visualization isn't always about analytics and exactness. Sometimes visualization is just about getting people to think.

  • Google Releases Insights for Search – India Likes Data

    August 6, 2008  |  Online Applications

    google-insights

    Google announced Insights for Search yesterday. Think Google Trends but with more information and more useful. Type in some search terms and get the rundown on interest over time based on search volume, regional interest, and related searches. It's geared towards advertisers using AdWords, but it can still be interesting to outsiders.

    For example, I put in a search for data + visualization + design + statistics and got the above. Apparently interest for all of those subjects (i.e. FlowingData) is on the decline and India sure loves its data. I'm packing my bags to India as we speak.

    [via TechCrunch]

  • New Version of Flare Visualization Toolkit Released

    July 31, 2008  |  Software, Visualization

    A new version of Flare, the data visualization toolkit for Actionscript (which means it runs in Flash), was just released yesterday with a number of major improvements from the previous version. The toolkit was created and is maintained by the UC Berkeley Visualization Lab and was one of the first bits of Actionscript that I got my hands on. The effort-to-output ratio was pretty satisfying, so if you want to learn Acitonscript for data visualization, check out Flare. The tutorial is a good place to start.

    Here are some sample applications created with Flare:

    [Thanks, Jeff]

  • Watching Our Twitter World – twittervision Redux

    July 28, 2008  |  Mapping, Projects, Software

    I've always liked twittervision. I'm not sure what it is, but it's strangely mesmerizing, getting a tiny peak into others' lives. This weekend, I recreated twittervision with a little bit of style for good measure. Say hello to Twitter World.

    The Data

    Twitter World shows updates from the Twitter public timeline, and makes use of the twittervision API for location. Until I get whitelisted for the Twitter API, I'm polling Twitter and twittervision every six minutes to keep things fresh. Hopefully neither putters out.

    The Implementation

    Like my visualization showing the spread of Walmart, I used Modest Maps (+ OpenStreetMap) to map things out, and I used TweenFilterLite to animate. I had all the gears in place and everything working nicely a couple of hours in - but that was with a flat XML file. The hard part was feeding the thing live data and then making sure everything was synchronized. That took, um, too much time.

    In any case, not bad for a weekend project.

    PS. Don't forget to follow me on Twitter :)

  • Map With All the Common APIs at Once – Mapstraction

    July 8, 2008  |  Mapping, Software

    For those who want more out of the commonly-used mapping APIs from Google, Yahoo, Microsoft, etc, but don't want to get too heavy on the programming, Mapstraction is for you. Mapstraction is a javascript mapping abstraction library that lets you easily use different mapping APIs all at once (or switch between them).

    This means you can use functionality from one API and apply it to another, or you can just put a whole bunch of synced maps on one page like above. Other features include geocoding, polylines, marker filters, and GeoRSS and KML, so go for it. Go map crazy.

    [via ReadWriteWeb]

  • Microsoft Excel Can’t Handle Clinton and Obama’s Moola

    May 28, 2008  |  Software

    MoneyThe Internet has made it easier to donate to presidential campaigns, so much so that the Federal Election Commission has had a hard time keeping up with the seemingly sudden influx of data they have to process.

    The campaign finance reports filed by Obama and Clinton have grown so massive that they’ve strained the capacity of the Federal Election Commission, good government groups, the media and even software applications to process and make sense of the data.

    Hold up. Even computers are buckling under the pressure? The first things that came to mind were crashing servers and tech maintenance pulling their graying hair out. Reading on though, "software" is a reference to Microsoft Excel 2003, which can't handle data files larger than 65,536 rows or 256 columns.

    Phew, that was close. I mean, come on, this is nation-wide data. Give me a MySQL dump for Pete's sake.

    Anyways, tt's certainly a good indicator for how times have changed data-wise. Excel 2007 can handle more. And on that note – it's still possible to open John McCain's monthly reports in Excel 2003.

    [Thanks, David]

  • What Do You Primarily Use to Analyze and/or Visualize Data? [POLL]

    May 19, 2008  |  Polls, Software

    In elementary school through high school, I always used Microsoft Excel for my charts and graphs (and use it to clean data every now and then). In undergrad, I learned all of my programming in C++ and Java and did a little bit of engineering stuff in MATLAB. When statistics rolled along, I always analyzed data using R.

    Then I got into data visualization, and for a while it was all about Processing. When I interned for The New York Times, I used a lot of Adobe Illustrator (and still really enjoy playing with it). Lately, I've been immersed in Actionscript.

    So what do you use to make sense of data?

    If your weapon of choice isn't listed, I'd be interested to know what your "other" tool is in the comments, because, well, there's always more fun stuff to learn.

    {democracy:3}
  • Discover, Share, Publish, Distribute, and Subscribe to Data With blist

    May 12, 2008  |  Online Applications

    blist logoToday, Kevin Merritt, founder and CEO of blist, provides some background on putting data in the hands of mainstream users.

    blist is not a company of modest ambitions. We want to democratize working with data much as PowerPoint and Visio have empowered mainstream users to create their own presentations and diagrams. Before these breakthroughs in innovation, mainstream users sketched free hand and asked professionals in central resource pools (art departments and engineering departments) to turn drawings into foil transparencies and blueprints.
    Continue Reading

  • How to Learn Actionscript (Flash) for Data Visualization

    April 21, 2008  |  Software

    A while back, I asked, "What is the best way to learn Actionscript for data visualization?" As I've had Actionscript staring me in the face for the past two weeks, I can attest to the idea that the best way to learn is by doing i.e. immersing yourself in a project with a deadline looming in the dark behind you. There have been, however, a few things that have made my life a little easier as I strive for coding nirvana.

    My Only Desktop Reference

    Essential Actionscript 3.0I have stacks of books on the floor, in the closet, and on my bookshelf, but there's one book that has stayed within in arm's reach as I learn - Colin Moock's Essential Actionscript 3.0. This is usually the first place I go to look when I'm stuck on a bug or am not sure where to begin. Moock's explanations are very clear and he provides plenty of useful examples without getting too specific.

    When I first started, I read the first section "Actionscript from the Ground Up," which helped me familiarize myself with core concepts like packages, classes, and just the basic ideas of how things work. I feel like one of the hardest parts of learning any programming language is figuring out how all the components talk to each other, so this first section helped a lot. I skimmed the rest of the book, and now it's my only desktop reference.

    I'm also starting to hear great things about Learning ActionScript 3.0: A Beginner's Guide by Shupe and Rosser, but I haven't got to look at it yet.

    Flare Visualization Toolkit

    FlareJeffrey Heer's Flare visualization toolkit seems to come out at just the right time specifically for me. Seriously, the timing couldn't have been better. For instant gratification, go through the tutorial, which covers a few Actionscript basics and straightforward examples for mainly, reading in data and animating and transitioning objects.

    After the tutorial, try to build some of your own visualizations and applying what you learned from the tutorial. Finally, when you're more comfortable, dive into the Flare code to see how things work.

    Modest Maps for Flexible Mapping


    Modest Maps
    For those interested in mapping, Modest Maps has helped me a good bit. From the site:

    Our intent is to provide a minimal, extensible, customizable, and free display library for discriminating designers and developers who want to use interactive maps in their own projects. Modest Maps provides a core set of features in a tight, clean package, with plenty of hooks for additional functionality.

    They're not lying. It provides the basic map functionality like pan and zoom, but it's open, so you can do whatever you want from there. I've been using Flare and Modest Maps together to take the best of both worlds, I guess you could say. There's also the Yahoo! Maps Actionscript API, but I haven't tried it. I don't know if it's as flexible as Modest, but I like the idea of owning all of my code.

    Adobe Flex Builder for Actionscript Development

    Flex Builder 3Flex Builder has been extremely helpful while coding. The name might suggest it's only for Flex projects, but it's pretty darn good for Actionscript projects. The serious Actionscript people I've talked to only seem to use Flex. The other option is to use your text editor of choice and install the free Flex SDK, but it's more complicated (and I've never tried it).

    The downside of Flex is that it's kind of expensive, pricing at just under $250 and even more for the pro version. However, on the flip side, Flex Builder Pro 3 is free to all education customers.

    Last Thoughts

    Finally, let's not forget about Adobe's Actionscript 3.0 language and components reference. In addition to Moock's book, this is the other indispensable resource. And of course there's all the online resources you'll find ala Google.

    This is pretty much what I've been immersed in for the past two weeks. It's definitely a sharp learning curve, but once I got the hang of things, it's been pretty fun and nice to see my data moving along.

    Anyways, I'm just now starting to kick the tires. I am sure there are many of you who have been at this for a while and who know a ton more than I do. What references or resources do you recommend for Flash/Actionscript beginners like myself?

  • Your Notes, Snapshots, and Memories Accessible From Everywhere – Evernote

    April 2, 2008  |  Online Applications

    I just signed up for an EverNote account, which lets you store all of your notes online from all of your devices - tablet, paper, mobile phone, laptop, PDA.
    Continue Reading

  • Facebook Security Upgrade Rendered Useless – Private Photos Leaked

    March 25, 2008  |  Online Applications

    Leaky Faucet

    Just when you thought it was safe to upload those photos from that wild Friday night to Facebook, this happens:

    A security lapse made it possible for unwelcome strangers to peruse personal photos posted on Facebook Inc.'s popular online hangout, circumventing a recent upgrade to the Web site's privacy controls.

    The dumbest part is how easy it has been all this time to find private photos. All it took was a modified URL with a photo ID to "hack" into Paris Hilton, Mark Zuckerberg, or anyone else's private albums. I don't know the whole story, but given Facebook's excellent reputation, you'd think that they would know better. The security hole has been plugged for now, and I am sure the Facebook group is working hard to make sure there are no other leaky areas.

    This leak probably couldn't have been more poorly timed for Facebook with the release of their new security measures as well as MySpace's not so distant and a bit too familiar photo breach.

    This really makes you wonder - what's next?

    Photo by Meredith Farmer

    [via ReadWriteWeb]

  • toolbox-thumb

    Putting Analysis Online With StatCrunch and Covariable [Review]

    Are online statistical tools sufficient to analyze our complex datasets?
Unless otherwise noted, graphics and words by me are licensed under Creative Commons BY-NC. Contact original authors for everything else.