Frederik Seiffert provides this nifty tool, LastHistory, to visualize your Last.fm listening history. Mouse over songs and find repeated track sequences. The visualization itself isn't all that useful, but it gets interesting when you hook your calendar and photos in with music. LastHistory lets you replay songs synched with your photos, and your slideshow suddenly gains a new dimension.

I don't use Last.fm, so LastHistory isn't much use to me, but I'd be interested in hearing what others think. Give it a try for your self though (for Mac only), and leave your thoughts in the comments below. Is it a reason to start using Last.fm?

---------
World Progress Report - 4 days left to order

Anatoly Zenkov provides this nifty tool (Mac and PC) to track your mouse pointer. Really simple. Just start it, let it run, minimize the window, and carry on as usual. In the end, you get this image that looks something like a Pollock. Circles show areas where the pointer didn't move while the tracks show movement.

The above is my own activity during the past couple of hours. I was just randomly browsing while watching the Laker game, so it's kind of all over the place. Run while using a single application for a while, and you might see something like Zenkov's tracks on Photoshop.

What do your tracks look like?

[via feltron]

Update: Anatoly has setup an actual site for the app instead of it sitting in the dropbox boonies of boontown. Go there for the most up-to-date info.

---------
World Progress Report - 4 days left to order

Nebul.us is an online application, currently in private beta, that aggregates and visualizes your online activity. Enter your information for Twitter, Facebook, Flickr, etc and install a plugin in Firefox to record your browsing behavior. Get something that looks like the above, sort of a donut-polar area chart hybrid. Nebul.us calls it a cloud.

While your personal cloud can be interesting to a small extent, the strength of Nebul.us will most likely come out of the social aspect. There's a social cloud to see your friends' activity and then there's a public cloud that lets you see what's trending across the network.

This of course depends on how many people use the service as well as what those users select to share. Right now, for example, the public cloud (above) isn't all that interesting. I see people are going to Twitter, Google, and Facebook. Nothing new there.

Finally, as with most of these types of things, we always have to ask, "Does this visualization do any better than a simple list?" As it stands now, no. The cloud isn't really doing anything for me. If I see something interesting, I'll just bookmark it on del.icio.us or tweet it.

But hey, they're still in the very early stages. There's a lot more work to be done.

Watch the clip below for more:

---------
World Progress Report - 4 days left to order

Ben Fogarty uses a stacked area chart to represent his life. To us outsiders looking in it's not much more than that, but to Ben I am sure there is a story in every peak and valley. It's like a "this is your life" slideshow in data.

This is the drive behind your.flowingdata. I don't think YFD is even remotely close yet to developing a personal narrative, but it's something to shoot for. I can imagine a lifetime of data replaying and watching it unfold like a movie. That'd be amazing. Then again, I might also end up like Jerry in Act 3 of Episode 88 in This American Life. Fingers crossed for the former.

[via WeLoveDataVis]

---------
World Progress Report - 4 days left to order

The brand new version of your.flowingdata (YFD) is coming soon, and of course, as a FlowingData reader, you get the first peak. Newer readers might not know what I'm talking about. Well, it's an online application that lets you collect data about yourself via Twitter.

Follow @yfd on Twitter to be the first to try it out when it's ready.

But wait, let me back up a bit...

Way back in October of last year (2008), I wrote a quick and dirty script - my first Twitter bot - to record my weight and eating habits. The user interface was one page that showed my progress.

That quickly evolved into a simple application that others could use (and some are still using) that could record eating and weight the same way I was. There was also a new tracker to record feelings. It was the birth of YFD so to speak.

In March of this year (2009), I put up a new version with more trackers, an upgraded UI, and a more reliable backend. This version actually got a bit more attention than I expected.

More importantly though, I got a lot of great feedback from all of you - what worked, what didn't work, and what else would be cool. Thanks for that.

A Sneak Peak

So I took in all that feedback and everything else I learned and went to work on a new version of YFD, which is what I've been working on for the past two months on and off.

I won't go into all the details. I'll do that when the new version is up, but here are a few screenshots:

As you can see, the new version is all about flexibility and exploration. While there's still a lot to be done before I'll be really happy with it, I like where things are headed, and I think you all will like the new features.

Be sure to follow YFD on Twitter to stay up-to-date on when you can start collecting and exploring data about yourself.

Thoughts or suggestions? Please leave them in the comments below.

P.S. If you're a YFD test user, feel free to continue recording data as you are now. All of your data will be available in the new version.

---------
World Progress Report - 4 days left to order

But all the self-surveillance tools so far are mostly about a single dataset or two at most. You track your weight and what you eat, but it's more complex than that. Life is complicated and data is an abstraction of life after all. Do you eat when you're depressed or are you depressed when you eat? Do you feel better if you exercise? What about sleep? How much sleep and exercise is best for you? What days should you exericse and how many days in a row and for how long? What truly makes you happy? I want my self-surveillance application to not only give me the ability to find these answers but to give them to me with very little effort on my part.

Process It and Find Information

There's no need for me to go over all the personal tools available online again, but they all share a common theme. They concentrate on a single aspect of your life, and the more generalized ones don't do much to relate multiple data streams. For example, RescueTime only measures activity on your computer, Bedpost is specifically for sex, and Motionbased is only for exercise. I'm not trying to knock these tools though. The area is relatively new, so we're all just getting started. My own project, your.flowingdata, is not even close to where I want it to be. Applications like Swivel (which I realize isn't specifically for self-surveillance) and me-trics do some correlating, but it's very basic and often not very useful. Daytum uses the Google Charts API, and has managed to make it look good, but there's no data processing on their part, which is again, the theme we're seeing across all self-surveillance tools.

It's the show-it-all approach. This works to extent, but the human brain can only process so much. I'm specifically thinking about the non-professional majority who don't necessarily know statistical methods. Even if you do, woudln't be nicer if the computer/application did it for you (in an intelligent way)? I mean, if all the data are on the server anyways, we might as well do something useful with it.

Digital Self in a Physical World

Okay, more data processing. Check. The ultimate self-surveillance tool has also gotta be ubiquitous. I don't want to have to wait to get to my computer to record something. I want to log it right then and there. Luckily mobile phones make that possible. With SMS and Internet connectivity practically everywhere, we can log data pretty easily. Twitter has made it easier.

There's still the tough part of remembering to send data over to the servers. A lot of it is manual data entry still, and that leads to sparse data. It also takes a while for anything worthwhile to show up in the graphs and plots, so people get bored quick. You're dubbed a "data geek" if you keep at it.

I don't know if we'll ever get rid of that stigma, but I look at Facebook and Twitter as my glimmers of hope. There's a whole culture around the two. Everyone thought Twitter was a super nerdy thing to do (my wife still thinks it's weird), but millions of people are using it and the (mainstream) community is growing at a rapid pace.

Facebook in particular has managed to intertwine itself with real life. I hung out with a very non-tech crowd this past weekend, and I heard about Facebook more than I ever had in my life. People use it to stay updated on their friends' lives as well as keep others in the know, and update their status multiple times per day. After an event in the real world, many feel it's necessary to upload pictures to Facebook immediately after. It's an interesting culture. Of course, people don't use Facebook to broadcast their weight or when just went doodie, but they are uploading and entering info (or data depending on who you ask) frequently, and I don't see why that culture couldn't extend to more data-ish things. Some people will cry information overload, but that's what data processing is for.

Make It Exploratory

The computer shouldn't do all the processing though. It should do a lot, but not everything. We are after all pretty good at finding patterns (even when they're not there). When you've been collecting data about yourself for a long while, you're going to need some exploratory tools. Without them, the pie graph and bar charts will get out of hand, and you're pretty much back at where you started - a boat load of data with nowhere to go. I'm thinking something like a Many Eyes for automatic personal data. You would then tell the computer to keep an eye on things when you find something interesting in the visualization tools. Imagine all your data streams in one place where you could explore, analyze, bookmark, etc.

The Perfect Self-surveillance Tool

In the end, I want all of my data in one place with some machine learning in the background and the ability to analyze and visualize easily and thoroughly. We're not quite there yet, but I'm looking forward to when we do. Information overload? No. Better-educated decisions and a completely different view of ourselves and our surroundings? Definitely.

What's in your perfect self-surveillance tool?

---------
World Progress Report - 4 days left to order

As many of you know, I've been working on a project that lets you collect data about yourself via direct messages on Twitter. It's called your.flowingdata (YFD). It started with just weight and sleep, but it's slowly growing. I recently added entertainment, potty time, smoking, mood, and something I call YFD pulse. You can now also download your data in CSV format.

Here's my YFD pulse...

Here's when I go... you know...

Now here's my sleep habits...

Alright, so maybe that's more information that you needed to know. But anyways, if you'd like to help me test your.flowingdata, I'm opening it up to the first 300 people. Just login at your.flowingdata with your Twitter info, and you'll be good to go. It's very tip of the iceberg at this point in development, but I'd love to hear what you all think.

UPDATE, 11:34am - It seems the Twitter whitelisting hasn't kicked in yet, so there's going to be some lag. Stay tuned. Feel free to continue logging data. No data will be lost while we wait. Sorry for the inconvenience.

UPDATE, 12:41pm - You can now login and create a YFD account. You can also start logging data. Although it won't appear, no data will be lost. I'm still waiting to get a response from Twitter.

UPDATE, 1:34pm - It looks like we're back in business. Updates will be slightly delayed until I'm absolutely sure, but feel free to create an account and record data.

UPDATE, 7:40pm - We're now back at the normal refresh rate of every 3 minutes. I'll increase the frequency in a couple of days.

UPDATE, 8:47pm - That's it for this wave of invites. Thanks everyone for the hugely positive response. Follow @yfd to stay updated on the next wave.

---------
World Progress Report - 4 days left to order

After Nicholas Felton's ever popular 2005, 2006 and 2007 annual report on himself, you knew this was coming. The 2008 Feltron Annual Report is now up for your viewing pleasure. There's a lot more mapping, data, and pages this time around.

The discipline it takes to collect that much data about yourself is quite the feat, but if you'd like to give it a try, you'll be interested in Nicholas' other project, Daytum, which is an application that allows you to do just that.

[via kottke]

---------
World Progress Report - 4 days left to order

I started tracking what I eat and my weight using Twitter in an effort to shed 10 pounds and consume less. It's already been (a really fast) two months since I started this experiment — I've lost 7 pounds so far. While there are a number of factors that can contribute to weight loss (and gain), I think the simple act of tweeting raised my awareness just enough to make me feel guilty for eating that bag of chips in the middle of the afternoon.

I Need Your Help

So I got enough positive response to keep going with this idea. I expanded my application so that anyone with a Twitter account can monitor his or her weight and eating habits. I also added two more services — sleep (above) and mood (below).

I'm not quite ready to open it up to everyone just yet. You all will be the first to know though. In the meantime, I need some people to test the application under the following non-daunting conditions:

• You have a Twitter account and actually know how to use it
• After two weeks of use, I can email you a few questions about what works and what doesn't

Ideally you'd want to lose some weight, change your sleeping habits, or are interested in your emotions. It's sorta useless otherwise. Well, I guess pure curiosity could be another reason. Anyways, if you'd like to test it out, let me know in the comments. Leave your Twitter username and use a valid email address so that I can send you the info to get started.

UPDATE: This isn't an app that processes your Twitter feed. Rather, you send direct messages to a bot and the app does the rest of the work - a similar process you might have seen on services like mycrocosm, Daytum, or FoodFeed.

---------
World Progress Report - 4 days left to order

The Gist of It

Creating my own Twitter bot was pretty straightforward (much more than I thought it'd be), mostly because Twitter provides an API and the resources to make it that way.

I wanted something really simple that I could play around with. I just wanted to be able to send a direct message to my Twitter bot, and from there, it would store my data. OK, so here are the basic steps I took:

1. Create Twitter account for bot
2. Turn on email notification for direct messages only
3. Check email periodically for new direct messages
4. Parse direct messages and store in database

Create a Twitter Account (and Email Address)

The first step is easy. Create a Twitter account specifically for your bot. The account name should be short and easy to remember. Make sure you enter an IMAP email address that is only for your bot. You could put in a general purpose email address, but it'll make your life a lot easier if the email address was specifically for Twitter.

Turn on Email Notifications

Once you've setup your bot account, turn on email notification via the Twitter options menu. For now, tell Twitter to only send you notifications when your bot receives direct messages and not when someone new follows.

Check Email and Do Something with Messages

Here's where the actual code comes in. Here's the general framework. I've left out some details that will be specific to your own purposes.

 
from imaplib import *
from email.Parser import Parser
import datetime, time, email, email.Utils
import re
 
# Connect to email server
server = IMAP4("__EMAIL_SERVER.COM__")
server.login("__EMAIL_ACCOUNT_NAME__", "__EMAIL_PASSWORD__")
r = server.select("INBOX")
 
# Find only new mail (i.e. new direct messages)
r, data = server.search(None, "(NEW)")
 
# If there are new direct messages:
if len(data[0]) > 0:
 
    p = Parser()
 
    # Loop through new emails
    for num in data[0].split():
 
        # Who email is from (Should be one line, broken for display only)
        r, data = server.fetch(num, '(BODY[HEADER.FIELDS
            (DATE SUBJECT FROM X-TwitterEmailType X-TwitterSenderScreenName
            X-TwitterCreatedAt X-TwitterRecipientScreenName)])')
        msg = p.parsestr(data[0][1])
        who = msg.__getitem__('From')
        matchemail = re.compile(r'[\w\-][\w\-\.]+@[\w\-][\w\-\.]+[a-zA-Z]{1,4}')
        email_addy = matchemail.findall(who)[0]
 
        # Twitter username
        twitter_un = msg.__getitem__('X-TwitterSenderScreenName')
 
        # If the email is a direct message sent from Twitter
        if msg.__getitem__('X-TwitterEmailType') == 'direct_message':
 
	    # When direct message sent, convert to epoch seconds
            twitter_time = msg.__getitem__('X-TwitterCreatedAt').strip()
            time_tuple = email.Utils.parsedate(twitter_time)
            epoch_seconds = time.mktime(time_tuple)
 
            # Get body of email sent by Twitter
            r, data = server.fetch(num, '(RFC822.TEXT)')
            body = data[0][1]
            twitter_dm = body.split("\r\n\r\n")[0].strip()
 
            # Do something with the twitter direct message...
            # Parse it...
            # Store it in a database?...
 
# Logout of email server
server.logout()
 

I run this script every 30 minutes with a cron. You could of course run it more frequently. The important part of this code though is that Twitter attaches its own special headers (e.g. X-TwitterEmailType). If you wanted your bot to automatically follow users that followed it, you could check the EmailType and then use the Twitter API to follow a Twitter user. For my simple purposes though, I only cared about direct messages.

That's all. There is of course plenty of room for improvement. Like I said, you could make this useful to lots of users by making your bot automatically follow those who follow it. Users can only direct message another Twitter user, if he is following. I would also delete emails that have already been read and stored somewhere so that the INBOX doesn't pile up. Yup.

Did I miss anything?

---------
World Progress Report - 4 days left to order