40 Essential Tools and Resources to Visualize Data
One of the most frequent questions I get is, "What software do you use to visualize data?" A lot of people are excited to play with their data, but don't know how to go about doing it or even start. Here are the tools I use or have used and resources that I own or found helpful for data visualization – starting with organizing the data, to graphs and charts, and lastly, animation and interaction.
Organizing the Data
Data are hardly ever in the format that you need them to be in. Maybe you got a comma-delimited file and you need it to be in XML; or you got an Excel spreadsheet that needs to go into a MySQL database; or the data are stuck on hundreds of HTML pages and you need to get it all together in one place. Data organization isn't incredibly fun, but it's worth getting to know these tools/languages. The last thing you want is to be restricted by data format.
PHP
PHP was the first scripting language I learned that was well-suited for the Web, so I'm pretty comfortable with it. I oftentimes use PHP to get CSV files into some XML format. The function fgetcsv() does just fine. It's also a good hook into a MySQL database or calling API methods.
RESOURCES:
Python
Most computer science types - at least the ones I've worked with - scoff at PHP and opt for Python mostly because Python code is often better structured (as a requirement) and has cooler server-side functions. My favorite Python toy is Beautiful Soup, which is an HTML/XML parser. What does that mean? Beautiful Soup is excellent for screen scraping.
RESOURCES:
MySQL
When I have a lot of data - like on the magnitude of the tends to hundreds of thousands - I use PHP or Python to stick it in a MySQL database. MySQL lets me subset on the data on pretty much any way I please.
RESOURCES:
R
Ah, good old R. It's what statisticians use, and pretty much nobody else. Everyone else has it installed on their computer, but haven't gotten around to learning it. I use R for analysis. Sometimes though, I use it to extract useful subsets from a dataset if the conditions are more complex than those I'd use with MySQL and then export them as CSV files.
RESOURCES:
Microsoft Excel
We all know this one. I use Excel from time to time when my dataset is small or if I'm in a point-and-click mood.
Charts and Graphs
Alright, the data are processed, formatted, and ready to go. Now it's time to visualize. The software I use for static charts and graphs depends on the task at hand, so I try not to limit myself to anyone piece of software. For example, R is good for quick results, but no good for a Web application.
Adobe Illustrator
I use Adobe Illustrator for publication-level graphics. I learned how to use it when I was at The Times out of necessity and have been enjoying it since. You can manipulate every element of a graph with a simple click and a drag - which can be a blessing and a curse.
RESOURCES:
R
If you have a particular type of (non-animated, non-interactive) statistical visualization in mind, R has probably got it. R is free with countless libraries available. If you can't find a library to suit your needs, you can always script it yourself. One cool thing about R is that you can save your graphics as PDF and then polish it in Adobe Illustrator.
RESOURCES:
PHP Graphics Library
I've only had limited experience the the PHP GD library. There are several PHP graphing packages available, but I haven't found one that I liked a whole lot, so I'm usually more satisfied drawing my own graphs with the GD library. The Sparklines PHP graphing library isn't half bad either.
RESOURCES:
- Official PHP GD Reference
- PHP GD Resource
- Sparklines PHP graphing library
- JpGraph - PHP Graph Creating Library
HTML + CSS + Javascript
You can surprisingly do quite a bit with some simple HTML and CSS. You can make graphs and of course tables as well as control colors and sizes. For example, a lot of the tag clouds you see on the Web are just HTML and CSS. Throw Javascript in to the mix and you've got yourself a party i.e. interaction capabilities.
RESOURCES:
- Accessible Data Visualization with Web Standards (A List Apart)
- 8 CSS Techniques for Charting Data
- jQuery Sparklines
- Google Maps API
- Marking Up a Tag Cloud | 24 Ways
Flash/Actionscript
Flash and Actionscript is better known for animating and moving data, but it can be used for static stuff too. It's pretty good if you want to add interaction to your visualization like highlighting or filtering. I've done some stuff from scratch and also played around with Flare, the Actionscript visualization toolkit.
RESOURCES:
Microsoft Excel
It's pretty rare that I use Excel for graphics. If I need something really quick though and the data are already in an Excel spreadsheet, I'll click that graph button.
RESOURCES:
Animating the Data
There are several options to create animated and interactive data visualization, but these are the only ones I use (and for the most part, dominate what you see on the Web).
Processing
Yeah, it's called Processing. I've seen mostly designers use it, but there's no reason it can't be used elsewhere. Processing uses a canvas metaphor where you draw and make sketches and then get a Java applet out of it. Processing was created to make programmatic goodness available to non-programmers.
RESOURCES:
- Official Processing Extended Reference
- Processing: A Programming Handbook for Visual Designers and Artists
- Learning Processing: A Beginner's Guide to Programming Images, Animation, and Interaction
Flash/Actionscript
Flash and Actionscript has been my point of interest lately – mostly because the Java applet is dead as far the Web is concerned. The interactive/animated visualization you see from places like The New York Times, Stamen Design, and web applications are usually implemented with Flash and Actionscript. Not sure if it's Flash? The tell tale sign is a simple right click on whatever you're looking at. Take a look at my previous post on How to Learn Actionscript for Data Visualization for more details.
RESOURCES:
- How to Learn Actionscript for Data Visualization
- Modest Maps Display and Interaction Library
- Flare Visualization Toolkit
- TweenFilterLite by GreenSock
- ActionScript 3.0 Cookbook: Solutions for Flash Platform and Flex Application Developers
- Adobe Flex Builder 3.0
Phew, that was a lot. I started this out as a list of 10 tools and resources, and it just kept growing. I didn't realize I use so many things. It just goes to show that for any given job, there's a tool that's right and one that's not.
The amazing thing is that these are only the tools I use. There are lots of others out there. Do you use something that's not on the list to visualize data or know of another resource that would be useful?
Nice selection of resources, you may want to add Open Flash Chart, a GPL library for animated data visualization in Flash.
This is a great post Nathan, it’s given me lots of pointers to get going in this fascinating area. May I suggest you post an updated version of this annually for other newbies.
Thanks Nathan. This was perfect.
@Stuart – Good idea. I’ll take that suggestion
I was clicking through my bookmarked visualization blogs this morning thinking ” Wouldn’t it be great if there were a list of suggested programs for various visualization tasks.” Of course what is the chance that a new post would happen to be just what I am looking for. Happily enough here it is, so thanks a ton!
@JSmith – I have a side job as a mind reader. It doesn’t pay that well, but it’s gratifying.
Great article! Three more that could be added are AJAX, JSON, and Django. A great example of these put to good use are the popular LA Times interactives:
Homicide map: http://www.latimes.com/news/lo.....micidemap/
California War Dead: http://projects.latimes.com/wardead/
Nice list! I, too, enjoy the simplicity of PHP/MySQL. I’m currently studying Actionscript and Flex with the idea of using AMFPHP to glue things together…
Been putting off learning Python, maybe because Perl is so well established (or Python’s enforced indentation reminds me too much of Fortran…) Still, if Python is good enough for Google… ;)
http://sujitreddyg.wordpress.c.....ng-amfphp/
Great post Nathan. I tend to stick too much to what I know, when some other tools may be best suited. Thanks for the Illustrator tip !
Nathan, great post.
Thanks for sharing the resources and links.
From my point of view one is missing:
Whenever I have to analyze and visualize lots of data stored in a large data base quickly and efficiently, Tableau is always my favorite (www.tableausoftware.com). It is easy to learn, easy to use and produces impressive results.
@Robert – I was wondering if someone was going to mention that. They seem to have quite the following.
Thanks Nathan!
Your last couple of posts (this and the one about storytelling) have really been golden. I’m glad to see a blog that doesn’t just post neat links but also contributes some personal knowledge as well. I look forward to more.
Ps. Thanks for the helpful email. There’s still a lot I need to figure out but I think it gives me a good foundation to work on.
Nathan, great post, and thanks for the link.
I would suggest the online charting also, like ManyEyes or Swivel.
Wiow, that looks just like my teenagers room! Wow!
Jiff
http://www.privacy-center.be.tc
Crystal Reports
As someone who does this for a living, this is a poor article.
Where is SVG? The visualization standard of choice for Wikipedia?
Also, MS Excel? I don’t think so. There were a bunch of others on here that were really strange. Flash for one.
I would suggest including Splunk, which can take data from disparaging sources and allow you to export it in any format needed by various visualization packages.
sorry, Nick, you lose.. arrogance and narrow-mindedness will not get you very far..
Hey Nathan, you missed Silverlight Charts.
“If you have a particular type of (non-animated, non-interactive) statistical visualization in mind, R has probably got it”
Hmmm … it is quite easy to create animated graphics in R – generate a sequence of PNG files in a loop, then run an image processor over the files to create a movie.
Example:
#Random walks on a coloured field
n <- 100
x <- 1:n
y <- 1:n
z <- outer(sin(x/n), sin(y/n),”*”)
z <- outer(sin(4*x/n), sin(4*y/n),”*”)
image(z, col=rainbow(n))
x <- rep(0.5, 10)
y <- rep(0.5, 10)
for (i in 10:99) {
x <- x + runif(10, -0.01, 0.01)
y <- y + runif(10, -0.01, 0.01)
png(filename = paste(“c:\00\\”, i, “.png”, sep=”"), width = 480, height = 480,
units = “px”, pointsize = 12, bg = “white”, res = NA,
restoreConsole = TRUE)
image(z, col=rainbow(n))
points(x,y, pch=19, col=rainbow(100)[i], cex=3)
dev.off()
}
#Create a GIF using ImageMagick
image.magick <- “D:\\data\\bin\\iMagick\\convert.exe”
params <- “-delay 9 -loop 0 c:\00\\*.png c:\00\\animated.gif”
system(paste(image.magick, params))
@Dugite – Thanks. Using the onEnterFrame functionality is also another way to do animation in R – I just typically don’t think of R as an animation powerhouse :)
A nice list of resources to visualize and manage data. Great work!
What about the Java environment? We have problems keep data straight between Java POJOs and web services.
We started to roll our own visual organization app but went with active endpoints and their activeVOS application.
http://www.activevos.com/demo.php
saved our jobs and now I drink less coffee. :)
Thanks for posting on this! Plenty here to explore, as I’m still in the hunt for the perfect solution (aren’t we all?). I’ve been working with Inkscape a little bit (open source illustrator-type package) and it is quite robust. I’m also a big fan of SVG for final web format, but I wish it were univerally adopted (c’mon Microsoft!)
You might want to add Birdeye:
http://code.google.com/p/birdeye/
Great piece.
There’s a software tool I use that is an incredible help. AWhere has a suite of services, from inexpensive software to highly customizable SaaS tools that let you dynamically map data to learn about the geography.
I love it as a marketer, and have turned on several colleagues, who use it for a wide range of uses.
this is a great post dealing with an often difficult topic. the bad thing about visualizing data is it might make sense to you, but you need to make it immediately understandable for anyone that will use it.
Have you heard of Prefuse and it’s Flash port, Flare?
They are powerful, open source, and have beautifully designed, object-oriented APIs. Badass demos behind the links!
@Daniel – I have. Flare is actually already in the list of Flash/Actionscript resources :)
There’s a free GD-based PHP chart class library at http://pchart.sourceforge.net/ – still under development, but has some nice graphics.
Thank you for great list of resources!
You can also add Origramy flash component
It is XML powered, has JavaScript support and example for php usage. I use it in my work and find it really useful
This is a great set of tools – for power users. What about a set of simpler tools for simpler users? (I’m currently playing with DIA, which is for me complicated enough).
Also – I tend to work on problems (disaster management related) that don’t necessarily have a dataset or quantitative information. What I’m often trying to do is provide a visual explanation, for instance of a chronogram, or a plan.
And do you ever use ArcMap or similar GIS to make all those neat maps that have been popping up? Just wondering.
@Mike: For my own projects, I’ve just been using open street map for map tiles, but yeah, to get into more details and cooler maps, ArcGIS shape files would be the next step
This is a great article: I’d add Manifold GIS and the GDAL utilities in FWTools for manipulating spatial data.
Let’s not forget Microsoft Visio! The 2007 Pro version has some exciting Data Graphics capabilities that let you link data from Excel, Access, SharePoint, SQL Server or other OLEDB/ODBC data sources to Visio shapes on a drawing page.
This linked data can be be visualized as adornments attached to shapes, or even affect the appearance of shapes. The classic example being: green if sales over $100k, red if sales under $25k, but you can change the geometry of shapes too, based on data.
Visio’s ShapeSheet, which lets visualization enthusiasts create intelligent graphical elements without having to code, has been fascinating and enticing semi-technical to technical users for over a decade.
Your site is a really great resource for data visualization, and is one that I’ll be referring my Visio-fan readers too in the near future!
Wow! Impressing! Having tried some of these tools, I’m sure others are worth consideration. I also have something to add. For exaple for presenting statistical data in charts u can use Flycharts – http://flycharts.net I like it for usability.
I’ll try this tools and processes out. Thanks. Great post.
I’ll try this tools/processes out. Thanks! Great post!
How come SAS is never mentioned? It’s good for both organizing and charting.