While doing research on the process of rebuilding New Orleans after Hurricane Katrina and the U.S. Army Corps of Engineers, I’ve run across a frequent critic close and knowledgeable watcher of the New Orleans rebuild: Robert Bea. I don’t know much about him except that he seems like a very nice man. I found this on his Berkeley homepage:

The world needs engineers who….

• whose truth cannot be bought,
• whose word is their bond,
• who put character and honesty above wealth,
• who do not hesitate to take chances,
• who will not lose their identity in a crowd,
• who will be as honest in small things as in great things,
• who will make no compromise with wrong,
• whose ambitions are not confined to their own selfish desires,
• who will not say they do it “because everybody else does it,”
• who are true to their friends through good report and evil report, in adversity as well as in prosperity,
• who do not believe that shrewdness and cunning are the best qualities for winning success,
• who are not ashamed to stand for the truth when it is unpopular, and · who have integrity and wisdom in addition to knowledge.

Please help me to be this kind of engineer.

Bob Bea

This can certainly be applied to statisticians as well. Please help me be that kind of statistician.

UPDATE: Just did some back and forth email with Professor Bea. He IS a nice man.

If I’ve learned anything in my first month at The Times, it’s that ArcGIS and Microsoft Excel are not worthless.

For a while now, since I started grad school, I had this beef against Microsoft Excel. I hated how everyone used it and how I didn’t have the money to buy the Office suite or even cared enough to want to buy it. It seemed so limited in what it could do compared to a quickly setup MySQL database.

Then last year, I took this crash course on ArcGIS. It was four days, eight hours a day of mapping. I hated ArcGIS after that workshop. The whole software suite seemed sluggish, bloated, and so not worth my time.

Today I saw some ArcGIS and Excel proficiency I had never seen before. My co-worker flew through giant spreadsheets, punched in formulas, and joined columns left and right. It was quite the scene. Once the data were prepared in Excel, she shot it over to ArcGIS. She quickly loaded a shape file for all counties in the Tri-state region, changed some limits, and voila, a few seconds later we had the map we needed. Put in some labeling, some numbers, and the graphic was complete.

Yes, ArcGIS and Excel are worthwhile.

I have so much to learn.

If you subscribe to Freakonomics, you probably already know that it’s moved to the NYTimes domain. Stephen J. Dubner and Steven D. Levitt are the blog authors, who co-wrote the book that goes along with the blog. I read the book, which dug into data and revealed a lot of interesting things like sumo wrestlers cheating and race/career correlations. Admittedly though, I totally forgot that there was a blog until I saw the ad on the NYTimes site.

I think this’ll be great to promote data awareness just as the book has. Of course, now on NYTimes, a lot more people are going to be reading the blog.

One downside though, being on the NYTimes site, it’s a limited feed, and that’s just kind of annoying. Wah, wah, wah. Yes, I like to complain sometimes.

UPDATE: I found the essay! Programmers Need To Learn Statistics Or I Will Kill Them All by Mr. Zed Shaw

There was this online essay that I read by a guy in the computer science/electrical engineering field who totally loves statistics. He read text books, and truly spoke like someone who respects data. I thought I bookmarked it, but now have no clue where the heck it is. Argh :(. If anyone knows who I’m talking about, please tell me!

He worked with a company where everyone thought they “knew” statistics. Automated reports would give them numbers, and they’d fully trust them. That was statistics to the computer engineers. Crunch some numbers and see what the software gives me. As a result, these engineer-types really pissed off the author of the article. Read More

I recently put together a timeline for Senator Hillary Rodham Clinton’s changing views on the Iraq war. In 2002, she voted in support of the war. In 2006, her language was a bit non-committal, as far as setting a deadline to get troops out of Iraq. Now, in 2007, she’s firmly set on getting troops out of Iraq by some deadline. The goal of the timeline is to show this change.

Here’s the important lesson I learned during this task — even though it’s easy to put a timeline together, it still has to tell a story. Think about the purpose of the timeline. Usually, you want to show some change or progression over time. The tinting on the above timeline is for events during which Senator Clinton shows a definite change in her stance. The hope is that the reader keeps going left to right.

If you don’t keep the story in mind, the timeline is no longer as useful. It’s just a bunch of text arranged in time order, which is sort of what the above timeline looked like after my first jab at it. I put tinting on the events i.e. the things that weren’t quotes from Mrs. Clinton. In retrospect, such tinting plainly defeats the purpose of this particular timeline, which went with a story that discussed the change. Duh.

I made this graphic early last week, or actually, maybe it was during my first week. In any case, they finally ran the story, and my graphic is on the front page of The Times Online (as of 1:39am Eastern time).

You can read the article. It’s pretty interesting. In a nutshell, prisons are getting crowded, so states are shipping inmates to out-of-state private prisons. For example, California is sending prisoners all the way to Wheelright, Kentucky.

Jorge Luis Borges wrote this really good fictional short story in 1944 called Funes, the Memorious. It’s about a boy, Funes, who isn’t incredibly bright until one day he falls off his horse and hits his head. After the accident Funes has finds that he suddenly has an amazing memory with which he remembers every single detail of every moment in his life.

His memory is so vivid that at one point he sees a dog, and a moment later the dog seems different. Funes remembers the way each hair stood on the the dog’s back, the direction of the breeze, what direction the dog’s tail was pointed, the perspiration on his own body, where everyone else was, etc. That dog could not possibly be the same dog that he saw a moment ago.

Funes not only remembered every leaf on every tree of every wood, but even every one of the times he had perceived or imagined it. He determined to reduce all of his past experience to some seventy thousand recollections, which he would later define numerically. Two considerations dissuaded him: the thought that the task as interminable and the thought that it was useless.

Trying to Remember Too Much

At this day and age, when so much of everything is stored in database and everything is logged, is it possible to remember too much? Technology has enabled us to surveil others, video tape every moment of our life, store every email, take a seemingly endless river of pictures, record conversations, and log data out the wazoo.

Sure, it’s great to have it, but what use can you make of a year’s worth of data? What about ten years? Or dare I say, a century’s worth of data?

This is when visualization becomes important. It’s our duty to make the ocean of data available without letting the ocean’s never-ending vastness overwhelm the data explorers. Otherwise, our technological memory becomes like that of Funes’, and all is lost. OK, cue the dramatic music… now.

Admittedly, ever since the Spring quarter ended, I’ve either been preparing for my internship at The Times or have been occupied by the internship. I haven’t given much thought to my dissertation topic, which in the most vaguest of terms will somehow encompass three things:

• Social Data Visualization
• Eco-Visualization
• Visualization of my Life

I have yet to figure out how to tie the three together in a worthwhile way or even whether I will include all three. Wrapped around the three will be data sharing. I got to thinking a little bit about visualizing my life in data today.

My adviser forwarded me this info design piece, by Gregory Dizzia (which was apparently also featured on infosthetics):

First off, this is a cool piece. If you haven’t seen it, go to the site and download the pdf. It’s a simple idea. Document past relationships — how they began, how they ended, what happened in between. The information is organized very well. At a glance, you can see how many relationships Greg has had in his life and all the one night stands he had after his mid-life, long-term relationship. The design is attractive and I could relate to the information, so I was drawn in to look more.

Dig a little deeper, and you’ll see that there’s not just one engagement ring during that long-term relationship with Sarah. There’s a second one during his very first girlfriend, Megan. Although, I’m a little wary of calling Megan a girlfriend since it was during Greg’s tender years at age 9 to 11. Stuff like that makes me want to know more.

Was he really engaged? Was it an arranged marriage or something? What do those breakup symbols really mean?

Life Visualization Appeal

Right off, Greg’s piece drew me in, because (1) it was pretty, and (2) I could relate to the data, and (3) there was a very human factor. This could probably be generalized to all types of successful visualization, but (2) and (3) are, I think, synonymous with life viz. That’s two out of three things that are automatic. Plus, as the visualizer I have a very strong emotional attachment to the data.

NOW, what happens when we have 100 people’s relationships to visualize? 1000? That’s when it gets really interesting and social data visualization makes its way into the picture. Well, something to think about.

Tired of looking at my New York Times graphics yet? Too bad. Here’s another one for my your viewing pleasure.

CUNY schools are planning to raise their SAT math scores to 510 for their top-tier schools and to 500 for the rest. Believe it or not, the current cutoff for all schools is 480. Some say the increase in standards is good for the school to improve reputability. Others argue that the new cutoffs single out a lot of minorities since the high school education system is uneven.

Currently, lots of students are coming into CUNY schools unprepared to take college-level math courses, and the college ends up teaching remedial courses like pre-algebra. That’s just SAD. It’s probably more important to focus on improving the high school education system than it is to try to get unqualified students into college.

I had a chance to browse through some of my subscribed feeds today, and I saw a post called Noisy Subways by Kaiser over at Junk Charts blog. So I clicked, since it isn’t one of those full feeds, and then I saw The New York subway report card. I smiled, because, well, I made that chart just a few days ago!

Just a disclaimer: The Times chart was just The New York Times version of the original Straphangers report:

Anyways, there was bit of a discussion, which again, I found very amusing. I felt kind of special in a way.

There were two main points to the post – 1. Noisy data; and 2. Chart is hard to read. I’m very tired right now, so I’ll just say a few things.

Yes, the data is really noisy, but why shouldn’t it be? We shouldn’t assume that all six variables are positively correlated. It’s very possible for a line to be very reliable, but have no seats. One could argue that the lines with more people HAVE to be more reliable, because if something goes wrong, more people are going to get screwed.

Secondly – sure, the chart is a bit hard to read at a glance, but who’s the audience? New Yorkers are the audience, and the first thing that they’re going to do is look for their subway line. That’s what I did. With the audience in mind, I think the chart serves its purpose.

Most of the commenters provided decent ideas for alternative graphics. My opinion is that with this kind of data, it’s up for grabs. Audience is key though for charts, graphs, plots, maps, etc in a newspaper. Spiders and whiskers won’t make sense to many people. You’d be amazed of how many people don’t know how to read a scatter plot. The public is getting better though. They’ll get there.

As for the person who left the comment about the gaps in the chart. I’m going to assume that was in haste. Some lines are tied, hence some blanks spaces.

Welp, that was fun. Yawwwwn. Time for bed.

My second graphic was in The Times Metro section today (Tuesday, July 24, pg B2). It’s an annual report card compiled by the Straphangers Campaign for every New York subway line. The No. 1 line was coincidentally ranked best while the C and the W (one of the lines I take) were near the bottom.

This is just really amusing to me. Above is a bar plot, from Google Reader, of the number of items I’ve read in the past 30 days, with each bar representing a day. Quite easy to see when I had a little bit too much time on my hands. Right when the internship starts, the number of items read plummets. I miss my subscribed feeds =(.

It’s six days in, and I’m starting to get used to Adobe Illustrator. It’s one honker of a program, so I’m picking up things as I go along, but on the upside, I’m really glad I went through some of Illustrator lessons to at least familiarize myself with layers, etc.

I think I’m getting closer to the point where it’s less about “How do I do this?” and more about “What am I going to show?” Don’t get me wrong. There’s A LOT I still don’t know how to do, but at least I know enough to figure out a good amount on my own. Just a lot to figure out about The Times graphics style — font, sizes, color, etc.

The administrative stuff is the hardest part of all though. While I’m working on a graphic I have to keep all the necessary people updated i.e. the reporter of the story of whom I am making the graphic for. I got scolded today, because a reporter didn’t know that her story was put on hold. I didn’t know that I was her only contact link. Lesson learned. I’m just going to contact everyone from now on. Better to provide too much information than too little (in this case, at least).

Once a graphic is completed, I have to print out five copies or so and hand them out to all of the necessary people. Next, update the graphic schedule, and then place it in the active list. It’s strange that even though we’re all equipped with these super awesome computers that I still have to walk upstairs and hand-deliver copies of a graphic. I guess nothing can replace human contact.

and here it is. Floyd Norris has a weekly editorial called Off the Charts. This week’s was titled A Blockbuster Seller Overseas: Stakes in Corporate America. It’s about the increase in the amount of money foreign investors are putting into American businesses. Check out the Business section in The New York Times! There’s also an online version.

It was ridiculous how many changes had to be made to my first pass at the graphic. I suck (but I’m getting better, really!). Color, bar width, grid style, font size, axis size, alignment, plus and minus signs, spacing, and area fill.

While writing statistical reports for class, I think it was easy to get away with so-so graphics. Just plug some data into R, use the plot function, and ta-da. I’ll never look at charts and graphs the same way again.

Every day I learn a lot, and every day I get better. For most of the day today, I worked on a single graphic (that hopefully runs in the paper). I gave it to the person in charged, and oh man, there was a lot to change. Fonts, labels, fill colors, bar widths, spacing, layer orientation, size… on and on and on. I think it might have been faster for him to make the graphic himself than it was for him to fix mine.

Sigh. Gotta practice.

The graphic above is the number of daily steps I’ve taken since I started wearing a pedometer. Can you tell when I moved to the city and was forced to walk to the subway and work?

So I started on a graphic today, and it might actually be in the paper. I’m pretty excited to see my first graphic published. I won’t say what it is or when until I actually find out if it gets published to save myself from any embarrassment, but nevertheless, cool to think about.

In other news, I got to see a coworker do his stuff with some mapping and what not, whizzing through Adobe Illustrator like it was part of him. His attention to detail and his ability to do it quickly were what impressed me the most. I have a lot to learn and a lot to do.

It’s also comforting to know that previous interns didn’t know any Adobe Illustrator either, so I don’t feel as dumb anymore.

Ok, so I said I would update after my first day, but I got home around 8p, and was just too tired to do anything. I forgot what it’s like to continuously work for 8 hours a day. It’s 9:30p right now, and honestly, I’m ready to pass out.

My first two days have been interesting, and it’s very clear that I have a lot to learn, in terms of visualizing graphics. I have yet to create an actual graphic. Rather I’ve been sifting through Iraq data, looking for stuff that’s interesting, and then putting it in a spreadsheet. I was also given an article to read that could possibly benefit from a graphic, but it turns out that the writer already got some amazing photos and a small map, so that was a no-go.

I think my strength is R, so I’m going to try to improve (um, learn) mapping in R. I think if I can do that, I will be of much more help. I was asked today if I could do maps, but unfortunately, I’ve only done very basic things, and the task was time-sensitive. Sigh.

Am I rambling? I feel like I’m rambling. I need to sleep. But I need to learn R. Alright then.

On a completely random note, I was able to see Matthew Carter, a master font designer, who has designed fonts like Georgia and Verdana, speak today. I never really put much thought into type faces, but wow, there’s a whole lot that goes into it. A lot of subtleties that involve making more text fit on a page without cluttering or techniques to make text easier to read on a newspaper or from a magazine.

Oh yeah, I also saw my idols — the IBM Visual Communications lab. I didn’t get a chance to talk to them though, but they looked like a friendly bunch.

In honor of my New York Times induction day, a visualization of The Wealthiest Americans Ever. You think good ol’ Billy would be there at the top of the list, worth $82 billion, there have been a few who have preceded the software giant e.g. John D. Rockefeller worth a crazy $192 billion. Just think how many Jack in the Box tacos you could buy with that kind of money.

Two months flew by in a hurry in between Spring quarter ending and my New York Times internship beginning. I arrived in New York City today and I’m starting at the times tomorrow. For the next 10 weeks, I’m going to be working in the graphics department, and I anticipate it’s going to have to be a pretty steep learning curve.

I don’t really know what I’m going to be doing yet, so I’ve familiarized myself with Adobe Illustrator and Flash and tried to pick up a few more skills in R. Originally, I was thought I was going to become some kind of expert in Illustrator and/or Flash, but after some time with the books, I’ve got a long a road ahead. Not being so hot in either Illustrator or Flash, I’m a little nervous, but I guess I’ll just wait and see.

Updates on my first day tomorrow.

Yes, more mapping. Map, map, map. amMap offers a Flash-based mapping tool that you can download and customize to your liking.

Ammap is an interactive flash map creation software. Use this tool to show locations of your offices, routes of your journeys, create your distributor map. Photos or illustrations can be used instead of maps, so you can make different presentations, e-learning tools and more.

There’s some smooth browsing and zooming, and it’s pretty sleek. Those who appreciate simplicity will appreciate amMap. Plus, it’s free :) Read More