A different analytical wall

Posted to Statistics  |  Nathan Yau

In reference to the wall between reporting data and understanding it, Martin Theus proposes a different one:

Once you start to explore the data, the whole thing stops to be linear but gets to be very iterative, jumping over the wall every now and then. I.e., you may find out that the data cleaning is insufficient, or the model you have in mind needs some other transformation of the data, or you might want to collect additional or other data altogether.

The wall does exist, but I think it is more separating two kinds of people / thinking.

Theus finishes:

One thing is for sure: we won’t succeed if analysts continue to build useful but technically insufficient tools and computer scientists still build fancy tools that merely help the analysts.

Or even better: analyst and tool builder become the same person. That’ll take much longer though, so communication is a good place to start.

[Theusrus]

2 Comments

  • > Or even better: analyst and tool builder become the same person.
    >
    That is a good approach, but the commonly used tools come from mega companies like IBM, SAS or SPSS, having loads of programmers who never spoke to a real user, and sales personal who care for their bonus more than the customer’s needs.

    There is hope though, when you look at the rise of R. High quality and managed open source projects might be the solution for the future.

  • Business Systems Analysts come close to a resource who can bridge the gaps between, business, data, and tool requirements. Truth is, every environment is different. In one environment an MS Excel solution where the most complex feature is advanced filtering may suffice. Others may demand pivot table functionality. Others may need a cube containing aggregates from a 50,000,000 data source – and drill through to boot. Obviously that’s the reporting side. Advanced analytics is another story.

    The reporting/analytics domain requires experienced resources who learn quickly, are passionate, can communicate, ‘understand’ how tools work, and not be afraid of jumping in to learn a new technology when required.

Favorites

The Best Data Visualization Projects of 2011

I almost didn’t make a best-of list this year, but as I clicked through the year’s post, it was hard …

Divorce Rates for Different Groups

We know when people usually get married. We know who never marries. Finally, it’s time to look at the other side: divorce and remarriage.

Where Bars Outnumber Grocery Stores

A closer look at the age old question of where there are more bars than grocery stores, and vice versa.

Watching the growth of Walmart – now with 100% more Sam’s Club

The ever so popular Walmart growth map gets an update, and yes, it still looks like a wildfire. Sam’s Club follows soon after, although not nearly as vigorously.