A different analytical wall

October 14, 2010

Topic

Statistics

In reference to the wall between reporting data and understanding it, Martin Theus proposes a different one:

Once you start to explore the data, the whole thing stops to be linear but gets to be very iterative, jumping over the wall every now and then. I.e., you may find out that the data cleaning is insufficient, or the model you have in mind needs some other transformation of the data, or you might want to collect additional or other data altogether.

The wall does exist, but I think it is more separating two kinds of people / thinking.

Theus finishes:

One thing is for sure: we won’t succeed if analysts continue to build useful but technically insufficient tools and computer scientists still build fancy tools that merely help the analysts.

Or even better: analyst and tool builder become the same person. That’ll take much longer though, so communication is a good place to start.

[Theusrus]

2 Comments

  • Martin — October 14, 2010 at 10:03 am

    > Or even better: analyst and tool builder become the same person.
    >
    That is a good approach, but the commonly used tools come from mega companies like IBM, SAS or SPSS, having loads of programmers who never spoke to a real user, and sales personal who care for their bonus more than the customer’s needs.

    There is hope though, when you look at the rise of R. High quality and managed open source projects might be the solution for the future.

  • Mike Smith — October 31, 2010 at 2:23 pm

    Business Systems Analysts come close to a resource who can bridge the gaps between, business, data, and tool requirements. Truth is, every environment is different. In one environment an MS Excel solution where the most complex feature is advanced filtering may suffice. Others may demand pivot table functionality. Others may need a cube containing aggregates from a 50,000,000 data source – and drill through to boot. Obviously that’s the reporting side. Advanced analytics is another story.

    The reporting/analytics domain requires experienced resources who learn quickly, are passionate, can communicate, ‘understand’ how tools work, and not be afraid of jumping in to learn a new technology when required.