How do you compare music visually? You can break it down into data by quantifying the notes, volume, etc and then visualize it with timescapes (above). The horizontal axis represents musical time, from the beginning to end of a piece. Large blocks show similarities to other pieces and smaller noisy chunks show more “fleeting” similarities.
Researches from the Centre for the History and Analysis of Recorded Music (CHARM) use these timescapes to highlight the work of Joyce Hatto, who was described by Richard Dyer of the Boston Globe as “the greatest living pianist that almost no one has ever heard of.”
The discussion that follows is kind of over my head, but it’s worth a look for the methods themselves.
The above is timescape for Arthur Rubinstein’s 1939 recording of Chopin’s Mazurka Op. 68 No. 3. Notice the big chunks in orange and black representing some musical sameness to his own recording in 1966.
This apparently, isn’t all that surprising, but then compare Hatto’s recording of Maurka Op. 68 No. 3 to Eugen Indjic’s (who I am guessing is recognized as an accomplished pianist). They are virtually indistinguishable.
Is this the picture of greatness? (Update: or fraud? Read the comments below.)
[Thanks, Remzi]
How do you interprete this ? If you cannot why then create it ? For the mere sake of beauty ? But if Infographics is, as the term says, about informing then you should change the name. You might prefer beautygraphics, or playgraphics.
@Frank – I think this is one of those tools that is much more useful if you’re an expert in the area, in this case music.
So this relates to a scandal in classical music circles a couple years ago, when it was discovered a couple of months after Hatto’s death that many of her recordings were actually recordings made by other artists that had been digitally manipulated and then published by her husband as her own playing. These graphics make the point clear.
Wow, they really wrote that article to be as incomprehensible as possible. Aziridine is right about the Hatto controversy, but I believe the issue was that they were NOT digitally manipulated: I heard at one point that GraceNote actually identified tracks played “by” her as their originals and that was part of the issue.
The article from CHARM seems to be trying so hard to not make accusations against Hatto that they mislead people about what the actual controversy was.
hmm, although wikipedia claims that there was dig. manip., so what I heard might be wrong. I posted this on my blog here:
http://noiseforairports.com/post/332334096/from-the-centre-for-the-history-and-analysis-of
The article is pussyfoots around the issue of fraud to an almost ridiculous extent.
As some other commenters have pointed out, what the article is really saying is that they’re pretty certain that the recordings attributed to Hatto and Indjic are the same recording – that the Hatto release was probably not her playing at all, but rather a re-release of the Indjic under her name, but changed to be at a slightly different tempo.
It seemed pretty obvious to me that the point of the article is to challenge Joyce Hatto’s credibility. Besides the timescapes that demonstrate extreme similarity between pieces and the Pearson correlations that demonstrate her recordings are more similar to the same pieces placed by Indjic than an exact reissue, the authors conclude with this:
“We have documented the similarities between CACD 20012 and Calliope 3321, which in our opinion demonstrate beyond reasonable doubt that, despite the packaging, both recordings are of the same performance (or set of performances).”
So, CACD 20012 (Hatto) and Calliope 3321 (Indjic) are the same, albeit at a slightly different tempo.
According to the article, timescapes show a visual representation of the similarity of musical pieces without regard to tempo. So Nick, the authors are just claiming that the Hatto pieces are previous recordings at a different speed.
The issue I think people are having is not the conclusion of the article—as you quote, the basics are clear (although I don’t think they explicitly say that the tempo differs).
The issue is the very strange way the article presents its findings and “pussyfoots around the issue of fraud,” as Chris said.
Is part of the problem one of musical semantics? And/or one of academic remove within discussions of classical music? I think the article comes across as a pretty straightforward investigation of the situation, but it does use language that is typical of the genre and avoids direct confrontation (while assuming the reader knows about the underlying controversy). If a tabloid or media expose were published on this, it would probably use more directly confrontational language. Perhaps we’ve become inured to that way of writing.
Yeah, they do mention explicitly the issue of speed. “they play at slightly different speeds; in the case of Op. 17 No. 4 the Hatto version plays about 0.7% slower, whereas in the case of Op. 68 No. 3 it is 2.8% slower—and in the case of another mazurka, Op. 24 No. 2, it is 1.2% faster. (Because our timescapes are based on relative timing, they show the similarities between the recordings despite these changes.)”
I don’t agree that they ‘pussyfoot around.’ They are music researchers. I thought they did a nice job of presenting their argument, evidence, and conclusion.
wow, thanks for the background all.
England’s libel laws might have something to do with the vagueness of the article.
From what I remember the guy only ever admitted patching recordings to cover-up his wife’s moans of pain. Not wholescale copying..
Alas, it was wholesale fraud, which tainted the modest reputation of Joyce Hatto’s earlier recordings.
(The story is on The Guardian’s archive:
http://www.guardian.co.uk/culture/2009/jul/31/the-great-piano-scam-review and http://www.guardian.co.uk/uk/2007/feb/27/musicnews.music)
The ultra-academic tone of the article shouldn’t surprise much: it was, after all, produced by an musicological think tank whose participants were drawn from several universities and research groups. Perhaps it was a sop to the music world’s sensitivities that they rehearsed the entire background of Joyce Hatto’s life in such detail and ended the article in such studied neutrality.
Once they got to discussing their analytical method, with examples showing how Rubenstein’s own performances of the same piece could vary over time, and how only a direct reissue of a recording could be virtually identical to another, I thought the case was closed.
The visualizations demonstrated that her performances were impossibly identical to someone else’s, with “the small markings at the bottom of [the figures] merely reflect[ing] limitations in the accuracy of data capture.”
On another page on the CHARM website, they state that, in reference to the visualization techniques highlighted by Nathan:
“An unanticipated outcome of this work was the discovery that the Concert Artists recording of the Mazurkas issued in the name of Joyce Hatto was in fact a slightly modified version of the recording by Eugen Indjic; this was the first proof of the infamous Hatto hoax, which became generally known in February 2007.”
http://www.charm.rhul.ac.uk/projects/p2_3.html
That seems fairly unambiguous.
Also, ignoring the hoax, CHARM has a page that discusses the visualization tools themselves:
http://www.charm.rhul.ac.uk/analysing/p9_4.html
They explain the triangular visualizations, created by Craig Snapp at Stanford, as follows:
“Sapp developed a means of representing harmonic structure through what he called keyscapes: triangular figures in which the base corresponds to the moment-to-moment course of the music, with successively higher layers showing averaged values based on the layer below, with the apex of the triangle consisting of a single value representing the average value for the piece as a whole. The result is to provide a visual impression of the strength of particular hamonies at particular points in the music at both a local and global level.”
That makes sense to me, and seems useful. And quite possibly applicable to other fields besides music.
Craig Sapp: http://ccrma.stanford.edu/~craig/
Two papers of his on this keyscape technique and its applications:
http://portal.acm.org/citation.cfm?id=1095534.1095544
https://ccrma.stanford.edu/~craig/papers/01/
It’s not clear to me why the graphs are triangular.
From one of the links above:
“During his doctoral studies at Stanford, Sapp developed a means of representing harmonic structure through what he called keyscapes: triangular figures in which the base corresponds to the moment-to-moment course of the music, with successively higher layers showing averaged values based on the layer below, with the apex of the triangle consisting of a single value representing the average value for the piece as a whole.”
Yeah, I don’t completely get it either… maybe if I knew more about music..
They don’t have to be triangular; it just makes the most sense.
CHARM has another page which describes the visualization techniques more fully: http://www.mazurka.org.uk/ana/timescape/
If you read that, you’ll see that these triangular plots are means of visualizing the correlations of all the subsequences of two (or more) sequences of data. The data being compared in the original article are analyses of musical recordings (not the recordings themselves) — things like tempo, phrasing, harmonic structure, etc.
The bottom of each triangle shows how the analysis of the first piece compares to the analysis of the second piece (or the analyses of the other pieces, depending on the total number involved), by comparing the smallest possible subsequences. If two piece analyses are being compared, then the colors represent the degree of correlation between them; if one piece analysis is being compared to many, then the colors represent which piece best correlates to the first.
The second row, working up, compares slightly larger subsequences, and represents the results in color in the same way. The third row up again compares larger subsequences, and so on, until you reach the top row (the apex of the triangle), in which case the comparison is between one entire sequence (the largest possible subsequence, in other words) and another.
If you represent each comparison as one block of color, then you have a decreasing number of blocks per row, moving upwards, and thus a triangle makes sense.
The page linked to above also contains these “hierarchical correlation plots” in bell and rectangular shapes, but I’m not sure why you’d want to use either form. The correlation plot seems to naturally be a triangle.
They even provide an online tool to generate your own “scape plots”: http://www.mazurka.org.uk/software/online/scape/
Of course, I just read all this now, so no promises about my accuracy. :)
This reminds me of visual binary diffs (PDF) Dan Kaminsky produced for a talk he did a while back, but he used rectangles.
My mistake, the author of the original paper is Jonathan Helfman.
Pingback: Weekly Roundup: Design Related Links #2 « Discovery Session… by Gerard Dolan