Looking Beyond Sound: Unsupervised Analysis of Musician Videos

In this work, we focus on visual information conveyed by performing musicians. While musicians are playing, their movement relates to their musical performance. As such, analysis of this information can support structural characterization and timeline indexing of a recorded performance, especially in cases when such analyses are not trivially computed from the musical audio. We propose an unsupervised visual analysis method, in which visual novelty is inferred from motion orientation histograms of regions of interest. Considering our method in a case study on audiovisually recorded jam sessions, we show that our analysis of the visual channel yields promising and meaningful performance-related information, including information complementary to the audio channel.