Exploiting Instrument-wise Playing/Non-Playing Labels for Score Synchronization of Symphonic Music

Score synchronization with an audio-visual recording of a symphonic music performance is usually done by solving an audio-to-MIDI alignment problem. In this paper we investigate what role visual channel can have in this process. In particular, we focus on the possibility to represent both the score and the performance by the information about what instrument is active at a given time stamp. More specifically, we investigate to what extent instrument-level 'play' (P) and 'non-play' (NP) labels are informative in the synchronization process. After introducing the P/NP-based representation of the music piece, both at the score and performance level, we define an efficient way of computing the distance between the two representations, which serves as input for the synchronization step based on dynamic time warping. In parallel with assessing the effectiveness of the proposed representation, we also study its robustness when missing and/or erroneous labels occur. Our experimental results show that P/NP-based music piece representation is informative for performance-to-score synchronization and may benefit the existing audio-only approaches.