In the demo a set of fully working web-based prototypes developed in the context of the EU FP7 PHENICX Project (http://phenicx.upf.edu) will be presented. The Phenicx project is about innovating the classical music experience providing them with a multimodal, multi-perspective and multilayer interactive engagement, before, during and after the concert. In this demo we present some prototypes that are related with the post concert experience.

We have recorded a set of classical pieces performed by top level orchestras, including some data modalities like multi-channel audio, video, motion capture sensors, midi and text. Once all data streams have been time-synchronized, we have performed the following analysis on the data:

- Low-level and high-level audio descriptors for each individual audio source
- Description of conductor gestures based on the motion capture sensors
- Score to performance alignment
- Audio source separation
- Musical structure analysis of the performed piece

Then all these data is uploaded to the repovizz web repository (repovizz.upf.edu) that allows visualization and sharing of the data over the network. A set of customized web-based visualizations have been designed to build the prototypes that will be shown in this demo. Multimodal data streams are accessed on-line using the repovizz web API and html5 is used for the visualizations of the multimodal data and descriptors extracted from the performances. The web-audio API is used to handle the audio rendering in the client to mix between the different audio channels obtained from the different recorded audio sources or from the automatic isolation of instruments performed in the analysis step.

The visualizations available include (all web-based):

- Scrolling piano roll visualization of the musical score while audio is playing
- Orchestra layout visualization showing instrument activity and loudness while playing audio
- Audio focus to hear individual instruments playing alone
- Multi-perspective video angle selection during the concert
- 3D render of the conductor body

Here is a live example of the orchestra layout visualization including the isolation of instruments as an example of one of the prototypes that will be shown during the demo: http://repovizz.upf.edu/phenicx

Video showing some of the repovizz orchestra visualizations developed in the context of PHENICX: https://www.youtube.com/watch?v=c7pmDvkKY7A#t=168

The FP7 PHENICX project focuses on creating a new digital classical concert experience, improving the accessibility of classical music concert performances by enhancing and enriching them in novel digital ways, In this paper, we present the project’s foreseen use cases. Subsequently, we summarize initial use case feedback from two different user groups. Despite the early stage of the project, the feedback already gives important insight into real-world considerations to make for interactive music content consumption solutions.

Many recent approaches to musical source separation rely on model-based inference methods that take into account the signal’s harmonic structure. To address the particular case of low-latency bass separation, we propose a method that combines harmonic decomposition using a Tikhonov regularization-based algorithm, with the peak contrast analysis of the pitch likelihood function. Our experiment compares the separation performance of this method to a naive low-pass filter, a state-of-the-art NMF-based method and a near-optimal binary mask. The proposed low-latency method achieves results similar to the NMF-based high-latency approach at a lower computational cost. Therefore the method is valid for real-time implementations.

This work explores the multidimensional, ambiguous and temporal characteristics of tonality from a holistic perspective. The approach relies on interfacing pitch-spaces with time vs. time-scale descriptions. In this combined representation, the spatial and temporal hierarchies of tonality are evidenced simultaneously and in relation to each other. A visual exploration method is proposed for the analysis of tonal context in music works, using a simple model of tonal induction. A geometrical colouring solution, based on the topology of the pitch-space, approaches the perceptual correlation between the tonal properties and the visual representation. A relational taxonomy is proposed for describing tonal ambiguity, which leads to extending the method for the analysis of music based on tonal systems beyond the major-minor paradigm. Two perceptual studies are approached from this descriptive framework. The first study evidences the impact of time-scale in a simple model of tonal induction, and analyses the mathematical artefacts introduced by evaluations in scaled spaces. In the second study, a model of contextual instability is proposed and discussed in relation to the modelling of tonal tension. The analysis and representation methods are then generalised, through a set-class theoretical domain, in order to be applied with any pitch-based music.

Modern digital multimedia and internet technology have radically changed the ways people find entertainment and discover new interests online, seemingly without any phys- ical or social barriers. Such new access paradigms are in sharp contrast with the traditional means of entertainment. An illustrative example of this is live music concert perfor- mances that are largely being attended by dedicated audi- ences only.

This papers introduces the PHENICX project, which aims at enriching traditional concert experiences by using state- of-the-art multimedia and internet technologies. The project focuses on classical music and its main goal is twofold: (a) to make live concerts appealing to potential new au- dience and (b) to maximize the quality of concert experi- ence for everyone. Concerts will then become multimodal, multi-perspective and multilayer digital artifacts that can be easily explored, customized, personalized, (re)enjoyed and shared among the users. The paper presents the main scientific objectives on the project, provides a state of the art review on related research and presents the main chal- lenges to be addressed.

In this technical demo we present repoVizz (http://repovizz.upf.edu), an integrated online system capable of structural formatting and remote storage, browsing, exchange, annotation, and visualization of synchronous multi-modal, time-aligned data. Motivated by a growing need for data-driven collaborative research, repoVizz aims to resolve commonly encountered diculties in sharing or browsing large collections of multi-modal data. At its current state, repoVizz is designed to hold time-aligned streams of heterogeneous data: audio, video, motion capture, physiological signals, extracted descriptors, annotations, et cetera. Most popular formats for audio and video are supported, while Broadcast WAVE or CSV formats are adopted for streams other than audio or video (e.g., motion capture or physiological signals). The data itself is struc tured via customized XML les, allowing the user to (re-)organize multi-modal data in any hierarchical manner, as the XML structure only holds metadata and pointers to data files. Datasets are stored in an online database, allowing the user to interact with the data remotely through a powerful HTML5 visual interface accessible from any standard web browser; this feature can be considered a key aspect of repoVizz since data can be explored, annotated, or visualized from any location or device. Data exchange and upload/download is made easy and secure via a number of data conversion tools and a user/permission management system.

Drums modelling is of special interest in musical source separation because of its widespread presence in western popular music. Current research has often focused on drums separation without specifically modelling the other sources present in the signal. This paper presents an extensive study of the use of regularizations and constraints to drive the factorization towards the separation between percussive and non-percussive music accompaniment. The proposed regularizations control the frequency smoothness of the basis components and the temporal sparseness of the gains. We also evaluated the use of temporal constraints on the gains to perform the separation, using both ground truth manual annotations (made publicly available) and automatically extracted transients. Objective evaluation of the results shows that, while optimal regularizations are highly dependent on the signal, drum event position contains enough information to achieve a high quality separation.