hmilogo
HUMAN MUSIC INTERACTION

Short description:

Our current Human Music Interaction activities focus on the Virtual Conductor. The virtual conductor can conduct human musicians in a live performance interactively. Using knowledge of the musical piece (tempo, volume, the different voices, etc...) the musicians are led through the piece and corrected when certain types of mistakes occur. The project serves as a novel interaction context in which new aspects of Human-Virtual Human interaction can be investigated.

Keywords:

Virtual Conductor, Human Music Interaction, Audio Analysis, Embodied Agents

Research Themes:

Some themes present in this work are :

Description

CURRENT STATE

The virtual conductor described in this report can conduct human musicians in a live performance interactively. The conductor can conduct 1-, 2-, 3- or 4-beat patterns. Tempo changes can be indicated in such a way that musicians can follow the change. Dynamics are supported by changing the amplitude of the conducting gestures, so that music that should be loud will make the conductor conduct bigger and music that should be played softly will be conducted smaller. These signals to musicians all are given before the actual change occurs, so that musicians are prepared that the tempo or dynamics will change. Accents are indicated by conducting the preparation of a beat bigger. Furthermore the conductor is able to correct the musicians when they are slowing down or speeding up too much.

The conductor has been evaluated several times with groups of human musicians. The musicians could follow the tempo and dynamic changes of the conductor reasonably well. The conductor could interact successfully with the musicians, correcting their tempo if they played too fast or to slow.

ACTIVITIES

Pieter Bos made for his MSc thesis the first complete version of the Virtual Conductor as it is described under "Current State".

Two students are currently working on their MSc thesis in the context of the Virtual Conductor: Mark ter Maat and Rob Ebbers. Work on the VC involves various activities, among which those described here.

Analyzing Conducting by Humans

To implement the right behavior in a VC we need to know how different intentions are expressed by human conductors, and when.

Music Perception Technology

To conduct effectively, the VC needs to analyze the music in real time using algorithms implemented and/or adapted from literature.

Multi Modal Generation

To correct, for example, the tempo of the musicians in real time, the movements of the conductor need not only be internally synchronized and consistent, but also synchronized to the music as it is being played. This needs development of animation technology, adaptive (re)planning, synchronization to extrapolations of incoming information, etc…

Evaluation with Human Musicians

Tests with human musicians are conducted to evaluate the interaction models and the perception technology.

MEDIA

A recent video of the Virtual Conductor in action (.wmv, 8Mb).
A video presentation (.mov, 19Mb).
Poster (1.3Mb) Poster presentation
Foto's gemaakt tijdens de NIRICT kickoff bijeenkomst Foto's gemaakt tijdens de NIRICT kickoff bijeenkomst

Publications related to this showcase are:

P. Bos, D. Reidsma, Z.M. Ruttkay and A. Nijholt Interacting with a Virtual Conductor, in Proc. of 5th International Conference on Entertainment Computing, R. Harper, M. Rauterberg and M. Combetto (eds), Lecture Notes in Computer Science, volume 4161, Springer Verlag, Berlin, ISBN 3-540-45259-1, ISSN 0302-9743, pp. 25-30, 2006 [ BiBTeX ] [EprintsDOI>  Download PDF document
P. Bos Interacting with a Virtual Conductor, mastersthesis, 2007 [ BiBTeX ] [EprintsDownload PDF document
A. Nijholt, D. Reidsma, R.M. Ebbers and M. ter Maat The Virtual Conductor: Learning and Teaching about Music, Performing, and Conducting, in Proceedings Learning technologies in the Information society. The 8th IEEE International Conference on Advanced Learning Technologies, IEEE Computer Society, Los Alamitos, ISBN 978-0-7695-3167-0, pp. 897-899, 2008 [ BiBTeX ] [EprintsDOI>  Download PDF document
M. ter Maat, R.M. Ebbers, D. Reidsma and A. Nijholt Beyond the Beat: Modelling Intentions in a Virtual Conductor, in INTETAIN '08: Proceedings of the 2nd international conference on INtelligent TEchnologies for interactive enterTAINment, ACM Digital Library, ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), Brussels, Belgium, ISBN 978-963-9799-13-4, pp. 12-21, 2008 [ BiBTeX ] [Eprints[Official URL]  Download PDF document
D. Reidsma, A. Nijholt and P. Bos Temporal Interaction between an Artificial Orchestra Conductor and Human Musicians, in BNAIC 2008, Proceedings 20th Belgian-Netherlands Conference on Artificial Intelligence, A. Nijholt, M. Pantic, M. Poel and G.H.W. Hondorp (eds), University of Twente, Enschede, the Netherlands, ISSN 1568-7805, pp. 405-406, 2008 [ BiBTeX ] [EprintsDownload PDF document
D. Reidsma, A. Nijholt and P. Bos Temporal interaction between an artificial orchestra conductor and human musicians, Computers in Entertainment, 6(4):1-22, ISSN 1544-3574, 2008 [ BiBTeX ] [EprintsDOI>  Download PDF document

HMI-members working on this showcase are:

Former HMI-members:

old Parlevink website   colophon   [Back] .