Our current Human Music Interaction activities focus on the Virtual
The virtual conductor can conduct human musicians in a live
performance interactively. Using knowledge of the musical piece
(tempo, volume, the different voices, etc...) the musicians are led
through the piece and corrected when certain types of mistakes occur.
The project serves as a novel interaction context in which new aspects
of Human-Virtual Human interaction can be investigated.
Virtual Conductor, Human Music Interaction, Audio Analysis, Embodied AgentsSome themes present in this work are :
The virtual conductor described in this report can conduct human
musicians in a live performance interactively. The conductor can
conduct 1-, 2-, 3- or 4-beat patterns. Tempo changes can be
indicated in such a way that musicians can follow the change.
Dynamics are supported by changing the amplitude of the conducting
gestures, so that music that should be loud will make the conductor
conduct bigger and music that should be played softly will be
conducted smaller. These signals to musicians all are given before
the actual change occurs, so that musicians are prepared that the
tempo or dynamics will change. Accents are indicated by conducting
the preparation of a beat bigger. Furthermore the conductor is
able to correct the musicians when they are slowing down or speeding
up too much.
The conductor has been evaluated several times with groups of
human musicians. The musicians could follow the tempo and dynamic
changes of the conductor reasonably well. The conductor could
interact successfully with the musicians, correcting their tempo
if they played too fast or to slow.
Bos made for his MSc thesis the first complete version of the
Virtual Conductor as it is described under "Current State".
Two students are currently working on their MSc thesis in the
context of the Virtual Conductor: Mark
ter Maat and Rob
Ebbers. Work on the VC involves various
activities, among which those described here.
To implement the right behavior in a VC we need to know how
different intentions are expressed by human conductors, and when.
To conduct effectively, the VC needs to analyze the music in real
time using algorithms implemented and/or adapted from literature.
To correct, for example, the tempo of the musicians in real time,
the movements of the conductor need not only be internally
synchronized and consistent, but also synchronized to the music as it
is being played. This needs development of animation technology,
adaptive (re)planning, synchronization to extrapolations of incoming
Tests with human musicians are conducted to evaluate the
interaction models and the perception technology.
A recent video of the Virtual Conductor in action (.wmv, 8Mb).
A video presentation (.mov, 19Mb).
Foto's gemaakt tijdens de NIRICT kickoff bijeenkomst
|P. Bos, D. Reidsma, Z.M. Ruttkay and A. Nijholt Interacting with a Virtual Conductor, in Proc. of 5th International Conference on Entertainment Computing, R. Harper, M. Rauterberg and M. Combetto (eds), Lecture Notes in Computer Science, volume 4161, Springer Verlag, Berlin, ISBN 3-540-45259-1, ISSN 0302-9743, pp. 25-30, 2006 [ BiBTeX ]  |
|A. Nijholt, D. Reidsma, R.M. Ebbers and M. ter Maat The Virtual Conductor: Learning and Teaching about Music, Performing, and Conducting, in Proceedings Learning technologies in the Information society. The 8th IEEE International Conference on Advanced Learning Technologies, IEEE Computer Society, Los Alamitos, ISBN 978-0-7695-3167-0, pp. 897-899, 2008 [ BiBTeX ]  |
|M. ter Maat, R.M. Ebbers, D. Reidsma and A. Nijholt Beyond the Beat: Modelling Intentions in a Virtual Conductor, in INTETAIN '08: Proceedings of the 2nd international conference on INtelligent TEchnologies for interactive enterTAINment, ACM Digital Library, ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), Brussels, Belgium, ISBN 978-963-9799-13-4, pp. 12-21, 2008 [ BiBTeX ]  [Official URL] |
|D. Reidsma, A. Nijholt and P. Bos Temporal Interaction between an Artificial Orchestra Conductor and Human Musicians, in BNAIC 2008, Proceedings 20th Belgian-Netherlands Conference on Artificial Intelligence, A. Nijholt, M. Pantic, M. Poel and G.H.W. Hondorp (eds), University of Twente, Enschede, the Netherlands, ISSN 1568-7805, pp. 405-406, 2008 [ BiBTeX ]  |