Our current Human Music Interaction activities focus on the Virtual
Conductor.
The virtual conductor can conduct human musicians in a live
performance interactively. Using knowledge of the musical piece
(tempo, volume, the different voices, etc...) the musicians are led
through the piece and corrected when certain types of mistakes occur.
The project serves as a novel interaction context in which new aspects
of Human-Virtual Human interaction can be investigated. Virtual Conductor, Human Music Interaction, Audio Analysis, Embodied Agents Some themes present in this work are :
The virtual conductor described in this report can conduct human
musicians in a live performance interactively. The conductor can
conduct 1-, 2-, 3- or 4-beat patterns. Tempo changes can be
indicated in such a way that musicians can follow the change.
Dynamics are supported by changing the amplitude of the conducting
gestures, so that music that should be loud will make the conductor
conduct bigger and music that should be played softly will be
conducted smaller. These signals to musicians all are given before
the actual change occurs, so that musicians are prepared that the
tempo or dynamics will change. Accents are indicated by conducting
the preparation of a beat bigger. Furthermore the conductor is
able to correct the musicians when they are slowing down or speeding
up too much.
The conductor has been evaluated several times with groups of
human musicians. The musicians could follow the tempo and dynamic
changes of the conductor reasonably well. The conductor could
interact successfully with the musicians, correcting their tempo
if they played too fast or to slow.
Pieter
Bos made for his MSc thesis the first complete version of the
Virtual Conductor as it is described under "Current State".
Two students are currently working on their MSc thesis in the
context of the Virtual Conductor: Mark
ter Maat and Rob
Ebbers. Work on the VC involves various
activities, among which those described here.
To implement the right behavior in a VC we need to know how
different intentions are expressed by human conductors, and when.
To conduct effectively, the VC needs to analyze the music in real
time using algorithms implemented and/or adapted from literature.
To correct, for example, the tempo of the musicians in real time,
the movements of the conductor need not only be internally
synchronized and consistent, but also synchronized to the music as it
is being played. This needs development of animation technology,
adaptive (re)planning, synchronization to extrapolations of incoming
information, etc…
Tests with human musicians are conducted to evaluate the
interaction models and the perception technology. |
|
A recent video of the Virtual Conductor in action (.wmv, 8Mb).
|
|
|
A video presentation (.mov, 19Mb).
|
|
|
Poster (1.3Mb)
|
|
|
Foto's gemaakt tijdens de NIRICT kickoff bijeenkomst
|
|
Former HMI-members: |
