The Virtual Dancer is an interactive dancing agent. It dances together
with the user, aligning the appropriate dance moves in real time to the
beat of the music, adapting its style to what it observes from the
user through real time computer vision. By alternating patterns of
following the user with taking the lead with new moves,
the system attempts to achieve a mutual dancing interaction. ECA, beat tracking, virtual rap dancer, audio/video input, virtual reality Some themes present in this work are :The Virtual Dancer is an interactive dancing agent. It dances together
with the user to the beat of the music. It will adapt its performance
to whatever the human user is doing, who is observed using real time
computer vision. Below some elements of the system are discussed.
A prerequisite for a virtual dancer is the ability to interpret the
music to find the beats to which the dance should be aligned. For this
we implemented the beat detection algorithm described in the
publications of Anssi Klapuri. This algorithm will detect the tempo
and beat in the music played for the dancer.
Once the beat is known, the virtual dancer should be able to dance
along. For this we constructed a database of many different dance
moves, collected using motion capturing or created manually, using
parameterized animation (see also the videos to the right of this
page). The virtual dancer selects the most appropriate dance moves
from the database, given the observations of the movement
characteristics of the human dancer. These moves are then timed to the
beat, by locally warping its timing so that the beat positions in the
move match with those in the music. For example, in a complex clapping
animation, the clap-points are aligned exactly to the predicted beat
times, so the dancer will clap to the beat of the music. The
transition from one move to the other is made using an IK-generated
stepping motion and interpolation techniques.
In the mean time, the system observes the movements of the human dance
partner using the computer vision system ParleVision.
Using several robust processors, the system extracts global
characteristics about the movements of the human dancer, such how much
(s)he moves around or how much (s)he waves with the arms. Such
characteristics can then be used to select moves from the database
that are in some way "appropriate" to the dancing style of the human
dancer.
By alternating patterns of following the user with taking
the lead with new moves, the system attempts to achieve a mutual
dancing interaction where both human and virtual dancer influence each
other. Finding the appropriate nonverbal interaction patterns that
achieve this is one of the longer term issues that is being addressed
in the context of this showcase.
|
The work on the Virtual Dancer has been presented in several videos,
some of which are downloadable from here. The first video was produced
for the recent CHI2006
interactivity session. The second movie
was a submission for GALA2005, where the system did not too bad in the
final rankings. The other movies show the process of
animating the dance motions captured in a Vicon lab and an example of
parametrized animation. Note: the movies are usually quite large
(10-30 Mb).
|
CHI 2006 Interactivity Submission (19Mb).
|
|
|
GALA2005 submission video (23Mb).
|
|
|
Motion capture and animation for the Virtual Dancer (10Mb).
|
|
|
Parameterized animation (1Mb).
|
|
|
Poster (8Mb)
|
|
Former HMI-members: |
