"Upbeat" - A practical test of nonverbal synchronization

Title:"Upbeat" - A practical test of nonverbal synchronization
Institute:University of Twente (HMI)
Place:Enschede The Netherlands
Type:Capita selecta and Research Topics
End date:not present
HMI ContactDennis Reidsma


The Human Media Interaction group works on the development of a Virtual Conductor. Conducting orchestras is a subtle and involved process. This is very much evident on the level of phrasing and expression, things that may still be too complicated to properly implement in a computer. However, even the basic task of 'getting the orchestra to start playing in the right tempo' is already very difficult. After all, when the conductor only signals 'now!' when the first note should sound, she is too late. The orchestra is not prepared, will start too late, and will have no idea how fast they should play.

A good conductor uses a preparatory movement, called the 'upbeat'. When properly executed, the upbeat gives the orchestra all the information they need to know: the tempo on which they must start, the moment when they will have to start playing (a bit after the upbeat), and even the expression with which they should play. A good conductor can conduct an upbeat, then stop, and still the orchestra will be able to start playing in the correct tempo.

Previous work on the HMI Virtual Conductor discussed how human conductors execute upbeats, based on literature, interviews and observation of recorded conducting sessions. But how can you translate this knowledge into a practical module for the virtual conductor that is actually able to successfully start an orchestra? It is not trivial to implement such tight synchronized behaviour in practice, nor is it trivial to perform the testing (metrics, measurements, aligned recording and playing, ...)

The goal of this Capita Selecta assignment is to execute a well designed test for different upbeat patterns in different situations to find significant conclusions about which pattern(s) works best in which situations.

To be done:

  • Design experiment, based on theoretical background and design suggestions from earlier work

  • Design good upbeat conducting patterns, grounded in available theory, with parameterizations or variations where relevant

  • Determine measurements and metrics that allow you to define what a ‘best’ upbeat pattern is and that allow you to draw quantitative conclusions about timing and tempo

  • Set up test with appropriate variations and conditions

  • Execute tests, interpret results

  • Write a Capita Selecta paper

If the assignment is carried out with a sufficient level of quality, the supervisors will assist in getting the work published in an appropriate scientific venue.