Measuring Multimodal Synchrony for Human-Computer Interaction

Title:Measuring Multimodal Synchrony for Human-Computer Interaction
Institute:University of Twente (HMI)
Place:Enschede The Netherlands
Type:Capita selecta and Research Topics
End date:not present
HMI ContactDennis Reidsma


People adapt to each other in interaction. They mirror each other’s gestures and postures; they converge in their choice of vocabulary; they do not always speak at the same time as another person; and show their mutual adaptation in many other ways. This is called entrainment or synchrony.

Synchrony is an important element in non-verbal human-human interaction. It has a role as one of the social signals that help building – and maintaining – a relation between people, be it for the duration of the conversation, or longer term. As such, it may also be a useful indicator of someone’s attitude towards the conversation, or towards their conversation partner.

Ramseyer developed a way to measure the amount of nonverbal movement synchrony (how much people move together during a conversation).

The goal of this assignment is to look closer at patterns of forming and breaking synchrony between people. To do this, we need to move from the global view of the interaction taken by Ramseyer et al. to a more local view. It is not enough to say that the overall interaction displayed significant synchrony. Instead of just looking at an overall measure for whole conversation, we must look if we can relate specific patches of high synchrony to specific conversational events in the interaction. We also want to be able to relate points with an above average high level of synchrony to the quality of interaction at these points in the interaction.

This assignment will involve looking at the literature for nonverbal synchrony, re-implementing Ramseyers method (example code is available), and looking at actual data of human conversations (both the content, and the result of the analysis method).