Project Name: Augmented Multi-party Interaction
January 1, 2004
December 31, 2006
"AMI is concerned with new multimodal technologies to support human
in the context of smart meeting rooms and remote meeting assistants.
aims to enhance the value of multimodal meeting recordings and to make
interaction more effective in real time. These goals will be achieved
new tools for computer-supported cooperative work and by designing new
search and browse meetings as part of an integrated multimodal group
captured from a wide range of devices."
[See also the
official web page where this text came from].
The AMI project is about multi modal, multi party interaction. IDIAP
(Martigny, Switzerland), TNO and the University of Edinburgh host
rooms that contain an extensive setup of microphones and camera's. In
rooms the interactions between people during meetings are recorded.
are annotated for many modalities. Examples of these
annotations are speech
transcription, emotion, gestures, dialogue acts and posture. This
information is used
for (at least) two purposes: as training material for recognition
algorithms and as
evidence on the basis of which theoretical models of human multi party
can be developed.
The AMI project is the context for many different activities of the
HMI research group. The following list
gives a few examples of ongoing work.
For more information about the AMI project, follow one of the links
below or contact one of the HMI researchers mentioned on the next
- The Virtual Meeting Room
is a 3D virtual environment displaying the AMI
meeting room with
avatars as meeting participants. The most obvious use of this
environment is to literally replay (fragments of) the recorded
meetings. More interesting though are applications
such as simple validation of interaction models and/or recognition
results ("does the
predicted behaviour look sensible when displayed in the VMR?"), the
ability to restructure
meetings (e.g. displaying all arguments around a certain decision in
discussion, smoothing over the transitions between the original
fragments of the meeting).
- By Gaze Experiment another application of the VMR is
algorithms are used to predict current speaker from gaze directions or
The results are compared to the performance of humans who get exactly
the same input as the
machine learning algorithms. To this end human judges are presented
with a screenshot of a
Virtual Meeting Room setting that shows four neutral avatars with
correct head orientation.
For each screenshot they have to say who they think is the current
speaker. In this setup,
all information that is not available to the machine learning
algorithms, such as facial
expressions, is suppressed for the human judges as well.
- Annotation tools:
HMI has also been involved in the development of better tools for
manual annotation of
aspects such as dialogue acts, named entities, gaze targets or
- Research into Addressing:
In the context of dialogue act recognition work is being done on
addressing behaviour. How
do people indicate at whom they address a certain utterance? Can the
- Pose and Gesture Detection
Within HMI a computer vision platform has been developed. One aspect
of this platform
involves 3D pose recognition from single camera images. The output of
this pose recognition
is now used for the development of gesture recognition in meeting
The following HMI-member(s) is/are coordinator of this Project
Here you can find the publications
The following ShowCases are associated with this Project