About Me

Professor Socially Intelligent Computing at the University of Twente, my research interests cover both the machine analysis of human (conversational) behaviour and the generation of human-like (conversational) behaviour by virtual agents and robots. I am especially interested in the nonverbal and paraverbal aspects of dialogue and what these signals reveal about the mental state (cognitive, affective, social). These topics are explored both from a computational perspective and as basic research in the humanities, reflecting my training as a computational linguist. But my interests are much broader as you may learn from these pages and my publications.

After my studies of Linguistics, Computer Science and Computational Linguistics at the University of Antwerp I moved to the Institute of Dutch Lexicology in Leyden, to develop tools for enriching natural language databases. After a couple of years I went on to the Utrecht University and got involved in the big European project Eurotra on Machine Translation. After coordinating a follow-up EU project, I started my PhD project on a logical approach to natural language analysis and parsing (Type Logical Grammar).

At the University of Twente I started working on embodied dialogue systems (aka virtual agents or embodied conversational agents). This made my interests shift from pure linguistic analysis to body language, from text analysis to real-time human-machine interaction, and from the logical analysis to a much broader concern with emotion and social relations in mediated human-human and in human-computer interaction.

Over the years, I have had the opportunity to expand investigations in the full bandwith of multimodal communication. In the BrainGain project I was able to look at brain signals as a mode of human computer interaction and in the current COMMIT/ projects I am exploring interaction through “touch”.

I have been involved in several European projects such as Humaine, AMI and AMIDA, COST 2102, Semaine, SSPNET, SERA, Aria Valuspa and De-ENIGMA. Dialogue management, human behaviour modeling, cognitive modeling, human-robot interaction, methodological issues in corpus studies and annotation, are some of the topics dealt with in these projects. An interesting Dutch national projects I was involved in include BrainGain in which neural correlates of a user’s experience are explored for Human Computer Interaction using Brain Computing Interfaces and GATE dealing with cognitive models of virtual agents in training applications.

Currently, I am the project leader of the COMMIT project Interaction for Universal Access. In this project we not only explore the use of virtual agents for training, but also investigate interactive playgrounds for children. In the COMMIT project VIEWW I am studying mediated touch.

Several new activities started this year that continue the research themes mentioned above. Take a look at:

  • Aria Valuspa: a European Project on intelligent virtual agents for assisting in information retrieval tasks (QA)
  • 3TU Humans & Technology: the Centre H&T started this year. Telepresence, smart spaces for happy and healthy living. These are the themes. I am the scientific director.
  • AAAC: the Association for the Advancement of Affective Computing is organising the next conference on Affective Computing and Intelligent Interaction. I am currently president of the association.

Topics of research: embodied conversational agents, social signal processing, ambient intelligence, affective computing spoken dialogue systems, human computer interaction, backchannels, brain computer interfaces, interactive playgrounds, virtual and augmented reality.