Titel: The user effects of using textual cues to increase image viewing attention Instituut: University of Twente (HMI) Plaats: Enschede Land: Nederland Begindatum: 15-01-2008 Voltooid: Nee Begeleider: Mariët Theune Onderwerpen: Multimodal Interactions, Speech and Language Technology, Human factors

Beschrijving:

The research proposal is to study the effects on readers of text cues to adjacent pictures. This thesis project is oriented on improving multimodal answer presentations as those used in the medical question answering system IMIX. IMIX is a system that automatically generates answers to medical user questions and it includes a picture to the textual answer. Cues could aid readers in processing pictures and integrating them with the textual information. Peeck (1993) discusses the effects of pictures on learning from illustrated text and the benefits of instructual interventions on learning from illustrations. An instructual intervention is a type of study instruction; it guides a student to more intensively study a picture. Such instructions can be as simple as telling a student to study a particular picture. Instructions can also be elaborate by giving specific assignments which require a picture to be studied in detail. A more elaborate instruction should consequently require more intensive processing of the information. One of the available instruction methods are cues. In this experiment we will study textual cues. Cues generally draw the user attention to a specific object or process. Textual cues refer to a picture when this picture has become relevant in the text, for example; “See figure 1” and “In figure 1 spots are visible on the boys face”. We will focus on the effects of cues and whether cues increase the processing of pictures and illustrated text in answer presentations like those used in IMIX. However there is also the possibility for adverse effects. Cues could also disturb the reading process as they direct the attention of the reader away from the text. Thus cues could also distract. In the worst case this will negatively influence the learning process. This study aims to create a better understanding of the effects of the cues on readers. To observe the immediate behaviour of the reader we intend to employ the use of an eye-tracker. An eye tracker will show where the reader has looked, for how long, and in which order. This will allow us to compare the behaviour between participants in different experiment configurations. Participants will also be asked to give a subjective rating on the informativity of each answer presentation. Each participant will also take part in a post-test to measure the retention of information of each answer presentation. Peeck, J. (1993). Increasing picture effects in learning from illustrated text. Learning and Instruction, 3(3):227-238. Capita Selectaopdracht Titel: A study of three models for image-text relations Instituut: University of Twente (HMI) Plaats: Enschede Land: Nederland Begindatum: 12-09-2006 Voltooid: Ja Verslag: http://hmi.ewi.utwente.nl/verslagen/capita-selecta/CS-Kornalijnslijper-David.pdf Begeleider: Mariët Theune, Wauter Bosma Onderwerpen: Multimedia Retrieval, Speech and Language Technology Beschrijving: Images and texts placed together in media are likely to be related. These modes share a relation. Various applications which use image-text combinations can benefit from a model which describes the features of this image-text relation. This research examines three models which can benefit the IMIX question answering system. IMIX answers medical questions and uses image-text relations to retrieve suitable images for its generated answers. Twenty image-text samples from a medical information corpus are annotated to create a better understanding of the three models, showing the advantages and disadvantages of the models and their individual differences.