Measuring Behavior 2005: Barkhuysen et al.

Audiovisual cues to finality

P. Barkhuysen, E. Krahmer and M. Swerts

Tilburg University, Tilburg, The Netherlands

During spoken interactions, conversation participants are able to adequately detect when their partner finishes a turn so that they can elegantly take over, without much overlap or delay. Previous research showed that speakers partly base such decisions on intonational cues (e.g. low-ending contours are reserved for turn-final position). The question explored here is whether there also exist visual cues, such as facial expressions, that speakers exploit to signal whether or not they want to continue their turn.

We collected utterances from 8 speakers, who were asked simple questions that elicited sequences of nouns (varying in length) during an interview in front of a camera. A selection of these were presented to participants during a perception experiment in one of the following modes: the original film containing audio and video, only the visual part of the material or only the auditory part. The participants’ task was to indicate as fast as possible when they felt a speaker’s utterance had reached its end. To get a baseline performance, the actual experiment was preceded by a test in which subjects had to respond to the end of (bimodal or unimodal) presentations of still pictures and/or monotonuous sounds without finality cues.

Participants were able to predict the end of the utterance, and showed the fastest reactions in the audiovisual condition. In the baseline session, however, reaction times were slower for the audiovisual stimuli. These findings suggest that cognitive load is higher when subjects need to focus on two modes at the same time. However, our perceptual system is facilitated when two different information sources (visual, auditory) are congruent in their cue value. Further results will be discussed at the workshop.

Paper presented at Measuring Behavior 2005 , 5^th International Conference on Methods and Techniques in Behavioral Research, 30 August - 2 September 2005, Wageningen, The Netherlands.