Analysis of the human multimodal emotional signals to an interactive computer

R. Ciceri, S. Balzarotti and P. Colombo

Laboratory of Communication Psychology, Catholic University of Sacred Heart, Milan, Italy

In «The Media Equation» Reeves and Nass argue that human-machine interaction is inherently natural and social, so that the rules of human-human interaction apply to human-machine interaction: in many ways people seem to respond psychologically to interactive computers as they were human actors and not tools. The increasing use of computers and machines that support the human user in any kind of task has nowadays transformed them into important and constitutive members of the physical and social environment with which people interact. For these reasons, researchers from several disciplines have turned their attention to emotion and its place in the interaction between human and artificial agents.

The present study tries to examine what kind of emotional responses the subject is motivated to express while interacting with an artificial agent if he/she believes that this agent is able to understand his/her emotional states. For this purpose different kind of computer games were programmed to elicit specific emotional appraisals and two different conditions were used. 30 subjects were asked to use the computer where in one condition the avatar provided a simulated intelligent feedback, while in the second only guided the subject across the different tasks.

Multimodal synchronized data were captured: physiological data, posture, facial expressions, vocal behavior. All observational data were codified frame by frame (25 fps) using The Observer^® 5.0 (Noldus Information Technology bv, The Netherlands). This analysis has previously required the elaboration of a coding grid and thus the selection of behavioral units to be extracted. As for Human behavior, four macro-categories were then considered: facial movements (the fundamental muscle movements that comprise Facial Action Coding System were selected), posture, gaze direction, vocal behavior. The computer game events and kind of computer task were also considered. Theme^tm (PatternVision Ltd., Iceland) was used for the detection of recurrent patterns. This is an in-progress study and data are still under elaboration. The paper includes initial results of the analysis of non-verbal communicative signals.

Paper presented at Measuring Behavior 2005 , 5^th International Conference on Methods and Techniques in Behavioral Research, 30 August - 2 September 2005, Wageningen, The Netherlands.

© 2005 Noldus Information Technology bv