Measuring cooperations between modalities
in human multimodal behavior
J.C.M. Martin
CHM AMI, LIMSI-CNRS, Orsay, France
When communicating with someone else, we use several different modalities in a cooperative way, such as speech, gestures, facial expression and posture. The mechanisms underlying this multimodality are not completely identified or understood. Similarly, we do not fully understand the behavior a subject might display when faced with a system that allows him or her to use different modalities. The study of such multimodality is currently driving collaborative research by LREC and IST-ISLE.
Following previous work on the manual annotation of human multimodal behavior, we have developed two tools :
- grammar for describing annotations of multimodal behavior
- software for computing behavioral metrics from annotations
According to our grammar
(an XML DTD), these annotations are composed of several sections. The first
section describes the features of the objects referred to by the subject in
the corpus (Figure 1). Each of the following sections contains the annotation
of a multimodal segment, itself composed of several sub-sections (one for each
modality).
Figure 1.
Frame
of a multimodal corpus.
A software package has been developed to parse these annotations and compute multimodal behavioral metrics (e.g. rates of complementarity, redundancy and equivalence). It follows these steps:
- Parse the annotation and build an internal representation of the corresponding tree
- Assign a 'salience' value to each (object, reference) couple, according to rules such as 'if the reference contains the fully specified name of the object, assign value 1.0 to the salience value'
- Assign a priori fixed values to weights for each modality
- Compute the average salience value in each single multimodal segment across all modalities
- Compute the average salience value for each object across all modalities
- Compute behavioral metrics
40 samples taken in several corpora have been manually annotated and parsed by our software. In the near future, we intend to improve the annotation by using a tool such as Anvil. This will allow us to annotate larger corpora, and to integrate the grammar and metrics in a methodology for analysing multimodal behavior across different applications.
IST-ISLE http://isle.nis.sdu.dk/
References
- Kipp, M. (2001). Anvil - A Generic Annotation Tool for Multimodal Dialogue. Eurospeech 2001, September 2001.
- Martin, J.C.; Grimard, S.; Alexandri, K. (2001). On the annotation of multimodal behavior and the computation of cooperation between modalities. Fifth Conference on Autonomous Agents.
- Workshop: 'Multimodal Resources and Multimodal Systems Evaluation'. LREC 2002, June 2002.
Paper presented at Measuring Behavior 2002
, 4th International Conference on Methods and Techniques in Behavioral
Research, 27-30 August 2002, Amsterdam, The Netherlands
© 2002
Noldus Information Technology bv