This presentation will deal with the measurement of the cooperation between gesture and speech.

First, we will explain TYCOON, a framework that has been proposed for the analysis of multimodal behavior [5]. This framework is based on a typology made of six primitive types of cooperation: equivalence, specialization, transfer, redundancy, complementarity and concurrency. We have already applied this framework to the analysis of the multimodal behavior of subjects in a Wizard of Oz experiment at the Stanford Research Institute [2].

Secondly, we will describe a video corpus developed within the European project "Chameleon" [4]. In this corpus, twelve subjects have been recorded while making pointing gestures on a map and speaking constrained commands (i.e. "who is in this office?"). The aim of this corpus was to identify the features (hand shape, dynamics, etc.) of pointing gestures in a multimodal context in order to design a gesture recognition [1,3].

Finally, we will discuss how the TYCOON framework could be used for the analysis of the cooperations between pointing gestures and constrained speech in such a corpus. Referenceable objects of the map will be identified (i.e. rooms, walls, etc.). Rules will be proposed for computing the salience of this objects in gestural or spoken references (i.e. repeating pointing gestures could be considered as a way to increase the salience of the pointed object in the reference). Then, the rate at which the subject's behavior is either redundant or complementary could be computed with the following formula: a global salience value has to be computed over all referents rk of all the commands Cj expressed by the subject; then this number is divided by the number of referents expressed by the subject as observed in the corpus.

References

1. Braffort, A.; Gherbi, R. (2000). Video-tracking and recognition of pointing gestures using Hidden Markov Models. IEEE INES'98 (Budapest, 1998).
2. Cheyer, A.; Julia, L.; Martin, J.C. (1998). A unified framework for constructing multimodal experiments and applications Proc. 2^nd Int. Conf. on Cooperative Multimodal Communication, Theory and Applications (CMC'98), 28-30 January 1998, Tilburg, The Netherlands. URL: www.limsi.fr/Individu/martin/ publications/download/cmc98-1.ps.
3. Gherbi, R.; Braffort, A. (1999). Pointing gestue intepretation in a multimodal context. 3^rd International Gesture Workshop (Gif-sur-Yvette, France, 1999). Springer.
4. Gherbi, R.; Braffort, A. (2000). Methodology for the design and evaluation of a gesture recognition system. Proc. RFIA'2000 Conference (Paris, 1-3 February 2000), I-47, I-56. [in French]
5. Martin, J.C.; Beroule, D. (1993). Types et buts de coopération entre modalités. Actes des Cinquiemes Journées sur l'Ingénierie des Interfaces Homme-Machine (IHM'93), 19-20 Octobre 1993, Lyon, France.

© 2000 Noldus Information Technology b.v.