A multimedia system for temporally situated perceptual psycholinguistic analysis
F. Quek1 and D. McNeill2
1Department
of Electrical Engineering and Computer Science, Wright State University, Dayton,
OH, U.S.A.
2Departments of Psychology and Linguistics, University of Chicago,
Chicago, IL, U.S.A.
Perceptual analysis of video (analysis by unaided ear and eye) plays an important role in such disciplines as psychology, psycholinguistics, linguistics, anthropology, and neurology. In the specific domain of psycholinguistic analysis of gesture and speech, researchers micro-analyze videos of subjects using a high-quality video cassette recorder that has a digital freeze capability down to the specific frame. Such analyses are very labor-intensive and slow. We present a multimedia system for perceptual analysis of video data using a multiple, dynamically linked representation model. The system components are linked through a time portal with a current time focus. The system provides mechanisms to analyze overlapping hierarchical interpretations of the discourse, and integrates visual gesture analysis, speech analysis, video gaze analysis, and text transcription into a coordinated whole. The various interaction components facilitate accurate multi-point access to the data. While this system is currently used to analyze gesture, speech and gaze in human discourse, the system described may be applied to any other field where careful analysis of temporal synchronies in video is important.
Poster presented at Measuring Behavior 2000, 3rd International Conference on Methods and Techniques in Behavioral Research, 15-18 August 2000, Nijmegen, The Netherlands