Measuring Behavior 2005: Reidsma et al.

Building efficient, focused annotation tools for large scale annotation efforts: A demonstration of three new tools

D. Reidsma, N. Jovanovic and D. Hofs

Department of Electrical Engineering, Mathematics and Computer Science, University of Twente, Enschede, The Netherlands

The creation of large, richly annotated, multimodal corpora of human interactions is an expensive and time consuming task. Therefore, support from annotation tools that make the annotation process more efficient is required, especially if the annotation effort involves really large amounts of data. Tools which are very efficient for one particular annotation task are not necessarily so for other tasks. Making one tool to handle every conceivable annotation will result in a monolithic, unwieldy tool. Therefore we investigated how different properties of specific annotation tasks can have an impact on the design of a tool focused on that general class of tasks. At the symposium ‘Annotating and Measuring Meeting Behavior’ we present our view on the considerations that should drive the design of new tools geared to specific tasks. In this paper we describe three new annotation tools which have already been developed using that method: a tool for markup of structured discourse entities and relations in transcriptions, a tool for labeling of continuous, non-overlapping phenomena in audio and/or video and an adaptation of the FeelTrace tool for labeling of emotion. The tools all make use of the same corpus exchange format and API so they can operate on one shared corpus, each contributing different parts of the annotation data. Due to the considerations used in the design, the tools are easily optimized to a different annotation strategy, allowing the annotation process to run as fast and as smooth as possible. The fact that the different tools are designed as a configuration of interoperating modules rather than as single tools makes it possible to reuse many of the design considerations for the development of focused annotation tools for a different class of annotation task.

Paper presented at Measuring Behavior 2005 , 5^th International Conference on Methods and Techniques in Behavioral Research, 30 August - 2 September 2005, Wageningen, The Netherlands.