In the last twenty years of research we have tried to establish a behavioral repertoire of human communication on the bases of traditional coding with behavior categories. This research was basically unsuccessful, and we realized that:

• the meaning of signals seems to depend from too many variables,
• signals show high inter- and intraindividual variation,
• one signal can be substituted by another signal,
• signal combinations and combinations of combinations are possible,
• signals can be used independent from context,
• non-verbal behavior seems to be highly probabilistic,
• non-verbal behavior is norm-governed in different cultures, and
• the same category of behavior can be displayed qualitatively different.

These facts lead to an "atomizing effect" in behavior categories - for an exact description and a complete repertoire coverage the categories will become smaller and smaller eventually until every single performed behaviour has its own category [2]. In addition to these problems traditional coding has to apply information reduction in order to be feasible. Pattern recognition in behavior is only possible by adding a "world model" or "a priori" knowledge to the behavior. Thus, observing with defined behavior categories is already a data reduction, and it seems likely that the information put in the construction of a behavior definition is highly biased in favor of the research question.

In this talk I will introduce a method of behavior analysis which is independent from human categorical perception. The new method for the assessment of quantitative and qualitative description of non-verbal behavior does not use any assumptions on the structure and organization of behavior itself. The applied algorithm (motion energy detection) can analyze the number of movements, duration, size, speed, complexity and information content from videotaped episodes in body movements and voice on a real-time frame-to-frame basis. These parameters then can be translated in expressiveness and emphasis scores.

Figure 1. Screenshot from E-motion 2.0 currently implemented on a Power Macintosh G3. The picture in the upper left is the original from a current study on the attractiveness of body rhythms, the frame shows the region of interest where motion energy detection takes place. The picture on the upper right is a result window for the first filtering steps (noise, sharpening). The picture in the lower left shows the results of motion-energy detection. Red color is the difference between the last picture and the current picture. Blue colors denote differences from now minus 25 frames (1 second). This type of information can be fed to neural networks for movement pattern recognition, or for calculating overall movement speed and acceleration.

Figure 2. This picture shows differences between single video frames from a movie of a dancing female. The red color is the difference between the current frame (t) and the frame before (t-1), blue is the difference between the two frames at t-25 and t-24. The colored areas show changes in the pictures - the size of the areas thus denotes the amount of motion. Motion energy is used in this study for detection of overall body rhythms and cyclic pattern stability.

The areas of current applications are:

• detection of intentions and motivations in interactions among humans [3],
• detection of synchronisation and rapport in interactions, in combination with Theme, a pattern detection algorithm [4],
• classification of the "big five" personality traits and depression in structured interviews,
• gender recognition and classification of gender identification from gait patterns, and
• attractiveness of body movements during dancing (Figures 1, 2).

I will present examples and first results from these research areas and suggest extensions to current communication theories, where the syntactic elements are replaceable by qualitative changes of movements. These qualitative changes could determine the meaning of signals independent from its content. The mechanisms used for signal transmission are probably provided by "mirror neurons", which fire in the frontal lobe (rostral part of the inferior area 6) of the receiver with the same patterns the producer of the movements uses to generate the movements [1]. Thus the brain could be able to decode body movements automatically without adding additional categorical information. If this is the case, current communication theories are basically wrong or an insufficient framework for the explanation of human communication if they rely only on signals in their traditional definition.

References

1. Gallese, V.; Fadiga, L.; Rizolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593-609.
2. Grammer, K.; Fieder, M.; Filova, V. (1997). The communication paradox and possible solutions. In A. Schmitt, K. Atzwanger, K. Grammer & K. Schäfer (Eds.). New Aspects of Human Ethology, 91-120. New York: Plenum.
3. Grammer, K.; Honda, R.; Schmitt, A.; Juette, A. (1999). Fuzziness of nonverbal courtship communication unblurred by motion energy detection. Journal of Personality and Social Psychology, 77, 509-524.
4. Magnusson, M.S. (2000). Discovering hidden time patterns in behavior: T-patterns and their detection. Behavior Research Methods, Instruments & Computers, 32, 93-110.