Ecological validity in usability testing

G. Lindgaard¹, K. BharrathSingh², B. Tsuji¹ and S. Khan¹

¹Human Oriented Technology Lab, Department of Psychology, Carleton University, Ottawa, Canada
²CTO Office, Nortel Networks, Ottawa, Canada

The traditional focus on measuring work productivity that characterized standard usability measures of efficiency and effectiveness is not always relevant in new or emerging technologies. In particular, measurements of user preferences, satisfaction, or conversely, user frustration, occupy a far more central role in discretionary domains such as mobility or entertainment. This paper addresses the issues of selecting suitable behavioral usability metrics to address ecological validity in novel technologies. In the first two experiments, a modern version of walkie-talkie technology was evaluated for usability. Since this technology is intended for emergency situations in which users are expected to be intolerant of delays, the first challenge was to identify one or more behavioral measures that would capture user frustration. Two artificial situations were created; one in which delays should not matter to the user, and one in which we expected users to exhibit signs of frustration. We believed that frustration would be measurable in the number of additional, unnecessary key presses during a fictitious interaction with another person. The first experiment yielded no results, and even after priming the subjects with a video clip portraying a very serious and urgent situation and designing the interaction to play on this urgency, no results were obtained. The third experiment used the metaphor of interactive TV and also the number of key presses as a measure of impatience. Again, this yielded no results. We conclude that the lack of ecological validity may account for this; subjects who find themselves in a laboratory where they are asked to sign an informed consent form and are subsequently paid for their participation are unlikely to respond to the artificially created situation in a manner that resembles reality. The question is therefore whether we can hope to obtain valid behavioral responses to these technologies in a laboratory setting at all. We speculate on ways in which frustration with emerging technologies may be measurable.

Paper presented at Measuring Behavior 2005 , 5^th International Conference on Methods and Techniques in Behavioral Research, 30 August - 2 September 2005, Wageningen, The Netherlands.

© 2005 Noldus Information Technology bv