Traditional usability testing is conducted in an ad hoc manner, essentially yielding observer opinions about usability problems that exist in the product being tested. In this way, a test user is observed interacting with the software product or prototype. As the test user encounters difficulties, the observer takes notes about where the trouble occurred and then suggests what usability problem is creating or contributing to the user's difficulties. The observers (generally usability professionals) record the data in a narrative or list format, and then report it to the product developers as an accurate view of human interactions with the system [1,2].

Recent studies indicate that different usability test professionals cannot replicate each other's results: each identifies different usability problems in the product being tested, with little overlap between them [3,4]. In addition to this replication issue, traditional usability testing and data recording fail to generate quantitative data, avoid specific behavioral data, and do not test users of different levels in ways that can be compared with one another.

The author has developed a quantitative approach to usability testing: the Optimal Path Test Method. This method creates a distinction between user errors (i.e. observable events that occur during testing) and usability problems (i.e. underlying design issues that contribute to the occurrence of user errors). Expanding on a traditional doctrine that different tests should be conducted for novice users as opposed to expert users, the Optimal Path Test Method has been applied to three levels of users performing exactly the same task, thereby generating data suitable for comparison across levels. Analyzing statistical contrasts by group level and by test element revealed not only where usability problems existed, but also provided empirical underpinnings for determining the type of usability problem present.

Replacing traditional usability testing with the structured Optimal Path Test Method, and testing and comparing users of varying experience, renders a data set that supports multiple levels of analysis of human behavior during usability testing. It also makes connections between the behavior and the actual problems that exist in the product under test. Moreover, the Optimal Path Test Method provides results that are repeatable and consistent across testers.

References

1. Faulkner, L.L. (2002). Developing the Optimal Path Test Method: A Quantifiable Approach to Usability. Proceedings of the International Conference on Practical Software Quality Techniques/Practical Software Testing Techniques, 2002.
2. Grosvenor, L.J. (1999). Software Usability: Challenging the Myths and Assumptions in an Emerging Field. Unpublished Master's Thesis, The University of Texas at Austin.
3. Bailey, B. (2001). How reliable is usability performance testing? User Interface Design Update, September 2001.
4. Molich, R.; Bevan, N.; Curson, I.; Butler, S.; Kindlund, E.; Miller, D.; Kirakowski, J. (1998). Comparative evaluation of usability tests. Proceedings of the Usability Professionals' Association.

Paper presented at Measuring Behavior 2002 , 4^th International Conference on Methods and Techniques in Behavioral Research, 27-30 August 2002, Amsterdam, The Netherlands

© 2002 Noldus Information Technology bv