Studying usage of complex applications

J. Kort¹, M.G.D. Steen¹, H. de Poot², H. ter Hofte² and I. Mulder²

¹TNO Information and Communication Technology, Groningen, The Netherlands
²Telematica Instituut, Enschede, The Netherlands

Innovative products and services are characterized by an increased complexity in technology and functionality. This increased complexity is well known in product development and research areas like: personal networks, intelligent home environments, ubicom, CSCW, etc.

Increased complexity poses new problems for usage studies. Current research methods and tools are not always suf.cient to deliver the desired insights. They are too resource consuming to evaluate the complete product or service, represent a limited number of research interests and are not applicable in complex environments like daily life.

Furthermore, many developments directly related to usage have become increasingly important in complex products and services (e.g. personalization, adaptability and context awareness). To study usage in this context, existing methods are often not sufficient because of their narrow focus on task performance or usability.

The first steps towards developing an integrated framework for usage analysis, addressing issues raised by complex product design, will be made during the CHI2005 workshop ‘Usage analysis: Combining Logging and Qualitative Methods’.

Related research is performed in the Freeband User Experience project (FRUX), part of the research program Freeband Communication. A context aware application is developed, which supports police officers in retrieving the right information at the right time, based on the officer’s social relations with colleagues. How to study the usage of this application is a problem currently addressed.

In FRUX new usage analysis tools, integrated in a measurement platform, are developed (partially based on the CHI2005 workshop results). These tools consist of a combination of qualitative (e.g. experience sampling (ESM), questionnaires) as well as quantitative (e.g. log files, automatic registration of social interactions and sensor data) methods to enhance research possibilities and insights in the usage of applications.

In this paper we present these tools, their conceptualization and design and the preliminary results of their application during a small scale pilot. We conclude with suggestions for further developments of the tools and platform.

Paper presented at Measuring Behavior 2005 , 5^th International Conference on Methods and Techniques in Behavioral Research, 30 August - 2 September 2005, Wageningen, The Netherlands.

© 2005 Noldus Information Technology bv