In the scientific community, well-documented evaluation methods are primarily those in which experts make the system assessments. Many of these describe methods such as thinking aloud or interviews with test-users. In contrast, methods based on recording logfile data, or observing interactions between test subjects and the system, are hardly ever documented. Despite this, such tests are frequently conducted, because the data they produce are reliable and many long-term effects can only be proven in this way. The toolset we have developed is designed to act as a guideline, by providing a set of tools for those steps of the evaluation that occur systematically, i.e. recording and analysing data.

Logfiles show snapshots of system states. To assess user behavior, it is necessary to examine all of the interaction data. This work could be simplified by structuring the data in advance. Developing prototypes of technical systems is often restricted by limited conditions (e.g. platform, development surroundings). Therefore, it is not possible to give developers instructions concerning the concrete realization of their system. To use our toolset, only the structure of the logfiles produced must be variable; the toolset is not designed to replace professional tools for statistically evaluating the data (e.g. SPSS, Excel). Between these two steps (i.e. developing the system and performing statistical calculations) lies the real problem for the tester, where (s)he must view, structure, complete and summarize the data. To achieve this, (s)he often uses specific new macros, scripts or self-made tools, which must be adapted for every new question. Our toolset fundamentally simplifies and structures this work.

As already explained, the structure of the produced logfiles within the system must be variable. However, one firm requirement is that the data must be saved as XML files. To achieve this, we have designed an XML-DTD. Despite having a simple structure that can be easily created by the system, it is also powerful and flexible enough to handle every imaginable type of logfile data. The use of XML made it possible to develop a general tool for browsing logfiles. As well as representating all entries in tabular form, it is also possible to switch different filters on or off within this table.

By improving the browser to an editor in this way, new information (e.g. names of conditions and events, identification of hierarchical levels) can be entered and saved in the logfile. It is also possible to record details regarding the knowledge of the evaluator about process content and application events within the logfile. An Application Knowledge Base (AKB) can be built, eventually making it possible to perform an automated evaluation. The tester must deal only with those sections in the logfile that have not already been taken into account by the AKB. The application of the AKB is illustrated in Figure 1.

Figure 1. Use of the editor and the AKB.

Paper presented at Measuring Behavior 2002 , 4^th International Conference on Methods and Techniques in Behavioral Research, 27-30 August 2002, Amsterdam, The Netherlands

© 2002 Noldus Information Technology bv