Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 361

measuring efficiency and productivity, and hence giving an indication of usability.
There are several others too, though, and they are usually concerned with either
performance (quantitative measures) or process (qualitative measures). There is
also the concept of user experience, related to how much satisfaction the user
obtains from the system. Here we briefly describe the measures that you are most
likely to encounter. Often you will use several complementary methods to mea-
sure usability, such as a combination of task performance times, and a usability
survey.

13.5.1 Task Time

Task performance time is widely used as a measure of efficiency within the fields
of HCI and human factors and ergonomics. The task is usually one of three: a
small cognitively manageable task, often referred to as a unit task (Card et al.
1983 ); a standard, predefined benchmark task (that you use to assess efficiency for
similar artifacts); or a task scenario (as described above).
It is easy to determine task time using a stop watch, for example, or using time
stamps if you are recording task performance. Time is a measure that is widely
understood, and is easy to analyze statistically. Time is generally used as a
measure in summative evaluations of the final system. Where the performance
time is relatively insensitive, however, it can be costly to carry out evaluations,
because you will have to run the test many times to be able to draw solid statistical
conclusions from the results.
Remember that usability is not only concerned with how easy something is to
use, but also how easy it is to learn to use. Task times can also be used to
determine how long it takes to learn to use a system. Normally some threshold
level of performance is defined in advance, and the length of time it takes
to reach that threshold is measured. Alternatively, the length of time it takes
to recover from observable errors can be measured: you would expect to see
this time reduce as people learn how to do the task and how to manage the
errors.
One of the main problems of using time measures is that they are not easily
compared unless all the contextual elements (tasks, level of expertise, lighting
conditions, and so on) are kept constant. The corollary of this is that if you want to
compare times when you cannot fully control the contextual effects, you have to
convert the data into a more stable metric, i.e., one that is not so easily affected by
changes in these elements. One way of doing this, which was proposed by Whiteside
et al. ( 1985 ), is to calculate a score in the range 1-100 as shown in Eq. ( 13.1 ):

Score ¼ 1 = T ð Þ P C ð 13 : 1 Þ

where T = time,C= constant based on fastest expert time, and P = percentage
of task completed.