Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 362

13.5.2 Errors
Errors can be measured quantitatively (by simply counting them) or qualitatively (by
noting the different types of error). Whilst time is best suited to summative evalu-
ations, error measures can be used in both summative and formative evaluations.
As we have already seen in Chap. 10 , however, errors are not easy to define,
and they can be hard to count too. This is particularly true when observing expert
behavior. One of the key aspects of expert performance is that they often detect
and recover their own errors before the effects of the error become apparent to
outside observers. So if you watch an expert perform a task, you may not even
realize that they have made an error.
We can distinguish many types of errors—slips, mistakes, violations, mode errors
(e.g., problems with grayed out menu items), discrimination errors (e.g., selecting the
wrong menu item because of ambiguous labels), and so on. The types of errors will
vary depending on which taxonomy of errors you use (see Chap. 10 for examples).
13.5.3 Verbal Protocols
Verbal protocols can be a useful way of understanding the issues that confront
users as they try to tackle particular problems using some artifact. Some care is
needed when reading about verbal protocols, because many people use the terms
talk aloud and think aloud interchangeably. Strictly speaking you usually want
people to produce talk aloud reports, reflecting the things that are in their short
term memory as they do the task; if they generate think aloud reports, this suggests
that they are processing things more deeply and (possibly) rationalizing their
decisions and actions before they verbalize them.
The two main types of verbal protocols are concurrent, which are taken whilst
the person performs the task, and retrospective, where the person describes what
they did after completing the task. In concurrent verbal protocols, the user is asked
to talk about information as it comes to mind, to ''say out loud everything that you
say to yourself'' (Ericsson and Simon 1980 , 1993 ). The user should not be
reflecting upon their own behavior and providing explanations of causality. While
this kind of reflective behavior (referred to as introspection) may provide some
useful insights, these insights are not considered valid data because they are easily
influenced by other aspects, such as expected task performance, social pressure,
and the user's (often incorrect) theories of how their own mind works. Talking
aloud about content is generally regarded as being more objective than thinking
aloud, which usually involves introspecting about the process.
Providing concurrent protocols can be hard for users, but they are more reliable
than other types of verbal protocol. When you take concurrent verbal protocols,
you should ask the user to practice providing a concurrent verbal protocol whilst
carrying out a simple task, such as an arithmetic addition or counting the windows
in their childhood home (Ericsson and Simon 1993 , appendix).