Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 363

You may find it easier to collect concurrent protocols by having two users work
together on a task. The natural dialogue that takes place (assuming that dialogue
occurs or is required for the task) will encapsulate the information they are using to
do the task. Another possible variation is to use expert commentary. Here one
expert describes what the user is doing as they perform the task.
Retrospective protocols can also be used, and these are taken after the task has
been performed. They tend to be more useful when people can watch a video or
pictorial record—we discuss visual protocols in the next section—of their per-
formance to help them remember what they did. This helps them to recognize their
actions, rather than just having to recall them from memory. Although subjects
may find it easier to provide retrospective verbal protocols, they can lead people to
provide post hoc rationalizations of actions that they now perceive to be incorrect
or that they performed instinctively.
Another way that you can interrogate what users are doing is by using pop-up
menus (Feurzeig and Ritter 1988 ). This idea has not been fully tested, however,
and does not have the same level of theoretical support as concurrent verbal
protocols. The obvious criticism is that the pop-up menu interrupts the task, and
may break the user's flow of activity because it draws their attention away from
the task. A similar but more intrusive approach is to freeze the task and ask users
about what they are doing at that particular point in time. This latter technique has
been used in measuring situation awareness (Endsley 1995 ).
13.5.4 Video Protocols
Video protocols (also called visual protocols) involve making a video recording of
users as they carry out some prescribed task. The recording is often made using
multiple cameras positioned to capture different aspects of performance, such as
what is currently shown on the screen, the position of the user's hands, and a more
general view that shows both the user and the system together. Sometimes the
recordings are made directly from the monitor. Although video protocols provide
very rich data, the fact that they are being video recorded does make some users
feel under pressure, and can lead to unnatural behavior.
The main problem with video protocols is that analyzing them can be very hard
and is very time-consuming. Typically, analysis can take anywhere between 10
and 100 times as long as the duration of the recording.
As noted above, video protocols can be shown to users to help in the collection
of retrospective verbal protocols. This technique is sometimes called auto-con-
frontation, because the users are shown the video recording and asked to explain
their behavior.
Video protocols can be shown to developers to let them see the sorts of
problems that real users encounter with their system. It is arguably better, though,
to let the developers watch users try to use their product in a usability laboratory in
real time. Both of these methods are generally much more effective than simply