Errors: An Inherent Part of Human-System Performance - Foundations for Designing User-Centered Systems - page 285

operating the system being used. Whilst it might be possible to use operators from
the relevant domain, their experience is liable to be biased towards the particular
system they normally work with (as opposed to your new system). The practicality
of getting access to operators for the length of time needed to conduct the
experiments also mitigates against using this approach. In addition, if subjects can
identify the purpose of the experiment they may behave more cautiously, to guard
against performing erroneous actions.
It can also be difficult to choose an appropriate experimental task and setting.
Complex systems often require operators to perform multiple tasks, sometimes
simultaneously. The difficulty is to find a complex system (or an appropriate
simulation) that can be readily deployed under laboratory conditions. The system
would also need to be familiar to the subjects to fulfill the criterion regarding
expertise. Unfortunately, laboratories that have their own high fidelity simulations,
such as Halden's nuclear power plant simulator (Hollnagel et al. 1996 ) are still the
exception rather than the rule.
10.2.2 Field-Based Observation
The second method is to carry out longitudinal observation of experienced oper-
ators. The main advantage of this method is that it guarantees the ecological
validity of the data. In general, observation is a valid technique, particularly if the
aim is to investigate human reliability per se. In light of Mach's ( 1905 ) observation
about the relationship between knowledge and error, there is a lot to be learned
about operator performance under abnormal system operating conditions from
observing performance under normal operating conditions.
Observational research tends to focus on one specific aspect of operator
behavior, however, rather than on operator behavior per se. The observational
method, therefore, has a number of drawbacks. The first is the relatively low
frequency of occurrence of human error. Although there may be a deterministic
element to the occurrence of human error, it is very difficult to predict precisely
when a set of events or actions giving rise to an observable error will occur. So
there is no guarantee that human error will occur, even during extended periods of
observation.
The second drawback is the high costs associated with extended observation
and the subsequent data analysis. Even if it could be guaranteed that human error
would occur once in every 24 h period, for example, then gathering enough data
for 100 errors would require 2,400 h of recordings. Because analysis of recorded
data takes an order (or two) of magnitude longer than the recording, the time to
gather and to analyze such a large amount of data quickly becomes prohibitive.
The third drawback is the inherent adaptability of human behavior. One of the
characteristics of experienced operators is their ability to detect and recover from
potential erroneous actions. In other words, recovery is performed before
unwanted consequences arise. Unless verbal reports are also recorded, which can