Errors: An Inherent Part of Human-System Performance - Foundations for Designing User-Centered Systems - page 286

be used to try and identify these recoveries, it can be difficult to detect where a
recovery has taken place.
The fourth drawback is that the presence of an outside observer may affect the
operator's behavior. In particular, operators may be more careful than usual if they
know their performance will be recorded for subsequent analysis. This may reduce
the frequency of occurrence of human error, thereby requiring even longer periods
of observation to gather enough data.
10.2.3 Archive Data
The third method is to use existing available data. Archive accident and incident
reports, for example, have been successfully used within human error research
(e.g., Pew et al. 1981 ; Rasmussen 1980 ; Wagenaar and Groeneweg 1987 ; Woods
1984 ). As with the other data collection methods, there are several pros and cons to
using archive data (Chappell 1994 ).
The first advantage of archive data is that the data are real in that they are
typically provided by participants in the incident. The second is that there are large
numbers of observations available (in contrast to accident data). The third is that
the data have high ecological validity because it relates to real incidents that
occurred under normal operating conditions. Finally, the cost of collecting the data
is generally low, because the data already exist.
The main disadvantage of using archive data is that they usually have not been
gathered for the specific purpose of the investigation at hand. The data therefore
often have to be re-ordered and possibly re-represented before it can be appro-
priately analyzed. The second disadvantage is that the detail in the reports may not
have been validated. The third is that the data may be subject to reporter biases, in
terms of who reports, and the information that gets reported which may be biased
by selective retrieval and rational reconstruction (Ericsson and Simon 1993 ).
10.2.4 Selecting the Most Appropriate Data Collection
Method
The aim of each of the methods described above is to generate data that can be
used to develop theories and models of human performance. Over the last
20 years, since the earliest work there has been a shift towards an increased use of
real world data (e.g., Hutchins 1995 ), especially when studying expert perfor-
mance in dynamic environments. The ideal method, however, would combine the
contextual richness of real world situations with some of the experimental control
of laboratory conditions. The work on the HEAP (Hollnagel et al. 1996 ) comes
close to this ideal. The HEAP involved observing teams of operators running a
high fidelity nuclear power plant simulator. Even with the HEAP work, however,