Evaluation as a Field - Evaluation Methods in Biomedical Informatics - page 35

mated task was carried out properly or whether the advice provided was
clinically or scientifically valid. Because society does not routinely expect
critics to agree, the potential lack of interobserver agreement does not
invalidate this approach. Although they tend to be more informal and tend
to reflect less direct experience with the resource than would be the case
in a complete “art criticism” study, software reviews that routinely appear
in technical journals and magazines are examples of this approach in
common practice.

Professional Review Approach

The professional review approach is the well-known “site visit” approach
to evaluation. It employs panels of experienced peers who spend several
days in the environment where the resource is deployed. Site visits often
are directed by a set of guidelines specific to the type of project under study
but sufficiently generic to accord the reviewers a great deal of control over
the conduct of any particular visit. They are generally free to speak with
whomever they wish and to ask of these individuals whatever they consider
important to know. They may request documents for review. Over the
course of a site visit, unanticipated issues may emerge. Site visit teams fre-
quently have interim meetings to identify these emergent questions and
generate ways to explore them. As a field matures, it becomes possible to
articulate formal review criteria that could be the focus of site visits, sup-
porting application of the professional review approach. In biomedical
informatics, the evolving evaluation criteria for computer-based patient
records 22

is one example of such guidelines.

Responsive/Illuminative Approach

The responsive/illuminative approach seeks to represent the viewpoints of
those who are users of the resource or an otherwise significant part of the
environment where the resource operates. 23 The goal is understanding, or
“illumination,” rather than judgment. The methods used derive largely from
ethnography. The investigators immerse themselves in the environment
where the resource is operational. The designs of these studies are not
rigidly predetermined. They develop dynamically as the investigators' expe-
rience accumulates. The study team begins with a minimal set of orienting
questions; the deeper questions that receive thorough ongoing study evolve
over time. Many examples of studies using this approach can be found in
the literature of biomedical informatics. 24-27

Self-Test 2.1
The answers to these exercises appear at the end of this chapter.