Determining What to Study - Evaluation Methods in Biomedical Informatics - page 75

Matching What Is Evaluated to the Type of
Information Resource
There are many types of information resources containing many functional
components. Clearly, it is impossible to generate an exhaustive list of every-
thing that can be studied for every kind of resource, but as previously men-
tioned, the appendices to this chapter provide samples of some of the issues
that can be addressed for a range of information resource components and
complete resources. Not all of these attributes can or should be measured
for every component or resource, and it often requires much thought about
the purpose of the evaluation itself to produce a relevant list of issues to
pursue. Because facilities for evaluation are always limited, it may be
helpful to rank the items listed in the appendices in the order of their likely
contribution to answering the questions the evaluation is intended to
resolve. Often, as discussed in Chapter 2, priorities are set not by the inves-
tigators, but by the stakeholders in the evaluation. The investigators' role is
then to initiate a process that leads to a consensus about what the priority
issues should be.
Matching How Much Is Evaluated to
the Stage in the Life Cycle
Evaluation, defined broadly, takes place throughout the resource develop-
ment cycle: from defining the need to monitoring the continuing impact of
a resource once it is deployed. The place of evaluation in the various devel-
opmental phases is illustrated in Figure 3.3. Different issues are explored,
at different degrees of intensity, at each stage of resource development.
Prior to any resource development, as discussed earlier, there may be
very active formal evaluation to establish needs. During the early phases of
actual resource development, informal feedback and exploration of proto-
types is associated with code development and debugging. A single proto-
type then emerges for more formal testing, with problems being fed back
to the development team. Eventually, it passes preset criteria of adequacy,
and its effects on users can be tested in a more formal way—though often
still under controlled “laboratory” conditions. Once safety is ensured and
there is reason to believe that the information resource is likely to bring
benefit, its impact can be studied in a limited field test prior to wider dis-
semination. Once disseminated, it is valuable to monitor the effects of the
resource on the institutions that have installed it and evaluate it for poten-
tial hazards that may only come to light when it is in widespread use—a
direct analogy with postmarketing surveillance of drugs for rare side effects.
Evaluation is integral to information resource development, and ade-
quate resources must be allocated for it when time and money are budgeted
for a development effort. Evaluation cannot be left to the end of a project.
However, it is also clear that the intensity of the evaluation effort should be