Evaluation as a Field - Evaluation Methods in Biomedical Informatics - page 42

data confidentiality. This concern meant that the network installations
had to be stand-alone and would not have any connectivity to other
networked environments. Furthermore, because of differences in
networking technology, it was not practical to directly link the T-
HELPER network with the existing computer systems at each site.
Instead, special-purpose point-to-point links were made to the regis-
tration and laboratory systems at ABC and XYZ.
IV. System evaluation : The primary goal of the evaluation was to
determine if the T-HELPER II system increased the rate of accrual
of eligible and potentially eligible patients to clinical protocols. In
addition, the evaluation group examined physicians' knowledge
and acceptance of the computer as an adjunct to patient care and in
collaborative research. It then correlated these attitudes with use of
and compliance with the T-HELPER systems.
Why It May Not Work Out as the topics Suggest
If we did possess the “evaluation machine”, described earlier in this chapter,
life would be easier, but not perfect. We would design and implement our
information resources, let the machine tell us what would have happened
had the resources not been implemented, and then compare the two sce-
narios. The difference would, of course, be a measure of the “effect” of the
resource; but there may be many other factors, not detectable by the
machine, that are important to investigate. It has been shown throughout
this chapter how the unavailability of the evaluation machine, and other
factors, have led many creative individuals to devise an assortment of
evaluation approaches. Because of the richness and diversity of these
approaches, it is safe to say that an informative study probably can be
designed to address any question of substantive interest in informatics.
However, even the best-designed studies do not work out as planned.
One of the worst failure scenarios occurs when apparently meritorious
studies—studies of important issues and that are well designed—are not
carried out. Resistance to the conduct of a study can develop either before
the study is actually begun or during its progress. There are two principal
reasons why this occurs. In both cases, it can be seen that attention to the
roles in an evaluation (Figure 2.5) and the importance of advance negotia-
tion of an evaluation contract can both signal problems and help the study
designers navigate through them.
• Sometimes we would rather not know or “ fear of the clear ”: Some, perhaps
many, resource developers believe they have more to lose than to gain
from a thorough ongoing study of their information resource. This belief
is more likely to occur in the case of a resource perceived to be func-