Challenges of Evaluation in Biomedical Informatics - Evaluation Methods in Biomedical Informatics - page 11

with severe head injuries, which varied from 15% to 85% among teaching
hospitals, even within California.** In addition, getting busy physicians to
give their opinions about the correct management of patients for compari-
son with a decision support system's advice may take as much as a full year. 22
Healthcare providers practice under strict legal and ethical obligations
to give their patients the best care available, to them no harm, keep them
informed about the risks of all procedures and therapies, and maintain con-
fidentiality. These obligations often impinge on the design of evaluation
studies. For example, because healthcare workers have imperfect memories
and patients take holidays and participate in the unpredictable activities of
real life, it is impossible to impose a strict discipline for data recording, and
study data are often incomplete. Before any field studies of an information
resource can be undertaken, healthcare workers and patients are entitled
to a full explanation of the possible benefits and disadvantages of the
resource. Even if it is a randomized trial, since half the patients or health-
care workers will be allocated randomly to the intervention group, all need
to be counseled and give their consent prior to being enrolled in the study.
Similar challenges to evaluation apply to the educational domain. For
example, students in professional schools are in a “high-stakes” environ-
ment where grades and other measures of performance can shape the tra-
jectory of their future careers. Students will be understandably attracted to
information resources they perceive as advantageous to learning and averse
to those they perceive to offer little or no benefit at a great expenditure of
time. Randomization or other means of arbitrary assignment of students to
groups, for purposes of evaluative experiments, may be seen as anathema.
Similar feelings may be seen in biological research, where competitiveness
among laboratories may mitigate against controlled studies of information
resources deployed in these settings.
Problems Deriving from the Complexity of
Computer-Based Information Resources
From a computer-science perspective, an important goal of evaluating an
information resource is to verify that the program code faithfully performs
those functions it was designed to perform. One approach to this goal is to
try to predict the resource's function and impact from knowledge of the
program's structure. However, although software engineering and formal
methods for specifying, coding, and testing computer programs have
become more sophisticated, programs of even modest complexity challenge
these techniques. Since we cannot logically or mathematically derive a
program's function from its structure, we often are left with exhaustive
“brute-force” techniques for program verification.
** Personal communication with Bryan Jennett, 1990.