Challenges of Evaluation in Biomedical Informatics - Evaluation Methods in Biomedical Informatics - page 12

The task of verifying a program (obtaining proof that it performs all and
only those functions specified) using brute-force methods increases expo-
nentially according to the program's size and the complexity of the tasks it
performs. This is an “NP-hard” problem (see glossary). Put simply, to verify
a program using brute-force methods requires application of every combi-
nation of possible input data items and values for each in all possible
sequences. This entails at least n factorial trials of the program, where n is
the number of input data items.
A range of computer-based information resources has been applied to
biomedicine (Table 1.1), each with different target users, input data, and
goals. Even though they are increasingly commonplace, computer-based
information resources remain a relatively novel technology in biomedicine,
with a legacy of 30 to 40 years. With any relatively new technology, novel
challenges arise. For example, practitioners may not use a decision support
system until it has been shown to be valuable, but persons need to use the
system in order to demonstrate its value. We call this phenomenon the
“evaluation paradox.” Moreover, many information resources do not reach
their maximum impact until they are fully integrated with other informa-
tion resources that operate in the same work environment and until they
become part of routine work practice. 23
In some projects, the goals of the new information resource are not pre-
cisely defined. Developers may be attracted by technology and produce
applications of it without first demonstrating the existence of a clinical, sci-
entific, or educational problem that the application is designed to address. 15
An example was a conference entitled “Medicine Meets Virtual Reality:
Discovering Applications for 3D Multimedia” [our italics]. The lack of a
clear need for the information resource makes some biomedical informat-
ics projects difficult or impossible to evaluate, and we will see in later chap-
ters that understanding the need for a proposed information resource is
often the driving question of an evaluation study.

TABLE 1.1. Range of computer-based information resources in medicine.
Clinical data systems Clinical knowledge systems

Clinical databases Computerized textbooks (e.g., Scientific

American Medicine on CD-ROM)

Communications systems (e.g., picture Teaching systems (e.g., interactive multimedia

archiving and communication systems) anatomy tutor)

On-line signal processing (e.g., 24-hour Patient simulation programs (e.g., interactive

ECG analysis system) active acid-base metabolism simulator)

Alert generation (e.g., ICU monitor, drug Passive knowledge bases (e.g., MEDLINE

interaction system) bibliographic system)

Laboratory data interpretation Patient-specific advice generators (e.g.,

MYCIN antibiotic therapy advisor)

Medical image interpretation Medical robotics