Challenges of Evaluation in Biomedical Informatics - Evaluation Methods in Biomedical Informatics - page 2

edge (e.g., medical knowledge bases and decision-support systems). For
basic and clinical research applications, information resources include
genomic and proteomic databases, tools to analyze these data and correlate
genotypes with phenotypes, and methods for identifying patients eligible
for clinical trials.* In health education, there are Internet-based programs
for distance learning and virtual reality simulations for learning medical
procedures. There is clearly a range of biomedical information and com-
munication resources to evaluate.
To further complicate the picture, each information resource has many
aspects that can be evaluated. The technically minded might focus on inher-
ent characteristics, asking such questions as: “How many columns are there
per database table?” or “How many probability calculations per second can
this resource sustain?” Clinicians, researchers, and students might ask more
pragmatic questions, such as: “Is the information in this resource completely
up-to-date?” or “How much time must I invest in becoming proficient with
this resource, and will it do anything to help me personally?” Those with a
broader perspective might wish to understand the impact of these resources
on organization or management, asking questions such as: “How well does
this electronic patient record support clinical audit?” or “Will sophisticated
educational simulations change the role of the faculty in teaching?” Thus,
evaluation methods in biomedical informatics must address not only a
range of different types of information resources, but also a range of ques-
tions about them, from the technical characteristics of specific systems to
their effects on people and organizations.
In this topic, we do not exhaustively describe how each possible evalua-
tion method can be used to answer each kind of question about each kind
of information resource. Instead, we describe the range of techniques avail-
able and focus on those that seem most useful in biomedical informatics.
We introduce, in detail, methods, techniques, study designs, and analysis
methods that apply across a range of evaluation problems. The methods
introduced in this volume are applicable to the full range of information
resources directed at health care, biomedical research, and education. The
examples we introduce and discuss will come primarily from clinical and
educational application. While there is a great deal of interest in a rapidly
developing set of information resources to support bioinformatics and clin-
ical research, there is relatively little experience with the evaluation of these
tools.
In the language of software engineering, our focus is much more on soft-
ware validation than software verification. Validation means checking that
the “right” information resource was built, which involves both determin-
ing that the original specification was right and that the resource is per-
* In this volume, the term “bioinformatics” will be used to refer to the use of infor-
mation resources in support of biological research. In this parlance, bioinformatics
is a subset of the broader domain of biomedical informatics.