Challenges of Evaluation in Biomedical Informatics - Evaluation Methods in Biomedical Informatics - page 8

For example, one large laboratory information system company saw $90
million wiped off its market value in 2003, when the Food and Drug Admin-
istration (FDA) published a warning about a fault in their product on the
FDA home page. 8 Developers of these resources will need to increasingly
comply with a comprehensive schedule of testing and monitoring proce-
dures, which may form an obligatory core of evaluation methods in the
future.
Biomedicine is well known to be a complex domain, with students spend-
ing a minimum of seven years to gain professional certification in medicine.
It is not atypical for a graduate student in molecular biology to spend five
years in pursuit of a PhD and follow this with a multi-year postdoctoral
training experience. In the United States, professional training in other
fields, such as nursing and pharmacy, has lengthened in recognition of the
nature and complexity of what is to be learned. For example, a single inter-
nal medicine textbook contains approximately 600,000 facts, 9 and practic-
ing medical experts have as many as two million to five million facts at their
fingertips. 10 Medical knowledge itself 9 and methods of healthcare delivery
and research change rapidly, so the goal posts for a biomedical information
resource may move significantly during the course of an evaluation study.
The problems that biomedical information resources assist in solving are
complex, highly variable, and difficult to describe. Patients often suffer from
multiple diseases, which may evolve over time and at differing rates, and
they may undergo a number of interventions over the course of the study
period. Two patients with the same disease can present with very different
sets of clinical signs and symptoms. There is variation in the interpretation
of patient data among medical centers. What may be regarded as an
abnormal result or an advanced stage of disease in one setting may pass
without comment in another because it is within their laboratory's normal
limits or is an endemic condition in their population. Thus, simply because
an information resource is safe and effective when used in one center on
patients with a given diagnosis, one is not entitled to prejudge the results
of evaluating it in another center or in patients with a different disease
profile.
Biological researchers have largely solved the very complex problem of
understanding genome structure, but now they face an even more complex
problem in understanding genome function: how gene expression affects
health and disease and the implications of person-to-person variation in
genetic structure. Biomedical information resources are challenged to help
scientists manage the enormous amounts of data—three billion DNA base-
pairs in the genome, and approximately 30,000 genes whose structure can
vary across billions of people—that are now of fundamental interest to bio-
logical research. In education, the challenges encountered by students striv-
ing to be clinicians and researchers are both profound and idiosyncratic.
The subject matter itself is sophisticated and changing rapidly. What a
person can learn is largely determined by what that person already knows