Challenges of Evaluation in Biomedical Informatics - Evaluation Methods in Biomedical Informatics - page 10

in order to demonstrate the causal link between changes in clinical prac-
tice and changes in patient outcome.
In some cases, changes in work processes resulting from introduction of
an information resource are difficult to interpret because the resulting
improved information management or decision-taking merely clears one
log jam and reveals another. An example of this situation occurred during
the evaluation of the ACORN ( A dmit to C CU OR N ot) chest pain
decision-aid, designed to facilitate more rapid and accurate diagnosis of
patients with acute ischemic heart disease in the emergency room. 14
Although ACORN allowed emergency room staff to rapidly identify
patients requiring admission to the cardiac care unit (CCU), it uncovered
an existing problem: the lack of beds in the CCU and delays in transferring
other patients out of them. 15
In the clinical domain, the processes of decision-making are complex and
have been studied extensively. 16,17 Clinicians make many kinds of deci-
sions—including diagnosis, monitoring, choice of therapy, and prognosis—
using incomplete and fuzzy data, some of which are appreciated intuitively
and not recorded in the clinical notes. If an information resource generates
more effective management of both patient data and medical knowledge,
it may intervene in the process of medical decision-making in a number of
ways; consequently, it may be difficult to decide which component of the
resource is responsible for the observed changes. Often this does not
matter, but if one component is expensive or hard to create, understanding
why and how the resource brings benefit becomes important. Understand-
ing why a resource brings benefit can be much more difficult than deter-
mining the magnitude of this benefit.
Data about individual patients typically are collected at several locations
and over periods of time ranging from an hour to decades. Unfortunately,
clinical notes usually contain only a subset of what was observed and seldom
contain the reasons why actions were taken. 18 Because reimbursement agen-
cies often have access to clinical notes, the notes may even contain data
intended to mislead chart reviewers or conceal important facts from the
casual reader. 19,20 Thus, evaluating an electronic medical record system by
examining the accuracy of its contents may not give a true picture.
There is a general lack of “gold standards” in health care. For example,
diagnoses are rarely known with 100% certainty, partly because it is uneth-
ical to do all possible tests in every patient, or even to follow up patients
without good cause, and partly because of the complexity of the human
biology. When attempting to establish a diagnosis or the cause of death, even
if it is possible to perform a postmortem examination, correlating the
observed changes with the patients' symptoms or findings before death may
prove impossible. Determining the “correct” management for a patient is
even harder, as there is wide variation even in so-called consensus opin-
ions, 21 which is reflected in wide variations in clinical practice even in neigh-
boring areas. An example is the use of endotracheal intubation in patients