Determining What to Study - Evaluation Methods in Biomedical Informatics - page 72

tion resources, often focusing on decision support tools that pose some of
the most extreme challenges. Other topics have explored more technical
health technology assessment or organizational approaches to evaluation
methods. 23-26
The Special Issue of Safety
Before disseminating any biomedical information resource that stores and
communicates data or knowledge and is designed to influence real-world
practice decisions, it is important to check that it is safe when used as
intended. In the case of new drugs, it is a statutory duty of developers to
perform extensive in vitro testing and in vivo testing in animals before any
human receives a dose. For information resources, equivalent safety tests
might include measuring how fast the information resource functions com-
pared to current procedures and estimating how often it corrupts or
retrieves erroneous data or furnishes incorrect advice. It may be necessary
to repeat these measurements following any substantial modifications to
the information resource, as the correction of errors may itself generate
more errors or uncover previously unrecognized problems.
Examining an information resource for safe operation is particularly
important when evaluating those that attempt to directly influence the deci-
sions made by front-line health professionals. Ensuring that such a resource
is safe requires measurement of how often it gives poor advice using data
representative of patients in whose management it is intended to assist and
comparing the advice given with the decisions made by current decision-
makers, as seen by expert judges.
The advice or output generated by most information resources depends
critically on the quality and quantity of data available to it, and thus, at least
partly, on the manner in which the resources are used by practitioners. Prac-
titioners who are untrained, in a hurry, or exhausted at 3:00 am may all
obtain less-reliable output because of the poor quality of data they input.
Thus, to generate valid results, functional tests must put the resources in
actual users' hands or in the hands of people with similar knowledge, skills,
and experience if real users are not available.
Evaluation Strategy: Deciding What and
How Much to Study
Matching What Is Evaluated to Decisions
That Need to be Made
A recurrent and troublesome issue in many evaluation studies is not
choosing what to measure, or even which methods to use in a specific
study, but how to balance the often-competing demands of the different