Biomedical Engineering Reference
In-Depth Information
TABLE 1.2. Possible questions that may arise during evaluation of a medical infor-
mation resource.
Questions about the resource
Questions about the impact of the resource
Is there a clinical need for it?
Do people use it?
Does it work?
Do people like it?
Is it reliable?
Does it improve users' efficiency?
Is it accurate?
Does it influence the collection of data?
Is it fast enough?
Does it influence users' decisions?
Is data entry reliable?
For how long do the observed effects last?
Are people likely to use it?
Does it influence users' knowledge or skills?
Which parts cause the effects?
Does it help patients?
How can it be maintained?
Does it change use of healthcare facilities?
How can it be improved?
What might ensue from widespread use?
evaluations can readily fall prey to the same delusion. In addition, it may
be unclear what kind of test material or users to include in such a study.
Often study designers are faced with trade-offs between selecting test mate-
rial or users with high fidelity to the real-world practice—those who can
help achieve adequate experimental control in the study and those who are
available and willing to participate. Finally, one of the more important
determinants of the results of an evaluation study is the manner in which
test material is abstracted and presented to users. For example, one would
expect differing results in a study of an information resource's accuracy
depending on whether the test data were abstracted by the resource devel-
oper or by the intended users.
There are many reasons for performing evaluations, ranging from assess-
ing a student's mastery of new subject matter to making national health
policy decisions or understanding the implications of a technical change on
resource performance. There are inevitably many actors in evaluation
studies (see Figure 1.1), including information resource developers, users,
and patients, all of whom may have different perspectives on which ques-
tions to ask and how to interpret the answers. Table 1.2 lists some sample
questions that may arise about the resource itself and its impact on users,
patients, and the healthcare system. The multiplicity of possible questions
creates challenges for the designers of evaluation studies. Any one study
inevitably fails to address some questions and may fail to answer adequately
some questions that are explicitly addressed.
Addressing the Challenges of Evaluation
No one could pretend that evaluation is easy. This entire topic describes
ways that have been developed to solve the many problems discussed in
this chapter. First, evaluators should recognize that a range of evaluation
Search WWH ::
Custom Search