Determining What to Study - Evaluation Methods in Biomedical Informatics - page 68

• Does using the resource improve influence information quality or
communication?
• Does using the resource influence user knowledge or skills?
• Does using the resource improve their work?
• For clinical information resources, does using the resource change out-
comes for patients?
• How does the resource influence the whole organization and relevant
subunits?
• Do the overall benefits and costs or risks differ for specific groups of
users, departments, and the whole organization?
• How much does the resource really cost the organization?
• Should the organization keep the resource as it is, improve it, or replace
it?
• How can the resource be improved, at what cost, and what benefits would
result?
To each of the above questions, one can add “Why or why not?” to get a
broader understanding of what is happening because of use of the resource.
This evaluation scenario, suggesting a problem impact study, is often what
people think of first when the concept of evaluation is introduced. However,
it has been shown in this chapter that it is one of many evaluation scenar-
ios, arising relatively late in the life cycle of an information resource. When
these impact-oriented evaluations are undertaken, they usually result from
a realization by stakeholders who have invested significantly in an infor-
mation resource, that the benefits of the resource are uncertain and there
is a need to justify recurring costs. These stakeholders usually vary in the
kind of evaluation methods that will convince them of the impacts that the
resource is or is not having. Many such stakeholders will wish to see quan-
tified indices of benefits or harms stemming from the resource—for
example, the number of users and daily uses, the amount the resource
improves productivity or reduces costs, or perhaps other benefits such as
reduced waiting times to perform key tasks or procedures, lengths of hos-
pital stay, or occurrence of adverse events. Such data are collected through
the kind of objectivist studies discussed in Chapters 7 and 8 and the rele-
vant economic methods described in Chapter 11. Other stakeholders may
prefer to see evidence of perceived benefit and positive views of staff, in
which case staff surveys, focus groups, and unstructured interviews may
prove the best evaluation methods. Often, a combination of many methods
is necessary to extend the goal of the investigation from understanding what
impact the resource has to why this impact occurs—or fails to occur.
If the investigator is pursuing objectivist methods, deciding which of the
possible effect variables to include in an impact study and developing ways
to measure them can be the most challenging aspect of an evaluation study
design. (These and related issues receive the attention of five full chapters
of this topic.) Investigators usually wish to limit the number of effect mea-