Determining What to Study - Evaluation Methods in Biomedical Informatics - page 67

• Is the resource being used at all; if so, by whom?
• Are these the intended users, and if not, why not?
• Is the information being captured or communicated by the resource of
good quality (accurate, timely)?
• What are users' attitudes and beliefs about using the resource in day-to-
day work? Do these beliefs depend on the users' background, work
assignments, or role in the organization?
• Does the information resource appear to be causing any problems? What
are these problems, and how often do they occur?
• Do there appear to be benefits from the use of the resource? What are
they?
• Do the benefits of the resource derive from features unique to a limited
set of users or the specific areas where it has been deployed?
The evaluation approach under this scenario can employ a mixture
of study types using a mix of methods. Subjectivist studies could be useful
to get an initial impression about who uses the resource and what they
perceive as its benefits and problems. These studies may identify focused
areas for further more-detailed study; or if the objectives of the study
are limited, subjectivist studies may be an end in themselves. If the usage
or likely impact (positive or negative) of the resource is of interest,
further studies can be conducted using either objectivist or subjectivist
methods to answer in greater detail questions about the resource for the
organization. Studies of usage rates may be important if there is evidence
of limited user effect, and could be followed by usability studies and both
lab and field-based studies of resource function to identify where the
resource needs improvement. If the results of effect studies are promising,
it may be beneficial to study whether the need for the resource exists in
other parts of the organization where it is currently not available or is not
being used. If these needs assessment studies are positive, widespread
deployment of the system across the organization can take place, whereas
if the results of these studies are less promising, the resource may have only
limited utility.
Scenario 4: A Commercial Resource Has Been
Deployed Across a Large Enterprise, and There Is
Need to Understand Its Impact on Users
as Well as on the Organization
The type of evaluation questions that arise here include:
• In what fraction of occasions when the resource could have been used
was it actually used?
• Who uses it, why, are these the intended users, and are they satisfied with
it?