Determining What to Study - Evaluation Methods in Biomedical Informatics - page 66

and how it might be improved, but also potential broader concerns and
unexpected issues that may include user privacy and long-term issues
around user training and working relationships.
• Studying users while they carry out a list of predesigned tasks using the
information resource. Methods for studying users include watching over
their shoulder; video observation (sometimes with several video cameras
per user); think out-loud protocols (asking the user to verbalize their
impressions as they navigate and use the system); and automatic logging
of keystrokes, navigation paths, and time to complete tasks. 9
• Use of validated questionnaires to capture user impressions, often before
and after an experience with the system, with one example being the
Telemedicine Preparedness questionnaire. 10
• Specific techniques to explore how users might improve the layout or
design of the software. For example, to help understand what users think
of as a “logical” menu structure for an information resource, investiga-
tors can use the card-sorting technique. This entails listing each function
available on all the menus on a separate card and then asking users to
sort these cards into several piles according to which function seems to
go with which. 11
Depending on the aim of a usability study, it may suffice to employ a
small number of potential users. Nielsen has shown that, if the aim is only
to identify major software faults, the proportion identified rises quickly up
to about five or six users, then much more slowly to plateau at about 15 to
20 users. 11,12 Five users will often identify 80% of software problems.
However, investigators conducting such small studies, useful though they
may be for software development, cannot then expect to publish them in a
scientific journal. The achievement in this case is having found answers to
a very specific question about a specific software prototype. This kind of
local reality test is unlikely to appeal to the editors or readers of a journal.
By contrast, the results of formal laboratory function studies, which typi-
cally employ more users, are more amenable to journal publication.
Scenario 3: A Locally Developed Information Resource
Has Been Deployed Within an Organization, But No
One Really Knows How Useful It Is Proving to Be
The situation here is quite different from the preceding scenario. Here, the
system is already deployed in one part of the organization, so it has already
moved well beyond the prototype stage. The key issues are whether the
resource is being used, by whom, whether this usage is appropriate, and
what benefits the resource use is bringing to the organization. With refer-
ence to Tables 3.1 and 3.2, this scenario typically calls for field-user effect
studies.
Typical evaluation questions here might include: