Determining What to Study - Evaluation Methods in Biomedical Informatics - page 60

In the paragraphs above, the term “sampling” has been introduced for
both tasks and users. The technical issues of sampling will not be discussed
here; however, it is important to establish that, in real evaluation studies,
tasks and users are always sampled from some real or hypothetical popu-
lation. Sampling occurs, albeit in different ways, in both subjectivist and
objectivist studies. Sampling of users and tasks are major challenges in eval-
uation study design. It is never possible, practical, or desirable to try to study
everyone doing everything possible with an information resource. Choos-
ing representative samples of users and tasks will challenge the resource
with a reasonable spectrum of what is expected to occur in normal prac-
tice. Under some circumstances, it is also important to know what will
happen to the resource's usability, functions, or impact if the resource
encounters extremes of user ability, data quality, disease incidence, or task
structure. For these purposes, modeling techniques can sometimes be used
to simulate what might happen under these extremes; or extreme atypical
circumstances can be deliberately created by the investigator to provide a
stress test of the resource. Sampling issues and their implications will be
returned to on several occasions in Chapters 4 through 8.
It is perhaps extreme to state that every evaluation approach, of the
eight listed in Chapter 2, can apply to all of the nine study types introduced
in Tables 3.1 and 3.2. There is certainly potential to use both objectivist
and subjectivist approaches across the spectrum of study types. At the
two extremes, for example, both “need validation” studies and “health-
care impact” studies provide many opportunities to apply both sub-
jectivist and objectivist approaches. Some study types are predisposed
toward specific approaches. For example, design validation studies invite
use of the professional review approach. The following sections of this
chapter expand on a selected set of the study types presented in Tables 3.1
and 3.2.
Self-Test 3.1
For each of the following hypothetical evaluation scenarios, list which of
the nine types of studies listed in Table 3.1 they include. Some scenarios
may include more than one type of study.
1. An order communications system is implemented in a small hospital.
Changes in laboratory workload are assessed.
2. The developers of the order communications system recruit five
potential users to help them assess how readily each of the main functions
can be accessed from the opening screen and how long it takes users to
complete them.
3. A study team performs a thorough analysis of the information
required by psychiatrists to whom patients are referred by a community
social worker.