Determining What to Study - Evaluation Methods in Biomedical Informatics - page 74

TABLE 3.3. Sample questions at differing levels of evaluation.
Level of evaluation Sample question

A lab report, guideline, email or What is the impact on lab report content of the

other component of the introduction of bedside terminals?

information being processed
The information resource as a How long does it take for lab results to get to the

whole bedside terminal?

A patient or case Are patients satisfied when their nurse uses a bedside

terminal?

A health professional Are nurses satisfied when they use a bedside terminal?

A clinical team How are communication patterns within the team

influenced by the introduction of bedside terminals?

Part of a healthcare delivery Is the throughput of the ICU affected by the

organization (e.g., ward) introduction of bedside terminals?

The entire healthcare delivery How has the introduction of bedside terminals affected

organization the rate of adverse events across the hospital, and

thus its indemnity position?

Choosing the Level of Evaluation
One of the fundamental choices required when planning any evaluation
study is the level or scale at which to focus the study. For example, in clin-
ical fields, this level can be one or more of the following: a lab report, guide-
line, email or other component of the information being processed or
communicated, the resource as a whole, a patient or case, a health profes-
sional such as a nurse or physician, a multidisciplinary clinical team, or part
or all of a healthcare delivery organization such as a hospital ward or the
whole hospital. To illustrate this, Table 3.3 shows some sample questions
that might prompt investigators to study each of these levels.
It is important to realize that logistical factors often require studies to be
conducted at higher levels of scale. This occurs when individual objects such
as patients or health professionals interact and thus cannot be separated
out for study. An example would be studying the impact of an antenatal
information kiosk by providing a password to half the women attending a
clinic. Because those women without a password could either “borrow” one
or look over the shoulders of women with a password using the kiosk, it
would be logistically difficult to restrict usage to the intended group. Even
if the “control” women failed to share passwords, women typically share
their experiences and information in an antenatal clinic; therefore, it would
be naïve to assume that if the kiosk were only made available to half the
women, the other half would be completely ignorant of its contents. Similar
arguments apply to studying the effect of an educational course for health
professionals or the impact of a new set of reference databases. These inter-
actions require that the investigator raise the level of the evaluation to focus
on groups rather than individuals.