Evaluation as a Field - Evaluation Methods in Biomedical Informatics - page 26

be sensitive to the distinction between what is necessary (to inform key
decisions) versus what is “merely” interesting. 5
3. Look for intended and unintended effects . Whenever a new informa-
tion resource is introduced into an environment, there can be many conse-
quences. Only some of them relate to the stated purpose of the resource.
During a complete evaluation, it is important to look for and document
effects that were anticipated, as well as those that were not, and continue
the study long enough to allow these effects to manifest. The literature of
innovation is replete with examples of unintended consequences. During
the 1940s, rural farmers in Georgia were trained and encouraged to pre-
serve their vegetables in jars in large quantities to ensure they would have
a balanced diet throughout the winter. The campaign was so successful that
the number of jars on display in the farmers' homes became a source of
prestige. Once the jars became a prestige factor, however, the farmers were
disinclined to consume them, so the original purpose of the training was
subverted. 6 On a topic closer to home, the QWERTY keyboard became a
universal standard even though it was actually designed to slow typing out
of concern about jamming the keys of a manual typewriter, a mechanical
device that has long since vanished. 7
4. Study the resource while it is under development and after it is deployed.
In general, the decisions evaluation can facilitate are of two types. Forma-
tive decisions are made for the purpose of improving an information
resource. These decisions usually are made while the resource is under
development, but they can also be made after the resource is deployed.
Summative decisions made after a resource is deployed in its envisioned
environment deal explicitly with how effectively the resource performs in
that environment. It can take many months, and sometimes years, for a
deployed resource to stabilize within an environment. Before conducting
the most useful summative studies, it may be necessary for investigators to
allow this amount of time to pass.
5. Study the resource in the laboratory and in the field. Completely dif-
ferent questions arise when an information resource is still in the labora-
tory and when it is in the field. In vitro studies, conducted in the developer's
laboratory, and in situ studies, conducted in an ongoing clinical or educa-
tional environment, are both important aspects of evaluation.
6. Go beyond the developer's point of view. The developers of an infor-
mation resource usually are empathic only up to a point and often are not
predisposed to be detached and objective about their system's performance.
Those conducting the evaluation often see it as part of their job to get close
to the end-users and portray the resource as the users experience it. 8
7. Take the environment into account. Anyone who conducts an evalua-
tion study must be, in part, an ecologist. The function of an information
resource must be viewed as an interaction between the resource, a set of
“users” of the resource, and the social/organizational/cultural “context,”