Evaluating Information Visualizations - Information Visualization: Human-Centered Issues and Perspectives - page 22

3 Choosing an Evaluation Approach

A recent call for papers from the information visualization workshop, Beyond Time
and Errors (BELIV06) held at Advanced Visual Interfaces 2006, stated that “ Con-
trolled experiments remain the workhorse of evaluation but there is a growing sense
that information visualization systems need new methods of evaluation, from longitu-
dinal field studies, insight based evaluation and other metrics adapted to the percep-
tual aspects of visualization as well as the exploratory nature of discovery ” [7]. The
purpose of this section is to encourage people to consider more broadly what might be
the most appropriate research methods for their purposes. To further this purpose a
variety of types of empirical research that can be usefully conducted are briefly out-
lined and these differing types are discussed in terms of their strengths and weak-
nesses. This discussion draws heavily from McGrath's paper Methodology Matters
[50] that was initially written for social scientists. However, while social scientists
work towards understanding humans as individuals, groups, societies and cultures, in
information visualization - similarly to HCI - we are looking to learn about how
information visualizations do or do not support people in their information tasks
and/or how people conduct their information related tasks so that visualization can be
better designed to support them. To gain this understanding we sometimes study peo-
ple using information visualization software and sometimes it may be important to
study people independently of that software, to better understand the processes we are
trying to support.
There are some commonalities to all studies. They all must start with some ques-
tion or questions that will benefit from further study. Also, they all must relate their
research questions to the realm of existing ideas, theories and findings. These ideas,
theories, and concepts are needed to relate the new study to existing research. For
example, the results from a new study might be in contrast to existing ideas, in
agreement with existing ideas, or offer an extension of or variation to existing ideas.
A study must also have a method. This is what this section is about - possible types of
empirical methodologies.
All methods offer both advantages and disadvantages. One important part of em-
pirical research is choosing the most appropriate research methods for your content,
your ideas, and your situation. The fact that methods both provide and limit evidence
suggests that making use of a wide variety of methodologies will, in time, strengthen
our understandings. Thus, both conducting a greater variety of studies and encourag-
ing this by publishing research that employs a greater variety of methodologies will
help to develop a better understanding of the value of information visualization and its
potential in our communities.
When conducting a study there are three particularly desirable factors: generaliza-
bility, precision, and realism [50]. Ideally, one would like all of these factors in one's
results. However, existing methodologies do not support the actualization of all three
simultaneously. Each methodology favours one or two of these factors, often at the
expense of the others; therefore the choice of a methodology for a particular goal is
important. To define these terms (as used in McGrath [50]):

Generalizability : a result is generalizable to the extent to which it can apply to
other people (than those directly in the study) and perhaps even extend to other
situations.

•