Evaluating Information Visualizations - Information Visualization: Human-Centered Issues and Perspectives - page 29

External Validity: Can we generalize the study results to other people/places/
times? External validity is concerned with the extent to which the result of a study
can be generalized. If a study has good internal and construct validity the results apply
to the setting, time, and participants being studied. The extent to which the results
apply beyond the immediate setting, time and participants depends, for participants,
on the participant sample and the population from which it was drawn. For instance,
in practice it is common to draw participants from the geographic region in which the
study is run. Does this mean that the results only apply to people from that region? If
culture has a possible impact on the results, they may not generalize. If one addresses
the need to include cultural variation by recruiting participants from different cultures
from a university's foreign students, one might have at least partially addressed the
need to run the study across cultural variations but now have limited the demographic
to university students which may introduce its own skew. Understanding the popula-
tion to which one would like to be able to generalize the study results and successfully
obtaining an appropriate participant sample is a difficult issue. This does not mean we
can not learn from more specific participant samples. It does mean that reporting the
demographics of the sample and being cautious about generalizations is important.
Participant sample choice is just one factor influencing external validity. Setting in-
cludes other factors such as noise, interruption, and distractions. Possible temporal
factors include events that occurred before or are anticipated after the experiment.
Ecological Validity: Ecological validity discussions focus on the degree to which the
experimental situation reflects the type of environment in which the results will be
applied. This concept relates strongly to McGrath's concept of realism. It is distinct
from the idea of external validity, in that external validity is concerned with whether
the experimental results generalize to other situations, while ecological validity is
concerned with how closely the experimental settings matches the real setting in
which the results might be applied. Thus it is possible to have good ecological valid-
ity; the study is conducted on site, but that the results are applicable only to that site.
This would indicate poor external validity in that the results do not generalize beyond
the specific setting.

4.3 Quantitative Studies Summary Remarks

The number of quantitative studies in information visualization is increasing. Early
examples include the series of studies done by Purchase and her collaborators that
examine the impact of graph drawing aesthetics on comprehension and usability [58,
59, 60, 61]. Dumais et al. [16] explored use of context techniques in web search. For-
lines et al. [23] looked at the effect of display configuration on relationship between
visual search and information visualization tasks. Recently, Willet et al. [81] studied
embedding information visualizations in widgets.
Quantitative experiments have formed the backbone of experimental science and it
is to be expected that they will continue to do so. However, it is relatively easy to find
fault in any given experiment because all factors can not usually be completely con-
trolled. If they are completely controlled, external and ecological validity can be im-
pacted. This is particularly true for studies involving humans. Designing and working
with experiments is often a matter of making choices about what factors are important
and understanding the strengths and limitations of any given study and its results. As