Evaluating Information Visualizations - Information Visualization: Human-Centered Issues and Perspectives - page 27

Fig. 2. A simple schematic of the traditional experimental process.

response to the manipulation of the independent variable. The aspects to be
measured are often called metrics. Common metrics include: speed, accuracy,
error rate, satisfaction, etc.

• Application of Statistics: The results collected can then be analysed through the

application of appropriate statistics. It is important to remember that statistics
tell us how sure we can be that these results could (or could not) have happened
by chance. This gives a result with a relative degree of certainty. There are many
good references such as Huck [33].

These steps sound deceptively simple but doing them well requires careful and rigor-
ous work. For instance, it is important that the study participants are valued, that they
are not over-stressed, and that they are given appropriate breaks, etc. Also, exactly
what they are being asked to do must be clear and consistent across all participants in
your study. Since small inconsistencies such as changes in the order of the instruc-
tions can affect the results, the common recommendation is that one scripts the expla-
nations. Perhaps most importantly, to eliminate surprises and work out the details, it is
best to pilot - run through the experiment in full - repeatedly.

4.2 Quantitative Challenges

Even though these types of experiments have been long and effectively used across all
branches of science, there remain many challenges to conducting a useful study. We
mention different types of commonly-discussed errors and validity concerns and re-
late these to the McGrath's discussion as outlined in Section 3. In this discussion we
will use a simple, abstract example of an experiment that looks at the effect of two
visualization techniques, VisA and VisB, on performance in search. There are several
widely discussed issues that can interfere with the validity of a study.
Conclusion Validity: Is there a relationship? This concept asks whether within the
study there is a relationship between the independent and the dependent variables.
Important factors in conclusion validity are finding a relationship when one does not
exist (type I error) and not finding a relationship when one does exist (type II error).