Evaluating Information Visualizations - Information Visualization: Human-Centered Issues and Perspectives - page 26

The experiments or studies involve a rigorous process of hypothesis development,
identification and control of the independent variables, observation and measurement
of the dependent variables, and application of statistics which enable the declaration
of the confidence with which the results can be taken. In these formal studies or con-
trolled evaluations, the experimenter controls the environment or setting, manipulates
chosen factor(s) or variable(s) - the independent variable(s) - in order to be able to
measure and observe the affect this manipulation has on one or more other factors -
the dependent variable(s). Ideally no other factors change during the experiment.
Once the changes to the dependent variables have been measured, statistical methods
can be applied to understand the relative importance of the results. Done with suffi-
cient thoroughness, this process can arrive at facts about which we can be relatively
certain. The application of this scientific process will to try to reduce the overall com-
plexity by fine tuning particular questions or hypotheses, using these hypotheses to
allow one to cull some of the complexity by trying to eliminate as many of the extra-
neous variables as possible. Traditionally experiments of this type are used to shed
light on cause and effect relationships; that is, to discover whether changes in some
factor result in changes to another factor.
This idea that we can observe simpler, more manageable subsets of the full com-
plex process is appealing, and it is clear from centuries of experiments that much can
be learnt in this manner.

4.1 Quantitative Methodology

Since quantitative empirical evaluations have evolved over the centuries the method-
ology has become relatively established (Figure 2). This brief overview is included
for completeness; the interested reader should refer to the many good topics on this
subject [15, 17, 33]. This methodology includes:

• Hypothesis Development: Much of the success of a study depends on asking an

interesting and relevant question. This question should ideally be of interest to
the broader research community, and hopefully answering it will lead to a deeper
or new understanding of open research questions. Commonly the importance of
the study findings results from a well thought through hypothesis, and formulat-
ing this question precisely will help the development of the study.

• Identification of the Independent Variables: The independent variables are

the factors to be studied which may (or may not) affect the hypothesis. Ideally
the number of independent variables is kept low to provide more clarity and pre-
cision in the results.

• Control of the Independent Variables: In designing the experiment the ex-

perimenter decides the manner in which the independent variables will be
changed.

• Elimination of Complexity: In order to be clear that it is actually the change in

the independent variable that caused the study's result, it is often the case that
other factors in the environment need to be controlled.

• Measurement of the Dependent Variables: Observations and measurements

are focused on the dependent variables as they change or do not change in