Methodology III: Empirical Evaluation - Foundations for Designing User-Centered Systems - page 349

alternative hypothesis (H 1 ) and the null hypothesis are mutually exclusive, so in
our case it would be that there is a difference between the usability of application
A and of application B. The multi-dimensional nature of usability means that
experiments are often set up with several pairs of hypotheses, one for each
dimension of usability. Some user studies might have multiple hypotheses to be
tested.
13.2.3 Identifying the Dependent and Independent Variables
An independent variable is a factor that is independent of user behavior, and which
can be varied by the person carrying out the evaluation (or the experiment, more
generally). The dependent variable is the thing that depends on the user's behavior,
or on the changes in the independent variable(s). Within user centered design, the
sorts of things that would often be considered as independent variables include the
type of input device (e.g., touch screen or keyboard), or different web page
designs. The dependent variables are often more limited, however, to those things
that are generally taken to measure the usability of a system (e.g., efficiency,
effectiveness, satisfaction, ease of learning, and workload).
13.2.4 What Type of Evaluation: Formative or Summative?
There are two basic types of user-based evaluation: formative and summative.
Each of these has a different purpose, and takes place at a different stage of
development, as described below. They each help to remove different types of
uncertainty, and to reduce the risk that the system will not be usable, or will be
unacceptable to the end users.
Formative evaluation can take place at any point during development. It is used
to help designers refine and form their designs. The focus of formative evaluation
is to identify problems and potential solutions. In this type of evaluation the
desired result is an indication of any problems that there may be in using the
system, possibly with some indication of their frequency of occurrence. The
designers can then use these frequencies to help rate the severity of the problems
so a decision can be made about which problems should be fixed first.
Summative evaluation is concerned with assessing the success of the finished
system or product, summarizing its overall impact and effectiveness. It is often
used to test for any fixes that may be needed before the system is released, and to
assess future releases. The end result may be some sort of usability score, and
individual organizations may have their own particular threshold values for
acceptability. One useful metric is to require that novice users of a new system are
able to demonstrate performance levels that are some predetermined percentage of
expert levels on the same system. The performance levels are measured using a set