Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 43

• Areimprovementsbeingmadefromonedesigniterationtothenext?

• What usability issues can you expect to remain after the product is

launched?

The most appropriate situation to run a formative usability study is when an
obvious opportunity to improve the design presents itself. Ideally, the design pro-
cess allows for multiple usability evaluations. If there's no opportunity to impact
the design, then running a formative test is probably not a good use of time or
money. Generally, though, selling the value of formative usability shouldn't be a
problem. Most people will see the importance of it. The biggest obstacles tend to
be a limited budget or time rather than a failure to see the value.

3.1.2 Summative Usability
Continuing with our cooking metaphor, summative usability is about evaluat-
ing the dish after it comes out of the oven. The usability specialist running a
summative test is like a food critic who evaluates a few sample dishes at a res-
taurant or perhaps compares the same meal in multiple restaurants. The goal
of summative usability is to evaluate how well a product or piece of functional-
ity meets its objectives. Summative testing can also be about comparing several
products to each other. Although formative testing focuses on identifying ways
of making improvements, summative testing focuses on evaluating against a set
of criteria. Summative usability evaluations answer these questions:

• Didwemeettheusabilitygoalsoftheproject?

• Whatistheoverallusabilityofourproduct?

• Howdoesourproductcompareagainstthecompetition?

• Havewemadeimprovementsfromoneproductreleasetothenext?

Running a successful summative usability test should always involve some fol-
low-up activities. Just seeing the metrics is usually not enough for most organiza-
tions. Potential outcomes of a summative usability test might be securing funding
to enhance functionality on your product, launching a new project to address
some outstanding usability issues, or even benchmarking changes to the user
experience against which senior managers will be evaluated. We recommend that
follow-up actions be planned along with any summative usability study.

FORMATIVE AND SUMMATIVE USABILITY TESTING

The terms formative and summative were borrowed from the classroom environment,
where formative assessment is done on an ongoing basis by a teacher every day in the
classroom (think informal observation and “pop quizzes”), while summative assessment
is done at the end of some significant period of time (think “final exams”). The earliest
application of these terms to usability testing appears to be in a paper presented by Tom
Hewett at a conference at the University of York in the United Kingdom (Hewett, 1986).
This was also when one of us (Tullis) first met Tom Hewett, mainly because we were the
only two Americans at the conference! We've been friends ever since.