Issue-Based Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 111

5.4.5 Issues by Task
Issues can also be analyzed at a task level. You might be interested in which tasks
lead to the most issues, and you can report the number of unique issues that occur
for each. This will identify the tasks you should focus on for the next design itera-
tion. Alternatively, you could report the frequency of participants who encounter
any issue for each task. This will tell you the pervasiveness of a particular issue.
The greater the number of issues for each task, the greater the concern should be.
If you have assigned a severity rating to each issue, it might be useful to ana-
lyze the frequency of high-priority issues by task. This is particularly effective if
you want to focus on a few of the biggest problems and your design efforts are
oriented toward specific tasks. This is also helpful if you are comparing different
design iterations using the same tasks.
5.5 CONSISTENCY IN IDENTIFYING USABILITY ISSUES
Much has been written about consistency and bias in identifying and prioritizing
usability issues. Unfortunately, the news is not so good. Much of the research shows
that there is very little agreement on what a usability issue is or how severe it is.
Perhapsthemostexhaustivesetofstudies,calledCUE(ComparativeUsability
Evaluation) has been coordinated by Rolf Molich. To date, nine separate CUE
studies have been conducted, dating back to 1998. Each study was set up in a
similar manner. Different teams of usability experts all evaluated the same design.
Each team reported their findings, including the identification of the usability
issues,alongwiththeirdesignrecommendations.Thefirststudy,CUE-1(Molich
et al., 1998), showed very little overlap in the issues identified. In fact, only 1 out
of the 141 issues was identified by all four teams participating in the study, and
128 out of the 141 issues were identified by single teams. Several years later, in
CUE-2,theresultswerenomoreencouraging:75%ofalltheissueswerereported
byonly1of9usabilityteams(Molichetet al.,2004).CUE-4(Molich&Dumas,
2008) showed similar results: 60% of all the issues were identified by only 1 of
the17differentteamsparticipatinginthestudy.Morerecently,CUE-8focusedon
the consistency of how UX metrics are used and the conclusions that are drawn.
CUE-8—HOW PRACTITIONERS MEASURE WEBSITE
USABILITY
by Rolf Molich , Dialog Design
Fifteen experienced professional usability teams simultaneously and independently
measuredabaselinefortheusabilityofthecarrentalwebsiteBudget.com.This
comparative study documented a wide difference in measurement approaches. The 8-10
teams that used similar and well-established approaches reached surprisingly similar
results.