Issue-Based Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 106

Alternatively, it's possible to consider
three or even four dimensions, such as
impact to the user experience, predicted
frequency of occurrence, impact on the
businessgoals,andtechnical/implemen-
tation costs. For example, you might
combine four different three-point
scales:

Few users
experiencing a
problem

Many users
experiencing a
problem

Small impact on
the user
experience

Low severity

Medium severity

Large impact on
the user
experience

Medium severity High severity

• Impact on the user experience (0 =
low, 1 = medium, 2 = high)
• Predicted frequency of occurrence
(0 = low, 1 = medium, 2 = high)

Figure 5.1 Severity rating scale taking into account problem frequency and
impact on the user experience. Adapted from Nielsen (1993).

• Impactonthebusinessgoals(0= low, 1 = medium, 2 = high)

• Technical/implementationcosts(0= high, 1 = medium, 2 = low)

Byaddingupthefourscores,younowhaveanoverallseverityratingranging
from 0 to 8. Of course, a certain amount of guesswork is involved in coming up
with the levels, but at least all four factors are being taken into consideration. Or,
if you really want to get fancy, you can weight each dimension based on some
sort of organizational priority.

5.3.3 Using a Severity Rating System
Once you have settled on a severity rating system, you still need to consider a
few more things. First, be consistent: Decide on one severity rating system and
useitforallyourstudies.Byusingthesameseverityratingsystem,youwillbe
able to make meaningful comparisons across studies, as well as help train your
audience on the differences between the severity levels. The more your audience
internalizes the system, the more persuasive you will be in promoting design
solutions.

Second,communicateclearlywhateachlevelmeans.Provideexamplesof
each level as much as possible. This is particularly important for other usability
specialists on your team who might also be assigning ratings. It's important that
developers, designers, and business analysts understand each severity level. The
morethe“nonusability”audienceunderstandseachlevel,theeasieritwillbeto
influence design solutions for the highest priority issues.

Third, try to have more than one usability specialist assign severity ratings to
each issue. One approach that works well is to have the usability specialists inde-
pendently assign severity ratings to each of the issues and then discuss any of the
issues where they gave different ratings and try to agree on the appropriate level.

Finally, there's some debate about whether usability issues should be tracked
aspartofalargerbug-trackingsystem(Wilson&Coyne,2001).Wilsonargues
that it is essential to track usability issues as part of a bug-tracking system because
it makes the usability issues more visible, lends more credibility to the usability
team,andmakesitmorelikelythattheissueswillberemedied.Coynesuggests