Self-Reported Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 128

tasks, studies, user groups, and so on. This is common practice among most UX
professionals as well as market researchers. Even though rating-scale data are not
technically interval data, many professionals treat it as interval. For example, we
assume the distance between a 1 and a 2 on a Likert scale is the same as the dis-
tance between a 2 and a 3 on the same scale. This assumption is called degrees of
intervalness . We also assume that a value between any two of the scale positions
has meaning. The bottom line is that it is close enough to interval data that we
can treat it as such.
When analyzing data from rating scales, it's always important to look at the
actual frequency distribution of the responses. Because of the relatively small
number of response options (e.g., 5-9) for each rating scale, it's even more
important to look at the distribution than it is for truly continuous data such as
task times. You might see important information in the distribution of responses
that you would totally miss if you just looked at the average. For example, let's
assume you asked 20 users to rate their agreement with the statement “This web-
site is easy to use” on a 1 to 7 scale, and the resulting average rating was 4 (right
in the middle). You might conclude that the users were basically just lukewarm
about the site's ease of use. But then you look at the distribution of the ratings
and you see that 10 users rated it a “1” and 10 rated it a “7”. So, in fact, no one
was lukewarm. They either thought it was great or they hated it. You might then
want to do some segmentation analysis to see if the people who hated it have
anything in common (e.g., they had never used the site before) vs the people
who loved it (e.g., long-time users of the site).
WHAT NUMBER SHOULD RATING SCALES START WITH?
Regardless of whether you show numbers for each scale value to the user , you will
normally use numbers internally for analysis. But what number should the scales start
with, zero or one? It generally doesn't matter, as long as you report what the scale is
whenever showing mean ratings (e.g., a mean of 3.2 on a scale of 1 to 5). But there are
some cases where it's convenient to start the scale at zero, particularly if you want to
express the ratings as percentages of the best possible rating. On a scale of 1 to 5, a rating
of 5 would correspond to 100%, but a rating of 1 does not correspond to 20%, as some
might think (e.g., calculating the percentage by multiplying the rating by 20, which is
wrong). On a scale of 1 to 5, 1 is the lowest possible rating, so it should correspond
to 0%. Consequently, we find it's easier to keep our sanity by internally numbering
rating scales starting at zero, so that a rating of 0 corresponds to 0%.
Another way to analyze rating-scale data is by looking at top-box or top-
2-box scores. Assume you're using a rating scale of 1 to 5, with 5 mean-
ing “Strongly Agree.” The sample data in Figure 6.1 illustrate the calculation
of top-box and top-2-box scores. A top-box score would be the percentage of
participants who gave a rating of 5. Similarly, a top-2-box score would be the