Self-Reported Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 126

6.2.5 Biases in Collecting Self-Reported Data
Some studies have shown that people who are asked directly for self-reported
data, either in person or over the phone, provide more positive feedback than
when asked through an anonymous web survey (e.g., Dillman et al., 2008). This
is called the social desirability bias (Nancarrow & Brace, 2000), in which respon-
dents tend to give answers they believe will make them look better in the eyes of
others. For example, people who are called on the phone and asked to evaluate
their satisfaction with a product typically report higher satisfaction than if they
reported their satisfaction levels in a more anonymous way. Telephone respon-
dents or participants in a usability lab essentially want to tell us what they think
we want to hear, and that is usually positive feedback about our product.

Therefore, we suggest collecting post-test data in such a way that the modera-
tor or facilitator does not see the user's responses until after the participant has left.
This might mean either turning away or leaving the room when the user fills out
the automated or paper survey. Making the survey itself anonymous may also elicit
more honest reactions. Some UX researchers have suggested asking participants in
a usability test to complete a post-test survey after they get back to their office or
home. This can be done by giving them a paper survey and a postage-paid envelope
to mail it back or by e-mailing a pointer to an online survey. The main drawback of
this approach is that you will typically have some drop-off in terms of who com-
pletes the survey. Another drawback is that it increases the amount of time between
users' interaction with the product and their evaluation via the survey, which could
have unpredictable results.

6.2.6 General Guidelines for Rating Scales
Crafting good rating scales and questions is difficult; it's both an art and a sci-
ence. So before you go off on your own, look at existing sets of questions, such
as those in this chapter, to see if you can't use those instead. But if you decide
that you need to create your own, here are some general points to consider:

• Multiple scales to help “triangulate.” When creating scales to assess a

specific attribute such as visual appeal, credibility, or responsiveness, the
main thing to remember is that you will probably get more reliable data
if you can think of a few different ways to ask participants to assess the
attribute. In analyzing the results, you would average those responses
together to arrive at the participant's overall reaction for that attribute.
Likewise, the success of questionnaires that include both positive and
negative statements to which participants respond would suggest the
value of including both types of statements.

• Odd or even number of values? The number of values to use in rating

scales can be a source of heated debate among UX professionals. Many
of the arguments center on the use of an even or odd number of points
on the scale. An odd number of points has a center, or neutral, point,
whereas an even number does not, thus forcing the user slightly toward
one end or the other on the scale. We believe that in most real-world