Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 51

3.3.8 Creating an Overall Positive User Experience
Some products strive to create an exceptional user experience. It's simply not
enough to be usable. These products need to be engaging, thought-provoking,
entertaining, and maybe even slightly addictive. The aesthetics and visual appeal
usually play important roles as well. These are products that you tell a friend
about and are not embarrassed to mention at a party. Their popularity usually
grows at phenomenal rates. Even though the characteristics of what constitutes
a great user experience are subjective, they are still measurable.
Although some performance metrics may be useful, what really matters is
what the user thinks, feels, and says with respect to his or her experience. In
some ways, this is the opposite perspective of measuring usability of a critical
product. If the user struggles a little at first, it may not be the end of the world.
What matters is how the user feels at the end of the day. Many self-reported met-
rics must be considered when measuring the overall user experience.
Satisfaction is perhaps the most common self-reported metric, but it may not
always be the best one. Being “satisfied” is usually not enough. One of the most
valuable self-reported metrics we've used relates to the participant's expectation.
The best experiences are those that exceed a participant's expectations. When the
participant says something is much easier, more efficient, or more entertaining
than expected, you know you are onto something.
Another set of self-reported metrics relates to future use. For example, you
might ask questions related to likelihood to purchase, recommend to a friend,
or use in the future. The Net Promoter Score is a widely used metric to measure
likelihood of future use. Another interesting set of metrics relates to subcon-
scious reactions that users may be having. For example, if you want to make
sure your product is engaging, you can look at physiological metrics. Changes
in pupil diameter can be used to gauge the level of arousal or, if you're try-
ing to eliminate stress as much as possible, you can measure heart rate or skin
conductance.
3.3.9 Evaluating the Impact of Subtle Changes
Not all design changes have an obvious impact on user behavior. Some design
changes are much more subtle, and their impact on user behavior is less clear.
Small trends, given enough users, can have huge implications for a large popu-
lation of users. The subtle changes may involve different aspects of the visual
design, such as font choice and size, placement, visual contrast, color, and image
choice. Nonvisual design elements, such as subtle changes to content or termi-
nology, can also have an impact on the user experience.
Perhaps the best way to measure the impact of subtle design changes is
through live-site metrics from A/B tests. A/B testing involves comparing a con-
trol design against an alternative design. For websites, this usually involves
diverting a (usually a small) portion of web traffic to an alternative design and
comparing metrics such as traffic or purchases to a control design. An online