Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 49

awareness of a specific piece of content or functionality. This is certainly true
for online advertisements, but it's also true for products that have important
but underutilized functionality. There can be many reasons why something is
not noticed or used, including some aspect of the visual design, labeling, or
placement.
First, we recommend monitoring the number of interactions with the ele-
ment in question. This is not foolproof, as a participant might notice some-
thing but not click on it or interact with it in some way. The opposite would
not be very likely: interaction without noticing. Because of this, data can
help confirm awareness but not demonstrate lack of awareness. Sometimes
it's useful to ask for self-reported metrics about whether the participants
noticed or were aware of a specific design element. Measuring noticeability
involves pointing out specific elements to the participants and then asking
whether they had noticed those elements during the task. Measuring aware-
ness involves asking the participants if they were aware of the feature before
the study began. However, data are not always reliable (Albert & Tedesco,
2010). Therefore, we don't recommend that this be your sole measure; you
should complement it with other data sources.
Memory is another useful self-reported metric. For example, you can show
participants several different elements, only one of which they had actually seen
previously, and ask them to choose which one they saw during the task. If they
noticed the element, their memory should be better than chance. But perhaps
the best way to assess awareness, if you have the technology available, is through
the use of behavioral and physiological metrics such as eye-tracking data. Using
eye-tracking technology, you can determine the average time spent looking at
a certain element, the percentage of participants who looked at it, and even
the average time it took to first notice it. Another metric to consider, in the
case of websites, is a change in live website data. Looking at how traffic pat-
terns change between different designs will help you determine relative aware-
ness. Simultaneous testing of alternative designs (A/B testing) on live sites is an
increasingly common way to measure how small design changes impact user
behavior.
3.3.6 Problem Discovery
The goal of problem discovery is to identify major usability issues. In some sit-
uations you may not have any preconceived ideas about what the significant
usability issues are with a product, but you want to know what annoys users.
This is often done for a product that is already built but has not gone through
usability evaluation before. A problem discovery study also works well as a peri-
odic checkup to get back in touch with how users are interacting with your prod-
uct. A discovery study is a little different from other types of usability studies
because it is generally open ended. Participants in a problem discovery study
may be generating their own tasks, as opposed to being given a list of specific
tasks. It's important to strive for realism as much as possible. This might involve