Introduction - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 12

Myth 2: UX Metrics Cost Too Much Money
Some people believe that the only way to get reliable UX data is to outsource the
study to a market research firm or UX/design consultancy. Although this may
be helpful in some situations, it can also be quite costly. Many reliable metrics
don't cost an arm and a leg. Even as part of your everyday testing, you can col-
lect incredibly valuable data on the frequency and severity of different usability
issues. You can also collect huge amounts of quantitative data by sending out
short e-mail surveys to fellow employees or a panel of targeted users. Also, some
of the best analysis tools are actually free on the web. Although money does help
in certain situations, it is by no means necessary to get some great metrics.
Myth 3: UX Metrics are not Useful When Focusing on Small
Improvements
Some project team members may question the usefulness of metrics when they
are interested in only some fairly small improvements. They may say it's best to
focus on a narrow set of improvements and not worry about metrics. They may
not have any extra time or budget to collect any UX metrics. They may say that
metrics have no place in a rapid-pace iterative design process. Analyzing usabil-
ity issues is an obvious and incredibly valuable solution. For example, looking at
the severity and frequency of usability issues and why they occur is an excellent
way to focus resources during the design process. This approach saves the project
both money and time. You can easily derive UX metrics based on previous stud-
ies that might help you answer key usability questions. UX metrics are useful for
large and small projects alike.
Myth 4: UX Metrics Don't Help us Understand Causes
Some people argue that metrics don't help us understand the root cause of user
experience problems. They assume (incorrectly) that metrics serve only to high-
light the magnitude of the problem. But if they concentrate on only success
rates or completion time data, it's easy to see why some might have this percep-
tion. Metrics, however, can tell you much more about the root cause of usabil-
ity issues than you might initially think. You can analyze verbatim comments
to reveal the source of the problem and how many users experience it. You can
identify where in the system users experience a problem and use metrics to tell
where and even why some problems occur. Depending on how the data are
coded and the methods used, there is a wealth of UX data that can help reveal
the root cause of many UX issues.
Myth 5: UX Metrics are Too Noisy
One big criticism of UX metrics is that the data are too “noisy.” Too many vari-
ables prevent getting a clear picture of what's going on. The classic example
of “noisy” data is measuring task completion time in an automated usability
study when the participant goes out for a cup of coffee or, worse, home for the
weekend. Although this may happen on occasion, it should not deter you from