Issue-Based Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 99

CHAPTER 5
Issue-Based Metrics

99

CONTENTS
5.1 WHAT IS A USABILITY ISSUE?

100

5.1.1 Real Issues versus False Issues 101

5.2 HOW TO IDENTIFY AN ISSUE 102

5.2.1 In-Person Studies 102

5.2.2 Automated Studies 103

5.3 SEVERITY RATINGS 103

5.3.1 Severity Ratings Based on the User Experience 104

5.3.2 Severity Ratings Based on a Combination of Factors 105

5.3.3 Using a Severity Rating System 106

5.3.4 Some Caveats about Rating Systems 107

5.4 ANALYZING AND REPORTING METRICS FOR USABILITY ISSUES 107

5.4.1 Frequency of Unique Issues 108

5.4.2 Frequency of Issues Per Participant 109

5.4.3 Frequency of Participants 109

5.4.4 Issues by Category 110

5.4.5 Issues by Task 111

5.5 CONSISTENCY IN IDENTIFYING USABILITY ISSUES 111

5.6 BIAS IN IDENTIFYING USABILITY ISSUES 113

5.7 NUMBER OF PARTICIPANTS 115

5.7.1 Five Participants is Enough 115

5.7.2 Five Participants is Not Enough 117

5.7.3 Our Recommendation 118

5.8 SUMMARY 119

Most user experience professionals probably consider identifying usability
issues and providing design recommendations the most important parts of their
job. A usability issue might involve confusion around a particular term or piece
of content, method of navigation, or just not noticing something that should
be noticed. These types of issues, and many others, are typically identified as
part of an iterative process in which designs are being evaluated and improved
throughout the design and development process. This process provides tremen-
dous value to product design and is the cornerstone of the UX profession.