Issue-Based Metrics - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 113

measuring. While both moderated and unmoderated testing have opportunities for
things to go wrong, it is more difficult to detect and correct these with unmoderated
testing. Further studies of how data contamination can be prevented and how
contaminated data can be cleaned efficiently are required.

For unmoderated measurements, the ease of use and intrusiveness of the remote
tool influence measurements. Some teams complained about clunky interfaces. We
recommend that practitioners demand usable products for usability measurements.

Practitioner's Takeaway from CUE-8
CUE-8confirmedanumberofrulesforgoodmeasurementpractice.Perhapsthe
mostinterestingresultfromCUE-8isthattheseruleswerenotalwaysobservedbythe
participating professional teams.
• Adherestrictlytopreciselydefinedmeasurementproceduresforquantitativetests.

• Reporttimeontask,success/failurerate,andsatisfactionforquantitativetests.

• Excludefailedtimesfromaveragetaskcompletiontimes.

• Understandtheinherentvariabilityfromsamples.Usestrictparticipantscreening

criteria.Provideconfidenceintervalsaroundyourresultsifthisispossible.Keep
in mind that time on task is not distributed normally and therefore confidence
intervals as commonly computed on raw scores may be misleading.

• Combinequalitativeandquantitativefindingsinyourreport.Presentwhat

happened (quantitative data) and support it with why it happened (qualitative
data). Qualitative data provide considerable insight regarding the serious obstacles
that users faced and it is counterproductive not to report this insight.

• Justifythecompositionandsizeofyourparticipantsamples.Thisistheonlyway

you have to allow your client to judge how much confidence they should place in
your results.

• Whenusingunmoderatedmethodologiesforquantitativetestsensurethatyou

can distinguish between extreme and incorrect results. Although unmoderated
testing can exhibit a remarkable productivity in terms of user tasks measured with a
limited effort, quantity of data is no substitute for clean data.

Further Information
The17-pagerefereedpaper“RentaCarinJust0,60,240or1,217Seconds?Comparative
UsabilityMeasurement,CUE-8”describestheresultsofCUE-8indetail.Thepaperis
freelyavailableintheNovember2010issueofthe Journal of Usability Studies .

5.6 BIAS IN IDENTIFYING USABILITY ISSUES
Manydifferentfactorscaninfluencehowusabilityissuesareidentified.Carolyn
Snyder (2006) provides a review of many of the ways usability findings might
be biased. She concludes that bias cannot be eliminated, but it must be under-
stood. In other words, even though our methods have flaws, they are still useful.