Planning - Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics - page 60

3.5.3 Data Collection
It's important to think about how the data are going to be collected. You should
plan well in advance how you are going to capture all the data that you need for
your study. The decisions you make may have a significant impact on how much
work you have to do further down the road when you begin analysis.

In the case of a lab test with a fairly small number of participants, Excel prob-
ably works as well as anything for collecting data. Make sure you have a template
in place for quickly capturing the data during the test. Ideally, this is not done by
the moderator but by a note taker or someone behind the scenes who can enter
data quickly and easily. We recommend that data be entered in numeric format
as much as possible. For example, if you are coding task success, it is best to code
itasa“1”(success)and“0”(failure).Dataenteredinatextformatwilleventu-
ally have to be converted, with the exception of verbatim comments.

The most important thing when capturing data is for everyone on the usabil-
ity team to know the coding scheme extremely well. If anyone starts flipping
scales (confusing the high and low values) or does not understand what to
enter for certain variables, you will have to either recode or throw data out. We
strongly recommend that you offer training to others who will be helping you
collect data. Just think of it as inexpensive insurance to make sure you end up
with clean data.

For studies involving larger numbers of participants, consider using a data-
capture tool. If you are running an online study, data are typically collected
automatically. You should also have the option of downloading the raw data
into Excel or various statistical programs such as SAS and SPSS.

3.5.4 Data Cleanup
Datararelycomeoutinaformatthatisinstantlyreadytoanalyze.Somesortof
cleanup is usually needed to get your data in a format that allows for quick and
easyanalysis.Datacleanupmayincludethefollowing.

• Filtering data . You should check for extreme values in the data set. The

most likely culprit will be task completion times (in the case of online
studies). Some participants may have gone out to lunch in the middle
of the study so their task times will be unusually large. Also, some par-
ticipants may have taken an impossibly short amount of time to com-
plete the task. This is likely an indicator that they were not truly engaged
in the study. Some general rules for how to filter time data are included
in Section 4.2. You should also consider filtering out data for partici-
pants who do not reflect your target audience or where outside factors
impacted the results. We've had more than a few usability testing ses-
sions interrupted by a fire drill!

• Creating new variables . Building on the raw data set is very useful. For

example, you might want to create a top-2-box variable for self-reported
rating scales by counting the number of participants who gave one of the