Information Technology Reference
In-Depth Information
from 5 to 10% of the participants in an online study to be in it only for the
compensation.
4.2.4 Issues to Consider When Using Time Data
Some of the issues to think about when analyzing time data is whether to look
at all tasks or just successful tasks, what the impact of using a think-aloud pro-
tocol might be, and whether to tell test participants that time is being measured.
ONLY SUCCESSFUL TASKS OR ALL TASKS?
Perhaps the first issue to consider is whether you should include times for only
successful tasks or all tasks in the analysis. The main advantage of only includ-
ing successful tasks is that it is a cleaner measure of efficiency. For example, time
data for unsuccessful tasks are often very difficult to estimate. Some users will
keep on trying until you practically unplug the computer. Any task that ends
with the participant giving up or the moderator “pulling the plug” is going to
result in highly variable time data.
The main advantage of analyzing time data for
all
tasks, successful or not, is
that it is a more accurate reflection of the overall user experience. For example,
if only a small percentage of users were successful, but that particular group was
very efficient, the overall time on task is going to be low. Therefore, it is easy to
misinterpret time-on-task data when only analyzing successful tasks. Another
advantage of analyzing time data for all tasks is that it is an independent mea-
sure in relation to task success data. If you only analyze time data for successful
tasks, you're introducing a dependency between the two sets of data.
A good rule is that if the participant always determined when to give up on
unsuccessful tasks, you should include all times in the analyses. If the moderator
sometimes decided when to end an unsuccessful task, then use only the times
for the successful tasks.
USING A CONCURRENT THINK-ALOUD PROTOCOL
Another important issue to consider is whether to use a concurrent think-aloud
protocol when collecting time data (i.e., asking participants to think aloud while
they are going through the tasks). Most usability specialists rely heavily on a con-
current think-aloud protocol to gain important insight into the user experience.
But sometimes a think-aloud protocol leads to a tangential topic or a lengthy
interaction with the moderator. The last thing you want to do is measure time
on task while a participant is giving a 10-minute diatribe on the importance of
fast-loading web pages. When you want to capture time on task but also use a
concurrent think-aloud protocol, a good solution is to ask participants to “hold”
any longer comments for the time between tasks. Then you can have a dialog
with the participant about the just-completed task after the “clock is stopped.”
Search WWH ::
Custom Search