Information Technology Reference
In-Depth Information
granular approach is to break out each level according to whether assistance was
given or not. Below is an example of six different levels of completion:
•
Completesuccess
o
With assistance
o
Without assistance
•
Partialsuccess
o
With assistance
o
Without assistance
•
Failure
o
User thought it was complete, but it wasn't
o
User gave up
If you do decide to use levels of success, it's important to clearly define the
levels beforehand. Also, consider having multiple observers independently
assess the levels for each task and then reach a consensus.
A common issue when measuring levels of success is deciding what consti-
tutes “giving assistance” to the participant. Here are some examples of situations
we define as giving assistance:
•
Moderatortakestheparticipantbacktoahomepageorresetstoanini-
tial (pretask) state. This form of assistance may reorient the participant
and help avoid certain behaviors that initially resulted in confusion.
•
Moderatoraskstheparticipantprobingquestionsorrestatesthetask.This
may cause the user to think about her behavior or choices in a different way.
•
Moderatoranswersaquestionorprovidesinformationthathelpsthe
participant complete the task.
•
Participantseekshelpfromanoutsidesource.Forexample,thepartici-
pant calls a phone representative, uses some other website, consults a
user manual, or accesses an online help system.
Level of success can also be examined in terms of the user experience. We
commonly find that some tasks are completed without any difficulty, whereas
others are completed with minor or major problems along the way. It's impor-
tant to distinguish between these different experiences. A four-point scoring
method can be used for each task:
1 = No problem. The user completed the task successfully without any dif-
ficulty or inefficiency.
2 = Minor problem. The user completed the task successfully but took a
slight detour. He made one or two small mistakes but recovered quickly and
was successful.
3 = Major problem. The user completed the task successfully but had major
problems. She struggled and took a major detour in her eventual successful
completion of the task.
4 = Failure/gave up. The user provided the wrong answer or gave up before
completing the task or the moderator moved on to the next task before suc-
cessful completion.
Search WWH ::
Custom Search