Information Technology Reference
In-Depth Information
Table 2. Actual number of flaws and severity
NHE
MHE
Appl. 1 Appl. 2
Mean % for both appl.s
Appl. 1 Appl. 2
Mean % for both appl.s
Cosmetic
5
7
6
5
5.75
5.375
Minor
5.5
6.5
6
11
13.25
12.125
Major
6.5
6.5
6.5
9
10.25
9.625
Catastrophe
5
8
6.5
1
8.75
4.875
Total of flaws
22
28
26
38
the mobile device by identifying usability
problems and prioritizing them according
to Nielsen's SRS (Table 2). Presentation of
the two applications to be evaluated was
counterbalanced to avoid any order effect.
While evaluating the mobile device, each
usability evaluator was asked to 'think
aloud' to explain what s/he was trying to
do and to describe why s/he was taking the
action. Their comments were recorded by
one of the evaluation moderators.
•
Debriefing session:
This focused on the
evaluators' experiences of the process, and
providing an opportunity to probe where
behavior was implicit or puzzling to the
researchers.
Nielsen's heuristics. The additional flaws found
by applying mobile heuristics were usually dif-
ferent from the ones identified by using Nielsen's
heuristics; also, the problems identified by each
expert in the mobile heuristics condition were a
small number from a larger set of usability difficul-
ties presented by the two applications, although
we could find some overlaps (problems pointed
out by more than one expert), which supports the
idea of inter-expert consistency when applying
mobile heuristics.
Severity of Flaws and Distribution
As depicted by Table 2 and Figure 5, Nielsen's
heuristics have produced a more equally distrib-
uted severity ranking of problems detected for
both applications. On the other hand, the mobile
heuristics have produced a more positive evalu-
ation of Appl.1 (61% of problems are considered
minor or cosmetic) while for Appl.2 the ranking
seems to be equally distributed among the four
severity levels. Considering the mean values in
Figure 5, it does appear that Nielsen's heuristics
identify fewer Minor and Major flaws compared to
the mobile heuristics. It also seems that Nielsen's
heuristics have a relatively even distribution of
severity ranking for the problems identified.
Nielsen's heuristics could therefore do a moderate
job of identifying flaws at any design level. The
mobile heuristics do seem to be especially good
at identifying Minor and Major flaws rather than
those at the extremes.
Data Analysis
The data collected were analyzed both quali-
tatively and quantitatively. Comparison of HE
effectiveness in the 2 experimental conditions
was assessed.
Number of Flaws and Variation among Experts
From Table 1, it appears that the use of the mo-
bile heuristics has increased the number of flaws
identified in the analysis of both applications, and
has reduced variation among experts' analyses. In
comparing the type of flaws detected by using the
two different sets of heuristics, we did not find
evidence of problems identified only by using
Search WWH ::
Custom Search