A Semi-Automated Approach to the Content Analysis of Experience Narratives - Modeling Users' Experiences with Interactive Systems - page 131

that are clustered together. This information may assist the researcher in interpreting
the theme of each cluster and proceed in further grouping of the clusters.
Using the three most dominant terms for each cluster, six individuals were asked
to assign each of the nine clusters to one of the five overall categories: stimulation,
learnability, long-term usability, usefulness and social experiences .
Figure 6.8 displays the percentage of narratives being correctly classified, i.e.
assigned to the same category as in the hand-coded analysis of the initial study
Karapanos et al. (2009c), both for the optimal approach that employs the distribution
of narratives across the five categories and for the approach that employs human
raters in assigning the 9 clusters to five categories. The 95% confidence intervals
depict the uncertainty derived from the human classification of the nine clusters into
the five overall categories.

1 1

Optimalvalue
Averagevalue(humanraters)

Optimalvalue
Averagevalue(humanraters)

0.9 0.9

0.8 0.8

0.7 0.7

0.6 0.6

0.5 0.5

0.4 0.4

0.3 0.3

0.2

0.1

0

LSALSAonRestrictedtermsConceptAnalysis

Fig. 6.8 Percentage of narratives being correctly classified, i.e. assigned to the same category
as in the hand-coded analysis of the initial study Karapanos et al. (2009c), both for the opti-
mal approach and for the approach that employs human raters in assigning the 9 clusters to
five categories. The 95% confidence intervals depict the uncertainty derived from the human
classification of the nine clusters into the five overall categories.

A number of insights are gained from this analysis. First, note the substantial
difference in interrater agreement between the optimal mapping process (based on
narratives distribution) and the process that involved human raters. This difference
is larger in the two approaches that involved LSA as these procedures perform less
favorably and are more ambiguous in characterizing the clusters through the three
most dominant terms.
Next, the proposed method performs substantially better than traditional LSA,
i.e. 66% of narratives were correctly classified versus 24% in the case of LSA when
human raters are involved, while the optimal mapping process results in respective
scores of 88% (Interrater agreement K=0.85 (Fleiss et al., 2003)) for the proposed
method and 63% (Interrater agreement K=0.54) for traditional LSA.