A Semi-Automated Approach to the Content Analysis of Experience Narratives - Modeling Users' Experiences with Interactive Systems - page 114

Contrary to idiosyncratic insights, generalized knowledge requires a pre-
processing step: that of assessing the similarity between different experiences.
While it is well-acknowledged that every experience is unique and non-repeatable
(Forlizzi and Battarbee, 2004; Hassenzahl and Tractinsky, 2006; Wright and Mc-
Carthy, 2008), different experiences share similar features: some might refer to the
same product feature; others might be motivated by the same human need despite
the seemingly different nature (Hassenzahl, 2008); others might refer to the same
product quality; some are filled with positive emotions while others are dominated
by negative ones. These features may form the basis for assessing the similarity of
different experience narratives.
In assessing the similarity between different narratives, content analysis tech-
niques (Krippendorff, 2004; Hsieh and Shannon, 2005) may be employed in identi-
fying key concepts in the data, identifying a hierarchical structure among the con-
cepts and classifying narratives into broad categories. Such approaches are laborious
as the researcher needs to process all narratives in identifying the key concepts in
the data.
A number of automated approaches to semantic similarity assessment have been
proposed in the field of Information Retrieval and can potentially assist the analyst
in this task. Such approaches typically rely on vector space models (Salton et al.,
1975) in which the degree of semantic similarity between documents is related to
the degree of term co-occurrence across the documents. As we will argue in this
chapter, these approaches exhibit a number of limitations when one is concerned
about analyzing self-reported experience narratives. First, they assume a homogene-
ity in the style of writing across documents which does not hold in this context as
the vocabulary and verbosity of documents might substantially vary across different
participants. Second, similarity is computed based on the co-occurrence of all terms
that appear in a pool of documents while in the qualitative analysis of experience
narratives the researcher is typically interested only in a limited set of words that
refer to the phenomena of interest. As a result, words that are of minimal interest to
the researcher may shadow the semantic relations that researchers are pursuing at
identifying. Third, these automated approaches lack an essential part of qualitative
research, that of interpretation. As different participants may use different terms or
even phrases to refer to the same latent concept, an objectivist approach that relies
purely on semantics will evidently fail in capturing the relevant concepts.
In this chapter, we propose a partially automated approach which combines tra-
ditional content analysis techniques (Strauss and Corbin, 1998) with computational
approaches to assess the semantic similarity between documents (Salton et al.,
1975). In identifying the concepts which will form the basis for computing the simi-
larity between narratives, the proposed approach combines existing domain-specific
knowledge with open-coding procedures that aim at identifying constructs that are
not captured by existing measurement models. Domain-specific knowledge, on the
other hand, for instance within the field of user experience, may be found in psy-
chometric scales measuring perceived product qualities and emotional responses to
products. Thus, only concepts that are relevant to the given context are considered
in computing the similarity between narratives. At the same time, the process is