Cross-Cultural Design and Evaluation Framework - Cross-Cultural Human-Computer Interaction and User Experience Design - page 84

clustered according to their areas, that is, personal information, layout, color, symbol,
and look and feel. When appropriate, we used closed questions. However, because
we carried an exploratory pilot study, we did not want to constrain our respondents
and offered mostly open questions. The questionnaire was supported by examples of
UI components.
To get data as reliable as possible, we wanted to limit the respondent's adaptation
to a foreign culture. According to Smith (2011), tests should be done locally to
prevent users from absorbing the majority culture, and should be carried on with
a local moderator, sharing the language and cultural background (Ibid., p. 33). For
that reason, we worked with students who were enrolled at a local university (Dalian
Maritime University in China and Charles University in Prague, the Czech Republic)
and were born and lived in the target cultures of our study. Also, the moderator of the
interviews was a native speaker at each location.
For each of the interviewed groups we chose a sample consisting of 20 respon-
dents, evenly split between females and males. The Chinese respondents had a mean
age of 23 years, while the Czechs had a mean age of 26 years. For this kind and
depth of research, 20 respondents make for an adequate saturated sample size (Guest
et al., 2006; Nielsen and Landauer, 1993; Mason, 2010). The results were analyzed
both qualitatively and quantitatively with regard to the threshold of significance.
The threshold of significance was set to 10%. Only results of more than 10% from
one another were taken as different. This level is based on Sauro (2005), where
there is an 8% margin of error for results from 20 respondents. We took into ac-
count only results greater than or equal to 60%. The open questions were then ana-
lyzed with content analysis and contextual analysis across multiple questions where
appropriate.
The research was carried on in the following manner:
• Users were presented with a localized questionnaire of 53 questions divided
into groups: personal info, layout, color, symbols, look and feel.
• The questionnaire was supported by a moderator , who would give the neces-
sary background to the user, and present him or her a choice of images/color
samples, when appropriate.
• The testing of the Chinese sample was carried out in the user lab , with a
note taker sitting besides the moderator. The Czech sample was interviewed
in a computer lab , and the respondents answered the questionnaire directly
by typing.
• In the user lab, users were taped by 3 cameras and their voices were
recorded.
• Supportive note taking (2-3 people), and/or observation were done in an
adjacent room.
• As a data-collecting and reporting platform, we chose SurveyGizmo.
com .
• One session took about 90 min .
• After each testing day, data from the note takers was checked and merged
with the data on the server.