Information Technology Reference
In-Depth Information
3.7 Analysis Procedures
For hypothesis testing, we use a one-way repeated-measures ANOVA (analysis
of variances). It aims to determine the variation within subjects that is caused
by the method. Additionally, we carry out a dependent t-test with acceptance
level p
<
.05. It is used to get a different view on the data and to assess potentially
confounding factors that might have influenced the performance of the subjects.
To assess reliability of the questionnaire, we use Cronbach's alpha. It deter-
mines the internal consistency of the three questionnaire items measuring one
hypothesis. The video data is analyzed by two independent reviewers. They com-
pare their results and (if needed) resolve conflicts by negotiation. Cohen's Kappa
is used to determine the inter-rater agreement before negotiation to assess the
quality of our coding guidelines.
4 Experiment Execution and Data Collection
The experiment design was executed in December 2009 at a trade school in
Potsdam. Slots were offered to the students by short teasers given in the classes.
All subjects were at the age of nineteen to twenty-one. Students could choose
to swap one lecture unit for experiment participation (about 1h). We expected
to test industrial clerks only, but only ten volunteered. Thus, we opened up the
experiment to oce clerks as well. We ended up testing 7 oce clerks and 10
industrial clerks within the week.
Fig. 4.
Photos from the experiment execution. Subject giving interview (left) and mod-
eling with t.BPM (right). Taped by the video cameras.
Each experiment run started with a short informal warm-up chat and after-
wards followed the design as outlined in Section 3.2. One experimenter ran the
experiment, the other one operated the cameras and observed the situation to
ensure a stable treatment. Fig. 4 depicts the two experimental tasks as taped by
the cameras. One video taping went wrong, leading to a sample size of sixteen
for the video coding hypotheses.
5DaAnlys
We explain the analysis techniques used and the results found in this section.
We reason about the results in Section 6.
