Information Technology Reference
In-Depth Information
questionnaire based variables aggregate to one large principal component. These
measures rely on self-perception of the subjects and therefore describe one side
of the coin. Moreover, the time for
talking
V
1
and
silence
V
2
strongly correlate
with the amount of
reviews
V
6
done. It indicates the degree to which people were
involved with the task. Finally,
corrections
V
7
builds a single factor. Overall, the
measurement validation calls for a more thought-out hypothesis decomposition
and clever selection of measurement instruments.
5.4 Hypothesis Testing
We use the repeated-measures ANOVA to determine the effect of our indepen-
dent variable (method) within each individual per dependent variable. In other
words, to what extend did the method influence the performance of each individ-
ual? Fig. 5 illustrates how our data is partitioned. From the overall variability
(
SS
T
), we identify the performance difference within participants (
SS
W
)and
canfurtherdistinguishthevariationcausedbythetreatment(
SS
M
)andthe
variation not explained by our treatment(
SS
R
).
SS
T
=
SS
B
+
SS
W
Total Variation
SS
B
SS
W
=
SS
M
+
SS
R
Between Participants Variation
Within Participants Variation
SS
M
SS
R
Variation caused
by method
Other variation
(Error)
Fig. 5.
Data partitioning for rep.-measures ANOVA. Drawing adopted from [6] p.463
The ratio of explained to unexplained variability in our dataset is described
by
F
=
SS
M
df
M
/
SS
R
df
R
.Where
df
are the degrees of freedom calculated from the
number of different methods (
df
M
=2-1=1) and the participant number (
df
R
=17-
1=16). The critical ratio
F
.
05
(
df
M
,df
R
) is the value to pass before the result is
actually significant with an acceptance level of p
<
.05. For our variables collected
in questionnaires
F
.
05
(1
,
16)
>
4
.
49 is a significant result, for the video codings
we only have N=16 thus
F
.
05
(1
,
15)
>
4
.
54 is a significant ratio. In Table 1 we
sorted the variables according to descending
F
.
05
ratios. We also report
SS
B
,
SS
M
,
SS
R
and
η
2
(eta squared). The value of
η
2
=
SS
M
SS
W
describes the ratio of
variation within the subjects that can be explained by the treatment method. It
is an effect size measure.
Furthermore we conduct a dependent t-test to create a different view on the
data, see Table 2. It compares the groups doing t.BPM and interviews by their
mean scores (
V
=in minutes,
Q
=Likert scale [1..5]), the statistical significance
of this difference (one-tailed with acceptance level p
<
.05) and the confidence
interval. The upper and lower boundaries indicate that the real mean difference
