Information Technology Reference
In-Depth Information
Table 2
0.6
Minimum and maximum phases assigned for each instructional m
ethod
Instructional method
Minimum
Maximum
Nominal group technique
2
4
Debate
1
5
Reflective deliberation
2
4
Invited expert
2
4
WebQuest
2
5
Table 20.7
Frequency of postings in each phase of the practical inquiry model for Coders 1 and 2
broken down by instructional methods
Nominal group
technique
Reflective
deliberation
Debate
Invited expert
WebQuest
Coder
1
Coder
2
Coder
1
Coder
2
Coder
1
Coder
2
Coder
1
Coder
2
Coder
1
Coder
2
Phase
1
2
1
0
0
0
0
0
0
0
0
2
13
7
12
12
3
3
2
1
60
38
3
79
66
45
44
25
27
17
18
142
150
4
42
46
5
7
10
9
7
6
80
81
5
19
28
0
0
0
0
1
0
13
28
Totals
155
148
62
63
38
39
27
25
295
297
Table 20.8
Coder agreement of cognitive presence by instructional method (Cohen's Kappa and
testing of independence between coders)
All groups
Debate
Invited
NGT
Reflective
Webquest
0.570
0.557
0.638
0.230
0.366
0.590
P
b
< 0.001
∗
P
b
< 0.001
∗
P
b
P
b
(9)
=
626.64
a
,
P
< 0.001
∗
=
0.214
=
0.087
(9)
=
362.93
a
,
P
< 0.001
∗
∗
Significant at 0.05 level.
a
Fisher Exact's Test could not be computed due to insufficient memory.
b
P
-values were computed by Fisher's Exact Tests due to small expected cell sizes.
quite small for the nominal group technique and reflective deliberation methods
(chance-corrected agreement is 23.0 and 36.6% respectively). For testing of inde-
pendence between the coders for the nominal group technique and the reflective
deliberation methods, Fisher's Exact Test
P
-values show they are not significant
(Table 20.8). Although it has been argued that statistically unacceptable agreement
between raters does not mean that the coding decisions are so inconsistent as to be
worthless (Crocker & Algina, 1986), the data on nominal group technique in par-
ticular needs to be interpreted with caution as the coders strength of agreement is,
at best, fair. An explanation for this result could be that because the nominal group
technique was the first instructional method the coders analyzed they did not, yet,
completely understand where the conceptual boundaries were for each of phase of
