Chemistry Reference
In-Depth Information
Table 3 Inter-rater
agreement data for whether an
interpretation of item 6 is a
good indicator of the intended
scale
Rater 2
A good indicator or not
Yes
No
Total for rater 2
Rater 1
Yes
40
1
41
No
3
6
9
Total for rater 1
43
7
50
respondents interpret the survey item (Schwarz,
1999
). Specifically, we need to
know whether students understand and respond to item 6 as an indicator of
chemistry attitude on the intellectual accessibility subscale as expected.
Fifty KU students who took the ASCIv2 were willing to provide anonymous
written responses to the second author regarding their interpretation of the meaning
of item 6. The first author and a chemistry graduate student with qualitative coding
experience independently coded all the feedback. If the student interpreted item 6 as
expected for the intellectual accessibility scale, the feedback was coded as 1. If not,
the code was 0. From Table
3
, both raters agreed on 46 out of the 50 total codes,
with percent agreement at 92 %. Inter-rater reliability calculated by Cohen
s kappa
is 0.70, which is considered substantial strength of agreement (Landis & Koch,
1977
).
Both raters believed that a large portion of students (40/50) interpreted item 6 to
involve considerations of whether chemistry is difficult or not, which does fit the
intended scale of intellectual accessibility. Both raters also agreed that six students
did not interpret the item as a good indicator of the intended subscale, but they
disagreed on the other four students
'
feedback. Overall, then, ten students explicitly
interpreted this item in terms that seemed outside intellectual accessibility to at least
one rater.
Three chemistry graduate students engaged in a consensus coding process to
identify the common characteristics in the ten student interpretations that were not
in good alignment with the intended subscale. Out of the ten interpretations, one
student interpreted the meaning of challenging as including strong negative emo-
tional arousal. He wrote, “Challenging for me it means challenging (frustrating) in
negative side of it, and usually frustrating does not lead to a positive result as far as
what sort of despair and melancholy in the soul.” This means he treats this item as
more on the emotional satisfaction rather than on the intellectual accessibility
subscale. Five student responses were in alignment with neither intended subscale.
Three of these students interpreted item 6 as chemistry relating to daily life, for
example, “Chemistry helps us to understand how the things around us work.” The
other two students viewed the item as indicating the role of chemistry in discovery,
e.g., “Challenge in chemistry appears to me on how new discovery can help,
improve, and solve of today life.” These interpretations are both off target for
intellectual accessibility.
The other four students mentioned multiple elements in their responses, such as
the effort needed to learn chemistry, the lab work, competition for a high score, the
learning strategy, the grading system, and so on. For example, one student wrote, “It
let you try to do your best to get the best result; [e]specially in the lab you try to get
'
Search WWH ::
Custom Search