Hardware Reference
In-Depth Information
Fig. 5.
Respondent answer (non-annotated)
not considered an error because the respondent correctly labelled the table.
Respondent A2 achieved the fewest Errors and both the most Correct Elements
and fewest Missing Elements in Question 2.
The corresponding snippet from respondent B1, who had used a non-annotated
version, is transcribed in Figure 5. Here, the respondent B1 failed to identify from
the text that the “install()” action should fail due to the prior installation of a
conflicting package. Indeed respondent B1 didn't correctly identify “install()” as
an action at all, instead specifying the package name “vcron” combined with the
error detail as data to be verified. In comparing these answers with the reference
answer in Figure 4 one element was missed by respondent A2 while four elements
were missed by respondent B1 in Figure 5. Finally, it should also be noted that
respondent B1 performed better when using annotated texts and respondent A2
performed worse when using non-annotated texts. The next section summarises
the overall results for the experiment.
4.3 Results
The results gathered from the respondents answers, are summarised in Table 1.
For clarity, the
row number
is included in column 1. Columns 2, 3 and 4 introduce
the
question number
,which
group
is responding (A or B) and the
type
of descrip-
tion provided in the group's question. Columns 5, 6 and 7 contain the arithmetic
mean of the counts for each group's
Errors, Correct Elements
and
Missing El-
ements
, respectively. The presence of
Errors
indicates
Over-Specification
while
that of
Missing Elements
indicates
Under-Specification
. In all cases,
Correct El-
ements
plus
Missing Elements
equals
Total Element s
of the “ideal” answer.
We analysed both the data element and the structural element of the re-
sponses. An
Error
occurs whenever a response is matched against the “ideal”
answer and a mistake is identified. A mistake may be identified in either the data
element or the structural element. All mistakes that occur in the data element
are counted as errors, whereas only the first occurrence is counted as an error
in the structural element. For example, if we matched an individual's response
against the “ideal” response and discovered that a data element “fcron” had
been included by a respondent three times; the first two match the “ideal” re-
sponse counting as
Correct
but the third element would be incorrect and count
as one error.
Each row in Table 1 presents the results of one group for a particular question.
For example, Row 1 represents the arithmetic mean of responses from Group B
for Question 1 (annotated). The use of median would not reverse the overall
results.
