Hardware Reference
In-Depth Information
4 Annotations Experiment
This experiment was designed to evaluate the impact of annotations on the pro-
cess of authoring acceptance tests. The scenario used to write the question de-
scriptions given to respondents concerned the management of software packages
on a computer system, such as GNU/Linux [38]. There were six participants,
each experienced in computing as either a postgraduate or professional. How-
ever, none had prior experience of writing FIT tables. All were given a short,
two-hour training session on FIT Tables and ATDD. Participants were tasked
to create tests using either annotated descriptions or from non-annotated plain
text descriptions. The plain text descriptions serve as a reference for comparison
against annotations. The only difference between descriptions was the presence
or absence of annotations. Each participant was randomly assigned to Group A
and Group B, with each group assigned in total three participants and receiving
four questions. Group B started with annotated descriptions while Group A were
given a non-annotated version. For subsequent exercises the groups alternated
between annotated and non-annotated. Apart from a common assignment of
question, to their group, participants worked alone. In providing these descrip-
tions, the first author acted in the role of a customer on an agile project. The
experiment considered annotations in paper-based experiment in isolation aside
from usability considerations of prototypes.
4.1 Design
For comparison purposes, the first author wrote reference tests, providing an
“ideal” test description against which the participants' tests were compared.
Each was in the form of high-level descriptions of how a system should func-
tion, including handling of error conditions and intended to be of approximately
equal di
culty: Question 1 covered
initial bootstrap
of the package management
system; Question 2 covered
installation
of new packages; Question 3 covered
re-
moval
of packages; Question 4 covered
upgrading
of packages. The metrics used
to assess the experiment were gathered under the following headings:
-
Errors
: elements that should not appear in the test. From participants' an-
swers, all error occurrences counted towards the average.
-
Correct Elements
: From participants' answers an elements first occurrence.
Participants were free to reuse structural elements (for example the first
row in a FIT Table) as this only affects readability. However, repeated data
elements are counted as
Errors
. Presence of a data element irrespective of
corresponding structural element was enough for it to count as correct, so
two penalising respondents twice.
-
Missing Elements
: defined as elements that were omitted by the participants
compared to the reference test.
-
Time
: amount of time taken to complete FIT table.
