Information Technology Reference
In-Depth Information
the specification and to mark words, word sequences or sentences that, in his/her
opinion, would cause problems if we model the system behavior.
2. Then, every subject was given a set of questions generated as presented in Sec-
tion 3.2, and asked, for every generated question, to evaluate if the question ad-
dresses a genuine problem of the specification.
3. Similarly to step 1, every subject was given 15 (out of 41) scenarios from the In-
strument Cluster Specification [5]. The 15 scenarios were selected in such a way
that they represent typical system behavior. Thus, the results obtained with the 15
scenarios can be extrapolated to all scenarios provided in the specification.
Analogously to step 1, every subject was asked to read the specification and to mark
words, word sequences or sentences that, in his/her opinion, would cause problems
if we model the system behavior.
4. Analogously to step 2, every subject was given a set of generated questions and
asked, for every generated question, to evaluate if the question addresses a genuine
problem of the specification.
In order to address different representations of the generated questions, the subjects
were separated in two groups: One group was given questions referring to the specifi-
cation text for the Steam Boiler and questions referring to the model for the Instrument
Cluster. The other group, inversely, was given questions referring to the model for the
Steam Boiler and questions referring to the specification text for the Instrument Cluster.
Due to this setting, we avoid two potential threats to validity of the results:
-
Every subject was given one set of questions referring to the model and one set of
questions referring to the specification. Thus, the differences in subjects' responses
cannot be attributed to the differences in the representation of the questions.
-
Every subject was given one set of questions about the Steam Boiler Specification,
and one set of questions about the Instrument Cluster Specification. This way, we
avoid learning effects that would become important if we would have given two
sets of questions about the same specification to the same subject.
The evaluation was performed in groups, but the subjects were not allowed to commu-
nicate with each other. In total, every subject spent approximately two hours performing
the above steps 1-4.
4.2
Experiment Results
The results of the experiment are presented in Table 6. The numbers in the cells repre-
sent the number of manually marked problems or generated/confirmed questions. The
number of questions generated for the Steam Boiler Specification (automaton-based)
coincide for the questions referring to the model and to the text, as the differences in
the representations are not too big. For the Instrument Cluster (MSCs), however, differ-
ences in representations result in the different number of generated questions (cf. Sec-
tion 3.1).
Questions generated for the Steam Boiler Specification (automaton-based) were con-
sidered by the most subjects as spurious. With one notable exception, most subjects
