Information Technology Reference
In-Depth Information
spreadsheet created with the OntRep approach. The results were evaluated with de-
scriptive statistics in Excel and R and are described in the following section.
5 Results
The following subsections describe the results of the pilot study regarding require-
ments categorization and conflict analysis.
5.1 Requirements Categorization
In order to evaluate the requirements categorization task, we took the categorization
result of the requirements engineering expert as reference solution for comparing the
results of the manual and automated approaches.
Table 1.
Results of manual and automated req. categorization
Individuals
avg./std.dev.
Groups
avg./std.dev.
OntRep
MANUAL
AUTOMATED
1. Overfulfilled
9.5/3.9
12.5/3,5
6.0
2. Correct
5.7/2.7
6.0/2.8
8.0
Sum 1. & 2.
15.2/2.3
18,5/0.7
14.0
3. Partly correct
2.0/1.1
0.0/0.0
2.0
4. False
5.8/1.7
4.5/0.7
7.0
Table 1 summarizes the results of the manual and automated requirements categori-
zation approaches: the rows in the table contain the quality levels of the categorization:
“overfulfilled” means that a requirement was categorized into all correct categories but
also into one or more additional ones, “correct” means that a requirement was catego-
rized in the right categories. “Partially correct” means that a requirement was catego-
rized in some but not all of the correct categories, “false” means that a requirement was
categorized into wrong categories but not the right ones. The group results are better
than the individual results: the number of false categorizations is reduced, and the
number of correct and overfulfilled categorization is increased. Overfulfillment is not a
problem, because all requirements are categorized into the right categories, and into
some more categories, but this is just additional information which is allowed.
Categorization with OntRep was more accurate, i.e., 8 requirements (more than
with the manual approach) have been categorized into the right categories without
categorizing them in additional categories. On the other hand, comparing the sum of
correctly categorized requirements (overfulfilled + correct) shows the lowest value for
automation. Further, the number of false categorizations is also the highest. This is
due to the fact, 4 requirements were not categorized at all. The reason therefore is that
the terms used in these requirements could not be mapped to the categories, neither
through substrings, synonyms or hyponyms.
