Robust Requirements Analysis in Complex Systems through Machine Learning - Trustworthy Eternal Systems via Evolving, Software Data and Knowledge - page 52

&

&

&

&

&

&

&

% % % %

Fig. 2. Requisite Classification Results

Figure 2 reports the classification results, in terms of accuracy , i.e. the percent-
age of correctly classified requisites. Different colors reflect the different adopted
feature models. The first three histograms provide results when classifiers are
trained over requisite from one single scenario (i.e. EAU, NUM and FREMM
respectively) and applied to the other remaining scenarios. The second group
of histograms shows results when classifiers are trained over two scenarios (i.e.
(EAU-NUM), (EAU-FREMM) and (FREMM-NUM)) and applied to requisites
in the single remaining scenario. Finally, the last group shows results from an
in-domain setting, when the 80% of requisites from all scenarios are used to
train classifiers, while the remaining 20% are used as test set. In all experi-
ments, SVM parameters are estimated over an held-out 20% of the training
data. Results, especially when lexical and grammatical features are considered,
i.e. the BoW + N-words + N-POS model, are very good and an accuracy higher
of 93% is achieved. Moreover, the system robustness is very promising, as ac-
curacy higher than 85% is reached even in out-of-domain tests. Errors refer to
reasonable and genuinely ambiguous cases. For example, the system labels as
FNC both requisites “ The CMS shall display the progress of each engagement. ”
and “ The CMS shall display single manoeuvre request within ... ”, although this
latter is associated to OPR. Once a specific requisite is located, the Informa-
tion Extraction (IE) System (Fig. 1) carries out the extraction of its relevant
information, as a slot-filling process over the reference templates. Templates are
automatically generated from the analysis of a Domain Ontology , which pro-
vided a model of the application domain as well as an abstraction of individual
requirement types. These types ontologically determine different capabilities , i.e.
desired characteristics of a target system. Moreover, the ontology provides hi-
erarchies that group capabilities according to their semantics and the expected
grain of analysis. Coarse grained capabilities refer to high level system charac-
teristics, such as such Resource Management , that in turn groups together
several fine-grained capabilities. These latter specialize the considered aspects,
e.g. Navigation Radar (NAV), that specializes the notion of Resource Man-
agement in Navigation Radar systems . The IE system is asked to associate a req-
uisite like “ The CMS shall monitor information transmitted by the Navigation
Radar ” to the NAV template, recognizing its finer-grained aspect. The database