Robust Requirements Analysis in Complex Systems through Machine Learning - Trustworthy Eternal Systems via Evolving, Software Data and Knowledge - page 51

4.1 Experimental Evaluation

Machine-learning techniques for requirement analysis described in the previous
Sections have been implemented in a Requirement Analysis System, according
to the Architecture shown in Figure 1. The resulting adaptive system has been
applied to a real scenario, i.e. the requirement analysis of a Naval Combat Sys-
tems, focusing on the SW system, namely the Combat Management System
(CMS). This Section provides the empirical evaluation of system functionalities,
such as the Requirement Identification (RI) and Information Extraction (IE)
as applied to the CMS requirement analysis. Requisites here refer to different
aspects of the CMS, such as Functional Requirements (FNC) or Performance
requirements (PRF). The dataset adopted in our tests is made of 4,727 anno-
tated requirements, related to three different scenarios, called EAU, FREMM
and NUM . Each requisite has been labeled according to one of the five requisite
types, which are specific aspects of the resulting system, such as FNC or PRF,
asshowninTable1.

Tabl e 1. Requisite Types

ABBR Type Number

NFC Non-Functional Requirements 74

DCC Design and Construction Constraints 288

OPR Operator requirements 2,587

PRF Performance Requirements 249

FNC Functional requirements 1,529

Total 4,727

The Requirement Identification system of Figure 1 has been trained to recog-
nize and characterize requirements. The module applies Support Vector Machine
classification to associate each requirement its suitable specific class, reflected
into the corresponding type. Different models of observable text properties al-
lowed to investigate different linguistic information and to identify the most
informative representations for the learning algorithm:
- The Bag-of-Word ( BoW ) model mainly accounts for the lexical information:
requisites are mapped into sets of words, neglecting word order, i.e. syntactic
information.
- The N-gram of Words ( N-Words ) model provides a first form of grammatical
information, by mapping short word sequences into n -grams of words.
- A Bag-of-Word and N-gram of Part-of-Speech ( N-POS ) introduces gram-
matical information as it attaches part-of-speech to n -grams, by further
generalizing the sequences of words in the textual requisite.
- The Comprehensive ( BoW + N-Words + N-POS ) model accounts for all
the previous information, i.e. as it includes Bag-of-Words, n -grams of Words
and n -grams of Part-of-Speeches.

The objective of the experiments is also to measure and compare the adap-
tion capabilities of SVM classifiers to different scenarios: the idea is that SVMs
should be able to induce meaningful classification models from the data avail-
able in a specific scenario, i.e. the in-domain scenario, but also provide accurate
predictions even when applied to different, i.e. out-of-domain , scenarios.