Information Technology Reference
In-Depth Information
methods and tools is always a challenging task [37-39] as one has to use subjects that
participate actively in the study as well as have different backgrounds and experiences
that can influence the results obtained.
The goal of our case first case study (Sect. 5.1) was to evaluate the
time-
effectiveness
(Sect. 5.1.1) of using EA-Miner as well as its
accuracy
(measured by
precision and recall - Sect. 5.1.2) for undertaking some AORE activities comparing it
to a manual approach. Measuring accuracy in terms of precision and recall is
commonly used in related works in order to assess the identification capabilities of
tools [12, 36] (Sect. 7). The evaluation was conducted using two requirements
documents and two different techniques (manual vs. EA-miner based) for identifying
and structuring the concepts into an AORE model based on viewpoints.
Our second case study (Sect. 5.2) aimed at exposing EA-Miner to a real industrial
exercise using the tool to conduct an AORE analysis on a set of documents, provided
by Siemens, related to a toll collection system. The goal of this case study was not
comparative as the one in Sect. 5.1 but was to investigate how the tool behaved, what
benefits it brought and what challenges it faced in a real-world situation.
5.1 First Case Study: EA-Miner Versus Manual Approach
The two requirements documents used in the first case study are part of real world
system specifications. The first is a well-known specification document of a light
control system [40] that is commonly used in evaluating requirements approaches, as
for example in [34]. The size of this file is approximately 11 pages (3,671 words).
The second document is a description of a library system used in Lancaster
University. In terms of structure the library document is different than the light
system as it is structured in a “user manual” style highlighting in detail the use of the
system. The size of this file is larger than the previous - approximately 29 pages
(6,504 words).
The manual analysis
conducted by one requirements engineer was compared
against an EA-Miner based analysis done by a second requirements engineer. After
both engineers finished their tasks they sent their results to a third senior requirements
engineer who compared the results. The role of the senior engineer was to check the
accuracy verifying what concepts were correctly identified by both engineers.
Different engineers performed the manual and EA-Miner based analysis because if
the same engineer was used than s/he would learn about the document and the second
analysis would be biased. We were also careful enough to select engineers with a
similar level of expertise in AORE (both engineers are PhD students at the same
group at Lancaster University and both have their research topics related to AORE).
The evaluation shows very promising results about EA-Miner that highlight the
contributions this tool offers for AORE.
5.1.1 Results of Time-Effectiveness
In order to clarify this variable it is important to explain what task each engineer
conducted. The general task of both was to mine a requirements document (such as
the one in Fig. 2a) in order to:
Search WWH ::
Custom Search