Information Technology Reference
In-Depth Information
Considering the accuracy results we gathered data related to the identification of
viewpoints and early aspects. The data is shown in Table 3, 4 and 5 and represent
different comparison approaches which are explained next. First we give general
explanations to help understand the tables:
As explained before the total list of correct abstractions was assumed to be the list
of the senior engineer who compared the results given to him/her. This number is
used in the recall calculation in all tables and is assumed to be:
Total correct viewpoints: 11 for light control system and 8 for library system;
Total correct early aspects: Seven for light control and eight for library.
For EA-Miner's viewpoint's precision and recall it is important to mention that the
threshold filter (i.e., selecting the n most significant viewpoints based on
frequency) defines the number of candidate viewpoints. This is done because in
large documents the number of candidate viewpoints can be very long (Sect. 3) and
yields a direct impact on the results. Another reason we set a threshold is because
we wanted the accuracy results to remain consistent with the time analysis done
previously that considered the identification to be fully automated (frequency
filtering is an operation offered by EA-Miner). Therefore for Tables 3 and 4 we
consider:
Total number of viewpoint candidates for EA-Miner: 10 for both light
control and library in Table 3 (highlighted in bold).
Total number of viewpoint candidates for EA-Miner: 20 for both light
control and library in Table 4 (highlighted in bold).
In Table 5 we considered that the engineer using EA-Miner spent some more time
using the tool (e.g., observing different thresholds), the guidelines shown in
Sect. 4.3 as well as his/her personal experience. We did not record the exact time
(even though it was still significantly less than the manual approach) spent for this,
therefore it does not comply with the data on Sect. 5.1. The reason we present this
data is to investigate the results of the combination of EA-Miner with user
knowledge and process guidance.
The only data that varies in all tables are precision and recall of viewpoints
(highlighted in all tables) as they are influenced by the threshold.
Tables 3 and 4 shows the results with a threshold of 10 and 20 viewpoints
respectively. For the identified viewpoints, recall tends to improve with the threshold size
since EA-Miner tends to list more correct viewpoints. For example in the light control
system the correct viewpoints listed increases from two (Table 3) to seven (Table 4)
which means that five correct viewpoints are between the 11th and 20th most significant.
This improves the recall from 18.2 to 63.6% which is superior than the manual approach.
Precision data for EA-Miner's identified viewpoints is highly influenced by the
threshold. If we increase the threshold too much (e.g., 100, 200) it is likely that
precision will be low as the total number of viewpoints identified is much higher
when compared with the total of correct identified. On the other hand, if we set the
threshold too low (e.g., 5, 10) precision can vary depending if the small list is a
“good” list. For example, in Table 3, precision of EA-Miner is 20% for the light
control and 50% for the library system meaning that in the latter case even though the
list was small, it was “good”. Table 4 shows that increasing the threshold improves
precision for the light control and decreases for the library system.
Search WWH ::




Custom Search