Evaluating Case Selection Algorithms for Analogical Reasoning Systems - Foundations on Natural and Artificial Computation - page 350

Ta b l e 1 . Decision score results of the experiments: err. is the error rate, ρ is the reduction rate,
time (in seconds), Sens is the sensitivity, Spec. is the specificity and κ is the Kappa coefficient.

Iris Wine Yeast Breast Cancer Abalone

Err. ρ Err. ρ Err. ρ time Err. ρ time Sens. Spec. κ Err. ρ time

None 0,05 0 0,03 0 0,48 0 0 0,28 0 0 0,49 0,82 0,3 0,8 0 0

Rand. 25% 0,04 0,25 0,04 0,25 0,5 0,25 0,02 0,3 0,25 0 0,42 0,81 0,24 0,79 0,25 0,18

Rand. 50% 0,05 0,5 0,06 0,5 0,52 0,5 0,05 0,33 0,5 0 0,44 0,77 0,2 0,79 0,5 0,38

Rand. 75% 0,07 0,75 0,07 0,75 0,5 0,75 0,08 0,33 0,75 0 0,33 0,81 0,15 0,8 0,75 0,57

CNN 0,08 0,85 0,08 0,89 0,52 0,33 0,05 0,33 0,46 0 0,44 0,77 0,21 0,81 0,11 1,26

RNN 0,07 0,85 0,06 0,85 0,52 0,33 0,06 0,34 0,46 0 0,42 0,76 0,18 0,81 0,11 2,91

ENN 0,05 0,04 0,04 0,04 0,42 0,49 0,01 0,24 0,24 0 0,29 0,94 0,24 0,74 0,83 0,02

RENN 0,05 0,04 0,04 0,04 0,42 0,49 0,03 0,24 0,24 0 0,29 0,94 0,24 0,74 0,83 0,08

All-KNN 0,06 0,09 0,03 0,05 0,45 0,57 0,01 0,28 0,4 0 0,36 0,87 0,23 0,77 0,85 0,05

IB2 0,16 0,95 0,12 0,96 0,53 0,44 0,01 0,37 0,61 0 0,47 0,69 0,16 0,81 0,19 0,01

IB3 0,17 0,88 0,14 0,82 0,53 0,44 139,09 0,39 0,56 9,05 0,44 0,68 0,12 0,81 0,19 1132,19

Shrink 0,09 0,9 0,11 0,91 0,53 0,42 0,06 0,37 0,58 0 0,41 0,72 0,13 0,81 0,16 1,67

DROP1 0,05 0,94 0,07 0,9 0,48 0,76 0,45 0,26 0,83 0,04 0,38 0,89 0,28 0,78 0,83 2,8

DROP2 0,05 0,94 0,07 0,87 0,49 0,81 0,49 0,26 0,81 0,04 0,37 0,89 0,26 0,78 0,89 3,01

DROP3 0,05 0,88 0,07 0,82 0,52 0,32 0,2 0,34 0,5 0,02 0,51 0,72 0,23 0,82 0,1 0,46

SPEA2 0,05 0,57 0,05 0,54 0,5 0,51 37,33 0,31 0,52 8,2 0,44 0,79 0,22 0,8 0,5 183,15

NSGA-II 0,06 0,5 0,05 0,51 0,51 0,5 42,48 0,28 0,51 9,32 0,47 0,82 0,3 0,8 0,5 204,14

It is worht mentioning that the MOEA algorithms have similar behaviour and achieve
similar results in the reduction rate and the error. Their reductions are near to the half
size of case memory in all cases, and the results are a slightly higher than control exper-
iments. They show the same result in every experiment so although they are not the best
in any case, they are the most regular respecting the instances distribution. Their weak-
ness is the execution time expended of them. The rest of methods are highly efficient
respect to them.

5 Conclusions

The aim of this work is to provide support to choose the most suitable case selection
method for a given memory case to solve a particular problem. To this end, we propose
an evaluation methodology based on the size of the case memory after the selection
process, the efficiency of the method, and the suitability of the case memory to solve
the problem.
In some works, the evaluation of the case selection process focuses on the valida-
tion of the complete system using an arbitrary repetition of the Hold-Out technique
[22,23,7]. Our methodology proposes the use of a K-NN classifier, avoiding the use of
the complete system. The works described in [21,25,16] also suggest the same strategy
but the case selection process is stopped when the final case memory reaches a deter-
minate size. Unlike our methodology, the case memory size must be known beforehand
and this assumption could not be acceptable in some domains. Moreover we suggest
the use of a Coss-Validation, providing a more flexible approach by tuning the folder
size.