Feature Evaluation by Filter, Wrapper, and Embedded Approaches - Feature Selection for Data and Pattern Recognition - page 39

more than five important variables are eliminated the performance gets worse quickly
and irrecoverably. We can reduce significantly more lower ranking variables before
the performance degrades below a satisfactory level.

3.6.3 Comparison of Feature Reduction Results

For both types of classifiers we can also observe the differences and similarities in
the performance in the feature reduction process while following the two defined
rankings, as shown in Fig. 3.5 . The displayed values correspond to the calculated
difference in classification accuracy for the currently considered number of features,
Relief-based reduction minus DRSA-based reduction. For ANN classifiers the dif-
ferences are calculated only with respect to median classification accuracies.
For artificial neural networks for both increasing and decreasing orders for both
rankings the results are close as the differences are equal to or close to zero. The
discrepancies are visible when there are fewer than a half of features left. Then
reduction of those with lower ranks returns better results for Relief (at maximum
for five variables, when the difference in classification accuracy is 35%), and for
elimination of higher ranking attributes DRSA-based ordering is more advantageous
(at maximum outperforms by 25% for 8, 7 and 9 remaining features).
For rule classifiers for the decreasing order for both rankings the classification
accuracies are almost the same for the first 16-17 reduced variables. Then, for
fewer than 8 inputs, Relief ranking based selection of attributes is outperformed
by DRSA-based reduction. For elimination of higher ranking variables in most cases
the results are better while employing Relief than DRSA-based ranking.
As can be seen in presented graphs, executing sequential backward reduction of
characteristic features driven by ranking of these features obtained previously results
in several cases with significant gains in terms of lower dimensionality, increased
predictive accuracies of the constructed classification systems, and decreased stor-
age requirements. Observations based on performance allow for evaluation of used
attributes and estimation of their relevance for particular tasks.

(a ) (b )

Fig. 3.5 Differences in classification for reduction of characteristic features along Relief and
embedded DRSA ranking, for decreasing orders (Most series), and increasing orders (Least series)
for: a ANN classifier, b DRSA classifier