Weighting of Features by Sequential Selection - Feature Selection for Data and Pattern Recognition - page 84

Fig. 5.4 Classification accuracy observed for decision algorithms in sequential backward elimina-
tion process of attributes and rules, in relation to the number of considered features
are very close. The fact that here the correct classification ratio is lower than before
results from the different attitude to preference orders of features. In forward selec-
tion it was treated as one more element to be established, thus better adjusted to the
task under study, while for backward elimination in the initial stage the preference
orders are arbitrarily assigned and remain unchanged throughout all processing.
The two types of classifiers used in experiments have very distinctive properties,
and the differences between them are also visible in obtained by them rankings of
considered variables, which shows bias that all wrappers are prone to. For example,
for ANN the feature “not” is rejected as the first one, while for DRSA classifier it
is kept almost to the end. When we compare orderings of variables from forward
with backward selection for rule classifiers, even though the type of the inducer
employed is the same, we cannot say that they are reversed. All these observations
illustrate how different perspectives in which attributes are considered can change
relationships and dependencies detectable among studied elements, to the point of
completely different relevance of the same features and their resulting weightings.
5.6 Concluding Remarks
Backward and forward sequential search are two approaches to feature selection, with
the opposite starting points in the feature space. In forward selection we commence
with the empty set of variables to which we add one element at a time. In backward
elimination we begin with the entire set of attributes, which are reduced one by
one. The two procedures are relatively simple to apply, even though could be time
consuming, depending on the number of available variables to be tested, and a type
of inducer used in validation of candidate feature subsets.
In the experiments dedicated to selection of variables two distinctively different
classifiers were employed, connectionist approach of artificial neural networks and
rule-based exploiting rough set theory incorporating dominance relation.