A Geometric Approach to Feature Ranking Based Upon Results of Effective Decision Boundary Feature Matrix - Feature Selection for Data and Pattern Recognition - page 53

Fig. 4.2 A piece of true decision boundary, its linear approximation and the local discriminative
direction N ij = m i − m j

code vector in M a class label. The classification rule associated with an LVQ is:

N

ʨ LVQ : R .

Note the Nearest Neighbor nature of this classification rule: each vector in

ₒ y , x ₒ L (ʩ( x ))

N
is assigned to the same class as its nearest code vector. Thus, decision regions are
defined by the union of Voronoi regions of code vectors with the same label. Note
also that the decision boundary is defined only by those hyperplanes

R

S i , j such that

m i and m j have different labels.
An LVQ can be trained to find an approximation of the Bayes boundary. LVQ
training algorithms have been originally proposed by Kohonen [ 22 ]. Here we use a
more recent algorithm known as Bayes VQ (BVQ), formally defined as a gradient
descent algorithm for the minimization of the error probability. It strongly resem-
bles Kohonen's LVQ2.1, however, formal derivation introduces also some modifica-
tions that improve performances and robustness. The BVQ algorithm is an iterative
punishing-rewarding adaptation schema. At each iteration, the algorithm considers a
sample randomly picked from the training set. If the sample turns out to fall “on” the
decision boundary, then the position of the two code vectors determining the bound-
ary is updated, moving the code vector with the same label of the sample towards
the sample itself and moving away that with a different label. Since the decision
boundary is a null measure subspace of the feature space, we have zero probability
to get samples falling exactly on it. Thus, an approximation of the decision boundary
is made, considering those samples falling close to it. Due to lack of space we cannot
report the BVQ algorithm here. The algorithm is described in [ 9 ].
Having a trained LVQ, the calculus of the feature rank is straightforward and is
given by the following BVQ-based Feature Ranking (BVQ-FR) Algorithm 1.