Discrimination - Neural Networks: Methodology and Applications - page 354

Fig. 6.16. Forces on the hyperplane. At iteration t , the example k incorrectly
classified attracts the hyperplane. Its contribution to the weight update is indicated
by the vector c k ( t ) y k x k , which is added to w ( t )togive w ( t +1)

that acts on the hyperplane. Note that that force derives from a potential,
which is nothing but the partial cost V (this is why the latter is usually called
potential in the literature written by physicists working in this field; we will
not use this term here in order to avoid confusions with the neuron potential).
Since the hyperplane contains the origin of the extended input space, this force
will make it turn around the origin. If V ( z ) is a non-increasing function of
its argument, then c k ≥ 0. As we can see on Fig. 6.16, if the stability of

x k is negative, then the force attracts the hyperplane towards x k , like if the
example tried to pass to the other side of the hyperplane. Conversely, if the
stability of x k is positive, the example repels the hyperplane.

Remark. The rotation angle is proportional to the learning rate µ .Ifitis
too large, the effect of the force may be excessive and produce oscillations
upon successive iterations.

The hyperplane orientation is stabilized, and the algorithm converges,
when a balance between the forces produced by examples on both sides of
the hyperplane is reached. If the partial cost V is zero for positive stabilities,
only the wrongly classified patterns produce forces, which are attractive, on
the hyperplane. If V = 0 for positive stabilities, as in the Minimerror algo-
rithm, the correctly classified patterns also produce (repulsive) forces on the
hyperplane.
If the examples of the training set are not linearly separable, training with
partial costs that diverge at negative stabilities may exhibit convergence prob-
lems. In non-separable problems, the misclassified patterns attract the hyper-
plane with forces that are stronger the farther the examples. The hyperplane
orientation may thus oscillate during successive iterations and never stabilize.