Discrimination - Neural Networks: Methodology and Applications - page 355

To avoid this problem, the learning rate µ may be decreased throughout the
learning procedure. The same remark applies to on-line (adaptive) training: in
that case, the solution not only depends on the rate at which µ was modified,
but also on the order of presentation of the patterns.

6.5 Beyond Linear Separation

We have seen that a perceptron can only implement linear separation of the
inputs. To overcome that limitation, multilayered networks were introduced
in Chap. 1. However, other possibilities exist. One of them is to use non-
linear potentials. An example is presented in the next section. It allows the
implementation of spherical separation surfaces using a generalized percep-
tron with the same number of parameters as a linear perceptron. Clearly,
restricting to hyperspheres is still an important limitation that can be over-
come by using two very different approaches. Either the network complexity is
increased through the addition of hidden neurons using incremental methods,
or we increase the perceptron complexity is increased, as in “support vector
machines”. The latter can find discriminant surfaces of arbitrary shapes at
the price of learning a larger number of parameters.

6.5.1 Spherical Perceptron

A hyperspherical discriminant surface may be implemented through a simple
generalization of the linear perceptron. Let us define a spherical potential or
activity,

v S = N

w i ) 2 w 0 .

( x i − −

i =1

The sum over i is the square of the distance between the input x and the
weight vector in input space w =[ w 1 , w 2 ,..., w N ] T , which is the center of a
hypersphere of radius w 0 . The perceptron output is

σ S =sign( v S ) .

Thus, σ S 1ifit

is inside (see Fig. 6.17). Notice that the spherical perceptron has the same
number of parameters as the linear perceptron. Only the definition of the
potential is different. All the training algorithms for the linear perceptron
can thus be easily transposed to the spherical perceptron, by introducing the
following expression:

= +1 if the point x is outside the hypersphere, and σ S = −

z S = y k v S
for the aligned field of example k . It is important to emphasize that, in this
case, the weights must not be normalized, since that would impose that the
hypersphere center be at a fixed distance of the origin, given by the normal-
ization constant.