Discrimination - Neural Networks: Methodology and Applications - page 344

pointing out that Hebb's rule has very poor performances in Machine Learn-
ing. However, even if its interest for applications is mainly historical, we will
show that it is possible to accelerate the convergence of some algorithms by
initializing the weights with Hebb's rule.

Remark. If the weights were not normalized after each iteration to satisfy the
constraint = constant, it would be possible to control the convergence,

and stop the algorithm as soon as the corrections become parallel to the
weights, that is, when w ( t +1)

w

· w ( t )= w ( t +1) w ( t ) (within the accuracy

limits imposed by the considered application).

In the rest of this section we review some partial costs proposed in the
literature.

6.4.4 Cost Functions for the Perceptron

The cost function that seems most appropriate intuitively is the number of
training errors. The corresponding partial cost is shown on Fig. 6.9, and can
be written as

V ( z )= Θ ( − z ) ,

where Θ ( u ) is the Heaviside function defined at the beginning of this chapter.
It takes the value 1 if the example is incorrectly classified, and 0 otherwise.
At its minimum, the total cost is the smallest fraction of examples incorrectly
classified. This cost function is not differentiable, and cannot be minimized
using a gradient descent. Its minimization is performed by combinatorial op-
timization techniques, or using simulated annealing, described in Chap. 8.

Fig. 6.9. Partial cost corresponding to the number of training errors