Neural Networks without Training for Optimization - Neural Networks: Methodology and Applications - page 478

development truncated around a saddle point (see below): that is the main
difference between simulated annealing and mean field annealing.
The performances of mean field annealing are sometimes improved by per-
forming some normalization on groups of neurons [Herault et al. 1989; Peter-
son et al. 1989]. Some constraints of the type one output out of N equal to 1
are then naturally taken into account, without resorting to additional penalty
terms in the energy function. Similarly to microcanonical annealing [Herault
et al. 1993], it is possible to build a microcanonical mean field annealing al-
gorithm [Lee et al. 1991b].
The equations of the mean field approximation, giving the neuron outputs
between − 1 and +1, are defined as follows:

y i =tanh

, i =1 ,...,N.

1
T

∂E ( y )
∂y i

−

In mean field theory, the temperature is fixed, and it is chosen around the
critical temperature T c , which is generally very di cult to estimate. That
is the reason why it is preferable to use the mean field annealing with a
decreasing temperature according to a given schedule during convergence [van
den Bout et al. 1989, 1990; Peterson et al. 1988, 1989; Peterson 1990; Herault
et al. 1989, 1991]. Contrary to simulated annealing, mean field annealing is
a deterministic and intrinsically parallel method, described by the following
system of differential equations:

y i −

tanh

, i =1 ,...,N.

d y i
d t =

1
T

∂E ( y )
∂y i

− µ i −

8.6.5.6 Pulsed Neural Networks

Pulsed neural networks [Herault 1995c] do not suffer the limitations of the
Hopfield networks in terms of constraint violation. They are defined with
models of binary neurons that are more general than those used in the Hopfield
neural networks. In a Hopfield network, the non-linearity of a neuron is only
in its activation function: its potential is a linear function of inputs, and its
output is a non-linear function of its potential. In a pulsed neural network,
the potential of a neuron is a function, possibly non-linear of its inputs, and
the motion equation has a more general form,

y i = Φ [ v i ( k − 1) ,y 1 ( k − 1) ,...,y N ( k − 1)] .

The dynamics associated with that type of networks alternates several con-
straint satisfaction phases, during which the network converges towards valid
solutions, and pulsation phases, during which the network tries to escape
from local minima to go to better minima. In practice, the network proposes
regularly some valid solutions (if they exist) of satisfactory quality. The se-
lected solution will then be chosen among the set of proposed solutions. That