Neural Networks without Training for Optimization - Neural Networks: Methodology and Applications - page 468

where y is the vector of neuron outputs, i.e., the state vector of the system:
y =[ y 1 ,y 2 ,...,y N ] T .
That function is an N -variable function that generally has a large number
of local minima.
A natural link between such a function and the energy function of a com-
binatorial problem can be found. That is the reason why recurrent neural
networks are interesting for solving optimization problems.

8.6.4.2 Analog Hopfield Neural Networks

Hopfield described a continuous (called analog) version of the above binary
recurrent neural network [Hopfield 1984]. In that case, the associated energy
function is defined as

y i

N N N N

1
2

I i y i + α
γ

Ψ − 1 ( y )d y , E ( y )= − w ij y i y j −

0

i =1 j =1 i =1 i =1

where α is a positive real number, and Ψ is the activation function of the
neurons.
Generally, the last term in that energy function is negligible with respect
to the previous ones, when the slope γ is large, or when α is small.
Hopfield and Tank first applied that type of neural network to combina-
torial optimization [Hopfield et al. 1985].
A potentially interesting feature of that type of network is the fact that
they can give rise to the hardware implementation of analog ASICs, by in-
terconnecting a set of resistors, some non-linear amplifiers with symmetric
outputs, external current sources and some capacitors [Newcomb et al. 1995].
The equations that govern the evolution of a continuous neuron i is the
following:

⎧
⎨

d v i
d t = µ i

∂E ( y )
∂y i

− α i v i −

y i =tanh µ T

,

⎩

where µ i =1 /τ i is a positive real number which parameterizes the convergence
speed, α i is a positive real number, T is the temperature (inverse of the slope at
the origin of the neuron's activation function) and E ( y ) is the energy function
of the problem, which is not necessarily quadratic.
The derivative of the energy function E versus time can be written, from
the above equations, as

d v i
d t

2

N N N

d E
d t =

∂E
∂y i

d y i
d t =

d y i
d v i

d y i
d t .

− τ i − α i v i

i =1 i =1 i =1