Information Technology Reference
In-Depth Information
8.6.5.3 Probabilistic Networks
To avoid getting trapped in a high-energy local minimum, another approach
consists in defining probabilistic equations, by adding a noise term in the
motion equations of analog neurons. That noise term is such that its influence
decreases during the convergence of the network [Asai 1995]. The modified
equation of an analog neuron thus becomes
(1
, i
=1
...,N,
d
y
i
d
t
=
y
i
)
2
∂E
(
y
)
∂y
i
−
µ
i
−
+
c
(
t
)
N
i
where the
N
i
are non-correlated sources of white noise with zero mean, and
c
(
t
) provides the decreasing law of the noise. Typically,
c
(
t
) has the following
form:
c
(
t
)=
c
0
exp
.
t
τ
−
In practice, it is necessary to add an uncorrelated noise source to each neu-
ron, and to decrease gradually its power during the convergence. The benefit,
as with simulated annealing, is to provide the network with the capacity of
escaping from high-energy local minima, and thus to converge to much better
solutions. That is the reason why that technique is sometimes called hardware
simulated annealing.
8.6.5.4 Boltzmann Machine
The Boltzmann machine was first described in 1984 and 1985 [Hinton et al.
1984; Ackley et al. 1985]. It can be considered as a combination of the prin-
ciples of simulated annealing with those of binary Hopfield neural networks.
Its architecture is similar to that of a binary Hopfield neural network.
The energy of a Boltzmann network can be expressed under the same form
as the energy of a binary Hopfield neural network,
N
N
1
2
E
(
y
)=
−
w
ij
y
i
y
j
.
i
=1
j
=1
The first step of the updating of a neuron
i
consists in computing the
energy variation generated by its change of state. The energy variation asso-
ciated with the change from 0 to 1 of a neuron output is given by
N
∆
E
i
=
E
y
i
=1
−
E
y
i
=0
=
−
w
ij
v
j
.
j
=1
j
=
i
Then, the state of the neuron changes with the probability given by the
Metropolis criterion used in simulated annealing. During the convergence of
Search WWH ::
Custom Search