Neural Networks without Training for Optimization - Neural Networks: Methodology and Applications - page 450

energy) of this physical system. In order to get close to those fundamental
states, some results of statistical physics are used.

Two hypotheses on the physical system gave rise to two families of opti-
misation methods: the canonical analysis and the microcanonical analysis.

8.5.1 Canonical Analysis

Canonical analysis assumes that the physical system under investigation is
not isolated: it can exchange heat with its environment. As a consequence, we
will include in the methods a temperature parameter, denoted by T .
At a given temperature, the thermodynamical equilibrium of such a system
is a state such that the free energy of the system is minimum. This free energy
is defined as the difference between the internal energy and the temperature-
entropy product. Moreover, it is well known that the entropy is zero at zero
temperature. Those results can be used to solve an optimisation problem by
associating the cost function of the problem with the internal energy of the
system.
During the search of an optimum, it is important to explore a large terri-
tory in the solution space: to this end, thermal noise is used.
Finally, statistical physics teaches us that, at thermodynamical equilib-
rium, the system states are distributed according to a Boltzmann law: the
probability that the system be in a state of energy E 0 is given by

exp

E 0
kT

−

P ( E 0 )= .

Z ( T )

In this equation, k is the Boltzmann constant and Z ( T ) is a normaliza-
tion function ensuring that the sum of the probabilities of all the accessible
energies is 1. From this property, it is clear that the most probable states at
thermodynamical equilibrium are those of minimal energy. This property is
of great interest to solve optimisation problems.

Remark. Thenumberofpossiblestatesis finite since we are considering a
combinatorial problem: therefore the energy states are discrete . As a conse-
quence, the notion of probability can be used; for a problem with continuous
variables, where the energies have continuous values, the notion of probability
density function, introduced in Chap. 2, should be used.

8.5.1.1 Simulated Annealing

Moreover, metallurgy teaches us that a good way to reach low energy states
of a solid consists in heating the material, and then letting it cool slowly.
That process, called annealing, forces the system evolution towards low energy
states; a slow cooling process (as opposed to a fast cooling, called quenching)