Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 106

Note that as we mentioned previously, x i w ij x j and

i both appear in the double-sum term in the har-
mony equation, and we assume that w ij = w ji due to
symmetry, so this cancels out the term. The result is

just the same as equation 3.11, meaning that updating
the activations of a network of these units is indeed the
same as maximizing the harmony in the network.
The preceding analysis can also be made for a net-
work using sigmoidal units, which are more similar to
the point neuron activation function we have been us-
ing (see section 2.5.4). For this to work out mathemat-
ically, the harmony/energy equation needs to be aug-
mented with a stress or entropy term, which reflects
the extent to which the activation states of units in the
network are “undecided” (e.g., in the middle of their
range of values). The resulting overall equation is called
goodness in harmony-based parlance, and free energy
in physics-based lingo. However, the results of adding
this additional term do not typically affect the ordinal
relationship between the harmonies of different states
— in other words, harmony and goodness are usually
fairly redundant measures. Thus, we will restrict our
focus to the simpler harmony term.
Given that the point neuron activation function used
in Leabra will result in the increased activity of units
connected by strong weights, just like the simpler cases
above, it is not surprising that the use of this function
will tend to increase the overall harmony of the network.
Thus, we can understand the general processing in these
networks as performing constraint satisfaction by work-
ing to maximize the overall harmony. The exercises that
follow explore this idea.

Local Minimum

Global Minimum

Figure 3.26: A local minimum in the energy function. Noise
can shake the system (represented by the ball) so that it finds
the global minimum (or at least a better local minimum).

are referred to as maxima of the harmony function (or
minima of the energy function), and they correspond to
attractor states.
The process of converging on an attractor over set-
tling (e.g., as represented by figure 3.18) is thus iso-
morphic to the process of updating the activations and
improving the harmony of the network's states. Thus,
constraint satisfaction and the associated mathematics
of the harmony or energy function provides a nice for-
malization of the notion of an attractor. In the next sec-
tion, we will see how noise and inhibition interact with
this constraint satisfaction/attractor process.

3.6.2 The Role of Noise

Noise (e.g., in the membrane potential or activation val-
ues) can play an important role in constraint satisfac-
tion. Basically, noise helps to keep things from get-
ting stuck . Think about how you get ketchup or Parme-
san cheese out of their containers — you shake them.
This shaking is a form of noise, and it keeps the system
(ketchup) from getting stuck in a suboptimal state. Sim-
ilarly, noise added to the activations of units can prevent
the network from getting stuck in a suboptimal state that
fails to satisfy as many of the constraints as other, more
optimal states (figure 3.26). These suboptimal states are
called local maxima or local minima (depending on
whether you're using harmony or energy, respectively),
as compared to the (somewhat mythical) global max-
ima (or minima), which is the most optimal of all pos-
sible states (i.e., having the maximum harmony value
possible). In all but the simplest networks, we must
typically accept only local maxima. However, by using
noise, we are more likely to find better local maxima.

3.6.1 Attractors Again

In section 3.4.4, we introduced the notion of an attrac-
tor , which is a stable activation state that the network
will tend to settle into from a range of different start-
ing states (the attractor basin). We can now relate the
notion of an attractor to the constraint-satisfaction ideas
presented above. Specifically, the tendency of the ac-
tivation updates to maximize the harmony of the net-
work means that the network will tend to converge on
the most harmonious states possible given a particular
set of input constraints. These most harmonious states