Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 107

a) No KWTA b) KWTA

In actual neurons, there is plenty of naturally occur-
ring noise in the precise timing of output spikes. How-
ever, when we use the rate-code functions instead of
discrete spikes, we lose this source of noise (even in
the noisy XX1 function, which builds the noise into the
shape of the function — see section 2.5.4). Thus, in
the relatively rare cases where we have a network that
is suffering from excessive local maxima problems, we
can add some additional noise back into the activation
function (or we can switch to spikes). However, when
dealing with ambiguous stimuli with truly equal pos-
sible interpretations (see the Necker cube example be-
low), noise is also necessary to “break the tie.”
In especially tough constraint satisfaction problems,
we may need to resort to a special technique called
simulated annealing (Kirkpatrick, Gelatt, & Vecchi,
1983), which simulates the slow cooling methods used
to create high-quality metals. In a neural network, this
can be simulated by gradually reducing the level of
noise as the network settles. The idea is that early on in
processing, you want to explore a wide range of differ-
ent activation states in search of better maxima, which
is facilitated by relatively high levels of noise. Then,
as you home in on a good state, you want to reduce the
noise level and ease your way into the maximum. There
is a facility for specifying an annealing schedule in the
simulator that controls the noise level as a function of
the number of processing cycles performed during set-
tling, but we do not use this in any of the simulations
for this text.

Unit 1 Unit 1

Figure 3.27: Illustration in two dimensions (units) of how
kWTA inhibition can restrict the search space to a smaller
subset of possible patterns. a) represents a network without
kWTA inhibition, which can explore all possible combina-
tions of activation states. b) shows the effect of kWTA inhi-
bition in restricting the activation states that can be explored,
leading to faster and more effective constraint satisfaction.

wide range of different overall activity levels, a network
with kWTA inhibition can only produce states with a
relatively narrow range of overall activity levels (fig-
ure 3.27).
The advantages of so restricting the search space is
that the network will settle faster and more reliably into
good states, assuming that these good states are among
those allowed by the kWTA algorithm . Thus, one has to
adopt the idea of sparse distributed representations and,
through learning or other means, have a network where
all the important possible representations are within the
range of activities allowed by the kWTA function. We
will see that such sparse distributed representations are
quite useful for the cognitive tasks studied in later chap-
ters, and that the network with kWTA inhibition does
indeed settle faster and more reliably into good con-
straint satisfaction solutions (see Vecera & O'Reilly,
1998 for a comparison between networks with and with-
out kWTA constraints).

3.6.3 The Role of Inhibition

The set point-like kWTA inhibitory function has an
important impact on the constraint satisfaction perfor-
mance of the network. Basically, constraint satisfac-
tion is a form of parallel search , where the network
searches through a number of different possible states
before finding one that satisfies the constraints fairly
optimally. Because each unit is updating in parallel,
the search proceeds in parallel instead of sequentially
visiting a huge number of distinct states in sequence.
Viewed in this way, the role of kWTA inhibition is to
restrict the search space dramatically. Thus, instead
of having the possibility of going into states with a

3.6.4 Explorations of Constraint Satisfaction: Cats

and Dogs

We will begin our exploration with a simple semantic
network intended to represent a (very small) set of re-
lationships among different features used to represent a
set of entities in the world. In our case, we represent
some features of cats and dogs: their color, size, fa-
vorite food, and favorite toy. The network contains in-
formation about a number of individual cats and dogs,
and is able to use this information to make generaliza-