Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 104

will revisit these ideas in chapter 8, in a model of early
visual processing and learning.
An important limitation of the Kohonen network de-
rives from the rigidity of its representation — it does
not possess the full power of distributed representations
because it cannot “mix and match” different units to
represent different combinations of features. However,
we will see later how we can build a neighborhood bias
into the lateral connectivity between units, which ac-
complishes effectively the same thing as the Kohonen
network, but does so within the more flexible kWTA
framework described above. Such an approach is more
similar to that of von der Malsburg (1973), which also
used explicit lateral connectivity.
Finally, a number of models have been constructed
using units that communicate both excitation and inhi-
bition directly to other units (McClelland & Rumelhart,
1981; Grossberg, 1976, 1978). The McClelland and
Rumelhart (1981) model goes by the name of interac-
tive activation and competition (IAC), and was one of
the first to bring a number of the principles discussed
in this chapter to bear on cognitive phenomena (namely
the word superiority effect described previously). Inter-
active activation is the same as bidirectional excitatory
connectivity, which provided top-down and bottom-up
processing in their model. Grossberg (1976, 1978) was
a pioneer in the development of these kinds of bidirec-
tionally connected networks, and was one of the first to
document many of the properties discussed above.
These models have some important limitations, how-
ever. First, the fact that special inhibitory neurons were
not used is clearly at odds with the separation of exci-
tation and inhibition found in the brain. Second, this
kind of direct inhibition among units is not very good
at sustaining distributed representations where multiple
units are active and yet competing with each other. The
case where one unit is active and inhibiting all the others
is stable, but a delicate balance of excitation and inhi-
bition is required to keep two or more units active at
the same time without either one of them dominating
the others or too many units getting active. With sep-
arate inhibitory neurons, the inhibition is summed and
“broadcast” to all the units within the layer, instead of
being transmitted point-to-point among excitatory neu-
rons. This summed inhibition results in smoother, more

consistent activation dynamics with distributed repre-
sentations, as we saw in the earlier explorations.

3.6 Constraint Satisfaction

Having explored the distinct effects of excitation and
inhibition, we can now undertake a more global level
of analysis where bidirectional excitation and inhibition
can be seen as part of a larger computational goal. This
more global level of analysis incorporates many of the
more specific phenomena explored previously, and con-
solidates them under a unified conceptual and mathe-
matical framework.
The overall perspective is called constraint satisfac-
tion , where the network can be seen as simultaneously
trying to satisfy a number of different constraints im-
posed on it via the external inputs from the environ-
ment, and the weights and activation states of the net-
work itself. Mathematically, it can be shown that sym-
metrically connected bidirectional networks with sig-
moidal activation functions are maximizing the extent
to which they satisfy these constraints. The original
demonstration of this point was due to Hopfield (1982,
1984), who applied some ideas from physics toward
the understanding of network behavior. These Hopfield
networks were extended with more powerful learning
mechanisms by in the Boltzmann machine (Ackley,

Hinton, & Sejnowski, 1985).
The crucial concept borrowed from physics is that of
an energy function . A physical system (e.g., a crystal)
has an energy function associated with it that provides a
global measure of energy of the system, which depends
on its temperature and the strength of the interactions
or connections between the particles (atoms) in the sys-
tem. A system with a higher temperature has particles
moving at faster velocities, and thus higher energy. The
interactions between particles impose the constraints on
this system, and part of the energy of the system is a
function of how strong these constraints are and to what
extent the system is obeying them or not — a system
that is not obeying the constraints has higher energy,
because it takes energy to violate the constraints. Think
of the constraints as gravity — it takes energy to oppose
gravity, so that a system at a higher elevation above the
ground has violated these constraints to a greater ex-