Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 98

tially different from the previous case (a difference of
1-2%).

adds significantly to the amount of computation re-
quired for a given simulation. In addition, we might
be able to do this in such a way as to avoid the need
to use slow time constants in updating the units, which
would save even more processing resources.
The class of inhibitory functions that we adopt here
are known as k-winners-take-all (kWTA) functions
(Majani, Erlarson, & Abu-Mostafa, 1989), of which
we develop two different versions ( basic kWTA and
average-based kWTA ). A kWTA function ensures that
no more than k units out of n total in a layer are ac-
tive at any given time. From a biological perspective, a
kWTA function is attractive because it captures the set
point property of the inhibitory interneurons, where the
activity level is maintained through negative feedback
at a roughly constant level (i.e., k ). From a functional
perspective, a kWTA function is beneficial in that it en-
forces the development of sparse distributed representa-
tions, whose benefits were discussed previously.
There is an important tension that arises in the kWTA
function between the need to apply a firm set point con-
straint on the one hand, and the need for flexibility in
this constraint on the other. We know that a firm activa-
tion constraint is needed to prevent runaway excitation
in a bidirectionally connected network. We will also see
in the next chapter that a strong inhibitory constraint is
important for learning. Nevertheless, it also seems rea-
sonable that the network would benefit from some flex-
ibility in determining how many units should be active
to represent a given input pattern.
One general solution to this firmness/flexibility trade-
off that is implemented to varying extents by both ver-
sions of kWTA functions is to make the k parameter an
upper limit , allowing some flexibility in the range be-
tween0and k units, and in the graded activation values
of the k active units. This upper limit quality results
from the impact of the leak current, which can be set to
be strong enough to filter out weakly excited units that
might nevertheless be in the top k of the units. Thus,
perhaps a more accurate name for these functions would
be k-or-less WTA. In addition, the two different ver-
sions of kWTA functions represent different tradeoffs
in this balance between a firm constraint and flexibility,
with the basic kWTA function being more firm and the
average-based kWTA function being more flexible.

Next, set input_pct to 25, Apply , NewInput , and
Run .
Again, you should observe only modest increases in
activity level.
Thus, the network appears to be relatively robust to
changes in overall input excitation, though it does show
some effect. Perhaps a more dramatic demonstration
comes from the relatively small differences between the
initial activity level in the hidden units compared to the
subsequent level after the input from the second hidden
layer has kicked in. It is this approximate set point be-
havior, where the system tends to produce a relatively
fixed level of activity regardless of the magnitude of the
excitatory input, that is captured by the inhibition func-
tions described in the next section.

, !

Question 3.12 Explain in general terms why the system
exhibits this set point behavior.

Last, you can also change the activation function,
using the ActFun button.
You should see that the same basic principles apply
when the units use a spiking activation function.

, !

To continue on to the next simulation, you can leave
this project open because we will use it again. Or, if you
wish to stop now, quit by selecting Object/Quit in the
PDP++Root window.

3.5.3 The k-Winners-Take-All Inhibitory Functions

We saw in the previous section that the appropriate
combination of feedforward and feedback inhibition
gives rise to the controlled activation of excitatory neu-
rons, even when they are bidirectionally connected. Be-
cause the inhibitory interneurons are essentially just
sensing the overall level of excitation coming into a
layer and delivering some roughly proportional level
of inhibition, it would seem plausible that the general
effects of these inhibitory neurons could be summa-
rized by directly computing the inhibitory current into
the excitatory neurons using an inhibitory function that
takes into account this overall level of excitation. Do-
ing so would avoid the need to explicitly simulate the
inhibitory neurons and all of their connectivity, which