Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 95

expect to get results similar to those just obtained for
g_bar_i.hidden , but be careful — inhibition upon
inhibitory neurons could have interesting consequences.

A more intuitive (but somewhat inaccurate in the de-
tails) way of understanding the effect of inhibition on
inhibitory neurons is in terms of the location of the
thermostat relative to the AC output vent — if you
place the thermostat very close to the AC vent (while
you are sitting some constant distance away from the
vent), you will be warmer than if the thermostat was
far away from the AC output. Thus, how strongly the
thermostat is driven by the AC output vent is analo-
gous to the g_bar_i.inhib parameter — larger val-
ues of g_bar_i.inhib are like having the thermo-
stat closer to the vent, and will result in higher levels of
activation (greater warmth) in the hidden layer, and the
converse for smaller values.

First Run with a g_bar_i.inhib of 4 for compar-
ison. Then decrease g_bar_i.inhib to 3 and Run ,
and next increase g_bar_i.inhib to 5 and Run .
With a g_bar_i.inhib of 3, you should see that
the excitatory activation drops, but the inhibitory level
stays roughly the same! With a value of 5, the excita-
tory activation level increases, but the inhibition again
remains the same. This is a difficult phenomenon to un-
derstand, but the following provide a few ways of think-
ing about what is going on.
First, it seems straightforward that reducing the
amount of inhibition on the inhibitory neurons should
result in more activation of the inhibitory neurons. If
you just look at the very first blip of activity for the
inhibitory neurons, this is true (as is the converse that
increasing the inhibition results in lower activation).
However, once the feedback inhibition starts to kick in
as the hidden units become active, the inhibitory activ-
ity returns to the same level for all runs. This makes
sense if the greater activation of the inhibitory units for
the g_bar_i.inhib = 3 case then inhibits the hidden
units more (which it does, causing them to have lower
activation), which then would result in less activation of
the inhibitory units coming from the feedback from the
hidden units. This reduced activation of the inhibitory
neurons cancels out the increased activation from the
lower g_bar_i.inhib value, resulting in the same
inhibitory activation level. The mystery is why the hid-
den units remain at their lower activation levels once the
inhibition goes back to its original activation level.
One way we can explain this is by noting that this
is a dynamic system, not a static balance of excitation
and inhibition. Every time the excitatory hidden units
start to get a little bit more active, they in turn acti-
vate the inhibitory units more easily (because they are
less apt to inhibit themselves), which in turn provides
just enough extra inhibition to offset the advance of the
hidden units. This battle is effectively played out at
the level of the derivatives (changes) in activations in
the two pools of units, not their absolute levels, which
would explain why we cannot really see much evidence
of it by looking at only these absolute levels.

, !

Set g_bar_i.inhib back to 4 before continuing
(or hit Defaults ).

Roles of Feedforward and Feedback Inhibition

Next we assess the importance and properties of
the feedforward versus feedback inhibitory projec-
tions by manipulating their relative strengths. The
inhib_ctrl control panel has two parameters that
determine the relative contribution of the feedforward
and feedback inhibitory pathways: scale.ff applies
to the feedforward weights from the input to the in-
hibitory units, and scale.fb applies to the feedback
weights from the hidden layer to the inhibitory units.
These parameters uniformly scale the strengths of an
entire projection of connections from one layer to an-
other, and are the arbitrary wt_scale.rel ( r k ) rela-
tive scaling parameters described in section 2.5.1.

Set scale.ff to 0, effectively eliminating the feed-
forward excitatory inputs to the inhibitory neurons from
the input layer.

, !

Question 3.11 (a) How does this affect the behavior
of the excitatory and inhibitory average activity levels?
(b) Explain this result. (Hint: think about the an-
ticipatory effects of feedforward inhibition.) Next, set
scale.ff back to .35 and scale.fb to0toturn
off the feedback inhibition. (c) Now what happens? (d)
Try finding a value of scale.ff (in increments of .05)
that gives roughly the same activity level as the initial
default system — how does this differ from the initial