Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 99

Our primary interest in constructing an inhibition
function is to achieve a simple and computationally ef-
fective approximation to the behavior of an actual cor-
tical network, specifically its set point nature as repre-
sented by the kWTA function. Thus, as long as our
function captures this target behavior, the details of how
we implement it need not be particularly biologically
plausible themselves. Indeed, as will be clear in the
next section, the implementation is not particularly bio-
logically plausible, but it is efficient, and directly im-
plements the kWTA objective. We can only tolerate
this kind of implausibility by knowing that the resulting
function approximates a much more plausible network
based directly on the inhibitory interneurons known to
exist in the cortex — we assume that all of our kWTA
models could be reimplemented using this much more
realistic form of inhibition and function in qualitatively
the same way (only they would take a lot longer to run!).

this computation for the inhibitory conductance at the
threshold ( gi ), we get:

(3.2)

where ge represents the excitatory input minus the con-
tribution from the bias weight. Networks learn much
better using this ge value here, presumably because
it allows the bias weights to override the kWTA con-
straint, thereby introducing an important source of flex-
ibility.
In the basic version of the kWTA algorithm ( KWTA
in the simulator), we simply compute g i as a value in-
termediate between the gi

values for the k and k +1th
units as sorted by level of excitatory conductance ( g e ).
This ensures that the k+1th unit remains below thresh-
old, while the kth unit is above it. Expressed as a for-
mula, this is:

(3.3)

kWTA Function Implementation

where the constant 0 < q < 1 determines where ex-
actly to place the inhibition between the k and k +1th
units. A value of q = :25 is typically used, which en-
ables the kth unit to be reasonably far above the in-
hibitory threshold depending on how the g i

First, it should be clear that the k units active in a kWTA
function are the ones that are receiving the most excita-
tory input ( g e ). Thus, the first step in computing both
versions of the kWTA functions is to sort the units ac-
cording to g e (note that in practice a complete sort is
not necessary, as will become clear, so the computa-
tion can be optimized considerably). Then, a layer-wide
level of inhibitory conductance ( g i ) is computed such
that the top k units will (usually) have above-threshold
equilibrium membrane potentials with that value of g i ,
while the rest will remain below firing threshold. This
inhibitory conductance is then used by each unit in the
layer when updating their membrane potentials.
Thus, to implement kWTA we need to be able to
compute the amount of inhibitory current that would put
a unit just at threshold given its present level of excita-
tory input, which we write as gi ,where ￿ is the thresh-
old membrane potential value. The necessary equation
should be familiar to you from the exercises in chap-
ter 2, where you used the equilibrium membrane poten-
tial equation (2.9) to compute the threshold level of leak
current. Thus, the specific form of the point neuron ac-
tivation function is important for enabling the appropri-
ate amount of inhibition to be easily computed. Doing

terms are

distributed.
To see the importance of the shape of the distribution,
we have plotted in figure 3.24 several different possi-
ble such distributions. These plots show the values of

(which are monotonically related to g e )fortherank
ordered units. Thus, the g i value computed by equa-
tion 3.3 can be plotted directly on these graphs, and it is
easy to see how far the upper k units are above this in-
hibitory value. The greater this distance (shown by the
bracket on the Y axis in the graphs), the more strongly
activated these units will be.
As you can see, the strongest activity (greatest dis-
tance above g i ) is produced when there is a clear separa-
tion between the most active units and those that are less
active. This is a desirable functional property for the
inhibition function, because the activation then reflects
something like the “confidence” in the active units be-
ing definitively more excited. Furthermore, those units
that are just above the g i will have weak or no activa-
tion values, even if they are at a rank position above the