Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 100

a) b) c)

g i

g i

g i

g Θ g Θ g Θ

i i i

k k+1 k k+1 k k+1

0 n 0 n 0 n

Figure 3.24: Possible distributions of level of excitation across units in a layer, plotted on the Y axis by gi

(which is monotonically
related to excitatory net input g e ), and rank order index on the X axis. The basic kWTA function places the layer-wide inhibitory
current value g i between the k and k+1th most active units, as shown by the dotted lines. The shape of the distribution significantly
affects the extent to which the most highly activated units can rise above the threshold, as is highlighted by the brackets on the Y
axis, which indicate the distance between the most active unit and the inhibitory threshold. a) Shows a standard kind of distribution,
where the most active units are reasonably above the inhibition. b) Has many strongly activated units below the threshold, resulting
in a small excitatory-inhibitory differential for the most activated units. c) Has few strongly active units, resulting in a very large
differential for the most activated units.

threshold, depending on the strength of the leak cur-
rent and/or their bias weights. Thus, there is some room
for flexibility in the kWTA constraint.
The second version of the kWTA function, the
average-based kWTA function ( KWTA_AVG in the simu-
lator), provides still greater flexibility regarding the pre-
cise activity level in the layer, while still providing a
relatively firm upper limit. The tradeoff in lack of pre-
cision about the exact activity level is often worth the
advantages to the network in having a bit more flexibil-
ity in its representations. In this version, the layer-wide
inhibitory conductance g i is placed in between the aver-
age gi

and for the remaining n ￿ k units:

(3.5)

n ￿ k

Then we can just plug these averages into the same ba-
sic formula as used before for placing the inhibitory
conductance somewhere in between these two values:

(3.6)

n ￿ k n ￿ k

The q value in this case is a much more important
variable than in the basic kWTA function, because the
two averaged inhibitory conductances typically have a
greater spread and the resulting g i value is not specifi-
cally guaranteed to be between the k and k +1 th units.
Thus, it is typically important to adjust this parameter
based on the overall activity level of a layer ( ￿ ), with
a value of .5 being appropriate for activity levels of
around 25 percent, and higher values for lower (sparser)
levels of activity (e.g., .6 for 15%).
Depending on the distribution of excitation (and
therefore gi ) over the layer, there can be more or less
than k units active under the average-based kWTA func-
tion (figure 3.25). For example, when there are some
strongly active units just below the k rank order level,
they will tend to be above threshold for the computed

val-
ues for the remaining n-k units. This averaging makes
the level of inhibition a function of the distribution of
excitation across the entire layer, instead of a function
of only two units ( k and k +1 ) as was the case with
the basic WTA function (equation 3.3). We will see in a
moment that this imparts a greater degree of flexibility
in the overall activity level.
The expression for the average of gi

values for the top k units and the average gi

for the top k

units is:

(3.4)