Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 101

a) b) c)

g < k g < k

i i

g < n−k g < k

g i

i i

g i

g < n−k

i

g i

g < n−k

Actual
activity

Actual
activity

i

g i g i g i

k k+1 k k+1 k k+1

0 n 0 n 0 n

Figure 3.25: Plots of the average-based kWTA inhibitory function for the same distributions of excitatory input as the previous
figure. The dashed lines represent the average gi values for the top k units ( hgi i k ) and the remaining n ￿ k values ( hgi i n ￿ k ).

Although case a) has the inhibition in a similar place as the basic kWTA function (between k and k +1 ), b) results in a lower
level of inhibition than basic kWTA with more than k units becoming active (shown by the actual activity arrow), and c) results in
a higher level of inhibition than basic kWTA with fewer than k units becoming active (again shown by actual activity arrow).

inhibition, resulting in more than k total units active
(figure 3.25b). In contrast, when the lower portion of
the top k units are not strongly excited, they will tend to
be below threshold with the computed g i value, result-
ing in fewer than k units active.
Thus, when the learning algorithm is shaping the rep-
resentations in the network, it has a little bit more flex-
ibility in choosing appropriate overall levels of activa-
tion for different cases. The downside of this flexibil-
ity is that activation levels are less tightly controlled,
which can be a problem, especially in layers having
very sparse activity levels. Thus, we typically use the
average-based kWTA function for layers with 15 or
more percent activity, and the basic kWTA function for
sparser layers (especially localist layers having only one
active unit).
Finally, we know from our previous explorations that
the inhibitory-interneuron based inhibition that more
closely reflects the kind of inhibition actually present in
the cortex is a complex function of a number of param-
eters. The cortex has likely evolved appropriate param-
eters for achieving a reasonable balance between flexi-
bility and firmness in activation control. Therefore, we
are not too concerned with the extra degree of freedom
afforded to the modeler in selecting an appropriate ver-
sion of the kWTA function. Ideally, as computational
resources get more powerful, it will become feasible to
use more biologically plausible forms of inhibition.

3.5.4 Exploration of kWTA Inhibition

Now, let's explore the kWTA inhibition functions and
compare their behavior with that of the previous net-
works.

Open the project inhib.proj.gz in chapter_3
to begin. (Or, if you already have this project open from
the previous exercise, reset the parameters to their de-
fault values using the Defaults button.) Make sure the
graph log is open ( View , GRAPH_LOG ). Set network to
BIDIR_EXCITE to select the bidirectionally connected
network, and Run .
This should reproduce the standard activation graph
for the case with actual inhibitory neurons.

, !

Now, set the inhib_type to KWTA_INHIB instead
of the default UNIT_INHIB to use the basic kWTA func-
tion described above. Hit Apply .
The k value of this function is set according to the
kwta_pct value in the inhib_ctrl control panel
(this percentage is automatically translated into a corre-
sponding k value, 15 in this case, by the software).

Press Run .
Notice that roughly the same level of activity results
(the inhibitory activity is at zero, because these units are
not used for this function). Also, the activity function is
somewhat smoother, because the kWTA function effec-
tively does a perfect job of anticipating the appropriate
level of inhibition required. Further, the activation of
the hidden layer starts much earlier because we can use

, !