Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 103

3.5.6 Other Simple Inhibition Functions

Continue to increase the hidden_k parameter, and
Run .
You should observe that this provides increasingly
distributed patterns in the hidden layer (with some vari-
ation in the total number of active units due to the ef-
fects of ties as just discussed). The advantage of con-
trolling the number of active units through the kWTA
inhibition function instead of the trial-and-error manip-
ulation of the g_bar_l parameter is that you have
precise and direct control over the outcome. Once
this parameter is set, it will also apply regardless of
changes in many network parameters that would affect
the g_bar_l parameter.
Now, let's explore the use of kWTA inhibition in the
distributed, feature-based network.

, !

Historically speaking, the predecessor to the kWTA in-
hibition function was the single winner-takes-all (WTA)
function. A simple case of WTA was used in the
competitive learning algorithm (Rumelhart & Zipser,
1986; Grossberg, 1976), in which the most excited
unit's activity is set to 1, and the rest to zero. A “softer”
version of this idea was developed by Nowlan (1990)
using some of the same kinds of Bayesian mathemat-
ics discussed in section 2.7. In this case, units were
activated to the extent that their likelihood of generat-
ing (predicting) the input pattern was larger than that of
other units. Thus the activity of each unit had the form:

(3.7)

Set network to DISTRIBUTED_NETWORK instead
of LOCALIST_NETWORK .Set hidden_k back to 1,
Apply , and press Run .
Notice how, except when there were ties, we were
able to force this distributed network to have a single
hidden unit active — this would have been very difficult
to achieve by setting the g_bar_l parameter due to the
fine tolerances involved in separating units with very
similar levels of excitation.

where l j is the likelihood measure ( P (datajh) )for
each unit, and the denominator comes from replacing
P(data) with a sum over the conditional probabilities
for all the mutually exclusive and exhaustive hypothe-
ses (as represented by the different units).
This mutual exclusivity assumption constitutes a sig-
nificant limitation of this type of model. As we saw
above, distributed representations obtain their consid-
erable power by enabling things to be represented by
multiple cooperating units. The exclusivity assumption
is inconsistent with the use of such distributed represen-
tations, relegating the single WTA models to using lo-
calist representations (even when the units have graded
“soft” activation values). Nevertheless, these simpler
models are important because they admit to a firm math-
ematical understanding at the level of the entire net-
work, which is not generally possible for the kWTA
function.
A related form of activation function to the simple
WTA function is the Kohonen network (Kohonen,
1984), which builds on ideas that were also proposed
by von der Malsburg (1973). Here, a single “winner”
is chosen as before, but now a neighborhood of units
around this winner also gets to be active, with the acti-
vation trailing off as a function of distance from the win-
ner. When used with learning, such networks exhibit
many interesting properties deriving from their built-in
tendency to treat neighboring items in similar ways. We

Do a Cluster on the CLUSTER_HIDDEN unit activ-
ities.

, !

Question 3.14 (a) How well does one feature do at
representing the similarity structure of the digits? (b)
What value of the hidden_k parameter produces a
cluster without any collapsed distinctions? (c) How
many of the hidden patterns actually have that many
units active? (d) What mechanism explains why not ev-
ery pattern had hidden_k units active? (e) Keeping
the hidden_k value you just found, what happens to
the activity levels and clustering when you reduce the
leak current to 1? 0?

Go to the PDP++Root window. To continue on to
the next simulation, close this project first by selecting
.projects/Remove/Project_0 . Or, if you wish to
stop now, quit by selecting Object/Quit .