Hebbian Model Learning - Computational Explorations in Cognitive Neuroscience - page 139

Parameter Manipulations prevent units from developing less selective representa-

tions of multiple lines. This is why we have a default
value of .5 for this parameter.

Now, let's explore the effects of some of the param-
eters in the control panel. First, let's manipulate the
wt_gain parameter, which should affect the contrast
(and therefore selectivity) of the unit's weights.

Switch to using a savg_cor value of 1, and then
Batch run the network.
You should observe results very similar to those when
you decreased wt_off — both of these manipulations
reduce the level of correlation that is necessary to pro-
duce strong weights.

, !

Set wt_gain to 1 instead of 6, Apply , and Batch
run the network.

, !

Question 4.6 (a) What statistics for the number of
uniquely represented lines did you obtain? (b) In
what ways were the final weight patterns shown in the
weight grid log different from the default case? (c) Ex-
plain how these two findings of hidden unit activity and
weight patterns are related, making specific reference
to the role of selectivity in self-organizing learning.

Set savg_cor back to .5, and then set wt_mean to

.5.
This sets the initial random weight values to have a
mean value of .5 instead of .25.

, !

Batch run the network, and pay particular attention
to the weights.
You should see that this ended up eliminating the
loser units, so that every unit now codes for a line. This
result illustrates one of the interesting details about self-
organizing learning. In general, the CPCA rule causes
weights to increase for those input units that are active,
and decrease for those that are not. However, this qual-
itative pattern is modulated by the soft weight bound-
ing property discussed earlier — larger weights increase
less rapidly and decrease more rapidly, and vice versa
forsmallerweights.
When we start off with larger weight values, the
amount of weight decrease will be large relative to the
amount of increase. Thus, hidden units that were active
for a given pattern will subsequently receive less net in-
put for a similar but not identical pattern (i.e., a pattern
containing 1 of the 2 lines in common with the previ-
ous pattern), because the weights will have decreased
substantially to those units that were off in the original
pattern but on in the subsequent one. This decreased
probability of reactivation means that other, previously
inactive units will be more likely to be activated, with
the result that all of the units end up participating. This
can sometimes be a useful effect if the network is not
drawing in sufficient numbers of units, and just a few
units are “hogging” all of the input patterns.
Finally, let's manipulate the learning rate parameter
lrate .

, !

Set wt_gain back to 6, change wt_off from 1.25
to 1, Apply , and run a Batch . To make the effects of
this parameter more dramatic, lower wt_off to .75 and
Batch again.

, !

Question 4.7 (a) What statistics did you obtain for
these two cases (1 and .75)? (b) Was there a noticeable
change in the weight patterns compared to the default
case? (Hint: Consider what the unique pattern statistic
is looking for.) (c) Explain these results in terms of the
effects of wt_off as adjusting the threshold for where
correlations are enhanced or decreased as a function
of the wt_gain contrast enhancement mechanism. (d)
Again, explain why this is important for self-organizing
learning.

Set wt_off back to 1.25 (or hit Defaults ).
Now, let's consider the savg_cor parameter, which
controls the amount of renormalization of the weight
values based on the expected activity level in the send-
ing layer. A value of 1 in this parameter will make the
weights increase more rapidly, as they are driven to a
larger maximum value (equation 4.18). A value of 0
will result in smaller weight increases. As described
before, smaller values of savg_cor are appropriate
when we want the units to have more selective rep-
resentations, while larger values are more appropriate
for more general or categorical representations. Thus,
using a smaller value of this parameter should help to

, !

First, set wt_mean back to .25, and then set lrate
to .1 instead of .01 and do a Batch run.

, !