Hebbian Model Learning - Computational Explorations in Cognitive Neuroscience - page 130

Now we want to compare the conditionalizing aspect
of CPCA with the unconditional PCA algorithm. Let's
assume that each event ( Right and Left ) has an equal
probability of appearing in the environment.

Set Evt Label in the environment to

FreqEvent::freq ,

and set p_right to .7, (and

, !

Apply )

— you should see the frequencies of

for the right lines and :3=3(= :1) for
the left ones. Then, Run .
You should see that the right lines all have weights of
around .2333 and the left lines have weights around .1.
Although this is the correct result for representing the
conditional probabilities, this result illustrates a couple
of problems with the CPCA learning algorithm. First,
when units represent categories of features instead of
single instances, the weights end up being diluted be-
cause the receiving unit is active for several different
input patterns, so the conditional probabilities for each
individual pattern can be relatively small. Second, this
dilution can be compounded by a receiving unit that has
somewhat less than perfect selectivity for one category
of features (right) over others (left), resulting in rela-
tively small differences in weight magnitude (e.g., .233
for right versus .1 for left). This is a real problem be-
cause units are generally not very selective during the
crucial early phases of learning for reasons that will be-
come clear later.
Thus, in some sense, the CPCA algorithm is too faith-
ful to the actual conditional probabilities, and does not
do enough to emphasize the selectivity of the receiv-
ing unit. Also, these small overall weight values reduce
the dynamic range of the weights, and end up being in-
consistent with the weight values produced by the task
learning algorithm described in chapter 5. The next sec-
tion shows how we can deal with these limitations of the
basic CPCA rule. After that, we will revisit this simula-
tion.

Set p_right to .5, Apply , and Run .
This will simulate the effects of a standard (uncon-
ditional) form of PCA, where the receiving unit is ef-
fectively always on for the entire environment (unlike
CPCA which can have the receiving unit active when
only one of the lines is present in the environment).

, !

Question 4.2 (a) What result does p_right =.5 lead
to for the weights? (b) Does this weight pattern sug-
gest the existence of two separate diagonal line features
existing in the environment? Explain your answer. (c)
How does this compare with the “blob” solution for the
natural scene images as discussed above and shown in
figure 4.8?

Question 4.3 (a) How would you set p_right to sim-
ulate the hidden unit controlled in such a way as to
come on only when there is a right-leaning diagonal
line in the input, and never for the left one? (b) What
result does this lead to for the weights? (c) Explain
why this result might be more informative than the case
explored in the previous question. (d) How would you
extend the architecture and training of the network to
represent this environment of two diagonal lines in a
fully satisfactory way? Explain your answer.

The simple environment we have been using so far
is not very realistic, because it assumes a one-to-one
mapping between input patterns and the categories of
features that we would typically want to represent.

Go to the PDP++Root window. To continue on to
the next simulation, close this project first by selecting
.projects/Remove/Project_0 . Or, if you wish to
stop now, quit by selecting Object/Quit .

Switch the env_type from ONE_LINE to the
THREE_LINES environment in the control panel (and
Apply ). Then, do View , EVENTS to see this environ-
ment.
Notice that there are now three different versions of
both the left and right diagonal lines, with “upper” and
“lower” diagonals in addition to the original two “cen-
ter” lines. In this environment, p_right is spread
among all three types of right lines, which are conceived
of as mere subtypes of the more general category of
right lines (and likewise for 1-p_right and the left
lines).

, !

, !

4.7 Renormalization and Contrast Enhancement

The CPCA algorithm results in normalized (0-1)
weight values, which, as we saw in the previous sec-
tion, tend to not have much dynamic range or selectiv-
ity , limiting the effectiveness of this form of the learning
algorithm. These problems can be remedied by intro-