Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 79

hit Update in this window if bias.wt was already se-
lected, to update the display).

Do Object/Close to remove the cluster plot win-
dows (cluster plots are the only windows that you should
close in this way, because they are temporary).
The next step is to run the case where there are multi-
ple instances of each digit (the NOISY_DIGITS case).

, !

Question 3.2 Explain how these bias weights con-
tribute to producing the originally observed hidden unit
activities.

Set env_type to NOISY_DIGITS , Apply and then
do View , EVENTS to see the different noisy versions of
the digit images. Press Run to present these to the net-
work.
You should see that the appropriate hidden unit is ac-
tive for each version of the digits (although small lev-
els of activity in other units should also be observed in
some cases).

, !

Cluster Plots

Next we will produce some cluster plots like those
shown previously.

Run the network with the biases on again. Hit the
Cluster button in the xform_ctrl control panel, and
select CLUSTER_DIGITS .
You should get a window containing the cluster plot
for the similarity relationships among the digit images.

, !

Now do CLUSTER_HIDDEN .
You should see the same plot as shown in
figure 3.8b. You can compare this with the
CLUSTER_NOISY_DIGITS cluster plot of the input
digit images (same as figure 3.8a). This clearly shows
that the network has collapsed across distinctions be-
tween different noisy versions of the same digits, while
emphasizing the distinctions between different digit cat-
egories.

For comparison, click on View in the control panel
and select EVENTS ).
Compare the amount of overlap between activated
pixels in the digit images with the cluster plot results.

Iconify the events window when done.
Next let's look at the similarity relationships among
the hidden unit representations of the digits.

Selectivity and Leak

Do Cluster again, selecting CLUSTER_HIDDEN
this time.
You should get a cluster plot that looks much like
that shown in figure 3.8b, except there is just one label
for each digit category. This shows that the network
has transformed the complex patterns of input similarity
into equally distinct hidden representations of the digit
categories.
Note that we have binarized the hidden unit activa-
tion values (i.e., changed values greater than .5 to 1,
and those less than .5 to 0) for the purposes of cluster-
ing. Otherwise, small differences in the activation val-
ues of the units would distract from the main structure
of the cluster plot (which is actually the same regardless
of whether the activations are binarized or not). For the
present purposes, we are more interested in whether a
detector has fired or not, and not in the specific activa-
tion value of that detector, though in general the graded
values can play a useful role in representations, as we
will discuss later.

In the detector exploration from the previous chapter
(section 2.6.3), we saw that manipulating the amount
of leak conductance altered the selectivity of the unit's
response. By lowering the leak, the unit responded in
a graded fashion to the similarity of the different digit
images to the detector's weight pattern. Let's see what
kinds of effects this parameter has on the behavior of
the present network. The control panel shows that the
leak conductance ( g_bar_l ) for the hidden units has
been set to a value of 6.

Reduce the ( g_bar_l ) for the hidden units from 6
to 5 and Run (still using NOISY_DIGITS ).

, !

Question 3.3 (a) What happens generally to the hid-
den activations with this reduction in leak value? (b)
How does this affect the cluster plot of hidden unit ac-
tivities? (c) How about for a g_bar_l of 4? (d) If
the goal of this network was to have the same hidden
representation for each version of the same digit, and
different representations for different digits, how does