Language - Computational Explorations in Cognitive Neuroscience - page 339

Network Connectivity and Learning We can trace the link from this Ortho_Code “i”

unit up to the hidden layer to see how it is processed
there. We will view the weights of a hidden unit that
the 'i' unit projects strongly to.

Now, let's explore the connectivity and weights of the
trained network.

Click on r.wt and click on some units on the left
hand side of the Ortho_Code layer.
Notice that these units receive from the left-most por-
tion of the orthography input. We can use a trick of
viewing nothing in the NetView to see the “skeleton” of
the network as shown in figure 10.14, and see how these
connections align with the orthography slots.

Select PickUnit again and select HID_I .
In addition to viewing the weights into the hidden
unit, the original Ortho_Code 'i' unit, plus another
we will examine in a moment, are selected (highlighted
with a dashed white border). You can see that this hid-
den unit is strongly driven by the Ortho_Code unit
that we were just viewing, and also that it receives a
very clear pattern in the central vowel slot from the
phonological output layer. Because the weights in the
network are generally symmetric, we can often inter-
pret what a unit projects to by looking at what it receives
from. However, to be sure, let's also look at the sending
weights.

, !

, !

Click on the r.wt button using the middle mouse
button (or Shift plus the left mouse button) to unselect
it and see the skeleton, and then toggle back and forth
by reselecting r.rwt .
You should see that the left-most Ortho_Code
units are receiving from the left-most 3 letter slots,
where each letter slot is a 3 x 9 group of units. You
should also see that the other groups of units as you
progress to the right within Ortho_Code receive from
overlapping groups of 3 letter slots.

, !

Select s.wt , which will show the sending weights.
By switching back and forth between s.wt and
r.wt , you can see that they are generally, but not ex-
actly, symmetric (the Hebbian learning component is
only approximately symmetric, but the error-driven is
always symmetric).
To interpret the phonological output pattern that this
hidden unit wants to produce, we need to again look at
the vowel phonology.

Click back to viewing r.wt with the left button, and
click on Ortho_Code units all throughout the layer.
As you click on these Ortho_Code units, pay atten-
tion to the patterns of weights. You should notice that
there are often cases where the unit has strong weights
from the same input letter(s) across two or three of the
slots. We have preselected some examples of different
patterns of connectivity that we can step through now.

, !

Do View , VOWELS . Then, click on the i button and
then scroll down and middle-click (or shift-click) on the I
button.
You should see that this particular hidden unit wants
to produce either an /i/ or /I/ sound, which are the two
main pronunciations of the letter 'i', just as we would
expect from the input to this unit from the invariant 'i'
detector in the Ortho_Code layer. Further, we can
also see that this hidden unit receives from other units
in Ortho_Code that also code for the letter “i.”

Press PickUnit on the ss_ctrl control panel,
and then select OC_I1 .
This unit (the one selected toward the right side of
the Ortho_Code layer) is a good example of a single-
letter invariant unit. It clearly represents the letter “i”
in all 3 slots, thus providing a locally spatially invariant
representation of the letter “i,” much like the units in the
V2 layer of the object recognition model from chapter 8.
When the network's hidden layer learns about the pro-
nunciation consequences associated with the letter “i”
in a particular position using a representation like this,
this learning will automatically generalize to all 3 loca-
tions of the letter “i”. This invariant coding is just the
kind of thing that the PMSP hand-tuned input represen-
tations were designed to accomplish, and we can see
that this network learned them on its own.

, !

Do PickUnit again and select OC_I2 , and then
click back to viewing r.wt instead of s.wt .
Now you can see the weights of the unit in the mid-
dle of the Ortho_Code layer that is strongly inter-
connected with the hidden layer unit — this unit also
represents the letter “i” in any of 3 input locations.
You can also go back to HID_I and verify that the
other Ortho_Code units that this hidden unit receives
strongly from also have invariant “i” representations.

, !