Language - Computational Explorations in Cognitive Neuroscience - page 337

Phon

rate decrease over time) are all characteristic of larger
networks trained on large corpora.

10.4.2 Exploring the Model

[Note: this simulation requires a minimum of 128Mb of
RAM to run.]

Hidden

Open the project ss.proj.gz (ss stands for
spelling-to-sound) in chapter_10 to begin.
Again, this very large network, which looks just like
that shown in figure 10.14, is not constructed. We will
build and load a trained network (which took a couple
of weeks to train!).

Ortho_Code

Ortho

Do LoadNet in the overall ss_ctrl control panel.

Figure 10.14: The actual network, in skeleton view. There
are 7 slots of 3x9 (27) input units (total of 189), 5x84 or 420
units in the first hidden layer (Ortho Code), 600 units in the
second hidden layer, and 7 slots of 2x10 (20) phonology out-
put units (140 total).

, !

Reading Words

First, we will see that the network can read words that
are presented to it, using a standard list of probe words
developed by Taraban and McClelland (1987).

quency. This square root compression of the frequency
(also used in some of the PMSP models) enables the
network to train on the low-frequency words in a rea-
sonable amount of time (it takes several weeks as it is).
PMSP demonstrated qualitatively similar results with
actual and square-root compressed frequencies. Be-
cause each word could appear in any of the positions
in the input such that the entire word still fit within the
7 slots, shorter words appeared in more different posi-
tions than longer ones. However, the word itself, and
not the word-position combination, was subject to the
frequency manipulation, so shorter words did not ap-
pear more often.
The activity constraints were set to 15 percent activ-
ity in the hidden layers, and the proportion of Hebbian
learning was set to .0005 — this value was necessar-
ily small to prevent the constant pressure of the Heb-
bian component from swamping the error term and pre-
venting successful learning of the corpus. Further, as
with the object recognition model and other large net-
works, the learning rate was reduced after 300 epochs
(from .01 to .001) to prevent “thrashing” (i.e., where
subsequent weight changes cause undue interference
with previous ones). All other parameters were stan-
dard. These nonstandard parameters (15% activity in-
stead of 25%, smaller Hebbian learning, and learning

Do StepTest in the overall control panel.
The first word the network reads is “best,” presented
at the left-most edge of the input. You can read this
word from the input pattern by identifying individual
letters within each of the 4 activated 3x9 slots. Each
of these 3x9 slots corresponds to one location and con-
tains 26 units corresponding to one of the 26 letters of
the alphabet (with one extra unit). The slots begin with
the “a” unit in the lower left, “b” to the right of “a,”
and “c” to the right of “b,” then “d,”“e,” and “f” above
them, and so on. To familiarize yourself with the layout
of the input patterns, verify that “best” is in fact the pat-
tern presented. The output pattern produced, /bbbestt/
is the correct pronunciation. To verify this, we can view
the phonology patterns and then compare each phoneme
slot's output with the appropriate patterns.

, !

Press View on the ss_ctrl overall control panel,
and select CONSONANTS .
The initial 3 slots (i.e., the onset) should all have
/b/'s, which we can find by examining the correspond-
ing phonological pattern.

Click on the b button in the consonant window.
You should see that this pattern matches that in the
first 3 slots of the output. Next, let's look at the coda,
which should have an /s/ and 2 /t/s.