Language - Computational Explorations in Cognitive Neuroscience - page 349

to capture without questionable manipulations of the
training corpus or other parameters. These other mod-
els look more like the Bp network in the figure, with
overregularization beginning quite early relative to the
level of responding. Furthermore, increasing the learn-
ing rate in Bp uniformly advances both responding and
overregularization, preserving the same basic relation-
ship.
The second critical feature is the overall level of over-
regularization and its tendency to continue on at a low,
sporadic rate over an extended period of time, which
is an important characteristic of the human data. The
Leabra network shows this characteristic, but the Bp
network exhibits a rapidly resolving overregularization
period, consistent with the purely gradient-descent na-
ture of Bp. The extended overregularizations in Leabra
can be attributed to a dynamic competition between
the regular and irregular mappings that is played out
on each settling trial, and is affected by small weight
changes that effectively prime the regular mapping
(O'Reilly & Hoeffner, in preparation). Thus, during
a protracted period of learning, the Leabra network is
dynamically balanced on the “edge” between the regu-
lar and irregular mappings, and can easily shift between
them, producing the characteristic low rate of sporadic
overregularizations.
To provide a more representative quantitative assess-
ment of these two critical features, O'Reilly and Ho-
effner (in preparation) ran 25 random networks through
the early correct period and recorded the number of
valid responses to past-tense irregulars prior to the first
and second overregularization (figure 10.21). Again,
a standard backpropagation (Bp) network was com-
pared with Leabra, with three different levels of Heb-
bian learning: none (H0), .001 (as explored above), and
.005. The results confirm that Leabra exhibits a signif-
icantly more substantial early correct period compared
to Bp, and further shows that Hebbian learning has little
effect either way.
O'Reilly and Hoeffner (in preparation) also ran 5
of each type of network for a full 300 epoch period
and counted the total number of overregularizations
produced (figure 10.22). This also confirms our pre-
vious single-network results — the interactivity and
inhibitory competition in Leabra facilitate a dynamic

Early Correct Responding

100

75

50

25

0

Bp L H0 L H001 L H005

To 1st OR
To 2nd OR

Figure 10.21: Total number of phonologically valid re-
sponses made by the network ( N =25 ) to past-tense irreg-
ulars prior to the first and second overregularization, for four
different networks: backpropagation (Bp) and Leabra (L) with
three levels of Hebbian learning, none (H0), .001, and .005.
This provides a measure of the extent of the early correct pe-
riod prior to the onset of overregularization. The interactivity
and inhibitory competition in Leabra appear to contribute sig-
nificantly relative to a generic backpropagation network (Bp),
but Hebbian learning does not seem to have a strong effect
either way.

Total Overregularizations

200

150

100

50

0

Bp L H0 L H001 L H005

Figure 10.22: Total number of overregularizations made
over training for same networks as in the previous figure
( N = 5 ). As before, there is a main effect of Leabra ver-
sus backpropagation (Bp), with Leabra producing a substan-
tially larger number of overregularizations. Too much Heb-
bian learning (.005) reduces the overregularization in Leabra,
but smaller amounts have no effect.