Language - Computational Explorations in Cognitive Neuroscience - page 345

the environmental level of analysis, while the critiques
of these models have emphasized the mechanistic level.
In this section, we explore a model that more directly
addresses the critiques by showing that certain mech-
anistic neural network principles naturally lead to the
U-shaped overregularization phenomenon. Specifically,
these mechanistic principles are the familiar ones that
we have explored throughout this text, including inter-
activity (bidirectional connectivity), inhibitory compe-
tition, and Hebbian learning.

(see box 5.2 for a summary of the standard feedforward
backpropagation algorithm). Such networks react to the
competition between the regular and irregular mappings
strictly at the level of the weight changes, and not in
the activation dynamics taking place on individual tri-
als. These weight changes are slow to accumulate and
tend to produce “blends” of two mutually exclusive al-
ternatives instead of choosing discretely between them
on a trial-by-trial basis.
Furthermore, the backpropagation learning mecha-
nism is explicitly designed to perform gradient-descent
in error, which should produce a monotonically improv-
ing, not U-shaped, learning curve for the past tense
(Hoeffner, 1997). Thus, when the slowly accumu-
lating weight-based competition is combined with the
gradient-descent nature of purely error-driven learning,
the backpropagation network's weights tend to mono-
tonically resolve the competition in favor of the explic-
itly trained irregular form. This results in a monotoni-
cally decreasing level of overregularizations, and not a
U-shaped curve.
We explore these ideas in a model that instanti-
ates the past tense mapping illustrated in figure 10.17.
We compare the results using Leabra with those of
a standard backpropagation network to test the idea
that Leabra's distinctive features (especially interactiv-
ity and inhibitory competition) are important for pro-
ducing a more realistic U-shaped curve.

A Competitive, Priming-Based Model

The central intuition behind this approach is that the
regular and irregular mappings of a given irregular form
are in a state of competition, with the explicit training
on the irregular item favoring the irregular form, and all
of the training on regular forms conspiring to support
the regular inflection. In the presence of interactive,
competitive activation dynamics, the regular and irreg-
ular mappings can compete as two mutually-exclusive
activation states on the time scale of an individual pro-
duction trial. This activation-based competition makes
the network susceptible to priming-like effects (e.g., as
explored in chapter 9), where random variations in the
prior training history can favor the production of one
form over another. For example, a string of regular pro-
ductions will tend to favor the production of an overreg-
ularization for a subsequent irregular form.
The simulations demonstrate that an interactive,
competitive network is highly sensitive to these
priming-like effects, and by extension show that this re-
sults in a high rate of past-tense overregularization in a
model of inflectional morphology. Interactivity plays an
important role by producing attractor dynamics (chap-
ter 3), where the regular and irregular mappings are two
separate attractors, and small changes in the network
weights can make relatively large changes in the attrac-
tor landscape, leading to an overregularization. These
ideas were first developed in small-scale networks such
as the weight-priming model explored in chapter 9, and
then in the large scale model that we explore here.
Interestingly, almost all previous past-tense models
have used standard feedforward backpropagation net-
works, which lack both interactivity and competition

10.5.1 Basic Properties of the Model

The model has a semantic input projecting through
a hidden layer to the phonological output layer (fig-
ure 10.19). The general structure and approach for this
model are based on the work of Hoeffner (1997, 1992),
which also provided the initial corpus of words for this
model. The network is trained to produce the appropri-
ate pronunciation of 389 different monosyllabic English
verbs (90 irregulars and 299 regulars) in the past tense
and the four other types of inflection in the English verb
inflection system, for a total of 1945 training items.
The five different English verb inflections are shown
in table 10.10. These inflections convey extra seman-
tic information about each verb. Thus, the semantic
input for the model has two main components (fig-