Language - Computational Explorations in Cognitive Neuroscience - page 369

a study by Arieli, Sterkin, and Aertsen (1996), who
showed that the apparently noisy responses of neurons
to discrete inputs can be explained by taking into ac-
count the ongoing state of activation in the network.
Getting these kinds of temporal integration dynamics
right in the model may require that we use units that
have a much closer approximation to the true dynam-
ics of cortical neurons. These dynamics are also likely
to depend on more detailed parameter settings (e.g., in
regulating the balance between hysteresis and accom-
modation, as discussed in chapter 2). Thus, it may be
more practical to continue to use a specialized context
representation even when modeling temporal integra-
tion phenomena that may not actually depend on such
representations.
One important contrast between our model and the
original SG model is in the speed of learning. Our
model achieved reasonable levels of performance after
about 10,000 training sequences, and we stopped train-
ing at an apparent asymptote after 25,000 sequences. In
contrast, the original SG model required 630,000 total
training sequences. We can attribute some of this differ-
ence to the benefits of the self-organizing, model learn-
ing aspects of Leabra (Hebbian learning and inhibitory
competition) in networks having multiple hidden lay-
ers, as discussed in chapter 6. Furthermore, temporally
extended processing can be viewed as adding more in-
tervening steps in the propagation of error signals, so
these model-learning constraints are likely to be even
more important in this model.
Finally, we should note that there are a number of
limitations of this model. As we saw, its performance
is not perfect. Furthermore, the model requires external
control mechanisms, like resetting the context after each
sentence, that probably should be learned (and more re-
alistically, should be integrated together with paragraph
and higher levels of temporal integration). One avenue
for further exploration would be to incorporate some
of the more powerful frontally mediated context updat-
ing and maintenance mechanisms that are discussed in
chapters 9 and 11.
Perhaps the biggest limitation of the model is the rel-
atively simplistic form of both the semantic and syntac-
tic features. To demonstrate truly a flexible and pow-
erful form of sentence-level constraint satisfaction pro-

cessing, one would need to rule out the possibility that
the network is doing something akin to “memoriza-
tion.” Although even this relatively simple environment
has a combinatorial space of more than 8,000 surface
forms (which argues against the memorization idea),
one would ideally like to use something approach-
ing the semantic and syntactic complexity managed
by an elementary-school child. One optimistic sign
that something like this could plausibly be achieved
comes from the relatively rapid learning exhibited by
the Leabra version of this model, which allows for the
possibility that more complex environments could still
be learned in reasonable amounts of time.

10.8 Summary

Our explorations of language illustrated some important
general principles of computational cognitive neuro-
science. For example, a central theme of this chapter is
that words are represented in a distributed lexicon ,and
not stored in a single canonical lexicon as in traditional
accounts. We explored the distributed lexical compo-
nents of orthography (written word forms), phonol-
ogy (spoken word forms composed of phonemes ), and
semantics (representations of word meanings). Vari-
ous forms of dyslexia (reading impairments), including
surface , phonological ,and deep dyslexia, can be un-
derstood in terms of damage to various pathways within
an interactive distributed lexicon model.
Another theme is that neural networks can capture
both regularities and exceptions within a unitary sys-
tem, and that they can generalize based on the regu-
larities, for example in producing humanlike pronunci-
ations for nonwords . A neural network is ideally suited
for modeling the complex continuum of regularities and
exceptions in the English orthography to phonology
mapping. The tension between regulars and exceptions
can be played out over the timecourse of learning, as
in the English past-tense inflectional system . Here,
there is evidence of a U-shaped curve over develop-
ment, where irregulars are initially produced correctly,
and are then sporadically overregularized , followed by
eventual mastery. Both Hebbian learning and inhibitory
competition contribute to a reasonable account of the
U-shaped curve in our model.