Information Technology Reference
In-Depth Information
in the case of BCM, the notion that everything occurs in
the environment with the same basic frequency seems
problematic.
The popular independent components analysis (ICA)
algorithm (Bell & Sejnowski, 1995) takes an approach
to model learning that is similar in many ways to BCM.
ICA was designed to do blind source separation —
separating a set of independent signals that have been
mixed together, as in a cocktail party of voices recorded
by a set of microphones. ICA requires that the number
of hidden units be equal to the number of underlying
sources (features) in the environment (as in BCM), and
it also requires that this number be equal to the number
of input units (i.e., the input-hidden weight matrix must
be square). Furthermore, ICA learns by making the hid-
den units maximally independent from each other (as
defined by mutual information), so that what a given
unit learns is highly dependent on what the other units
have learned. Under appropriate conditions, ICA can do
an excellent job of picking out independent components
in the inputs. However, like BCM, ICA suffers when its
constraints do not fit the environment.
In contrast with BCM and ICA, the CPCA + kWTA
algorithm distributes its constraints more evenly be-
tween those that apply at the individual unit level,
and those that depend on the interactions between the
units. Specifically, CPCA plus the contrast enhance-
ment bias encourages the individual unit to specialize
in what it represents, and to emphasize the strongest
correlations (principal component) while deemphasiz-
ing weaker ones. The kWTA function then interacts
with these unit-level constraints in shaping the overall
development of representations. This more balanced
distribution of constraints makes CPCA + kWTA much
less dependent on the precise numbers of hidden units,
for example.
the world, and then learn based on the difference be-
tween what was generated and what is actually being
perceived. One advantage of this learning mechanism
is that it requires the internal model to fit as precisely
as possible the actual input patterns. Thus, it should
in principle lead to fewer representations of spurious
correlations, and “hog” units may not be as much of a
problem because they will produce a worse fittothe
details of specific patterns. Further, generative mod-
els can be easily understood in terms of the Bayesian
statistical framework, because the likelihood term that
plays an important role in this framework is essentially
like a generative model in that it expresses the extent to
which the hypothesis (i.e., internal model) would have
produced the data (i.e., the actual perceptual input).
Although appealing in many respects, there are some
problems with generative models. For example, gen-
erative models require that there be a clear directional-
ity and hierarchy to processing. Thus, a given layer in
the network must be considered a internal model of the
layer below it, and as something to be modeled by the
layer above it. The processing associated with these dif-
ferent relationships is different, so the kind of interac-
tive, bidirectional constraint satisfaction processing that
we explored in chapter 3 is not really feasible in gener-
ative models (at least not current versions). Not only is
this at odds with the known biology, but we will also
see in the second part of the topic that many cognitive
phenomena depend on bidirectional constraint satisfac-
tion processing and do not easily admit to the more rigid
hierarchical structuring required by generative models.
Despite these current limitations, it may be possible that
more cognitively and biologically plausible generative
models will be developed in the future.
4.10
Summary
In this chapter we explored in detail one approach based
on Hebbian learning that achieves the model learn-
ing objective of developing an internal model of the
important structural features of the environment (i.e.,
things that are strongly correlated ). Because Hebbian
learning is also biologically plausible, it satisfies both
computational and biological constraints for a learning
mechanism useful for cognitive neuroscience modeling.
4.9.6
Generative Models
Generative models are an important class of self-
organizing learning models based on the idea of recog-
nition by synthesis (Dayan et al., 1995; Saul et al.,
1996; Carpenter & Grossberg, 1987; Ullman, 1994).
The idea is to generate some top-down image of what
you are perceiving based on your internal model of
Search WWH ::




Custom Search