Information Technology Reference
In-Depth Information
ically analyzable, but less powerful framework. It is
important to appreciate that although summary math-
ematical analyses can be useful (and are very strongly
emphasized by the neural network community), they are
by no means a good predictor of what actually works in
practice, and vice versa — many algorithms that have
no corresponding concise mathematical analysis work
quitewell,andmanythatdohavesuchananalysisdo
not work well at all. Because we have found that CPCA
+ kWTA does indeed work very well across a wide
range of cognitively relevant tasks, the need for addi-
tional mathematical confirmation of this fact remains
somewhat diminished (though we are by no means say-
ing that such an analysis, if possible, would not be ex-
tremely welcome).
In the next section, we discuss a number of impor-
tant points of contact with more easily analyzed frame-
works that further our basis for understanding the essen-
tial principles behind CPCA + kWTA. Finally, we will
see in the next chapter that error-driven learning does
admit to a more complete overall analysis, so that if we
use CPCA + kWTA in the context of this form of learn-
ing, we can have a reasonable mathematical assurance
that something useful will result.
Question 4.8 (a)
Does this tenfold increase in learn-
ing rate have any noticeable effect on the network, as
measured by the unique pattern statistics and the weight
patterns shown in the grid log?
(b)
Explain why this
might be the case, comparing these results to the effects
of learning rate that you observed in question 4.1.
This exercise should give you a feel for the dynam-
ics that underly self-organizing learning, and also for
the importance of contrast enhancement for the CPCA
algorithm to be effective. More generally, you should
now appreciate the extent to which various parameters
can provide appropriate (or not) a priori biases on the
learning process, and the benefit (or harm) that this can
produce.
To stop now, quit by selecting
Object/Quit
in the
PDP++Root
window.
,
!
4.8.2
Summary and Discussion
This exploration demonstrates that the combination of
CPCA Hebbian learning and the kWTA competitive ac-
tivation function does produce a useful model of the
correlational structure of a simple input domain. We
will see several other similarly successful demonstra-
tions in later chapters, with environments that are much
more complex and based directly on real-world data.
Despite these successes, it is important to note that we
have not provided an overarching mathematical analy-
sis that proves that CPCA + kWTA will do something
sensible.
Unfortunately, such an analysis remains essentially
impossible as long as anything resembling the kWTA
activation function is involved, because any analytical
treatment would quickly come up against the intractable
combinatorial explosion caused by the complex interde-
pendencies among hidden units imposed by this func-
tion. For example, we saw in the previous exploration
that these interdependencies (e.g., a balance of com-
petition and cooperation) were essential for successful
learning, so we are not inclined to abandon the general
kWTA approach.
Thus, in this particular instance, we have opted for re-
lying on a conceptual-level understanding of the learn-
ing algorithm instead of resorting to a more mathemat-
4.9
Other Approaches to Model Learning
CPCA Hebbian learning used together with the compet-
itive kWTA activation function is but one of many pos-
sible instantiations of the general objectives of model
learning. Its advantages are its relative simplicity and
consistency with the network properties developed in
chapter 3, the biological plausibility of Hebbian learn-
ing, and, perhaps most importantly, its ability to develop
sparse distributed representations (the virtues of which
were evident in the preceding exploration). In this sec-
tion, we will summarize a number of other important
approaches to model learning. This should provide a
broader perspective on the field, and a better apprecia-
tion for the relative properties of CPCA + kWTA and
where they came from.
We begin with model learning algorithms that are
very similar to CPCA + kWTA, and discuss some al-
ternative ways of viewing the objectives of this type of
learning.
Then, we cover some other approaches that
Search WWH ::
Custom Search