Unsupervised Learning - Hierarchical Neural Networks for Image Interpretation - page 112

Table 5.3. Learning a hierarchy of sparse features - classification of abstract features. Zero-
reject substitution rates of different classifiers that input the upper four layers of the learned
hierarchy of sparse features.

classifier TST VAL

KNN 15 3.59 3.64

FFNN 1920 − 10 1.71 1.66

FFNN 1920 − 16 − 10

1.99 1.77

FFNN 1920 − 32 − 10 1.71 1.73

FFNN 1920 − 64 − 10 1.67 1.68

FFNN 1920 − 128 − 10 1.65 1.49

tures. One can see that the performance of the neural networks is better for the more
abstract features.
In the second experiment, the top four layers of the feed-forward pyramid are
fed into a 1920 − 128 − 10 FFNN to classify the digits. After 120 epochs of online-
training with a learning rate of η = 0 . 01 , a zero-reject substitution rate of 1.65%
on the test set and a rate of 1.49% on the validation set was observed. Table 5.3
shows the results for different numbers of hidden units, as well as for a network
without hidden units and a KNN classifier. These rates compare favorably to the
results published in [21] for the same dataset. One can also reject ambiguous digits
by looking at the two best classes. The substitution rate drops to 0.55% when 2.52%
of the validation set are rejected and to 0.21% for 7.9% rejects. Figure 5.7 shows the
substitution-reject curve of this classifier compared to the structural classifier and a
time-delay neural network (TDNN) classifier [21]. Clearly, the classifier that uses
the features extracted by the Neural Abstraction Pyramid performs about as well
as the combination of the other two classifiers. The figure also shows the results
when the new classifier is combined sequentially [22] with the other two. Now the
zero-reject substitution rate drops to 1.17%. The substitution rate can be reduced to
0.30% with 3.60% and to 0.11% with 9.20% rejects. These results are the best the
author knows for this dataset.

5.5 Discussion

This chapter presented an unsupervised learning algorithm for the design of the
forward projections in the Neural Abstraction Pyramid. The algorithm was applied
to a dataset of handwritten digits to produce a sequence of increasingly abstract
digit representations. The emerging feature detectors are meaningful and can be
interpreted in terms of detected combinations of digit substructures. This leads to
a hierarchical image description that is distributed and sparse. When looking at the