Unsupervised Learning - Hierarchical Neural Networks for Image Interpretation - page 102

5.2 Learning a Hierarchy of Sparse Features
In order to make the Neural Abstraction Pyramid approach to image interpretation
work, a sequence of increasingly abstract models of the potential image content is
needed. In the last chapter, such models were designed manually. In Section 4.3.2,
for instance, the Gestalt approach was used to construct weight templates for the
extraction of foreground/background features, step edges, and oriented lines. The
feature cells in that network cooperated and competed to achieve binarization of
handwritten digits.
In the previous section, several unsupervised learning methods were discussed.
They produced new representations from a set of data vectors. The representations
found by the different methods have various properties that sometimes are contra-
dictory. For example, while PCA tries to preserve variance, SFA focuses on the
least variant features. All methods discussed so far are single-step transformations.
In contrast, the Neural Abstraction Pyramid represents the image content on differ-
ent levels of abstraction. Hence, a sequence of transformations is needed to extract
features which become increasingly abstract.
One way to produce such features is by repeated application of an unsupervised
learning technique. This imposes a constraint on the learning method: Its output
must be admissible as input for the next stage of learning. Hence, unsupervised
learning methods that drastically change the nature of the representation cannot be
used for this task. Another constraint is that features within a level should be treated
equally. This excludes methods which produce an ordered sequence of features.
In the following, I present an unsupervised learning method that produces rep-
resentations with the following desired properties:
- Completeness: All salient features of the input image should be represented.
- Sparseness: The value of a feature should be zero at most positions and high at
only a few positions.
- Fairness: All feature arrays of a layer should contribute approximately equally
to the representation.
The proposed method is based on Hebbian weight updates and lateral competi-
tion. It can be applied repeatedly to learn a hierarchy of sparse features.
Training starts at Layer 1 of the pyramid that analyzes small windows of Layer 0
representations. It proceeds upwards from layer to layer. Using the topmost repre-
sentation on layer ( l − 1) as input, it learns weight templates for the forward projec-
tions of feature cells that reside on layer l . Since the number of layers is logarithmic
in the image size, only a few steps are needed to train the entire hierarchy.
5.2.1 Network Architecture
Figure 5.1 illustrates the architecture of the network that is used for the unsuper-
vised learning of a hierarchy of sparse features. It is a Neural Abstraction Pyramid
(compare to Chapter 4) with six layers. Each layer l consists of 4 · 2 l excitatory
feature arrays and a feature sum array. All but the bottom layer also contain an array