Supervised Learning - Hierarchical Neural Networks for Image Interpretation - page 115

6. Supervised Learning
In the last chapter, supervised learning has already been used to classify the outputs
of a Neural Abstraction Pyramid that was trained with unsupervised learning. In
this chapter, it is discussed how supervised learning techniques can be applied in
the Neural Abstraction Pyramid itself.
After an introduction, supervised learning in feed-forward neural networks is
covered. Attention is paid to the issues of weight sharing and the handling of net-
work borders, which are relevant for the Neural Abstraction Pyramid architecture.
Section 6.3 discusses supervised learning for recurrent networks. The difficulty of
gradient computation in recurrent networks makes it necessary to employ algorithms
that use only the sign of the gradient to update the weights.
6.1 Introduction
Supervised learning is more precisely defined than unsupervised learning. Given
a training set of input/output vector pairs ( x i , y i ) , the goal of supervised learning
is to produce for unseen inputs x j (generated from the same distribution) outputs
o j which are as close as possible to the desired outputs y j . It does not suffice to
memorize the training set since generalization to new examples is desired.
Two supervised learning problems can be distinguished:
• Classification: Here, the output vector y represents the class of an object to rec-
ognize. The examples are assigned to a discrete number of classes. If the classifi-
cation system is also able to produce a classification confidence, this quantity can
be used to reject ambiguous examples.
• Function approximation: This problem is also known as regression. Here, the
input/output examples are samples from a function y = f ( x ) . The output of the
learning machine is continuous.
Classification can be viewed as a special case of function approximation. For
example, an approximation to the characteristic function of a set can be used to
classify whether or not examples belong to the class described by the set.
6.1.1 Nearest Neighbor Classifier
One particularly simple supervised classifier is the nearest-neighbor (NN) classi-
fier [46]. It assigns an input vector x i to the class y k of the training vector x k that is