Neural Networks: An Overview - Neural Networks: Methodology and Applications - page 13

since the training of the network should “discover” the similarities between
elements of the database, and translate them into vicinities in the new data
representation or “map.” The most popular feedforward neural networks with
unsupervised training are the “self-organizing maps” or “Kohonen maps”.
Chapter 7 is devoted to self-organizing maps and their applications.
1.1.3 The Fundamental Property of Neural Networks with
Supervised Training: Parsimonious Approximation
1.1.3.1 Nonlinear in Their Parameters, Neural Networks Are
Universal Approximators
Property. Any bounded, su ciently regular function can be approximated
uniformly with arbitrary accuracy in a finite region of variable space, by a
neural network with a single layer of hidden neurons having the same activa-
tion function, and a linear output neuron [Hornik 1989, 1990, 1991].
That property is just a proof of existence and does not provide any method
for finding the number of neurons or the values of the parameters; furthermore,
it is not specific to neural networks. The following property is indeed specific
to neural networks, and it provides a rationale for the applications of neural
networks.
1.1.3.2 Some Neural Networks Are Parsimonious
In order to implement real applications, the number of functions that are
required to perform an approximation is an important criterion when a choice
must be made between different models. It will be shown in the next section
that the model designer ought always to choose the model with the smallest
number of parameters, i.e., the most parsimonious model.
Fundamental Property
It can be shown [Barron 1993] that, if the model is nonlinear with respect to
its parameters, it is more parsimonious than if the model is linear with respect
to its parameters.
More specifically, it can be shown that the number of parameters neces-
sary to perform an approximation with a given accuracy varies exponentially
with the number of variables for models that are linear with respect to their
parameters, whereas it increases linearly with the number of variables if the
model is not linear with respect to its parameters.
Therefore, that property is valuable for models that have a “large” number
of inputs: for a process with one or two variables only, all nonlinear models are
roughly equivalent from the viewpoint of parsimony: a model that is nonlinear
with respect to its parameters is equivalent, in that respect, to a model that
is linear with respect to its parameters.