Modeling Methodology: Dimension Reduction and Resampling Methods - Neural Networks: Methodology and Applications - page 204

modeling, the conventional method of input dimension reduction is termed
principal component analysis (PCA): the latter consists in projections, and
is limited to linear varieties. To process nonlinear representations, we will de-
scribe an alternative method, termed curvilinear component analysis (CCA),
which may be considered as a “nonlinear” extension of PCA. It is similar to
the “Kohonen maps” (Chap. 7), but it is more flexible than them, since the
structure of the projection space is not imposed.
Resampling methods aim at performing estimations estimates when the
probability distributions of the variables to be analyzed are not known. In the
problems raised by regression, particularly regression by neural networks, they
allow estimations of the generalization error, and they lead to e cient and
robust assessments of the variability of the network with respect to the data;
that is the key element of the bias-variance dilemma (described in Chap. 2),
which arises in the generation of any statistical model. Those advanced tech-
niques are computer-intensive, but the increased speed of computers makes
them more and more popular. A new method will be described, combining the
bootstrap and early stopping (described in the previous chapter) to automate
and monitor the training of neural networks.

3.2 Preprocessing

3.2.1 Preprocessing of Inputs

In the previous chapter, we mentioned that the values of model variables are
generally expressed in different units and have different orders of magnitude.
It is therefore necessary to pre-process those values so that they have the same
influence on the design of the model. Therefore, variables must be centered
and reduced or at least normalized. The preprocessing described in the sec-
tion “Input normalization” of Chap. 2 transforms the input components into
variables with zero average and unit standard deviation.

Standardize or Reduce

For distributions with uniform an d centered inputs, the ratio between stan-

dardization and reduction is only √ 3 for the standard deviati on . The standard

deviation of a uniform distribution over an interval l is l/ 2 √ 3 and standard-
ization over the same interval divides the variable by l/ 2.

Boolean Variables

The values 0 and 1 of Boolean variables should be transformed into 1and

+1 respectively; variables resulting from fuzzy encoding should be subject to
similar processing.
Figure 3.1 shows the effect of preprocessing. It corresponds to a shift of
the centre of gravity of the scatter diagram followed by standardization of the
dispersion of values on each axis without altering the distribution of points.

−