Modeling Methodology: Dimension Reduction and Resampling Methods - Neural Networks: Methodology and Applications - page 225

Statistics of the Generalization Error

The difference between the training error computed on the bootstrapped base
and the testing error evaluated on the initial base is considered to be a random
variable that represents the difference between the training error and the
generalization error.
Statistics are produced for all those differences (1 per bootstrapped base)
in order to estimate the probability law of the difference between the training
and the generalization error.
We denote by B the initial database and by B b , b =1 ,...,N the set of
bootstrapped bases. Denote by ε b the training error on bootstrapped base
k, and by ε b the error of the same network computed on the initial base
B . The difference δ b

ε b between the errors may be considered as a
random variable that arises from overtraining. That difference may also be
viewed as the bias that appears when estimating the generalization error by
the training error. The expectation value δ and the variance σ 2
δ

= ε b −

of the bias

may be estimated on the set of values of δ b ,

B B

δ b −

δ 2 .

1
B

1

ε b k , δ = δ b , σ 2

δ b = ε b − =

δ

B − 1

b =1 b =1

3.6.4 The NeMo Method

The algorithm proposed above was programmed in the NeMo software. The
bootstrap is associated with early stopping for automatic monitoring of the
training of the network.

The NeMo Tool

NeMo is a tool developed by the Systems and Structure Modeling Department
of the Study Centre at Saclay using the Stuttgart neural network simulator
(SNNS) available on http://www.ra.informatik.uni-tuebingen.of/SNNS ,which
is designed to simplify neural network learning and testing tasks.
The user chooses the number of training cycles N c and the number of
bootstrapped bases B . NeMo performs B training cycles and saves the average
quadratic training and test errors for each cycle. NeMo analyses the training
and test error profiles in order to select the most appropriate value for the
number of cycles.

Modeling of Errors

The average quadratic error EQM r is calculated from the centered and re-
duced output variables (estimated and measured). Therefore, the analysis of
the error deals with the part of the variance that is not explained by the model
or coe cient of nondetermination that was described in the section on output
preprocessing.