Neural Networks Approach - Computational Intelligence in Time Series Forecasting - page 114

SSE-Vs-Epoch with
Validation data set

Early Stopping

SSE-Vs-Epoch
with training data set

0

Epoch

Figure 3.16. Early stopping of training

But still, the dilemma remains: in order to stop the training process, how do we
realize that the network has learnt all the required knowledge from the training data
and has reached its maximum generalization? Then, from learning theory we know
that after reaching the point of maximum generalization, the network - although
learning more and more from the training set - will start impairing the related test
set performance (Figure 3.16) due to its overtraining (Vapnik, 1995). To prevent
this, the method of early stopping with cross-validation has been suggested by
Prechelt (1998).
Cross-validation is a traditional statistical procedure for random partitioning of
collected data into a training set and a test set , and for further partitioning of the
training set into the estimation set and the validation set . It is obvious that, if only
a restricted data set is available, the partition of the entire set reduces the size of the
training set. This, again, makes the location of the early stopping point difficult.
For managing this problem, a predicate or a stopping criterion should be found
that can indicate when to stop the training.
Prechelt (1998), using the error function (or the objective function) E , training
error E tr (as the average error per example across the training set), and the test and
validation errors E t and E v respectively, has defined three possible stopping
criteria:

x Stop as soon as the generalization loss exceeds a threshold value İ , i.e .
when g ( ) t ! İ , where the error function g ( ) t is based on the lowest

loss loss

validation set error E and the validation error

E . opt

x Stop as soon as the quotient

g t
Pt

()
()

loss

! İ ,

tr

P tr ( t )

where is the training progress defined by