Modeling with Neural Networks: Principles and Model Design Methodology - Neural Networks: Methodology and Applications - page 134

2.6.1.2 Computation of the Jacobian Matrix

In the section devoted to the training of a model that is not linear with respect
to its parameters, it was shown that the gradient of the cost function can easily
be computed by backpropagation,
∂J
∂w i

= ∂ ( y p − =

g ( x , w )) 2
∂w i

g ( x , w )) ∂g ( x , w )
∂w i

− 2( y p − .

g ( x , w ) is equal to 1/2, then the gradient of the
cost function is equal to the gradient of the output. Thus, the Jacobian matrix
can easily be computed by backpropagating a modeling error equal to 1/2. The
extra computation time incurred by the computation of the Jacobian matrix
is marginal, since it is performed once per training, whereas backpropagation
is performed at each training epoch.

If the modeling error y p −

2.6.1.3 Computation of the Rank of the Jacobian Matrix

The rank of the matrix can be computed by a variety of methods [Press et al.
1992]. They will not be described here. In the section devoted to the effect of
withdrawing an example from the training set, we describe a technique that
is convenient in the framework of model selection.

2.6.2 A Global Approach to Model Selection: Cross-Validation
and Leave-One-Out

2.6.2.1 Introduction

As discussed in a previous section, model selection should be based on the
comparison of the generalization errors of the candidate models, but the gener-
alization error, just as the regression function, cannot be computed: therefore,
it must be estimated.
The most natural idea consists in performing model selection on the basis
of the mean square error on the training set (TMSE),

N T

1
N T

E T = ( r k ) 2 ,

k =1

where r k g ( x k , w ), and where

the summation is performed over all N T examples of the training set. That is a
bad idea: as discussed previously, the modeling error on the training set can be
made as small as desired by just adding hidden neurons, which is detrimental
to generalization. Thus, the value of E T

is the modeling error on example k : r k = y p −

is not a suitable selection criterion.