Modeling with Neural Networks: Principles and Model Design Methodology - Neural Networks: Methodology and Applications - page 106

to their parameters, such as neural networks, will be discussed. Finally, regu-
larization techniques, which aim at avoiding overfitting when training with a
small number of examples, will be discussed.

2.5.1 Training Models that are Linear with Respect to Their
Parameters: The Least Squares Method for Linear
Regression

We assume that the measurements of the quantity to be modeled can be
viewed as realizations of a random variable Y p that is an a ne function of
variables which have been selected in an earlier step: Y p = ζ T w p + B ,where
ζ is the vector of the variables of the model, of known dimension q, where w p
is the vector (non random but unknown) of the parameters of the model, and
where B is a random vector whose expectation value is zero. Therefore, the
regression function is linear with respect to the variables of the model
E ( Y p )= ζ T w p .
We want to design a model g ( ζ , w )= ζ T w , given a set of N measurements of
the quantity of interest y p ,k =1to N

that are a set of realizations of the
random variable Y p , and given a set of corresponding measurements of the
inputs

{ }

ζ k ,k =1to N

{ } .

2.5.1.1 Nonadaptive (Batch) Training of Models that are Linear
with Respect to Their Parameters

Because there is a wealth of textbooks on the subject (see for instance [Seber
1977; Antoniadis et al. 1992; Draper et al. 1998], no proof will be given in the
present section.

Minimizing the Least Squares Cost Function. The Normal Equations

The minimum of the following cost function is sought

J ( w )= N

y P −

g ( ζ k , w ) 2 ,

k =1

with g ( ζ , w )= ζ T w . In such a model, the number of parameters q is equal to
the number of inputs n .
The matrix of observations is the matrix Ξ whose column i ( i =1to q )
is the vector ξ i whose components are the N measurements of the i th input:
therefore, it has N rows and q columns,

⎡
⎣

⎤
⎦

⎡
⎣

⎤
⎦

ζ 1 ... ζ 1 ( ζ 1 ) T

...
...
...
( ζ N ) T

n
... ... ...
... ... ...
... ... ...
ζ 1 ... ζ n

= ( ζ 1 ) ... ( ζ n ) ,

Ξ = =