Modeling with Neural Networks: Principles and Model Design Methodology - Neural Networks: Methodology and Applications - page 94

that assumption does not lead to a satisfactory model, then one must resort to
a model that is nonlinear with respect to its variables, such as a polynomial,
a neural network, etc.
Whatever the assumption on the mathematical form of the model, the
problem of modeling, in the present context, is, by essence, the problem of
estimating the parameters of the most satisfactory model, given the available
data. How one can decide whether a model is satisfactory or not, is a major
methodological problem that is considered in the present chapter.

2.3.2 Introduction to the Design Methodology

In all the following, a model whose vector of variables is x and whose vector
of parameters is w will be denoted by g ( x , w ). If there exists a parameter
vector w p such that the model is identical to the regression function g ( x ,w p ) ≡

E Y ( x ), then the family of functions g ( x, w ) contains the regression function,
and the model g ( x, w ) is said to be “true”. If such is not the case, then a
model will be sought, that is as close as possible to the regression function
E Y ( x ). Training is the algorithmic procedure whereby the parameters of such
a model are sought, for a given family of functions (for instance, for the family
of neural networks with three inputs and two hidden neurons).
Therefore, the design of a nonlinear black-box model requires the achieve-
ment of several tasks, including

• variable selection, i.e., the selection of the components of vector x in g ( x ,

w ); that task is carried out in two steps:
1. the reduction of the dimension of the input vector;
2. the selection of relevant variables, i.e., of the variables whose influ-
ence on the quantity to be modeled is larger than the influence of the
disturbances;

• the estimation of the parameters w of the model g ( x , w ), i.e. the training

of a model; this is also carried out in two steps:
1. the choice of a family of functions within which the model is sought
(for instance, the family of neural networks with three hidden neurons,
the family of polynomials of degree 4, etc.);
2. the training of one or several models within the chosen family;

• the selection of the best model and the estimation of its performances;

if that best model is not satisfactory, another family of model is chosen
(for instance, the number of hidden neurons is increased or decreased, the
degree of the polynomial is increased or decreased, etc.), and the process
is iterated to the second step of the previous task.

The latter step makes machine learning modeling different from conventional
statistical modeling: in statistical modeling, the “best” model is the model
whose parameters are estimated with the best accuracy. In machine learning,
the best model is the model that generalizes best, the exact values of the
parameters being of little or no interest.