Modeling with Neural Networks: Principles and Model Design Methodology - Neural Networks: Methodology and Applications - page 170

g ( k )= ϕ NNN ( g ( k − 1) ,g ( k − 2) ,u ( k − 1) , w ) ,

where w is the vector of parameters, of dimension 19.
Its TMSE is 0.092 and the mean square error on the test set is 0.15. For
each structure, 50 trainings were performed with different parameter initial-
izations. Additional hidden neurons generate overfitting, and a higher order
does not improve the performance. The parameters are estimated with a semi-
directed algorithm using the Levenberg-Marquardt optimization algorithm.

2.7.4.2 State-Space Model

In view of the result obtained with an input-output model, models of order 2
seem satisfactory. Two possibilities arise,

• model with two state variables (not measured, in the present application),

• model in which one of the state variables is the output (hence that state

variable is measured).

As in the previous case, models trained under the state noise assumption give
poor results when operated as simulators.
Table 2.3 shows the best results obtained on the test set after semidi-
rected training, for a network with three hidden neurons, optimized by the
Levenberg-Marquardt algorithm.

Table 2.3. Results obtained after semidirected training of a network with three
hidden neurons, with the Levenberg-Marquadt optimization algorithm

Mean square error Mean square error

on the training set on the test set

Network with no measured state
variable 0.091 0.18

Network whose output is one of
the state variables 0.071 0.12

Therefore, the best model is a network whose output is one of the state
variables. Its equation is

x 1 ( k )= ϕ 1

( x 1 ( k − 1) ,x 2 ( k − 1) ,u ( k − 1))

NN

x 2 ( k )= ϕ 2

( x 1 ( k − 1) ,x 2 ( k − 1) ,u ( k − 1))

NN

y ( k )= x 2 ( k ) .

The network has 26 parameters, but is has better performances, on the
test set, than an input-output network with 19 parameters. That is an exper-
imental illustration of the parsimony of state-space models, which allow the
use of a larger number of parameters without overfitting.