Modeling with Neural Networks: Principles and Model Design Methodology - Neural Networks: Methodology and Applications - page 156

Fig. 2.35. Modeling error for a process with output noise, after training according
to the output noise assumption
Output Noise Assumption
We first consider the (correct) assumption that noise can be modeled as out-
put noise. Therefore, the ideal predictor is recurrent. Figure 2.35 shows the
modeling error after training a recurrent neural network with 5 hidden neu-
rons. The modeling error is white noise with amplitude 0.5: by making the
right assumption and using the appropriate structure and training (recurrent
neural network and semidirected training), the modeling error is equal to the
noise, which is the best achievable result.
State Noise Assumption
Now, we consider the (wrong) assumption that the noise is state noise. Accord-
ing to that assumption, a feedforward neural network with 5 hidden neurons is
trained. Figure 2.36 shows the resulting modeling error: its amplitude is larger
than 0.5. As expected, the result is not as satisfactory as the result obtained
with the output noise assumption, since we made the wrong assumption. It
should be clearly understood that this is not a “technical” problem (too few
or too many hidden neurons, ine cient optimization algorithm, inappropriate
training set, etc.), but a basic problem: even with the best training algorithm,
Fig. 2.36. Modeling error for a process with output noise, after training according
to the state noise assumption