Information Technology Reference
In-Depth Information
Fig. 2.23.
Virtual leave-one-out scores for 500 different models
•
For models whose Jacobian matrix does not have full rank, each model is
shown as a point in a plane: the horizontal axis is the TMSE, and the verti-
cal axis is the virtual leave-one-out score (estimation of the generalization
error of the model); note that the vertical scale is logarithmic.
•
For models whose Jacobian matrix does not have full rank, the correspond-
ing points are shown below the graph, on a horizontal scale that shows the
TMSE's of those models.
Note that
•
The Jacobian matrix of the model with smallest TMSE does not have full
rank: that model must be discarded.
•
In the present example, 70% of the minima found do not have a Jacobian
matrix with full rank.
•
The estimate of the generalization error varies by several orders of mag-
nitude, which requires a logarithmic scale for
E
p
. Models with very high
virtual leave-one-out scores are very “specialized” on one or several points,
with leverages very close to 1.
Figure 2.24 shows the outputs of the model that has the smallest value
of
E
T
and of the model that has the smallest value of
E
p
(shown as a gray
circle and a gray triangle respectively on Fig. 2.23). Note that the model
with minimal
E
T
gives a prediction that is less smooth than the model with
minimal
E
p
. Therefore, the latter is more satisfactory; however, it is the most
satisfactory model among models that have four
hidden neurons
.Inorderto
finalize the selection, that model must be compared to the best models found
for different complexities
Figure 2.25 shows the virtual leave-one-out scores and the TMSE's of
the best networks, found by the above procedure, for complexities increasing
from 0 hidden neuron (linear model) to 5 hidden neurons. As additional in-
formation, the graph also displays the standard deviation of noise (which, in
general, would be unknown in a real application). As expected, the TMSE
Search WWH ::
Custom Search