The Mountain Car Problem - Design of Experiments for Reinforcement Learning - page 100

Table 5.5 Parameter settings for the experimental design and kriging metamodeling for the
mountain car problem.
Description Value

Number of variables ( d ) 5

Number of design points 300 (60 d )

Replications per design point 5

Minimum number of convergent replications 3

Kriging model form Linear (universal kriging)

Covariance kernel Matérn 5/2

Covariance parameter range [0, 2]

each learning attempt. Parameters and settings for the kriging experimental design
and model are provided in Table 5.5 . For each of the convergent subregions, we use
an experimental design consisting of 300 design points, with 5 replications at each
design point, created using Latin Hypercube sampling (LHS) from the subregion
bounds corresponding to those listed in Table 5.4 .
Table 5.6 shows summary statistics and kriging model metrics for each of the
experiments for the convergent subregions, and model parameters for each of the
kriging models are provided in the Appendix in Table C.2. In general, the ranges of
the responses of each of the subregions are similar, with the exception of subregion
11, which seems to generally have larger performance values. Based on the R 2 and
Q 2 , the metamodels for subregions 6 and 11 seem to fit the response surface well,
and these metamodels are also created from the largest number of points. The other
metamodels for subregions 13, 25, and 32 may need more convergent design points
to improve the model fit. As the sequential CART procedure is a random algorithm
because of the experimental designs (which affects the resulting CART models), the
subregions defined as convergent may be not be as convergent as thought, especially
with a small number of design points, and this is shown by the low convergence
proportions. This could be due to experimental design sampling such that, for any
iteration of the sequential CART algorithm, the number of design points that fall
within a convergent subregion is relatively small compared to the total number used
at that iteration.
The sensitivity analysis (Fig. 5.6 ) shows that there is no consistency for the pa-
rameter sensitivities across the five convergent subregions (see Sect. 4.2.6 for a
description of the sensitivity indices). We also see that some subregions have a sin-
gle and very dominant parameter (subregions 6 and 11), whereas other subregions
have multiple parameters that have a great effect on the response (subregions 13,
25, and 32). These results suggest that the shape of the response surface in different
regions of the parameter space is quite different, and what might be an important pa-
rameter in one subregion may not be important in another. We also see that the three
sensitivity indices that we use are generally consistent, especially when parameters
are ranked based on their sensitivities.