The Mountain Car Problem - Design of Experiments for Reinforcement Learning - page 97

ʵ

332 332

0.6 1

225 225

113 113

111 111

6 6

ʳ

332 332

0.9 1

225 225

113 113

111 111

6 6

ʻ

332 332

0 1

225 225

113 113

111 111

6 6

ʱ ratio

332 332

1 8

225 225

113 113

111 111

6 6

ʱ mag

332 332

0.0005 0.01

225 225

113 113

111 111

6 6

Fig. 5.3 Parameter ranges for the convergent subregions for the mountain car problem. Each pa-
rameter explore is shown as a group of horizontal bars . Each convergent subregion is specified by a
small number within these blocks of horizontal bars. The bounds of the initial parameter ranges are
show on the left and right sides of the figure. The horizontal bars for each parameter and subregion
indicate the size and location of the respective parameter.

design based on the respective parameter bounds. More specifically, each node in the
tree corresponds to a low region of points from the sequential CART algorithm; high
regions are not further explored and are discarded in this algorithm, and these nodes
are therefore not shown. Nodes that are lower in the tree in this figure represent smaller
parameter subregions than nodes higher in the tree. When parameter subregions (i.e.,
nodes) have a convergence proportion greater than ʸ ˆ = 90 %, the node is considered

to have converged and further experimentation of that subregion is terminated. All
nodes in the experimental tree are assigned a number based on when they were
explored in the large experimental scheme, such that lower numbered nodes were
explored earlier and larger numbered nodes are explored later in the experiment.
We further screened the convergent domains to have them contain at least 20
design points and have a dimensionality ratio of at most 20, and this resulted in
five convergent subregions, shown as blue squares in Fig. 5.2 . Figure 5.3 shows the