Methodology - Design of Experiments for Reinforcement Learning - page 79

run. At each iteration, 20 new points (10 number of variables) were sampled within

each region of the leaves of the CART models labeled as ˉ L , again with 4 replications
at each design point, and these points were also sampled using LHS. An alternative
method for selecting subsequent design points within a subregion could be to choose
points based on their proximity to previous points in subregion, though this is likely
a more involved optimization problem.
The R package rpart (Therneau et al. 2012 ) was used for the CART modeling.
Two CART model parameters were modified from their default values. These include
the minimum number of points required for a split in the CART model and the
minimum number of points required to be in a leaf node, and these values were
set to 10 % and 5 %, respectively, of the total number of points used in the CART
model for the current subregion being explored. For all subregions, the proportion of
points labeled as ˉ L was set to 80 % of the total number of points. Subregions were
considered to be converged if their convergence proportion ʲ surpassed 0.75. These
settings, or settings that are very similar, seemed to be acceptable for the problems
of interest in this work, however they are likely not optimal for all problems. For
example, if successful design runs are very rare and the true subregion of interest is
very small, the minimum number of points in a CART leaf would likely have to be
reduced.
Figure 4.3 shows how the sequential CART procedure proceeds from the initial-
ization at iteration 0 through iteration 7 where three convergent subregions are found.
These figures show how the subregions become smaller and smaller and narrow in
the lower valued and convergent regions of the parameter space.
Table 4.3 shows statistics of the convergent subregions that were found from
the sequential CART approach. These tables show the bounding regions for each
subregion along each dimension, the number of points that fall into each region, and
some of the sizes and shapes of the subregions. Subregion 13 is more rectangular
shaped than the other subregions (based on the dimensionality ratio), but it is also
much larger than the other regions (based on the average radius and the sum of the
radii), and this can be seen in Fig. 4.4 which shows the final convergent subregions
outlined in red dotted lines.
Additionally, the parameter ranges of these convergent subregions can be repre-
sented visually and compactly as in Fig. 4.5 . This figure shows, for each dimension
x 1 and x 2 of the parameter space, the extent and location of the parameter ranges of
each of the subregions. This plot shows that subregions 13 has the largest domain,
whereas subregions 23 and 28 are much smaller. This plot also shows that the con-
vergent subregions fall in quite different and non-adjacent regions of the parameter
space.
As the sequential CART procedure is a random process due to the design run
sampling, we assessed the variability of the bounds of the convergent subregions for
the 2-D camelback function. The sequential CART procedure was run 20 times, and
the convergent subregions were manually mapped/paired to the low regions of the of
the function. For this part, we labeled the middle region of the function as subregion
1, the upper left low region as subregion 2, and the lower right low region as subregion
3. In some cases, no convergent subregions were found that appropriately match the

×