Methodology - Design of Experiments for Reinforcement Learning - page 71

approach but for the purpose of optimization. Other proportions could be used as
well, but it should be noted that the algorithm will progress differently; for example, if
a larger proportion of the points was considered to be in the low group, the algorithm
would find convergent subregions slower. The exact effects of this split proportion
should be evaluated under different problems, although this is left for future work.
The subregions of the CART model that are labeled as low are then explored in
subsequent experimentation iterations. For each low CART model leaf q , a new set
of design points X (1)

q ,0 are created within the subregion defined by leaf A (1) q ,0 , where this

subregion is bounded by the CART model splits. As mentioned, the second subscript
value in these variables indicates the parent subregion of the current subregion q .
These new design points X (1)

q ,0 are then run through the simulation, and corresponding

numerical results Y (1)

q ,0 and result types (1) q ,0 are obtained. Design runs (and their

numerical results and result types) from X (0)
0

that fall with leaf q (call these entities

, and (0 q , respectively) are appended to the runs from X (1) X (0 q , Y (0)

q ,0 resulting in a
set of points X q ,0 and associated responses Y q ,0 , low / high indicator variables Z q ,0 ,
and convergence indicators q ,0 . Design points are appended so that design runs
from previous iterations are not merely thrown away. Another CART model T (1)

q

q ,0 is
then constructed that models Z q ,0 as a function of X q ,0 . The next iteration proceeds
by considering all q leaf nodes from iteration 1 as new parent nodes from iteration 2,
sampling within these subregions, obtaining and computing responses, and creating
new CART models.
The convergent subregions of the parameter space are of interest and convergent
runs usually have lower numerical results Y , and this sequential CART procedure is
built on this assumption. Consequently, the leaves of the CART model that are la-
beled as low are of primary interest because these subregions will likely have greater
proportions of convergent reinforcement learning runs compared to outside of these
subregions. As sequential experimentation proceeds, the proportion of convergent
runs ʲ ( i )

q , p within leaves labeled as low is likely to increase. The sequential experi-
mentation procedure continues finding and exploring subregions until the proportion
of convergent runs with leaves labeled as low surpasses some threshold proportion
of convergence ʸ ˈ . These subregions that surpass this threshold convergence pro-
portion are then considered to be convergent subregions, which can then be analyzed
or used in further and/or additional experimentation.
This algorithm is presented in Algorithm 2 and Fig. 4.1 shows an example of how
this algorithm proceeds in a 2-dimensional setting. Note that the sequential CART ex-
perimentation procedure itself could also be viewed as a tree, where parameter space
subregions are children of parameter space subregions from parent subregions of
previous iterations, and this visualization representation of the experimental process
is used in the experiments performed later in this work.