Discussion - Design of Experiments for Reinforcement Learning - page 149

procedure can be better understood. Additionally, the performance of this pro-
cedure also depends on the underlying problem being investigated. Future work
should therefore explore the effects of algorithm parameters and evaluate the per-
formance of sequential CART on different functions with various characteristics.
While this is the first use of the sequential CART procedure, we feel confident
in its value for future applications based on its performance in the challenging
problems presented in this work.

￿ Experimental efficiency: Experimental efficiency was not a high priority in this

work, as we were interested in obtaining the most information about the rein-
forcement learning problems and we were not really limited by a computational
budget. That said, we believe that the experimental procedure could be made more
efficient, both for sequential CART and for kriging metamodeling. Improving the
sequential CART efficiency could be done by more aggressively reducing the
parameter space (i.e., selecting a smaller proportion of points to be considered
ˉ L ), or using additional subregion pruning procedures; these avenues are also
related to better understanding the dynamics of sequential CART. It is possible
though, that in attempting to increase the experimental efficiency of these meth-
ods, the accuracy of the resulting convergent subregions and kriging models may
be affected; however, this is not known for certain at this point. As mentioned,
however, an obvious future direction is parallelizing this algorithm, which would
likely be able to find convergent subregions with greater accuracy and in less
elapsed time.

￿ Sequential kriging experimentation: The kriging metamodels were based on

a single large experimental design, but sequential experimentation could reduce
the total number of design points required. However, as mentioned, preliminary
experiments using established resampling techniques resulted in poor candidate
design points to be evaluated, and thus a new, possibly application-specific,
resampling technique may need to be developed.

References

Bect, J., Ginsbourger, D., Li, L., Picheny, V., & Vazquez, E. (2012). Sequential design of computer
experiments for the estimation of a probability of failure. Statistics and Computing , 22(3),
773-793.
Bichon, B. J., Eldred, M. S., Swiler, L. P., Mahadevan, S., & McFarland, J. M. (2008). Efficient
global reliability analysis for nonlinear implicit performance analysis. AIAA (American Institute
of Aeronautics and Astronautics) Journal , 46(10), 76-96.
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and Regression
Trees . New York, NY: Chapman & Hall.
Chapman, W. L., Welch, W. J., Bowman, K. P., Sacks, J., & Walsh, J. E. (1994). Arctic sea ice
variability: Model sensitivities and a multidecadal simulation. Journal of Geophysical Research ,
99(C1), 919-935.
Chen, E. J. & Lin, M. (2014). Design of experiments for interpolation-based metamodels. Simulation
Modelling Practice and Theory , 44, 14-25.