Design of Experiments - Design of Experiments for Reinforcement Learning - page 59

is largely restricted to the design and analysis for deterministic computer models
(Sacks et al. 1989 ; Santner et al. 2003 ; Chen et al. 2006 ). Such models produce
the same response every time for the same set of input parameters, and thus the
experimental error from classical experiments is not present. Furthermore, because
of the absence of experimental error, classical design of experiments practices such
as randomization, replication, and blocking are not necessary.
The work herein consists of simulations that are based on a learning algorithm
that is not deterministic, and experimental error is present in the response variables
from the simulations. The presence of experimental error in these experiments means
that some practices of classical design of experiments techniques must be utilized to
deal with experimental error, and in particular, replication. Thus, the experiments in
this work have qualities of both classical and contemporary design of experiments,
and techniques from both approaches will need to be employed. In this section, we
describe approaches that are commonly used design of experiments techniques for
computer simulations.
Metamodeling Metamodeling, also known as surrogate modeling, is the process of
developing a statistical model that approximates the response of a system based on a
set of input variables or parameters (Chen et al. 2006 ; Barton 2009 ; Sudret 2012 ). A
response surface model, as described above, is one example of a metamodel. The use
of metamodeling has grown in large part due to the increased use of computationally
expensive computer simulations (e.g., each run takes hours to days of computation
time). In these cases, the primary use of metamodeling is to develop a model that
sufficiently approximates the response of the simulation but that can be evaluated
'on-demand', essentially minimizing the use of or replacing the computationally
expensive simulation. Although acting as a surrogate model for the purposes of
prediction, optimization, or model tuning is often the purpose of metamodels, their
use is not limited to approximating simulation responses. Depending on the statistical
form of the metamodel, it may provide more concrete and interpretable knowledge
of the response of simulations, and such knowledge is of particular interest in this
work. Similar to classical design of experiments, the development of the metamodel
requires careful consideration of two tasks: determining the experimental runs that
are used to build the metamodel, and specifying the form of metamodel (Sacks
et al. 1989 ; Chen et al. 2006 ; Barton 2009 ). As previously mentioned, there may
not be an experimental design or statistical model that perfectly complements the
qualities of the experiment, and this holds for metamodeling as well, which often
requires compromises on the design or modeling approach. An exhaustive review of
metamodling is beyond the scope of this work, however, below we discus some of
the experimental designs and statistical models used in metamodeling. The reader is
directed to works by Chen et al. ( 2006 ) and Barton ( 2009 ) for excellent reviews that
thoroughly compare these designs and models.
Experimental Designs of Metamodels Common experimental designs for meta-
models include response surface method designs and Number-Theoretic Methods
(NTM) (Chen et al. 2006 ). Design methods for response surface methods are
mentioned above. Number-Theoretic Methods can be referred to as a space-filling