Discussion - Design of Experiments for Reinforcement Learning - page 146

effects in reinforcement learning using simple one-factor-at-a-time experiments
(Gatti et al. 2011a ) and a full-factorial design (Gatti et al. 2013 ) in other problem
domains. The work presented herein attempted to more comprehensively explore
the effects of reinforcement learning parameters in additional domains to gain an
understanding of the utility of this learning method.

￿ Identification of a new class of scenarios in computer experiments (DoE): As

far as is known, the study of computer simulations that can potentially diverge
has not been considered from a design of experiments perspective, and their
study in this work is novel. While the problems we study are concerned only
with reinforcement learning, there are other fields that use computer simulations
that have the potential to not converge; for example, finite element modeling or
computational fluid dynamics. The methods developed herein may be useful in
those other fields as well.

￿ New application area of computer experiments (DoE): Stochastic computer

experiments are relatively commonplace for methods such as finite element analy-
sis. Although some learning algorithms are commonly acknowledged as stochastic
algorithms, they are not studied using design of experiments approaches. This
work considers learning algorithms to be another method that can be classified
and studied as a stochastic computer experiment.

￿ Novel domain applications (RL): This work is the first to apply a pure rein-

forcement learning approach to learning the tandem truck backer-upper problem.
Additionally, along with our brief introductory work that applied a pure reinforce-
ment learning approach to learning the single truck backer-upper problem (Gatti
and Embrechts 2014 ), we are the first to explore this problem as well. Although
these problems are in silico problems, they are more challenging and closer to
real world problems than the classic reinforcement learning benchmark problems,
such as the mountain car problem. These domains could be used as new benchmark
problems when developing and evaluating reinforcement learning methods.

￿ Development of custom reinforcement learning code (M): The code developed

in this work allows for the inclusion of additional domains, learning algorithms,
and representations. The addition of problem domains is relatively simple because
of the structured and consistent format of the code and because of the extensive
infrastructure of accessory functions. The addition of other learning algorithms
and representation is also straightforward. Using the code is also simple and
follows consistent procedures for each domain and representation, and thus this
code could be made available to the reinforcement learning community.

￿ Explicit definition and description of domain characteristics (RL): As far

as is known, the domain characteristics described in Sect. 2.2.1 have not been
formally defined in the literature. We considered problem domains on a finer
scale and explicitly define domain characteristics. The motivation for this is our
belief that a successful reinforcement learning application lies at the intersection
of specific combinations of domain characteristics and learning algorithm and
representation parameter settings.