Design of Experiments - Design of Experiments for Reinforcement Learning - page 53

Desired
information

Coneption
process

Analytic
method(s)

Practical
application

Experiment/response
characteristics

Experimental
design

Fig. 3.1 The design and practical application processes of design of experiments. The experimental
design process starts from the desired information, which, along with characteristics of the exper-
iment and response, helps select a potential analytic method, and then directs the experimental
design to be run. The practical application works in the reverse manner, beginning with testing
all design points in the experimental design, analyzing the data, and finally obtaining the desired
information.

the response, then limits the analytic methods that could be used. Once a particular
analytic method is selected, an appropriate and well-paired experimental design can
be constructed.
While the conception of an experimental approach proceeds in a top-down manner,
as in Fig. 3.1 , the practical application and implementation of these steps work in
the reverse order. The application of the design of experiments begins by performing
the experiment and collecting data. These data are then analyzed using statistical
methods, which provides the desired information.
Figure 3.1 may suggest that the conception of the experimental procedure is a
clear and step-wise process, consisting of identifying particular information, which
leads to a single analytic method, which then leads to a single experimental design.
However, this is hardly the case in practice. In ideal cases, there is a particular analytic
method that is well-suited to the characteristics of the experiment and data, and this
method provides all of the desired information; this intersection is represented by the
overlapping region in Fig. 3.2 . However, in many cases, there may not be a single
analytic method that yields all of the desire information, or that is ideally suited for
the peculiarities of the data collected, and there is often not a perfect overlapping
region as in Fig. 3.2 . Rather, each analytic method has advantages and disadvantages
with regards to the information that it can provide and the characteristics of the data
to which it is applied. Thus, a trade-off must therefore be made that considers all
aspects of the design of experiments procedure.
In general, design of experiments techniques can be categorized as classical and
contemporary (Fig. 3.3 ), or alternatively, as real-world (physical) experiments and
computer experiments that are either deterministic or stochastic (Kleijnen 2008b ).