Design of Experiments for the Mountain Car Problem - Design of Experiments for Reinforcement Learning - page 171

0.6 0.7 0.8 0.97 0.98 0.99 0.7 0.8 0.9

0.6 0.7 0.8 0.97 0.98 0.99 0.7 0.8 0.9

ʳ

ʻ ʵ

Fig. B.2 Boxplots of convergence episode and performance versus ʻ , ʳ , and . This figure
previously appear in Gatti et al. (2013).

Table B.2 Analysis of variance for the logarithm of the episode at convergence; the colon between
variables indicates a variable interaction ( SS = sums of squares).
Model term SS F -ratio p -value

ʻ 0.207 48.864 <0.001

ʳ 0.231 54.603 <0.001

0.295 69.557 <0.001

ʻ : ʳ 0.037 8.812 0.003

ʻ : 0.016 3.752 0.054

ʳ : 0.005 1.164 0.282

Residuals 0.747

Table B.3 Analysis of variance for the logarithm of the mean performance.
Model term

SS F -ratio p -value

ʻ 0.000 0.000 0.996

ʳ 0.043 20.275 <0.001

0.005 2.368 0.126

Residuals 0.381

B.4 Discussion

This study employs design of experiments and statistical analysis to aid in under-
standing the behavior of TD( ʻ ) in a specific domain. Experimental design

D 1 shows
that the network empirically converges to a stable solution in a fairly small region of
the parameter space, however, convergence is not guaranteed. Experimental design
D 2 indicates that significant effects are of low order. While all parameters have an