The Truck Backer-upper Problem - Design of Experiments for Reinforcement Learning - page 121

0.0084 0.0084

0.00601 0.00659

8 3.5

1.94 1.94

0.298 0.194

0.0706 0.0706

0.988 0.988

0. 9

0.5

0.1 0.2 0 .1

0. 1

0.957 0.961

1 0.965

0.974 0.888

ʱ mag ʱ mag

n hnodes ʱ ratio ʳ n hnodes ʱ ratio ʳ

ʻ ʻ

a b

Subregion 12: Threshold: 59.0 Subregion 25: Threshold: 58.6

0.01 0.01

0.0084 0.0084

8 8

2.72 2.72

0.24

0.0619 0.166

0 0.0712

0.985 0.985

0 . 9

0. 8

0 .8

0.9

0.3

0.1 0.2

0.2 0.2

0.1 0.1

0.957 0.957

0.949 0.949

0.888 0.888

0.41

ʱ mag ʱ mag

n hnodes ʱ ratio ʳ n hnodes ʱ ratio ʳ

ʻ ʻ

c d

Subregion 31: Threshold: 58.9 Subregion 32: Threshold: 58.6

0.00 0.20 0.40 0.60 0.80 1.00

Fig. 6.8 Response surface projections for convergent subregions 12, 25, 31, and 32. The contour
lines correspond to the probability that the surface lies above the threshold value of the subregion.

number of convergent parameter subregions that are able to learn how to control the
truck and back it into a specific location with a specific orientation.
We found that most of the ranges of the parameters within these convergent sub-
regions are consistent with those used in other applications of the TD( ʻ ) learning
algorithm, however, we found that ʻ must be set relatively low, which goes against
what is generally recommended for the TD( ʻ ) algorithm. The reason for this may
be related to what knowledge is actually learned, i.e., the set of actions to avoid
jack-knifing, although this was not specifically tested or known for sure.