The Tandem Truck Backer-Upper Problem - Design of Experiments for Reinforcement Learning - page 135

to learn more challenging variants of this problem, though we believe that starting
with weights from a more general problem would likely be a more efficient training
scheme.
In the other domains investigated in this work, convergence was relatively fre-
quent, especially compared to the TTBU domain. Hence, we could use a moving
average (over a number of episodes) of the number of time steps to the goal as a per-
formance metric. In the TTBU domain, however, convergence is very infrequent, and
using a similar moving average metric would yield too few points to create a CART
model. This motivated the use of an alternative performance metric. Creating CART
models from the same moving average performance metric as in the other domains
would often only result in a tree stump (root node) and not a tree. The alternative
performance metric that we used was based on the maximum of the moving pro-
portion of reaching the goal. Values closer to 0.0 indicated very good performance,
and values closer to 1.0 indicated very poor performance where little or nothing was
learned. By using this metric in the sequential CART procedure, we were able to find
two parameter subregions that had good learning performance, though not perfect
performance. The use of an alternative performance metric shows the flexibility of
sequential CART such that it can be tailored to the characteristics of the responses
of the problem at hand.
Kriging metamodels were created for the mountain car and TBU problems, how-
ever, we did not do this for the tandem truck backer-upper problem. The primary
reason for this is because we do not consider the problem to be solved, and any
metamodel would not be an accurate representation of the true performance. Addi-
tionally though, we found that the responses (i.e., number of time steps to the goal)
for the same design points were extremely variable, and that a metamodel could not
accurately model the response surface. This variability could be due to the fact that
the problem was relaxed with loose goal tolerances, and it is possible that an accurate
metamodel could be created after additional training is used to create a more accurate
controller.
References
Riid, A., Leibak, A., & Rustern, E. (2006). Fuzzy backing control of truck and two trailers.
In Proceedings of the IEEE International Conference on Computational Cybernetics (ICCC),
Budapest, Hungary, 20-22 August (pp. 1-6). doi: 10.1109/ICCCYB.2006.305690
Tanaka, K., Taniguchi, T., & Wang, H. O. (1997). Model-based fuzzy control for two-trailers
problem: Stability analysis and design via linear matrix inequalities. In Proceedings of the 6th
International Conference on Fuzzy Systems, Barcelona, Spain, 1-5 July (pp. 343-348). doi:
10.1109/FUZZY.1997.616392
Widrow, B. & Lamego, M. M. (2000a). Neurointerfaces: Applications. In IEEE Adaptive Systems for
Signal Processing, Communications, and Control Symposium, Lake Louise, Alberta, Canada,
1-4 October (pp. 441-444). doi: 10.1109/ ASSPCC.2000.882515
Widrow, B. & Lamego, M. M. (2000b). Neurointerfaces: Principles. In IEEE Adaptive Systems for
Signal Processing, Communications, and Control Symposium, Lake Louise, Alberta, Canada,
1-4 October (pp. 315-320). doi: 10.1109/ASSPCC.2000.882492