Design of Experiments for the Mountain Car Problem - Design of Experiments for Reinforcement Learning - page 166

Appendix B
Design of Experiments for the Mountain
Car Problem

This chapter previously appeared as: Gatti et al. (2013). An empirical analysis of
reinforcement learning using design of experiments. In Proceedings of the 21st
European Symposium on Artificial Neural Networks, Computational Intelligence and
Machine Learning (ESANN), Bruges, Belgium, 24-26 April (pp. 221-226). Bruges,
Belgium: ESANN. This work could be considered a proof-of-concept for applying
a design of experiments approach to analyzing reinforcement learning by analyzing
both convergence and performance, which has grown into the work described in this
dissertation.

B.1 Introduction

Reinforcement learning has had a handful of successes in challenging domains,
including backgammon (Tesauro 1995) and helicopter control (Ng et al. 2004). How-
ever, this learning method has not had nearly the success of other machine learning
approaches, and this may be due to our limited understanding of the complex in-
teractions between the learning algorithms, functional representations, and domain
characteristics. Theoretical analysis of reinforcement learning is limited to rather
simplistic scenarios (Tsitsiklis and Roy 1996), and this motivates the use of empiri-
cal methods to understand the behavior of reinforcement learning (Bhatnagar et al.
2009; Gatti et al. 2011a; Sutton and Barto 1998). However, empirical studies often
use simple parameter studies and assess effects by comparative observations (Sutton
and Barto 1998), which cannot easily reveal parameter interactions. A more efficient
and statistically rigorous approach is to use formal design of experiments approaches
(Montgomery 2008). The purpose of this study is to use a design of experiments ap-
proach to understand the effects of three parameters of the TD( ʻ ) algorithm when
using a neural network to learn the mountain car domain.