Methodology - Design of Experiments for Reinforcement Learning - page 77

We should note that other metrics could be developed to quantify other aspects
of a learning curve. For instance, persistence could be used to assess the stability
of learning over a period of time, although the use of a moving window in the
computation of the three metrics that are used (level, range, and slope) does assess
persistence to some degree. A separate persistence measure was not used primarily
because the learning algorithm explored in this work, TD( ʻ ), is known to be unstable
such that the agent can unlearn and performance can decrease after reaching a high
level (Gatti et al. 2011a ; Gatti and Embrechts 2012 ).

4.1.5 Example: 2-D 6-hump Camelback Function

A 2-dimensional function is used to illustrate the sequential CART modeling
procedure. The function of interest is the 2-D 6-hump camel back function:

4 x 1 +

x 1
3

x 1 x 2 + − 4 x 2 x 2 2 . 1 x 1 +

y = − 4 + (4.3)

which is normally defined over x 1 2, 2]. This function

is augmented to fit the context for when the sequential CART procedure is ap-
plicable for the type of problem of interest. Specifically, this function should be
non-deterministic and there should be some convergence indicator that is asso-
ciated with each response value. We therefore modify this function and include
an additional Gaussian-distributed zero-mean noise term. Additionally, a conver-
gence indicator variable ˈ was created based the response value y . The augmented
camelback function and the convergence equations are:

∈ [ − 3, 3], x 2 ∈ [ −

4 x 1 +

x 1
3

x 1 x 2 + − 4 x 2 x 2 +

2 . 1 x 1 + y = − 4 + N (0, 0 . 2) (4.4)

U (0, 1) < 1

1

ˈ = − (4.5)

1 + e − 3 . 5 y + 2 . 5

The deterministic version of this function has six minima, four local minima and two
global minima. Figure 4.2 shows a contour plot of the deterministic and noisy versions
of this function, where the low regions are indicated by lighter colors. The two global
minima are equal to

− 1 . 0316, which are located at ( x 1 , x 2 ) = ( − 1 . 0898, 0 . 7126)

and (0 . 0898, 0 . 7126). The noisy augmented function has low regions around these

locations. Convergence is most likely in these regions as well, and becomes less
likely away from these locations as the function value increases. The goal of the
sequential CART procedure is to find subregions, defined by variables x 1 and x 2 ,
that have a high proportion of convergent responses (i.e., ˈ

−

1).
Table 4.2 provides a summary of the parameters and settings used in the sequential
CART procedure for this example problem. The parameters and settings used for
this example and for the domain problems that follow were selected based on initial

=