Up the Down Staircase: Hierarchical Reinforcement Learning - Realtime Data Mining - page 109

κ 1 RAL δ

kk 1
RAL

RAL 1

kk 1

1

1 κ 1 RAL δ

kk 1
RAL

1

1

k 1 k R δ AL

RAL

RAL 1

1

¼

k 1 k R δ AL

1

1

RAL

1

k 1 k R δ AL 1

1 αγ

¼

1 R

RAL

k 1 k δ AL

Ak k 1
1 αγ R

R δ 1

1 αγ

¼

k 1 k δ AL

γε
1 αγ γε

1
1 αγ

,

where we invoked upon the identity
κ 1 RAL ¼ κ 1 ð ð 1 αγ ÞI Þ ¼ 1 :

Hence, we obtain

2

γ ε

E ðÞ LBkk 1 kk 1 kk 1

kk 1 :

Þγ 1 γ

1

1 α þ ε ð

1 (1 αγ

If the stronger condition ε < γ ) holds, we attain the stronger bound

2

γ ε

1

E ðÞ

kk 1

Þγ 1 γ 1 α þ ε ð

from essentially the same calculation.

□
The message of this result is as follows: the closer the transition probability
matrix is to a block-diagonal matrix composed of rank-1-blocks corresponding to
the classes of the partition, the faster converges the AMG procedure. With regard to
recommendation engines, we may conclude that the method is sensibly applicable
if a major part of the transitions is between products in the same class. Furthermore,
the behavior within the classes should be almost memoryless, that is, the transition
to a product within a class hardly depends on the previous product.