Up the Down Staircase: Hierarchical Reinforcement Learning - Realtime Data Mining - page 98

Fig. 6.4 (a) Sparse grid and function in 2D, (b) sparse grid in 3D [Zu00]
We will describe sparse grids in detail in Chap. 7 and summarize once again the
advantages of wavelets. Wavelets are (asymptotically) optimal:
• For the signal processing and compression of smooth signals
• For the solution of the central classes of differential and integral equations
• For the approximation of multidimensional smooth functions
In a certain sense they represent the optimal compromise between particles
(nodal basis) and waves (multi-scale basis): by this means for the first time we
can optimally solve multidimensional operator equations and optimally transmit,
smooth, and analyze their solution functions. Speaking philosophically, the revo-
lutionary wave-particle dualism of quantum physics (Chap. 1 ) has its approxima-
tion theory counterpart here. Therefore, multi-scale bases and wavelets represent
one of the hottest research topics in numerical analysis. For that reason we want to
use this groundbreaking approach for RL too.
6.1.2 Algebraic Approach
After so much euphoria on the topic of hierarchical bases and wavelets, we now
encounter the first problems. In the case of the recommendation engine, in accor-
dance with the modeling in Chap. 4 , our states and actions are inherently discrete.
Therefore, we cannot directly change over to a continuous state-value and action-
value function v(s) and q(s, a) . We know however that the Bellman equation ( 3.6 )is
the discrete counterpart to the Hamilton-Jacobi-Bellman differential equation. The
question then arises of whether despite this the multilevel approach is useful here.
The answer is provided by access via algebraic multigrid methods (AMG).
These constitute the algebraic counterpart to the analytical multigrid method