Building a Recommendation Engine: The XELOPES Library - Realtime Data Mining - page 273

State state ¼ states.getState(i);
long is ¼ state.getIndex();
ActionSet as ¼ asSet.getActionSet(state);
boolean cnAs ¼ as.isCountable(); // true
while ( as.nextAction() ) {
Action action ¼ as.readAction();
long ia ¼ action.getIndex();
// ...use state (or index is) and action (or index ia) ...//

}

} ■

State-Value Function, Action-Value Function

The classes StateValueFunction and ActionValueFunction are used for the state-
value function v(s) and the action-value function q(s,a), respectively.
We start with the state-value function v(s) . In this topic, we mainly work with
discrete state and action sets and use a tabular representation of the state-value
function. However, as pointed out in Sect. 6.1.1 and used in Sect. 10.4.2 , in many
applications, v(s) is explicitly represented by a function, in most cases constructed
by regression. Thus, we use a MiningModel (Sect. 12.1.3 ) to store the function. This
means that in principle all data mining models of XELOPES can be used for v(s) ,
especially the regression models like linear and polynomial regression, regression
trees, neural networks, or even sparse grids. For different reasons, StateValue-
Function does not directly extend MiningModel but uses a variable function of
the class MiningModel to store the function.
To handle the important special case of tabular representations, the RL package
contains a special mining model - TableMiningModel - to store all pairs of
argument and function value { x , f(x) } directly. The class StateValueTable extends
TableMiningModel for state-value functions, i.e., to store all pairs { s , v(s) }.
Depending on whether the number of states is constant or not, it uses an array or
a hash table to store the function values. The StateValueTable is the default mining
model of StateValueFunction , i.e., if no other mining model is passed to

StateValueFunction , this one is used.
We describe the central methods of StateValueFunction. The first

public double getValue(State state) throws

MiningException;
returns the function value of state . Obviously, this method just calls function.
applyModelFunction(state) to invoke the mining function call.

public void setValue(State state, double value) throws
MiningException;