Database Reference
In-Depth Information
MiningVector
although this might be useful in some conceptual sense as we will
discuss next.
The class
StateActionVector
, which also extends
IndexedMiningVector
, repre-
sents a mining vector composed of a state and an action. This class is required to
model action-value functions within the XELOPES framework as
MiningModel
,as
will be described below. There are also other similar classes that extend
IndexedMi-
ningVector
and represent compositions of states, actions, and rewards. Further
examples are
StateRewardVector
(state and reward),
SampleVector
(state, action,
reward, next state), and
SarsaVector
(extends
SampleVector
by additional next
action and thus represents state, action, reward, next state, next action). All these
classes in the end extend
MiningVector
what is extremely helpful because we can
use them in the same
learn
and
apply
methods of the agent.
StateSet, ActionSet, StateActionSet
The class
StateSet
models the state set
S
, and
ActionSet
the actions set
A(s)
of all
actions available in state
s
. Both are containers of mining vectors of states and
actions, respectively. The class
ActionSet
further contains a reference to its state
s
.
Because the number of valid states and actions may be infinite and even
uncountable,
MiningVectorSet
is used to store the vectors (Sect.
12.1.2
). In case
of discrete problems, a
MiningInputStream
as subtype of
MiningVectorSet
is used
to store the vectors; we then call the mining vector set
countable
what is indicated
by the
isCountable
method. For countable vector sets, there exist specific methods
like
nextState
(
nextAction
) and
readState
(
readAction
) which repeat the methods of
MiningInputStream
but are more simple to use because explicit-type conversions
(e.g., to
State
object or
long
type) can be avoided. Additionally, the methods
addState
(
addAction
) not only add a new
state
(
action
) to the set but can automat-
ically update the index of the
state
(
action
). Of course, using the
getMiningIn-
putStream
method also the “classic” stream methods can be used to access the states
and actions.
The interface
StateActionSet
represents a state set
S
and the action sets
A(s)
for
all states
s
of
S
. The method
getStateSet
returns the state set, and the method
getActionSet(State s
) the action set for the specified state
s.
Typically,
StateActionSet
is implemented by environments and contains all admissible states
and actions.
Example 12.18
The example assumes countable state and action sets and iterates
over all states and corresponding actions of state-action set
.
We demonstrate
different vector iteration and access methods for the state and action sets which
are similar to mining input streams.
StateActionSet asSet
¼
...// reference to state-action set
StateSet states
¼
asSet.getStates();
boolean cnSt
¼
states.isCountable(); // true
for (long i
¼
0; i
<
states.getStatesNumber(); i++) {
