Database Reference
In-Depth Information
Chapter 4
Recommendations as a Game: Reinforcement
Learning for Recommendation Engines
Abstract We describe the application of reinforcement learning to recommendation
engines. At this, we introduce RE-specific empirical assumptions to reduce
the complexity of RL in order to make it applicable to real-live recommendation
problems. Especially, we provide a new approach for estimating transition probabil-
ities of multiple recommendations based on that of single recommendations.
The estimation of transition probabilities for single recommendations is left as an
for testing online recommendations.
An effective approach to using reinforcement learning for recommendation engines
is described below. In the simplest case, the product detail views form the states, the
recommended products the actions, and the rewards the clicks or purchases of
the products. The goal consists (depending on the chosen reward) in maximizing
the activity (clicks) or the success (sales).
Figure
4.1
illustrates the use of RL for product recommendations in a web shop
and shows the interaction between the recommendation engine and the user. Here,
the optimal proven recommendations are marked with “*.”
In the first and third steps, the recommendation engine is following the
proven recommendations (exploitation); in the second step, a new recommendation
is tried out (exploration). The user ignores the first recommendation, but accepts
the second and third. The feedback arrows symbolize the updating of the
recommendations.
Although this modeling may appear self-evident, it nevertheless represents a
highly complex task. Firstly, web shops generally offer very many products, as
a rule between a few thousand up to a few million (for instance, at a bookshop).
Many of these products have virtually no transaction history, i.e., they have scarcely
ever been bought; indeed, some have never even been clicked on. Furthermore,
