Databases Reference
In-Depth Information
4.5.4 Results and discussion
Experiments on real data
Table 4.7 shows the Mean Absolute Error values for the different algorithms and the data sets from the
user study in increasing order
7
. It is important to recall that the density levels of the data sets are fixed,
whereas the amount of explicitly available tag preference data is varied as described in Section 4.5.1.
Algorithm
US-real US-mixed
US-pred
regress-tag-ui
0.46
0.47
0.49
regress-tag
0.48
0.49
0.49
funk-svd
0.58
0.58
0.58
cosine-tag
0.59
0.59
0.60
0.59
0.59
0.59
item-item
0.60
0.59
0.60
cosine-tag-ui
Table 4.7: MAE results for real tag preference data.
We can see that the
regress-tag-ui
scheme proposed in this chapter to exploit user- and item-
specific tag preferences leads to the smallest MAE values for all data set variations. The improvement
over the previous
regress-tag
method is particularly strong when only real rating data is used. For
the situation in which we only rely on estimates of item-specific tag preferences (US-pred), our method
is minimally better or at least on a par with the previous method. When using only half of the existing
ratings, MAE values somewhere in the middle between the two extremes can be observed. As a result,
the numbers indicate that the accuracy constantly increases when more real tag preferences are entered
into the system.
Another observation on this relatively dense data from the user study is that the
cosine-tag-ui
method performs slightly worse than the
cosine-tag
andeventhe
item-item
algorithm. For the data
set US-pred,
item-item
also slightly outperforms
cosine-tag
.
Table 4.8 shows the corresponding values for precision, recall, and the F1-measure. The values were
obtained using a 10-fold cross-validation on the relatively small data set.
Algorithm
US-real
US-mixed
US-pred
F1
Prec.
Recall
F1
Prec.
Recall
F1
Prec.
Recall
regress-tag-ui
84.07 84.66
83.50
84.43 85.03
83.86
83.30 83.90
82.73
83.30 83.90
82.73
83.30 83.90
82.73
83.30 83.90
82.73
regress-tag
72.39 72.99
71.82
71.97 72.57
71.41
72.39 72.99
71.82
funk-svd
68.90 69.50
68.33
68.90 69.50
68.33
68.90 69.50
68.33
cosine-tag
68.66 69.26
68.10
68.23 68.83
67.67
68.05 68.65
67.49
cosine-tag-ui
68.05 68.65
67.49
68.05 68.65
67.49
67.30 67.99
66.82
item-item
Table 4.8: F1, precision, and recall results for real tag preference data.
Again, the numbers show that
regress-tag-ui
slightly outperforms
regress-tag
and the other
algorithms and is significantly better than the other methods. The neighborhood- and similarity-based
methods show the poorest results on this metric and are also outperformed by the SVD-based algorithm.
Note that there are virtually no differences in the observed numbers across the different data sets, which
can be accounted to the characteristics of the dense user study data set and the chosen evaluation metric.
MovieLens data
Table 4.9 shows the MAE numbers for the different MovieLens data sets. Note that unlike the three
data sets considered above the different MovieLens data sets have different density levels as described in
Section 4.5.1.
7
Note that we only report two decimal places in the tables due to standard error of the entries and randomness effects.
However, the order of the entries is based on the third decimal place.
