Our SAGs: Design Aspects and Improvements - Semantic Acquisition Games: Harnessing Manpower for Creating Semantics - page 119

2. Stopword probability (interval between 0 and 1). The method queries the sus-
picious term for the average co-occurrence with a set of 10 manually selected
reference terms (with a small mutual co-occurrence on Web). Alternatively, the
document frequency of the term can be used. If the co-occurrence with them is
high, the terms are more likely to be stopwords.
It is then up to the game administrator to judge, whether the term should be banned.
However, these measures allows a quick filtering of the candidates.
During the game deployment, the number of banned terms increased from 230 to
430. The “a posteriori” heuristics was able to identify 30 terms not suitable for the
game. The rest of the banned terms was inferred by game administrator according
to these 30 terms. For example, in one case, the suggested problematic term was
a number (unfortunately, present in a dictionary). This naturally led the administrator
to also ban other numbers.
8.1.2.3 A Posteriori Cheating Detection: A Generalization
Based on what we described above, we formulate the general a posteriori cheating
detection method for semantics acquisition games. It is a regression based anomaly
detection. It consist of the following steps:
1. Collect player produced problem solutions (artifacts).
2. Assign these artifacts with score the players received for creating them.
3. Measure the “usefulness” of the solution. This is problem-dependent, in all cases
however, the “usefulness” can be substituted by measuring the participation of
each solution on the consensus ( consensus rates ), i.e. how large is the support of
players for this solution in comparison to the solution which is supported most?
4. Create a relation of between usefulness (x-axis) and the score (y-axis). Compute
its regression of an order considered optimal (e.g. linear, if the players are meant
to be rewarded linear to the value they provide).
5. Identify outliers above the regression (high scores), these are the suspicious solu-
tions.
Instead of artifacts themselves, the behavioral patterns (of the players) that led to
the artifacts may also be considered. The behavioral pattern is an abstract sequence
of player's actions that somehow characterize player's behavior in the game (for
example, typing and deleting the same word repeatedly). It might be viable if the
game mechanics are not so simple and may be combined in many ways to create
problem solutions. If for example, a pattern has led to a suspicious solution, it may
be a good idea to investigate where else this pattern occurred.
8.1.3 Evaluating the Appeal to the Player
We also conducted series of experiments to examine how players perceive the Little
Search Game in terms of attractiveness [ 4 , 5 ], which is one of two key factors in
quantifying the SAG impact [ 6 ] (the second one is throughput , i.e. number of problem