Game Development Reference
In-Depth Information
these, we initially manually reviewed the use of terms in the game and later let
a posteriori cheating detection heuristics (described below) to identify them for us.
Over time, the ban list grew and became an effective tool for preventing the abuse of
stopwords.
Unfortunately, some terms had to be excluded from the game (and thus the result-
ing termnetwork) even though they arguably had semanticmeaning and legitimacy to
be used as negative terms in certain tasks (e.g., “restaurant -menu” or “school -table”,
where “menu” and “table” are banned due to being common in web content and
HTML code). Such terms had to be sacrificed in order to keep the game fair; though
they could still be used as task words.
8.1.2.2 A Posteriori Cheating Detection
Regarding the second type of threat—the hacking of the game's implementation—no
prevention rules were possible. While this was a problem-dependent issue, it led us
to a cheating detection scheme which is generalizable to other semantics acquisition
games.
Our
a posteriori cheating detection heuristics
suggests suspicious players and
potential stopwords to game administrators. The premise is that
suspicious player
actions
(such is the use of stopwords)
1. would result in high ladder rankings of the cheaters,
2. would result in repetition of a same player behavior for different tasks and
3. would result to marginal contributions to the usefulness of the game's output.
The more these conditions are satisfied for one player, the more is this player suspi-
cious for performing “unwanted actions”. For the Little Search Game, the heuristics
works following these steps:
1. Collect highest ranking players (their number is parametric).
2. Collect their best attempts (sets of negative search terms) for tasks where they
ranked high.
3. Drop terms appearing in only one task.
4. Mark “suspicious” terms, i.e. terms that significantly decreased the number of
results for more than one task term. Pass them to administrator.
For each of the suspicious terms, two measures are computed to aid the administrator
in decidingwhether to ban the term (or the player that used it) or not. The first measure
is universal for any SAG, the second is problem dependent—related only to the LSG:
1. The consensus participation (interval between 0 and 1). How many players agree
on this term for this task (relative to other terms players agree on)? If the suspicious
term (artifact in general) is used by other players too for the same task, it is likely
not abusive.
Search WWH ::
Custom Search