CityLights: A Method for Music Metadata Validation - Semantic Acquisition Games: Harnessing Manpower for Creating Semantics - page 82

However, in order to validate the tags in the dataset, we need it for individual tags,
not groups of tags. The individual tag validation is achieved by variation in the game
question composition. For each question that features the same track, the subset of
it's original tags is always different, causing a variation in the descriptiveness of the
whole choice set and therefore, a space for really good, descriptive tags to “prove”
their power.
This variation effect is best illustrated by an example: consider a music track
decorated with 4 tags within the input dataset: nice , awesome , Beethoven and classic .
Consider that only the Beethoven tag is assigned correctly, while other tags are
subjective or vague. The question featuring this particular track appears 4 times in
the game, each time with a different 3-member subset of the tags above:
1. Beethoven, awesome, nice
2. nice, classic, Beethoven
3. awesome, classic, Beethoven
4. awesome, nice, classic.
Suppose the tag Beethoven (featured in the first three occurrences of this question)
has a sufficient descriptive power that it convinces players to pick the presented tag
set each time as a first choice. Each time this happens, all of the tags presented in the
choice will have their support increased. However, when the fourth option comes by,
the player will not select it (because of the vague and non-descriptive nature of its
tags) and will select another choice instead. Therefore, after all four combinations,
the truly correct and descriptive tag Beethoven will have a higher support value, than
the other tags.
A valid design question with this game might be: why did we chose to compose
choices of multiple tags, and not single tags? The answer is a practical one: with
only one tag, the game would be hardly playable for the player. A high number of
wrong or noisy tags would make the guessing very hard or a not challenging random
betting. If the player has more tags in each choice (in our experiments, we used 5),
he encounters “good” ones more easily and has therefore more successful attempts.
6.1.3 More on the Game's Features
The basic choice question task forms the core element of the CityLights SAG. It
is, however, complemented with other gameplay features that enhance the metadata
validation abilities of the game and encapsulate its purpose into the storyline.
Explicit tag “rule-out”
The question answering can be considered as a form of implicit user feedback on the
validity of the featured tags (the player solves different task from which the validity
is inferred). However, we also offer the player to rule out the wrong tags explicitly.
For a small point reward, the player has an option to mark tags that confused his
decision making: