CityLights: A Method for Music Metadata Validation - Semantic Acquisition Games: Harnessing Manpower for Creating Semantics - page 86

ans w ∗ con f

nbr

Δ support =−

The player rules-out a tag after a “correct choice”.

Δ support =− tag w

The player marks a positive tag after an incorrect choice.

(was not included in the game release).

Δ support = tag c

The player buys a tag

(was not included in the game release).

Δ support =− buy

The important question about the validation process is: when the validation of a
single tag ends? Naturally, it ends after its support reaches one of the thresholds.
However, this may take a long time even if the tag retrieves a lot of feedback, because
the individual feedback actions might be contradictory. Ultimately, the tag might
circulate in the game forever, “wasting” the power of the crowd. Therefore, we
chose to put the tags out of the game after a critical number of feedback actions
has affected it, reaching neither upper or lower threshold. This critical number was
a constant which we handpicked as a estimated number of feedback actions needed
for a very good or very bad tag multiplied by ten. Tags that have been drawn out of
the process this way, were declared as “not confident” cases.

6.2 Experiments: Evaluation of the Metadata
Validation Ability

To validate our approach, we have developed and deployed the CityLights as a web
application and conducted an open, unsupervised experiment with real users. We
then computed the correctness of the tag validation (against the expert validations in
the same corpus). Then we conducted the simulations to achieve best results. In this
section, we describe these experiments.

Hypotheses

The method is able to correctly validate the given tag assignments to music tracks.
The primary target function is the correctness i.e. the ratio of correctly evaluated tags
to all evaluated tags. Secondary, our goal was to track the confidence , i.e. how much
of the featured tags the method was confident to evaluate to all tags it featured.