State-of-the-Art: Design of the Semantics Acquisition Games - Semantic Acquisition Games: Harnessing Manpower for Creating Semantics - page 104

7.3 Evaluation of SAGs: Attention, Retention
and Throughput
A SAG designer may consider a formal measurement of the effectiveness of his
game. In evaluating general game, two aspects are usually measured:
1. Attraction , or a number of players (per unit of time) the game is able to attract
to play for trial. The game must be appealing for the player from the very first
experience with it. If it is not, though it may even be the best game ever created in
it's later stages, it will never attract the player to play it and it therefore reduces
it's own potential. This happens for instance when the game rules are complex
and the player firstly need to understand them to start playing, which most players
refuse to do.
2. Retention , or how long (averagely) the players remain attracted to the game and
return to it for more playing. A game might entertain the player for only limited
amount of time and after that, the player ceases to play it. From this point of view,
the games based on social experience of the players have more potential than self-
challenge (which is exceed when player reaches his limits) and competition-based
games (which may lead to the player's frustration that he is not the winner). Of
course, this depends on the prevailing characteristics of target player groups (e.g.
males are more responsive to competition while females to socialization [ 16 ]).
Multiplication of these two measures results in the total number of man-hours that
players spent playing the game and a natural goal of game designers is to increase
this factor as much as possible.
The semantics acquisition games have the very same goal. Their effectiveness is
however, determined by more factors, namely:
1. Game throughput or how many task instances (in average) is a player able to
solve within 1h of playing (this measure was introduced by von Ahn [ 28 ]who
complemented it with average lifetime play, which corresponds to our retention).
2. Average quality of solutions delivered . This complements the previous quanti-
tative factor. For some tasks, the quality of valid artifacts is the is irrelevant, for
example an information, whether an image contains human figure or not cannot
have multiple levels of quality. On the other hand, we can define a quality measure
correlated, for example, to tag specificness in a game that deliver image tags.
7.4 SAG Design at a Glance: Discussion
We see that the field of design of semantics acquisition games comprises several
design perspectives, from which we can look at them. We have identified and
described six of them (they are summarized in a virtual six-dimensional space in the
Fig. 7.1 ). Each dimension represents a particular design issue which each SAG has to
overcome and is accompanied with several solution types or “design patterns” used