Little Search Game: Lightweight Domain Modeling - Semantic Acquisition Games: Harnessing Manpower for Creating Semantics - page 57

Research questions. Is there a variety in relationship types of LSG term network?
What relationship types are present there? How many of the LSG relationships are
not present in the existing knowledge base?
Hypothesis. The examined relationships are semantically sound.
Data. We took 400 strongest LSG network term relationships. The ConceptNet
base was accessed via online REST API.
Process. The judges (individually) evaluated each of the LSG term relationships.
First they evaluated semantic soundness itself (with values 2-“strong”, 1-”weak” and
0-“no”), then they assigned one of the 23 relationship types. Then, the individual
contributions were merged. The soundness scores were summed. All relationships
with sum equal to 4 or 3 were declared as sound, with sum equal 2 as “disputable”
and others were declared not sound. When judges matched on the relationship type,
the type was accepted, otherwise a general “related to” type was assigned to the
relationship.
Results. The repeated soundness evaluation rendered 80%of LSG relationships as
sound, 8% as “disputable” and 12% as not sound. This is worse result than with our
initial soundness evaluation. In this experiment however, we took more relationships
including “weaker ones” (according to w value provided by the game itself). When
we took only first 100 relationships (instead of 400), the “sound” portion of the set
returned to 93% (with “not sound” dropping to 1%).
The confrontation with the ConceptNet shown, that only 41% (164) of the 400
examined LSG term relationships were present in the knowledge base (which argues,
that there is still space for finding new ways how to extend such bases).
The Fig. 4.6 compares the distribution of relationship types assigned by judges
(over all LSG relationships) and distribution of relationship types assigned by Con-
ceptNet (over those LSG relationships known to it). The LSG relationships that were
recognized by ConceptNet were predominantly of 6 (of the total 23) types, while
other types were not used so often. The LSG term network relationships, however,
appear to have much richer distribution according to manual evaluation.
4.4 The TermBlaster
We created the game TermBlaster 2 as a modification of the original Little Search
Game. By this we followed two main goals.
1. To move from domain of general semantics to a specific domain and demonstrate
its semantics acquisition capabilities there. For this, we choose the domain of
software engineering education.
2. Make the game more attractive to players.
2 The authors wish to thank the bachelor student, Marek Kiss, for helping with implementation of
the game.