Automatic Evaluation of Ontologies - Natural Language Processing and Text Mining - page 192

• Hierarchy or taxonomy. An ontology typically includes a hierarchical is-a or sub-

sumption relation between concepts. Although various other relations between
concepts may be also defined, the is-a relationship is often particularly important
and may be the focus of specific evaluation efforts.

• Other semantic relations. The ontology may contain other relations besides is-a,

and these relations may be evaluated separately. This typically includes measures
such as precision and recall.

• Context level. (1) An ontology may be part of a larger collection of ontologies, and

may reference or be referenced by various definitions in these other ontologies. In
this case it may be important to take this context into account when evaluating
it [26, 3, 21]. (2) Another form of context is the application where the ontology
is to be used; basically, rather than evaluate the ontology per se, it may be more
practical to evaluate it within the context of a particular application, and to
see how the results of the application are affected by the use of the ontology in
question. Instead of focusing on an individual application, one may also focus
on evaluation from the point of view of the individual users or the organization
(e.g., company) that will use the ontology [7].

• Syntactic level. Evaluation on this level may be of particular interest for on-

tologies that have been mostly constructed manually. The ontology is usually
described in a particular formal language and must match the syntactic require-
ments of that language (use of the correct keywords, etc.). Various other syntactic
considerations, such as the presence of natural-language documentation, avoid-
ing loops between definitions, etc., may also be considered [8]. Of all aspects of
ontology evaluation, this is probably the one that lends itself the most easily to
automated processing.

• Structure, architecture, design. Unlike the first three levels on this list, which

focus on the actual sets of concepts, instances, relations, etc. involved in the
ontology, this level focuses on higher-level design decisions that were used dur-
ing the development of the ontology. This is primarily of interest in manually
constructed ontologies. Assuming that some kind of design principles or crite-
ria have been agreed upon prior to constructing the ontology, evaluation on
this level means checking to what extent the resulting ontology matches those
criteria. Structural concerns involve the organization of the ontology and its
suitability for further development (e.g., addition of new concepts, modification
or removal of old ones) [8, 9]. For some applications, it is also important that
the formal definitions and statements of the ontology are accompanied by ap-
propriate natural-language documentation, which must be meaningful, coherent,
up-to-date and consistent with the formal definitions, su ciently detailed, etc.
Evaluation of these qualities on this level must usually be done largely or even
entirely manually by people such as ontological engineers and domain experts.

The following table summarizes which approaches from the list at the beginning of
Section 11.2 are commonly used for which of the levels discussed in this subsection.
The next few subsections will present more details about the various approaches
and levels of ontology evaluation.