Automatic Evaluation of Ontologies - Natural Language Processing and Text Mining - page 200

document title, description, URL, and the main block of the text (for practical
purposes, such text is often represented as a vector using, e.g., the TF-IDF
weighting under the vector space model of text representation).
• Data-model ontologies where concepts are tables in a data base and instances
are data records (such as in a database schema). In this setting, datatypes and
attributes in the above-mentioned formal definition of an ontology are straight-
forward analogies to the types and attributes (a.k.a. fields or columns) in a data
base management system.
Evaluation can be incorporated in this theoretical framework as a function that
maps the ontology O to a real number, e.g., in the range [0 , 1]. However, as has
been seen in Section 11.2, a more practical approach is to focus the evaluation on
individual components of the ontology O (which correspond roughly to the levels of
ontology evaluation discussed in Section 11.2). Results of the evaluation of individual
components can later be aggregated into a combined ontology evaluation score [6].
• The datatypes and their values (i.e., T , V , ≤ T , and ι T ) would typically not be

evaluated; they are merely the groundwork on which the rest of the structure
can stand.

• A lexical- or concept-level evaluation can focus on C , I , ι C , and possibly some

instance attributes from ι A .

• Evaluation of the concept hierarchy (is-a relationship) would focus on the ≤ C

partial order.

• Evaluation of other semantic relations would focus on R , ι R , and the concept

and instance attributes.

• One could also envision evaluation focusing on particular attributes; for example,

whether a suitable natural-language name has been chosen for each concept. This
kind of evaluation would take ι C and the attributes as input and assess whether
the concept attributes are suitable given ι C and the instance attributes.

• Application- or task-based evaluation could be formalized by defining the appli-

cation as a function A ( D, O ) which produces some output given its input data D
and the ontology O . By fixing the input data D , any evaluation function defined
on the outputs of A becomes de facto an evaluation function on O . However, the
practical applicability of such a formalization is debatable.

• Evaluation based on comparison to a gold standard can be incorporated into this

theoretical framework as a function defined on a pair of ontologies (effectively
a kind of similarity measure, or a distance function between ontologies). Simi-
larly, data-driven evaluation can be seen as a function of the ontology and the
domain-specific data corpus D , and could even be formulated probabilistically
as P ( O|D ).

11.4 Architecture and Approach

We have developed an approach to ontology evaluation primarily geared to enable
automatic evaluation of an ontology that includes instances of the ontology con-
cepts. The approach is based on the gold standard paradigm and its main focus is
to compare how well the given ontology resembles the gold standard in the arrange-
ment of instances into concepts and the hierarchical arrangement of the concepts
themselves. It is similar to the other existing ontology evaluation methods based on