Automatic Evaluation of Ontologies - Natural Language Processing and Text Mining - page 191

The remainder of this chapter is structured as follows. In Section 11.2, we present
an overview of related work on ontology evaluation. We describe the main approaches
to ontology evaluation and show different techniques that are used to evaluate differ-
ent aspects or levels of an ontology. In Section 11.3, we refer to a formal framework
for defining an ontology and show how various aspects of evaluation can be incorpo-
rated into such a framework. In Section 11.4, we present our approach to evaluating
a hierarchic ontology by comparing it to a gold standard. In Section 11.5, we present
experiments that explore how our evaluation measure responds to various modifi-
cations in the case of a large real-world topic ontology. Finally, in Section 11.6, we
present some guidelines for future work.

11.2 Survey of Ontology Evaluation Approaches

Various approaches to the evaluation of ontologies have been considered in the lit-
erature, depending on what kind of ontologies are being evaluated and for what
purpose. Broadly speaking, most evaluation approaches fall into one of the following
categories:
• approaches based on comparing the ontology to a gold standard (which may
itself be an ontology; e.g., [16]);
• approaches based on using the ontology in an application and evaluating the
results (e.g., [22]);
• approaches involving comparisons with a source of data (e.g., a collection of
documents) about the domain that is to be covered by the ontology (e.g., [2]);
• approaches where evaluation is done by humans who try to assess how well the
ontology meets a set of predefined criteria, standards, requirements, etc. (e.g.,
[15]).
In addition to the above categories of evaluation, we can group the ontology eval-
uation approaches based on the level of evaluation, as described in the following
subsections.

11.2.1 Ontology Evaluation at Different Levels

An ontology is a fairly complex structure and it is often more practical to focus on the
evaluation of different levels of the ontology separately rather than trying to directly
evaluate the ontology as a whole. This is particularly true if the emphasis is on
having the evaluation proceed automatically rather than being entirely carried out
by human users/experts. Another reason for the level-based approach is that when
automatic learning techniques have been used in the construction of the ontology, the
techniques involved are substantially different for the different levels. The individual
levels have been defined variously by different authors (e.g., [8, 9, 3, 22, 6]), but these
various definitions tend to be broadly similar and usually involve the following levels:
• Lexical, vocabulary, or data layer. Here the focus is on which concepts, instances,

facts, etc. have been included in the ontology, and the vocabulary used to repre-
sent or identify these concepts. Evaluation on this level tends to involve compar-
isons with various sources of data concerning the problem domain (e.g., domain-
specific text corpora), as well as techniques such as string similarity measures
(e.g., edit distance).