Automatic Evaluation of Ontologies - Natural Language Processing and Text Mining - page 198

(e.g., tenders, grant applications, etc.). It could be particularly useful in situations
where we are faced with a considerable number of ontologies roughly relevant to
our domain in interest and wish to select the best ontology (or a few good ones).
However, this type of approaches may still have di culties such as the need for
much manual involvement by human experts, for the presence of a gold standard
ontology, etc. In effect, the general problem of ontology evaluation has been deferred
or relegated to the question of how to evaluate the ontology with respect to the
individual evaluation criteria.
Burton-Jones et al. [3] propose an approach of this type, with 10 simple criteria
such as syntactical correctness, clarity of vocabulary, etc. (a brief description of the
way used to compute a numeric score for each attribute is included in parentheses):

• lawfulness (i.e., frequency of syntactical errors),

• richness (how many of the syntactic features available in the formal language

are actually used by the ontology),

• interpretability (do the terms used in the ontology also appear in WordNet?),

• consistency (how many concepts in the ontology are involved in inconsistencies),

• clarity (do the terms used in the ontology have many senses in WordNet?),

• comprehensiveness (number of concepts in the ontology, relative to the average

for the entire library of ontologies),

• accuracy (percentage of false statements in the ontology),

• relevance (number of statements that involve syntactic features marked as useful

or acceptable to the user/agent),

• authority (how many other ontologies use concepts from this ontology),

• history (how many accesses to this ontology have been made, relative to other

ontologies in the library/repository).

As can be seen from this list, this methodology involves criteria from most of the
levels discussed in Section 11.2.1. A downside of this approach is that there is little
in it to help us ascertain to what extent the ontology matches the real-world state of
the problem domain to which is refers (or indeed if it really deals with the domain we
are interested in; it could be about some entirely unrelated subject; but this problem
can be at least partially addressed by text categorization techniques, as used, e.g.,
in [21]). The accuracy criterion in the list above provides a way to take the accuracy
into account when computing the overall ontology score, but it's usually di cult
to compute the percentage of false statements otherwise than by examining them
all manually. On the positive side, the other criteria listed above can be computed
automatically (although some of them assume that the ontology under consideration
belongs to a larger library or repository of ontologies, and that metadata such as
access history is available for the repository). Fox et al. [7] propose another set of
criteria, which is however geared more towards manual assessment and evaluation of
ontologies. Their criteria involve: functional completeness (does the ontology contain
enough information for the application at hand?), generality (is it general enough
to be shared by multiple users, departments, etc.?), e ciency (does the ontology
support e cient reasoning?), perspicuity (is it understandable to the users?), preci-
sion/granularity (does it support multiple levels of abstraction/detail?), minimality
(does it contain only as many concepts as necessary?).
An even more detailed set of 117 criteria is described in [15], organized in a three-
level framework. The criteria cover various aspects of the formal language used to
describe the ontology, the contents of the ontology (concepts, relations, taxonomy,