Automatic Evaluation of Ontologies - Natural Language Processing and Text Mining - page 197

has several drawbacks: (1) it allows one to argue that the ontology is good or bad
when used in a particular way for a particular task, but it's di cult to generalize
this observation (what if the ontology is used for a different task, or differently for
the same task?); (2) the evaluation may be sensitive in the sense that the ontology
could be only a small component of the application and its effect on the outcome
may be relatively small (or depend considerably on the behavior of the other com-
ponents); (3) if evaluating a large number of ontologies, they must be su ciently
compatible that the application can use them all (or the application must be suf-
ficiently flexible), e.g., as regarding the format in which the ontology is described,
the presence and names of semantic relations, etc. If it is necessary to adapt the
application somewhat for each ontology that is to be evaluated, this approach to
evaluation can quickly become very costly.
11.2.6 Data-Driven Evaluation
An ontology may also be evaluated by comparing it to existing data (usually a
collection of textual documents) about the problem domain to which the ontology
refers. For example, Patel et al. [21] proposed an approach to determine if the on-
tology refers to a particular topic, and to classify the ontology into a directory of
topics: one can extract textual data from the ontology (such as names of concepts
and relations, or other suitable natural-language strings) and use this as the input
to a text classification model. The model itself can be trained by some of the stan-
dard machine learning algorithms from the area of text classification; a corpus of
documents on a given subject can be used as the input to the learning algorithm.
Another data-driven approach has been proposed by Brewster et al. [2]. First, a
set of relevant domain-specific terms are extracted from the corpus of documents, for
example using latent semantic analysis. The amount of overlap between the domain-
specific terms and the terms appearing in the ontology (e.g., as names of concepts)
can then be used to measure the fit between the ontology and the corpus. Measures
such as precision or recall could also be used in this context.
In the case of more extensive and sophisticated ontologies that incorporate a lot
of factual information (such as Cyc, see, e.g., www.cyc.com), the corpus of docu-
ments could also be used as a source of “facts” about the external world, and the
evaluation measure is the percentage of these facts that can also be derived from
information in the ontology.
11.2.7 Multiple-Criteria Approaches
Another family of approaches to ontology evaluation deals with the problem of
selecting a good ontology (or a small short-list of promising ontologies) from a given
set of ontologies, and treats this problem as essentially a decision-making problem.
Therefore, techniques familiar from the area of decision support systems can be used
to help us evaluate the ontologies and choose one of them. Usually, these approaches
are based on defining several decision criteria or attributes; for each criterion, the
ontology is evaluated and given a numerical score. Additionally a weight is also
assigned (in advance) to each criterion, and an overall score for the ontology is then
computed as a weighted sum of its per-criterion scores. This approach is analogous to
the strategies used in many other contexts to select the best candidate out of many