Information Technology Reference
In-Depth Information
Verification & Validation Metrics
Heuristic
Quantitative
Information Theoretic
Graph Theoretic
Logic (Consistency)
Fig. 8.10
Taxonomy of verifi cation and validation metrics for the results generated by intelligent
agents that leverage conceptual knowledge collections
8.4.1
Heuristic Methods
Heuristic metrics are probably the most common approach to verifying or validat-
ing the output of intelligent agents such as in silico hypothesis discovery tools. In
this case, we use the term heuristic to refer to “rules of thumb” or more formally,
rules that are informed by the expertise or commonly held knowledge of human
SMEs. The advantages of using heuristics are the ability to incorporate domain-
specifi c knowledge or conventions, and their simplicity (i.e., knowledge engineers
or experts manually review the knowledge collection to determine if the contents
are consistent with the heuristics). However, since such measures are diffi cult to
automate or scale to larger data sets, such heuristic techniques are limited in their
tractability when applied to “big data” contexts. Furthermore, heuristically compar-
ing “quality” across multiple hypotheses or underlying knowledge collections is
diffi cult, as a result of the relative and qualitative nature of the evaluation. Specifi c
heuristic criteria for verifying or validating the output of intelligent agents have
previously been proposed by Gruber [
32
] and include the following factors:
Clarity
Coherence
Extendibility
Minimal encoding bias
Minimal deviation from ontological commitment, where ontological commit-
ment refers to the situation were all observable actions of a knowledge-based
system utilizing the given ontology are consistent with the relationships and
defi nitions contained within that ontology.
8.4.2
Quantitative Methods
Quantitative methods of evaluating the results generated by intelligent agents are
best suited for measuring both multi-source agreement and the degree of interre-
latedness of ensuing hypotheses. Such measures can include simple statistics such
as the precision, accuracy and chance-corrected agreement of the multiple sources
Search WWH ::
Custom Search