In Silico Hypothesis Discovery - Translational Informatics: Realizing the Promise of Knowledge-Driven Healthcare - page 136

Table 8.1 Differentiation of types of agreement in multi-expert KA studies. In this model, the use
of the “same” nomenclature or distinctions refers to the sources or experts using semantically
similar or compatible means of describing or classifying concepts in a domain. Similarly, the use
of “different” nomenclature or distinctions refers to the sources or experts using semantically
dissimilar or incompatible means of describing or classifying concepts in a domain
which possible relationships between entities are enumerated or otherwise defi ned
within the underlying knowledge collections. The logical, or axiomatic consis-
tency of the relationships that comprise a hypothesis is often used as a measure
of the accuracy of the output of the agent, again as defi ned by the correspondence
of axioms that may be derived from the source knowledge collection(s) with
the hierarchical and semantic assertions that make up such conceptual knowl-
edge. Finally, multiple-source or expert agreement is most commonly used to
validate the utility or impact of the output of the intelligent agent in “real world”
application-oriented scenarios. This later set of measures is a critical criterion
when attempting to measure the likely utility or impact of results generated by an
intelligent agent. Unfortunately, there is not a single approach for measuring mul-
tiple-source, or expert agreement - since most evaluation methods corresponding
to this type of metric involve the engagement of multiple (human) subject matter
experts (SMEs). Instead, metrics must be chosen based upon variables such as
data type as well as the number and types of knowledge sources being used. Most
importantly, such analyses must be formulated in a manner consistent with the
relative importance of four different types of agreement: (1) consensus; (2) cor-
respondence; (3) confl ict; and (4) contrast. Defi nitions of each of these types of
agreement are provided in Table 8.1 . A detailed discussion of the techniques that
may be applied to measure agreement can be found in the reviews provided by
Hripcsak et al. [ 30 , 31 ].
At the highest level, the specifi c methods that can be used to satisfy the types of
evaluation measures introduced above can be organized into a taxonomy consist-
ing of the following major categories: heuristic, quantitative, information theoretic,
graph theoretic and logical (Fig. 8.10 ). Brief descriptions of the techniques included
in each category are provided below: