Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 181

FIGURE 6.8. Possible design for blinded mutual audit.
Mutual Audit
With the blinded mutual audit design, each judge reviews data from some
fraction of the objects (usually test cases), giving his or her response. Data
from each object are reviewed by at least two judges. Each opinion then is
graded for correctness either by the same group of judges 28 or by a second
group, 29 without knowing who originally provided it. This technique allows
evaluators to calculate the inter- and intrajudge agreement (and thus error)
rates and to ensure efficient use of the judges' time. Although we used a
simplified version of it in the exercise in Chapter 5, the study of Hypercritic
employed a variant of the mutual audit technique.
The mutual audit can be used in a pure measurement study, or, as shown
in Figure 6.8, it can be employed in a hybrid measurement/demonstration
study where a set of test cases is reviewed by a set of expert judges, an infor-
mation resource, and the persons who provided the actual care on these
cases.
Delphi Technique
The Delphi technique, as it applies to developing a consensus judgment of
some attribute of a clinical case, is illustrated in Figure 6.9. Each judge
reviews the data from all cases independently and records his or her opinion
on a report form. The forms are passed to a moderator, who extracts the
consensus opinion for each case and returns them and the case data to the
judges for a second opinion, usually without informing them of their