Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 184

Self-Test 6.5
1. Each (of two) objects-by-observations matrixes would have 111 rows
(for cases as objects) and three columns (for judges as observations).
One matrix would be generated for actual care cases and the other for
TraumAID's recommendations.
2. There is no compelling evidence for rater tendency errors. The mean
ratings of the judges are roughly equal and near the middle of the scale.
Central tendency effects can be ruled out because the standard devia-
tions of the ratings are substantial.
3. From a reliability standpoint, the ratings are more than adequate.
However, the validity of the ratings must be questioned because the
judges are from the institution where TraumAID was developed.
4. The data seem to suggest that TraumAID's advice is accurate, as the
judges preferred how TraumAID would have treated the patients over
how the patients were actually treated. However, the concern about
validity of the ratings would cast some doubt on this conclusion.
Self-Test 6.6
Item 1: Accuracy should be defined. The response categories should be
replaced by alternatives that are more behavioral or observable.
Item 2: Ten response options are too many. The respondent needs to know
whether 1 or 10 corresponds to a high level of satisfaction. The numeri-
cal response options have no verbal descriptors.
Item 3: “No opinion” does not belong on the response continuum. Having
no opinion is different from having an opinion that happens to be midway
between “strongly agree” and “strongly disagree.”
Item 4: The logic of the response options does not match the stem.
There are not enough response options, and they are not well spaced
semantically.
References
1. Isaac S, Michael WB. Handbook in Research and Evaluation. San Diego: EdITS,
1995.
2. Cureton EE, D'Agostino RB. Factor Analysis, an Applied Approach. Hillsdale,
NJ: Lawrence Erlbaum, 1983.
3. Kim J, Mueller CW. Factor Analysis: What It Is and How to Do It. Newbury
Park, CA: Sage, 1978.
4. Brennan RL, Fienberg S, Liveseley D, Rolph J. Generalizability Theory.
Newbury Park, CA: Sage, 2001.
5. Shavelson RJ, Webb NM. Generalizability Theory: A Primer. Newbury Park,
CA: Sage, 1991.
6. Anderson JG, Jay SJ, Schweer HM, et al. Why doctors don't use computers: some
empirical findings. J R Soc Med 1986;79:142-144.