Enhancing Cognitive Screening in Geriatric Care: Use of an Internet-Based System

ABSTRACT

Cognitive screening measures for age-related cognitive impairment have been found to have only fair validity, and the risks of harm even may outweigh the benefits at this time (U.S. Preventative Service Task Force, 2003). A large-scale project designed to assess elder care in Primary Care Physician offices noted that dementia evaluation and treatment was one of the most overlooked aspects of care. Taken together, these studies cited the lack of time and technical expertise in test administration as the most prominent barriers to the accurate detection of dementia in Primary Care Physician offices. It was for these reasons that the Cognitive Screening Test (CST) was created. The CST requires no physician time or training to administer or interpret it. The current study investigated the clinical utility of this cognitive screening system by comparing the results of 102 patients to those of expert geriatricians, using consensus conference methods for diagnosis. Overall clinical utility demonstrated scores at .80 or above for sensitivity, specificity, and positive and negative predictive power. In contrast, the MMSE had only a .38 sensitivity. A Receiver Operating Curve (ROC) analysis indicated a .863 accuracy rating for the predetermined cut score on the CST.

Validation of a dementia screening test in geriatric practice

The growing prevalence of dementia makes it imperative that Primary Care Physicians become more able and more comfortable to diagnose dementia, particularly in its earlier stages. Early diagnosis gives physicians the opportunity to initiate treatment with acetylcholinesterase inhibitors, which have shown to delay cognitive and functional declines (Mohs et al., 2001) and to delay nursing home placement (Schneider, 2000; Turner, 2003), all the while producing infrequent, temporary side effects (Boise, Morgan, Kaye,& Camicioli, 1999). It allows patients and families to plan for future care needs, financial needs, and legal needs associated with a lengthy, debilitating illness (Boise et al., 1999; Knopman, Donohue,& Gutterman, 2000), and families and patients may be given insight into the behavioral and personality changes that may occur and the safety precautions that eventually may be necessary (Knopman, et al., 2000; Sano & Weber, 2003). On the whole, primary care physicians continue to have difficulty detecting the majority of dementia cases among their patients. In previous studies, less than 30% of patients have been correctly identified as cognitively impaired, including even those with poor cognitive performance (Boise, Neal, & Kaye, 2004; Olafsdottir, Skoog, & Marcusson, 2000; Valcour, Masaki, & Blanchette, 2002). In an effort to increase the detection of cognitive impairment and generally to increase the quality of patient care, variables have been identified that impede this process. In a review, Reuben, Roth, Kamberg, and Wenger (2003) identified physician time constraints and a lack of physician technical expertise with cognitive screening administration and interpretation as major factors that impede effective dementia screening. Additionally, compounding the issue is that the standard primary care model of medical practice relies heavily on the patient’s symptoms report. However, lack of awareness of symptoms is a common feature of Alzheimer’s disease; thus, even when queried directly, patients often deny or minimize cognitive problems (Zanetti et al., 1999).

Conducting and interpreting brief cognitive screening is problematic in Primary Care Practices, even in practices where screening tests such as the Mini-Mental Status Examination (MMSE) are used. It is well established that the MMSE is sensitive to the effects of education and age (Anthony, LeResche, Niaz, Von Korff, & Folstein, 1982; Crum, Anthony, Bassett, & Folstein, 1993; Monsch et al., 1995; Spreen & Strauss, 1998; Tombaugh & McIntyre, 1992; Tombaugh, McDowell, Krisjansson, & Hubbley, 1996; Uhlmann & Larson, 1991). As a result of these findings, normative data have been published that make such considerations (Crum et al., 1993; Tombaugh et al., 1996). However, interpretation that considers age and education rarely occurs in clinical practice. More often than not, single-cut scores are used as a basis to distinguish the impaired from intact patients; for example, a cut score of 23/24 on the MMSE has been suggested to indicate cognitive impairment (Folstein, Folstein, & McHugh, 1975). This cut score also has been found appropriate in population-based studies (Ganguli et al., 1993). However, depending on the age and educational characteristics of the patient, this cut score consistently has demonstrated low sensitivity or specificity (Monsch, et al., 1995; Spreen & Strauss, 1998; Tombaugh & McIntyre, 1996). Due to these test constraints, cognitive screening measures for age-related cognitive impairment have been found to have only fair validity, and the risks of harm even may outweigh the benefits at this time (U.S. Preventative Service Task Force, 2003).

Ideally, a screening device for dementia will contain many memory items, since this is the domain of cognition typically impaired; it will require little in the way of administration and interpretation by those not trained in psycho-metrics and will be based on normative data that are meaningful to clinical interpretation. In response to both the problematic characteristics of traditional screening measures and the qualities desired in screening devices, a computerized screening battery was created. The Cognitive Screening Test (CST), an Internet-based program, is easily administered and requires no interpretation by the administrator. Conceptually, CST may provide a superior method of screening for cognitive impairment, because it instantly can take into consideration the factors that have shown to significantly affect screening interpretation in order to maximize sensitivity and specificity. Thus, even given equal qualities of the test instrument itself, an Internet-based system has the advantage of being able to utilize corrections for variables such as age and education in sophisticated ways. The purpose of this is to present preliminary data analyses using the CST system in order to compare its validity to other traditional screening devices and to demonstrate other qualities of the CST that lend to its incremental validity over other devices.

Two hypotheses were proposed for the study:

1. The CST will demonstrate a high level of clinical agreement (over .8 on combined indices of sensitivity, specificity, and positive and negative predictive power) with expert physician consensus diagnoses. 2. The CST will demonstrate superior clinical utility data than will the MMSE using a single cut score.

methods

Participants and clinic Procedures

The sample consisted of 102 patients from a specialized geriatric clinic; characteristics of the sample are listed in Table 1. The clinic is located in a suburban setting and typically attracts European American patients who have above-average educational levels. The attending physicians at this clinic are all geriatric fellow trained and board certified. The only patients who were not approached about the study were those who had significant motoric, visual, or cognitive impairments that prevent them from using a computer or understanding directions, and individuals who were not fluent in English.

There are two groups of patients at this clinic, both of which were represented in this study. The first includes those who have established the clinic for primary medical care; the second are individuals who have been referred to the geriatric memory clinic by their own physicians in order to address specific geriatric issues. Evaluation of all patients is always very thorough. When first admitted to the clinic, there is a comprehensive review of symptoms and a physical examination, including a detailed neurological examination. This would be repeated for primary care patients if cognitive status were in question. Medication review and an informant interview are conducted. Medical, family, social, and occupational histories are taken. Functional assessment, laboratory tests, and neuroimaging are part of the clinical workup for cases of suspected dementia. Included in the screening of patients who are screened for cognitive decline is the Mini-Mental State Examination.

Table 1. Sample demographics

measures

CST Normative sample and scoring Rules

The CST norms and cut scores were derived from normative data collected completely independently of the current study. The characteristics of the normative data were a sample size of 105 with a mean age of 73 (s.d.=7) and a mean educational level of 12 years .Seventy percent of the normative sample was made up of non-Hispanic whites, and 30% was minority elders (primarily African Americans) .The normative data were employed in developing the cut scores as follows. Multiple regression analyses were conducted for each subtest of interest (learning total, memory total, executive function—total errors and mean response time), with subtest performance as the dependent variable and many potentially significant determinants added as predictors in the equation (e.g., health history variables, demographics, etc.). Of all of the variables included, age, ethnicity, education, and gender alone significantly predicted subtest performance.

Using the normative data beta weights and constants, the significant demographic variables were plugged into the general linear model (a+bx) to derive an expected score for each individual in the current sample. This expected score then was subtracted from the individual’s raw score, which was divided by the standard error of the estimate to produce a z-score that describes the patient’s deviation from the mean score. An Impairment Index then was derived for each subject as follows: z scores were converted to Impairment Index scores by summing subtest results according to the following: z scores > -2.0 were assigned a subtest score of 0; z scores > -2.5 (but less than -2.0) were assigned a subtest score of .5; z scores < -2.5 were assigned a subtest score of 1. A total score of 1.5 or more (summed across the three subtests) indicates a high probability of cognitive impairment and represents the cut score for impairment in these analyses. See Table 2 for a visual illustration of the translation of z-scores into this impairment index. In summary, the analyses from the cases reported here provide a cross validation of the equations obtained from the normative derivation sample.

Table 2. Translation of z-scores into the cognitive impairment index

cognitive screening Test (CST)

The CST is composed of several subtests. The following describes each one and explains the rationale for its development. Total administration time is approximately 15 minutes.

• Keyboard Proficiency Subtests (2 minutes). The first of these two subtests consists of a green ball presented on the screen. The participant is instructed to press the spacebar repeatedly until the green ball turns red. The second involves a series of numbers appearing on the screen; the participant is instructed to press the number key that corresponds with the number shown. These subtests are administered solely for the purpose of acclimating the patients to the nature of the tests.

• Learning and Memory Subtests (7-8 minutes). Individuals learn the placement of nine household objects placed in a virtual cabinet. In the course of three learning trials, they are quizzed about the location of each object. The number of correct responses summed over all three trials is referred to as the Learning subtest. After a delay involving the interpolation of the Executive Function subtest, a final inquiry is made about the location of the objects. This final recall is referred to as the Memory subtest. See Figure 1 for an illustration of this task.

• Executive Function Subtest (6-8 minutes). There are two components to this task. For the first, numbers from 0 to 9 are presented one at a time. Individuals are instructed to press the 0 or 1 key when the respective number is presented. Test takers are asked not to respond to other numbers. The second part consists of the numbers again being presented, but the participants are instructed to press the 0 key in response to the appearance of a 1 and to press the 1 key in response to the appearance of a 0. Again, subjects are told not to respond to numbers other than 0 or 1. The second part of this test taps executive function, with its emphasis on self-monitoring, and response inhibition and flexibility (Lezak, 1995). See Figure 2 for an illustration of the Executive Function subtest. Both the number of errors and the mean response time for the second part of this task are used to compute the CST decision.

Procedure

Recruitment and CST Administration

During medical appointments, patients were approached by their physicians about volunteering for the study. Those who agreed to participate then were provided informed consents and ad ministered the CST protocol by a graduate student in psychology. Administration typically took 15 minutes. A double-blind procedure was employed, as physicians did not disclose participant diagnoses to the CST administrator, nor did the administrator ever inform physicians about their performance. All participants were administered the CST (n=102).

Figure 1. Illustration of the “memory cabinet” used in the memory and learning subtests of the CST

Figure 2. The “executive function” subtest of the CST; this figure illustrates what patients view and which responses are appropriate in each task

Part 1 of Executive Function Subtest

consensus Diagnosis conference

Consensus diagnosis conferences by the attending physicians were conducted, during which clinical impressions, lab record reviews, and the MMSE scores were utilized in order to determine diagnoses; physicians were blinded to the patients’ performances on the CST. DSM-IV criteria were utilized for diagnosis of dementia (DSM-IV, 1994); those who were diagnosed with Mild Cognitive Impairment (MCI) met Petersen’s consensus criteria (Petersen et al., 1999). Using these criteria and methods of assessment, 35.3% of the sample was determined to be demented, 10.8% to have MCI, and 53.9% to be cognitively intact.

statistical Analyses

In order to determine the concordance between all of the CST and the consensus diagnoses, scores above and below the cut points were plotted, and accuracy was computed manually.

Part 2 of Executive Function Subtest

results

The demographic characteristics of this sample are presented in Table 1. The final sample of 102 participants administered the CST was examined. An overall 83% concordance was found between the geriatrician consensus diagnosis and the CST predetermined cut score (see previous section titled CST Scoring and Decision Rules). Specifically, there was 80% sensitivity, 87% specificity, 88% positive predictive power (ppp), and 79% negative predictive power (npp) (see Table 3). In order to determine further whether the predetermined cut score (Impairment Index of 1.5 summed across subtests) was the most accurate in predicting cognitive status, a receiver operating characteristic (ROC) curve was examined. The predetermined cut score that had been employed did, in fact, produce the highest overall accuracy among the range of possible scores. The area under the curve (AUC) value for all possible accuracy values as determined by the ROC curve was .861 (SE=.039) (see Figure 3). Thus, there was support for hypothesis 1: clinical accuracy of the CST.

In order to compare the clinical utility of the Cognitive Screening Test to another screening test that utilized a single cut score, comparisons were made between the CST and the Mini Mental State Examination (MMSE). Using the single cut-score method, sensitivity of 37.7%, specificity of 97.6%, ppp of 95.2%, and npp of 55.4% were found for the MMSE. Taken together, these indices demonstrate that a single cut score in this clinic population leads to an unacceptable number of false negatives and very poor sensitivity.

Table 3. Screening device accuracy

Figure 3. Receiver operating characteristic (ROC) curve for the cognitive impairment index of the CST; Area under the curve (AUC)=.861 (SE=.039), p<.001

Table 4. Mean scores on cognitive tests by impairment status

Note. *Signifies that the intact and MCI groups are significantly different.Signifies that the intact and demented groups are significantly different. No tests were significantly different between the MCI and demented groups.

Figure 4. Mean IADL scores plotted as a function of the cognitive impairment index scores

Figure 5. ROC curve for MMSE scores. AUC=.764 (SE=.049), p<.001

In order to better determine the ability of the CST and MMSE to detect early dementia, an analysis of variance (ANOVA) function was performed among those diagnosed as demented, those diagnosed with Mild Cognitive Impairment (MCI), and those considered cognitively intact by the geriatricians. Significantly different scores were found between those considered intact and those considered demented by all tests. The only test that was significantly different among those who were intact and those with MCI was the CST. A follow-up analysis was conducted among only the MCI patients to determine the effectiveness of the different screens in detecting early dementia. Of the 11 MCI patients, nine (81.8%) were correctly identified by the CST as likely impaired, whereas only three (27.3%) of these were correctly identified by the MMSE.

discussion

This study demonstrated that the Cognitive Screening Test was administered easily in a physician’s clinic setting and had a high rate of concordance with expert geriatrician diagnosis with an overall hit rate of 83%. The exciting implications of these data are related to the use of technology for cognitive screening in a way that reduces clinician burden (i.e., time and expertise) while improving accuracy of the screening tool. As already stated, physicians and other healthcare professionals do not need to be concerned with administering the measure or interpreting results. Studies have reported that physicians may choose not to take part in increased care innovations if they do not feel that they possess the skills to do it correctly (Reuben et al., 2003); thus, a procedure that does not require physicians to learn new skills (i.e., cognitive testing) may more likely be utilized. Additionally, the CST does not take significantly more time than other pencil-and-paper measures and can be administered with little help by a medical technician or other person who can navigate to the Web site and respond to any questions or anxieties of the patient. This translates to little time investment on the part of the physician. Finally, there is no additional equipment to purchase, as the entire system will be available on the Internet. A standard CPU, keyboard, modem, and monitor are used.

There are limitations to this model. The CST is not a diagnostic tool and should not be used for diagnostic purposes. In order to establish a true cognitive impairment, follow-up assessment should be thorough, and diagnoses should use accepted clinical criteria. It is important to note that the major thrust of this study was not to compare the quality of the CST and the MMSE. All cognitive screening tests are affected by variables such as age, education quantity, and education quality. The advantage of the CST is that the inclusion of these variables in a sophisticated way can be brought into the interpretation of a given patient’s single test score. This ability is simply not possible with any paper or pencil test, unless the test interpreter is very sophisticated in his or her approach to interpretation. Given these advantages of the CST, it was not surprising that the clinical utility of the CST exceeded that of the MMSE when using a single cut score.

Figure 6. ROC curve for CLOX1 and CLOX2 scores. .CLOX1 AUC=.683 (SE=.060), p=.005; CLOX2 AUC=.707(SE=.058), p=.002