abstract
One in nine adults in the United States has chronic kidney disease (CKD). Randomized studies show electronic health (e-health) systems improve health outcomes in chronic disease. This study describes a systematic evaluation of available CKD Web sites. We evaluated Web sites for educating patients with CKD, focusing on three specific design issues: usability, quality, and content. Thirty Web sites were evaluated between April and July 2004. Cohen’s kappa and intraclass correlation of quartile rankings for two independent evaluators were calculated. Mean score for evaluator 1 was 7.293 (standard error 0.511) and for evaluator 2 was 8.189 (0.413). Cohen’s kappa for the 2 evaluators’ total scores for all 30 Web sites is 0.7671, and intraclass correlation is 0.7703. In general, results show that it is possible to identify Web sites more likely to provide a positive educational experience for CKD patients. Further evaluation is needed to investigate the utility of variably ranked Web sites as educational interventions.
Introduction
Medical informatics has traditionally consisted of applications to improve information exchange, decision-making, and time management for health care professionals. As the fundamental relationship between patients and physicians has changed from paternalistic to partnership, the informatics focus is shifting towards the consumer (Eysen-bach, 2000). The objectives of this transition are to empower patients with appropriate medical knowledge, education, and the tools for self-management. This may be particularly effective in enhancing patient and health care provider partnerships in the management of chronic disease (Celler, & Lovell, 2003). According to the Pew Research Center, a nonpartisan organization that provides information on national and global trends, 70% of the health care consumers, who are influenced by online information, report that the Internet affected their decision making about health care (Fox, & Rainie, 2002). Still, very little is known about how patients judge health information obtained on the Internet. Moreover, data suggest that while health care consumers are facile at obtaining answers to directed health questions, they seldom assess the quality or source of the information. Consumers also tend not to remember Web sites from where they acquired the information (Eysenbach & Kohler, 2002). Thus, health care provider-sanctioned Web instruction is needed to help alert and direct patients to information that is valid and accurate.
One in nine adults in the United States has chronic kidney disease (CKD) (Coresh, Astor, & et al. 2003). This creates a tremendous public health burden, with more than $22 billion in spending on the end-stage renal disease (ESRD) program in 2002 (USRDS, 2002). Slowing the rate of loss of kidney function in later stages of CKD by 1030% could save $5 to $30 billion dollars in health care expenditures over the next decade (Trivedi, & Pang, 2002). Prior to end-stage disease, there may be opportunities to slow progression and delay disease complications, but patient education is vital. Time constraints prevent clinicians from taking on the task of repetitive patient education and counselling. However, customized Web sites available via the Internet may be able to fill this need by providing novel ways to reach patients with CKD.
Randomized studies have found that e-health systems can improve knowledge, self-care, quality of life, coping skills, health care participation, and health outcomes in chronic disease, as well as self-management behaviors, including diet, exercise, medication taking, and health utilization (Gustafson, & Hawkins, 1999a; Gustafson, & McTavish, 1999b). However, very little is known about the selection of e-health Web sites and the assessment of medical information obtained by patients (Eysenbach & Kohler, 2002). There are a number of CKD patient education sites available on the World Wide Web, some of which are sponsored by not-for-profit organizations and some by private industry. Few are written at a level so as to be useful to the general population, and many do not comply with available standards for health Web sites (Calderon, & Zadshir, 2004a; Calderon, & Zadshir, 2004b; Jaffery, & Becker, 2004).
Additionally, as research in the field ofe-health expands, it is becoming increasingly clear that in order to capitalize on the potential power of the Internet as an educational tool, innovative methods for communicating with patients is needed. Marill, Miller, and Kitendaugh (2006) report on the ongoing challenges to the National Library of Medicine’s Medline Plus, highlighting the dynamic process inherent in health information Web site design.
Although more literature is emerging on optimizing human-computer interactions for specialty clinical areas and the education of chronic disease patients, to date, much of this has been led by the cancer and mental health arenas (Bader, & Strickman-Stein, 2003; Chelf, & Deshler, 2002; Chernecky, & Macklin, 2006; Chou, & Lin, 2004), with limited contribution in the nephrology literature. The purpose of this manuscript is to describe a systematic evaluation of readily available CKD Web sites. A more comprehensive review ofthese sites revealed significant deficits that are likely to impair their usefulness as educational tools.
methods
Based on our previous work, relevant literature in other disease states, and published guidelines for evaluation of e-health Web sites ( Kim, & Eng, 1999; Robinson, & Patrick, 1998; Winker, & Flanagin, 2000), a system was developed for evaluating Web sites aimed at educating patients with CKD. Three distinct Web-based design issues were addressed: usability, quality, and content.
1. Usability: Defined on the basis of the ease with which a user can access clinically relative material (ease of navigation) and readability (reading level), was assessed with the following tools:
a. Reading level, which was assessed using the Flesch-Kincaid grade level formula (Kincaid, Rogers, & Chissom, 1975).
b. Ease of navigation, which was assessed by the number of links needed to get to clinically relevant material. Clinically relevant material is defined as containing information directly related to aspects of CKD management most likely to improve clinical outcomes, as elaborated below. Number of links was defined as the number of mouse-clicks needed to arrive at such material from the home page.
2. Quality is a complex measure. Defining and assessing the quality ofhealth education Web sites remains an elusive goal (Gattoni & Si-cola, 2005). Various criteria have been used to evaluate the quality of health information on Web sites. These have arisen through expert consensus or arbitrary design. In this study, we defined quality specifically on the basis of adherence to those shared areas that these groups describe as essential when judging the quality of e-health sites (Kim et al., 1999; Robinson et al., 1998; Winker et al., 2000). In this sense, the opinions of area experts are key in judging the quality of relevant CKD Web sites.
Quality questions (Yes/No):
1. Authorship/source: Are the sources of health information mentioned with credentials listed?
2. Quality/best-available evidence: Is it clear whether information is based on scientific evidence (studies), expert consensus, or personal/professional experience/opinion?
3. Privacy: Is there a privacy policy?
4. Updated: Is the date of last modification listed?
5. Contact: Is a link to a valid contact form/email available?
6. Ownership/sponsorship: Are all sources of funding listed?
7. Advertising: Is there an advertising policy?
8. Editorial content: Is there an editorial policy/description of content review?
3. Content had no standard measures to evaluate what information is appropriate to convey to health care consumers visiting a CKD Web site. Evaluations of other e-health education sites for other disease entities have tended to focus on five to eight clinical areas ( Croft & Peterson 2002; Impicciatore, Pandolfini, 1997). We judged a CKD Web site’s content according to how accurately it addresses six areas. These are based on specific management goals according to the kidney disease outcome quality initiative (K/DOQI) (National Kidney Foundation K/DOQI, 2004), emphasizing aspects of CKD management most likely to improve clinical outcomes. At this point, we focus on each of the areas that are grouped under high blood pressure, CKD complications, and nutrition. a. High blood pressure. Hypertension is both a significant underlying cause and complication of CKD. Blood pressure control is recognized as among the most important factors for slowing the progression of CKD and is associated with other positive outcomes, including the prevention of cardiovascular disease.
Questions (Yes/Nd):
1. Hypertension: Are specific blood pressure goals addressed? b. Complications of CKD. Among the kidney’s functions is the production of two hormones, erythropoietin, a glycoprotein hormone that stimulates the production of red blood cells by stem cells in bone marrow, and 1,25 dihydroxy vitamin D3, a steroid compound necessary for normal bone growth. Consequently, two of the most common complications of CKD are anaemia and bone disease, or renal osteodystrophy.
Questions (Yes/No):
2. Anemia: Are target hemoglobin/hematocrit goals addressed?
3. Bone disease: Is vitamin D replacement therapy addressed?
c. Nutrition. Diet is one of the most modifiable factors that can affect the control of all of the above, thereby offering the potential to have profound effects on both CKD progression and morbidities. Therefore, our content tool evaluates whether CKD Web sites addresses some of the nutritional areas most relevant to CKD, namely sodium, phosphorus, and protein intake.
Questions (Yes/No):
4. Sodium: Are specific dietary sodium goals addressed?
5. Phosphorus: Is appropriate phosphate binder use addressed?
6. Protein: Are specific dietary protein intake goals addressed?
It is important to note that as the appropriateness of CKD Web site content is something that cannot be assumed by the Web site owner nor assessed by the consumer (CKD patient), there is the need for content validation by experts, irrespective of target audience. Therefore, in designing our study, two nephrologists were to perform the Web site evaluation, drawing on their expertise in assessing CKD Web site content. Other aspects of evaluation are objective including “reading level” and “number of clicks,” but it was vital that experts in nephrology assess the appropriateness of content.
Accordingly, the investigators also felt that it was crucial to attempt as exhaustive a search of CKD Web sites as possible, using multiple common search engines. For instrument testing and validation, however, the evaluation was led to exploring the National Kidney Disease Education Program Chronic Kidney Disease Compendium (NKDEP) (2004) for additional relevant Web sites to support instrument testing and validation.
Using five conventional search engines (Excite, Google, HotBot, Lycos, and Yahoo), we entered four search terms (kidney disease, chronic kidney disease, chronic kidney failure, and renal failure) and identified the first 40 uniform resource locators (URLs), the World Wide Web address of a site on the Internet, from each search engine. Of these, 110 were redundant, leaving 90 discrete sites. Twenty-one sites were directed toward the care of animals, 18 were obsolete or pertained to nondisease states, 19 were oriented toward providers rather than patients, and two were focused solely on end-stage renal disease. For the final evaluation, this left 30 Web sites designed for educating patients with CKD.
In the data analysis, we calculated quartile rankings for each of the three areas for two independent evaluators for all 30 Web sites using Microsoft 2000 Excel. Cohen’s kappa and intra-class correlation were calculated using the online statistics toolbox of the Department of Obstetrics and Gynaecology at the Chinese University of Hong Kong (Department of Obstetrics and Gynaecology 2005 [accessed 2005 March 23]).
Results
Preliminary content Validity and Reliability Testing
Initially, five Web sites not among those identified by the search described above were identified using the NKDEP (2004). Two nephrologists from University of Wisconsin-Madison evaluated these sites. Revisions were made based on these pilot evaluations, and this process was repeated with additional Web sites identified in the same manner, allowing for very good correlation among evaluators (r=0.925).
Specifically, revisions to the “usability” construct included the “ease of navigation” concept in addition to “readability.” During piloting, readability was assessed using both the Fry readability scale and the Flesch-Kincaid grade level formula. We found these tools to correlate very well (r=0.857), which is consistent with previous correlations between the Flesch-Kincaid formula and other readability scales (Kincaid et al., 1975; Paasche-Orlow, Taylor, 2003). As well, the Flesch-Kincaid formula is significantly easier to use, as it is an embedded tool in Microsoft Word. Therefore the Flesch-Kincaid formula was subsequently chosen for the 30 Web site evaluation. Accommodation for visual impairment was removed as none of the Web sites allowed for this. Content was revised to better reflect specific goals and published guidelines of the National Kidney Foundation, which in turn are based on best evidence or, in the absence of evidence, expert opinion (K/DOQI, 2004). Finally, quality was revised to allow for greater objectivity. All sections were simplified dramatically. Piloting the system in this manner has enabled us to render it a more objective instrument, improving inter-rater reliability and has allowed us to address content validity.
Following piloting of the evaluation system, scoring was weighted to allow each area of evaluation to have similar weight for a median score (Figure 1). Usability scores were the composite of readability and ease of navigation, with readibility, which is key to usability, given greater weight. Because reading grade levels are inversely related to readibility (i.e., the higher the reading grade level, the lower the readibility) and the number of links needed to locate useful material is likewise inversely related to ease of navigation (i.e., the greater the number of links needed, the lower the ease of navigation), the following adjustments were made: the grade level (out of 12) was mul-tipled by a factor of 0.2, and then subtracted from 4. The number of links needed was multiplied by a factor of 0.1, and then subtracted from 1. This formula allowed for the usability score, like the content and quality, to be higher with a lower reading grade level/number of links needed, as well as providing greater relative weighting of readability.
All 30 Web sites were evaluated between April and July 2004. Total scores were calculated for all 30 Web sites for each ofthe two evaluators. Mean score for evaluator 1 was 7.293 (standard error 0.511) and for evaluator 2 was 8.189 (0.413). Each evaluator’s score was assigned scores accordingly to quartiles (i.e., 1 through 4; see Figure 2). Two indices were calculated to assess for interranking reliability.
Cohen’s kappa measures concordance between two rankings on the same objects using an ordinal scale. In calculation, differences in scoring are usually weighted according to the differences between the two evaluators. A correlation of 0.6-0.8 is typically considered to be highly significant, with a correlation of 0.8-1.0 almost perfect. The Cohen’s kappa for the two evaluators’ total scores for all 30 Web sites is 0.7671. Intraclass correlation evaluates the level of agreement between rankings in measurements. The coefficient represents concordance, where 1 is perfect agreement and 0 is no agreement at all. The intraclass correlation for the two evaluators’ total scores for all 30 Web sites is 0.7703.
discussion
The young field of e-health is changing rapidly. As Danaher, McKay, and Seeley (2005) point out in a recent analysis of information architecture design of health behavior Web sites) that despite increasing sophistication, there is as yet no standardized, universally accepted or validated process for designing the structure of health-related Web sites. Despite this, e-health Web sites designed to educate patients with CKD may be a powerful tool in the nephrologists’ armamentarium. Currently available and easily accessible, CKD education Web sites present significant limitations as they do not appear to comply with available standards of e-health and are not constructed to be helpful for the majority of end-users who might benefit from them. Despite this, it is possible to identify Web sites that may be more likely to provide a positive educational experience for patient and families.
In previous work, we identified differences in readability and overall e-health Web site quality between nonprofit and industry-owned organizations (Jaffery & Becker, 2004). The present study is the first to our knowledge to attempt an exhaustive evaluation of the quality of CKD education Web sites in multiple realms. Calderon and colleagues identified 12 such sites that fulfilled at least three of six “domains of CKD information” and also found significant limitations of readability (Calderon et al., 2004a, b). While there have been many different criteria used by diverse groups to evaluate Web sites in a variety of specialties and subspecialties, readability is usually included and is nearly universally felt to be too high (high-school level or above) (Berland, Elliott, 2001; Friedman, Hoffman-Goetz, 2004; Griffin, McKenna, 2004; Kusec, Brborovic, 2003). Not surprisingly, Birru, et al. (2004) showed that adults with low health literacy are limited in their ability to gain information from health Web sites.
Our study has several limitations. Only two evaluators performed the Web site evaluations. While correlations were strong, it is possible that different results could be seen with additional evaluations. This study also did not address the issue of languages other than English; this is clearly a potential area for improvement, most notably for Spanish -peaking minorities, a disproportionate number of whom have CKD.
Many of the readily identified sites on the Web are either directed toward health care professionals, rather than consumers, or geared toward specific kidney diseases or distinct CKD complications. Because of this, the present evaluation may be biased against some of the more likely sites to be visited by patients, such as those arising more frequently from common search engines searches, but eliminated from our preestablished exclusion criteria. However, this bias should actually favor the Web sites that were chosen, as they fulfilled criteria more relevant to patients with CKD.
Finally, the present analysis does not specifically address the utility of these sites as educational interventions. While it is tempting to use information such as that found in this evaluation to make recommendations to patients and providers about specific Web site usage, in fact, none of these sites has been proven to enhance patient knowledge of CKD or aid in patient care. Further evaluation will be needed to investigate the relative utility of variably ranked Web sites as educational interventions, as well as strategies for more effectively steering patients and families toward Web sites most useful to them.
