An increasing amount of very useful quantitative evidence from health care research is available to practitioners. New research findings continually expand the knowledge base of what does more good than harm for patients, and institutional forces, both professional and financial, are accelerating the adoption of research findings. More and more information is available on issues related to such important clinical topics as screening and diagnostic tests, preventive and therapeutic interventions, prognosis and clinical prediction, risk of adverse outcomes, improvement in quality of care, and cost-effectiveness of tests and treatments. Clinical application of this evidence has lagged, however, for a number of reasons.1 First, evidence from research is often not definitive or covers only some aspects of practice. Second, clinicians are often slow to adopt research findings, even those that are well validated. Third, resources may be inadequate or too poorly organized in the local setting to permit implementation. Fourth, clinicians may be unfamiliar with the concepts that lie behind the application of quantitative reasoning to clinical care. This topic addresses the last of these barriers: principles and methods for quantitative reasoning.
Lack of precision in clinical thinking is beginning to yield to several encouraging developments—in particular, clinicians increasingly applying principles of critical appraisal to evidence in the medical literature; formulation of methods for medical decision analysis; increasing clinical comfort with terms such as sensitivity, specificity, likelihood ratio, number needed to treat, and confidence interval (CI); and creation of print and electronic resources that minimize the effort that clinicians must make to find and interpret valid quantitative evidence when it is needed. These developments notwithstanding, the possibility of miscom-munication is still considerable. A 2003 study of primary care physicians reported that just over 50% of respondents were able to answer questions about critical appraisal of methods and interpretation of results of studies focusing on treatments and diagnostic tests.2 Patients are entitled to expect clearer thinking from their physicians, especially because many patients have difficulty themselves interpreting information about risks, benefits, and prognoses provided by their doctors.3 Moreover, the current health care environment increasingly demands that physicians be able to justify clinical policies and decisions with an evidence-based, quantitative approach.
We have two principal goals in this topic. The first is to provide a basic explanation of the measurements used in critical appraisal of the literature and the ways in which physicians interpret these measurements in evidence-based clinical decision making. With the advent of electronic access to MED-LINE and its clinical subsets, specialized compendia of studies (e.g., Clinical Evidence4 and Physicians’ Information and Education Resource [PIER]5), systematic reviews of studies (e.g., the Cochrane Library6), and alerting services for new, clinically relevant evidence (e.g., bmjupdates+7 and MEDSCAPE Best Evidence alerts8), the current best evidence for clinical practice is becoming more and more accessible to clinicians.
The second goal is to introduce the topic of medical decision analysis. Clinicians use decision analysis in two ways. One way is essentially indirect: reliance on products of decision analyses conducted by others. For example, practice guidelines increasingly influence many of the quick, straightforward decisions that occur in daily practice. Many of these guidelines are based on formal decision analyses. The second way of employing decision analysis is more direct: using the tools of decision analysis to assist in making major decisions about the care of an individual patient. Although few physicians spend the hours required to conduct a formal decision analysis from scratch, some tools of decision analysis (e.g., likelihood ratios of test results) are easy to apply; moreover, some decision analyses are accessible on a desktop or palmtop computer and only require the clinician to enter the clinical findings required by the decision tree.
It is important to understand the intent of this effort to achieve precision and quantitation in measurement and decision making: to enhance the quality of care by making it more tailored to the individual patient. Anything that can be measured, even if only qualitatively, can be counted and turned into a clinically useful quantitative measure. For example, a study might classify clinical outcomes only qualitatively (e.g., as satisfactory or unsatisfactory), but if the numbers of participants in the study who fall into one or the other of the two outcome states are counted, the result then becomes quantitative. If physicians can define individual states and measure them quantitatively (e.g., by using a continuous scale to assess functional status), they can describe individual patient status more precisely and therefore can make finer distinctions between groups of patients. By placing patients in distinctive groups, physicians can achieve one of the great goals of patient care: to inform patients of the choices between alternative treatments by the known predictors of response to those treatments.
What is the role of the individual practitioner in retrieving and evaluating evidence from research and incorporating it into individual clinical decisions? The answer to this innocuous question distills the angst of contemporary health care. In some settings, the practitioner has the freedom to act as circumstances dictate, whereas in others (e.g., certain managed care settings), someone else tries to dictate how to translate research results into patient care. We believe that practitioners cede their responsibility for clinical decision making to others at great risk to their patients and themselves, because any clinical decisions must take into account not only the evidence available and the guidelines in force but also the patient’s unique circumstances and individual wishes. In today’s world, the freedom to determine the content of one’s practice is increasingly precious. To use this freedom responsibly, practitioners must have ready access to information that is based on current best evidence, must understand the basic principles of quantitative decision making and decision analysis, must be able to determine whether others have applied these principles appropriately in published works or in practice,and must be able to understand how to use evidence from research to make decisions in clinical practice.
Table 1 Abbreviated Users’ Guides for Appraisal of Medical Journal Articles
|
Purpose of Study |
Source of Data |
Method of Arriving at Findings |
Method of Reducing Bias of Findings |
|
Diagnosis |
Clearly identified comparison groups, all suspected of having the disorder, but one of which is free of the disorder |
Objective or reproducible diagnostic standard applied to all participants |
Blinded assessment of test and diagnostic standard |
|
Therapy |
Random allocation of patients to comparison groups |
Outcome measure of known or probable clinical importance |
Follow-up of > 80% of subjects |
|
Prognosis |
Inception cohort, early in the course of the disorder and initially free of the outcome of interest |
Objective or reproducible assessment of clinically important outcomes |
Follow-up of > 80% of subjects |
|
Causation |
Clearly identified comparison group for those who are at risk for, or for those having, the outcome of interest |
Blinding of observers of outcome to exposure; blinding of observers of exposure to outcome |
— |
|
Review |
Comprehensive search for relevant articles |
Explicit criteria for rating relevance and merit of studies |
Inclusion of all relevant studies |
How to Critically Evaluate Research Reports
To use numbers wisely in making decisions about patients, the physician must have some way of determining whether the numbers are derived from sound research. Detailed users’ guides for interpreting the medical literature are available9; in an effort to simplify this issue, we have provided an abbreviated set of such guides [see Table 1].10 Physicians may find these guides especially useful when reading research reports in the primary literature. However, when physicians are not getting and interpreting evidence themselves, they should look to evidence-based publications, such as Clinical Evidence and PIER; systematic review articles, such as those from the Cochrane Collaboration and clinical journals; and practice guidelines that use explicit criteria for evaluating evidence [see Table 1].
How to Apply Research Results to Patient Care
Once a physician is satisfied that the quantitative results from the relevant research were derived through sound methods, he or she can interpret them in light of the patient’s circumstances and use them to help determine the best way to proceed with management. The interpretation of research results takes five main forms: (1) measures of disease frequency, (2) measures of diagnostic certainty, (3) measures of diagnostic test performance and interpretation, (4) measures of the effects of treatment, and (5) measures of treatment outcomes adjusted for quality of life.
Measures of disease frequency
Clinically useful measures of disease frequency include incidence, prevalence, the case-fatality rate, the P value, and the CI [see Table 2]. The use of such terms is illustrated in more detail elsewhere (see below).
Measures of diagnostic certainty: use of probabilities
When asked how sure they are of their diagnoses, most physicians express their degree of certainty in words rather than numbers. A classic study illustrates the difficulty of this approach.11 The authors examined pathology and radiology reports and recorded various terms expressive of the probability of a disorder, such as "compatible with," "consistent with," "likely," "probably," and "pathognomonic." They then asked a group of clinicians to assign numerical probabilities to all of these terms. For each term (even "pathognomonic"), the range of probabilities stretched over half the scale. For example, to one physician, "likely" meant there was a 45% chance that the disease in question was present, whereas to another, "likely" meant the probability was higher than 90%. When diagnostic-test specialists were asked on two different occasions what they meant by these terms, the earlier and later answers were highly consistent for each individual specialist but highly inconsistent from one specialist to the next.
An alternative to using words to express the degree of diagnostic certainty is to use a number—namely, the probability that the diagnosis is present. A probability is a number between 0 and 1 that expresses the likelihood that an event will occur; 0 represents certainty that it will not occur, and 1 represents certainty that it will. Using probability to express diagnostic certainty has two key advantages. First, it facilitates precise communication. Comparison of probability estimates is a far more precise method of comparing degrees of diagnostic certainty than ex-changing verbal assessments.
|
Table 2 Clinically Useful Measures of Disease Frequency |
|
• Incidence: the proportion of new cases of a disorder occurring in a defined population during a specified period of time, typically 1 year. |
|
• Prevalence: the proportion of cases of a disorder at a designated point in time in a specified population. |
|
• Case-fatality rate: the proportion of cases of a specified disorder that are fatal during a specified period of follow-up (typically 1 yr) from the onset of the disorder. |
|
• Quality-adjusted life year (QALY): a measure of survival in which each year of a patient’s survival is discounted according to a measure (usually an index) of the patienf s quality of life. |
|
• P value: the probability of obtaining the observed data, or more unlikely data, when the null hypothesis is true. The P value does not indicate the magnitude of the effect of interest, or even its direction, nor does it indicate how much uncertainty is associated with the results. |
|
• Confidence interval (CI): the range of values of a true effect that is consistent with the data observed in a study. A common (although not entirely correct) interpretation of a 95% confidence interval is that 95% of the time, the true value lies within the stated range of values. |
Second, there exists an accurate method of calculating changes in the likelihood of disease as new information (e.g., a test result) becomes available. This method, Bayes’ theorem, should be one of the central principles that underlie medical practice. This claim may seem audacious to some readers, but we all recognize that the interpretation of new information about the patient moves us either away from or closer to a diagnosis and, therefore, away from or closer to the decision to use a specific treatment.
The probability of an event is not precisely the same thing as the odds of an event occurring, even though the two are mathematically equivalent ways of expressing diagnostic uncertainty. Habitues of the racetrack are reputed to use odds directly, but most clinicians are likely to find probabilities easier to use. Each of these measures can be readily converted to the other, as follows:
To use a test result quantitatively, a physician must first estimate the pretest probability of the disease. Unaided, physicians are not particularly good at this task. In a 1982 study, when primary care physicians were given clinical scenarios and asked for their estimates of the probabilities of given disorders, they provided estimates—quite confident ones—but their estimates did not agree with those of their fellow clinicians.12 Indeed, when individual physicians were tested subsequently with the same scenarios, their later estimates did not agree with their initial ones.
How does a physician estimate the probability that a patient’s chief complaint is a manifestation of a particular disease? The first step is to take a careful history and do a physical examination. From this point, the physician may take any of three basic approaches to estimating the probability of a disease13: (1) subjective estimation, (2) estimation based on the prevalence of disease in other patients with the same syndrome, or (3) application of clinical prediction rules.
Subjective Estimation
In principle, the physician can draw on personal experience with similar patients and use the estimated frequency of the disease in those patients. In practice, this approach is little more than a semiquantitative guess and is prone to error because of defective recall, as well as to bias in the application of the heuristics (i.e., the rules of thumb) for estimating probability. Examples of such heuristics are representativeness, by which one estimates a probability on the basis of the similarity of the patient’s signs and symptoms to the features of the classic description of the disease, and availability, by which one estimates a probability partly on the basis of how easy it is to recall similar cases. One very useful heuristic is anchoring and adjustment, by which one establishes an initial estimate (e.g., the prevalence of pulmonary embolism in 100 patients presenting to the emergency department with pleuritic chest pain) and then adjusts the estimate upward or downward by taking into account the patient’s findings (e.g., hypoxemia, unilateral leg swelling, or a history of cancer). Physicians can, in principle, calculate the extent of such adjustments by using Bayes’ theorem (see below).
Estimation Based on the Prevalence of Disease in Other Patients with the Same Syndrome
One antidote to the failures of subjective probability estimation is to base the estimate on accurate diagnoses established in a series of patients with the same clinical syndrome as the patient under consideration. The best example is the diagnosis of suspected coronary artery disease in patients with chronic chest pain. On the basis of the clinical history, the physician can place the patient in one of three categories: typical angina pectoris, atypical angina, or nonanginal chest pain. Many published studies have reported the frequency of angiographically proven coronary disease in patients with these syndromes. These studies have shown, for example, that in a man with atypical angina, the probability of significant coronary artery disease is approximately 0.70 (see below).
Application of Clinical Prediction Rules
Clinical prediction rules describe the key clinical findings that predict a disease and show how to use these findings to estimate the probability of disease in a patient. Such rules are based on analysis of a standardized set of data, including clinical findings and the final diagnosis, for each of many patients with a diagnostic problem. One type of clinical prediction rule uses regression analysis to identify the best clinical predictors and their diagnostic weights. The sum of the diagnostic weights corresponding to a patient’s findings is a score, and the probability of disease for each patient is equivalent to the prevalence of disease among patients with a similar score. A well-known example of this approach is the rule for estimating the probability of cardiac complications from noncardiac surgery.14 Another interesting example showed that the prevalence of coronary artery disease in patients with similar chest pain scores varied systematically according to the overall prevalence of coronary artery disease in several study populations.15 This study suggested that the probability of disease corresponding to a patient’s clinical history varies depending on whether the setting of care is a primary care practice or a referral practice. Diagnostic Strategies for Common Medical Problems16 is an excellent source of pretest probabilities, as is Evidence-Based Physical Diagnosis.
