Biomedical Engineering Reference
In-Depth Information
Multiple testing adjustments
- microarray data analysis has different chal-
lenges compared to classic statistics, which test for a few variables across
many replications (such as the simple example of the price of wheat in rela-
tion to monthly rainfall information over the past 50 years). In contrast,
microarray experiments have thousands of variables (genes or SNPs) and
very few replications (number of patients). Microarray experiments are
costly and even with an unlimited budget, samples and materials would be
limiting factors. Using statistics by defi nition at a
p
-value of 0.05, at least 5%
of the assays on the array will be signifi cant. The standard multiple testing
adjustments include the Bonferroni correction (Bonferroni, 1935,1936). The
Bonferroni correction controls the error rate for all the tests. If you perform
n
tests, and you want the overall test to be signifi cant at the α level of 0.05,
then for each individual test to be signifi cant, its
p
-value needs to be below
β = α/n
. However, in the case of microarray data,
n
is very large and there-
fore for an
n
of 100 000 (very common when looking at SNP chips), the
p
-values for individual tests would have to be below 0.0000005, a number
so small that some statistical packages don't even report it. A problem with
this correction is that the study-wide error rate it controls applies to the uni-
versal null hypothesis. Furthermore, though the two groups are identical for
all variables, we cannot distinguish which, if any, of the variables differs, and
the likelihood of the type II error (caused by not recognizing a signifi cant
difference) is greatly increased (Perneger, 1998). This method is judged as
too conservative by many.
There are other less conservative methods than the Bonferroni correc-
tion that can be used, such as the Holm-Bonferroni, the Sidak correction
and others (Dudoit
et al.
, 2002; Holm, 1979; Hochberg and Tamhane, 1987;
Shaffer, 1995; Westfall and Young, 1993). Another popular method to cor-
rect for the multiple testing is to control the FDR in the same way as it
is done with SAM. There are four main factors that infl uence the FDR of
a microarray study: (1) the proportion of true positives, (2) their distri-
bution, (3) the measured variability and (4) the sample size. The experi-
menter can only control the last parameter, sample size, with replication.
Since the shuffl ing method is used to estimate the null hypothesis, suffi -
cient replicates are needed to shuffl e the data enough times to correctly
estimate the FDR. When the number of truly differentiated genes is small,
or when the fold- changes are small, a large sample size is needed to con-
trol for the FDR. In small experiments involving less than fi ve arrays per
group, the true positives found would have to have large fold-changes
(about three times the standard deviation) in order to be found signifi cant
at a reasonable FDR level.
Finally, one can combine a p-value/FDR cut-off with a regular aver-
age fold-change cut-off. Though some genes or SNPs might be signifi cant,
if the overall change they produce is biologically insignifi cant, then they
