Introductory concepts - Practical Biomedical Signal Analysis Using MATLAB - page 17

• Use a resampling (bootstrap or permutation) test [Efron and Tibshirani,

1993]. In this type of test in principle any function of the data can be used
as a statistic. We need to formulate a statistical model of the process that
generates the data. The model is often very simple and relies on appro-
priate resampling of the original dataset, e.g., we draw a random sample
from the original dataset with replacement. It is crucial that the model
(the resampling process) conforms to the null hypothesis. The model is
simulated many times and for each realization the statistic is computed.
In this way the empirical distribution of the statistic is formed. The em-
pirical distribution of the statistic is used then to evaluate the probability
of observing the value of statistic equal or more extreme than the value
for the original dataset.

1.5.3 Multiple comparison problem
Let us assume that we perform a t test with the significance level α. This means
that under the true null hypothesis H 0 we can observe with the probability α value
greater than the critical:

P ( t ≥ t α )= α

(1.42)

P ( t < t α )= 1 − α

For n independent tests, the probability that none of them will not give t value greater
than the critical is

n . Thus the probability of observing at least one of the n
values exceeding the critical one (probability of the family wise error FWER) is:

( 1 − α )

P FW ER

n

P ( t ≥ t α ;in n tests )= = 1 − ( 1 − α ) (1.43)

Let us consider tests performed at significance level α 05, The probability com-

puted from (1.43) gives the chance of observing extreme values of statistic for data
conforming to the null hypothesis just due to fluctuations. For a single test the above
formula gives P

= 0 .

( t ≥ t α ;in 1 test )= 0 . 05, for n = 10, P ( t ≥ t α ;in 10 test ) ≈ 0 . 4,for

n 994.

In case of some dependence (e.g., correlation) among the tests, the above described
problem is less severe but also present. There is clearly a need to control the error of
false rejections of null hypothesis due to multiple comparison problem (MCP).
Table 1.1 summarizes the possible outcomes of m hypothesis tests. As a result of
the application of a test we obtain one of two true statements: true null hypothesis
is accepted (number of such cases is denoted U ) or false null hypothesis is rejected
(number of such cases is denoted S ). There is a possibility to commit one of two types
of errors. Type I error is when a true null hypothesis is rejected (the number of such
cases is denoted as V ); Type II error—the alternative hypothesis is true, but the null
hypothesis is accepted (denoted as T ). The total number of rejected null hypotheses
is denoted by R .
The total number of tested hypotheses m is known. The number of true null hy-
potheses m 0 and the number of false null hypotheses m 1

= 100, P ( t ≥ t α ; in 100 test ) ≈ 0 .

= m − m 0 are unknown.