Graphics Reference
In-Depth Information
hypothesis is rejected, we continue with the process. As soon as a certain hypoth-
esis can not be rejected, all the remaining hypotheses are maintained as supported.
The statistic for comparing the
i
algorithm with the
j
algorithm is:
k
(
k
+
1
)
z
=
(
R
i
−
R
j
)/
6
N
The value of z is used for finding the corresponding probability from the table
of the normal distribution (
p
-value), which is compared with the corresponding
value of
. Holm's method is more powerful than Bonferroni-Dunn's and it does
no additional assumptions about the hypotheses checked.
α
•
Hochberg [
11
] procedure: It is a step-up procedure that works in the opposite direc-
tion to Holm's method, comparing the largest
p
-value with ?, the next largest with
α/
2 and so forth until it encounters a hypothesis it can reject. All hypotheses with
smaller
p
values are then rejected as well. Hochberg's method is more powerful
than Holm's [
23
].
The post-hoc procedures described above allow us to know whether or not a
hypothesis of comparison of means could be rejected at a specified level of sig-
nificance
. However, it is very interesting to compute the
p
-value associated to
each comparison, which represents the lowest level of significance of a hypothesis
that results in a rejection. In this manner, we can know whether two algorithms are
significantly different and also get a metric of how different they are.
In the following, we will describe the method for computing these exact
p
-values
for each test procedure, which are called “adjusted
p
-values” [
30
].
α
•
The adjusted
p
-value for BonferroniDunn's test (also known as the Bonferroni
correction) is calculated by
p
Bonf
=
(
k
−
1
)
p
i
.
•
The adjusted
p
-value for Holm's procedure is computed by
p
Holm
=
(
p
i
.
Once all of them have been computed for all hypotheses, it will not be possible to
find an adjusted
p
-value for the hypothesis
i
lower than that for the hypothesis
j
,
j
k
−
i
)
i
. In this case, the adjusted
p
-value for hypothesis
i
is set equal to the p-values
associated to the hypothesis
j
.
<
•
The adjusted
p
-value for Hochberg's method is computed with the same formula
as Holm's, and the same restriction is applied in the process, but to achieve the
opposite, that is, so that it will not possible to find an adjusted
p
-value for the
hypothesis
i
lower than for the hypothesis
j
,
j
>
i
.
2.2.4.2 A Case Study: Performing Multiple Comparisons
In this section we carry out a toy example on the analysis of a multiple comparison
using the same ML algorithms as Sect.
2.2.3.2
: MLP, RBFN, SONN and LVQ.
In Table
2.6
we show the ranks obtained by each algorithm for Friedman test.
From this table we can observe that SONN is the algorithm with the lowest rank and
hence will act as the control algorithm.
