Graphics Reference
In-Depth Information
Iman-Davenport test
Iman and Davenport [
15
] proposed a derivation from the Friedman statistic given
that this last metric often produces a conservative effect not desired. The proposed
statistic is
2
F
(
n
−
1
)χ
F
ID
=
(2.4)
2
F
n
(
k
−
1
)χ
which is distributed according to an F distribution with
k
1 and
(
k
1
)(
N
1
)
degrees of
freedom. See Table A10 in [
24
] to find the critical values.
A drawback of the ranking scheme employed by the Friedman test is that it allows
for intra-set comparisons only. When the number of algorithms for comparison is
small, this may pose a disadvantage, since inter-set comparisons may not be mean-
ingful. In such cases, comparability among problems is desirable. The method of
aligned ranks [
12
] for the Friedman test overcomes this problem but for the sake of
simplicity we will not elaborate on such an extension.
Post-hoc procedures
The rejection of the null hypothesis in both tests described above does not involve
the detection of the existing differences among the algorithms compared. They only
inform us of the presence of differences among all samples of results compared. In
order to conducting pairwise comparisons within the framework of multiple com-
parisons, we can proceed with a post-hoc procedure. In this case, a control algorithm
(maybe a proposal to be compared) is usually chosen. Then, the post-hoc procedures
proceed to compare the control algorithm with the remain
k
−
1 algorithms. In the
following, we describe three post-hoc procedures:
•
Bonferroni-Dunn's procedure [
32
]: it is similar to Dunnet's test for ANOVA
designs. The performance of two algorithms is significantly different if the corre-
sponding average of rankings is at least as great as its critical difference (CD).
k
(
k
+
1
)
CD
=
q
(2.5)
α
6
N
is the critical value of
Q
for a multiple non-parametric comparison
with a control (Table B.16 in [
32
]).
The value of
q
α
•
Holm [
13
] procedure: for contrasting the procedure of Bonferroni-Dunn, we dis-
pose of a procedure that sequentially checks the hypotheses ordered according to
their significance. We will denote the
p
-values ordered by
p
1
,
p
2
,...
,intheway
that
p
1
≤
p
2
≤ ··· ≤
p
1. Holm's method compares each
p
i
with
α/(
ki
)
starting
from the most significant
p
-value. If
p
1
is lower than
α/(
k
1
)
, the corresponding
hypothesis is rejected and it leaves us to compare
p
2
with
α/(
k
2
)
. If the second
