Graphics Reference
In-Depth Information
the statement: “SONN obtains a classification rate better than RBFN, LVQ and MLP
algorithms with a
p
value lower than 0.05” is incorrect, since we do not prove the
control of the FWER. The SONN algorithm really outperforms MLP, RBFN and
LVQ algorithms considering classification rate in independent comparisons.
The true statistical signification for combining pairwise comparisons is given
by Eq.
2.2
:
p
=
P
(
reject H
0
|
H
0
true
)
=
1
−
P
(
Accept H
0
|
H
0
true
)
=
1
−
P
(
Accept A
k
=
A
i
,
i
=
1
,...,
k
+
1
|
H
0
true
)
k
−
1
=
1
−
P
(
Accept A
k
=
A
i
|
H
0
true
)
i
=
1
k
−
1
=
1
−
1
[
1
−
P
(
Reject A
k
=
A
i
|
H
0
true
)
]
i
=
k
−
1
=
1
−
1
(
1
−
p
H
i
)
(2.2)
i
=
2.2.4 Non-parametric Tests for Multiple Comparisons Among
More than Two Algorithms
When a new ML algorithm proposal is developed or just being taking as reference,
it could be interesting to compare it with previous approaches. Making pairwise
comparisons allows this analysis to be conducted, but the experiment wise error can
not be previously controlled. Furthermore, a pairwise comparison is not influenced
by any external factor, whereas in a multiple comparison, the set of algorithms chosen
can determine the results of the analysis.
Multiple comparison procedures are designed for allowing the FWER to be fixed
before performing the analysis and for taking into account all the influences that can
exist within the set of results for each algorithm. Following the same structure as
in the previous section, the basic and advanced non-parametrical tests for multiple
comparisons are described in Sect.
2.2.4.1
and their application on the case study is
conducted in Sect.
2.2.4.2
.
2.2.4.1 Friedman Test and Post-hoc Tests
One of themost frequent situations where the use of statistical procedures is requested
is in the joint analysis of the results achieved by various algorithms. The groups of
differences between these methods (also called blocks) are usually associated with
