Data Sets and Proper Statistical Analysis of Data Mining Techniques - Data Preprocessing in Data Mining - page 30

allows us to apply it over the means obtained by the algorithms in each data set,
without any assumptions about the sample of results obtained.

2.2.3.2 A Case Study: Performing Pairwise Comparisons

In the following, we will perform the statistical analysis by means of pairwise com-
parisons by using the results of performance measures obtained by the algorithms
taken as reference for this section: MLP, RBFN, SONN and LVQ. A similar yet more
detailed study can be found in [ 20 ].
In order to compare the results between two algorithms and to stipulate which
one is the best, we can perform a Wilcoxon signed-rank test for detecting differences
in both means. This statement must be enclosed by a probability of error, that is the
complement of the probability of reporting that two systems are the same, called the
p value [ 32 ]. The computation of the p value in the Wilcoxon distribution could be
carried out by computing a normal approximation [ 24 ]. This test is well known and
it is usually included in standard statistics packages (such as SPSS, R, SAS, etc.) as
well as in open source implementations like as in KEEL [ 9 ] (see Chap. 10 ) .
In Table 2.4 the ranks obtained by each method are shown. Table 2.5 shows the
results obtained in all possible comparisons of the algorithms considered in the
example study. We stress with bullets the winning algorithm in each row/column
when the p value associated is below 0.1 and/or 0.05.
The comparisons performed in this study are independent, so they never have to be
considered in a whole. If we try to extract a conclusion which involves more than one
comparison from the previous tables, we will lose control of the FWER. For instance,

Table 2.4 Ranks computed by the Wilcoxon test
(1) (2) (3) (4) (5)

MLP-CG-C (2) 19.0 - 16.0 0.0 15.0

RBFN-C (3) 11.0 5.0 - 0.0 5.0

SONN-C (4) 21.0 21.0 21.0 - 21.0

LVQ-C (5) 16.0 6.0 16.0 0.0 -

Row algorithm is the reference

Table 2.5 Summary of the Wilcoxon test
(1) (2) (3) (4) (5)

MLP-CG-C (2) - ◦

RBFN-C (3) - ◦

SONN-C (4) • • • - •

LVQ-C (5) ◦ -

•= the method in the row improves the method of the column
◦= the method in the column improves the method of the row. Upper diagonal of level significance
α = 0 . 9, Lower diagonal level of significance α = 0 . 95