Information Technology Reference
In-Depth Information
6.1.3 Experiments with Real-World Datasets
Experiments comparing MLPs with different risk functionals, applied to the
classification of real-world datasets, were reported in several works.
In the experiments reported in [198], datasets Diabetes, Wine and Iris
(UCI repository [13]) were used. Several MLPs with varying
n
h
were trained
and tested with the holdout method using
H
R
2
and MSE. The results of these
experiments are presented in Table 6.4. The superiorness of R
2
EE over MSE
for these datasets is clear. The bottom row of Table 6.4 also hints a stabler
design of R
2
EE than MSE.
Tabl e 6 . 4 Mean (standard deviation) of test error (%) for the experiments re-
ported in [198]. Last row is the standard deviation of the mean errors for all
n
h
values. Best results in bold.
Diabetes
Wine
Iris
n
h
R
2
EE
MSE
R
2
EE
MSE
R
2
EE
MSE
2
23.80(0.94)
28.40(4.87)
3.62(1.30)
9.72(10.60)
3
23.94(0.97)
27.25(4.72)
3.81(1.00)
4.27(3.77)
4.36(1.12) 4.72(4.75 )
4
23.99(1.52)
26.42(4.53)
1.94(0.72)
3.03(1.08)
4.43(1.30)
4.75(1.27)
5
23.80(1.04)
25.10(1.80)
2.50(1.01)
3.20(1.83)
4.38(1.34)
4.15(1.32)
6
24.10(1.33)
24.70(1.80)
2.47(1.20)
3.06(1.43)
4.30(1.16)
3.97(1.05)
7
24.10(0.90)
24.40(1.06)
2.44(1.00)
2.39(1.50)
4.41(1.42)
5.18(4.74)
8
23.90(0.71)
23.90(1.18)
2.16(0.92)
2.92(1.07)
4.31(1.27)
4.65(1.32)
9
24.30(1.42)
24.00(0.95)
2.22(0.83)
2.50(1.35)
10
23.60(0.86) 24.10(1.20)
2.31(0.51)
2.95(1.29)
11
24.02(1.00)
27.41(5.19)
12
24.93(3.24)
27.64(5.04)
STD
0.35
1.69
0.65
2.29
0.05
0.44
Experiments on two-class artificial and real-world datasets with MLPs us-
ing EE risks (R
2
EE, SEE), EXP, and the classical risks MSE and CE, were
reported in [215]. The datasets are shown in Table 6.5. Six of them are arti-
ficial checkerboard datasets, an example of which is shown in Fig. 6.10; both
2
4 (CB4x4) "boards" are used, with some percentage of
the less represented class given in parenthesis (e.g., CB2x2(10) has 10% of
the total number of instances assigned to one of the classes). The other six
datasets are real-world and available in [13].
All experiments with a given dataset used the same MLP architecture with
n
h
guaranteeing acceptable generalization. Regularization was performed by
early stopping (see, e.g., [26] on this issue) using a preliminary suite of 10
runs in order to choose an adequate number of epochs as well as an adequate
value of
h
(starting with initial values given by formula (6.8)). Details on all
these issues are given in [215].
×
2 (CB2x2) and 4
×
