Databases Reference
In-Depth Information
0.011111
0.016667
0.033333
0.1
0.2
0.5
1
2
5
0.19
0.18
0.17
0.16
0.15
0.14
0.13
0.12
0.11
0.1
0.09
−Inf
−8
−7
−6
−5
−4
−3
−2
−1
0
Fig. 2.
Single RLSC: Cross-validation experimentation in order to find the optimal
combination of kernel width and regularization parameter for madelon data set.
Vertical axis is the tenfold cross-validation error rate on training data, horizontal
axis is log
10
(
γ
), and each curve corresponds to a specific kernel width as
w
j
d
av
,where
w
j
are the numbers shown in the legend, and
d
av
is the average squared distance
between data samples. The optimal combination for this data set is
γ
=10
−
3
and
σ
2
=0
.
016667
d
av
of data sampled to train each LSC [19]. Our motivation in using random
kernels, or more precisely, random kernel widths, was to get rid of all these
tunable parameters in ensemble construction without sacrificing any of the
generalization performance.
Naturally, the kernel width cannot be completely random, but in a reason-
able range, which is determined by the data. We sampled the
σ
2
uniformly
in the range of [
d
2
min
/
4
,d
2
med
], where
d
med
is the median distance between
samples, and
d
min
is the minimum distance between samples. This was found
to be a reasonable range for all the five diverse challenge datasets.
The ensemble size was fixed to 200, and the fraction of training data to
train each LSC was fixed to 0.5. These were near-optimal values for ELSCs
according to our earlier experiments [19].
Ensemble output combination was done using PCA-regression. We exper-
imented also with plain regression using a mixture of training/OOB samples
or just the OOB-samples, but the differences were insignificant.
We present the final classification error rates in Table 4. Even though
there is no significant difference in validation error rates between using a sin-
gle RLSC with optimized parameters, an ELSCs with optimized parameters,
or an ELSC with random kernel width, the fact that the latter can be trained
without any necessary parameter/model selection makes it a desirable alter-
native.
