Databases Reference
In-Depth Information
Fig. 3.
Gaussian kernel compared to a supervised kernel using the Arcene dataset.
Left side
depicts the 100
×
100 Gaussian kernel matrix of the data set. The data
can be seen to consist of three clusters. Each cluster has samples from both classes.
Class identities of samples are depicted as the graphs below and between the kernel
matrix images. For ideal classification purposes, the kernel matrix should reflect
the similarity within a class and dissimilarity between classes. This can be seen on
the
right side
of the figure, where the proposed supervised kernel has split the first
cluster (
top left corner
) into the two classes nicely. Splits on the second and third
clusters are not that clean but still visible, and much more so than what can be seen
in the Gaussian kernel
7 Conclusion
We proposed a relatively straightforward approach to create powerful ensem-
bles of simple least square classifiers with random kernels. We used NIPS2003
feature selection challenge data to evaluate performance of such ensembles.
The binary classification data sets considered in the challenge originated in
different domains with number of variables ranging from moderate to ex-
tremely large and moderate to very small number of observations. We used
fast exploratory Random Forests for variable filtering as a preprocessing step.
The individual learners were trained on small random sample of data with
Gaussian kernel width randomly selected from relatively wide range of val-
ues determined only by basic properties of the corresponding dissimilarities
matrix. The random sample of data used to build individual learner was rela-
tively small. Modest ensemble size (less than 200) stabilized the generalization
error. We used consistent parameter settings for all datasets, and achieved at
least the same accuracy as the best single RLSC or an ensemble of LSCs with
fixed tuned kernel width. Individual learners were combined through simple
OOB post-processing PCA regression.
