Information Technology Reference
In-Depth Information
ers under the assumption that
multiple models are better than one if they are diverse. Popular ensemble approaches
include boosting and bagging [
50
]. Combination strategies for the multiple decisions
can be divided into two types: those that adaptively adjust to the training set based on
the performance of previous models as in boosting methods and those do not as in
bagging [
51
]. The bootstrap is widely used to estimate the standard error or confi-
Ensemble learning combines multiple trained classi
-
dence intervals of an estimate. Bagging is based on the bootstrap technique where the
predictions on bootstrapped samples are aggregated to form an ensemble hypothesis.
Boosting combines the predictions from re-sampled data based on the previous
model
s performance such that harder data samples for the system are more likely to
be sampled. Bagging has been shown to reduce the variance of classi
'
ers while
boosting can reduce both variance and bias. However, it is rarely shown that bagging
and boosting combined with feature selection can signi
cantly reduce ensemble
training time in practice. In our method, we adopt the boosting approach.
Instead of using one SVM in a very high dimensional feature space, we break the
feature space into subsets of low dimension feature spaces (also known as feature
vectors). Each feature vector established one SVM, and all SVMs form a com-
mittee. We allow overlap of features between different feature subsets. This scheme
combines the advantages of using a large number of features and keeping the
feature space small for single SVM in the committee. Each member in the com-
mittee has one vote for the classification, i.e., if the decision function of the SVM is
greater than 0, the vote is
. The majority vote is used
as the decision function of the committee. The committee approach generally
produces improved results, provided that the error rate for each member is less than
50 %. Figure
7
demonstrates how the SVM committee works. This is a committee
of three SVMs. In the
'
yes
'
, otherwise the vote is
'
no
'
ed data (big square and
big circle), but in the second and third SVM, they are correctly classi
first SVM, there are two misclassi
ed. By a
majority vote, a correct classi
cation is reached.
In order for the committee to achieve optimal performance, SVM members
should be able to compensate each other. If only a few top feature vectors are
selected, they usually tend to overlap each other and putting them together will not
enhance the differentiating power of the committee. Therefore, a large pool of
feature vectors should be available for committee member selection. We developed
a progressive feature vector selection method for this purpose [
52
]. The goal of the
feature vector selection is to generate a large pool of feature vectors to be used as
candidates for committee members. The task is to select K feature vectors with best
performance, and each vector has N features. Here K is a large number, and N is a
relatively small number. There are several commonly used feature selection
schemes, including exhaustive search, forward stepwise search, and genetic algo-
rithm [
20
]. Exhaustive search can be very time consuming if hundreds of features
are available as candidates. Forward stepwise search is easily trapped in local
minimum, and genetic algorithm is sensitive to the initial population.
We proposed a progressive search method to ef
ciently select a group of K best
N-feature vectors [
53
]. In this method, N-feature vectors are formed progressively
in N stages. In each stage, one more feature is added to the vectors selected from the
