Databases Reference
In-Depth Information
Forward selection
In forward selection you start with a regression model with no
features, and gradually add one feature at a time according to
which feature improves the model the most based on a selection
criterion. This looks like this: build all possible regression models
with a single predictor. Pick the best. Now try all possible models
that include that best predictor and a second predictor. Pick the
best of those. You keep adding one feature at a time, and you stop
when your selection criterion no longer improves, but instead gets
worse.
Backward elimination
In backward elimination you start with a regression model that
includes
all
the features, and you gradually remove one feature at
a time according to the feature whose removal makes the biggest
improvement in the selection criterion. You stop removing fea‐
tures when removing the feature makes the selection criterion get
worse.
Combined approach
Most subset methods are capturing some flavor of
minimum-
redundancy-maximum-relevance
. So, for example, you could
have a greedy algorithm that starts with the best feature, takes a
few more highly ranked, removes the worst, and so on. This a
hybrid approach with a filter method.
Selection criterion
There are a number of selection criteria you could choose from. As a
data scientist you have to select which selection criterion to use. Yes!
You need a selection criterion to select the selection criterion.
Part of what we wish to impart to you is that in practice, despite the
theoretical properties of these various criteria, the choice you make is
somewhat arbitrary. One way to deal with this is to try different se‐
lection criteria and see how robust your choice of model is. Different
selection criterion might produce wildly different models, and it's part
of your job to decide what to optimize for and why:
R-squared
Given by the formula
R
2
= 1−
∑
i
2
y
i
−
y
i
, it can be interpreted as the
2
∑
i
y
i
−
y
proportion of variance explained by your model.
