Methods wrapper evaluate subsets of variables which allows, unlike filter approaches, to detect the possible interactions between variables.

1. Wrappers for feature subset
selection
Amir Razmjou

2. Benefits of Feature Subset Selection
• Too many dimensions; Elimination of the
curse of dimensionality problem
• Improved model and classifier performance
• Simple Models and elimination of over-
fitting
• Faster training times.

3. The Process

4. Feature Selection Methods
Wrapper
use a search algorithm to search through the
space of possible features and evaluate each subset
by running a model on the subset
Risk of over fitting to the model
Computationally expensive
Embedded
Embedded in and specific to a model
Filter
Similar to Wrappers in the search approach
Simpler filter is evaluated

5. Hypothetical Concept

6. Definition 2

7. Contradiction 2

8. Definition 3

9. Contradiction 3
All features are irrelevant

10. Definition 4

11. Contradiction 4
every feature is relevant
Y=TRUE S1=X1=TRUE S2=X2=TRUE
Y=
X1 X2 X3 X4 = ! X2 X5 = !X3
X1 XOR X2
FALSE TRUE TRUE TRUE FALSE FALSE
FALSE TRUE TRUE FALSE FALSE TRUE
TRUE TRUE FALSE TRUE TRUE FALSE
TRUE TRUE FALSE FALSE TRUE TRUE
TRUE FALSE TRUE TRUE FALSE FALSE
TRUE FALSE TRUE FALSE FALSE TRUE
FALSE FALSE FALSE TRUE TRUE FALSE
FALSE FALSE FALSE FALSE TRUE TRUE

12. Definition 5 – Strong Relevance

13. Definition 6 – Weak Relevance

14. Conclusion 1

15. • Forward selection methods: these methods
start with one or a few features selected
according to a method specific selection
criteria. More features are iteratively added
until a stopping criterion is met.
• Backward elimination methods: methods of
this type start with all features and iteratively
remove one feature or bunches of features.

16. Relief
• Evaluates the worth of an attribute by
repeatedly sampling an instance and
considering the value of the given attribute for
the nearest instance of the same and different
class. Can operate on both discrete and
continuous class data.

17. Relief
• Relief does not help with redundant features.
If most of the given features are relevant to
the concept, it would select most of them
even though only a fraction are necessary for
concept description