Towards Principled Feature Selection: Relevancy, Filters and Wrappers

Ioannis Tsamardinos, Constantin F. Aliferis
Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, PMLR R4:300-307, 2003.

Abstract

In an influential paper Kohavi and John [7] presented a number of disadvantages of the filter approach to the feature selection problem, steering research towards algorithms adopting the wrapper approach. We show here that neither approach is inherently better and that any practical feature selection algorithm needs to at least consider the learner used for classification and the metric used for evaluating the learner’s performance. In the process we formally define the feature selection problem, re-examine the relationship between relevancy and filter algorithms, and establish a connection between Kohavi and John’s definition of relevancy to the Markov Blanket of a target variable in a Bayesian Network faithful to some data distribution. The theoretical results lead to principled ways of designing optimal filter algorithms of which we present one example.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR4-tsamardinos03a, title = {Towards Principled Feature Selection: Relevancy, Filters and Wrappers}, author = {Tsamardinos, Ioannis and Aliferis, Constantin F.}, booktitle = {Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics}, pages = {300--307}, year = {2003}, editor = {Bishop, Christopher M. and Frey, Brendan J.}, volume = {R4}, series = {Proceedings of Machine Learning Research}, month = {03--06 Jan}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/r4/tsamardinos03a/tsamardinos03a.pdf}, url = {https://proceedings.mlr.press/r4/tsamardinos03a.html}, abstract = {In an influential paper Kohavi and John [7] presented a number of disadvantages of the filter approach to the feature selection problem, steering research towards algorithms adopting the wrapper approach. We show here that neither approach is inherently better and that any practical feature selection algorithm needs to at least consider the learner used for classification and the metric used for evaluating the learner’s performance. In the process we formally define the feature selection problem, re-examine the relationship between relevancy and filter algorithms, and establish a connection between Kohavi and John’s definition of relevancy to the Markov Blanket of a target variable in a Bayesian Network faithful to some data distribution. The theoretical results lead to principled ways of designing optimal filter algorithms of which we present one example.}, note = {Reissued by PMLR on 01 April 2021.} }
Endnote
%0 Conference Paper %T Towards Principled Feature Selection: Relevancy, Filters and Wrappers %A Ioannis Tsamardinos %A Constantin F. Aliferis %B Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2003 %E Christopher M. Bishop %E Brendan J. Frey %F pmlr-vR4-tsamardinos03a %I PMLR %P 300--307 %U https://proceedings.mlr.press/r4/tsamardinos03a.html %V R4 %X In an influential paper Kohavi and John [7] presented a number of disadvantages of the filter approach to the feature selection problem, steering research towards algorithms adopting the wrapper approach. We show here that neither approach is inherently better and that any practical feature selection algorithm needs to at least consider the learner used for classification and the metric used for evaluating the learner’s performance. In the process we formally define the feature selection problem, re-examine the relationship between relevancy and filter algorithms, and establish a connection between Kohavi and John’s definition of relevancy to the Markov Blanket of a target variable in a Bayesian Network faithful to some data distribution. The theoretical results lead to principled ways of designing optimal filter algorithms of which we present one example. %Z Reissued by PMLR on 01 April 2021.
APA
Tsamardinos, I. & Aliferis, C.F.. (2003). Towards Principled Feature Selection: Relevancy, Filters and Wrappers. Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research R4:300-307 Available from https://proceedings.mlr.press/r4/tsamardinos03a.html. Reissued by PMLR on 01 April 2021.

Related Material