Bayes Optimal Feature Selection for Supervised Learning with General Performance Measures

Saneem Ahmed CG, Harikrishna Narasimhan Indian Institute of Science, Shivani Agarwal Indian Institute of Science
Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence, PMLR R13:179-188, 2015.

Abstract

The problem of feature selection is critical in several areas of machine learning and data analysis. Here we consider feature selection for supervised learning problems, where one wishes to select a small set of features that facilitate learning a good prediction model in the reduced feature space. Our interest is primarily in filter methods that select features independently of the learning algorithm to be used and are generally faster to implement than wrapper methods. Many common filter methods for feature selection make use of mutual information based criteria to guide their search process. However, even in simple binary classification problems, mutual information based methods do not always select the best set of features in terms of the Bayes error. In this paper, we develop a filter method that directly aims to select the optimal set of features for a general performance measure of interest. Our approach uses the Bayes error with respect to the given performance measure as the criterion for feature selection and applies a greedy algorithm to optimize this criterion. We demonstrate application of this method to a variety of learning problems involving different performance measures. Experiments suggest the proposed approach is competitive with several state-of-the-art methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR13-cg15a, title = {{B}ayes Optimal Feature Selection for Supervised Learning with General Performance Measures}, author = {CG, Saneem Ahmed and Science, Harikrishna Narasimhan Indian Institute of and Science, Shivani Agarwal Indian Institute of}, booktitle = {Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence}, pages = {179--188}, year = {2015}, editor = {Meila, Marina and Heskes, Tom}, volume = {R13}, series = {Proceedings of Machine Learning Research}, month = {12--16 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r13/main/assets/cg15a/cg15a.pdf}, url = {https://proceedings.mlr.press/r13/cg15a.html}, abstract = {The problem of feature selection is critical in several areas of machine learning and data analysis. Here we consider feature selection for supervised learning problems, where one wishes to select a small set of features that facilitate learning a good prediction model in the reduced feature space. Our interest is primarily in filter methods that select features independently of the learning algorithm to be used and are generally faster to implement than wrapper methods. Many common filter methods for feature selection make use of mutual information based criteria to guide their search process. However, even in simple binary classification problems, mutual information based methods do not always select the best set of features in terms of the Bayes error. In this paper, we develop a filter method that directly aims to select the optimal set of features for a general performance measure of interest. Our approach uses the Bayes error with respect to the given performance measure as the criterion for feature selection and applies a greedy algorithm to optimize this criterion. We demonstrate application of this method to a variety of learning problems involving different performance measures. Experiments suggest the proposed approach is competitive with several state-of-the-art methods.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Bayes Optimal Feature Selection for Supervised Learning with General Performance Measures %A Saneem Ahmed CG %A Harikrishna Narasimhan Indian Institute of Science %A Shivani Agarwal Indian Institute of Science %B Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2015 %E Marina Meila %E Tom Heskes %F pmlr-vR13-cg15a %I PMLR %P 179--188 %U https://proceedings.mlr.press/r13/cg15a.html %V R13 %X The problem of feature selection is critical in several areas of machine learning and data analysis. Here we consider feature selection for supervised learning problems, where one wishes to select a small set of features that facilitate learning a good prediction model in the reduced feature space. Our interest is primarily in filter methods that select features independently of the learning algorithm to be used and are generally faster to implement than wrapper methods. Many common filter methods for feature selection make use of mutual information based criteria to guide their search process. However, even in simple binary classification problems, mutual information based methods do not always select the best set of features in terms of the Bayes error. In this paper, we develop a filter method that directly aims to select the optimal set of features for a general performance measure of interest. Our approach uses the Bayes error with respect to the given performance measure as the criterion for feature selection and applies a greedy algorithm to optimize this criterion. We demonstrate application of this method to a variety of learning problems involving different performance measures. Experiments suggest the proposed approach is competitive with several state-of-the-art methods. %Z Reissued by PMLR on 04 October 2026.
APA
CG, S.A., Science, H.N.I.I.o. & Science, S.A.I.I.o.. (2015). Bayes Optimal Feature Selection for Supervised Learning with General Performance Measures. Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R13:179-188 Available from https://proceedings.mlr.press/r13/cg15a.html. Reissued by PMLR on 04 October 2026.

Related Material