Scalable, efficient, stepwise-optimal feature elimination in support vector machines

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

We address feature selection for support vector machines for the scenario in which the feature space is huge, i.e., 105-105 or more features, as may occur e.g. in a biomedical context working with 3-D (or 4-D) brain images. Feature selection in this case may be needed to improve the classifier's generalization performance (given limited training data), to reduce classification complexity, and/or to identify a minimum subset of features necessary for accurate classification, i.e., a set of putative "biomarkers". While there are a variety of techniques for SVM-based feature selection, many such may be unsuitable for huge feature spaces due to computational and/or memory requirements. One popular, lightweight scheme is recursive feature elimination (RFE) [5], wherein the feature with smallest weight magnitude in the current solution is eliminated at each step. Here we propose an alternative to RFE that is stepwise superior in that it maximizes margin (in the separable case) and minimizes training error rate (in the non-separable case), rather than minimizing weight magnitude. Moreover, we formulate an algorithm that achieves this stepwise maximum margin feature elimination without requiring explicit margin evaluation for all the remaining (candidate) features - in this way, the method achieves reduced complexity. To date, we have only performed experiments on (modestly dimensioned) UC Irvine data sets, which demonstrate better classification accuracy of our scheme (both training and test) over RFE. At the workshop, we will present results on huge feature spaces, for disease classification of 3-D MRI brain images and on other data domains.

Original languageEnglish (US)
Title of host publicationMachine Learning for Signal Processing 17 - Proceedings of the 2007 IEEE Signal Processing Society Workshop, MLSP
Pages75-80
Number of pages6
DOIs
StatePublished - Dec 1 2007
Event17th IEEE International Workshop on Machine Learning for Signal Processing, MLSP-2007 - Thessaloniki, Greece
Duration: Aug 27 2007Aug 29 2007

Other

Other17th IEEE International Workshop on Machine Learning for Signal Processing, MLSP-2007
CountryGreece
CityThessaloniki
Period8/27/078/29/07

Fingerprint

Support vector machines
Feature extraction
Brain
Biomarkers
Magnetic resonance imaging
Classifiers
Data storage equipment
Experiments

All Science Journal Classification (ASJC) codes

  • Computer Science(all)
  • Signal Processing

Cite this

Aksu, Y., Kesidis, G., & Miller, D. J. (2007). Scalable, efficient, stepwise-optimal feature elimination in support vector machines. In Machine Learning for Signal Processing 17 - Proceedings of the 2007 IEEE Signal Processing Society Workshop, MLSP (pp. 75-80). [4414285] https://doi.org/10.1109/MLSP.2007.4414285
Aksu, Yaman ; Kesidis, George ; Miller, David Jonathan. / Scalable, efficient, stepwise-optimal feature elimination in support vector machines. Machine Learning for Signal Processing 17 - Proceedings of the 2007 IEEE Signal Processing Society Workshop, MLSP. 2007. pp. 75-80
@inproceedings{7e669da0eea54a6a8e56f9c65f1807d0,
title = "Scalable, efficient, stepwise-optimal feature elimination in support vector machines",
abstract = "We address feature selection for support vector machines for the scenario in which the feature space is huge, i.e., 105-105 or more features, as may occur e.g. in a biomedical context working with 3-D (or 4-D) brain images. Feature selection in this case may be needed to improve the classifier's generalization performance (given limited training data), to reduce classification complexity, and/or to identify a minimum subset of features necessary for accurate classification, i.e., a set of putative {"}biomarkers{"}. While there are a variety of techniques for SVM-based feature selection, many such may be unsuitable for huge feature spaces due to computational and/or memory requirements. One popular, lightweight scheme is recursive feature elimination (RFE) [5], wherein the feature with smallest weight magnitude in the current solution is eliminated at each step. Here we propose an alternative to RFE that is stepwise superior in that it maximizes margin (in the separable case) and minimizes training error rate (in the non-separable case), rather than minimizing weight magnitude. Moreover, we formulate an algorithm that achieves this stepwise maximum margin feature elimination without requiring explicit margin evaluation for all the remaining (candidate) features - in this way, the method achieves reduced complexity. To date, we have only performed experiments on (modestly dimensioned) UC Irvine data sets, which demonstrate better classification accuracy of our scheme (both training and test) over RFE. At the workshop, we will present results on huge feature spaces, for disease classification of 3-D MRI brain images and on other data domains.",
author = "Yaman Aksu and George Kesidis and Miller, {David Jonathan}",
year = "2007",
month = "12",
day = "1",
doi = "10.1109/MLSP.2007.4414285",
language = "English (US)",
isbn = "1424415667",
pages = "75--80",
booktitle = "Machine Learning for Signal Processing 17 - Proceedings of the 2007 IEEE Signal Processing Society Workshop, MLSP",

}

Aksu, Y, Kesidis, G & Miller, DJ 2007, Scalable, efficient, stepwise-optimal feature elimination in support vector machines. in Machine Learning for Signal Processing 17 - Proceedings of the 2007 IEEE Signal Processing Society Workshop, MLSP., 4414285, pp. 75-80, 17th IEEE International Workshop on Machine Learning for Signal Processing, MLSP-2007, Thessaloniki, Greece, 8/27/07. https://doi.org/10.1109/MLSP.2007.4414285

Scalable, efficient, stepwise-optimal feature elimination in support vector machines. / Aksu, Yaman; Kesidis, George; Miller, David Jonathan.

Machine Learning for Signal Processing 17 - Proceedings of the 2007 IEEE Signal Processing Society Workshop, MLSP. 2007. p. 75-80 4414285.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Scalable, efficient, stepwise-optimal feature elimination in support vector machines

AU - Aksu, Yaman

AU - Kesidis, George

AU - Miller, David Jonathan

PY - 2007/12/1

Y1 - 2007/12/1

N2 - We address feature selection for support vector machines for the scenario in which the feature space is huge, i.e., 105-105 or more features, as may occur e.g. in a biomedical context working with 3-D (or 4-D) brain images. Feature selection in this case may be needed to improve the classifier's generalization performance (given limited training data), to reduce classification complexity, and/or to identify a minimum subset of features necessary for accurate classification, i.e., a set of putative "biomarkers". While there are a variety of techniques for SVM-based feature selection, many such may be unsuitable for huge feature spaces due to computational and/or memory requirements. One popular, lightweight scheme is recursive feature elimination (RFE) [5], wherein the feature with smallest weight magnitude in the current solution is eliminated at each step. Here we propose an alternative to RFE that is stepwise superior in that it maximizes margin (in the separable case) and minimizes training error rate (in the non-separable case), rather than minimizing weight magnitude. Moreover, we formulate an algorithm that achieves this stepwise maximum margin feature elimination without requiring explicit margin evaluation for all the remaining (candidate) features - in this way, the method achieves reduced complexity. To date, we have only performed experiments on (modestly dimensioned) UC Irvine data sets, which demonstrate better classification accuracy of our scheme (both training and test) over RFE. At the workshop, we will present results on huge feature spaces, for disease classification of 3-D MRI brain images and on other data domains.

AB - We address feature selection for support vector machines for the scenario in which the feature space is huge, i.e., 105-105 or more features, as may occur e.g. in a biomedical context working with 3-D (or 4-D) brain images. Feature selection in this case may be needed to improve the classifier's generalization performance (given limited training data), to reduce classification complexity, and/or to identify a minimum subset of features necessary for accurate classification, i.e., a set of putative "biomarkers". While there are a variety of techniques for SVM-based feature selection, many such may be unsuitable for huge feature spaces due to computational and/or memory requirements. One popular, lightweight scheme is recursive feature elimination (RFE) [5], wherein the feature with smallest weight magnitude in the current solution is eliminated at each step. Here we propose an alternative to RFE that is stepwise superior in that it maximizes margin (in the separable case) and minimizes training error rate (in the non-separable case), rather than minimizing weight magnitude. Moreover, we formulate an algorithm that achieves this stepwise maximum margin feature elimination without requiring explicit margin evaluation for all the remaining (candidate) features - in this way, the method achieves reduced complexity. To date, we have only performed experiments on (modestly dimensioned) UC Irvine data sets, which demonstrate better classification accuracy of our scheme (both training and test) over RFE. At the workshop, we will present results on huge feature spaces, for disease classification of 3-D MRI brain images and on other data domains.

UR - http://www.scopus.com/inward/record.url?scp=48149100965&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=48149100965&partnerID=8YFLogxK

U2 - 10.1109/MLSP.2007.4414285

DO - 10.1109/MLSP.2007.4414285

M3 - Conference contribution

SN - 1424415667

SN - 9781424415663

SP - 75

EP - 80

BT - Machine Learning for Signal Processing 17 - Proceedings of the 2007 IEEE Signal Processing Society Workshop, MLSP

ER -

Aksu Y, Kesidis G, Miller DJ. Scalable, efficient, stepwise-optimal feature elimination in support vector machines. In Machine Learning for Signal Processing 17 - Proceedings of the 2007 IEEE Signal Processing Society Workshop, MLSP. 2007. p. 75-80. 4414285 https://doi.org/10.1109/MLSP.2007.4414285