Learning Feature Nonlinearities with Regularized Binned Regression

Samet Oymak, Mehrdad Mahdavi, Jiasi Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

For various applications, the relations between the dependent and independent variables are highly nonlinear. Consequently, for large scale complex problems, neural networks and regression trees are commonly preferred over linear models such as Lasso. This work proposes learning the feature nonlinearities by binning feature values and finding the best fit in each quantile using non-convex regularized linear regression. The algorithm first captures the dependence between neighboring quantiles by enforcing smoothness via piecewise-constant/linear approximation and then selects a sparse subset of good features. We prove that the proposed algorithm is statistically and computationally efficient. In particular, it achieves linear rate of convergence while requiring near-minimal number of samples. Evaluations on real datasets demonstrate that algorithm is competitive with current state-of-the-art and accurately learns feature nonlinearities.

Original languageEnglish (US)
Title of host publication2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1452-1456
Number of pages5
ISBN (Electronic)9781538692912
DOIs
Publication statusPublished - Jul 2019
Event2019 IEEE International Symposium on Information Theory, ISIT 2019 - Paris, France
Duration: Jul 7 2019Jul 12 2019

Publication series

NameIEEE International Symposium on Information Theory - Proceedings
Volume2019-July
ISSN (Print)2157-8095

Conference

Conference2019 IEEE International Symposium on Information Theory, ISIT 2019
CountryFrance
CityParis
Period7/7/197/12/19

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Information Systems
  • Modeling and Simulation
  • Applied Mathematics

Cite this

Oymak, S., Mahdavi, M., & Chen, J. (2019). Learning Feature Nonlinearities with Regularized Binned Regression. In 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings (pp. 1452-1456). [8849541] (IEEE International Symposium on Information Theory - Proceedings; Vol. 2019-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISIT.2019.8849541