Regularization parameter selections via generalized information criterion

Yiyun Zhang, Runze Li, Chih Ling Tsai

Research output: Contribution to journalArticle

126 Citations (Scopus)

Abstract

We apply the nonconcave penalized likelihood approach to obtain variable selections as well as shrinkage estimators. This approach relies heavily on the choice of regularization parameter, which controls the model complexity. In this paper, we propose employing the generalized information criterion, encompassing the commonly used Akaike information criterion (AIC) and Bayesian information criterion (BIC), for selecting the regularization parameter. Our proposal makes a connection between the classical variable selection criteria and the regularization parameter selections for the nonconcave penalized likelihood approaches. We show that the BIC-type selector enables identification of the true model consistently, and the resulting estimator possesses the oracle property in the terminology of Fan and Li (2001). In contrast, however, the AIC-type selector tends to overfit with positive probability. We further show that the AIC-type selector is asymptotically loss efficient, while the BIC-type selector is not. Our simulation results confirm these theoretical findings, and an empirical example is presented. Some technical proofs are given in the online supplementary material.

Original languageEnglish (US)
Pages (from-to)312-323
Number of pages12
JournalJournal of the American Statistical Association
Volume105
Issue number489
DOIs
StatePublished - Mar 1 2010

Fingerprint

Information Criterion
Selector
Parameter Selection
Regularization Parameter
Bayesian Information Criterion
Akaike Information Criterion
Penalized Likelihood
Variable Selection
Oracle Property
Shrinkage Estimator
Model Complexity
Tend
Estimator
Bayesian information criterion
Regularization
Akaike information criterion
Information criterion
Simulation
Variable selection
Model

All Science Journal Classification (ASJC) codes

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Cite this

@article{721b85f026cc4c9b89b769a9f7228d4d,
title = "Regularization parameter selections via generalized information criterion",
abstract = "We apply the nonconcave penalized likelihood approach to obtain variable selections as well as shrinkage estimators. This approach relies heavily on the choice of regularization parameter, which controls the model complexity. In this paper, we propose employing the generalized information criterion, encompassing the commonly used Akaike information criterion (AIC) and Bayesian information criterion (BIC), for selecting the regularization parameter. Our proposal makes a connection between the classical variable selection criteria and the regularization parameter selections for the nonconcave penalized likelihood approaches. We show that the BIC-type selector enables identification of the true model consistently, and the resulting estimator possesses the oracle property in the terminology of Fan and Li (2001). In contrast, however, the AIC-type selector tends to overfit with positive probability. We further show that the AIC-type selector is asymptotically loss efficient, while the BIC-type selector is not. Our simulation results confirm these theoretical findings, and an empirical example is presented. Some technical proofs are given in the online supplementary material.",
author = "Yiyun Zhang and Runze Li and Tsai, {Chih Ling}",
year = "2010",
month = "3",
day = "1",
doi = "10.1198/jasa.2009.tm08013",
language = "English (US)",
volume = "105",
pages = "312--323",
journal = "Journal of the American Statistical Association",
issn = "0162-1459",
publisher = "Taylor and Francis Ltd.",
number = "489",

}

Regularization parameter selections via generalized information criterion. / Zhang, Yiyun; Li, Runze; Tsai, Chih Ling.

In: Journal of the American Statistical Association, Vol. 105, No. 489, 01.03.2010, p. 312-323.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Regularization parameter selections via generalized information criterion

AU - Zhang, Yiyun

AU - Li, Runze

AU - Tsai, Chih Ling

PY - 2010/3/1

Y1 - 2010/3/1

N2 - We apply the nonconcave penalized likelihood approach to obtain variable selections as well as shrinkage estimators. This approach relies heavily on the choice of regularization parameter, which controls the model complexity. In this paper, we propose employing the generalized information criterion, encompassing the commonly used Akaike information criterion (AIC) and Bayesian information criterion (BIC), for selecting the regularization parameter. Our proposal makes a connection between the classical variable selection criteria and the regularization parameter selections for the nonconcave penalized likelihood approaches. We show that the BIC-type selector enables identification of the true model consistently, and the resulting estimator possesses the oracle property in the terminology of Fan and Li (2001). In contrast, however, the AIC-type selector tends to overfit with positive probability. We further show that the AIC-type selector is asymptotically loss efficient, while the BIC-type selector is not. Our simulation results confirm these theoretical findings, and an empirical example is presented. Some technical proofs are given in the online supplementary material.

AB - We apply the nonconcave penalized likelihood approach to obtain variable selections as well as shrinkage estimators. This approach relies heavily on the choice of regularization parameter, which controls the model complexity. In this paper, we propose employing the generalized information criterion, encompassing the commonly used Akaike information criterion (AIC) and Bayesian information criterion (BIC), for selecting the regularization parameter. Our proposal makes a connection between the classical variable selection criteria and the regularization parameter selections for the nonconcave penalized likelihood approaches. We show that the BIC-type selector enables identification of the true model consistently, and the resulting estimator possesses the oracle property in the terminology of Fan and Li (2001). In contrast, however, the AIC-type selector tends to overfit with positive probability. We further show that the AIC-type selector is asymptotically loss efficient, while the BIC-type selector is not. Our simulation results confirm these theoretical findings, and an empirical example is presented. Some technical proofs are given in the online supplementary material.

UR - http://www.scopus.com/inward/record.url?scp=77952580367&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77952580367&partnerID=8YFLogxK

U2 - 10.1198/jasa.2009.tm08013

DO - 10.1198/jasa.2009.tm08013

M3 - Article

C2 - 20676354

AN - SCOPUS:77952580367

VL - 105

SP - 312

EP - 323

JO - Journal of the American Statistical Association

JF - Journal of the American Statistical Association

SN - 0162-1459

IS - 489

ER -