A proof of convergence of the concave-convex procedure using Zangwill's theory

Bharath Kumar Sriperumbudur, Gert R.G. Lanckriet

Research output: Contribution to journalComment/debate

38 Citations (Scopus)

Abstract

The concave-convex procedure (CCCP) is an iterative algorithm that solves d.c. (difference of convex functions) programs as a sequence of convex programs. In machine learning,CCCP is extensively used in many learning algorithms, including sparse support vector machines (SVMs), transductive SVMs, and sparse principal component analysis. Though CCCP is widely used in many applications, its convergence behavior has not gotten a lot of specific attention. Yuille and Rangarajan analyzed its convergence in their original paper; however, we believe the analysis is not complete. The convergence of CCCP can be derived from the convergence of the d.c. algorithm (DCA), proposed in the global optimization literature to solve general d.c. programs, whose proof relies on d.c. duality. In this note, we follow a different reasoning and show how Zangwill's global convergence theory of iterative algorithms provides a natural framework to prove the convergence of CCCP. This underlines Zangwill's theory as a powerful and general framework to deal with the convergence issues of iterative algorithms, after also being used to prove the convergence of algorithms like expectation-maximization and generalized alternating minimization. In this note, we provide a rigorous analysis of the convergence of CCCP by addressing two questions:When does CCCP find a local minimum or a stationary point of the d.c. program under consideration? and when does the sequence generated by CCCP converge? We also present an open problem on the issue of local convergence of CCCP.

Original languageEnglish (US)
Pages (from-to)1391-1407
Number of pages17
JournalNeural computation
Volume24
Issue number6
DOIs
StatePublished - Dec 1 2012

Fingerprint

Principal Component Analysis
Learning
Support Vector Machine
Machine Learning
Duality

All Science Journal Classification (ASJC) codes

  • Arts and Humanities (miscellaneous)
  • Cognitive Neuroscience

Cite this

@article{1b1750630b6d43669781f2ede46522c0,
title = "A proof of convergence of the concave-convex procedure using Zangwill's theory",
abstract = "The concave-convex procedure (CCCP) is an iterative algorithm that solves d.c. (difference of convex functions) programs as a sequence of convex programs. In machine learning,CCCP is extensively used in many learning algorithms, including sparse support vector machines (SVMs), transductive SVMs, and sparse principal component analysis. Though CCCP is widely used in many applications, its convergence behavior has not gotten a lot of specific attention. Yuille and Rangarajan analyzed its convergence in their original paper; however, we believe the analysis is not complete. The convergence of CCCP can be derived from the convergence of the d.c. algorithm (DCA), proposed in the global optimization literature to solve general d.c. programs, whose proof relies on d.c. duality. In this note, we follow a different reasoning and show how Zangwill's global convergence theory of iterative algorithms provides a natural framework to prove the convergence of CCCP. This underlines Zangwill's theory as a powerful and general framework to deal with the convergence issues of iterative algorithms, after also being used to prove the convergence of algorithms like expectation-maximization and generalized alternating minimization. In this note, we provide a rigorous analysis of the convergence of CCCP by addressing two questions:When does CCCP find a local minimum or a stationary point of the d.c. program under consideration? and when does the sequence generated by CCCP converge? We also present an open problem on the issue of local convergence of CCCP.",
author = "Sriperumbudur, {Bharath Kumar} and Lanckriet, {Gert R.G.}",
year = "2012",
month = "12",
day = "1",
doi = "10.1162/NECO_a_00283",
language = "English (US)",
volume = "24",
pages = "1391--1407",
journal = "Neural Computation",
issn = "0899-7667",
publisher = "MIT Press Journals",
number = "6",

}

A proof of convergence of the concave-convex procedure using Zangwill's theory. / Sriperumbudur, Bharath Kumar; Lanckriet, Gert R.G.

In: Neural computation, Vol. 24, No. 6, 01.12.2012, p. 1391-1407.

Research output: Contribution to journalComment/debate

TY - JOUR

T1 - A proof of convergence of the concave-convex procedure using Zangwill's theory

AU - Sriperumbudur, Bharath Kumar

AU - Lanckriet, Gert R.G.

PY - 2012/12/1

Y1 - 2012/12/1

N2 - The concave-convex procedure (CCCP) is an iterative algorithm that solves d.c. (difference of convex functions) programs as a sequence of convex programs. In machine learning,CCCP is extensively used in many learning algorithms, including sparse support vector machines (SVMs), transductive SVMs, and sparse principal component analysis. Though CCCP is widely used in many applications, its convergence behavior has not gotten a lot of specific attention. Yuille and Rangarajan analyzed its convergence in their original paper; however, we believe the analysis is not complete. The convergence of CCCP can be derived from the convergence of the d.c. algorithm (DCA), proposed in the global optimization literature to solve general d.c. programs, whose proof relies on d.c. duality. In this note, we follow a different reasoning and show how Zangwill's global convergence theory of iterative algorithms provides a natural framework to prove the convergence of CCCP. This underlines Zangwill's theory as a powerful and general framework to deal with the convergence issues of iterative algorithms, after also being used to prove the convergence of algorithms like expectation-maximization and generalized alternating minimization. In this note, we provide a rigorous analysis of the convergence of CCCP by addressing two questions:When does CCCP find a local minimum or a stationary point of the d.c. program under consideration? and when does the sequence generated by CCCP converge? We also present an open problem on the issue of local convergence of CCCP.

AB - The concave-convex procedure (CCCP) is an iterative algorithm that solves d.c. (difference of convex functions) programs as a sequence of convex programs. In machine learning,CCCP is extensively used in many learning algorithms, including sparse support vector machines (SVMs), transductive SVMs, and sparse principal component analysis. Though CCCP is widely used in many applications, its convergence behavior has not gotten a lot of specific attention. Yuille and Rangarajan analyzed its convergence in their original paper; however, we believe the analysis is not complete. The convergence of CCCP can be derived from the convergence of the d.c. algorithm (DCA), proposed in the global optimization literature to solve general d.c. programs, whose proof relies on d.c. duality. In this note, we follow a different reasoning and show how Zangwill's global convergence theory of iterative algorithms provides a natural framework to prove the convergence of CCCP. This underlines Zangwill's theory as a powerful and general framework to deal with the convergence issues of iterative algorithms, after also being used to prove the convergence of algorithms like expectation-maximization and generalized alternating minimization. In this note, we provide a rigorous analysis of the convergence of CCCP by addressing two questions:When does CCCP find a local minimum or a stationary point of the d.c. program under consideration? and when does the sequence generated by CCCP converge? We also present an open problem on the issue of local convergence of CCCP.

UR - http://www.scopus.com/inward/record.url?scp=84874088405&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84874088405&partnerID=8YFLogxK

U2 - 10.1162/NECO_a_00283

DO - 10.1162/NECO_a_00283

M3 - Comment/debate

C2 - 22364501

AN - SCOPUS:84874088405

VL - 24

SP - 1391

EP - 1407

JO - Neural Computation

JF - Neural Computation

SN - 0899-7667

IS - 6

ER -