Maximum entropy approach for optimal statistical classification

David Jonathan Miller, Ajit Rao, Kenneth Rose, Allen Gersho

Research output: Contribution to conferencePaper

2 Citations (Scopus)

Abstract

A global optimization technique is introduced for statistical classifier design to minimize the probability of classification error. The method, which is based on ideas from information theory and analogies to statistical physics, is inherently probabilistic. During the design phase, data are assigned to classes in probability, with the probability distributions chosen to maximize entropy subject to a constraint on the expected classification error. This entropy maximization problem is seen to be equivalent to a free energy minimization, motivating a deterministic annealing approach to minimize the misclassification cost. Our method is applicable to a variety of classifier structures, including nearest prototype, radial basis function, and multilayer perceptron-based classifiers. On standard benchmark examples, the method applied to nearest prototype classifier design achieves performance improvements over both the learning vector quantizer, as well as over multilayer perceptron classifiers designed by the standard back-propagation algorithm. Remarkably substantial performance gains over learning vector quantization are achieved for complicated mixture examples where there is significant class overlap.

Original languageEnglish (US)
Pages58-66
Number of pages9
StatePublished - Jan 1 1995
EventProceedings of the 5th IEEE Workshop on Neural Networks for Signal Processing (NNSP'95) - Cambridge, MA, USA
Duration: Aug 31 1995Sep 2 1995

Other

OtherProceedings of the 5th IEEE Workshop on Neural Networks for Signal Processing (NNSP'95)
CityCambridge, MA, USA
Period8/31/959/2/95

Fingerprint

Classifiers
Entropy
Multilayer neural networks
Backpropagation algorithms
Information theory
Vector quantization
Global optimization
Probability distributions
Free energy
Physics
Annealing
Costs

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Software
  • Electrical and Electronic Engineering

Cite this

Miller, D. J., Rao, A., Rose, K., & Gersho, A. (1995). Maximum entropy approach for optimal statistical classification. 58-66. Paper presented at Proceedings of the 5th IEEE Workshop on Neural Networks for Signal Processing (NNSP'95), Cambridge, MA, USA, .
Miller, David Jonathan ; Rao, Ajit ; Rose, Kenneth ; Gersho, Allen. / Maximum entropy approach for optimal statistical classification. Paper presented at Proceedings of the 5th IEEE Workshop on Neural Networks for Signal Processing (NNSP'95), Cambridge, MA, USA, .9 p.
@conference{c75d6b4b26a54eac84b151f32fd977c4,
title = "Maximum entropy approach for optimal statistical classification",
abstract = "A global optimization technique is introduced for statistical classifier design to minimize the probability of classification error. The method, which is based on ideas from information theory and analogies to statistical physics, is inherently probabilistic. During the design phase, data are assigned to classes in probability, with the probability distributions chosen to maximize entropy subject to a constraint on the expected classification error. This entropy maximization problem is seen to be equivalent to a free energy minimization, motivating a deterministic annealing approach to minimize the misclassification cost. Our method is applicable to a variety of classifier structures, including nearest prototype, radial basis function, and multilayer perceptron-based classifiers. On standard benchmark examples, the method applied to nearest prototype classifier design achieves performance improvements over both the learning vector quantizer, as well as over multilayer perceptron classifiers designed by the standard back-propagation algorithm. Remarkably substantial performance gains over learning vector quantization are achieved for complicated mixture examples where there is significant class overlap.",
author = "Miller, {David Jonathan} and Ajit Rao and Kenneth Rose and Allen Gersho",
year = "1995",
month = "1",
day = "1",
language = "English (US)",
pages = "58--66",
note = "Proceedings of the 5th IEEE Workshop on Neural Networks for Signal Processing (NNSP'95) ; Conference date: 31-08-1995 Through 02-09-1995",

}

Miller, DJ, Rao, A, Rose, K & Gersho, A 1995, 'Maximum entropy approach for optimal statistical classification' Paper presented at Proceedings of the 5th IEEE Workshop on Neural Networks for Signal Processing (NNSP'95), Cambridge, MA, USA, 8/31/95 - 9/2/95, pp. 58-66.

Maximum entropy approach for optimal statistical classification. / Miller, David Jonathan; Rao, Ajit; Rose, Kenneth; Gersho, Allen.

1995. 58-66 Paper presented at Proceedings of the 5th IEEE Workshop on Neural Networks for Signal Processing (NNSP'95), Cambridge, MA, USA, .

Research output: Contribution to conferencePaper

TY - CONF

T1 - Maximum entropy approach for optimal statistical classification

AU - Miller, David Jonathan

AU - Rao, Ajit

AU - Rose, Kenneth

AU - Gersho, Allen

PY - 1995/1/1

Y1 - 1995/1/1

N2 - A global optimization technique is introduced for statistical classifier design to minimize the probability of classification error. The method, which is based on ideas from information theory and analogies to statistical physics, is inherently probabilistic. During the design phase, data are assigned to classes in probability, with the probability distributions chosen to maximize entropy subject to a constraint on the expected classification error. This entropy maximization problem is seen to be equivalent to a free energy minimization, motivating a deterministic annealing approach to minimize the misclassification cost. Our method is applicable to a variety of classifier structures, including nearest prototype, radial basis function, and multilayer perceptron-based classifiers. On standard benchmark examples, the method applied to nearest prototype classifier design achieves performance improvements over both the learning vector quantizer, as well as over multilayer perceptron classifiers designed by the standard back-propagation algorithm. Remarkably substantial performance gains over learning vector quantization are achieved for complicated mixture examples where there is significant class overlap.

AB - A global optimization technique is introduced for statistical classifier design to minimize the probability of classification error. The method, which is based on ideas from information theory and analogies to statistical physics, is inherently probabilistic. During the design phase, data are assigned to classes in probability, with the probability distributions chosen to maximize entropy subject to a constraint on the expected classification error. This entropy maximization problem is seen to be equivalent to a free energy minimization, motivating a deterministic annealing approach to minimize the misclassification cost. Our method is applicable to a variety of classifier structures, including nearest prototype, radial basis function, and multilayer perceptron-based classifiers. On standard benchmark examples, the method applied to nearest prototype classifier design achieves performance improvements over both the learning vector quantizer, as well as over multilayer perceptron classifiers designed by the standard back-propagation algorithm. Remarkably substantial performance gains over learning vector quantization are achieved for complicated mixture examples where there is significant class overlap.

UR - http://www.scopus.com/inward/record.url?scp=0029238772&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0029238772&partnerID=8YFLogxK

M3 - Paper

SP - 58

EP - 66

ER -

Miller DJ, Rao A, Rose K, Gersho A. Maximum entropy approach for optimal statistical classification. 1995. Paper presented at Proceedings of the 5th IEEE Workshop on Neural Networks for Signal Processing (NNSP'95), Cambridge, MA, USA, .