Approximate maximum entropy joint feature inference consistent with arbitrary lower-order probability constraints: Application to statistical classification

David J. Miller, Lian Yan

    Research output: Contribution to journalArticlepeer-review

    8 Scopus citations

    Abstract

    We propose a new learning method for discrete space statistical classifiers. Similar to Chow and Liu (1968) and Cheeseman (1983), we cast classification/inference within the more general framework of estimating the joint probability mass function (p.m.f.) for the (feature vector, class label) pair. Cheeseman's proposal to build the maximum entropy (ME) joint p.m.f. consistent with general lower-order probability constraints is in principle powerful, allowing general dependencies between features. However, enormous learning complexity has severely limited the use of this approach. Alternative models such as Bayesian networks (BNs) require explicit determination of conditional independencies. These may be difficult to assess given limited data. Here we propose an approximate ME method, which, like previous methods, incorporates general constraints while retaining quite tractable learning. The new method restricts joint p.m.f. support during learning to a small subset of the full feature space. Classification gains are realized over dependence trees, tree-augmented naive Bayes networks, BNs trained by the Kutato algorithm, and multilayer perceptrons. Extensions to more general inference problems are indicated. We also propose a novel exact inference method when there are several missing features.

    Original languageEnglish (US)
    Pages (from-to)2175-2207
    Number of pages33
    JournalNeural computation
    Volume12
    Issue number9
    DOIs
    StatePublished - Sep 2000

    All Science Journal Classification (ASJC) codes

    • Arts and Humanities (miscellaneous)
    • Cognitive Neuroscience

    Fingerprint Dive into the research topics of 'Approximate maximum entropy joint feature inference consistent with arbitrary lower-order probability constraints: Application to statistical classification'. Together they form a unique fingerprint.

    Cite this