Unsupervised learning of parsimonious mixtures on large spaces with integrated feature and component selection

Michael W. Graham, David Jonathan Miller

    Research output: Contribution to journalArticle

    39 Scopus citations

    Abstract

    Estimating the number of components (the order) in a mixture model is often addressed using criteria such as the Bayesian information criterion (BIC) and minimum message length. However, when the feature space is very large, use of these criteria may grossly underestimate the order. Here, it is suggested that this failure is not mainly attributable to the criterion (e.g., BIC), but rather to the lack of "structure" in standard mixtures - these models trade off data fitness and model complexity only by varying the order. The authors of the present paper propose mixtures with a richer set of tradeoffs. The proposed model allows each component its own informative feature subset, with all other features explained by a common model (shared by all components). Parameter sharing greatly reduces complexity at a given order. Since the space of these parsimonious modeling solutions is vast, this space is searched in an efficient manner, integrating the component and feature selection within the generalized expectation-maximization (GEM) learning for the mixture parameters. The quality of the proposed (unsupervised) solutions is evaluated using both classification error and test set data likelihood. On text data, the proposed multinomial version - learned without labeled examples, without knowing the "true" number of topics, and without feature preprocessing - compares quite favorably with both alternative unsupervised methods and with a supervised naive Bayes classifier. A Gaussian version compares favorably with a recent method introducing "feature saliency" in mixtures.

    Original languageEnglish (US)
    Pages (from-to)1289-1303
    Number of pages15
    JournalIEEE Transactions on Signal Processing
    Volume54
    Issue number4
    DOIs
    StatePublished - Apr 1 2006

    All Science Journal Classification (ASJC) codes

    • Signal Processing
    • Electrical and Electronic Engineering

    Fingerprint Dive into the research topics of 'Unsupervised learning of parsimonious mixtures on large spaces with integrated feature and component selection'. Together they form a unique fingerprint.

  • Cite this