Joint parsimonious modeling and model order selection for multivariate gaussian mixtures

Scott C. Markley, David J. Miller

    Research output: Contribution to journalArticlepeer-review

    12 Scopus citations


    Multivariate Gaussian mixture models (GMMs) are widely for density estimation, model-based data clustering, and statistical classification. A difficult problem is estimating the model order, i.e., the number of mixture components, and model structure. Use of full covariance matrices, with number of parameters quadratic in the feature dimension, entails high model complexity, and thus may underestimate order, while naive Bayes mixtures may introduce model bias and lead to order overestimates. We develop a parsimonious modeling and model order and structure selection method for GMMs which allows for and optimizes over parameter tying configurations across mixture components applied to each individual parameter, including the covariates. We derive a generalized Expectation-Maximization algorithm for [(Bayesian information criterion (BIC)-based] penalized likelihood minimization. This, coupled with sequential model order reduction, forms our joint learning and model selection. Our method searches over a rich space of models and, consistent with minimizing BIC, achieves fine-grained matching of model complexity to the available data. We have found our method to be effective and largely robust in learning accurate model orders and parameter-tying structures for simulated ground-truth mixtures. We compared against naive Bayes and standard full-covariance GMMs for several criteria: 1) model order and structure accuracy (for synthetic data sets); 2) test set log-likelihood; 3) unsupervised classification accuracy; and 4) accuracy when class-conditional mixtures are used in a plug-in Bayes classifier. Our method, which chooses model orders intermediate between standard and naive Bayes GMMs, gives improved accuracy with respect to each of these performance measures.

    Original languageEnglish (US)
    Article number5447637
    Pages (from-to)548-559
    Number of pages12
    JournalIEEE Journal on Selected Topics in Signal Processing
    Issue number3
    StatePublished - Jun 2010

    All Science Journal Classification (ASJC) codes

    • Signal Processing
    • Electrical and Electronic Engineering


    Dive into the research topics of 'Joint parsimonious modeling and model order selection for multivariate gaussian mixtures'. Together they form a unique fingerprint.

    Cite this