Joint parsimonious modeling and model order selection for multivariate gaussian mixtures

Scott C. Markley, David Jonathan Miller

    Research output: Contribution to journalArticle

    10 Citations (Scopus)

    Abstract

    Multivariate Gaussian mixture models (GMMs) are widely for density estimation, model-based data clustering, and statistical classification. A difficult problem is estimating the model order, i.e., the number of mixture components, and model structure. Use of full covariance matrices, with number of parameters quadratic in the feature dimension, entails high model complexity, and thus may underestimate order, while naive Bayes mixtures may introduce model bias and lead to order overestimates. We develop a parsimonious modeling and model order and structure selection method for GMMs which allows for and optimizes over parameter tying configurations across mixture components applied to each individual parameter, including the covariates. We derive a generalized Expectation-Maximization algorithm for [(Bayesian information criterion (BIC)-based] penalized likelihood minimization. This, coupled with sequential model order reduction, forms our joint learning and model selection. Our method searches over a rich space of models and, consistent with minimizing BIC, achieves fine-grained matching of model complexity to the available data. We have found our method to be effective and largely robust in learning accurate model orders and parameter-tying structures for simulated ground-truth mixtures. We compared against naive Bayes and standard full-covariance GMMs for several criteria: 1) model order and structure accuracy (for synthetic data sets); 2) test set log-likelihood; 3) unsupervised classification accuracy; and 4) accuracy when class-conditional mixtures are used in a plug-in Bayes classifier. Our method, which chooses model orders intermediate between standard and naive Bayes GMMs, gives improved accuracy with respect to each of these performance measures.

    Original languageEnglish (US)
    Article number5447637
    Pages (from-to)548-559
    Number of pages12
    JournalIEEE Journal on Selected Topics in Signal Processing
    Volume4
    Issue number3
    DOIs
    StatePublished - Jun 1 2010

    Fingerprint

    Model structures
    Covariance matrix
    Classifiers

    All Science Journal Classification (ASJC) codes

    • Signal Processing
    • Electrical and Electronic Engineering

    Cite this

    @article{949a4d02dfc842889f4b12c3dc933285,
    title = "Joint parsimonious modeling and model order selection for multivariate gaussian mixtures",
    abstract = "Multivariate Gaussian mixture models (GMMs) are widely for density estimation, model-based data clustering, and statistical classification. A difficult problem is estimating the model order, i.e., the number of mixture components, and model structure. Use of full covariance matrices, with number of parameters quadratic in the feature dimension, entails high model complexity, and thus may underestimate order, while naive Bayes mixtures may introduce model bias and lead to order overestimates. We develop a parsimonious modeling and model order and structure selection method for GMMs which allows for and optimizes over parameter tying configurations across mixture components applied to each individual parameter, including the covariates. We derive a generalized Expectation-Maximization algorithm for [(Bayesian information criterion (BIC)-based] penalized likelihood minimization. This, coupled with sequential model order reduction, forms our joint learning and model selection. Our method searches over a rich space of models and, consistent with minimizing BIC, achieves fine-grained matching of model complexity to the available data. We have found our method to be effective and largely robust in learning accurate model orders and parameter-tying structures for simulated ground-truth mixtures. We compared against naive Bayes and standard full-covariance GMMs for several criteria: 1) model order and structure accuracy (for synthetic data sets); 2) test set log-likelihood; 3) unsupervised classification accuracy; and 4) accuracy when class-conditional mixtures are used in a plug-in Bayes classifier. Our method, which chooses model orders intermediate between standard and naive Bayes GMMs, gives improved accuracy with respect to each of these performance measures.",
    author = "Markley, {Scott C.} and Miller, {David Jonathan}",
    year = "2010",
    month = "6",
    day = "1",
    doi = "10.1109/JSTSP.2009.2038312",
    language = "English (US)",
    volume = "4",
    pages = "548--559",
    journal = "IEEE Journal on Selected Topics in Signal Processing",
    issn = "1932-4553",
    publisher = "Institute of Electrical and Electronics Engineers Inc.",
    number = "3",

    }

    Joint parsimonious modeling and model order selection for multivariate gaussian mixtures. / Markley, Scott C.; Miller, David Jonathan.

    In: IEEE Journal on Selected Topics in Signal Processing, Vol. 4, No. 3, 5447637, 01.06.2010, p. 548-559.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - Joint parsimonious modeling and model order selection for multivariate gaussian mixtures

    AU - Markley, Scott C.

    AU - Miller, David Jonathan

    PY - 2010/6/1

    Y1 - 2010/6/1

    N2 - Multivariate Gaussian mixture models (GMMs) are widely for density estimation, model-based data clustering, and statistical classification. A difficult problem is estimating the model order, i.e., the number of mixture components, and model structure. Use of full covariance matrices, with number of parameters quadratic in the feature dimension, entails high model complexity, and thus may underestimate order, while naive Bayes mixtures may introduce model bias and lead to order overestimates. We develop a parsimonious modeling and model order and structure selection method for GMMs which allows for and optimizes over parameter tying configurations across mixture components applied to each individual parameter, including the covariates. We derive a generalized Expectation-Maximization algorithm for [(Bayesian information criterion (BIC)-based] penalized likelihood minimization. This, coupled with sequential model order reduction, forms our joint learning and model selection. Our method searches over a rich space of models and, consistent with minimizing BIC, achieves fine-grained matching of model complexity to the available data. We have found our method to be effective and largely robust in learning accurate model orders and parameter-tying structures for simulated ground-truth mixtures. We compared against naive Bayes and standard full-covariance GMMs for several criteria: 1) model order and structure accuracy (for synthetic data sets); 2) test set log-likelihood; 3) unsupervised classification accuracy; and 4) accuracy when class-conditional mixtures are used in a plug-in Bayes classifier. Our method, which chooses model orders intermediate between standard and naive Bayes GMMs, gives improved accuracy with respect to each of these performance measures.

    AB - Multivariate Gaussian mixture models (GMMs) are widely for density estimation, model-based data clustering, and statistical classification. A difficult problem is estimating the model order, i.e., the number of mixture components, and model structure. Use of full covariance matrices, with number of parameters quadratic in the feature dimension, entails high model complexity, and thus may underestimate order, while naive Bayes mixtures may introduce model bias and lead to order overestimates. We develop a parsimonious modeling and model order and structure selection method for GMMs which allows for and optimizes over parameter tying configurations across mixture components applied to each individual parameter, including the covariates. We derive a generalized Expectation-Maximization algorithm for [(Bayesian information criterion (BIC)-based] penalized likelihood minimization. This, coupled with sequential model order reduction, forms our joint learning and model selection. Our method searches over a rich space of models and, consistent with minimizing BIC, achieves fine-grained matching of model complexity to the available data. We have found our method to be effective and largely robust in learning accurate model orders and parameter-tying structures for simulated ground-truth mixtures. We compared against naive Bayes and standard full-covariance GMMs for several criteria: 1) model order and structure accuracy (for synthetic data sets); 2) test set log-likelihood; 3) unsupervised classification accuracy; and 4) accuracy when class-conditional mixtures are used in a plug-in Bayes classifier. Our method, which chooses model orders intermediate between standard and naive Bayes GMMs, gives improved accuracy with respect to each of these performance measures.

    UR - http://www.scopus.com/inward/record.url?scp=77952602507&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=77952602507&partnerID=8YFLogxK

    U2 - 10.1109/JSTSP.2009.2038312

    DO - 10.1109/JSTSP.2009.2038312

    M3 - Article

    VL - 4

    SP - 548

    EP - 559

    JO - IEEE Journal on Selected Topics in Signal Processing

    JF - IEEE Journal on Selected Topics in Signal Processing

    SN - 1932-4553

    IS - 3

    M1 - 5447637

    ER -