Parsimonious topic models with salient word discovery

Hossein Soleimani, David Jonathan Miller

    Research output: Contribution to journalArticle

    15 Scopus citations

    Abstract

    We propose a parsimonious topic model for text corpora. In related models such as Latent Dirichlet Allocation (LDA), all words are modeled topic-specifically, even though many words occur with similar frequencies across different topics. Our modeling determines salient words for each topic, which have topic-specific probabilities, with the rest explained by a universal shared model. Further, in LDA all topics are in principle present in every document. By contrast, our model gives sparse topic representation, determining the (small) subset of relevant topics for each document. We derive a Bayesian Information Criterion (BIC), balancing model complexity and goodness of fit. Here, interestingly, we identify an effective sample size and corresponding penalty specific to each parameter type in our model. We minimize BIC to jointly determine our entire model - the topic-specific words, document-specific topics, all model parameter values, and the total number of topics - in a wholly unsupervised fashion. Results on three text corpora and an image dataset show that our model achieves higher test set likelihood and better agreement with ground-truth class labels, compared to LDA and to a model designed to incorporate sparsity.

    Original languageEnglish (US)
    Article number6871387
    Pages (from-to)824-837
    Number of pages14
    JournalIEEE Transactions on Knowledge and Data Engineering
    Volume27
    Issue number3
    DOIs
    StatePublished - Mar 1 2015

    All Science Journal Classification (ASJC) codes

    • Information Systems
    • Computer Science Applications
    • Computational Theory and Mathematics

    Fingerprint Dive into the research topics of 'Parsimonious topic models with salient word discovery'. Together they form a unique fingerprint.

  • Cite this