Hierarchical, Unsupervised Learning with Growing via Phase Transitions

David Jonathan Miller, Kenneth Rose

    Research output: Contribution to journalArticle

    15 Citations (Scopus)

    Abstract

    We address unsupervised learning subject to structural constraints, with particular emphasis placed on clustering with an imposed decision tree structure. Most known methods are greedy, optimizing one node of the tree at a time to minimize a local cost. By constrast, we develop a joint optimization method, derived based on information-theoretic principles and closely related to known methods in statistical physics. The approach is inspired by the deterministic annealing algorithm for unstructured data clustering, which was based on maximum entropy inference. The new approach is founded on the principle of minimum cross-entropy, using informative priors to approximate the unstructured clustering solution while imposing the structural constraint. The resulting method incorporates supervised learning principles applied in an unsupervised problem setting. In our approach, the tree "grows" by a sequence of bifurcations that occur while optimizing an effective free energy cost at decreasing temperature scales. Thus, estimates of the tree size and structure are naturally obtained at each temperature in the process. Examples demonstrate considerable improvement over known methods.

    Original languageEnglish (US)
    Pages (from-to)425-450
    Number of pages26
    JournalNeural Computation
    Volume8
    Issue number2
    DOIs
    StatePublished - Feb 15 1996

    Fingerprint

    Phase Transition
    Learning
    Cluster Analysis
    Entropy
    Costs and Cost Analysis
    Decision Trees
    Temperature
    Physics
    Unsupervised Learning
    Costs

    All Science Journal Classification (ASJC) codes

    • Arts and Humanities (miscellaneous)
    • Cognitive Neuroscience

    Cite this

    @article{82ebbfe5611d4adfbf0bcab1df24a07c,
    title = "Hierarchical, Unsupervised Learning with Growing via Phase Transitions",
    abstract = "We address unsupervised learning subject to structural constraints, with particular emphasis placed on clustering with an imposed decision tree structure. Most known methods are greedy, optimizing one node of the tree at a time to minimize a local cost. By constrast, we develop a joint optimization method, derived based on information-theoretic principles and closely related to known methods in statistical physics. The approach is inspired by the deterministic annealing algorithm for unstructured data clustering, which was based on maximum entropy inference. The new approach is founded on the principle of minimum cross-entropy, using informative priors to approximate the unstructured clustering solution while imposing the structural constraint. The resulting method incorporates supervised learning principles applied in an unsupervised problem setting. In our approach, the tree {"}grows{"} by a sequence of bifurcations that occur while optimizing an effective free energy cost at decreasing temperature scales. Thus, estimates of the tree size and structure are naturally obtained at each temperature in the process. Examples demonstrate considerable improvement over known methods.",
    author = "Miller, {David Jonathan} and Kenneth Rose",
    year = "1996",
    month = "2",
    day = "15",
    doi = "10.1162/neco.1996.8.2.425",
    language = "English (US)",
    volume = "8",
    pages = "425--450",
    journal = "Neural Computation",
    issn = "0899-7667",
    publisher = "MIT Press Journals",
    number = "2",

    }

    Hierarchical, Unsupervised Learning with Growing via Phase Transitions. / Miller, David Jonathan; Rose, Kenneth.

    In: Neural Computation, Vol. 8, No. 2, 15.02.1996, p. 425-450.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - Hierarchical, Unsupervised Learning with Growing via Phase Transitions

    AU - Miller, David Jonathan

    AU - Rose, Kenneth

    PY - 1996/2/15

    Y1 - 1996/2/15

    N2 - We address unsupervised learning subject to structural constraints, with particular emphasis placed on clustering with an imposed decision tree structure. Most known methods are greedy, optimizing one node of the tree at a time to minimize a local cost. By constrast, we develop a joint optimization method, derived based on information-theoretic principles and closely related to known methods in statistical physics. The approach is inspired by the deterministic annealing algorithm for unstructured data clustering, which was based on maximum entropy inference. The new approach is founded on the principle of minimum cross-entropy, using informative priors to approximate the unstructured clustering solution while imposing the structural constraint. The resulting method incorporates supervised learning principles applied in an unsupervised problem setting. In our approach, the tree "grows" by a sequence of bifurcations that occur while optimizing an effective free energy cost at decreasing temperature scales. Thus, estimates of the tree size and structure are naturally obtained at each temperature in the process. Examples demonstrate considerable improvement over known methods.

    AB - We address unsupervised learning subject to structural constraints, with particular emphasis placed on clustering with an imposed decision tree structure. Most known methods are greedy, optimizing one node of the tree at a time to minimize a local cost. By constrast, we develop a joint optimization method, derived based on information-theoretic principles and closely related to known methods in statistical physics. The approach is inspired by the deterministic annealing algorithm for unstructured data clustering, which was based on maximum entropy inference. The new approach is founded on the principle of minimum cross-entropy, using informative priors to approximate the unstructured clustering solution while imposing the structural constraint. The resulting method incorporates supervised learning principles applied in an unsupervised problem setting. In our approach, the tree "grows" by a sequence of bifurcations that occur while optimizing an effective free energy cost at decreasing temperature scales. Thus, estimates of the tree size and structure are naturally obtained at each temperature in the process. Examples demonstrate considerable improvement over known methods.

    UR - http://www.scopus.com/inward/record.url?scp=0012292950&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=0012292950&partnerID=8YFLogxK

    U2 - 10.1162/neco.1996.8.2.425

    DO - 10.1162/neco.1996.8.2.425

    M3 - Article

    AN - SCOPUS:0012292950

    VL - 8

    SP - 425

    EP - 450

    JO - Neural Computation

    JF - Neural Computation

    SN - 0899-7667

    IS - 2

    ER -