An extension of iterative scaling for decision and data aggregation in ensemble classification

Siddharth Pal, David Jonathan Miller

    Research output: Contribution to journalArticle

    4 Citations (Scopus)

    Abstract

    Improved iterative scaling (IIS) is an algorithm for learning maximum entropy (ME) joint and conditional probability models, consistent with specified constraints, that has found great utility in natural language processing and related applications. In most IIS work on classification, discrete-valued "feature functions" are considered, depending on the data observations and class label, with constraints measured based on frequency counts, taken over hard (0-1) training set instances. Here, we consider the case where the training (and test) set consist of instances of probability mass functions on the features, rather than hard feature values. IIS extends in a natural way for this case. This has applications (1) to ME classification on mixed discrete-continuous feature spaces and (2) to ME aggregation of soft classifier decisions in ensemble classification. Moreover, we combine these methods, yielding a method, with proved learning convergence, that jointly performs (soft) decision-level and feature-level fusion in making ensemble decisions. We demonstrate favorable comparisons against standard Adaboost.M1, input-dependent boosting, and other supervised combining methods, on data sets from the UC Irvine Machine Learning repository.

    Original languageEnglish (US)
    Pages (from-to)21-37
    Number of pages17
    JournalJournal of VLSI Signal Processing Systems for Signal, Image, and Video Technology
    Volume48
    Issue number1-2
    DOIs
    StatePublished - Aug 1 2007

    Fingerprint

    Data Aggregation
    Maximum Entropy
    Ensemble
    Entropy
    Agglomeration
    Scaling
    Conditional Model
    Adaptive boosting
    AdaBoost
    Probability Model
    Boosting
    Test Set
    Conditional probability
    Feature Space
    Repository
    Natural Language
    Learning systems
    Labels
    Aggregation
    Fusion

    All Science Journal Classification (ASJC) codes

    • Signal Processing
    • Information Systems
    • Electrical and Electronic Engineering

    Cite this

    @article{e7733b4f09e14a1e8182857a5ac70eae,
    title = "An extension of iterative scaling for decision and data aggregation in ensemble classification",
    abstract = "Improved iterative scaling (IIS) is an algorithm for learning maximum entropy (ME) joint and conditional probability models, consistent with specified constraints, that has found great utility in natural language processing and related applications. In most IIS work on classification, discrete-valued {"}feature functions{"} are considered, depending on the data observations and class label, with constraints measured based on frequency counts, taken over hard (0-1) training set instances. Here, we consider the case where the training (and test) set consist of instances of probability mass functions on the features, rather than hard feature values. IIS extends in a natural way for this case. This has applications (1) to ME classification on mixed discrete-continuous feature spaces and (2) to ME aggregation of soft classifier decisions in ensemble classification. Moreover, we combine these methods, yielding a method, with proved learning convergence, that jointly performs (soft) decision-level and feature-level fusion in making ensemble decisions. We demonstrate favorable comparisons against standard Adaboost.M1, input-dependent boosting, and other supervised combining methods, on data sets from the UC Irvine Machine Learning repository.",
    author = "Siddharth Pal and Miller, {David Jonathan}",
    year = "2007",
    month = "8",
    day = "1",
    doi = "10.1007/s11265-006-0009-6",
    language = "English (US)",
    volume = "48",
    pages = "21--37",
    journal = "Journal of Signal Processing Systems",
    issn = "1939-8018",
    publisher = "Springer New York",
    number = "1-2",

    }

    TY - JOUR

    T1 - An extension of iterative scaling for decision and data aggregation in ensemble classification

    AU - Pal, Siddharth

    AU - Miller, David Jonathan

    PY - 2007/8/1

    Y1 - 2007/8/1

    N2 - Improved iterative scaling (IIS) is an algorithm for learning maximum entropy (ME) joint and conditional probability models, consistent with specified constraints, that has found great utility in natural language processing and related applications. In most IIS work on classification, discrete-valued "feature functions" are considered, depending on the data observations and class label, with constraints measured based on frequency counts, taken over hard (0-1) training set instances. Here, we consider the case where the training (and test) set consist of instances of probability mass functions on the features, rather than hard feature values. IIS extends in a natural way for this case. This has applications (1) to ME classification on mixed discrete-continuous feature spaces and (2) to ME aggregation of soft classifier decisions in ensemble classification. Moreover, we combine these methods, yielding a method, with proved learning convergence, that jointly performs (soft) decision-level and feature-level fusion in making ensemble decisions. We demonstrate favorable comparisons against standard Adaboost.M1, input-dependent boosting, and other supervised combining methods, on data sets from the UC Irvine Machine Learning repository.

    AB - Improved iterative scaling (IIS) is an algorithm for learning maximum entropy (ME) joint and conditional probability models, consistent with specified constraints, that has found great utility in natural language processing and related applications. In most IIS work on classification, discrete-valued "feature functions" are considered, depending on the data observations and class label, with constraints measured based on frequency counts, taken over hard (0-1) training set instances. Here, we consider the case where the training (and test) set consist of instances of probability mass functions on the features, rather than hard feature values. IIS extends in a natural way for this case. This has applications (1) to ME classification on mixed discrete-continuous feature spaces and (2) to ME aggregation of soft classifier decisions in ensemble classification. Moreover, we combine these methods, yielding a method, with proved learning convergence, that jointly performs (soft) decision-level and feature-level fusion in making ensemble decisions. We demonstrate favorable comparisons against standard Adaboost.M1, input-dependent boosting, and other supervised combining methods, on data sets from the UC Irvine Machine Learning repository.

    UR - http://www.scopus.com/inward/record.url?scp=34547307226&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=34547307226&partnerID=8YFLogxK

    U2 - 10.1007/s11265-006-0009-6

    DO - 10.1007/s11265-006-0009-6

    M3 - Article

    AN - SCOPUS:34547307226

    VL - 48

    SP - 21

    EP - 37

    JO - Journal of Signal Processing Systems

    JF - Journal of Signal Processing Systems

    SN - 1939-8018

    IS - 1-2

    ER -