Feature selection in Web applications by ROC inflections and powerset pruning

F. M. Coetzee, E. Glover, S. Lawrence, C. L. Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations

Abstract

A basic problem of information processing is selecting enough features to ensure that events are accurately represented for classification problems, while simultaneously minimizing storage and processing of irrelevant or marginally important features. To address this problem, feature selection procedures perform a search through the feature power set to find the smallest subset meeting performance requirements. Major restrictions of existing procedures are that they typically explicitly or implicitly assume a fixed operating point, and make limited use of the statistical structure of the feature power set. We present a method that combines the Neyman-Pearson design procedure on finite data, with the directed set structure of the Receiver Operating Curves on the feature subsets, to determine the maximal size of the feature subsets that can be ranked in a given problem. The search can then be restricted to the smaller subsets, resulting in significant reductions in computational complexity. Optimizing the overall Receiver Operating Curve also allows for end users to select different operating points and cost functions to optimize. The algorithm also produces a natural method of Boolean representation of the minimal feature combinations that best describe the data near a given operating point. These representations are especially appropriate when describing data using common text-related features useful on the Web, such as thresholded TFIDF data. We show how to use these results to perform automatic Boolean query modification generation for distributed databases, such as niche metasearch engines.

Original languageEnglish (US)
Title of host publicationProceedings - 2001 Symposium on Applications and the Internet, SAINT 2001
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages5-14
Number of pages10
ISBN (Electronic)0769509428, 9780769509426
DOIs
StatePublished - 2001
EventSymposium on Applications and the Internet, SAINT 2001 - San Diego, United States
Duration: Jan 8 2001Jan 12 2001

Publication series

NameProceedings - 2001 Symposium on Applications and the Internet, SAINT 2001

Other

OtherSymposium on Applications and the Internet, SAINT 2001
CountryUnited States
CitySan Diego
Period1/8/011/12/01

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Software

Fingerprint Dive into the research topics of 'Feature selection in Web applications by ROC inflections and powerset pruning'. Together they form a unique fingerprint.

Cite this