Separating the wheat from the chaff

Applications of automated document classification using support vector machines

Vito D'Orazio, Steven T. Landis, Glenn Hunter Palmer, Philip A Schrodt

Research output: Contribution to journalArticle

21 Citations (Scopus)

Abstract

Due in large part to the proliferation of digitized text, much of it available for little or no cost from the Internet, political science research has experienced a substantial increase in the number of data sets and large-n research initiatives. As the ability to collect detailed information on events of interest expands, so does the need to efficiently sort through the volumes of available information. Automated document classification presents a particularly attractive methodology for accomplishing this task. It is efficient, widely applicable to a variety of data collection efforts, and considerably flexible in tailoring its application for specific research needs. This article offers a holistic review of the application of automated document classification for data collection in political science research by discussing the process in its entirety. We argue that the application of a two-stage support vector machine (SVM) classification process offers advantages over other wellknown alternatives, due to the nature of SVMs being a discriminative classifier and having the ability to effectively address two primary attributes of textual data: high dimensionality and extreme sparseness. Evidence for this claim is presented through a discussion of the efficiency gains derived from using automated document classification on the Militarized Interstate Dispute 4 (MID4) data collection project.

Original languageEnglish (US)
Article numbermpt030
Pages (from-to)224-242
Number of pages19
JournalPolitical Analysis
Volume22
Issue number2
DOIs
StatePublished - Jan 1 2014

Fingerprint

political science
ability
available information
proliferation
Internet
efficiency
event
methodology
costs
evidence

All Science Journal Classification (ASJC) codes

  • Sociology and Political Science
  • Political Science and International Relations

Cite this

@article{4502bf3cd0a44ffcbf2174fc26c65956,
title = "Separating the wheat from the chaff: Applications of automated document classification using support vector machines",
abstract = "Due in large part to the proliferation of digitized text, much of it available for little or no cost from the Internet, political science research has experienced a substantial increase in the number of data sets and large-n research initiatives. As the ability to collect detailed information on events of interest expands, so does the need to efficiently sort through the volumes of available information. Automated document classification presents a particularly attractive methodology for accomplishing this task. It is efficient, widely applicable to a variety of data collection efforts, and considerably flexible in tailoring its application for specific research needs. This article offers a holistic review of the application of automated document classification for data collection in political science research by discussing the process in its entirety. We argue that the application of a two-stage support vector machine (SVM) classification process offers advantages over other wellknown alternatives, due to the nature of SVMs being a discriminative classifier and having the ability to effectively address two primary attributes of textual data: high dimensionality and extreme sparseness. Evidence for this claim is presented through a discussion of the efficiency gains derived from using automated document classification on the Militarized Interstate Dispute 4 (MID4) data collection project.",
author = "Vito D'Orazio and Landis, {Steven T.} and Palmer, {Glenn Hunter} and Schrodt, {Philip A}",
year = "2014",
month = "1",
day = "1",
doi = "10.1093/pan/mpt030",
language = "English (US)",
volume = "22",
pages = "224--242",
journal = "Political Analysis",
issn = "1047-1987",
publisher = "Oxford University Press",
number = "2",

}

Separating the wheat from the chaff : Applications of automated document classification using support vector machines. / D'Orazio, Vito; Landis, Steven T.; Palmer, Glenn Hunter; Schrodt, Philip A.

In: Political Analysis, Vol. 22, No. 2, mpt030, 01.01.2014, p. 224-242.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Separating the wheat from the chaff

T2 - Applications of automated document classification using support vector machines

AU - D'Orazio, Vito

AU - Landis, Steven T.

AU - Palmer, Glenn Hunter

AU - Schrodt, Philip A

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Due in large part to the proliferation of digitized text, much of it available for little or no cost from the Internet, political science research has experienced a substantial increase in the number of data sets and large-n research initiatives. As the ability to collect detailed information on events of interest expands, so does the need to efficiently sort through the volumes of available information. Automated document classification presents a particularly attractive methodology for accomplishing this task. It is efficient, widely applicable to a variety of data collection efforts, and considerably flexible in tailoring its application for specific research needs. This article offers a holistic review of the application of automated document classification for data collection in political science research by discussing the process in its entirety. We argue that the application of a two-stage support vector machine (SVM) classification process offers advantages over other wellknown alternatives, due to the nature of SVMs being a discriminative classifier and having the ability to effectively address two primary attributes of textual data: high dimensionality and extreme sparseness. Evidence for this claim is presented through a discussion of the efficiency gains derived from using automated document classification on the Militarized Interstate Dispute 4 (MID4) data collection project.

AB - Due in large part to the proliferation of digitized text, much of it available for little or no cost from the Internet, political science research has experienced a substantial increase in the number of data sets and large-n research initiatives. As the ability to collect detailed information on events of interest expands, so does the need to efficiently sort through the volumes of available information. Automated document classification presents a particularly attractive methodology for accomplishing this task. It is efficient, widely applicable to a variety of data collection efforts, and considerably flexible in tailoring its application for specific research needs. This article offers a holistic review of the application of automated document classification for data collection in political science research by discussing the process in its entirety. We argue that the application of a two-stage support vector machine (SVM) classification process offers advantages over other wellknown alternatives, due to the nature of SVMs being a discriminative classifier and having the ability to effectively address two primary attributes of textual data: high dimensionality and extreme sparseness. Evidence for this claim is presented through a discussion of the efficiency gains derived from using automated document classification on the Militarized Interstate Dispute 4 (MID4) data collection project.

UR - http://www.scopus.com/inward/record.url?scp=84898622561&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84898622561&partnerID=8YFLogxK

U2 - 10.1093/pan/mpt030

DO - 10.1093/pan/mpt030

M3 - Article

VL - 22

SP - 224

EP - 242

JO - Political Analysis

JF - Political Analysis

SN - 1047-1987

IS - 2

M1 - mpt030

ER -