Measuring conference quality by mining program committee characteristics

Ziming Zhuang, Ergin Elmacioglu, Dongwon Lee, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

32 Citations (Scopus)

Abstract

Bibliometrics are important measures for venue quality in digital libraries. Impacts of venues are usually the major consideration for subscription decision-making, and for ranking and recommending high-quality venues and documents. For digital libraries in the Computer Science literature domain, conferences play a major role as an important publication and dissemination outlet. However, with a recent profusion of conferences and rapidly expanding fields, it is increasingly challenging for researchers and librarians to assess the quality of conferences. We propose a set of novel heuristics to automatically discover prestigious (and low-quality) conferences by mining the characteristics of Program Committee members. We examine the proposed cues both in isolation and combination under a classification scheme. Evaluation on a collection of 2,979 conferences and 16,147 PC members shows that our heuristics, when combined, correctly classify about 92% of the conferences, with a low false positive rate of 0.035 and a recall of more than 73% for identifying reputable conferences. Furthermore, we demonstrate empirically that our heuristics can also effectively detect a set of low-quality conferences, with a false positive rate of merely 0.002. We also report our experience of detecting two previously unknown low-quality conferences. Finally, we apply the proposed techniques to the entire quality spectrum by ranking conferences in the collection.

Original languageEnglish (US)
Title of host publicationProceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007
Subtitle of host publicationBuilding and Sustaining the Digital Environment
Pages225-234
Number of pages10
DOIs
StatePublished - Nov 29 2007
Event7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007: Building and Sustaining the Digital Environment - Vancouver, BC, Canada
Duration: Jun 18 2007Jun 23 2007

Publication series

NameProceedings of the ACM International Conference on Digital Libraries

Other

Other7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007: Building and Sustaining the Digital Environment
CountryCanada
CityVancouver, BC
Period6/18/076/23/07

Fingerprint

Digital libraries
Computer science
Decision making
heuristics
ranking
subscription
PC
computer science
social isolation
librarian
decision making

All Science Journal Classification (ASJC) codes

  • Software
  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences

Cite this

Zhuang, Z., Elmacioglu, E., Lee, D., & Giles, C. L. (2007). Measuring conference quality by mining program committee characteristics. In Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007: Building and Sustaining the Digital Environment (pp. 225-234). (Proceedings of the ACM International Conference on Digital Libraries). https://doi.org/10.1145/1255175.1255220
Zhuang, Ziming ; Elmacioglu, Ergin ; Lee, Dongwon ; Giles, C. Lee. / Measuring conference quality by mining program committee characteristics. Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007: Building and Sustaining the Digital Environment. 2007. pp. 225-234 (Proceedings of the ACM International Conference on Digital Libraries).
@inproceedings{0a04c97c17a947fdbb7126b0b879746f,
title = "Measuring conference quality by mining program committee characteristics",
abstract = "Bibliometrics are important measures for venue quality in digital libraries. Impacts of venues are usually the major consideration for subscription decision-making, and for ranking and recommending high-quality venues and documents. For digital libraries in the Computer Science literature domain, conferences play a major role as an important publication and dissemination outlet. However, with a recent profusion of conferences and rapidly expanding fields, it is increasingly challenging for researchers and librarians to assess the quality of conferences. We propose a set of novel heuristics to automatically discover prestigious (and low-quality) conferences by mining the characteristics of Program Committee members. We examine the proposed cues both in isolation and combination under a classification scheme. Evaluation on a collection of 2,979 conferences and 16,147 PC members shows that our heuristics, when combined, correctly classify about 92{\%} of the conferences, with a low false positive rate of 0.035 and a recall of more than 73{\%} for identifying reputable conferences. Furthermore, we demonstrate empirically that our heuristics can also effectively detect a set of low-quality conferences, with a false positive rate of merely 0.002. We also report our experience of detecting two previously unknown low-quality conferences. Finally, we apply the proposed techniques to the entire quality spectrum by ranking conferences in the collection.",
author = "Ziming Zhuang and Ergin Elmacioglu and Dongwon Lee and Giles, {C. Lee}",
year = "2007",
month = "11",
day = "29",
doi = "10.1145/1255175.1255220",
language = "English (US)",
isbn = "1595936440",
series = "Proceedings of the ACM International Conference on Digital Libraries",
pages = "225--234",
booktitle = "Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007",

}

Zhuang, Z, Elmacioglu, E, Lee, D & Giles, CL 2007, Measuring conference quality by mining program committee characteristics. in Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007: Building and Sustaining the Digital Environment. Proceedings of the ACM International Conference on Digital Libraries, pp. 225-234, 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007: Building and Sustaining the Digital Environment, Vancouver, BC, Canada, 6/18/07. https://doi.org/10.1145/1255175.1255220

Measuring conference quality by mining program committee characteristics. / Zhuang, Ziming; Elmacioglu, Ergin; Lee, Dongwon; Giles, C. Lee.

Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007: Building and Sustaining the Digital Environment. 2007. p. 225-234 (Proceedings of the ACM International Conference on Digital Libraries).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Measuring conference quality by mining program committee characteristics

AU - Zhuang, Ziming

AU - Elmacioglu, Ergin

AU - Lee, Dongwon

AU - Giles, C. Lee

PY - 2007/11/29

Y1 - 2007/11/29

N2 - Bibliometrics are important measures for venue quality in digital libraries. Impacts of venues are usually the major consideration for subscription decision-making, and for ranking and recommending high-quality venues and documents. For digital libraries in the Computer Science literature domain, conferences play a major role as an important publication and dissemination outlet. However, with a recent profusion of conferences and rapidly expanding fields, it is increasingly challenging for researchers and librarians to assess the quality of conferences. We propose a set of novel heuristics to automatically discover prestigious (and low-quality) conferences by mining the characteristics of Program Committee members. We examine the proposed cues both in isolation and combination under a classification scheme. Evaluation on a collection of 2,979 conferences and 16,147 PC members shows that our heuristics, when combined, correctly classify about 92% of the conferences, with a low false positive rate of 0.035 and a recall of more than 73% for identifying reputable conferences. Furthermore, we demonstrate empirically that our heuristics can also effectively detect a set of low-quality conferences, with a false positive rate of merely 0.002. We also report our experience of detecting two previously unknown low-quality conferences. Finally, we apply the proposed techniques to the entire quality spectrum by ranking conferences in the collection.

AB - Bibliometrics are important measures for venue quality in digital libraries. Impacts of venues are usually the major consideration for subscription decision-making, and for ranking and recommending high-quality venues and documents. For digital libraries in the Computer Science literature domain, conferences play a major role as an important publication and dissemination outlet. However, with a recent profusion of conferences and rapidly expanding fields, it is increasingly challenging for researchers and librarians to assess the quality of conferences. We propose a set of novel heuristics to automatically discover prestigious (and low-quality) conferences by mining the characteristics of Program Committee members. We examine the proposed cues both in isolation and combination under a classification scheme. Evaluation on a collection of 2,979 conferences and 16,147 PC members shows that our heuristics, when combined, correctly classify about 92% of the conferences, with a low false positive rate of 0.035 and a recall of more than 73% for identifying reputable conferences. Furthermore, we demonstrate empirically that our heuristics can also effectively detect a set of low-quality conferences, with a false positive rate of merely 0.002. We also report our experience of detecting two previously unknown low-quality conferences. Finally, we apply the proposed techniques to the entire quality spectrum by ranking conferences in the collection.

UR - http://www.scopus.com/inward/record.url?scp=36349003629&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=36349003629&partnerID=8YFLogxK

U2 - 10.1145/1255175.1255220

DO - 10.1145/1255175.1255220

M3 - Conference contribution

AN - SCOPUS:36349003629

SN - 1595936440

SN - 9781595936448

T3 - Proceedings of the ACM International Conference on Digital Libraries

SP - 225

EP - 234

BT - Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007

ER -

Zhuang Z, Elmacioglu E, Lee D, Giles CL. Measuring conference quality by mining program committee characteristics. In Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries, JCDL 2007: Building and Sustaining the Digital Environment. 2007. p. 225-234. (Proceedings of the ACM International Conference on Digital Libraries). https://doi.org/10.1145/1255175.1255220