Scholarly big data information extraction and integration in the CiteSeerχ digital library

Kyle Williams, Jian Wu, Sagnik Ray Choudhury, Madian Khabsa, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

30 Scopus citations

Abstract

CiteSeerχ is a digital library that contains approximately 3.5 million scholarly documents and receives between 2 and 4 million requests per day. In addition to making documents available via a public Website, the data is also used to facilitate research in areas like citation analysis, co-author network analysis, scalability evaluation and information extraction. The papers in CiteSeerχ are gathered from the Web by means of continuous automatic focused crawling and go through a series of automatic processing steps as part of the ingestion process. Given the size of the collection, the fact that it is constantly expanding, and the multiple ways in which it is used both by the public to access scholarly documents and for research, there are several big data challenges. In this paper, we provide a case study description of how we address these challenges when it comes to information extraction, data integration and entity linking in CiteSeer χ. We describe how we: aggregate data from multiple sources on the Web; store and manage data; process data as part of an automatic ingestion pipeline that includes automatic metadata and information extraction; perform document and citation clustering; perform entity linking and name disambiguation; and make our data and source code available to enable research and collaboration.

Original languageEnglish (US)
Title of host publication2014 IEEE 30th International Conference on Data Engineering Workshops, ICDEW 2014
PublisherIEEE Computer Society
Pages68-73
Number of pages6
ISBN (Print)9781479934805
DOIs
StatePublished - Jan 1 2014
Event2014 IEEE 30th International Conference on Data Engineering Workshops, ICDEW 2014 - Chicago, IL, United States
Duration: Mar 31 2014Apr 4 2014

Publication series

NameProceedings - International Conference on Data Engineering
ISSN (Print)1084-4627

Other

Other2014 IEEE 30th International Conference on Data Engineering Workshops, ICDEW 2014
CountryUnited States
CityChicago, IL
Period3/31/144/4/14

All Science Journal Classification (ASJC) codes

  • Software
  • Signal Processing
  • Information Systems

Fingerprint Dive into the research topics of 'Scholarly big data information extraction and integration in the CiteSeer<sup>χ</sup> digital library'. Together they form a unique fingerprint.

Cite this