Near duplicate detection in an academic digital library

Kyle Williams, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

20 Scopus citations

Abstract

The detection and potential removal of duplicates is desirable for a number of reasons, such as to reduce the need for unnecessary storage and computation, and to provide users with uncluttered search results. This paper describes an investigation into the application of scalable simhash and shingle state of the art duplicate detection algorithms for detecting near duplicate documents in the CiteSeerX digital library. We empirically explored the duplicate detection methods and evaluated their performance and application to academic documents and identified good parameters for the algorithms. We also analyzed the types of near duplicates identified by each algorithm. The highest F-scores achieved were 0.91 and 0.99 for the simhash and shingle-based methods respectively. The shingle-based method also identified a larger variety of duplicate types than the simhash-based method.

Original languageEnglish (US)
Title of host publicationDocEng 2013 - Proceedings of the 2013 ACM Symposium on Document Engineering
PublisherAssociation for Computing Machinery
Pages91-94
Number of pages4
ISBN (Print)9781450317894
DOIs
StatePublished - Jan 1 2013
Event2013 ACM Symposium on Document Engineering, DocEng 2013 - Florence, Italy
Duration: Sep 10 2013Sep 13 2013

Publication series

NameDocEng 2013 - Proceedings of the 2013 ACM Symposium on Document Engineering

Other

Other2013 ACM Symposium on Document Engineering, DocEng 2013
CountryItaly
CityFlorence
Period9/10/139/13/13

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Software

Cite this

Williams, K., & Giles, C. L. (2013). Near duplicate detection in an academic digital library. In DocEng 2013 - Proceedings of the 2013 ACM Symposium on Document Engineering (pp. 91-94). (DocEng 2013 - Proceedings of the 2013 ACM Symposium on Document Engineering). Association for Computing Machinery. https://doi.org/10.1145/2494266.2494312