HARRA: Fast iterative hashed record linkage for large-scale data collections

Hung Sik Kim, Dongwon Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

58 Scopus citations

Abstract

We study the performance issue of the "iterative" record linkage (RL) problem, where match and merge operations may occur together in iterations until convergence emerges. We first propose the Iterative Locality-Sensitive Hashing (ILSH) that dynamically merges LSH-based has tables for quick and accurate blocking. Then, by exploiting inherent characteristics within/across data sets, we develop a suite of I-LSH-based RL algorithms, named as HARRA (HAshed RecoRd linkAge). The superiority of HARRA in speed over competing RL solutions is thoroughly validated using various real data sets. While maintaining equivalent or comparable accuracy levels, for instance, HARRA runs: (1) 4.5 and 10.5 times faster than StringMap and R-Swoosh in iteratively linking 4,000 x 4,000 short records (i.e., one of the small test cases), and (2) 5.6 and 3.4 times faster than basic LSH and Multi-Probe LSH algorithms in iteratively linking 400,000 x 400,000 long records (i.e., the largest test case).

Original languageEnglish (US)
Title of host publicationAdvances in Database Technology - EDBT 2010 - 13th International Conference on Extending Database Technology, Proceedings
Pages525-536
Number of pages12
DOIs
StatePublished - May 19 2010
Event13th International Conference on Extending Database Technology: Advances in Database Technology - EDBT 2010 - Lausanne, Switzerland
Duration: Mar 22 2010Mar 26 2010

Publication series

NameAdvances in Database Technology - EDBT 2010 - 13th International Conference on Extending Database Technology, Proceedings

Other

Other13th International Conference on Extending Database Technology: Advances in Database Technology - EDBT 2010
CountrySwitzerland
CityLausanne
Period3/22/103/26/10

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

Cite this

Kim, H. S., & Lee, D. (2010). HARRA: Fast iterative hashed record linkage for large-scale data collections. In Advances in Database Technology - EDBT 2010 - 13th International Conference on Extending Database Technology, Proceedings (pp. 525-536). (Advances in Database Technology - EDBT 2010 - 13th International Conference on Extending Database Technology, Proceedings). https://doi.org/10.1145/1739041.1739104