Finding a haystack in haystacks - Simultaneous identification of concepts in large bio-medical corpora

Ying Liu, Lucian V. Lita, R. Stefan Niculescu, Prasenjit Mitra, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Since nearly all information is now created digitally, large text databases have become more prevalent than ever. Automatically mining information from these databases proves to be a challenge due to slow pattern/string matching techniques. In this paper we introduce a new, fast multi-string pattern matching method called the Block Suffix Shifting (BSS) algorithm, which is based on the well known Aho-Chorasick algorithm. The advantages of our algorithm include: the ability to exploit the natural structure of text, perform significant character shifting, avoid useless backtracking jumps, efficient matching time and avoid the typical "sub-string" false positive errors. Our algorithm is applicable to many fields with free text, such as the health care domain and the scientific document field. In this paper, we apply the BSS algorithm to health care data and mine hundreds of thousands of medical concepts from a large Electronic Medical Record (EMR) corpora simultaneously and efficiently. Experimental results show the superiority of our algorithm when compared with the top of the line multi-string matching algorithms (the Aho-Corasick and the Wu-Manber algorithm).

Original languageEnglish (US)
Title of host publicationSociety for Industrial and Applied Mathematics - 8th SIAM International Conference on Data Mining 2008, Proceedings in Applied Mathematics 130
PublisherSociety for Industrial and Applied Mathematics Publications
Pages668-679
Number of pages12
ISBN (Print)9781605603179
DOIs
StatePublished - Jan 1 2008
Event8th SIAM International Conference on Data Mining 2008, Applied Mathematics 130 - Atlanta, GA, United States
Duration: Apr 24 2008Apr 26 2008

Publication series

NameSociety for Industrial and Applied Mathematics - 8th SIAM International Conference on Data Mining 2008, Proceedings in Applied Mathematics 130
Volume2

Other

Other8th SIAM International Conference on Data Mining 2008, Applied Mathematics 130
CountryUnited States
CityAtlanta, GA
Period4/24/084/26/08

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Software
  • Signal Processing
  • Theoretical Computer Science

Fingerprint Dive into the research topics of 'Finding a haystack in haystacks - Simultaneous identification of concepts in large bio-medical corpora'. Together they form a unique fingerprint.

  • Cite this

    Liu, Y., Lita, L. V., Niculescu, R. S., Mitra, P., & Giles, C. L. (2008). Finding a haystack in haystacks - Simultaneous identification of concepts in large bio-medical corpora. In Society for Industrial and Applied Mathematics - 8th SIAM International Conference on Data Mining 2008, Proceedings in Applied Mathematics 130 (pp. 668-679). (Society for Industrial and Applied Mathematics - 8th SIAM International Conference on Data Mining 2008, Proceedings in Applied Mathematics 130; Vol. 2). Society for Industrial and Applied Mathematics Publications. https://doi.org/10.1137/1.9781611972788.61