Distributed In-Memory Processing of All k Nearest Neighbor Queries

Georgios Chatzimilioudis, Constantinos Costa, Demetrios Zeinalipour-Yazti, Wang Chien Lee, Evaggelia Pitoura

Research output: Contribution to journalArticle

12 Scopus citations

Abstract

A wide spectrum of Internet-scale mobile applications, ranging from social networking, gaming and entertainment to emergency response and crisis management, all require efficient and scalable All k Nearest Neighbor (AkNN) computations over millions of moving objects every few seconds to be operational. Most traditional techniques for computing AkNN queries are centralized, lacking both scalability and efficiency. Only recently, distributed techniques for shared-nothing cloud infrastructures have been proposed to achieve scalability for large datasets. These batch-oriented algorithms are sub-optimal due to inefficient data space partitioning and data replication among processing units. In this paper, we present Spitfire, a distributed algorithm that provides a scalable and high-performance AkNN processing framework. Our proposed algorithm deploys a fast load-balanced partitioning scheme along with an efficient replication-set selection algorithm, to provide fast main-memory computations of the exact AkNN results in a batch-oriented manner. We evaluate, both analytically and experimentally, how the pruning efficiency of the Spitfire algorithm plays a pivotal role in reducing communication and response time up to an order of magnitude, compared to three other state-of-the-art distributed AkNN algorithms executed in distributed main-memory.

Original languageEnglish (US)
Article number7337428
Pages (from-to)925-938
Number of pages14
JournalIEEE Transactions on Knowledge and Data Engineering
Volume28
Issue number4
DOIs
StatePublished - Apr 1 2016

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Computational Theory and Mathematics

Cite this