REMIAN: Real-Time and Error-Tolerant Missing Value Imputation

Qian Ma, Yu Gu, Wang Chien Lee, Ge Yu, Hongbo Liu, Xindong Wu

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Missing value (MV) imputation is a critical preprocessing means for data mining. Nevertheless, existing MV imputation methods are mostly designed for batch processing, and thus are not applicable to streaming data, especially those with poor quality. In this article, we propose a framework, called Real-time and Error-tolerant Missing vAlue ImputatioN (REMAIN), to impute MVs in poor-quality streaming data. Instead of imputing MVs based on all the observed data, REMAIN first initializes the MV imputation model based on a-RANSAC which is capable of detecting and rejecting anomalies in an efficient manner, and then incrementally updates the model parameters upon the arrival of new data to support real-time MV imputation. As the correlations among attributes of the data may change over time in unforseenable ways, we devise a deterioration detection mechanism to capture the deterioration of the imputation model to further improve the imputation accuracy. Finally, we conduct an extensive evaluation on the proposed algorithms using real-world and synthetic datasets. Experimental results demonstrate that REMAIN achieves significantly higher imputation accuracy over existing solutions. Meanwhile, REMAIN improves up to one order of magnitude in time cost compared with existing approaches.

Original languageEnglish (US)
Article number3412364
JournalACM Transactions on Knowledge Discovery from Data
Volume14
Issue number6
DOIs
StatePublished - Oct 2020

All Science Journal Classification (ASJC) codes

  • Computer Science(all)

Fingerprint

Dive into the research topics of 'REMIAN: Real-Time and Error-Tolerant Missing Value Imputation'. Together they form a unique fingerprint.

Cite this