Order-Sensitive Imputation for Clustered Missing Values

Qian Ma, Yu Gu, Wang Chien Lee, Ge Yu

Research output: Contribution to journalArticlepeer-review

4 Scopus citations

Abstract

The issue of missing values (MVs) has appeared widely in real-world datasets and hindered the use of many statistical or machine learning algorithms for data analytics due to their incompetence in handling incomplete datasets. To address this issue, several MV imputation algorithms have been developed. However, these approaches do not perform well when most of the incomplete tuples are clustered with each other, coined here as the Clustered Missing Values Phenomenon, which attributes to the lack of sufficient complete tuples near an MV for imputation. In this paper, we propose the Order-Sensitive Imputation for Clustered Missing values (OSICM) framework, in which missing values are imputed sequentially such that the values filled earlier in the process are also used for later imputation of other MVs. Obviously, the order of imputations is critical to the effectiveness and efficiency of OSICM framework. We formulate the searching of the optimal imputation order as an optimization problem, and show its NP-hardness. Furthermore, we devise an algorithm to find the exact optimal solution and propose two approximate/heuristic algorithms to trade off effectiveness for efficiency. Finally, we conduct extensive experiments on real and synthetic datasets to demonstrate the superiority of our OSICM framework.

Original languageEnglish (US)
Article number8330055
Pages (from-to)166-180
Number of pages15
JournalIEEE Transactions on Knowledge and Data Engineering
Volume31
Issue number1
DOIs
StatePublished - Jan 1 2019

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'Order-Sensitive Imputation for Clustered Missing Values'. Together they form a unique fingerprint.

Cite this