Abstract
The issue of missing values (MVs) has appeared widely in real-world datasets and hindered the use of many statistical or machine learning algorithms for data analytics due to their incompetence in handling incomplete datasets. To address this issue, several MV imputation algorithms have been developed. However, these approaches do not perform well when most of the incomplete tuples are clustered with each other, coined here as the Clustered Missing Values Phenomenon, which attributes to the lack of sufficient complete tuples near an MV for imputation. In this paper, we propose the Order-Sensitive Imputation for Clustered Missing values (OSICM) framework, in which missing values are imputed sequentially such that the values filled earlier in the process are also used for later imputation of other MVs. Obviously, the order of imputations is critical to the effectiveness and efficiency of OSICM framework. We formulate the searching of the optimal imputation order as an optimization problem, and show its NP-hardness. Furthermore, we devise an algorithm to find the exact optimal solution and propose two approximate/heuristic algorithms to trade off effectiveness for efficiency. Finally, we conduct extensive experiments on real and synthetic datasets to demonstrate the superiority of our OSICM framework.
| Original language | English (US) |
|---|---|
| Article number | 8330055 |
| Pages (from-to) | 166-180 |
| Number of pages | 15 |
| Journal | IEEE Transactions on Knowledge and Data Engineering |
| Volume | 31 |
| Issue number | 1 |
| DOIs | |
| State | Published - Jan 1 2019 |
Fingerprint
All Science Journal Classification (ASJC) codes
- Information Systems
- Computer Science Applications
- Computational Theory and Mathematics
Cite this
}
Order-Sensitive Imputation for Clustered Missing Values. / Ma, Qian; Gu, Yu; Lee, Wang-chien; Yu, Ge.
In: IEEE Transactions on Knowledge and Data Engineering, Vol. 31, No. 1, 8330055, 01.01.2019, p. 166-180.Research output: Contribution to journal › Article
TY - JOUR
T1 - Order-Sensitive Imputation for Clustered Missing Values
AU - Ma, Qian
AU - Gu, Yu
AU - Lee, Wang-chien
AU - Yu, Ge
PY - 2019/1/1
Y1 - 2019/1/1
N2 - The issue of missing values (MVs) has appeared widely in real-world datasets and hindered the use of many statistical or machine learning algorithms for data analytics due to their incompetence in handling incomplete datasets. To address this issue, several MV imputation algorithms have been developed. However, these approaches do not perform well when most of the incomplete tuples are clustered with each other, coined here as the Clustered Missing Values Phenomenon, which attributes to the lack of sufficient complete tuples near an MV for imputation. In this paper, we propose the Order-Sensitive Imputation for Clustered Missing values (OSICM) framework, in which missing values are imputed sequentially such that the values filled earlier in the process are also used for later imputation of other MVs. Obviously, the order of imputations is critical to the effectiveness and efficiency of OSICM framework. We formulate the searching of the optimal imputation order as an optimization problem, and show its NP-hardness. Furthermore, we devise an algorithm to find the exact optimal solution and propose two approximate/heuristic algorithms to trade off effectiveness for efficiency. Finally, we conduct extensive experiments on real and synthetic datasets to demonstrate the superiority of our OSICM framework.
AB - The issue of missing values (MVs) has appeared widely in real-world datasets and hindered the use of many statistical or machine learning algorithms for data analytics due to their incompetence in handling incomplete datasets. To address this issue, several MV imputation algorithms have been developed. However, these approaches do not perform well when most of the incomplete tuples are clustered with each other, coined here as the Clustered Missing Values Phenomenon, which attributes to the lack of sufficient complete tuples near an MV for imputation. In this paper, we propose the Order-Sensitive Imputation for Clustered Missing values (OSICM) framework, in which missing values are imputed sequentially such that the values filled earlier in the process are also used for later imputation of other MVs. Obviously, the order of imputations is critical to the effectiveness and efficiency of OSICM framework. We formulate the searching of the optimal imputation order as an optimization problem, and show its NP-hardness. Furthermore, we devise an algorithm to find the exact optimal solution and propose two approximate/heuristic algorithms to trade off effectiveness for efficiency. Finally, we conduct extensive experiments on real and synthetic datasets to demonstrate the superiority of our OSICM framework.
UR - http://www.scopus.com/inward/record.url?scp=85058226786&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85058226786&partnerID=8YFLogxK
U2 - 10.1109/TKDE.2018.2822662
DO - 10.1109/TKDE.2018.2822662
M3 - Article
AN - SCOPUS:85058226786
VL - 31
SP - 166
EP - 180
JO - IEEE Transactions on Knowledge and Data Engineering
JF - IEEE Transactions on Knowledge and Data Engineering
SN - 1041-4347
IS - 1
M1 - 8330055
ER -