Optimal transport, mean partition, and uncertainty assessment in cluster analysis

Jia Li, Beomseok Seo, Lin Lin

Research output: Contribution to journalArticle

Abstract

In scientific data analysis, clusters identified computationally often substantiate existing hypotheses or motivate new ones. Yet the combinatorial nature of the clustering result, which is a partition rather than a set of parameters or a function, blurs notions of mean, and variance. This intrinsic difficulty hinders the development of methods to improve clustering by aggregation or to assess the uncertainty of clusters generated. We overcome that barrier by aligning clusters via optimal transport. Equipped with this technique, we propose a new algorithm to enhance clustering by any baseline method using bootstrap samples. Cluster alignment enables us to quantify variation in the clustering result at the levels of both overall partitions and individual clusters. Set relationships between clusters such as one-to-one match, split, and merge can be revealed. A covering point set for each cluster, a concept kin to the confidence interval, is proposed. The tools we have developed here will help address the crucial question of whether any cluster is an intrinsic or spurious pattern. Experimental results on both simulated and real data sets are provided. The corresponding R package OTclust is available on CRAN.

Original languageEnglish (US)
Pages (from-to)359-377
Number of pages19
JournalStatistical Analysis and Data Mining
Volume12
Issue number5
DOIs
StatePublished - Oct 1 2019

Fingerprint

Optimal Transport
Cluster analysis
Cluster Analysis
Agglomeration
Partition
Uncertainty
Clustering
Bootstrap Method
Point Sets
Confidence interval
Baseline
Aggregation
Data analysis
Alignment
Quantify
Covering
Experimental Results

All Science Journal Classification (ASJC) codes

  • Analysis
  • Information Systems
  • Computer Science Applications

Cite this

@article{7f592866b4ee4aaa8b4fd9ea1542d279,
title = "Optimal transport, mean partition, and uncertainty assessment in cluster analysis",
abstract = "In scientific data analysis, clusters identified computationally often substantiate existing hypotheses or motivate new ones. Yet the combinatorial nature of the clustering result, which is a partition rather than a set of parameters or a function, blurs notions of mean, and variance. This intrinsic difficulty hinders the development of methods to improve clustering by aggregation or to assess the uncertainty of clusters generated. We overcome that barrier by aligning clusters via optimal transport. Equipped with this technique, we propose a new algorithm to enhance clustering by any baseline method using bootstrap samples. Cluster alignment enables us to quantify variation in the clustering result at the levels of both overall partitions and individual clusters. Set relationships between clusters such as one-to-one match, split, and merge can be revealed. A covering point set for each cluster, a concept kin to the confidence interval, is proposed. The tools we have developed here will help address the crucial question of whether any cluster is an intrinsic or spurious pattern. Experimental results on both simulated and real data sets are provided. The corresponding R package OTclust is available on CRAN.",
author = "Jia Li and Beomseok Seo and Lin Lin",
year = "2019",
month = "10",
day = "1",
doi = "10.1002/sam.11418",
language = "English (US)",
volume = "12",
pages = "359--377",
journal = "Statistical Analysis and Data Mining",
issn = "1932-1872",
publisher = "John Wiley and Sons Inc.",
number = "5",

}

Optimal transport, mean partition, and uncertainty assessment in cluster analysis. / Li, Jia; Seo, Beomseok; Lin, Lin.

In: Statistical Analysis and Data Mining, Vol. 12, No. 5, 01.10.2019, p. 359-377.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Optimal transport, mean partition, and uncertainty assessment in cluster analysis

AU - Li, Jia

AU - Seo, Beomseok

AU - Lin, Lin

PY - 2019/10/1

Y1 - 2019/10/1

N2 - In scientific data analysis, clusters identified computationally often substantiate existing hypotheses or motivate new ones. Yet the combinatorial nature of the clustering result, which is a partition rather than a set of parameters or a function, blurs notions of mean, and variance. This intrinsic difficulty hinders the development of methods to improve clustering by aggregation or to assess the uncertainty of clusters generated. We overcome that barrier by aligning clusters via optimal transport. Equipped with this technique, we propose a new algorithm to enhance clustering by any baseline method using bootstrap samples. Cluster alignment enables us to quantify variation in the clustering result at the levels of both overall partitions and individual clusters. Set relationships between clusters such as one-to-one match, split, and merge can be revealed. A covering point set for each cluster, a concept kin to the confidence interval, is proposed. The tools we have developed here will help address the crucial question of whether any cluster is an intrinsic or spurious pattern. Experimental results on both simulated and real data sets are provided. The corresponding R package OTclust is available on CRAN.

AB - In scientific data analysis, clusters identified computationally often substantiate existing hypotheses or motivate new ones. Yet the combinatorial nature of the clustering result, which is a partition rather than a set of parameters or a function, blurs notions of mean, and variance. This intrinsic difficulty hinders the development of methods to improve clustering by aggregation or to assess the uncertainty of clusters generated. We overcome that barrier by aligning clusters via optimal transport. Equipped with this technique, we propose a new algorithm to enhance clustering by any baseline method using bootstrap samples. Cluster alignment enables us to quantify variation in the clustering result at the levels of both overall partitions and individual clusters. Set relationships between clusters such as one-to-one match, split, and merge can be revealed. A covering point set for each cluster, a concept kin to the confidence interval, is proposed. The tools we have developed here will help address the crucial question of whether any cluster is an intrinsic or spurious pattern. Experimental results on both simulated and real data sets are provided. The corresponding R package OTclust is available on CRAN.

UR - http://www.scopus.com/inward/record.url?scp=85065815225&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065815225&partnerID=8YFLogxK

U2 - 10.1002/sam.11418

DO - 10.1002/sam.11418

M3 - Article

AN - SCOPUS:85065815225

VL - 12

SP - 359

EP - 377

JO - Statistical Analysis and Data Mining

JF - Statistical Analysis and Data Mining

SN - 1932-1872

IS - 5

ER -