research-article

Public Access

Comparing Clustering with Pairwise and Relative Constraints: A Unified Framework

Authors:

Xiaoli Z. Fern,

Teresa Vania Tjahja,

Rómer RosalesAuthors Info & Claims

ACM Transactions on Knowledge Discovery from Data (TKDD), Volume 11, Issue 2

Article No.: 22, Pages 1 - 26

https://doi.org/10.1145/2996467

Published: 03 December 2016 Publication History

Abstract

Clustering can be improved with the help of side information about the similarity relationships among instances. Such information has been commonly represented by two types of constraints: pairwise constraints and relative constraints, regarding similarities about instance pairs and triplets, respectively. Prior work has mostly considered these two types of constraints separately and developed individual algorithms to learn from each type. In practice, however, it is critical to understand/compare the usefulness of the two types of constraints as well as the cost of acquiring them, which has not been studied before. This paper provides an extensive comparison of clustering with these two types of constraints. Specifically, we compare their impacts both on human users that provide such constraints and on the learning system that incorporates such constraints into clustering. In addition, to ensure that the comparison of clustering is performed on equal ground (without the potential bias introduced by different learning algorithms), we propose a probabilistic semi-supervised clustering framework that can learn from either type of constraints. Our experiments demonstrate that the proposed semi-supervised clustering framework is highly effective at utilizing both types of constraints to aid clustering. Our user study provides valuable insights regarding the impact of the constraints on human users, and our experiments on clustering with the human-labeled constraints reveal that relative constraint is often more efficient at improving clustering.

References

[1]

Ehsan Amid, Aristides Gionis, and Antti Ukkonen. 2015. A kernel-learning approach to semi-supervised clustering with relative distance comparisons. In Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 219--234.

Digital Library

[2]

Korinna Bade and Andreas Nürnberger. 2014. Hierarchical constraints. Machine Learning 94, 3 (2014), 371--399.

Digital Library

[3]

Mahdieh Soleymani Baghshah and Saeed Bagheri Shouraki. 2009. Semi-supervised metric learning using pairwise constraints. In Proceedings of the 21st International Joint Conference on Artificial Intelligence. 1217--1222.

Digital Library

[4]

Sugato Basu, Mikhail Bilenko, and Raymond J. Mooney. 2004. A probabilistic framework for semi-supervised clustering. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD’04). 59--68.

Digital Library

[5]

Michel Benam and Jean-Yves Le Boudec. 2011. On mean field convergence and stationary regime. CoRR abs/1111.5710 (2011).

[6]

Mikhail Bilenko, Sugato Basu, and Raymond J. Mooney. 2004. Integrating constraints and metric learning in semi-supervised clustering. In Proceedings of the 21th International Conference on Machine Learning. 81--88.

Digital Library

[7]

Christopher M. Bishop. 2007. Pattern Recognition and Machine Learning (1st ed.). Springer.

Digital Library

[8]

Forrest Briggs, Xiaoli Z. Fern, and Raviv Raich. 2012. Rank-loss support instance machines for MIML instance annotation. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD’12). 534--542.

Digital Library

[9]

Shiyu Chang, Charu C. Aggarwal, and Thomas S. Huang. 2014a. Learning local semantic distances with limited supervision. In Proceedings of the 14th IEEE International Conference on Data Mining. 70--79.

Digital Library

[10]

Shiyu Chang, Guo-Jun Qi, Charu C. Aggarwal, Jiayu Zhou, Meng Wang, and Thomas S. Huang. 2014b. Factorized similarity learning in networks. In Proceedings of the 14th IEEE International Conference on Data Mining. 60--69.

Digital Library

[11]

Jason V. Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit S. Dhillon. 2007. Information-theoretic metric learning. In Proceedings of the 24th International Conference on Machine Learning. 209--216.

Digital Library

[12]

Inderjit S. Dhillon. 2001. Co-clustering documents and words using bipartite spectral graph partitioning. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD’01). ACM, 269--274.

Digital Library

[13]

Shifei Ding, Hongjie Jia, Liwen Zhang, and Fengxiang Jin. 2014. Research of semi-supervised spectral clustering algorithm based on pairwise constraints. Neural Computing and Applications 24, 1 (2014), 211--219.

[14]

Weisheng Dong, Xin Li, D. Zhang, and Guangming Shi. 2011. Sparsity-based image denoising via dictionary learning and structural clustering. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE, 457--464.

Digital Library

[15]

Charles W. Fox and Stephen J. Roberts. 2012. A tutorial on variational Bayesian inference. Artificial Intelligence Review 38, 2 (2012), 85--95.

[16]

Ryan Gomes, Andreas Krause, and Pietro Perona. 2010. Discriminative clustering by regularized information maximization. In Proceedings of Advances in Neural Information Processing Systems. 775--783.

Digital Library

[17]

Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research. In Human Mental Workload, Vol. 52: Advances in Psychology. North-Holland, 139--183.

[18]

Kaizhu Huang, Yiming Ying, and Colin Campbell. 2011. Generalized sparse metric learning with relative comparisons. Knowledge and Information Systems 28, 1 (2011), 25--45.

Digital Library

[19]

A. Jordan. 2002. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In Proceedings of Advances in Neural Information Processing Systems 14 (2002), 841.

[20]

Sepandar D. Kamvar, Dan Klein, and Christopher D. Manning. 2003. Spectral learning. In Proceedings of the 18th International Joint Conference on Artificial Intelligence. 561--566.

Digital Library

[21]

Nimit Kumar and Krishna Kummamuru. 2008. Semi-supervised clustering with metric learning using relative comparisons. IEEE Transactions on Knowledge and Data Engineering 20, 4 (2008), 496--503.

Digital Library

[22]

Tilman Lange, Martin H. C. Law, Anil K. Jain, and Joachim M. Buhmann. 2005. Learning with constrained and unlabelled data. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 731--738.

Digital Library

[23]

Martin H. C. Law, Alexander P. Topchy, and Anil K. Jain. 2005. Model-based clustering with probabilistic constraints. In Proceedings of SIAM Conference on Data Mining. 641--645.

[24]

Weizhong Li, Lukasz Jaroszewski, and Adam Godzik. 2001. Clustering of highly homologous sequences to reduce the size of large protein databases. Bioinformatics 17, 3 (2001), 282--283.

[25]

E. Y. Liu, Z. Zhang, and W. Wang. 2011. Clustering with relative constraints. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 947--955.

Digital Library

[26]

Eric Yi Liu, Zhishan Guo, Xiang Zhang, Vladimir Jojic, and Wei Wang. 2012. Metric learning from relative comparisons by minimizing squared residual. In Proceedings of the 12th IEEE International Conference on Data Mining. 978--983.

Digital Library

[27]

Yiyi Liu, Quanquan Gu, Jack P. Hou, Jiawei Han, and Jian Ma. 2014. A network-assisted co-clustering algorithm to discover cancer subtypes based on gene expression. BMC Bioinformatics 15, 1 (2014), 37.

[28]

Yi Liu, Rong Jin, and Anil K. Jain. 2007. BoostCluster: Boosting clustering by pairwise constraints. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 450--459.

Digital Library

[29]

Zhengdong Lu. 2007. Semi-supervised clustering with pairwise constraints: A discriminative approach. Journal of Machine Learning Research 2 (2007), 299--306.

[30]

Zhengdong Lu and Todd K. Leen. 2004. Semi-supervised learning with penalized probabilistic clustering. In Proceedings of Advances in Neural Information Processing Systems. 849--856.

Digital Library

[31]

MicrosoftResearch. 2005. Image Database. Retrieved from: http://research.microsoft.com/en-us/projects/ObjectClassRecognition/.

[32]

Alexis Mignon and Frédéric Jurie. 2012. Pcca: A new approach for distance learning from sparse pairwise constraints. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 2666--2672.

Digital Library

[33]

Blaine Nelson and Ira Cohen. 2007. Revisiting probabilistic models for clustering with pairwise constraints. In Proceedings of the 24th International Conference on Machine Learning. 673--680.

Digital Library

[34]

M. E. Nilsback and A. Zisserman. 2008. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing. 722--729.

Digital Library

[35]

J. C. Nunnally and Ira Bernstein. 1994. Psychometric Theory. McGraw Hill, Inc.

[36]

Yuanli Pei, Xiaoli Z. Fern, Rómer Rosales, and Teresa V. Tjahja. 2014. Discriminative clustering with relative constraints. http://arxiv.org/abs/1501.00037.

[37]

Rómer Rosales and Glenn Fung. 2006. Learning sparse metrics via linear programming. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 367--373.

Digital Library

[38]

L. Saul, T. Jaakkola, and M. Jordan. 1996. Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research 4 (1996), 61--76.

[39]

Mark Schmidt. 2012. L-BFGS software. Retrieved from: http://www.di.ens.fr/&sim;mschmidt/Software/minFunc.html.

[40]

Matthew Schultz and Thorsten Joachims. 2003. Learning a distance metric from relative comparisons. In Proceedings of Advances Neural Information Processing Systems. 41--48.

Digital Library

[41]

Noam Shental, Aharon Bar-Hillel, Tomer Hertz, and Daphna Weinshall. 2004. Computing Gaussian mixture models with EM using equivalence constraints. In Proceedings of Advances in Neural Information Processing Systems 16, 8 (2004), 465--472.

Digital Library

[42]

Parag Singla and Pedro Domingos. 2005. Discriminative training of Markov logic networks. In Proceedings of AAAI National Conference on Artificial intelligence (AAAI’05), Vol. 5. 868--873.

Digital Library

[43]

Kiri Wagstaff, Claire Cardie, Seth Rogers, and Stefan Schrödl. 2001. Constrained K-means clustering with background knowledge. In Proceedings of the 18th International Conference on Machine Learning. 577--584.

Digital Library

[44]

Dingding Wang, Shenghuo Zhu, Tao Li, Yun Chi, and Yihong Gong. 2011. Integrating document clustering and multidocument summarization. In Proceedings of ACM Transactions on Knowledge Discovery from Data (TKDD’11) 5, 3 (2011), 14.

Digital Library

[45]

Xiang Wang and Ian Davidson. 2010. Flexible constrained spectral clustering. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD’10). 563--572.

Digital Library

[46]

Xiang Wang, Jun Wang, Buyue Qian, Fei Wang, and Ian Davidson. 2014. Self-taught spectral clustering via constraint augmentation. In Proceedings of SIAM Conference on Data Mining. 416--424.

[47]

Shiming Xiang, Feiping Nie, and Changshui Zhang. 2008. Learning a Mahalanobis distance metric for data clustering and classification. Pattern Recognition 41, 12 (2008), 3600--3612.

Digital Library

[48]

Eric P. Xing, Andrew Y. Ng, Michael I. Jordan, and Stuart J. Russell. 2002. Distance metric learning with application to clustering with side-information. In Proceedings of Advances Neural Information Processing Systems. 505--512.

Digital Library

[49]

Xuesong Yin, Songcan Chen, Enliang Hu, and Daoqiang Zhang. 2010. Semi-supervised clustering with metric learning: An adaptive kernel method. Pattern Recognition 43, 4 (2010), 1320--1333.

Digital Library

[50]

Stella X. Yu and Jianbo Shi. 2004. Segmentation given partial grouping constraints. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 2 (Jan. 2004), 173--183.

Digital Library

[51]

Hong Zeng and Yiu-ming Cheung. 2012. Semi-supervised maximum margin clustering with pairwise constraints. IEEE Transactions on Knowledge and Data Engineering 24, 5 (2012), 926--939.

Digital Library

[52]

Xiangrong Zhang, Licheng Jiao, Fang Liu, Liefeng Bo, and Maoguo Gong. 2008. Spectral clustering ensemble applied to SAR image segmentation. IEEE Transactions on Geoscience and Remote Sensing 46, 7 (2008), 2126--2136.

Cited By

Bicego MHassan H(2024)An Extension of Random Forest-Clustering Schemes Which Works with Partition-Level ConstraintsPattern Recognition10.1007/978-3-031-78383-8_7(93-108)Online publication date: 2-Dec-2024
https://doi.org/10.1007/978-3-031-78383-8_7
Jiang ZZhan YMao QDu Y(2022)Semi-supervised clustering under a compact-cluster assumptionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3145347(1-1)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3145347
Zhang YLi XJia M(2022)Semi-supervised nonnegative matrix factorization with pairwise constraints for image clusteringInternational Journal of Machine Learning and Cybernetics10.1007/s13042-022-01614-713:11(3577-3587)Online publication date: 10-Sep-2022
https://doi.org/10.1007/s13042-022-01614-7
Show More Cited By

Index Terms

Comparing Clustering with Pairwise and Relative Constraints: A Unified Framework
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis
2. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

A Novel Semi-supervised Subtractive Clustering Method Using Pairwise Constraints
ICEICE '12: Proceedings of the 2012 Second International Conference on Electric Information and Control Engineering - Volume 02

In this paper, a novel semi-supervised subtractive clustering method using pair wise constraints is proposed. Because the traditional subtractive clustering method cannot obtain the better clustering accuracies, some pair wise constraints regarded as ...
Semi-supervised agglomerative hierarchical clustering using clusterwise tolerance based pairwise constraints
MDAI'10: Proceedings of the 7th international conference on Modeling decisions for artificial intelligence

Recently, semi-supervised clustering has been remarked and discussed in many researches. In semi-supervised clustering, pairwise constraints, that is, must-link and cannot-link are frequently used in order to improve clustering results by using prior ...
Semi-supervised Fuzzy c-Means Clustering Using Clusterwise Tolerance Based Pairwise Constraints
GRC '10: Proceedings of the 2010 IEEE International Conference on Granular Computing

Recently, semi-supervised clustering has been remarked and discussed in many research fields. In semi-supervised clustering, prior knowledge or information are often formulated as pairwise constraints, that is, must-link and cannot-link. Such pairwise ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Knowledge Discovery from Data

ACM Transactions on Knowledge Discovery from Data Volume 11, Issue 2

May 2017

419 pages

ISSN:1556-4681

EISSN:1556-472X

DOI:10.1145/3017677

Editor:
Philip S. Yu
University of Illinois at Chicago, USA

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 December 2016

Accepted: 01 September 2016

Revised: 01 April 2016

Received: 01 January 2015

Published in TKDD Volume 11, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
576
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)18

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bicego MHassan H(2024)An Extension of Random Forest-Clustering Schemes Which Works with Partition-Level ConstraintsPattern Recognition10.1007/978-3-031-78383-8_7(93-108)Online publication date: 2-Dec-2024
https://doi.org/10.1007/978-3-031-78383-8_7
Jiang ZZhan YMao QDu Y(2022)Semi-supervised clustering under a compact-cluster assumptionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3145347(1-1)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3145347
Zhang YLi XJia M(2022)Semi-supervised nonnegative matrix factorization with pairwise constraints for image clusteringInternational Journal of Machine Learning and Cybernetics10.1007/s13042-022-01614-713:11(3577-3587)Online publication date: 10-Sep-2022
https://doi.org/10.1007/s13042-022-01614-7
Huang DHu JLi TDu SChen H(2022)Consistency regularization for deep semi-supervised clustering with pairwise constraintsInternational Journal of Machine Learning and Cybernetics10.1007/s13042-022-01599-313:11(3359-3372)Online publication date: 7-Jul-2022
https://doi.org/10.1007/s13042-022-01599-3
Śmieja MStruski ŁFigueiredo M(2020)A classification-based approach to semi-supervised clustering with pairwise constraintsNeural Networks10.1016/j.neunet.2020.04.017Online publication date: May-2020
https://doi.org/10.1016/j.neunet.2020.04.017
Pham AXi J(2018)Differentially Private Semi-Supervised Learning With Known Class Priors2018 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2018.8622071(801-810)Online publication date: Dec-2018
https://doi.org/10.1109/BigData.2018.8622071
Gan HLi ZWu WLuo ZHuang R(2018)Safety-aware Graph-based Semi-Supervised LearningExpert Systems with Applications10.1016/j.eswa.2018.04.031107(243-254)Online publication date: Oct-2018
https://doi.org/10.1016/j.eswa.2018.04.031
Li ZTang Y(2018)Comparative density peaks clusteringExpert Systems with Applications10.1016/j.eswa.2017.11.02095(236-247)Online publication date: May-2018
https://doi.org/10.1016/j.eswa.2017.11.020
Ukkonen A(2017)Crowdsourced Correlation Clustering with Relative Distance Comparisons2017 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM.2017.148(1117-1122)Online publication date: Dec-2017
https://doi.org/10.1109/ICDM.2017.148

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents