Article

Information-theoretic co-clustering

Authors:

Inderjit S. Dhillon,

Subramanyam Mallela,

Dharmendra S. ModhaAuthors Info & Claims

KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 89 - 98

https://doi.org/10.1145/956750.956764

Published: 24 August 2003 Publication History

Abstract

Two-dimensional contingency or co-occurrence tables arise frequently in important applications such as text, web-log and market-basket data analysis. A basic problem in contingency table analysis is co-clustering: simultaneous clustering of the rows and columns. A novel theoretical formulation views the contingency table as an empirical joint probability distribution of two discrete random variables and poses the co-clustering problem as an optimization problem in information theory---the optimal co-clustering maximizes the mutual information between the clustered random variables subject to constraints on the number of row and column clusters. We present an innovative co-clustering algorithm that monotonically increases the preserved mutual information by intertwining both the row and column clusterings at all stages. Using the practical example of simultaneous word-document clustering, we demonstrate that our algorithm works well in practice, especially in the presence of sparsity and high-dimensionality.

References

[1]

P. Bradley, U. Fayyad, and C. Reina. Scaling clustering algorithms to large databases. In KDD'03. AAAI Press, 1998.]]

[2]

Y. Cheng and G. Church. Biclustering of expression data. In Proceedings ISMB, pages 93-103. AAAI Press, 2000.]]

Digital Library

[3]

T. Cover and J. Thomas. Elements of Information Theory. John Wiley & Sons, New York, USA, 1991.]]

Digital Library

[4]

I. S. Dhillon. Co-clustering documents and words using bipartite spectral graph partitioning. In Proceedings of The 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining(KDD-2001), pages 269-274, 2001.]]

Digital Library

[5]

I. S. Dhillon, J. Fan, and Y. Guan. Efficient clustering of very large document collections. In R. Grossman, C. Kamath, P. Kegelmeyer, V. Kumar, and R. Namburu, editors, Data Mining for Scientific and Engineering Applications, pages 357-381. Kluwer Academic Publishers, 2001.]]

[6]

I. S. Dhillon, S. Mallela, and R. Kumar. A divisive information-theoretic feature clustering algorithm for text classification. Journal of Machine Learning Research(JMLR): Special Issue on Variable and Feature Selection, 3:1265-1287, March 2003.]]

Digital Library

[7]

I. S. Dhillon and D. S. Modha. Concept decompositions for large sparse text data using clustering. Machine Learning, 42(1):143-175, January 2001.]]

Digital Library

[8]

R. El-Yaniv and O. Souroujon. Iterative double clustering for unsupervised and semi-supervised learning. In ECML-01, pages 121-132, 2001.]]

Digital Library

[9]

R. A. Fisher. On the interpretation of x2 from the contingency tables, and the calculation of p. J. Royal Stat. Soc., 85:87-94, 1922.]]

[10]

N. Friedman, O. Mosenzon, N. Slonim, and N. Tishby. Multivariate information bottleneck. In UAI-2001, pages 152-161, 2001.]]

Digital Library

[11]

J. A. Hartigan. Direct clustering of a data matrix. Journal of the American Statistical Association, 67(337):123-129, March 1972.]]

[12]

T. Hofmann. Probabilistic latent semantic indexing. In Proc. ACM SIGIR. ACM Press, August 1999.]]

Digital Library

[13]

T. Hofmann and J. Puzicha. Latent class models for collaborative filtering. In Proceedings of the International Joint Conference in Artificial Intelligence (IJCAI), 1999.]]

Digital Library

[14]

A. K. Jain and R. C. Dubes. Algorithms for Clustering Data. Prentice-Hall, Englewood Cliffs, NJ, 1988.]]

Digital Library

[15]

Ken Lang. News Weeder: Learning to filter netnews. In ICML-95, pages 331-339, 1995.]]

[16]

A. K. McCallum. Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. www.cs.cmu.edu/ mccallum/bow, 1996.]]

[17]

B. Mirkin. Mathematical Classification and Clustering. Kluwer Academic Publishers, 1996.]]

[18]

N. Slonim, N. Friedman, and N. Tishby. Agglomerative multivariate information bottleneck. In NIPS-14, 2001.]]

[19]

N. Slonim and N. Tishby. Document clustering using word clusters via the information bottleneck method. In ACM SIGIR, pages 208-215, 2000.]]

Digital Library

[20]

N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368-377, 1999.]]

Cited By

Chen WWang HZhang YDeng PLuo ZLi T(2024)T-Distributed Stochastic Neighbor Embedding for Co-Representation LearningACM Transactions on Intelligent Systems and Technology10.1145/362782315:2(1-18)Online publication date: 22-Feb-2024
https://dl.acm.org/doi/10.1145/3627823
Hu SLou ZYan XYe Y(2024)A Survey on Information BottleneckIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336634946:8(5325-5344)Online publication date: Aug-2024
https://doi.org/10.1109/TPAMI.2024.3366349
Zeng PLin Z(2024)scICML: Information-Theoretic Co-Clustering-Based Multi-View Learning for the Integrative Analysis of Single-Cell Multi-Omics DataIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2023.330598921:1(200-207)Online publication date: Jan-2024
https://doi.org/10.1109/TCBB.2023.3305989
Show More Cited By

Index Terms

Information-theoretic co-clustering

Recommendations

Spectral co-clustering ensemble

The goal of co-clustering is to simultaneously cluster the rows and columns of an input data matrix. It overcomes several limitations associated with traditional clustering methods by allowing automatic discovery of similarity based on a subset of ...
Non-Exhaustive, Overlapping Co-Clustering
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management

The goal of co-clustering is to simultaneously identify a clustering of the rows as well as the columns of a two dimensional data matrix. Most existing co-clustering algorithms are designed to find pairwise disjoint and exhaustive co-clusters. However, ...
A Survey of Co-Clustering
Co-clustering is to cluster samples and features simultaneously, which can also reveal the relationship between row clusters and column clusters. Therefore, lots of scientists have drawn much attention to conduct extensive research on it, and co-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

August 2003

736 pages

ISBN:1581137370

DOI:10.1145/956750

Conference Chair:
Lise Getoor
University of Maryland, College Park
,
General Chair:
Ted Senator
DARPA
,
Program Chairs:
Pedro Domingos
University of Washington
,
Christos Faloutsos
Carnegie Mellon University

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD03

Sponsor:

KDD03: The Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 24 - 27, 2003

Washington, D.C.

Acceptance Rates

KDD '03 Paper Acceptance Rate 46 of 298 submissions, 15%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

563
Total Citations
View Citations
4,668
Total Downloads

Downloads (Last 12 months)111
Downloads (Last 6 weeks)9

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen WWang HZhang YDeng PLuo ZLi T(2024)T-Distributed Stochastic Neighbor Embedding for Co-Representation LearningACM Transactions on Intelligent Systems and Technology10.1145/362782315:2(1-18)Online publication date: 22-Feb-2024
https://dl.acm.org/doi/10.1145/3627823
Hu SLou ZYan XYe Y(2024)A Survey on Information BottleneckIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336634946:8(5325-5344)Online publication date: Aug-2024
https://doi.org/10.1109/TPAMI.2024.3366349
Zeng PLin Z(2024)scICML: Information-Theoretic Co-Clustering-Based Multi-View Learning for the Integrative Analysis of Single-Cell Multi-Omics DataIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2023.330598921:1(200-207)Online publication date: Jan-2024
https://doi.org/10.1109/TCBB.2023.3305989
Ahmad AAhmad TMasood SSiddiqui MEl-Rahiem BPlawiak PAlblehai F(2024)A Probabilistic Approach for Extractive Summarization Based on Clustering Cum Graph Ranking MethodIEEE Access10.1109/ACCESS.2024.339225212(70464-70479)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3392252
Marchello GCorneli MBouveyron C(2024)A Deep Dynamic Latent Block Model for Co-clustering of Zero-Inflated Data MatricesJournal of Computational and Graphical Statistics10.1080/10618600.2024.2319162(1-39)Online publication date: 23-Feb-2024
https://doi.org/10.1080/10618600.2024.2319162
Chen YLei CLi CMa HHu N(2024)Robust convex biclustering with a tuning-free methodJournal of Applied Statistics10.1080/02664763.2024.2367143(1-16)Online publication date: 17-Jun-2024
https://doi.org/10.1080/02664763.2024.2367143
Yang WWang HZhang YLuo ZLi T(2024)A Nondominated Sorting Genetic Model for Co-clusteringInformation Sciences10.1016/j.ins.2024.120459(120459)Online publication date: Mar-2024
https://doi.org/10.1016/j.ins.2024.120459
Lanfermann FLiu QJin YSchmitt S(2024)Identification of energy management configuration concepts from a set of pareto-optimal solutionsEnergy Conversion and Management: X10.1016/j.ecmx.2024.10057622(100576)Online publication date: Apr-2024
https://doi.org/10.1016/j.ecmx.2024.100576
Yu GRen LWang JDomeniconi CZhang X(2024)Multiple clusterings: Recent advances and perspectivesComputer Science Review10.1016/j.cosrev.2024.10062152(100621)Online publication date: May-2024
https://doi.org/10.1016/j.cosrev.2024.100621
Amiri BKarimianghadim R(2024)A novel text clustering model based on topic modelling and social network analysisChaos, Solitons & Fractals10.1016/j.chaos.2024.114633181(114633)Online publication date: Apr-2024
https://doi.org/10.1016/j.chaos.2024.114633
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents