research-article

Toward autonomic grids: analyzing the job flow with affinity streaming

Authors:

Xiangliang Zhang,

Cyril Furtlehner,

Cecile Germain-Renaud,

Michèle SebagAuthors Info & Claims

KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 987 - 996

https://doi.org/10.1145/1557019.1557126

Published: 28 June 2009 Publication History

Abstract

The Affinity Propagation (AP) clustering algorithm proposed by Frey and Dueck (2007) provides an understandable, nearly optimal summary of a dataset, albeit with quadratic computational complexity. This paper, motivated by Autonomic Computing, extends AP to the data streaming framework. Firstly a hierarchical strategy is used to reduce the complexity to O(N^1+ε); the distortion loss incurred is analyzed in relation with the dimension of the data items. Secondly, a coupling with a change detection test is used to cope with non-stationary data distribution, and rebuild the model as needed. The presented approach StrAP is applied to the stream of jobs submitted to the EGEE Grid, providing an understandable description of the job flow and enabling the system administrator to spot online some sources of failures.

Supplementary Material

JPG File (p987-zhang.jpg)

Download
9.94 KB

MP4 File (p987-zhang.mp4)

Download
64.79 MB

References

[1]

F. Bergeaud and S. Mallat. Matching pursuit of images. In ICIP, pages 53--56, 1995.

Digital Library

[2]

F. Cao, M. Ester, W. Qian, and A. Zhou. Density-based clustering over an evolving data stream with noise. In SIAM Conference on Data Mining (SDM), pages 326--337, 2006.

[3]

G. Cormode, S. Muthukrishnan, and W. Zhuang. Conquering the divide: Continuous clustering of distributed data streams. In ICDE, pages 1036--1045, 2007.

[4]

M. Ester. A density-based algorithm for discovering clusters in large spatial databases with noise: the uniqueness of a good optimum for k-means. In SIGKDD, pages 226--231, 1996.

[5]

U. Fayyad, G. Piatetsky-Shapiro, and P. Smyth. From data mining to knowledge discovery: An overview. In Advances in Knowledge Discovery and Data Mining, pages 1--34. MIT Press, 1996.

Digital Library

[6]

B. Frey and D. Dueck. Clustering by passing messages between data points. Science, 315:972--976, 2007.

[7]

J. Gama, R. Rocha, and P. Medas. Accurate decision trees for mining highspeed data streams. In SIGMOD, pages 523--528, 2003.

Digital Library

[8]

J. Gama and P. P. Rodrigues. Stream-based electricity load forecast. In PKDD, pages 446--453, 2007.

[9]

S. Guha, A. Meyerson, N. Mishra, R. Motwani, and L. O'Callaghan. Clustering data streams: Theory and practice. TKDE, 15:515--528, 2003.

Digital Library

[10]

Z. Harchaoui, F. Bach, and E. Moulines. Kernel change-point analysis. In NIPS, 2008.

[11]

D. Hinkley. Inference about the change-point from cumulative sum tests. Biometrika, 58:509--523, 1971.

[12]

D. Judd, P. K. McKinley, and A. K. Jain. Large-scale parallel data clustering. IEEE Trans. Pattern Anal. Mach. Intell., 20:871--876, 1998.

Digital Library

[13]

M. Leone, Sumedha, and M. Weigt. Clustering by soft-constraint afinity propagation: Applications to gene-expression data. Bioinformatics, 23:2708, 2007.

Digital Library

[14]

M. Meila. The uniqueness of a good optimum for k-means. In ICML, pages 625--632, 2006.

Digital Library

[15]

S. Muthukrishnan, E. v. d. Berg, and Y. Wu. Sequential change detection on data streams. In ICDM Workshops, 2007.

Digital Library

[16]

E. Page. Continuous inspection schemes. Biometrika, 41:100--115, 1954.

[17]

N. Palatin, A. Leizarowitz, A. Schuster, and R. Wolff. Mining for misconfigured machines in grid systems. In SIGKDD, pages 687--692, 2006.

Digital Library

[18]

C. E. Rasmussen and C. K. Williams. Gaussian Processes for Machine Learning. MIT Press, 01 2006.

Digital Library

[19]

I. Rish, M. Brodie, and S. M. et al. Adaptive diagnosis in distributed systems. IEEE Trans. on Neural Networks, 16:1088--1109, 2005.

Digital Library

[20]

G. Schwarz. Estimating the dimension of a model. The Annals of Statistics, 6:461--464, 1978.

[21]

J. Villemonteix, E. Vazquez, M. Sidorkiewicz and E. Walter. Global optimization of expensive-to-evaluate functions: an empirical comparison of two sampling criteria. Journal of Global Optimization, vol 43 (2-3), p.373--389, 2009

Digital Library

[22]

X. Zhang, C. Furtlehner, and M. Sebag. Data streaming with afinity propagation. In ECML/PKDD, pages 628--643, 2008.

[23]

J. Andreeva, B. Gaidioz, J. Herrala, and et al. Dashboard for the LHC experiments. Journal of Physics: Conference Series, vol. 119, 2008.

[24]

Real Time Monitor: http://gridportal.hep.ph.ic.ac.uk/rtm/.

[25]

X. Zhang, C. Furtlehner, and M. Sebag. INRIA research report in progress.

Cited By

Faroughi ABoostani RTajalizadeh HJavidan R(2023)ARD-Stream: An adaptive radius density-based stream clusteringFuture Generation Computer Systems10.1016/j.future.2023.07.027149(416-431)Online publication date: Dec-2023
https://doi.org/10.1016/j.future.2023.07.027
Sui JLiu ZJung ALiu LLi X(2018)Dynamic Clustering Scheme for Evolving Data Streams Based on Improved STRAPIEEE Access10.1109/ACCESS.2018.28645536(46157-46166)Online publication date: 2018
https://doi.org/10.1109/ACCESS.2018.2864553
Chen CHe LZhang HZheng HWang L(2017)On the Accuracy and Parallelism of GPGPU-Powered Incremental Clustering AlgorithmsComputational Intelligence and Neuroscience10.1155/2017/25197822017Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1155/2017/2519782
Show More Cited By

Index Terms

Toward autonomic grids: analyzing the job flow with affinity streaming
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis
2. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

An Autonomic Workflow Management System for Global Grids
CCGRID '08: Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid

Workflow Management System is generally utilized to define, manage and execute workflow applications on Grid resources. However, the increasing scale complexity, heterogeneity and dynamism of Grid environment that includes networks, resources and ...
Initializing K-means Clustering Using Affinity Propagation
HIS '09: Proceedings of the 2009 Ninth International Conference on Hybrid Intelligent Systems - Volume 01

K-means clustering is widely used due to its fast convergence, but it is sensitive to the initial condition.Therefore, many methods of initializing K-means clustering have been proposed in the literatures. Compared with Kmeans clustering, a novel ...
A Support System for Clustering Data Streams with a Variable Number of Clusters
Special Section on Best Papers from SASO 2014 and Regular Articles

Many algorithms for clustering data streams that are based on the widely used k-Means have been proposed in the literature. Most of these algorithms assume that the number of clusters, k, is known and fixed a priori by the user. Aimed at relaxing this ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining

June 2009

1426 pages

ISBN:9781605584959

DOI:10.1145/1557019

General Chairs:
John Elder
Elder Research, Inc., USA
,
Françoise Soulié Fogelman
KXEN, France
,
Program Chairs:
Peter Flach
University of Bristol, UK
,
Mohammed Zaki
RPI, USA

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD09

Sponsor:

KDD09: The 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

June 28 - July 1, 2009

Paris, France

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
498
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Faroughi ABoostani RTajalizadeh HJavidan R(2023)ARD-Stream: An adaptive radius density-based stream clusteringFuture Generation Computer Systems10.1016/j.future.2023.07.027149(416-431)Online publication date: Dec-2023
https://doi.org/10.1016/j.future.2023.07.027
Sui JLiu ZJung ALiu LLi X(2018)Dynamic Clustering Scheme for Evolving Data Streams Based on Improved STRAPIEEE Access10.1109/ACCESS.2018.28645536(46157-46166)Online publication date: 2018
https://doi.org/10.1109/ACCESS.2018.2864553
Chen CHe LZhang HZheng HWang L(2017)On the Accuracy and Parallelism of GPGPU-Powered Incremental Clustering AlgorithmsComputational Intelligence and Neuroscience10.1155/2017/25197822017Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1155/2017/2519782
Saki FKehtarnavaz N(2016)Online frame-based clustering with unknown number of clustersPattern Recognition10.1016/j.patcog.2016.03.01057:C(70-83)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1016/j.patcog.2016.03.010
Zimmermann MNtoutsi ESpiliopoulou M(2015)Discovering and monitoring product features and the opinions on them with OPINSTREAMNeurocomputing10.1016/j.neucom.2014.04.079150:PA(318-330)Online publication date: 20-Feb-2015
https://dl.acm.org/doi/10.1016/j.neucom.2014.04.079
Zhang XFurtlehner CGermain-Renaud CSebag M(2014)Data Stream Clustering With Affinity PropagationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2013.14626:7(1644-1656)Online publication date: Jul-2014
https://doi.org/10.1109/TKDE.2013.146
Wang WGuyet TQuiniou RCordier MMasseglia FZhang X(2014)Autonomic intrusion detectionKnowledge-Based Systems10.1016/j.knosys.2014.06.01870:C(103-117)Online publication date: 1-Nov-2014
https://dl.acm.org/doi/10.1016/j.knosys.2014.06.018
Atwa WLi K(2014)Clustering Evolving Data Stream with Affinity Propagation AlgorithmDatabase and Expert Systems Applications10.1007/978-3-319-10073-9_38(446-453)Online publication date: 2014
https://doi.org/10.1007/978-3-319-10073-9_38
Wang CLai JHuang DZheng W(2013)SVStreamIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2011.26325:6(1410-1424)Online publication date: 1-Jun-2013
https://dl.acm.org/doi/10.1109/TKDE.2011.263
Jian-Peng Zhang Fu-Cai Chen Li-Xiong Liu Shao-Mei Li (2013)Online stream clustering using density and affinity propagation algorithm2013 IEEE 4th International Conference on Software Engineering and Service Science10.1109/ICSESS.2013.6615433(828-832)Online publication date: May-2013
https://doi.org/10.1109/ICSESS.2013.6615433
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents