research-article

Enabling comprehensive data-driven system management for large computational facilities

Authors:

James C. Browne,

Robert L. DeLeon,

Matthew D. Jones,

Steven M. Gallo,

Amin Ghadersohi,

Abani K. Patra,

William L. Barth,

Thomas R. Furlani,

Robert T. McLayAuthors Info & Claims

SC '13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Article No.: 86, Pages 1 - 11

https://doi.org/10.1145/2503210.2503230

Published: 17 November 2013 Publication History

Abstract

This paper presents a tool chain, based on the open source tool TACC_Stats, for systematic and comprehensive job level resource use measurement for large cluster computers, and its incorporation into XDMoD, a reporting and analytics framework for resource management that targets meeting the information needs of users, application developers, systems administrators, systems management and funding managers. Accounting, scheduler and event logs are integrated with system performance data from TACC_Stats. TACC_Stats periodically records resource use including many hardware counters for each job running on each node. Furthermore, system level metrics are obtained through aggregation of the node (job) level data. Analysis of this data generates many types of standard and custom reports and even a limited predictive capability that has not previously been available for open-source, Linux-based software systems. This paper presents case studies of information that can be applied for effective resource management. We believe this system to be the first fully comprehensive system for supporting the information needs of all stakeholders in open-source software based HPC systems.

References

[1]

Del Vento, D., Engel, T., Ghosh S., Hart, D., Kelly, R., Liu, S. and Valent, R. "System-level monitoring of floating-point performance to improve effective system utilization." In 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC11)

Digital Library

[2]

Furlani, T. R., Jones, M. D., Gallo, S. M., Bruno, A. E., Lu, C.-D., Ghadersohi, A., Gentner, R. J., Patra, A., DeLeon, R. L., von Laszewski, G., Wang, F., and Zimmerman, A., "Performance metrics and auditing framework using application kernels for high performance computer systems," Concurrency and Computation: Practice and Experience, 2012. {Online} Available: http://dx.doi.org/10.1002/cpe.2871 XDMoD: http://xdmod.ccr.buffalo.edu

[3]

http://www.thegeekstuff.com/2011/12/linux-performance-monitoring-tools/

[4]

K. A. Huck, A. D. Malony, S. Shende, and A. Morris. "Knowledge Support and Automation for Performance Analysis with PerfExplorer 2.0." Large-Scale Programming Tools and Environments, Special Issue of Scientific Programming, vol. 16, no. 2-3, pp. 123--134. 2008.

Digital Library

[5]

S. Shende and A. Malony. "The Tau Parallel Performance System." International Journal of High Performance Computing Applications, 20(2): 287--311.

Digital Library

[6]

Tau: http://www.cs.uoregon.edu/research/tau/home.php.

[7]

N. R. Tallent, J. M. Mellor-Crummey, L. Adhianto, M. W. Fagan, and M. Krentel. "HPCToolkit: performance tools for scientific computing." Journal of Physics: Conference Series, 125. 2008.

[8]

L. Djoudi, D. Barthou, P. Carribault, C. Lemuet, J.-T. Acquaviva, and W. Jalby. "Exploring Application Performance: a New Tool for a Static/Dynamic Approach." The Sixth Los Alamos Computer Science Institute Symp. 2005.

[9]

HPCToolkit: http://www.hpctoolkit.org/. Last accessed April 1, 2011.

[10]

Open|SpeedShop: http://www.openspeedshop.org/wp/.

[11]

M. Geimer, P. Saviankou, A. Strube, Z. Szebenyi, F. Wolf, B. J. N. Wylie: Further improving the scalability of the Scalasca toolset. In Proc. of PARA 2010: State of the Art in Scientific and Parallel Computing, Part II: Minisymposium Scalable tools for High Performance Computing, Reykjavik, Iceland, June 6--9 2010, volume 7134 of Lecture Notes in Computer Science, pages 463--474, Springer, 2012.

Digital Library

[12]

VTune: software.intel.com/intel-vtune-amplifier-xe

[13]

B. P. Miller, M. D. Callaghan, J. M. Cargille, J. K. Hollingsworth, R. B. Irvin, K. L. Karavanic, K. Kunchithapadam, and T. Newhall. "The Paradyn Parallel Performance Measurement Tool." IEEE Computer, 28:37--46. 1995.

Digital Library

[14]

M. Burtscher, B. D. Kim, J. Diamond, J. McCalpin, L. Koesterke, and J. Browne. "PerfExpert: An Easy-to-Use Performance Diagnosis Tool for HPC Applications." SC 2010 Int. Conference for High-Performance Computing, Networking, Storage and Analysis. November 2010.

Digital Library

[15]

PerfExpert: http://www.tacc.utexas.edu/perfexpert/.

[16]

http://clumon.ncsa.illinois.edu/

[17]

http://oss.sgi.com/projects/pcp/

[18]

http://ganglia.sourceforge.net/

[19]

http://www.nagios.org/

[20]

http://www.komodolabs.com/

[21]

https://e-reports-ext.llnl.gov/pdf/754636.pdf

[22]

https://computing.llnl.gov/linux/slurm/overview.html

[23]

http://www.splunk.com/

[24]

http://github.com/TACCProjects/tacc_stats

[25]

Hammond, J. "TACC_stats: I/O performance monitoring for the intransigent" In 2011 Workshop for Interfaces and Architectures for Scientific Data Storage (IASDS 2011)

[26]

Edward Chuah_, Arshad Jhumka_, Sai Narasimhamurthy, John Hammond, James C. Browne, Bill Barth Linking Resource Usage Anomalies with System Failures from Cluster Log Data (Submitted to SRDS 2013 -- Preprint available from mailto:[email protected])

Digital Library

[27]

J. L. Hammond, T. Minyard, and J. Browne, "End-to-end framework for fault management for open source clusters: Ranger," in Proceedings of ACM TeraGrid, no. 9, 2010.

Digital Library

[28]

Scott, D. W. "Multivariate Density Estimation". Wiley, New York, 1992.

Cited By

Захаров ИПанарин ОРыкованов СЗагидуллин РМалютин АШкандыбин ЮЕрмекова А(2021)Monitoring applications on the ZHORES cluster at SkoltechProgram Systems: Theory and ApplicationsПрограммные системы: теория и приложения10.25209/2079-3316-2021-12-2-73-10312:2(73-103)Online publication date: 2021
https://doi.org/10.25209/2079-3316-2021-12-2-73-103
ChuahM EJhumka AAlt SEvans RSuri N(2021)Failure Diagnosis for Cluster Systems using Partial Correlations2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00151(1091-1101)Online publication date: Sep-2021
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00151
Sperhac JDeLeon RWhite JJones MBruno AIvey RFurlani TBard JChaudhary V(2020)Towards Performant Workflows, Monitoring and Measuring2020 29th International Conference on Computer Communications and Networks (ICCCN)10.1109/ICCCN49398.2020.9209647(1-9)Online publication date: Aug-2020
https://doi.org/10.1109/ICCCN49398.2020.9209647
Show More Cited By

Recommendations

Large Scale Data Management on Grids
ADVCOMP '08: Proceedings of the 2008 The Second International Conference on Advanced Engineering Computing and Applications in Sciences

We explore the status, challenges and opportunities in providing a resource center focused on large scale data management on the EGEE Grid, in the context of the emerging Spanish e-Science Network. The application to bulky and relatively flat datasets ...
Tiered data management system: Accelerating data processing on HPC systems
Abstract
The explosion of scientific data generated from large-scale simulations and advanced sensors makes scientific workflows more complex and more data-intensive. Supporting these data-intensive workflows on high-performance computing systems presents ...
Highlights
- Optimizing I/O performance for scientific workflows.
- Data management systems on tiered storage architecture.
- Customizing data management strategies for different workflow access patterns.
- Data-aware task scheduling.
Data driven workflow planning in cluster management systems
HPDC '07: Proceedings of the 16th international symposium on High performance distributed computing

Traditional scientific computing has been associated with harnessing computation cycles within and across clusters of machines. In recent years, scientific applications have become increasingly data-intensive. This is especially true in the fields of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

November 2013

1123 pages

ISBN:9781450323789

DOI:10.1145/2503210

General Chair:
William Gropp
University of Illinois at Urbana-Champaign, Urbana, Illinois
,
Program Chair:
Satoshi Matsuoka
Tokyo Institute of Technology, Tokyo, Japan

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Office of Cyberinfrastructure

Conference

SC13

Sponsor:

SIGHPC
SIGARCH
IEEE-CS

SC13: International Conference for High Performance Computing, Networking, Storage and Analysis

November 17 - 21, 2013

Colorado, Denver

Acceptance Rates

SC '13 Paper Acceptance Rate 91 of 449 submissions, 20%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
185
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Захаров ИПанарин ОРыкованов СЗагидуллин РМалютин АШкандыбин ЮЕрмекова А(2021)Monitoring applications on the ZHORES cluster at SkoltechProgram Systems: Theory and ApplicationsПрограммные системы: теория и приложения10.25209/2079-3316-2021-12-2-73-10312:2(73-103)Online publication date: 2021
https://doi.org/10.25209/2079-3316-2021-12-2-73-103
ChuahM EJhumka AAlt SEvans RSuri N(2021)Failure Diagnosis for Cluster Systems using Partial Correlations2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00151(1091-1101)Online publication date: Sep-2021
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00151
Sperhac JDeLeon RWhite JJones MBruno AIvey RFurlani TBard JChaudhary V(2020)Towards Performant Workflows, Monitoring and Measuring2020 29th International Conference on Computer Communications and Networks (ICCCN)10.1109/ICCCN49398.2020.9209647(1-9)Online publication date: Aug-2020
https://doi.org/10.1109/ICCCN49398.2020.9209647
Li JAli GNguyen NHass JSill ADang TChen Y(2020)MonSTer: An Out-of-the-Box Monitoring Tool for High Performance Computing Systems2020 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER49012.2020.00022(119-129)Online publication date: Sep-2020
https://doi.org/10.1109/CLUSTER49012.2020.00022
Sperhac JDeLeon RFurlani TGallo SInnus MJones MPalmer JPatra APlessinger BRathsam RSimakov NWhite JChakraborty RDean G(2019)Managing computational gateway resources with XDMoDFuture Generation Computer Systems10.1016/j.future.2019.03.02998:C(154-166)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1016/j.future.2019.03.029
Sperhac JPlessinger BPalmer JChakraborty RDean GInnus MRathsam RSimakov NWhite JFurlani TGallo SDeLeon RJones MCornelius CPatra A(2018)Federating XDMoD to Monitor Affiliated Computing Resources2018 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER.2018.00074(580-589)Online publication date: Sep-2018
https://doi.org/10.1109/CLUSTER.2018.00074
Evans RBrowne JBarth W(2016)Understanding Application and System Performance Through System-Wide Monitoring2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW.2016.145(1702-1710)Online publication date: May-2016
https://doi.org/10.1109/IPDPSW.2016.145
Yoo WKoo MCao YSim ANugent PWu K(2016)Performance Analysis Tool for HPC and Big Data Applications on Scientific ClustersConquering Big Data with High Performance Computing10.1007/978-3-319-33742-5_7(139-161)Online publication date: 17-Sep-2016
https://doi.org/10.1007/978-3-319-33742-5_7
Birngruber EForai PZauner ABording CGamblin THansper V(2015)Total recallProceedings of the Second International Workshop on HPC User Support Tools10.1145/2834996.2835001(1-12)Online publication date: 15-Nov-2015
https://dl.acm.org/doi/10.1145/2834996.2835001
DeLeon RFurlani TGallo SWhite JJones MPatra AInnus MYearke TPalmer JSperhac JRathsam RSimakov Nvon Laszewski GWang FPeterson G(2015)TAS view of XSEDE users and usageProceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure10.1145/2792745.2792766(1-8)Online publication date: 26-Jul-2015
https://dl.acm.org/doi/10.1145/2792745.2792766
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents