research-article

A scalable observation system for introspection and in situ analytics

Authors:

Allen MalonyAuthors Info & Claims

ESPT '16: Proceedings of the 5th Workshop on Extreme-Scale Programming Tools

Pages 42 - 49

Published: 13 November 2016 Publication History

Get Access

Abstract

SOS is a new model for the online in situ characterization and analysis of complex high-performance computing applications. SOS employs a data framework with distributed information management and structured query and access capabilities. The primary design objectives of SOS are flexibility, scalability, and programmability. SOS provides a complete framework that can be configured with and used directly by an application, allowing for a detailed workflow analysis of scientific applications. This paper describes the model of SOS and the experiments used to validate and explore the performance characteristics of its implementation in SOSflow. Experimental results demonstrate that SOS is capable of observation, introspection, feedback and control of complex high-performance applications, and that it has desirable scaling properties.

References

[1]

Omar Aaziz, Jonathan Cook, and Hadi Sharifi. Push me pull you: Integrating opposing data transport modes for efficient hpc application monitoring. In Cluster Computing (CLUSTER), 2015 IEEE International Conference on, pages 674--681. IEEE, 2015.

Digital Library

Google Scholar

[2]

Anthony Agelastos, Benjamin Allan, Jim Brandt, Paul Cassella, Jeremy Enos, Joshi Fullop, Ann Gentile, Steve Monk, Nichamon Naksinehaboon, Jeff Ogden, et al. The lightweight distributed metric service: a scalable infrastructure for continuous monitoring of large scale computing systems and applications. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 154--165. IEEE Press, 2014.

Digital Library

Google Scholar

[3]

Guilherme da Cunha Rodrigues, Glederson Lessa dos Santos, Vinicius Tavares Guimaraes, Lisandro Zambenedetti Granville, and Liane Margarida Rockenbach Tarouco. An architecture to evaluate scalability, adaptability and accuracy in cloud monitoring systems. In Information Networking (ICOIN), 2014 International Conference on, pages 46--51. IEEE, 2014.

Crossref

Google Scholar

[4]

Todd Evans, William L Barth, James C Browne, Robert L DeLeon, Thomas R Furlani, Steven M Gallo, Matthew D Jones, and Abani K Patra. Comprehensive resource use monitoring for hpc systems with tacc stats. In Proceedings of the First International Workshop on HPC User Support Tools, pages 13--21. IEEE Press, 2014.

Digital Library

Google Scholar

[5]

Weiming Gu, Greg Eisenhauer, Eileen Kraemer, Karsten Schwan, John Stasko, Jeffrey Vetter, and Nirupama Mallavarupu. Falcon: On-line monitoring and steering of large-scale parallel programs. In Frontiers of Massively Parallel Computation, 1995. Proceedings. Frontiers' 95., Fifth Symposium on the, pages 422--429. IEEE, 1995.

Digital Library

Google Scholar

[6]

Kevin A Huck, Allen D Malony, Sameer Shende, and Alan Morris. Taug: Runtime global performance data access using mpi. In Recent Advances in Parallel Virtual Machine and Message Passing Interface, pages 313--321. Springer, 2006.

Digital Library

Google Scholar

[7]

Gregory Katsaros, Roland Kubert, and Georgina Gallizo. Building a service-oriented monitoring framework with rest and nagios. In Services Computing (SCC), 2011 IEEE International Conference on, pages 426--431. IEEE, 2011.

Digital Library

Google Scholar

[8]

Mahendra Kutare, Greg Eisenhauer, Chengwei Wang, Karsten Schwan, Vanish Talwar, and Matthew Wolf. Monalytics: online monitoring and analytics for managing large scale data centers. In Proceedings of the 7th international conference on Autonomic computing, pages 141--150. ACM, 2010.

Digital Library

Google Scholar

[9]

Matthew L Massie, Brent N Chun, and David E Culler. The ganglia distributed monitoring system: design, implementation, and experience. Parallel Computing, 30(7):817--840, 2004.

Crossref

Google Scholar

[10]

Rafael Keller Tesser and Philippe Olivier Alexandre Navaux. Dimvhcm: An on-line distributed monitoring data collection model. In Parallel, Distributed and Network-Based Processing (PDP), 2012 20th Euromicro International Conference on, pages 37--41. IEEE, 2012.

Digital Library

Google Scholar

[11]

Xuechen Zhang, Hasan Abbasi, Kevin Huck, and Allen Malony. Wowmon: A machine learning-based profiler for self-adaptive instrumentation of scientific workflows. Procedia Computer Science, 80:1507--1518, 2016.

Digital Library

Google Scholar

Cited By

View all

Do TPottier LFerreira da Silva RCaíno-Lores STaufer MDeelman E(2021)Assessing Resource Provisioning and Allocation of Ensembles of In Situ Workflows50th International Conference on Parallel Processing Workshop10.1145/3458744.3474051(1-10)Online publication date: 9-Aug-2021
https://dl.acm.org/doi/10.1145/3458744.3474051
Peterka TBard DBennett JBethel EOldfield RPouchard LSweeney CWolf M(2020)Priority research directions for in situ data managementInternational Journal of High Performance Computing Applications10.1177/109434202091362834:4(409-427)Online publication date: 1-Jul-2020
https://dl.acm.org/doi/10.1177/1094342020913628
Malony ARamesh SHuck KChaimov NShende S(2019)A Plugin Architecture for the TAU Performance SystemProceedings of the 48th International Conference on Parallel Processing10.1145/3337821.3337916(1-11)Online publication date: 5-Aug-2019
https://dl.acm.org/doi/10.1145/3337821.3337916
Show More Cited By

A scalable observation system for introspection and in situ analytics

Recommendations

An analysis of computational workloads for the ORNL Jaguar system
ICS '12: Proceedings of the 26th ACM international conference on Supercomputing

This study presents an analysis of science application workloads for the Jaguar Cray XT5 system during its tenure as a 2.3 petaflop supercomputer at Oak Ridge National Laboratory. Jaguar was the first petascale system to be deployed for open science and ...
Extreme-scale workflows: A perspective from the JLESC international community
Abstract
The Joint Laboratory for Extreme-Scale Computing (JLESC) focuses on software challenges in high-performance computing systems to meet the needs of today’s science campaigns, which often require large resources, consist of multiple tasks, and ...
Highlights
- We feature extreme-workflow systems developed by JLESC partners.
- We share the lessons learned while developing these extreme-scale workflows.
- We discuss open challenges and important research directions in extreme-scale workflows.
Introspection-based memory de-duplication and migration
VEE '13: Proceedings of the 9th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments

Memory virtualization abstracts a physical machine's memory resource and presents to the virtual machines running on it a piece of physical memory that could be shared, compressed and moved. To optimize the memory resource utilization by fully ...

Comments

Information & Contributors

Information

Published In

ESPT '16: Proceedings of the 5th Workshop on Extreme-Scale Programming Tools

November 2016

54 pages

ISBN:9781509039180

In-Cooperation

SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

IEEE Press

Publication History

Published: 13 November 2016

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SC16

Sponsor:

SIGHPC
IEEE-CS\DATC

SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 13 - 18, 2016

Utah, Salt Lake City

Acceptance Rates

Overall Acceptance Rate 2 of 4 submissions, 50%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
49
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)1

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Do TPottier LFerreira da Silva RCaíno-Lores STaufer MDeelman E(2021)Assessing Resource Provisioning and Allocation of Ensembles of In Situ Workflows50th International Conference on Parallel Processing Workshop10.1145/3458744.3474051(1-10)Online publication date: 9-Aug-2021
https://dl.acm.org/doi/10.1145/3458744.3474051
Peterka TBard DBennett JBethel EOldfield RPouchard LSweeney CWolf M(2020)Priority research directions for in situ data managementInternational Journal of High Performance Computing Applications10.1177/109434202091362834:4(409-427)Online publication date: 1-Jul-2020
https://dl.acm.org/doi/10.1177/1094342020913628
Malony ARamesh SHuck KChaimov NShende S(2019)A Plugin Architecture for the TAU Performance SystemProceedings of the 48th International Conference on Parallel Processing10.1145/3337821.3337916(1-11)Online publication date: 5-Aug-2019
https://dl.acm.org/doi/10.1145/3337821.3337916
Izadpanah RNaksinehaboon NBrandt JGentile ADechev D(2018)Integrating Low-latency Analysis into HPC System MonitoringProceedings of the 47th International Conference on Parallel Processing10.1145/3225058.3225086(1-10)Online publication date: 13-Aug-2018
https://dl.acm.org/doi/10.1145/3225058.3225086

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

An analysis of computational workloads for the ORNL Jaguar system

Extreme-scale workflows: A perspective from the JLESC international community

Introspection-based memory de-duplication and migration

Comments

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Recommendations

An analysis of computational workloads for the ORNL Jaguar system

Extreme-scale workflows: A perspective from the JLESC international community

Introspection-based memory de-duplication and migration

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations