research-article

MPI-focused Tracing with OTFX: An MPI-aware In-memory Event Tracing Extension to the Open Trace Format 2

Authors:

Michael Wagner,

Jens Doleschal,

Andreas KnüpferAuthors Info & Claims

EuroMPI '15: Proceedings of the 22nd European MPI Users' Group Meeting

Article No.: 7, Pages 1 - 8

https://doi.org/10.1145/2802658.2802664

Published: 21 September 2015 Publication History

Abstract

Performance analysis tools are more than ever inevitable to develop applications that utilize the enormous computing resources of high performance computing (HPC) systems. In event-based performance analysis the amount of collected data is one of the most urgent challenges. The resulting measurement bias caused by uncoordinated intermediate memory buffer flushes in the monitoring tool can render a meaningful analysis of the parallel behavior impossible. In this paper we address the impact of intermediate memory buffer flushes and present a method to avoid file interaction in the monitoring tool entirely. We propose an MPI-focused tracing approach that provides the complete MPI communication behavior and adapts the remaining application events to an amount that fits into a single memory buffer. We demonstrate the capabilities of our method with an MPI-focused prototype implementation of OTFX, based on the Open Trace Format 2, a state-of-the-art Open Source event tracing library used by the performance analysis tools Vampir, Scalasca, and Tau. In a comparison to OTF2 based on seven applications from different scientific domains, our prototype introduces in average 5.1% less overhead and reduces the trace size up to three orders of magnitude.

References

[1]

Argonne National Laboratories. Nek5000 website, http://nek5000.mcs.anl.gov.

[2]

D. Eschweiler, M. Wagner, M. Geimer, A. Knüpfer, W. E. Nagel, and F. Wolf. Open Trace Format 2: The Next Generation of Scalable Trace Formats and Support Libraries. In Applications, Tools and Techniques on the Road to Exascale Computing, pages 481--490, 2012.

[3]

M. Geimer, F. Wolf, B. J. Wylie, E. Ábrahám, D. Becker, and B. Mohr. The Scalasca Performance Toolset Architecture. Concurrency and Computation: Practice and Experience, 22(6):702--719, 2010.

Digital Library

[4]

B. Hess, C. Kutzner, D. van der Spoel, and E. Lindahl. GROMACS 4: Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation. Journal of Chemical Theory and Computation, 4(3):435--447, 2008.

[5]

T. Ilsche, J. Schuchart, J. Cope, D. Kimpe, T. Jones, A. Knüpfer, K. Iskra, R. Ross, W. E. Nagel, and S. Poole. Enabling Event Tracing at Leadership-Class Scale through I/O Forwarding Middleware. In Proc. of the 21th Intern. Symposium on High Performance Distributed Computing, pages 49--60, 2012.

Digital Library

[6]

A. Knüpfer, H. Brunst, J. Doleschal, M. Jurenz, M. Lieber, H. Mickler, M. S. Müller, and W. E. Nagel. The Vampir Performance Analysis Tool Set. In Tools for High Performance Computing, pp. 139--155, 2008.

[7]

A. Knüpfer and W. E. Nagel. Compressible Memory Data Structures for Event-based Trace Analysis. Future Gener. Comput. Syst., 22(3):359--368, 2006.

Digital Library

[8]

A. Knüpfer, C. Rössel, D. Mey, S. Biersdorff, K. Diethelm, D. Eschweiler, M. Geimer, M. Gerndt, D. Lorenz, A. Malony, W. E. Nagel, Y. Oleynik, P. Philippen, P. Saviankou, D. Schmidl, S. Shende, R. Tschüter, M. Wagner, B. Wesarg, and F. Wolf. Score-P: A Joint Performance Measurement Run-Time Infrastructure for Periscope, Scalasca, TAU, and Vampir. In Tools for High Performance Computing 2011, pages 79--91, 2012.

[9]

M. Lieber, V. Grützun, R. Wolke, M. S. Müller, and W. E. Nagel. Highly Scalable Dynamic Load Balancing in the Atmospheric Modeling System COSMO-SPECS+FD4. In Applied Parallel and Scientific Computing, pages 131--141, 2012.

Digital Library

[10]

G. Llort, J. Gonzalez, H. Servat, J. Gimenez, and J. Labarta. On-line Detection of Large-scale Parallel Application's Structure. In Parallel Distributed Processing (IPDPS), 2010 IEEE International Symposium on, pages 1--10, 2010.

[11]

Message Passing Interface Forum. MPI: A Message-Passing Interface Standard, Version 3.1, 2015. http://www.mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf.

[12]

K. Mohror and K. L. Karavanic. Evaluating Similarity-based Trace Reduction Techniques for Scalable Performance Analysis. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, SC '09, pages 55:1--55:12, 2009.

Digital Library

[13]

J. Mußler, D. Lorenz, and F. Wolf. Reducing the Overhead of Direct Application Instrumentation Using Prior Static Analysis. In Proceedings of the 17th International Conference on Parallel Processing - Volume Part I, Euro-Par'11, pages 65--76, Berlin, Heidelberg, 2011. Springer-Verlag.

Digital Library

[14]

S. Plimpton. Fast Parallel Algorithms for Short-Range Molecular Dynamics. Journal of Computational Physics, 117(1):1--19, 1995.

Digital Library

[15]

Sandia National Laboratories. Lammps website. http://lammps.sandia.gov.

[16]

S. S. Shende and A. D. Malony. The Tau Parallel Performance System. Int. J. on High Performance Computing Applications, 20(2):287--311, 2006.

Digital Library

[17]

Top 500 supercomputer sites, http://www.top500.org.

[18]

M. Wagner, J. Doleschal, A. Knüpfer, and W. E. Nagel. Selective Runtime Monitoring: Non-intrusive Elimination of High-frequency Functions. In Proc. of the International Conference on High Performance Computing & Simulation, pages 295--302, 2014.

[19]

M. Wagner, J. Doleschal, W. E. Nagel, and A. Knüpfer. Runtime Message Uniquification for Accurate Communication Analysis on Incomplete MPI Event Traces. In Proceedings of the 20th European MPI Users' Group Meeting, pages 123--128, 2013.

Digital Library

[20]

M. Wagner, A. Knüpfer, and W. E. Nagel. Enhanced Encoding Techniques for the Open Trace Format 2. Procedia Computer Science, 9:1979--1987, 2012.

[21]

M. Wagner, A. Knüpfer, and W. E. Nagel. Hierarchical Memory Buffering Techniques for an In-Memory Event Tracing Extension to the Open Trace Format 2. In Parallel Processing (ICPP), 2013 42nd International Conference on, pages 970--976, 2013.

Digital Library

[22]

M. Wagner and W. E. Nagel. Strategies for Real-Time Event Reduction. In Euro-Par 2012: Parallel Processing Workshops, pages 429--438, 2013.

Digital Library

Cited By

Luo JYan TXu QSun JSun G(2024)Siesta: Synthesizing Proxy Applications for MPI Programs2024 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER59578.2024.00009(14-26)Online publication date: 24-Sep-2024
https://doi.org/10.1109/CLUSTER59578.2024.00009
Wang CGuo YBalaji PSnir M(2023)Near-Lossless MPI Tracing and Proxy Application AutogenerationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.321594234:1(123-140)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TPDS.2022.3215942
Wang CBalaji PSnir Mde Supinski BHall MGamblin T(2021)PilgrimProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3476151(1-14)Online publication date: 14-Nov-2021
https://dl.acm.org/doi/10.1145/3458817.3476151
Show More Cited By

Index Terms

MPI-focused Tracing with OTFX: An MPI-aware In-memory Event Tracing Extension to the Open Trace Format 2

Recommendations

An Overhead Analysis of MPI Profiling and Tracing Tools
PERMAVOST '22: Proceedings of the 2nd Workshop on Performance EngineeRing, Modelling, Analysis, and VisualizatiOn Strategy

MPI performance analysis tools are important instruments for finding performance bottlenecks in large-scale MPI applications. These tools commonly support either the profiling or the tracing of parallel applications. Depending on the type of analysis, ...
Hierarchical Memory Buffering Techniques for an In-Memory Event Tracing Extension to the Open Trace Format 2
ICPP '13: Proceedings of the 2013 42nd International Conference on Parallel Processing

One of the most urgent challenges in event based performance analysis is the enormous amount of collected data. A real-time event reduction is crucial to enable a complete in-memory event tracing workflow, which circumvents the limitations of current ...
Benefits of Cross Memory Attach for MPI libraries on HPC Clusters
XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

With the number of cores per node increasing in modern clusters, an efficient implementation of intra-node communications is critical for application performance. MPI libraries generally use shared memory mechanisms for communication inside the node, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EuroMPI '15: Proceedings of the 22nd European MPI Users' Group Meeting

September 2015

149 pages

ISBN:9781450337953

DOI:10.1145/2802658

Conference Chair:
Jack Dongarra,
Program Chairs:
Alexandre Denis,
Brice Goglin,
Emmanuel Jeannot,
Guillaume Mercier

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Conseil Régional d'Aquitaine
Communauté Urbaine de Bordeaux
INRIA: INRIA Rhône-Alpes

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

EuroMPI '15

EuroMPI '15: The 22nd European MPI Users' Group Meeting

September 21 - 23, 2015

Bordeaux, France

Acceptance Rates

EuroMPI '15 Paper Acceptance Rate 14 of 29 submissions, 48%;

Overall Acceptance Rate 66 of 139 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
92
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Luo JYan TXu QSun JSun G(2024)Siesta: Synthesizing Proxy Applications for MPI Programs2024 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER59578.2024.00009(14-26)Online publication date: 24-Sep-2024
https://doi.org/10.1109/CLUSTER59578.2024.00009
Wang CGuo YBalaji PSnir M(2023)Near-Lossless MPI Tracing and Proxy Application AutogenerationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.321594234:1(123-140)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TPDS.2022.3215942
Wang CBalaji PSnir Mde Supinski BHall MGamblin T(2021)PilgrimProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3476151(1-14)Online publication date: 14-Nov-2021
https://dl.acm.org/doi/10.1145/3458817.3476151
Wang CSun JSnir MMohror KGonsiorowski E(2020)Recorder 2.0: Efficient Parallel I/O Tracing and Analysis2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW50202.2020.00176(1-8)Online publication date: May-2020
https://doi.org/10.1109/IPDPSW50202.2020.00176
Wagner MKnüpfer ANagel W(2016)OTFX: An In-memory Event Tracing Extension to the Open Trace Format 2Algorithms and Architectures for Parallel Processing10.1007/978-3-319-49956-7_1(3-17)Online publication date: 19-Nov-2016
https://doi.org/10.1007/978-3-319-49956-7_1

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents