research-article

Runtime Correctness Analysis of MPI-3 Nonblocking Collectives

Authors:

Tobias Hilbrich,

Matthias Weber,

Joachim Protze,

Bronis R. de Supinski,

Wolfgang E. NagelAuthors Info & Claims

EuroMPI '16: Proceedings of the 23rd European MPI Users' Group Meeting

Pages 188 - 197

https://doi.org/10.1145/2966884.2966906

Published: 25 September 2016 Publication History

Abstract

The Message Passing Interface (MPI) includes nonblocking collective operations that support additional overlap between computation and communication. These new operations enable complex data movement between large numbers of processes. However, their asynchronous behavior hides and complicates the detection of defects in their use. We highlight a lack of correctness tool support for these operations and extend the MUST runtime MPI correctness tool to alleviate this complexity. We introduce a classification to summarize the types of correctness analyses that are applicable to MPI's nonblocking collectives. We identify complex wait-for dependencies in deadlock situations and incorrect use of communication buffers as the most challenging types of usage errors. We devise, demonstrate, and evaluate the applicability of correctness analyses for these errors. A scalable analysis mechanism allows our runtime approach to scale with the application. Benchmark measurements highlight the scalability and applicability of our approach at up to 4,096 application processes and with low overhead.

References

[1]

C. Falzone, A. Chan, E. Lusk, and W. Gropp. Collective Error Detection for MPI Collective Operations. In B. Martino, D. Kranzlmüller, and J. Dongarra, editors, Recent Advances in Parallel Virtual Machine and Message Passing Interface, volume 3666 of Lecture Notes in Computer Science, pages 138--147. Springer Berlin Heidelberg, 2005.

Digital Library

[2]

T. Hilbrich, B. R. de Supinski, W. E. Nagel, J. Protze, C. Baier, and M. S. Müller. Distributed Wait State Tracking for Runtime MPI Deadlock Detection. In Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis, SC'13, pages 16:1--16:12, New York, NY, USA, 2013. ACM.

Digital Library

[3]

T. Hilbrich, F. Hänsel, M. Schulz, B. R. de Supinski, M. S. Müller, W. E. Nagel, and J. Protze. Runtime MPI Collective Checking with Tree-Based Overlay Networks. In Recent Advances in the Message Passing Interface, Lecture Notes in Computer Science. Springer Berlin Heidelberg, 2013.

Digital Library

[4]

T. Hoefler, A. Lumsdaine, and W. Rehm. Implementation and Performance Analysis of Non-blocking Collective Operations for MPI. In Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC '07, pages 52:1--52:10, New York, NY, USA, 2007. ACM.

Digital Library

[5]

G. R. Luecke, H. Chen, J. Coyle, J. Hoekstra, M. Kraeva, and Y. Zou. MPI-CHECK: A Tool for Checking Fortran 90 MPI Programs. Concurrency and Computation: Practice and Experience, 15(2):93--100, 2003.

[6]

Message Passing Interface Forum. MPI: A Message-Passing Interface Standard, Version 3.0. http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf, 2012. Last visited on 27/11/2013.

[7]

P. Ohly and W. Krotz-Vogel. Automated MPI Correctness Checking: What if there was a magic option? In Proceedings of the 8th LCI International Conference on High-Performance Clustered Computing, 2007.

[8]

J. Protze, T. Hilbrich, A. Knüpfer, B. R. de Supinski, and M. S. Müller. Holistic Debugging of MPI Derived Datatypes. In Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium, IPDPS '12, pages 354--365. IEEE Computer Society, Washington, DC, USA, 2012.

Digital Library

[9]

J. Protze, T. Hilbrich, M. Schulz, B. R. de Supinski, W. E. Nagel, and M. S. Müller. MPI Runtime Error Detection with MUST: A Scalable and Crash-Safe Approach. In 43nd International Conference on Parallel Processing (ICPP), Fifth International Workshop on Parallel Software Tools and Tool Infrastructures, Los Alamitos, CA, USA, 2014. IEEE Computer Society.

Digital Library

[10]

E. Saillard, P. Carribault, and D. Barthou. Combining Static and Dynamic Validation of MPI Collective Communications. In Proceedings of the 20th European MPI Users' Group Meeting, EuroMPI '13, pages 117--122, New York, NY, USA, 2013. ACM.

Digital Library

[11]

V. Samofalov, V. Krukov, B. Kuhn, S. Zheltov, A. V. Konovalov, and J. DeSouza. Automated Correctness Analysis of MPI Programs with Intel(r) Message Checker. In Parallel Computing: Current & Future Issues of High-End Computing, Proceedings of the International Conference ParCo 2005, volume 33 of John von Neumann Institute for Computing Series, pages 901--908. Central Institute for Applied Mathematics, Jülich, Germany, 2005.

[12]

S. Siegel. Using MPI-Spin to Model Check MPI Programs with Nonblocking Communication. In Recent Advances in Parallel Virtual Machine and Message Passing Interface (EuroPVM/MPI), 2006.

[13]

S. F. Siegel and T. K. Zirkel. Automatic Formal Verification of MPI-Based Parallel Programs. In Proceedings of the 16th ACM Symposium on Principles and Practice of Parallel Programming, PPoPP '11, pages 309--310, New York, NY, USA, 2011. ACM.

Digital Library

[14]

J. L. Träff and J. Worringen. Verifying Collective MPI Calls. In D. Kranzlmüller, P. Kacsuk, and J. Dongarra, editors, Recent Advances in Parallel Virtual Machine and Message Passing Interface, volume 3241 of Lecture Notes in Computer Science, pages 18--27. Springer Berlin Heidelberg, 2004.

[15]

S. S. Vakkalanka, S. Sharma, G. Gopalakrishnan, and R. M. Kirby. ISP: A Tool for Model Checking MPI Programs. In Proceedings of the 13th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP '07, pages 285--286, New York, NY, USA, 2008. ACM.

Digital Library

[16]

J. S. Vetter and B. R. de Supinski. Dynamic Software Testing of MPI Applications with Umpire. In Proceedings of the 2000 ACM/IEEE conference on Supercomputing, Supercomputing '00, Washington, DC, USA, 2000. IEEE Computer Society.

Digital Library

Cited By

El Karchi JChen HTehraniJamsaz AJannesari APopov MSaillard E(2024)MPI Errors Detection using GNN Embedding and Vector Embedding over LLVM IR2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS57955.2024.00059(595-607)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPS57955.2024.00059
Van Beeumen RIbrahim KKahanamoku–Meyer GYao NYang C(2022)Enhancing scalability of a matrix-free eigensolver for studying many-body localizationInternational Journal of High Performance Computing Applications10.1177/1094342021106036536:3(307-319)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1177/10943420211060365
Laurent MSaillard EQuinson M(2021)The MPI Bugs Initiative: a Framework for MPI Verification Tools Evaluation2021 IEEE/ACM 5th International Workshop on Software Correctness for HPC Applications (Correctness)10.1109/Correctness54621.2021.00008(1-9)Online publication date: Nov-2021
https://doi.org/10.1109/Correctness54621.2021.00008
Show More Cited By

Recommendations

Correctness Analysis of MPI-3 Non-Blocking Communications in PARCOACH
EuroMPI '15: Proceedings of the 22nd European MPI Users' Group Meeting

MPI-3 provide functions for non-blocking collectives. To help programmers introduce non-blocking collectives to existing MPI programs, we improve the PARCOACH tool for checking correctness of MPI call sequences. These enhancements focus on correct call ...
MPI correctness checking for OpenMP/MPI applications

The MPI interface is the de-facto standard for message passing applications, but it is also complex and defines several usage patterns as erroneous. A current trend is the investigation of hybrid programming techniques that use MPI processes and ...
MPI+Threads: runtime contention and remedies
PPoPP 2015: Proceedings of the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

Hybrid MPI+Threads programming has emerged as an alternative model to the “MPI everywhere” model to better handle the increasing core density in cluster nodes. While the MPI standard allows multithreaded concurrent communication, such flexibility comes ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EuroMPI '16: Proceedings of the 23rd European MPI Users' Group Meeting

September 2016

225 pages

ISBN:9781450342346

DOI:10.1145/2966884

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

EuroMPI 2016

EuroMPI 2016: The 23rd European MPI Users' Group Meeting

September 25 - 28, 2016

Edinburgh, United Kingdom

Acceptance Rates

Overall Acceptance Rate 66 of 139 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
89
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

El Karchi JChen HTehraniJamsaz AJannesari APopov MSaillard E(2024)MPI Errors Detection using GNN Embedding and Vector Embedding over LLVM IR2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS57955.2024.00059(595-607)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPS57955.2024.00059
Van Beeumen RIbrahim KKahanamoku–Meyer GYao NYang C(2022)Enhancing scalability of a matrix-free eigensolver for studying many-body localizationInternational Journal of High Performance Computing Applications10.1177/1094342021106036536:3(307-319)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1177/10943420211060365
Laurent MSaillard EQuinson M(2021)The MPI Bugs Initiative: a Framework for MPI Verification Tools Evaluation2021 IEEE/ACM 5th International Workshop on Software Correctness for HPC Applications (Correctness)10.1109/Correctness54621.2021.00008(1-9)Online publication date: Nov-2021
https://doi.org/10.1109/Correctness54621.2021.00008
Eassa FAlghamdi AHaridi SKhemakhem MAl-Ghamdi AAlsolami E(2020)ACC_TEST: Hybrid Testing Approach for OpenACC-Based ProgramsIEEE Access10.1109/ACCESS.2020.29910098(80358-80368)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2991009
Alghamdi AEassa F(2019)OpenACC Errors Classification and Static Detection TechniquesIEEE Access10.1109/ACCESS.2019.29354987(113235-113253)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2019.2935498
Protze JTerboven CMüller MPetiton SEmad NMurai HBoku T(2017)Runtime Correctness Checking for Emerging Programming ParadigmsProceedings of the First International Workshop on Software Correctness for HPC Applications10.1145/3145344.3145490(21-27)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3145344.3145490

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents