Computer Science > Distributed, Parallel, and Cluster Computing
[Submitted on 28 Dec 2020 (v1), revised 5 Jan 2021 (this version, v3), latest version 20 Apr 2021 (v8)]
Title:TEMPI: An Interposed MPI Library with a Canonical Representation of CUDA-aware Datatypes
View PDFAbstract:MPI Derived Datatypes are an abstraction that simplifies handling of non-contiguous data in MPI applications. These datatypes are recursively constructed at runtime from primitive Named Types defined in the MPI standard. More recently, the development and deployment of CUDA-aware MPI implementations has encouraged the transition of distributed high-performance MPI codes to use GPUs. These implementations allow MPI functions to directly operate on GPU buffers, easing integration of GPU compute into MPI codes. Despite substantial attention to CUDA-aware MPI implementations, they continue to offer cripplingly poor GPU performance when manipulating derived datatypes on GPUs. This work presents a new MPI library, TEMPI, to address this issue. TEMPI first introduces a common datatype to represent equivalent MPI derived datatypes. TEMPI can be integrated into existing MPI deployments as an interposed library. This library can be used regardless of MPI deployment and without modifying application code. Furthermore, this work presents a performance model of GPU derived datatype handling, demonstrating that previously preferred "one-shot" methods are not always fastest. Ultimately, the interposed-library model of this work demonstrates MPI_Pack speedup of up to 720,400x and MPI\_Send speedup of up to 59,000x compared to the MPI implementation deployed on a leadership-class supercomputer. This yields speedup of more than 20,000x in a 3D halo exchange.
Submission history
From: Carl Pearson [view email][v1] Mon, 28 Dec 2020 17:01:39 UTC (946 KB)
[v2] Tue, 29 Dec 2020 23:12:35 UTC (1,396 KB)
[v3] Tue, 5 Jan 2021 01:22:51 UTC (2,225 KB)
[v4] Mon, 11 Jan 2021 19:43:20 UTC (2,255 KB)
[v5] Wed, 13 Jan 2021 14:01:49 UTC (2,179 KB)
[v6] Thu, 21 Jan 2021 22:44:17 UTC (2,145 KB)
[v7] Fri, 16 Apr 2021 15:44:44 UTC (2,281 KB)
[v8] Tue, 20 Apr 2021 22:33:04 UTC (2,281 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.