short-paper

Open access

M&MMs: navigating complex memory spaces with hwloc

Authors:

Edgar A. León,

Andres Rubio ProañoAuthors Info & Claims

MEMSYS '19: Proceedings of the International Symposium on Memory Systems

Pages 149 - 155

https://doi.org/10.1145/3357526.3357546

Published: 30 September 2019 Publication History

Abstract

The complexity of the memory system has increased dramatically in the last decade. As a result, high-performance computers include multi-level, heterogeneous, and non-uniform memories, each with significantly different properties. For example, a memory system nowadays may include three types of memory: low-latency memory (DDR), high-bandwidth memory (HBM), and high-capacity memory (NVM)--not to mention multiple NUMA domains. Because of their significantly different characteristics and number, scientific application developers face a tremendous challenge: Leverage the memory system effectively to improve performance and productivity.

In this work, we present M&MMs, an interface to help manage the memory system complexity. It is comprised of a set of memory attributes and an API to express and manage the diverse memory characteristics using high-level metrics that are easy to understand. Our goal is to establish a building block to enable next-generation runtime systems, computing libraries, and scientific applications to leverage the best performance attributes of each memory, e.g., leverage the bandwidth of the fastest memory with the capacity of the largest memory. We believe M&MMs is a natural extension of hwloc--that focuses on the memory system--since hwloc exposes the locality of the hardware resources and it is the de facto standard for hardware topology discovery.

References

[1]

W. Allcock, B. Bernardoni, C. Bertoni, N. Getty, J. Insley, M. E. Papka, S. Rizzi, and B. Toonen. 2018. RAM as a Network Managed Resource. In 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). 99--106.

[2]

Mohamed Arafa, Bahaa Fahim, Sailesh Kottapalli, Akhilesh Kumar, Lily P Looi, Sreenivas Mandava, Andy Rudoff, Ian M Steiner, Bob Valentine, Geetha Vedaraman, et al. 2019. Cascade Lake: Next generation Intel Xeon scalable processor. IEEE Micro 39, 2 (2019), 29--36.

[3]

David A. Beckingsale and Richard D. Hornung. 2018. Umpire: Status Report and Future Development Plan. Technical Report LLNL-TR-754118. Office of Scientific and Technical Information.

[4]

François Broquedis, Jérôme Clet-Ortega, Stéphanie Moreaud, Nathalie Furmento, Brice Goglin, Guillaume Mercier, Samuel Thibault, and Raymond Namyst. 2010. hwloc: a Generic Framework for Managing Hardware Affinities in HPC Applications. In Proceedings of the 18th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP2010). IEEE Computer Society Press, Pisa, Italia, 180--186.

Digital Library

[5]

Christopher Cantalupo, Vishwanath Venkatesan, Jeff R. Hammond, and Simon Hammond. 2015. User Extensible Heap Manager for Heterogeneous Memory Platforms and Mixed Memory Policies. http://memkind.github.io/memkind/memkind_arch_20150318.pdf.

[6]

CORAL: Collaboration of Oak Ridge, Argonne and Livermore National Laboratories. 2013. Draft CORAL Build Statement of Work. RFP No. B604142, LLNL-PROP-636244. Office of Science and the National Nuclear Security Administration's Advanced Simulation and Computing (ASC) Program, U.S. Department of Energy. https://asc.llnl.gov/CORAL/RFP_components/02_draft_CORAL_Build_SOW_12-31-13.pdf

[7]

Fujitsu. 2019. Supercomputer Fugaku. ISC High Performance, Booth Presentation.

[8]

Brice Goglin. 2016. Exposing the Locality of Heterogeneous Memory Architectures to HPC Applications. In International Symposium on Memory Systems (MEMSYS'16). ACM, Washington, DC.

Digital Library

[9]

Edgar A. León and Matthieu Hautreux. 2018. Achieving Transparency Mapping Parallel Applications: A Memory Hierarchy Affair. In International Symposium on Memory Systems (MEMSYS'18). ACM, Washington, DC.

Digital Library

[10]

Ang Li, Weifeng Liu, Mads R. B. Kristensen, Brian Vinter, Hao Wang, Kaixi Hou, Andres Marquez, and Shuaiwen Leon Song. 2017. Exploring and Analyzing the Real Impact of Modern On-package Memory on HPC Scientific Kernels. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC'17). ACM, Denver, CO.

Digital Library

[11]

Aditya Narayan, Tiansheng Zhang, Shaizeen Aga, Satish Narayanasamy, and Ayse K. Coskun. 2018. MOCA: Memory Object Classification and Allocation in Heterogeneous Memory Systems. In Proceedings of the 2018 IEEE International Parallel and Distributed Processing Symposium. IEEE, Vancouver, BC, Canada.

[12]

NERSC. 2017. KNL Cache Mode Performance. https://www.nersc.gov/research-and-development/knl-cache-mode-performance-coe/

[13]

Harald Servat, Antonio Pena, German Llort, Estanislao Mercadal, Hans-Christian Hoppe, and Jesus Labarta. 2017. Automating the Application Data Placement in Hybrid Memory Systems. In Proceedings of the IEEE International Conference on Cluster Computing. Hawaii, USA.

[14]

Avinash Sodani, Roger Gramunt, Jesus Corbal, Ho-Seop Kim, Krishna Vinod, Sundaram Chinthamani, Steven Hutsell, Rajat Agarwal, and Yen-Chen Liu. 2016. Knights Landing: Second-Generation Intel Xeon Phi Product. IEEE Micro 36, 2 (March 2016), 34--46.

Digital Library

[15]

Alexander van Renen, Lukas Vogel, Viktor Leis, Thomas Neumann, and Alfons Kemper. 2019. Persistent Memory I/O Primitives. arXiv e-prints abs/1904.01614 (April 2019). arXiv:1904.01614 http://arxiv.org/abs/1904.01614

[16]

Sudharshan S. Vazhkudai, Bronis R. de Supinski, Arthur S. Bland, Al Geist, James Sexton, Jim Kahle, Christopher J. Zimmer, Scott Atchley, Sarp Oral, Don E. Maxwell, Veronica G. Vergara Larrea, Adam Bertsch, Robin Goldstone, Wayne Joubert, Chris Chambreau, David Appelhans, Robert Blackmore, Ben Casses, George Chochia, Gene Davison, Matthew A. Ezell, Tom Gooding, Elsa Gonsiorowski, Leopold Grinberg, Bill Hanson, Bill Hartner, Ian Karlin, Matthew L. Leininger, Dustin Leverman, Chris Marroquin, Adam Moody, Martin Ohmacht, Ramesh Pankajakshan, Fernando Pizzano, James H. Rogers, Bryan Rosenburg, Drew Schmidt, Mallikarjun Shankar, Feiyi Wang, Py Watson, Bob Walkup, Lance D. Weems, and Junqi Yin. 2018. The Design, Deployment, and Evaluation of the CORAL Pre-exascale Systems. In International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'18). IEEE, Dallas, Texas, Article 52, 12 pages.

Digital Library

[17]

Gwendolyn Voskuilen, Arun F. Rodrigues, and Simon D. Hammond. 2016. Analyzing Allocation Behavior for Multi-level Memory. In Proceedings of the Second International Symposium on Memory Systems (MEMSYS '16). ACM, New York, NY, USA, 204--207.

Digital Library

[18]

Sean Williams, Latchesar Ionkov, and Michael Lang. 2017. NUMA Distance for Heterogeneous Memory. In Proceedings of the Workshop on Memory Centric Programming for HPC (MCHPC'17). ACM, New York, NY, USA, 30--34.

Digital Library

Cited By

Köhler SHerzog BHerzog HVögele MWenzel LPolze AHönig T(2024)Carbon-Aware Memory PlacementACM SIGEnergy Energy Informatics Review10.1145/3698365.36983724:3(39-45)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3698365.3698372
Foyer CGoglin B(2021)Using Bandwidth Throttling to Quantify Application Sensitivity to Heterogeneous Memory2021 IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC)10.1109/MCHPC54807.2021.00008(9-16)Online publication date: Nov-2021
https://doi.org/10.1109/MCHPC54807.2021.00008

Index Terms

M&MMs: navigating complex memory spaces with hwloc
1. Computer systems organization
  1. Architectures
    1. Other architectures
      1. Heterogeneous (hybrid) systems
2. Hardware
  1. Emerging technologies
    1. Emerging interfaces
    2. Memory and dense storage
  2. Integrated circuits
    1. Semiconductor memory
      1. Non-volatile memory

Recommendations

Performance characterization of a DRAM-NVM hybrid memory architecture for HPC applications using intel optane DC persistent memory modules
MEMSYS '19: Proceedings of the International Symposium on Memory Systems

Non-volatile, byte-addressable memory (NVM) has been introduced by Intel in the form of NVDIMMs named Intel® Optane™ DC PMM. This memory module has the ability to persist the data stored in it without the need for power. This expands the memory ...
DEUCE: Write-Efficient Encryption for Non-Volatile Memories
ASPLOS'15

Phase Change Memory (PCM) is an emerging Non Volatile Memory (NVM) technology that has the potential to provide scalable high-density memory systems. While the non-volatility of PCM is a desirable property in order to save leakage power, it also has the ...
DEUCE: Write-Efficient Encryption for Non-Volatile Memories
ASPLOS '15

Phase Change Memory (PCM) is an emerging Non Volatile Memory (NVM) technology that has the potential to provide scalable high-density memory systems. While the non-volatility of PCM is a desirable property in order to save leakage power, it also has the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MEMSYS '19: Proceedings of the International Symposium on Memory Systems

September 2019

517 pages

ISBN:9781450372060

DOI:10.1145/3357526

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

MEMSYS '19

MEMSYS '19: The International Symposium on Memory Systems

September 30 - October 3, 2019

District of Columbia, Washington, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
328
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)16

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Köhler SHerzog BHerzog HVögele MWenzel LPolze AHönig T(2024)Carbon-Aware Memory PlacementACM SIGEnergy Energy Informatics Review10.1145/3698365.36983724:3(39-45)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3698365.3698372
Foyer CGoglin B(2021)Using Bandwidth Throttling to Quantify Application Sensitivity to Heterogeneous Memory2021 IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC)10.1109/MCHPC54807.2021.00008(9-16)Online publication date: Nov-2021
https://doi.org/10.1109/MCHPC54807.2021.00008

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents