research-article

Linux vs. lightweight multi-kernels for high performance computing: experiences at pre-exascale

Authors:

Kohei Tarumizu,

Takayuki Okamoto,

Masamichi Takagi,

Shinji Sumimoto,

Yutaka IshikawaAuthors Info & Claims

SC '21: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 11, Pages 1 - 13

https://doi.org/10.1145/3458817.3476162

Published: 13 November 2021 Publication History

Abstract

The long standing consensus in the High-Performance Computing (HPC) Operating Systems (OS) community is that lightweight kernel (LWK) based OSes have the potential to outperform Linux at extreme scale. To explore if LWKs live up to their expectation we developed IHK/McKernel, a lightweight multi-kernel OS designed for HPC, and deployed it on two high-end supercomputers to compare its performance against Linux. Oakforest-PACS, an Intel Xeon Phi (x86) based supercomputer, runs a moderately tuned Linux distribution. Fugaku, the world's fastest supercomputer at the time of writing this paper, is based on Fujitsu's A64FX (aarch64) CPU that runs a highly tuned Linux environment.

We discuss recent developments in our OS and provide a detailed description on the challenges of tuning Fugaku's Linux for high-end HPC. While in a moderately tuned environment McKernel significantly outperforms Linux (by up to approximately 2X), on Fugaku we observe an average of 4% speedup across all our experiments, with a few exceptions where the LWK outperforms Linux by up to 29%. As part of our evaluation we also disclose a full scale (158,976 compute nodes) noise profile of the Fugaku system.

Supplementary Material

MP4 File (Linux vs. Lightweight Multi-Kernels for High-Performance Computing_ Experiences at Pre-Exascale.mp4.mp4)

Presentation video

Download
137.50 MB

References

[1]

Andrea Arcangeli. 2020. Linux patch: Arm64: TLB: skip TLBI broadcast v2. https://lkml.org/lkml/2020/2/23/188.

[2]

ARM. 2021. Arm Architecture Reference Manual Armv8. https://developer.arm.com/documentation/ddi0487/latest/

[3]

Michael Bauer, Sean Treichler, Elliott Slaughter, and Alex Aiken. 2012. Legion: Expressing Locality and Independence with Logical Regions. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (Salt Lake City, Utah) (SC '12). IEEE Computer Society Press, Washington, DC, USA, Article 66, 11 pages.

Digital Library

[4]

BDEC Committee. 2017. The BDEC "Pathways to Convergence" Report. http://www.exascale.org/bdec/.

[5]

Pete Beckman, Marc Snir, Pavan Balaji, Franck Cappello, Rinku Gupta, Kamil Iskra, Swann Perarnau, Rajeev Thakur, and Kazutomo Yoshii. 2017. Argo: An Exascale Operating System. http://www.mcs.anl.gov/project/argo-exascale-operating-system.

[6]

Ron Brightwell, Ron Oldfield, Arthur B. Maccabe, and David E. Bernholdt. 2013. Hobbes: Composition and Virtualization As the Foundations of an Extreme-scale OS/R. In Proceedings of the 3rd International Workshop on Runtime and Operating Systems for Supercomputers (Eugene, Oregon) (ROSS).

[7]

Ron Brightwell, Kevin Pedretti, and Trammell Hudson. 2008. SMARTMAP: Operating System Support for Efficient Data Sharing Among Processes on a Multi-core Processor. In Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (Austin, Texas) (SC '08). IEEE Press, Piscataway, NJ, USA, Article 25, 12 pages.

[8]

R. Brightwell, R. Riesen, K. Underwood, T. B. Hudson, P. Bridges, and A. B. Maccabe. 2003. A performance comparison of Linux and a lightweight kernel. In 2003 Proceedings IEEE International Conference on Cluster Computing. 251--258.

[9]

Tsafrir Dan, Etsion Yoav, G. Feitelson Dror, and Kirkpatrick Scott. 2005. System noise, OS clock ticks, and fine-grained parallel applications. In 19th annual international conference on Supercomputing.

[10]

Kurt B. Ferreira, Patrick Bridges, and Ron Brightwell. 2008. Characterizing application sensitivity to OS interference using kernel-level noise injection. In SC '08: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing. 1--12.

[11]

Fujitsu Limited. 2019. White paper Advanced Software for the FUJITSU Supercomputer PRIMEHPC FX1000. https://www.fujitsu.com/downloads/SUPER/primehpc-fx1000-soft-en.pdf

[12]

Fujitsu Limited. 2021. A64FX® Microarchitecture Manual. https://github.com/fujitsu/A64FX/blob/master/doc/A64FX_Microarchitecture_Manual_en_1.4.pdf

[13]

Balazs Gerofi, Yutaka Ishikawa, Rolf Riesen, and Robert W. Wisniewski (Eds.). 2019. Operating Systems for Supercomputers and High Performance Computing. Vol. 1. Springer.

[14]

Balazs Gerofi, Rolf Riesen, Masamichi Takagi, Taisuke Boku, Yutaka Ishikawa, and Robert W. Wisniewski. 2018. Performance and Scalability of Lightweight Multi-Kernel based Operating Systems. In 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[15]

Balazs Gerofi, Rolf Riesen, Robert W. Wisniewski, and Yutaka Ishikawa. 2017. Toward Full Specialization of the HPC Software Stack: Reconciling Application Containers and Lightweight Multi-Kernels. In Proceedings of the 7th International Workshop on Runtime and Operating Systems for Supercomputers ROSS 2017 (Washingon, DC, USA) (ROSS '17). Association for Computing Machinery, New York, NY, USA, Article 7, 8 pages.

Digital Library

[16]

Balazs Gerofi, Aram Santogidis, Dominique Martinet, and Yutaka Ishikawa. 2018. "PicoDriver: Fast-Path Device Drivers for Multi-Kernel Operating Systems". In Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing (Tempe, Arizona) (HPDC '18). Association for Computing Machinery, New York, NY, USA, 2--13.

Digital Library

[17]

Balazs Gerofi, Akio Shimada, Atsushi Hori, and Yutaka Ishikawa. 2013. Partially Separated Page Tables for Efficient Operating System Assisted Hierarchical Memory Management on Heterogeneous Architectures. In 13th Intl. Symposium on Cluster, Cloud and Grid Computing (CCGrid).

[18]

Balazs Gerofi, Masamichi Takagi, Atsushi Hori, Guo Nakamura, Tomoki Shirasawa, and Yutaka Ishikawa. 2016. On the Scalability, Performance Isolation and Device Driver Transparency of the IHK/McKernel Hybrid Lightweight Kernel. In 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[19]

Balazs Gerofi, Masamichi Takagi, Yutaka Ishikawa, Rolf Riesen, Evan Powers, and Robert W. Wisniewski. 2015. Exploring the Design Space of Combining Linux with Lightweight Kernels for Extreme Scale Computing. In Proceedings of ROSS'15 (Portland, OR, USA). ACM, Article 5.

Digital Library

[20]

Mark Giampapa, Thomas Gooding, Todd Inglett, and Robert W. Wisniewski. 2010. Experiences with a Lightweight Supercomputer Kernel: Lessons Learned from Blue Gene's CNK. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[21]

V. E. Henson and U. M. Yang. 2002. BoomerAMG: A Parallel Algebraic Multigrid Solver and Preconditioner. https://codesign.llnl.gov/amg2013.php. Appl. Num. Math. 41 (2002), 155--177.

Digital Library

[22]

Torsten Hoefler, Timo Schneider, and Andrew Lumsdaine. 2010. Characterizing the Influence of System Noise on Large-Scale Applications by Simulation. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC '10). IEEE Computer Society, USA, 1--11.

Digital Library

[23]

T. Ichimura, K. Fujita, T. Yamaguchi, A. Naruse, J. C. Wells, T. C. Schulthess, T. P. Straatsma, C. J. Zimmer, M. Martinasso, K. Nakajima, M. Hori, and L. Maddegedara. 2018. A Fast Scalable Implicit Solver for Nonlinear Time-Evolution Earthquake City Problem on Low-Ordered Unstructured Finite Elements with Artificial Intelligence and Transprecision Computing. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. 627--637.

Digital Library

[24]

Intel Corporation. 2013. Intel Xeon Phi^™ Core Micro-architecture. https://software.intel.com/content/www/us/en/develop/articles/intel-xeon-phi-core-micro-architecture.html

[25]

Ken-Ichi Ishikawa, Yoshinobu Kuramashi, Akira Ukawa, and Taisuke Boku. 2017. CCS QCD Application. https://github.com/fiber-miniapp/ccs-qcd.

[26]

Joint Center for Advanced HPC (JCAHPC). 2017. Basic Specification of Oakforest-PACS. http://jcahpc.jp/files/OFP-basic.pdf.

[27]

Ian Karlin, Jeff Keasler, and Rob Neely. 2013. LULESH 2.0 Updates and Changes. Technical Report LLNL-TR-641973. Lawrence Livermore National Laboratory.

[28]

Takeharu Kato and Kouichi Hirai. 2019. K Computer. Springer Singapore, Singapore, 183--197.

[29]

Suzanne M. Kelly and Ron Brightwell. 2005. Software architecture of the light weight kernel, Catamount. In Cray User Group. 16--19.

[30]

Adam Lackorzynski, Carsten Weinhold, and Hermann Härtig. 2016. Decoupled: Low-Effort Noise-Free Execution on Commodity Systems. In Proceedings of the 6th International Workshop on Runtime and Operating Systems for Supercomputers (Kyoto, Japan) (ROSS '16). ACM, New York, NY, USA, Article 2, 8 pages.

Digital Library

[31]

J. Lange, K. Pedretti, T. Hudson, P. Dinda, Zheng Cui, Lei Xia, P. Bridges, A. Gocke, S. Jaconette, M. Levenhagen, and R. Brightwell. 2010. Palacios and Kitten: New high performance operating systems for scalable virtualized and native supercomputing. In IEEE International Symposium on Parallel Distributed Processing (IPDPS).

[32]

Lawrence Livermore National Laboratory. [n.d.]. The FTQ/FWQ Benchmark. https://asc.llnl.gov/sequoia/benchmarks/FTQ_summary_v1.1.pdf

[33]

J. Moreira, M. Brutman, J. Castano, T. Engelsiepen, M. Giampapa, T. Gooding, R. Haskin, T. Inglett, D. Lieber, P. McCarthy, M. Mundy, J. Parker, and B. Wallenfelt. 2006. Designing a Highly-Scalable Operating System: The Blue Gene/L Story. In SC 2006 Conference, Proceedings of the ACM/IEEE. 53--53.

[34]

Kengo Nakajima. 2003. Parallel Iterative Solvers of GeoFEM with Selective Blocking Preconditioning for Nonlinear Contact Problems on the Earth Simulator. In Proceedings of the 2003 ACM/IEEE Conference on Supercomputing (Phoenix, AZ, USA) (SC). ACM, New York, NY, USA.

Digital Library

[35]

NERSC. 2017. MILC. http://www.nersc.gov/research-and-development/apex/apex-benchmarks/milc/.

[36]

Sarp Oral, Feiyi Wang, David A. Dillow, Ross Miller, Galen M. Shipman, Don Maxwell, Dave Henseler, Jeff Becklehimer, and Jeff Larkin. 2010. Reducing Application Runtime Variability on Jaguar XT5. In Proceedings of CUG'10.

[37]

Jiannan Ouyang, Brian Kocoloski, John R. Lange, and Kevin Pedretti. 2015. Achieving Performance Isolation with Lightweight Co-Kernels. In Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing (Portland, Oregon, USA) (HPDC '15). ACM, New York, NY, USA, 149--160.

Digital Library

[38]

Yoonho Park, E. Van Hensbergen, M. Hillenbrand, T. Inglett, B. Rosenburg, Kyung Dong Ryu, and R.W. Wisniewski. 2012. FusedOS: Fusing LWK Performance with FWK Functionality in a Heterogeneous Environment. In Computer Architecture and High Performance Computing (SBAC-PAD), 2012 IEEE 24th International Symposium on. 211--218.

Digital Library

[39]

Kevin T. Pedretti, Michael Levenhagen, Kurt Ferreira, Ron Brightwell, Suzanne Kelly, Patrick Bridges, and Trammell Hudson. 2010. LDRD Final Report: A Lightweight Operating System for Multi-core Capability Class Supercomputers. Technical report SAND2010-6232. Sandia National Laboratories.

[40]

Fabrizio Petrini, Darren J. Kerbyson, and Scott Pakin. 2003. The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q. In SC '03: Proceedings of the 2003 ACM/IEEE conference on Supercomputing. 55.

Digital Library

[41]

Howard Pritchard, Duncan Roweth, David Henseler, and Paul Cassella. 2012. Leveraging the Cray Linux Environment Core Specialization Feature to Realize MPI Asynchronous Progress on Cray XE Systems. In Proceedings of Cray User Group (CUG).

[42]

Rolf Riesen, Ron Brightwell, Patrick G. Bridges, Trammell Hudson, Arthur B. Maccabe, Patrick M. Widener, and Kurt Ferreira. 2009. Designing and Implementing Lightweight Kernels for Capability Computing. Concurrency and Computation: Practice and Experience 21, 6 (April 2009).

Digital Library

[43]

Rolf Riesen, Arthur Barney Maccabe, Balazs Gerofi, David N. Lombard, John Jack Lange, Kevin Pedretti, Kurt Ferreira, Mike Lang, Pardo Keppel, Robert W. Wisniewski, Ron Brightwell, Todd Inglett, Yoonho Park, and Yutaka Ishikawa. 2015. What is a Lightweight Kernel?. In Proceedings of the 5th International Workshop on Runtime and Operating Systems for Supercomputers (Portland, OR, USA) (ROSS). ACM, New York, NY, USA.

Digital Library

[44]

RIKEN Center for Computational Science. 2021. Fugaku. https://www.r-ccs.riken.jp/en/fugaku/.

[45]

Subhash Saini and Horst D. Simon. 1994. Applications Performance Under OSF/1 AD and SUNMOS on Intel Paragon XP/S-15. In Proceedings of the 1994 ACM/IEEE Conference on Supercomputing (Washington, D.C.) (Supercomputing '94). IEEE Computer Society Press, Los Alamitos, CA, USA, 580--589.

[46]

M. Sato, Y. Ishikawa, H. Tomita, Y. Kodama, T. Odajima, M. Tsuji, H. Yashiro, M. Aoki, N. Shida, I. Miyoshi, K. Hirai, A. Furuya, A. Asato, K. Morita, and T. Shimizu. 2020. "Co-Design for A64FX Manycore Processor and "Fugaku"". In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. 1--15.

[47]

Taku Shimosawa, Balazs Gerofi, Masamichi Takagi, Gou Nakamura, Tomoki Shirasawa, Yuji Saeki, Masaaki Shimizu, Atsushi Hori, and Yutaka Ishikawa. 2014. Interface for Heterogeneous Kernels: A Framework to Enable Hybrid OS Designs targeting High Performance Computing on Manycore Architectures. In 21th Intl. Conference on High Performance Computing (HiPC).

[48]

Leslie G. Valiant. 1990. A Bridging Model for Parallel Computation. Commun. ACM 33, 8 (Aug. 1990), 103--111.

Digital Library

[49]

S. R. Wheat, A. B. Maccabe, R. Riesen, D. W. van Dresser, and T. M. Stallcup. 1994. PUMA: an operating system for massively parallel systems. In Proceedings of System Sciences'94, Vol. 2. 56--65.

[50]

Robert W. Wisniewski, Todd Inglett, Pardo Keppel, Ravi Murty, and Rolf Riesen. 2014. mOS: An Architecture for Extreme-scale Operating Systems. In Proceedings of the 4th International Workshop on Runtime and Operating Systems for Supercomputers (Munich, Germany) (ROSS). ACM, New York, NY, USA, Article 2.

Digital Library

[51]

Kazutomo Yoshii, Kamil Iskra, Harish Naik, Pete Beckmanm, and P. Chris Broekema. 2009. Characterizing the Performance of Big Memory on Blue Gene Linux. In Proceedings of the 2009 Intl. Conference on Parallel Processing Workshops (ICPPW). IEEE Computer Society, 65--72.

[52]

J. A. Zounmevo, S. Perarnau, K. Iskra, K. Yoshii, R. Gioiosa, B. C. V. Essen, M. B. Gokhale, and E. A. Leon. 2015. A Container-Based Approach to OS Specialization for Exascale Computing. In 2015 IEEE International Conference on Cloud Engineering. 359--364.

Cited By

Cui MPapadopoulou NPericàs M(2023)Analysis and Characterization of Performance Variability for OpenMP RuntimeProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624239(1614-1622)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624239
An ZLi DGuo YGao GRen YJia NHu X(2023)Towards OS Heterogeneity Aware Cluster Management for HPCProceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems10.1145/3609510.3609819(16-23)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.1145/3609510.3609819

Index Terms

Linux vs. lightweight multi-kernels for high performance computing: experiences at pre-exascale
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems

Recommendations

PicoDriver: fast-path device drivers for multi-kernel operating systems
HPDC '18: Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing

Lightweight kernel (LWK) operating systems (OS) in high-end supercomputing have a proven track record of excellent scalability. However, the lack of full Linux compatibility and limited availability of device drivers in LWKs have prohibited their wide-...
Exploring the Design Space of Combining Linux with Lightweight Kernels for Extreme Scale Computing
ROSS '15: Proceedings of the 5th International Workshop on Runtime and Operating Systems for Supercomputers

As systems sizes increase to exascale and beyond, there is a need to enhance the system software to meet the needs and challenges of applications. The evolutionary versus revolutionary debate can be set aside by providing system software that ...
A lightweight virtual machine monitor for Blue Gene/P

In this paper, we present a lightweight, micro-kernel-based virtual machine monitor (VMM) for the Blue Gene/P supercomputer. Our VMM comprises a small µ-kernel with virtualization capabilities and, atop, a user-level VMM component that manages virtual ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '21: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2021

1493 pages

ISBN:9781450384421

DOI:10.1145/3458817

General Chair:
Bronis R. de Supinski,
Program Chairs:
Mary Hall,
Todd Gamblin

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SC '21

Sponsor:

SIGHPC

SC '21: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 14 - 19, 2021

Missouri, St. Louis

Acceptance Rates

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
626
Total Downloads

Downloads (Last 12 months)73
Downloads (Last 6 weeks)4

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cui MPapadopoulou NPericàs M(2023)Analysis and Characterization of Performance Variability for OpenMP RuntimeProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624239(1614-1622)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624239
An ZLi DGuo YGao GRen YJia NHu X(2023)Towards OS Heterogeneity Aware Cluster Management for HPCProceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems10.1145/3609510.3609819(16-23)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.1145/3609510.3609819

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents