research-article

Open access

Container Adoption in Campus High Performance Computing at Texas A&M University

Authors:

Richard Lawrence,

Dhruva K. Chakravorty,

Wesley Brashear,

Joshua Winchell,

Honggao LiuAuthors Info & Claims

PEARC '24: Practice and Experience in Advanced Research Computing 2024: Human Powered Computing

Article No.: 6, Pages 1 - 7

https://doi.org/10.1145/3626203.3670550

Published: 17 July 2024 Publication History

All formats PDF

Abstract

Containers promise benefits for both researchers and cyberinfrastructure (CI) professionals, by offering pre-assembled software stacks with a lower barrier to entry. However, the benefits of containers are best realized in a well-developed CI ecosystem with support from administrators. In this paper, we report on the emerging container ecosystem supported by Texas A&M University’s High Performance Research Computing Group (HPRC)1; lessons learned from the perspective of a team that manages a diverse CI portfolio. Here we share our experience with the following: building a container ecosystem for HPRC computing resources, outreach to researcher communities, and documenting use cases that inspire uptake by others. HPRC container deployment has facilitated access to and support for a wide variety of research software at Texas A&M. Our progress is described from multiple perspectives that reflect the varied ways that our researchers and administrators utilize containers.

References

[1]

Carlos Arango Gutierrez, Remy Dernat, and John Sanabria. 2017. Performance Evaluation of Container-based Virtualization for High Performance Computing Environments. Revista UIS Ingenierías 18 (09 2017). https://doi.org/10.18273/revuin.v18n4-2019003

[2]

J. Armstrong, G. Hickey, M Diekhans, and et al.2020. Progressive Cactus is a multiple-genome aligner for the thousand-gen. Nature (2020). https://doi.org/10.1038/s41586-020-2871-y

[3]

Luciano Baresi, Giovanni Quattrocchi, and Nicholas Rasi. 2024. A qualitative and quantitative analysis of container engines. Journal of Systems and Software 210 (2024), 111965. https://doi.org/10.1016/j.jss.2024.111965

Digital Library

[4]

Erik Ferlanti, William J. Allen, Ernesto A. B. F. Lima, Yinzhi Wang, and John M. Fonner. 2023. Perspectives and Experiences Supporting Containers for Research Computing at the Texas Advanced Computing Center. In Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis (Denver, CO) (SC-W ’23). Association for Computing Machinery, New York, NY, USA, 155–164. https://doi.org/10.1145/3624062.3624587

Digital Library

[5]

Olivier Flauzac, Fabien Mauhourat, and Florent Nolot. 2020. A review of native container security for running applications. Procedia Computer Science 175 (2020), 157–164. https://doi.org/10.1016/j.procs.2020.07.025 The 17th International Conference on Mobile Systems and Pervasive Computing (MobiSPC),The 15th International Conference on Future Networks and Communications (FNC),The 10th International Conference on Sustainable Energy Information Technology.

[6]

David Y. Hancock, Jeremy Fischer, John Michael Lowe, Winona Snapp-Childs, Marlon Pierce, Suresh Marru, J. Eric Coulter, Matthew Vaughn, Brian Beck, Nirav Merchant, Edwin Skidmore, and Gwen Jacobs. 2021. Jetstream2: Accelerating cloud computing via Jetstream. In Practice and Experience in Advanced Research Computing (Boston, MA, USA) (PEARC ’21). Association for Computing Machinery, New York, NY, USA, Article 11, 8 pages. https://doi.org/10.1145/3437359.3465565

Digital Library

[7]

Zhenhua He, Sandra Nite, Joshua Winchell, Abhinand Nasari, Hieu Le, Jiao Tao, Dhruva Chakravorty, Lisa Perez, and Honggao Liu. 2023. Development of a Training Framework for Novel Accelerators. In 2023 IEEE Frontiers in Education Conference (FIE). 1–6. https://doi.org/10.1109/FIE58773.2023.10343498

[8]

Zhenhua He, Aditi Saluja, Richard Lawrence, Dhruva K. Chakravorty, Francis Dang, Lisa M. Perez, and Honggao Liu. 2023. Performance of Distributed Deep Learning Workloads on a Composable Cyberinfrastructure. In Practice and Experience in Advanced Research Computing (Portland, OR, USA) (PEARC ’23). Association for Computing Machinery, New York, NY, USA, 12 pages. https://doi.org/10.1145/3569951.3603632

Digital Library

[9]

Benjamin Holmes. 2019. Application lifecycle management for container-native development. https://developers.redhat.com/blog/2019/06/11/application-lifecycle-management-for-container-native-development. Accessed: 2024-05-16.

[10]

HPRC. 2024. Knowledge Base. https://hprc.tamu.edu/kb/

[11]

HPRC. 2024. Youtube Channel. https://www.youtube.com/channel/UCgeDEHE5GwkxYUGS0FDLmPw

[12]

Dave Hudak, Doug Johnson, Alan Chalker, Jeremy Nicklas, Eric Franz, Trey Dockendorf, and Brian L. McMichael. 2018. Open OnDemand: A web-based client portal for HPC centers. Journal of Open Source Software 3, 25 (2018), 622. https://doi.org/10.21105/joss.00622

[13]

R. Keller Tesser and E. Borin. 2022. Containers in HPC: a survey. J Supercomput (2022). https://doi.org/10.1007/s11227-022-04848-y

Digital Library

[14]

Gregory M. Kurtzer, Vanessa Sochat, and Michael W. Bauer. 2017. Singularity: Scientific containers for mobility of compute. PLOS ONE 12, 5 (05 2017), 1–20. https://doi.org/10.1371/journal.pone.0177459

[15]

Richard Lawrence, Tri M. Pham, Phi T. Au, Xin Yang, Kyle Hsu, Stuti H. Trivedi, Lisa M. Perez, and Dhruva K. Chakravorty. 2022. Expanding Interactive Computing to Facilitate Informal Instruction in Research Computing. The Journal of Computational Science Education 13 (April 2022), 50–54. Issue 1. https://doi.org/10.22369/issn.2153-4136/13/1/9

[16]

D. Moreau, K Wiebels, and C. Boettiger. 2023. Containers for computational reproducibility. Nat Rev Methods Primers (2023). https://doi.org/10.1038/s43586-023-00236-9

[17]

"National Academies of Sciences, Engineering, and Medicine". 2019. Reproducibility and Replicability in Science. The National Academies Press, Washington, DC. https://doi.org/10.17226/25303

[18]

Reid Priedhorsky, R. Shane Canon, Timothy Randles, and Andrew J. Younge. 2021. Minimizing privilege for building HPC containers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Missouri) (SC ’21). Association for Computing Machinery, New York, NY, USA, Article 32, 14 pages. https://doi.org/10.1145/3458817.3476187

Digital Library

[19]

A. Punjani, J. Rubinstein, D. Fleet, and et al.2017. cryoSPARC: algorithms for rapid unsupervised cryo-EM structure determination. Nat Methods (2017). https://doi.org/10.1038/nmeth.4169

[20]

Andrew Solis, William J. Allen, and Erik Ferlanti. 2022. Containerizing Visualization Software: Experiences and Best Practices. In Practice and Experience in Advanced Research Computing (Boston, MA, USA) (PEARC ’22). Association for Computing Machinery, New York, NY, USA, Article 22, 8 pages. https://doi.org/10.1145/3491418.3530769

Digital Library

[21]

X. Carol Song, Preston Smith, Rajesh Kalyanam, Xiao Zhu, Eric Adams, Kevin Colby, Patrick Finnegan, Erik Gough, Elizabett Hillery, Rick Irvine, Amiya Maji, and Jason St. John. 2022. Anvil - System Architecture and Experiences from Deployment and Early User Operations. In Practice and Experience in Advanced Research Computing (Boston, MA, USA) (PEARC ’22). Association for Computing Machinery, New York, NY, USA, Article 23, 9 pages. https://doi.org/10.1145/3491418.3530766

Digital Library

[22]

Yucheng Zhang and Lev Gorenstein. 2022. BioContainers on Purdue Clusters. In Practice and Experience in Advanced Research Computing (Boston, MA, USA) (PEARC ’22). Association for Computing Machinery, New York, NY, USA, Article 92, 2 pages. https://doi.org/10.1145/3491418.3535152

Digital Library

[23]

Yucheng Zhang, Lev Gorenstein, Payas Bhutra, and Ryan DeRue. 2022. Containerized Bioinformatics Ecosystem for HPC. 1–10. https://doi.org/10.1109/HUST56722.2022.00006

[24]

Gregory J. Zynda, Shweta Gopaulakrishnan, and John Fonner. 2021. RollingGantryCrane: Automation for unpacking containers into HPC environments. In 2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). 29–34. https://doi.org/10.1109/CANOPIEHPC54579.2021.00008

Cited By

Mishra SChakravorty DPerez LDang FLiu HWitherden F(2024)Impact of Memory Bandwidth on the Performance of AcceleratorsPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670540(1-9)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670540

Index Terms

Recommendations

Distributed Computing Education, Part 4: Training Infrastructure

Teaching distributed computing requires environments that provide adequate training infrastructure, or t-infrastructure. In practice, t-infrastructure includes the computing equipment, digital communications, software, data, and support staff necessary ...
Characterizing Containerized HPC Applications Performance at Petascale on CPU and GPU Architectures
High Performance Computing
Abstract
Containerization technologies provide a mechanism to encapsulate applications and many of their dependencies, facilitating software portability and reproducibility on HPC systems. However, in order to access many of the architectural features that ...
Perspectives and Experiences Supporting Containers for Research Computing at the Texas Advanced Computing Center
SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

Containers are becoming essential to support the diversity of scientific computing workloads at academic computing centers. Here, we offer perspectives and experiences from the Texas Advanced Computing Center on: the installation, configuration, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PEARC '24: Practice and Experience in Advanced Research Computing 2024: Human Powered Computing

July 2024

608 pages

ISBN:9798400704192

DOI:10.1145/3626203

Editors:
Shawn T. Brown
Hewlett-Packard Enterprise
,
Barr von Oehsen
Pittsburgh Supercomputing Center
,
Eric Adams
Purdue University
,
Eva Siegmann
Stony Brook University

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 July 2024

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

NSF (National Science Foundation)

Conference

PEARC '24

Sponsor:

PEARC '24: Practice and Experience in Advanced Research Computing

July 21 - 25, 2024

RI, Providence, USA

Acceptance Rates

Overall Acceptance Rate 133 of 202 submissions, 66%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
186
Total Downloads

Downloads (Last 12 months)186
Downloads (Last 6 weeks)44

Reflects downloads up to 27 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mishra SChakravorty DPerez LDang FLiu HWitherden F(2024)Impact of Memory Bandwidth on the Performance of AcceleratorsPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670540(1-9)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670540

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents