short-paper

Public Access

High-Performance Storage Support for Scientific Applications on the Cloud

Authors:

Gabriele Garzoglio,

Ioan RaicuAuthors Info & Claims

ScienceCloud '15: Proceedings of the 6th Workshop on Scientific Cloud Computing

Pages 33 - 36

https://doi.org/10.1145/2755644.2755648

Published: 16 June 2015 Publication History

Abstract

Although cloud computing has become one of the most popular paradigms for executing data-intensive applications (for example, Hadoop), the storage subsystem is not optimized for scientific applications. We believe that when executing scientific applications in the cloud, a node-local distributed storage architecture is a key approach to overcome the challenges from the conventional shared/parallel storage systems. We analyze and evaluate four representative file systems (S3FS, HDFS, Ceph, and FusionFS) on three platforms (Kodiak cluster, Amazon EC2 and FermiCloud) with a variety of benchmarks to explore how well these storage systems can handle metadata intensive, write intensive, and read intensive workloads.

References

[1]

K. Shvachko, et al. The hadoop distributed file system. In MSST, 2010.

Digital Library

[2]

S. Ghemawat, et al. The Google file system. In SOSP, 2003.

Digital Library

[3]

P. Carns, et al. Small-file access in parallel file systems. In IPDPS, 2009.

Digital Library

[4]

S3FS. https://code.google.com/p/s3fs/.

[5]

FUSE. http://fuse.sourceforge.net.

[6]

S. Weil, et al. Ceph: A scalable, high-performance distributed file system. In OSDI, 2006.

Digital Library

[7]

D. Zhao, et al. FusionFS: Toward supporting data-intensive scientific applications on extreme-scale distributed systems. In Big Data Conference, 2014.

[8]

S. Weil, et al. Crush: Controlled, scalable, decentralized placement of replicated data. In SC, 2006.

Digital Library

[9]

D. Zhao, et al. Distributed file systems for exascale computing. In SC, 2012.

[10]

D. Zhao, et al. Towards high-performance and cost-effective distributed storage systems with information dispersal algorithms. In CLUSTER, 2013.

[11]

D. Zhao, et al. Distributed data provenance for large-scale data-intensive computing. In CLUSTER, 2013.

[12]

D. Zhao, et al. Hycache+: Towards scalable high-performance caching middleware for parallel file systems. In CCGrid, 2014.

Digital Library

[13]

D. Zhao, et al. HyCache: A user-level caching middleware for distributed file systems. In IPDPSW, 2013.

Digital Library

[14]

D. Zhao, et al. Virtual chunks: On supporting random accesses to scientific data in compressible storage systems. In Big Data Conference, 2014.

[15]

D. Zhao, et al. Improving the i/o throughput for data-intensive scientific applications with efficient compression mechanisms. In SC, 2013.

[16]

Kodiak. https://www.nmc-probe.org/wiki/machines:kodiak.

[17]

Amazon EC2. http://aws.amazon.com/ec2.

[18]

FermiCloud. http://fclweb.fnal.gov/.

[19]

B. Welch, et al. Optimizing a hybrid SSD/HDD HPC storage system based on file size distributions. In MSST, 2013.

[20]

D. Nagle, et al. The Panasas activescale storage cluster: Delivering scalable high bandwidth storage. In SC, 2004.

Digital Library

[21]

D. Zhao, et al. Exploring reliability of exascale systems through simulations. In HPC, 2013.

Digital Library

[22]

F. Schmuck, et al. GPFS: A shared-disk file system for large computing clusters. In FAST, 2002.

Digital Library

[23]

P. Schwan. Lustre: Building a file system for 1,000-node clusters. In Linux symposium, 2003.

[24]

H. Wu, et al. A reference model for virtual machine launching overhead. IEEE Transactions on Cloud Computing, 2014.

[25]

H. Wu, et al. Modeling the virtual machine launching overhead under fermicloud. In CCGrid, 2014.

Digital Library

[26]

P. Carns, et al. PVFS: A parallel file system for linux clusters. In Annual Linux Showcase and Conference, 2000.

Digital Library

Cited By

Carneiro ABez JOsthoff CSchnorr LNavaux P(2023)Uncovering I/O demands on HPC platformsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2023.104744182:COnline publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1016/j.jpdc.2023.104744
Wang LZhao D(2022)Practical Federated Learning Infrastructure for Privacy-Preserving Scientific Computing2022 IEEE/ACM International Workshop on Artificial Intelligence and Machine Learning for Scientific Applications (AI4S)10.1109/AI4S56813.2022.00012(38-43)Online publication date: Nov-2022
https://doi.org/10.1109/AI4S56813.2022.00012
Jiang QLee YZomaya A(2020)The Limit of Horizontal Scaling in Public CloudsACM Transactions on Modeling and Performance Evaluation of Computing Systems10.1145/33733565:1(1-22)Online publication date: 4-Feb-2020
https://dl.acm.org/doi/10.1145/3373356
Show More Cited By

Index Terms

High-Performance Storage Support for Scientific Applications on the Cloud
1. Information systems
  1. Information retrieval
    1. Search engine architectures and scalability
      1. Distributed retrieval
      2. Peer-to-peer retrieval
  2. Information storage systems
    1. Storage architectures
      1. Distributed storage

Recommendations

Comparing FutureGrid, Amazon EC2, and Open Science Grid for Scientific Workflows

Scientists have many computing infrastructures available to conduct their research, including grids and public or private clouds. This article explores the use of these cyberinfrastructures to execute scientific workflows, an important class of ...
High-Performance Cloud Computing: A View of Scientific Applications
ISPAN '09: Proceedings of the 2009 10th International Symposium on Pervasive Systems, Algorithms, and Networks

Scientific computing often requires the availability of a massive number of computers for performing large scale experiments. Traditionally, these needs have been addressed by using high-performance computing solutions and installed facilities such as ...
Cloud-Aware Development of Scientific Applications
WETICE '14: Proceedings of the 2014 IEEE 23rd International WETICE Conference

The potential of cloud computing is still underutilized in the scientific computing field. Even if clouds probably are not fit for high-end HPC applications, they could be profitably used to bring the power of economic and scalable parallel computing to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ScienceCloud '15: Proceedings of the 6th Workshop on Scientific Cloud Computing

June 2015

46 pages

ISBN:9781450335706

DOI:10.1145/2755644

General Chairs:
Kyle Chard
University of Chicago & Argonne National Lab, USA
,
Alexandru Costan
Inria/IRISA, France
,
Bogdan Nicolae
IBM Research, Ireland
,
Lavanya Ramakrishnan
Lawrence Berkeley National Lab, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Arizona: University of Arizona
SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

KISTI
NSF CAREER
DOE

Conference

HPDC'15

Sponsor:

University of Arizona
SIGARCH

HPDC'15: The 24th International Symposium on High-Performance Parallel and Distributed Computing

June 16, 2015

Oregon, Portland, USA

Acceptance Rates

ScienceCloud '15 Paper Acceptance Rate 3 of 6 submissions, 50%;

Overall Acceptance Rate 44 of 151 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
324
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)11

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Carneiro ABez JOsthoff CSchnorr LNavaux P(2023)Uncovering I/O demands on HPC platformsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2023.104744182:COnline publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1016/j.jpdc.2023.104744
Wang LZhao D(2022)Practical Federated Learning Infrastructure for Privacy-Preserving Scientific Computing2022 IEEE/ACM International Workshop on Artificial Intelligence and Machine Learning for Scientific Applications (AI4S)10.1109/AI4S56813.2022.00012(38-43)Online publication date: Nov-2022
https://doi.org/10.1109/AI4S56813.2022.00012
Jiang QLee YZomaya A(2020)The Limit of Horizontal Scaling in Public CloudsACM Transactions on Modeling and Performance Evaluation of Computing Systems10.1145/33733565:1(1-22)Online publication date: 4-Feb-2020
https://dl.acm.org/doi/10.1145/3373356
Vef MSteiner RSalkhordeh RSteinkamp JVennetier FSmigielski JBrinkmann A(2020)DelveFS - An Event-Driven Semantic File System for Object Stores2020 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER49012.2020.00014(35-46)Online publication date: Sep-2020
https://doi.org/10.1109/CLUSTER49012.2020.00014
Abraham SPaul AKhan RButt A(2020)On the Use of Containers in High Performance Computing Environments2020 IEEE 13th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD49709.2020.00048(284-293)Online publication date: Oct-2020
https://doi.org/10.1109/CLOUD49709.2020.00048
Wang XAl-Mamun AYan FZhao D(2019)Toward Accurate and Efficient Emulation of Public Blockchains in the CloudCloud Computing – CLOUD 201910.1007/978-3-030-23502-4_6(67-82)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1007/978-3-030-23502-4_6
Montes DAñel JPena TUhe PWallom D(2017)Enabling BOINC in infrastructure as a service cloud systemGeoscientific Model Development10.5194/gmd-10-811-201710:2(811-826)Online publication date: 21-Feb-2017
https://doi.org/10.5194/gmd-10-811-2017
Xu RZhao KZhang PYuan DXie YYang Y(2017)A Novel Data Set Importance Based Cost-Effective and Computation-Efficient Storage Strategy in the Cloud2017 IEEE International Conference on Web Services (ICWS)10.1109/ICWS.2017.24(122-129)Online publication date: Jun-2017
https://doi.org/10.1109/ICWS.2017.24
Pulamolu KVenkatasubramanian DBhavani T(2017)An efficient resource optimization in intra-tenant heterogeneous hadoop cluster2017 International Conference on Intelligent Computing and Control Systems (ICICCS)10.1109/ICCONS.2017.8250736(324-329)Online publication date: Jun-2017
https://doi.org/10.1109/ICCONS.2017.8250736
Zhao KYuan DXie YYan LXu R(2017)An optimized data storage strategy by computational performance and monetary cost with data importance in the cloud2017 IEEE 21st International Conference on Computer Supported Cooperative Work in Design (CSCWD)10.1109/CSCWD.2017.8066733(433-438)Online publication date: Apr-2017
https://doi.org/10.1109/CSCWD.2017.8066733
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents