research-article

MasQ: RDMA for Virtual Private Cloud

Authors:

Kai ZhengAuthors Info & Claims

SIGCOMM '20: Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication

Pages 1 - 14

https://doi.org/10.1145/3387514.3405849

Published: 30 July 2020 Publication History

Abstract

RDMA communication in virtual private cloud (VPC) networks is still a challenging job due to the difficulty in fulfilling all virtualization requirements without sacrificing RDMA communication performance. To address this problem, this paper proposes a software-defined solution, namely, MasQ, which is short for "queue masquerade". The core insight of MasQ is that all RDMA communications should associate with at least one queue pair (QP). Thus, the requirements of virtualization, such as network isolation and the application of security rules, can be easily fulfilled if QP's behavior is properly defined. In particular, MasQ exploits the virtio-based paravirtualization technique to realize the control path. Moreover, to avoid performance overhead, MasQ leaves all data path operations, such as sending and receiving, to the hardware. We have implemented MasQ in the OpenFabrics Enterprise Distribution (OFED) framework and proved its scalability and performance efficiency by evaluating it against typical applications. The results demonstrate that MasQ achieves almost the same performance as bare-metal RDMA for data communication.

Supplementary Material

MP4 File (3387514.3405849.mp4)

20-Minute presentation video for MasQ: RDMA for Virtual Private Cloud

Download
52.86 MB

References

[1]

2019. Docker. https://www.docker.com/. (2019).

[2]

2019. Ftrace. https://www.kernel.org/doc/Documentation/trace/ftrace.txt. (2019).

[3]

2019. Graph 500. https://graph500.org/. (2019).

[4]

2019. High-Performance Big Data. http://hibd.cse.ohio-state.edu/#spark. (2019).

[5]

2019. HowTo Configure QoS over SR-IOV. https://community.mellanox.com/s/article/howto-configure-qos-over-sr-iov. (2019).

[6]

2019. Mellanox ConnectX-6 Dx. https://www.mellanox.com/products/ethernet-adapter-ic/connectx-6-dx-ic. (2019).

[7]

2019. Mellanox VMA. https://github.com/Mellanox/libvma. (2019).

[8]

2019. Monitoring Spark applications. https://spark.apache.org/docs/latest/monito-ring.html. (2019).

[9]

2019. MVAPICH: MPI over InfiniBand, Omni-Path, Ethernet/iWARP, and RoCE. http://mvapich.cse.ohio-state.edu/. (2019).

[10]

2019. Open vSwitch. https://www.openvswitch.org/. (2019).

[11]

2019. OSU HiBD Benchmarks. http://hibd.cse.ohio-state.edu/#microbenchmarks. (2019).

[12]

2019. Perftest package. https://community.mellanox.com/docs/DOC-2802. (2019).

[13]

2019. RDMA-bench. https://github.com/efficient/rdma_bench. (2019).

[14]

2019. Runtime options with Memory, CPUs, and GPUs. https://docs.docker.com/config/containers/resource_constraints/. (2019).

[15]

2019. Weave Net. https://www.weave.works/. (2019).

[16]

Fabrice Bellard. 2005. QEMU, a Fast and Portable Dynamic Translator. In Proceedings of the Annual Conference on USENIX Annual Technical Conference (ATEC '05). USENIX Association, Berkeley, CA, USA, 41--41.

Digital Library

[17]

Youmin Chen, Youyou Lu, and Jiwu Shu. 2019. Scalable RDMA RPC on Reliable Connection with Efficient Resource Sharing. In Proceedings of the Fourteenth EuroSys Conference 2019 (EuroSys '19). Association for Computing Machinery, New York, NY, USA, Article 19, 14pages. https://doi.org/10.1145/3302424.3303968

Digital Library

[18]

Inho Cho, Keon Jang, and Dongsu Han. 2017. Credit-Scheduled Delay-Bounded Congestion Control for Datacenters. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). ACM, New York, NY, USA, 239--252. https://doi.org/10.1145/3098822.3098840

Digital Library

[19]

Andrew R Curtis, Jeffrey C Mogul, Jean Tourrilhes, Praveen Yalagandula, Puneet Sharma, and Sujata Banerjee. 2011. DevoFlow: Scaling flow management for high-performance networks. In Proceedings of the ACM SIGCOMM 2011 conference. 254--265.

Digital Library

[20]

Aleksandar Dragojević, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. 2014. FaRM: Fast Remote Memory. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). USENIX Association, Seattle, WA, 401--414.

Digital Library

[21]

Shiqing Fan, Fang Chen, Holm Rauchfuss, Nadav Har'El, Uwe Schilling, and Nico Struckmann. 2017. Towards a Lightweight RDMA Para-Virtualization for HPC. In Proceedings of the Joined Workshops COSH 2017 and VisorHPC 2017.

[22]

Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mike Andrewartha, Hari Angepat, Vivek Bhanu, Adrian Caulfield, Eric Chung, et al. 2018. Azure Accelerated Networking: SmartNICs in the Public Cloud. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), Renton, WA.

Digital Library

[23]

Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew W. Moore, Gianni Antichi, and Marcin Wójcik. 2017. Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). ACM, New York, NY, USA, 29--42. https://doi.org/10.1145/3098822.3098825

Digital Library

[24]

InfiniBand Trade Association 2010. InfiniBand Architecture Specification Release 1.2.1 Annex A16: RoCE. InfiniBand Trade Association. Rev.1.2.1.

[25]

InfiniBand Trade Association 2014. InfiniBand Architecture Specification Release 1.2.1. InfiniBand Trade Association. Rev.1.2.1.

[26]

InfiniBand Trade Association 2014. InfiniBand Architecture Specification Release 1.2.1 Annex A17: RoCEv2. InfiniBand Trade Association. Rev.1.2.1.

[27]

Internet Engineering Task Force 2007. A Remote Direct Memory Access Protocol Specification. Internet Engineering Task Force. RFC5040.

[28]

N. S. Islam, M. W. Rahman, J. Jose, R. Rajachandrasekar, H. Wang, H. Subramoni, C. Murthy, and D. K. Panda. 2012. High Performance RDMA-based Design of HDFS over InfiniBand. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC '12). IEEE Computer Society Press, Los Alamitos, CA, USA, Article 35, 35 pages.

[29]

Cheng Jin, Abhinav Srivastava, and Zhi-Li Zhang. 2016. Understanding security group usage in a public IaaS cloud. In IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications. 1--9.

Digital Library

[30]

Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2014. Using RDMA Efficiently for Key-value Services. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM '14). ACM, New York, NY, USA, 295--306. https://doi.org/10.1145/2619239.2626299

[31]

Anuj Kalia, Michael Kaminsky, and David G Andersen. 2016. Design Guidelines for High Performance RDMA Systems. In 2016 USENIX Annual Technical Conference (USENIX ATC 16). 437--450.

Digital Library

[32]

Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. FaSST: Fast, Scalable and Simple Distributed Transactions with Two-Sided (RDMA) Datagram RPCs. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 185--201.

Digital Library

[33]

Daehyeok Kim, Tianlong Yu, Hongqiang Harry Liu, Yibo Zhu, Jitu Padhye, Shachar Raindel, Chuanxiong Guo, Vyas Sekar, and Srinivasan Seshan. 2019. FreeFlow: Software-based Virtual RDMA Networking for Containerized Clouds. In NSDI. 113--126.

[34]

Teemu Koponen, Martin Casado, Natasha Gude, Jeremy Stribling, Leon Poutievski, Min Zhu, Rajiv Ramanathan, Yuichiro Iwata, Hiroaki Inoue, Takayuki Hama, et al. 2010. Onix: A distributed control platform for large-scale production networks.

[35]

Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. 2019. HPCC: high precision congestion control. In Proceedings of the ACM Special Interest Group on Data Communication. 44--58.

[36]

Fangfei Liu and Ruby B. Lee. 2014. Random Fill Cache Architecture. In Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-47). IEEE Computer Society, USA, 203--215. https://doi.org/10.1109/MICRO.2014.28

[37]

M. Mahalingam, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, and C. Wright. 2014. Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks. https://www.rfc-editor.org/info/rfc7348. (August 2014).

[38]

Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Ionnis Koltsidas, and Thomas R Gross. 2015. A hybrid I/O virtualization framework for RDMA-capable network interfaces. ACM SIGPLAN Notices 50, 7 (2015), 17--30.

Digital Library

[39]

Adit Ranadive and Bhavesh Davda. 2012. Toward a paravirtual vRDMA device for VMware ESXi guests. VMware Technical Journal, Winter 2012 1, 2 (2012).

[40]

Rusty Russell. 2008. virtio: towards a de-facto standard for virtual I/O devices. ACM SIGOPS Operating Syst. Review (OSR (2008), 103.

Digital Library

[41]

Shin-Yeh Tsai, Mathias Payer, and Yiying Zhang. 2019. Pythia: Remote Oracles for the Masses. In Proceedings of the 28th USENIX Conference on Security Symposium (SEC '19). USENIX Association, USA, 693--710.

[42]

Shin-Yeh Tsai and Yiying Zhang. 2017. Lite kernel rdma support for datacenter applications. In Proceedings of the 26th Symposium on Operating Systems Principles. ACM, 306--324.

Digital Library

[43]

Dongyang Wang, Binzhang Fu, Gang Lu, Kun Tan, and Bei Hua. 2019. VSocket: Virtual Socket Interface for RDMA in Public Clouds. In Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2019). Association for Computing Machinery, New York, NY, USA, 179 C192. https://doi.org/10.1145/3313808.3313813

Digital Library

[44]

Zhenghong Wang and Ruby B. Lee. 2007. New Cache Designs for Thwarting Software Cache-Based Side Channel Attacks. SIGARCH Comput. Archit. News 35, 2 (June 2007), 494--505. https://doi.org/10.1145/1273440.1250723

Digital Library

[45]

Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015. Congestion Control for Large-Scale RDMA Deployments. SIGCOMM Comput. Commun. Rev. 45, 4 (Aug. 2015), 523--536. https://doi.org/10.1145/2829988.2787484

Digital Library

[46]

Danyang Zhuo, Kaiyuan Zhang, Yibo Zhu, Hongqiang Harry Liu, Matthew Rockett, Arvind Krishnamurthy, and Thomas Anderson. 2019. Slim:OS kernel support for a low-overhead container overlay network. In 16th USENIX Symposium on Networked Systems Design and Implementation NSDI 19. 331--344.

Cited By

Li XShu RXiong YRen F(2024)Software-based Live Migration for Containerized RDMAProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663416(52-58)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663416
Huang CXue FYu PWang XChen YWu THan LHan ZWang BGong XTian CDou WChen GYin H(2024)Minimizing Buffer Utilization for Lossless Inter-DC LinksIEEE/ACM Transactions on Networking10.1109/TNET.2024.344360032:6(4960-4975)Online publication date: Dec-2024
https://doi.org/10.1109/TNET.2024.3443600
Zhang ZXia CLiang CLi JYu CBie TMartin RDan DWang XLiu YGuan H(2024)Un-IOV: Achieving Bare-Metal Level I/O Virtualization Performance for Cloud Usage With Migratability, Scalability and TransparencyIEEE Transactions on Computers10.1109/TC.2024.337558973:7(1655-1668)Online publication date: Jul-2024
https://doi.org/10.1109/TC.2024.3375589
Show More Cited By

Index Terms

MasQ: RDMA for Virtual Private Cloud
1. Networks
  1. Network services
    1. Cloud computing
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles
        Cloud computing

Recommendations

NetLord: a scalable multi-tenant network architecture for virtualized datacenters
SIGCOMM '11

Providers of "Infrastructure-as-a-Service" need datacenter networks that support multi-tenancy, scale, and ease of operation, at low cost. Most existing network architectures cannot meet all of these needs simultaneously.

In this paper we present ...
NetLord: a scalable multi-tenant network architecture for virtualized datacenters
SIGCOMM '11: Proceedings of the ACM SIGCOMM 2011 conference

Providers of "Infrastructure-as-a-Service" need datacenter networks that support multi-tenancy, scale, and ease of operation, at low cost. Most existing network architectures cannot meet all of these needs simultaneously.

In this paper we present ...
Pre-Copy and post-copy VM live migration for memory intensive applications
Euro-Par'12: Proceedings of the 18th international conference on Parallel processing workshops

Virtualization technology provides a means for server consolidation, reducing the number of physical servers required for running a given workload. Virtual Machine (VM) live migration facilitates the transfer of a running (VM) between physical hosts ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '20: Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication

July 2020

814 pages

ISBN:9781450379557

DOI:10.1145/3387514

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Natural Science Foundation of China

Conference

SIGCOMM '20

Sponsor:

SIGCOMM

SIGCOMM '20: Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication

August 10 - 14, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
2,674
Total Downloads

Downloads (Last 12 months)210
Downloads (Last 6 weeks)23

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li XShu RXiong YRen F(2024)Software-based Live Migration for Containerized RDMAProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663416(52-58)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663416
Huang CXue FYu PWang XChen YWu THan LHan ZWang BGong XTian CDou WChen GYin H(2024)Minimizing Buffer Utilization for Lossless Inter-DC LinksIEEE/ACM Transactions on Networking10.1109/TNET.2024.344360032:6(4960-4975)Online publication date: Dec-2024
https://doi.org/10.1109/TNET.2024.3443600
Zhang ZXia CLiang CLi JYu CBie TMartin RDan DWang XLiu YGuan H(2024)Un-IOV: Achieving Bare-Metal Level I/O Virtualization Performance for Cloud Usage With Migratability, Scalability and TransparencyIEEE Transactions on Computers10.1109/TC.2024.337558973:7(1655-1668)Online publication date: Jul-2024
https://doi.org/10.1109/TC.2024.3375589
Rosa LFoschini LCorradi A(2024)Empowering Cloud Computing With Network Acceleration: A SurveyIEEE Communications Surveys & Tutorials10.1109/COMST.2024.337753126:4(2729-2768)Online publication date: Dec-2025
https://doi.org/10.1109/COMST.2024.3377531
Wei CLi XYang YJiang XXu TYang BWu TXu CLv YGao HZhang ZChen ZWang ZZhang ZZhu SChen WSchulzrinne HKohler EMaltz DMisra V(2023)Achelous: Enabling Programmability, Elasticity, and Reliability in Hyperscale Cloud NetworksProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604859(769-782)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604859
Rosa LGarbugli ACorradi ABellavista P(2023)INSANEProceedings of the 24th International Middleware Conference10.1145/3590140.3629105(57-70)Online publication date: 27-Nov-2023
https://dl.acm.org/doi/10.1145/3590140.3629105
Liu SWang QZhang JWu WLin QLiu YXu MCanini MCheung RHe JAamodt TJerger NSwift M(2023)In-Network Aggregation with Transport Transparency for Distributed TrainingProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3582016.3582037(376-391)Online publication date: 25-Mar-2023
https://dl.acm.org/doi/10.1145/3582016.3582037
Liu HLi WPang YPei RHu YLiu YSuo LLi K(2023)Accelerating Data Delivery of Latency-Sensitive Applications in Container Overlay NetworkIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330074534:12(3046-3058)Online publication date: Dec-2023
https://doi.org/10.1109/TPDS.2023.3300745
Du XXu KXu LZheng KShen MWu BLi T(2023)R-AQM: Reverse ACK Active Queue Management in Multitenant Data CentersIEEE/ACM Transactions on Networking10.1109/TNET.2022.319797331:2(526-541)Online publication date: Apr-2023
https://doi.org/10.1109/TNET.2022.3197973
Kampa TEl-Ankah AGrossmann D(2023)High Availability for virtualized Programmable Logic Controllers with Hard Real-Time Requirements on Cloud Infrastructures2023 IEEE 21st International Conference on Industrial Informatics (INDIN)10.1109/INDIN51400.2023.10218014(1-8)Online publication date: 18-Jul-2023
https://doi.org/10.1109/INDIN51400.2023.10218014
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents