research-article

Efficient wide area data transfer protocols for 100 Gbps networks and beyond

Authors:

Eric PouyoulAuthors Info & Claims

NDM '13: Proceedings of the Third International Workshop on Network-Aware Data Management

Article No.: 3, Pages 1 - 10

https://doi.org/10.1145/2534695.2534699

Published: 17 November 2013 Publication History

Abstract

Due to a number of recent technology developments, now is the right time to re-examine the use of TCP for very large data transfers. These developments include the deployment of 100 Gigabit per second (Gbps) network backbones, hosts that can easily manage 40 Gbps, and higher, data transfers, the Science DMZ model, the availability of virtual circuit technology, and wide-area Remote Direct Memory Access (RDMA) protocols. In this paper we show that RDMA works well over wide-area virtual circuits, and uses much less CPU than TCP or UDP. We also characterize the limitations of RDMA in the presence of other traffic, including competing RDMA flows. We conclude that RDMA for Science DMZ to Science DMZ transfers of massive data is a viable and desirable option for high-performance data transfer.

References

[1]

Aspera fasp. http://asperasoft.com/technology/transport/fasp/.

[2]

Connectx-2 EN with RDMA over Ethernet (RoCE). http://www.mellanox.com/related-docs/prod_software/ConnectX-2_RDMA_RoCE.pdf.

[3]

Gem: Network impairment emulator. http://www.spirent.com/Solutions-Directory/Impairments_GEM.

[4]

Infiniband trade association. http://www.infinibandta.org.

[5]

Ofed: Open fabrics enterprise distribution. https://openfabrics.org/ofed-for-linux-ofed-for-windows/ofed-overview.html.

[6]

Results from nasa high end computing (hec) wan file transfer experiments/demonstrations super computing 2012 (sc12). http://science.gsfc.nasa.gov/606.1/docs/SC12.pdf.

[7]

Infiniband. architecture specification release 1.2.1 annex a16: Roce. Infiniband Trade Association, 2010.

[8]

Intel direct ethernet transport (det). http://software.intel.com/en-us/articles/intel-direct-ethernet-transport, 2010.

[9]

P. Balaji. Sockets vs rdma interface over 10-gigabit networks: An in-depth analysis of the memory traffic bottleneck. In In RAIT workshop 04, page 2004, 2004.

[10]

C. Barakat, E. Altman, and W. Dabbous. On tcp performance in a heterogeneous network: A survey. IEEE Communications Magazine, 38: 40--46, 2000.

Digital Library

[11]

A. Barczyk, A. Mughal, and et. al. Efficient lhc data distribution across 100gbps networks. In Proceedings of the 2012 ACM/IEEE international Conference For High Performance Computing, Networking, Storage and Analysis, SC Companion, Nov 2012.

Digital Library

[12]

Biological and Environmental Research Network Requirements. Office of Biological and Environmental Research (BER), DOE Office of Science and the Energy Sciences Network, 2012.

[13]

D. Dalessandro and P. Wyckoff. A performance analysis of the ammasso rdma enabled ethernet adapter and its iwarp api. In In Proceedings of the IEEE Cluster 2005 Conference, RAIT Workshop, 2005.

[14]

E. Dart, L. Rotman, B. Tierney, M. Hester, and J. Zurawski. The science dmz: A network design pattern for data-intensive science. In IEEE/ACM Annual SuperComputing Conference (SC13), Denver CO, USA, 2013.

Digital Library

[15]

A. C. de Melo. The new linux 'perf' tools. http://vger.kernel.org/~acme/perf/lk2010-perf-paper.pdf, 2010.

[16]

GridFTP. http://www.globus.org/datagrid/gridftp.html.

[17]

Y. Gu and R. Grossman. Udt: Udp-based data transfer for high-speed wide area networks. Computer Networks (Elsevier) Volume 51, Issue 7, 2007.

Digital Library

[18]

V. Jacobsen, R. Braden, and D. Borman. Tcp extensions for high performance. RFC 1323, May 1992.

Digital Library

[19]

R. Kettimuthu, W. Liu, J. M. Link, and J. Bresnahan. A gridftp transport driver for globus xio. In PDPTA, pages 843--849, 2008.

[20]

E. Kissel and M. Swany. Evaluating high performance data transfer with rdma-based protocols in wide-area networks. In Proceedings of the 14th IEEE International Conference on High Performance Computing and Communications (HPCC-12), 2012.

Digital Library

[21]

P. MacArthur and R. D. Russell. A performance study to guide rdma programming decisions. In High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems (HPCC-ICESS), June 2012.

Digital Library

[22]

S. Molnár, B. Sonkoly, and T. A. Trinh. A comprehensive tcp fairness analysis in high speed networks. Comput. Commun., 32(13--14): 1460--1484, 2009.

Digital Library

[23]

I. Monga, C. Guok, W. E. Johnston, and B. Tierney. Hybrid networks: Lessons learned and future challenges based on esnet4 experience. In IEEE Communications Magazine, May 2011.

[24]

S. Narravula, H. Subramoni, P. Lai, R. Noronha, and D. K. Panda. Performance of hpc middleware over infiniband wan. In Proceedings of the 2008 37th International Conference on Parallel Processing, ICPP '08, pages 304--311, Washington, DC, USA, 2008. IEEE Computer Society.

Digital Library

[25]

N. S. V. Rao, W. Yu, W. R. Wing, S. W. Poole, and J. S. Vetter. Wide-area performance profiling of 10gige and infiniband technologies. In Proceedings of the 2008 ACM/IEEE conference on Supercomputing, SC '08, pages 14:1--14:12, Piscataway, NJ, USA, 2008. IEEE Press.

Digital Library

[26]

Y. Ren, T. Li, D. Yu, S. Jin, T. Robertazzi, B. Tierney, and E. Pouyoul. Protocols for wide-area data-intensive applications: Design and performance issues. In Proceedings of the 2012 ACM/IEEE international Conference For High Performance Computing, Networking, Storage and Analysis, Nov 2012.

Digital Library

[27]

B. Tierney, E. Kissel, M. Swany, and E. Pouyoul. Efficient data transfer protocols for big data. In E-Science (e-Science), 2012 IEEE 8th International Conference on, pages 1--9, 2012.

Digital Library

[28]

C. Villamizar and C. Song. High performance tcp in ansnet. SIGCOMM Comput. Commun. Rev., 24(5): 45--60, Oct. 1994.

Digital Library

[29]

W. Yu, N. Rao, P. Wyckoff, and J. Vetter. Performance of rdma-capable storage protocols on wide-area network. In 3rd Petascale Data Storage Workshop PSDW 2008, 2008.

[30]

W. Yu, Y. Tian, J. Vetter, N. S. Rao, T. Liu, and G. Shainer. Exio: Enabling globus on rdma networks - a case study with infiniband. Technical Report AU-CSSE-PASL/10-TR01, PASL, Dept. of CDDSSE, Auburn University, 2010.

[31]

W. Yu, Y. Tian, and J. S. Vetter. Efficient zero-copy noncontiguous i/o for globus on infiniband. In Proceedings of the 2010 39th International Conference on Parallel Processing Workshops, ICPPW '10, pages 362--368, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

Cited By

Mutter EShannigrahi S(2024)Science DMZ Networks: How Different Are They Really?2024 IEEE 49th Conference on Local Computer Networks (LCN)10.1109/LCN60385.2024.10639626(1-9)Online publication date: 8-Oct-2024
https://doi.org/10.1109/LCN60385.2024.10639626
Zuo TSun TZhu SLi WLu LDu ZZhang Y(2024)LoWAR: Enhancing RDMA over Lossy WANs with Transparent Error Correction2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS)10.1109/IWQoS61813.2024.10682853(1-10)Online publication date: 19-Jun-2024
https://doi.org/10.1109/IWQoS61813.2024.10682853
Chen YTian CDong JFeng SZhang XLiu CYu PXia NDou WChen G(2023)Swing: Providing Long-Range Lossless RDMA via PFC-RelayIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.321551734:1(63-75)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TPDS.2022.3215517
Show More Cited By

Index Terms

Efficient wide area data transfer protocols for 100 Gbps networks and beyond

Recommendations

Enhancing TCP performance in wide-area cellular wireless networks: transport level approaches
Wireless communications systems and networks

Internet technology-based architectures and protocols for supporting multimedia traffic over wireless networks are evolving. Since TCP (Transmission Control Protocol)/IP (Internet Protocol) is the standard network protocol stack on the Internet, its use ...
Long-haul secure data transfer using hardware-assisted GridFTP

Extreme-scale scientific collaborations require high-performance wide-area end-to-end data transports to enable fast and secure transfer of high data volumes among collaborating institutions. GridFTP is the de facto protocol for large-scale data ...
Flow and congestion control for reliable multicast communication in wide-area networks

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

NDM '13: Proceedings of the Third International Workshop on Network-Aware Data Management

November 2013

84 pages

ISBN:9781450325226

DOI:10.1145/2534695

General Chairs:
Mehmet Balman
VMware R&D & Lawrence Berkeley National Laboratory
,
Surendra Byna
Lawrence Berkeley National Laboratory
,
Brian L. Tierney
Energy Sciences Network & Lawrence Berkeley National Laboratory

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SC13

Sponsor:

SC13: International Conference for High Performance Computing, Networking, Storage and Analysis

November 17 - 21, 2013

Colorado, Denver

Acceptance Rates

NDM '13 Paper Acceptance Rate 9 of 14 submissions, 64%;

Overall Acceptance Rate 14 of 23 submissions, 61%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
314
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)2

Reflects downloads up to 04 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mutter EShannigrahi S(2024)Science DMZ Networks: How Different Are They Really?2024 IEEE 49th Conference on Local Computer Networks (LCN)10.1109/LCN60385.2024.10639626(1-9)Online publication date: 8-Oct-2024
https://doi.org/10.1109/LCN60385.2024.10639626
Zuo TSun TZhu SLi WLu LDu ZZhang Y(2024)LoWAR: Enhancing RDMA over Lossy WANs with Transparent Error Correction2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS)10.1109/IWQoS61813.2024.10682853(1-10)Online publication date: 19-Jun-2024
https://doi.org/10.1109/IWQoS61813.2024.10682853
Chen YTian CDong JFeng SZhang XLiu CYu PXia NDou WChen G(2023)Swing: Providing Long-Range Lossless RDMA via PFC-RelayIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.321551734:1(63-75)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TPDS.2022.3215517
Crichigno JKfoury EBou-Harb EGhani NCrichigno JKfoury EBou-Harb EGhani N(2021)Challenges and Open Research IssuesHigh-Speed Networks10.1007/978-3-030-88841-1_7(455-462)Online publication date: 22-Oct-2021
https://doi.org/10.1007/978-3-030-88841-1_7
Marrinan TRizzi SInsley JLong LRenambot LPapka M(2019)PxStream: Remote Visualization for Distributed Rendering Frameworks2019 IEEE 9th Symposium on Large Data Analysis and Visualization (LDAV)10.1109/LDAV48142.2019.8944369(37-41)Online publication date: Oct-2019
https://doi.org/10.1109/LDAV48142.2019.8944369
Crichigno JBou-Harb EGhani N(2019)A Comprehensive Tutorial on Science DMZIEEE Communications Surveys & Tutorials10.1109/COMST.2018.287608621:2(2041-2078)Online publication date: Oct-2020
https://doi.org/10.1109/COMST.2018.2876086
Hanford NAhuja VFarrens MTierney BGhosal D(2018)A Survey of End-System Optimizations for High-Speed NetworksACM Computing Surveys10.1145/318489951:3(1-36)Online publication date: 16-Jul-2018
https://dl.acm.org/doi/10.1145/3184899
Liu ZKettimuthu RFoster ILiu Y(2018)A Comprehensive Study of Wide Area Data Movement at a Scientific Computing Facility2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2018.00180(1604-1611)Online publication date: Jul-2018
https://doi.org/10.1109/ICDCS.2018.00180
Liu ZBalaprakash PKettimuthu RFoster IHuang HWeissman JIamnitchi AIosup A(2017)Explaining Wide Area Data Transfer PerformanceProceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing10.1145/3078597.3078605(167-178)Online publication date: 26-Jun-2017
https://dl.acm.org/doi/10.1145/3078597.3078605
Thyagaturu AMercian AMcGarry MReisslein MKellerer W(2016)Software Defined Optical Networks (SDONs)IEEE Communications Surveys & Tutorials10.1109/COMST.2016.258699918:4(2738-2786)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1109/COMST.2016.2586999
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents