research-article

Public Access

Zero-CPU Collection with Direct Telemetry Access

Authors:

Jonatan Langlet,

Sivaramakrishnan Ramanathan,

Gabriele Oliaro,

Michael Mitzenmacher,

Gianni AntichiAuthors Info & Claims

HotNets '21: Proceedings of the 20th ACM Workshop on Hot Topics in Networks

Pages 108 - 115

https://doi.org/10.1145/3484266.3487366

Published: 04 November 2021 Publication History

Abstract

Programmable switches are driving a massive increase in fine-grained measurements. This puts significant pressure on telemetry collectors that have to process reports from many switches. Past research acknowledged this problem by either improving collectors' stack performance or by limiting the amount of data sent from switches. In this paper, we take a different and radical approach: switches are responsible for directly inserting queryable telemetry data into the collectors' memory, bypassing their CPU, and thereby improving their collection scalability. We propose to use a method we call direct telemetry access, where switches jointly write telemetry reports directly into the same collector's memory region, without coordination. Our solution, DART, is probabilistic, trading memory redundancy and query success probability for CPU resources at collectors. We prototype DART using commodity hardware such as P4 switches and RDMA NICs and show that we get high query success rates with a reasonable memory overhead. For example, we can collect INT path tracing information on a fat tree topology without a collector's CPU involvement while achieving 99.9% query success probability and using just 300 bytes per flow.

References

[1]

Mohammad Alizadeh, Tom Edsall, Sarang Dharmapurikar, Ramanan Vaidyanathan, Kevin Chu, Andy Fingerhut, Vinh The Lam, Francis Matus, Rong Pan, Navindra Yadav, et al. 2014. CONGA: Distributed congestion-aware load balancing for datacenters. In Proceedings of the 2014 ACM conference on SIGCOMM. 503--514.

Digital Library

[2]

Emmanuel Amaro, Zhihong Luo, Amy Ousterhout, Arvind Krishnamurthy, Aurojit Panda, Sylvia Ratnasamy, and Scott Shenker. 2020. Remote Memory Calls. In Proceedings of the 19th ACM Workshop on Hot Topics in Networks. 38--44.

Digital Library

[3]

Arista. [n. d.]. Telemetry and Analytics. https://www.arista.com/en/solutions/telemetry-analytics. ([n. d.]). Accessed: 2021-06-24.

[4]

Infiniband Trade Association. [n. d.]. RoCEv2. https://cw.infinibandta.org/document/dl/7781. ([n. d.]). Accessed: 2021-05-12.

[5]

Ran Ben Basat, Xiaoqi Chen, Gil Einziger, Shir Landau Feibish, Danny Raz, and Minlan Yu. 2020. Routing Oblivious Measurement Analytics. In 2020 IFIP Networking Conference (Networking). IEEE, 449--457.

[6]

Ran Ben Basat, Xiaoqi Chen, Gil Einziger, and Ori Rottenstreich. 2020. Designing heavy-hitter detection algorithms for programmable switches. IEEE/ACM Transactions on Networking 28, 3 (2020), 1172--1185.

Digital Library

[7]

Ran Ben Basat, Sivaramakrishnan Ramanathan, Yuliang Li, Gianni Antichi, Minian Yu, and Michael Mitzenmacher. 2020. PINT: probabilistic in-band network telemetry. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 662--680.

Digital Library

[8]

Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, et al. 2014. P4: Programming protocol-independent packet processors. ACM SIGCOMM Computer Communication Review 44, 3 (2014), 87--95.

Digital Library

[9]

BROADCOM. [n. d.]. Trident Programmable Switch. https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56870-series. ([n. d.]).

[10]

Andrei Broder and Michael Mitzenmacher. 2004. Network applications of bloom filters: A survey. Internet mathematics 1, 4 (2004), 485--509.

[11]

Cisco. [n. d.]. Explore Model-Driven Telemetry. https://blogs.cisco.com/developer/model-driven-telemetry-sandbox. ([n. d.]). Accessed: 2021-06-24.

[12]

Hadi Esmaeilzadeh, Emily Blem, Renee St. Amant, Karthikeyan Sankaralingam, and Doug Burger. 2011. Dark silicon and the end of multicore scaling. In International Symposium on Computer Architecture (ISCA). ACM.

Digital Library

[13]

Nishant Garg. 2013. Apache kafka. Packt Publishing Ltd.

[14]

Michael T Goodrich and Michael Mitzenmacher. 2011. Invertible bloom lookup tables. In 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 792--799.

[15]

The P4.org Applications Working Group. [n. d.]. Telemetry Report Format Specification. https://github.com/p4lang/p4-applications/blob/master/docs/telemetry_report_latest.pdf. ([n. d.]). Accessed: 2021-06-23.

[16]

Chuanxiong Guo, Lihua Yuan, Dong Xiang, Yingnong Dang, Ray Huang, Dave Maltz, Zhaoyi Liu, Vin Wang, Bin Pang, Hua Chen, et al. 2015. Pingmesh: A large-scale system for data center network latency measurement and analysis. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 139--152.

Digital Library

[17]

Arpit Gupta, Rob Harrison, Marco Canini, Nick Feamster, Jennifer Rexford, and Walter Willinger. 2018. Sonata: Query-driven streaming network telemetry. In Proceedings of the 2018 conference of the ACM special interest group on data communication. 357--371.

Digital Library

[18]

Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazières, and Nick McKeown. 2014. I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks. In Networked Systems Design and Implementation (NSDI). USENIX Association.

[19]

Nikos Hardavellas. 2012. The rise and fall of dark silicon. In ;login:, Volume: 37. USENIX.

[20]

Chris Hare. 2011. Simple Network Management Protocol (SNMP). (2011).

[21]

Brandon Heller, Srinivasan Seetharaman, Priya Mahadevan, Yiannis Yiakoumis, Puneet Sharma, Sujata Banerjee, and Nick McKeown. 2010. Elastictree: Saving energy in data center networks. In Nsdi, Vol. 10. 249--264.

Digital Library

[22]

Qun Huang, Haifeng Sun, Patrick PC Lee, Wei Bai, Feng Zhu, and Yungang Bao. 2020. Omnimon: Re-architecting network telemetry with resource efficiency and full accuracy. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 404--421.

Digital Library

[23]

Huawei. [n. d.]. Overview of Telemetry. https://support.huawei.com/enterprise/en/doc/EDOC1000173015/165fa2c8/overview-of-telemetry. ([n. d.]). Accessed: 2021-06-24.

[24]

Infiniband Trade Association. 2015. InfiniBandTM Architecture Specification. (2015). Volume 1 Release 1.3.

[25]

Intel. [n. d.]. In-band Network Telemetry Detects Network Performance Issues. https://builders.intel.com/docs/networkbuilders/in-band-network-telemetry-detects-network-performance-issues.pdf. ([n. d.]). Accessed: 2021-06-04.

[26]

Intel. [n. d.]. Intel Tofino Series Programmable Ethernet Switch ASIC. https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet-switch/tofino-series/tofino.html. ([n. d.]). Accessed: 2021-05-12.

[27]

Intel. [n. d.]. Intel® Deep Insight Network Analytics Software. https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet-switch/network-analytics/deep-insight.html. ([n. d.]). Accessed: 2021-06-10.

[28]

Intel. [n. d.]. Intel® Ethernet Network Adapter E810-CQDA1/CQDA2. https://www.intel.com/content/www/us/en/products/docs/network-io/ethernet/network-adapters/ethernet-800-series-network-adapters/e810-cqda1-cqda2-100gbe-brief.html. ([n. d.]). Accessed: 2021-06-11.

[29]

Intel. [n. d.]. Intel® P4 Studio. https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet-switch/p4-suite/p4-studio.html. ([n. d.]). Accessed: 2021-06-08.

[30]

Vimalkumar Jeyakumar, Mohammad Alizadeh, Yilong Geng, Changhoon Kim, and David Mazières. 2014. Millions of little minions: Using packets for low latency network programming and visibility. ACM SIGCOMM Computer Communication Review 44, 4 (2014), 3--14.

Digital Library

[31]

Anurag Khandelwal, Rachit Agarwal, and Ion Stoica. 2019. Confluo: Distributed monitoring and diagnosis stack for high-speed networks. In 16th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 19). 421--436.

[32]

Changhoon Kim, Anirudh Sivaraman, Naga Katta, Antonin Bas, Advait Dixit, and Lawrence J Wobker. 2015. In-band network telemetry via programmable dataplanes. In ACM SIGCOMM.

[33]

Daehyeok Kim, Zaoxing Liu, Yibo Zhu, Changhoon Kim, Jeongkeun Lee, Vyas Sekar, and Srinivasan Seshan. 2020. Tea: Enabling stateintensive network functions on programmable switches. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 90--106.

Digital Library

[34]

Jan Kučera, Diana Andreea Popescu, Han Wang, Andrew Moore, Jan Kořenek, and Gianni Antichi. 2020. Enabling event-triggered data plane monitoring. In Proceedings of the Symposium on SDN Research. 14--26.

Digital Library

[35]

Yiran Li, Kevin Gao, Xin Jin, and Wei Xu. 2020. Concerto: cooperative network-wide telemetry with controllable error rate. In Proceedings of the 11th ACM SIGOPS Asia-Pacific Workshop on Systems. 114--121.

Digital Library

[36]

Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. 2019. HPCC: high precision congestion control. In Proceedings of the ACM Special Interest Group on Data Communication. 44--58.

[37]

Richard J Lipton. 1994. A new approach to information theory. In Annual Symposium on Theoretical Aspects of Computer Science. Springer, 699--708.

[38]

Zaoxing Liu, Antonis Manousis, Gregory Vorsanger, Vyas Sekar, and Vladimir Braverman. 2016. One sketch to rule them all: Rethinking network flow monitoring with univmon. In Proceedings of the 2016 ACM SIGCOMM Conference. 101--114.

Digital Library

[39]

Wassim Mansour, Nicolas Janvier, and Pablo Fajardo. 2019. FPGA implementation of RDMA-based data acquisition system over 100-Gb ethernet. IEEE Transactions on Nuclear Science 66, 7 (2019), 1138--1143.

[40]

Michael Mitzenmacher and Eli Upfal. 2017. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis. Cambridge university press.

[41]

Juniper Networks. [n. d.]. Overview of the Junos Telemetry Interface. https://www.juniper.net/documentation/us/en/software/junos/interfaces-telemetry/topics/concept/junos-telemetry-interface-oveview.html. ([n. d.]). Accessed: 2021-06-24.

[42]

NVIDIA. [n. d.]. NVIDIA Mellanox Spectrum Switch. https://www.mellanox.com/files/doc-2020/pb-spectrum-switch.pdf. ([n. d.]).

[43]

Jeff Rasley, Brent Stephens, Colin Dixon, Eric Rozner, Wes Felter, Kanak Agarwal, John Carter, and Rodrigo Fonseca. 2014. Planck: Millisecond-scale monitoring and control for commodity networks. ACM SIGCOMM Computer Communication Review 44, 4 (2014), 407--418.

Digital Library

[44]

Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C Snoeren. 2015. Inside the social network's (datacenter) network. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 123--137.

Digital Library

[45]

David Sidler, Zeke Wang, Monica Chiosa, Amit Kulkarni, and Gustavo Alonso. 2020. StRoM: smart remote memory. In Proceedings of the Fifteenth European Conference on Computer Systems. 1--16.

Digital Library

[46]

Praveen Tammana, Rachit Agarwal, and Myungjin Lee. 2018. Distributed network monitoring and debugging with switchpointer. In 15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18). 453--456.

[47]

Intel DPDK Validation team. [n. d.]. DPDK Intel NIC Performance Report Release 20.11. https://fast.dpdk.org/doc/perf/DPDK_20_11_Intel_NIC_performance_report.pdf. ([n. d.]). Accessed: 2021-05-07.

[48]

Mellanox Technologies. [n. d.]. ConnectX®-6 VPI Card. https://www.mellanox.com/files/doc-2020/pb-connectx-6-vpi-card.pdf. ([n. d.]). Accessed: 2021-05-12.

[49]

Nguyen Van Tu, Jonghwan Hyun, and James Won-Ki Hong. 2017. Towards onos-based sdn monitoring using in-band network telemetry. In 2017 19th Asia-Pacific Network Operations and Management Symposium (APNOMS). IEEE, 76--81.

[50]

Nguyen Van Tu, Jonghwan Hyun, Ga Yeon Kim, Jae-Hyoung Yoo, and James Won-Ki Hong. 2018. Intcollector: A high-performance collector for in-band network telemetry. In 2018 14th International Conference on Network and Service Management (CNSM). IEEE, 10--18.

[51]

Jonathan Vestin, Andreas Kassler, Deval Bhamare, Karl-Johan Grinnemo, Jan-Olof Andersson, and Gergely Pongracz. 2019. Programmable event detection for in-band network telemetry. In 2019 IEEE 8th international conference on cloud networking (CloudNet). IEEE, 1--6.

[52]

Xilinx. [n. d.]. Xilinx Embedded RDMA Enabled NIC. https://www.xilinx.com/support/documentation/ip_documentation/ernic/v3_0/pg332-ernic.pdf. ([n. d.]). Accessed: 2021-06-11.

[53]

Da Yu, Yibo Zhu, Behnaz Arzani, Rodrigo Fonseca, Tianrong Zhang, Karl Deng, and Lihua Yuan. 2019. dShark: A general, easy to program and scalable framework for analyzing in-network packet traces. In 16th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 19). 207--220.

[54]

Minlan Yu. 2019. Network telemetry: towards a top-down approach. ACM SIGCOMM Computer Communication Review 49, 1 (2019), 11--17.

Digital Library

[55]

Yu Zhou, Jun Bi, Tong Yang, Kai Gao, Jiamin Cao, Dai Zhang, Yangyang Wang, and Cheng Zhang. 2020. Hypersight: Towards scalable, high-coverage, and dynamic network monitoring queries. IEEE Journal on Selected Areas in Communications 38, 6 (2020), 1147--1160.

[56]

Yu Zhou, Chen Sun, Hongqiang Harry Liu, Rui Miao, Shi Bai, Bo Li, Zhilong Zheng, Lingjun Zhu, Zhen Shen, Yongqing Xi, et al. 2020. Flow event telemetry on programmable data plane. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 76--89.

Digital Library

[57]

Yibo Zhu, Nanxi Kang, Jiaxin Cao, Albert Greenberg, Guohan Lu, Ratul Mahajan, Dave Maltz, Lihua Yuan, Ming Zhang, Ben Y Zhao, et al. 2015. Packet-level telemetry in large datacenter networks. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 479--491.

Digital Library

Cited By

Hussain MCho B(2024)HPTCollector: high-performance telemetry collectorCluster Computing10.1007/s10586-024-04650-w27:10(14729-14744)Online publication date: 30-Jul-2024
https://doi.org/10.1007/s10586-024-04650-w
Shahout RFriedman RBen Basat R(2023)Together is Better: Heavy Hitters Quantile EstimationProceedings of the ACM on Management of Data10.1145/35889371:1(1-25)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588937
Chiesa MVerdi F(2023)Network Monitoring on Multi-Pipe SwitchesProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/35793217:1(1-31)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3579321
Show More Cited By

Index Terms

Zero-CPU Collection with Direct Telemetry Access

Index terms have been assigned to the content through auto-classification.

Recommendations

Direct Telemetry Access
ACM SIGCOMM '23: Proceedings of the ACM SIGCOMM 2023 Conference

Fine-grained network telemetry is becoming a modern datacenter standard and is the basis of essential applications such as congestion control, load balancing, and advanced troubleshooting. As network size increases and telemetry gets more fine-grained, ...
Age-based garbage collection

Modern generational garbage collectors look for garbage among the young objects, because they have high mortality; however, these objects include the very youngest objects, which clearly are still live. We introduce new garbage collection algorithms, ...
Garbage collection without paging
Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation

Garbage collection offers numerous software engineering advantages, but interacts poorly with virtual memory managers. Existing garbage collectors require far more pages than the application's working set and touch pages without regard to which ones are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HotNets '21: Proceedings of the 20th ACM Workshop on Hot Topics in Networks

November 2021

246 pages

ISBN:9781450390873

DOI:10.1145/3484266

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 November 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

HotNets '21

Sponsor:

SIGCOMM

HotNets '21: The 20th ACM Workshop on Hot Topics in Networks

November 10 - 12, 2021

Virtual Event, United Kingdom

Acceptance Rates

Overall Acceptance Rate 110 of 460 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
549
Total Downloads

Downloads (Last 12 months)154
Downloads (Last 6 weeks)13

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hussain MCho B(2024)HPTCollector: high-performance telemetry collectorCluster Computing10.1007/s10586-024-04650-w27:10(14729-14744)Online publication date: 30-Jul-2024
https://doi.org/10.1007/s10586-024-04650-w
Shahout RFriedman RBen Basat R(2023)Together is Better: Heavy Hitters Quantile EstimationProceedings of the ACM on Management of Data10.1145/35889371:1(1-25)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588937
Chiesa MVerdi F(2023)Network Monitoring on Multi-Pipe SwitchesProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/35793217:1(1-31)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3579321
Alhamed FScano DCastoldi PVegas Olmos JVershkov IPaolucci FCugini F(2023)P4 Telemetry collectorComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2023.109727227:COnline publication date: 1-May-2023
https://dl.acm.org/doi/10.1016/j.comnet.2023.109727

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents