research-article

Public Access

Sonata: query-driven streaming network telemetry

Authors:

Jennifer Rexford,

Walter WillingerAuthors Info & Claims

SIGCOMM '18: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

Pages 357 - 371

https://doi.org/10.1145/3230543.3230555

Published: 07 August 2018 Publication History

Abstract

Managing and securing networks requires collecting and analyzing network traffic data in real time. Existing telemetry systems do not allow operators to express the range of queries needed to perform management or scale to large traffic volumes and rates. We present Sonata, an expressive and scalable telemetry system that coordinates joint collection and analysis of network traffic. Sonata provides a declarative interface to express queries for a wide range of common telemetry tasks; to enable real-time execution, Sonata partitions each query across the stream processor and the data plane, running as much of the query as it can on the network switch, at line rate. To optimize the use of limited switch memory, Sonata dynamically refines each query to ensure that available resources focus only on traffic that satisfies the query. Our evaluation shows that Sonata can support a wide range of telemetry tasks while reducing the workload for the stream processor by as much as seven orders of magnitude compared to existing telemetry systems.

References

[1]

Antonakakis, M., April, T., Bailey, M., Bernhard, M., Bursztein, E., Cochran, J., Durumeric, Z., Halderman, J. A., Invernizzi, L., Kallitsis, M., et al. Understanding the Mirai botnet. In USENIX Security Symposium (2017).

Digital Library

[2]

Apache Thrift API. https://thrift.apache.org/.

[3]

Arashloo, M. T., Koral, Y., Greenberg, M., Rexford, J., and Walker, D. SNAP: Stateful network-wide abstractions for packet processing. In ACM SIGCOMM (2016).

Digital Library

[4]

Armbrust, M., Xin, R. S., Lian, C., Huai, Y., Liu, D., Bradley, J. K., Meng, X., Kaftan, T., Franklin, M. J., Ghodsi, A., et al. Spark SQL: Relational Data Processing in Spark. In ACM SIGMOD International Conference on Management of Data (2015).

Digital Library

[5]

Assignment 3, COS 561, Princeton University. https://github.com/Sonata-Princeton/SONATA-DEV/tree/tutorial/sonata/tutorials/Tutorial-1.

[6]

Bilge, L., Kirda, E., Kruegel, C., and Balduzzi, M. Exposure: Finding malicious domains using passive DNS analysis. In USENIX Network and Distributed System Security Symposium (2011).

[7]

Borders, K., Springer, J., and Burnside, M. Chimera: A declarative language for streaming network traffic analysis. In USENIX Security Symposium (2012).

Digital Library

[8]

Bosshart, P., Daly, D., Gibb, G., Izzard, M., McKeown, N., Rexford, J., Schlesinger, C., Talayco, D., Vahdat, A., Varghese, G., and Walker, D. P4: Programming Protocol-independent Packet Processors. ACM SIGCOMM Computer Communication Review 44, 3 (July 2014), 87--95.

Digital Library

[9]

Bosshart, P., Gibb, G., Kim, H.-S., Varghese, G., McKeown, N., Izzard, M., Mujica, F., and Horowitz, M. Forwarding metamorphosis: Fast programmable match-action processing in hardware for SDN. In ACM SIGCOMM (2013).

Digital Library

[10]

Cranor, C., Johnson, T., Spatschek, O., and Shkapenyuk, V. Gigascope: A stream database for network applications. In ACM SIGMOD International Conference on Management of Data (2003).

Digital Library

[11]

The CAIDA UCSD Anonymized Internet Traces 2016-09. http://www.caida.org/data/passive/passive_2016_dataset.xml.

[12]

Emmerich, P., Gallenmüller, S., Raumer, D., Wohlfart, F., and Carle, G. Moongen: A scriptable high-speed packet generator. In ACM Internet Measurement Conference (2015).

Digital Library

[13]

Estan, C., Savage, S., and Varghese, G. Automatically inferring patterns of resource consumption in network traffic. In ACM SIGCOMM (2003).

Digital Library

[14]

Fan, J., Xu, J., Ammar, M. H., and Moon, S. B. Prefix-preserving IP address anonymization: Measurement-based security evaluation and a new cryptography-based scheme. Computer Networks (2004).

Digital Library

[15]

Gil, T. M., and Poletto, M. MULTOPS: A data-structure for bandwidth attack detection. In USENIX Security Symposium (2001).

Digital Library

[16]

Gupta, A., Birkner, R., Canini, M., Feamster, N., MacStoker, C., and Willinger, W. Network Monitoring as a Streaming Analytics Problem. In ACM HotNets (2016).

Digital Library

[17]

Gurobi Solver. http://www.gurobi.com/.

[18]

Harrison, R., Qizhe, C., Gupta, A., and Rexford, J. Network-Wide Heavy Hitter Detection with Commodity Switches. In ACM Symposium on SDN Research (SOSR) (2018).

Digital Library

[19]

Hira, M., and Wobker, L. J. Improving Network Monitoring and Management with Programmable Data Planes. Blog posting, http://p4.org/p4/inband-network-telemetry/, September 2015.

[20]

Izzard, M. The Programmable Switch Chip Consigns Legacy Fixed-Function Chips to the History Books. https://goo.gl/JKWnQc, September 2016.

[21]

Javed, M., and Paxson, V. Detecting stealthy, distributed SSH brute-forcing. In ACM SIGSAC Conference on Computer & Communications Security (2013), pp. 85--96.

Digital Library

[22]

Jose, L., Yan, L., Varghese, G., and McKeown, N. Compiling packet programs to reconfigurable switches. In USENIX NSDI (2015).

Digital Library

[23]

Jose, L., Yu, M., and Rexford, J. Online measurement of large traffic aggregates on commodity switches. In Workshop on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services (March 2011).

Digital Library

[24]

Jung, J., Paxson, V., Berger, A. W., and Balakrishnan, H. Fast portscan detection using sequential hypothesis testing. In IEEE Symposium on Security and Privacy (2004), IEEE, pp. 211--225.

[25]

Kührer, M., Hupperich, T., Rossow, C., and Holz, T. Exit from hell? Reducing the impact of amplification DDoS attacks. In USENIX Security Symposium (2014).

Digital Library

[26]

Liu, Z., Manousis, A., Vorsanger, G., Sekar, V., and Braverman, V. One sketch to rule them all: Rethinking network flow monitoring with UnivMon. In ACM SIGCOMM (2016).

Digital Library

[27]

Madden, S., Franklin, M. J., Hellerstein, J. M., and Hong, W. TAG: A Tiny Aggregation Service for Ad-hoc Sensor Networks. In USENIX OSDI (2002).

Digital Library

[28]

Madden, S. R., Franklin, M. J., Hellerstein, J. M., and Hong, W. TinyDB: An Acquisitional Query Processing System for Sensor Networks. ACM Transaction on Database System 30, 1 (2005).

Digital Library

[29]

Moshref, M., Yu, M., Govindan, R., and Vahdat, A. Dream: Dynamic resource allocation for software-defined measurement. ACM SIGCOMM (2015).

Digital Library

[30]

Moshref, M., Yu, M., Govindan, R., and Vahdat, A. Scream: Sketch resource allocation for software-defined measurement. In ACM CoNEXT (2015).

Digital Library

[31]

Moshref, M., Yu, M., Govindan, R., and Vahdat, A. Trumpet: Timely and precise triggers in data centers. In ACM SIGCOMM (2016).

Digital Library

[32]

Mullin, J. K. Optimal Semijoins for Distributed Database Systems. IEEE Transactions on Software Engineering 16, 5 (1990).

Digital Library

[33]

Narayana, S., Arashloo, M. T., Rexford, J., and Walker, D. Compiling path queries. In USENIX NSDI (2016).

Digital Library

[34]

Narayana, S., Sivaraman, A., Nathan, V., Goyal, P., Arun, V., Alizadeh, M., Jeyakumar, V., and Kim, C. Language-directed Hardware Design for Network Performance Monitoring. In ACM SIGCOMM (2017).

Digital Library

[35]

Pa, Y. M. P., Suzuki, S., Yoshioka, K., Matsumoto, T., Kasama, T., and Rossow, C. IoTPOT: Analysing the rise of IoT compromises. In USENIX Workshop on Offensive Technology (2015).

Digital Library

[36]

Polychroniou, O., Sen, R., and Ross, K. A. Track join: Distributed joins with minimal network traffic. In ACM SIGMOD International Conference on Management of Data (2014).

Digital Library

[37]

An update on the Memcached/Redis benchmark. http://oldblog.antirez.com/post/update-on-memcached-redis-benchmark.html.

[38]

Apache Flink. http://flink.apache.org/.

[39]

Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines). https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines.

[40]

OpenSOC. http://opensoc.github.io/.

[41]

OpenSOC Scalability. https://goo.gl/CX2jWr.

[42]

Sonata's technical report. http://www.cs.princeton.edu/~arpitg/pdfs/sonata_tr.pdf.

[43]

The Bro Network Security Monitor. https://www.bro.org/.

[44]

The CAIDA Anonymized Internet Traces 2016 Dataset. https://www.caida.org/data/passive/passive_2016_dataset.xml.

[45]

Slowloris HTTP DoS. https://web.archive.org/web/20150426090206/http://ha.ckers.org/slowloris, June 2009.

[46]

Srivastava, U., Munagala, K., and Widom, J. Operator Placement for In-Network Stream Query Processing. In Symposium on Principles of Database Systems (2005).

Digital Library

[47]

Tammana, P., Agarwal, R., and Lee, M. Simplifying datacenter network debugging with PathDump. In USENIX OSDI (2016).

Digital Library

[48]

Apache Spark. http://spark.apache.org/.

[49]

Barefoot's Tofino. https://www.barefootnetworks.com/technology/.

[50]

P4 software switch. https://github.com/p4lang/behavioral-model.

[51]

Scapy: Python-based interactive packet manipulation program. https://github.com/secdev/scapy/.

[52]

SONATA Github. https://github.com/Sonata-Princeton/SONATA-DEV.

[53]

Sonata Queries. https://github.com/sonata-queries/sonata-queries.

[54]

Vinnakota, B. P4 with the Netronome Server Networking Platform. https://goo.gl/PKQtC7, May 2016.

[55]

Wu, Q., Strassner, J., Farrel, A., and Zhang, L. Network telemetry and big data analysis. Network Working Group Internet-Draft (2016 (Expired)).

[56]

Yu, M., Jose, L., and Miao, R. Software Defined Traffic Measurement with OpenSketch. In USENIX NSDI (2013).

Digital Library

[57]

Yuan, L., Chuah, C.-N., and Mohapatra, P. ProgME: Towards Programmable Network Measurement. In ACM SIGCOMM (2007).

Digital Library

[58]

Yuan, Y., Lin, D., Mishra, A., Marwaha, S., Alur, R., and Loo, B. T. Quantitative Network Monitoring with NetQRE. In ACM SIGCOMM (2017).

Digital Library

[59]

Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., and Stoica, I. Discretized streams: Fault-tolerant streaming computation at scale. In ACM SOSP (2013).

Digital Library

[60]

Zhu, Y., Kang, N., Cao, J., Greenberg, A., Lu, G., Mahajan, R., Maltz, D., Yuan, L., Zhang, M., Zhao, B. Y., and Zheng, H. Packet-level telemetry in large datacenter networks. In ACM SIGCOMM (2015).

Digital Library

Cited By

Kumar ABose ATiwari KMishra ADixit AKhan AVutukuru M(2024)Feasibility of Application Layer Header Parsing in eBPF and P42024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619855(475-481)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619855
Li EWu WZhaohua WLi ZNiu J(2024)SAROS: A Self-Adaptive Routing Oblivious Sampling Method for Network-wide Heavy Hitter DetectionProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663429(142-148)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663429
Liu KZhang JJiang ZZhang XGuo SBai YDong YZhang ZShi XWang LWei HWang ZPan YPan THuang T(2024)Hostmesh: Monitor and Diagnose Networks in Rail-optimized RoCE ClustersProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663426(122-128)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663426
Show More Cited By

Index Terms

Sonata: query-driven streaming network telemetry
1. Networks
  1. Network services
    1. Network monitoring

Recommendations

Concerto: cooperative network-wide telemetry with controllable error rate
APSys '20: Proceedings of the 11th ACM SIGOPS Asia-Pacific Workshop on Systems

Network-wide telemetry requires real-time analysis of a large amount of traffic. Telemetry systems use stream processors to support various applications, and Protocol Independent Switching Architecture switches to reduce the workload on stream ...
Video query processing in the VDBMS testbed for video database research
MMDB '03: Proceedings of the 1st ACM international workshop on Multimedia databases

The increased use of video data sets for multimedia-based applications has created a demand for strong video database support, including efficient methods for handling the content-based query and retrieval of video data. Video query processing presents ...
Query indexing with containment-encoded intervals for efficient stream processing

Many continual range queries can be issued against data streams. To efficiently evaluate continual queries against a stream, a main memory-based query index with a small storage cost and a fast search time is needed, especially if the stream is rapid. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '18: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

August 2018

604 pages

ISBN:9781450355674

DOI:10.1145/3230543

General Chairs:
Sergey Gorinsky
IMDEA, Spain
,
János Tapolcai
BME, Hungary

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SIGCOMM '18

Sponsor:

SIGCOMM

SIGCOMM '18: ACM SIGCOMM 2018 Conference

August 20 - 25, 2018

Budapest, Hungary

Acceptance Rates

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

185
Total Citations
View Citations
4,718
Total Downloads

Downloads (Last 12 months)553
Downloads (Last 6 weeks)121

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kumar ABose ATiwari KMishra ADixit AKhan AVutukuru M(2024)Feasibility of Application Layer Header Parsing in eBPF and P42024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619855(475-481)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619855
Li EWu WZhaohua WLi ZNiu J(2024)SAROS: A Self-Adaptive Routing Oblivious Sampling Method for Network-wide Heavy Hitter DetectionProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663429(142-148)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663429
Liu KZhang JJiang ZZhang XGuo SBai YDong YZhang ZShi XWang LWei HWang ZPan YPan THuang T(2024)Hostmesh: Monitor and Diagnose Networks in Rail-optimized RoCE ClustersProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663426(122-128)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663426
Liu KJiang ZZhang JGuo SZhang XBai YDong YLuo FZhang ZWang LShi XXu HBai YSong DWei HLi BPan YPan THuang TSekar VYu MSeneviratne AVeitch D(2024)R-Pingmesh: A Service-Aware RoCE Network Monitoring and Diagnostic SystemProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672264(554-567)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672264
Ding RLiu XYang SHuang QXie BSun RZhang ZCui BSekar VYu MSeneviratne AVeitch D(2024)RD-Probe: Scalable Monitoring With Sufficient Coverage In Complex Datacenter NetworksProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672256(258-273)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672256
Zheng HHuang CHan XZheng JWang XTian CDou WChen GSekar VYu MSeneviratne AVeitch D(2024)μMon: Empowering Microsecond-level Network Monitoring with WaveletsProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672236(274-290)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672236
Srivastava MHung SNamkung HLin KLiu ZSekar VVallina-Rodríguez NSuarez-Tángil GLevin DPelsser C(2024)Raising the Level of Abstraction for Sketch-Based Network Telemetry with SketchPlanProceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3689016(651-658)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3646547.3689016
Ji HXie KWen JZhang QXie GLiang W(2024)FineMon: An Innovative Adaptive Network Telemetry Scheme for Fine-Grained, Multi-Metric Data Monitoring with Dynamic Frequency Adjustment and Enhanced Data RecoveryProceedings of the ACM on Management of Data10.1145/36392672:1(1-26)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639267
Seufert MDietz KWehner NGeißler SSchüler JWolz MHotho ACasas PHoßfeld TFeldmann A(2024) Marina : Realizing ML-Driven Real-Time Network Traffic Monitoring at Terabit Scale IEEE Transactions on Network and Service Management10.1109/TNSM.2024.338239321:3(2773-2790)Online publication date: Jun-2024
https://doi.org/10.1109/TNSM.2024.3382393
Kfoury ECrichigno JBou-Harb E(2024)P4BS: Leveraging Passive Measurements From P4 Switches to Dynamically Modify a Router’s Buffer SizeIEEE Transactions on Network and Service Management10.1109/TNSM.2023.330633521:1(1082-1099)Online publication date: Feb-2024
https://doi.org/10.1109/TNSM.2023.3306335
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents