research-article

Validating datacenters at scale

Authors:

Karthick Jayaraman,

Nikolaj Bjørner,

Ashish Bhargava,

Paul-Andre C Bissonnette,

Aniruddha Parkhi,

Hanukumar Pinnamraju,

Neha Milind Raje,

Parag SharmaAuthors Info & Claims

SIGCOMM '19: Proceedings of the ACM Special Interest Group on Data Communication

Pages 200 - 213

https://doi.org/10.1145/3341302.3342094

Published: 19 August 2019 Publication History

Abstract

We describe our experiences using formal methods and automated theorem proving for network operation at scale. The experiences are based on developing and applying the SecGuru and RCDC (Reality Checker for Data Centers) tools in Azure. SecGuru has been used since 2013 and thus, is arguably a pioneering industrial deployment of network verification. SecGuru is used for validating ACLs and more recently RCDC checks forwarding tables at Azure scale. A central technical angle is that we use local contracts and local checks, that can be performed at scale in parallel, and without maintaining global snapshots, to validate global properties of datacenter networks. Specifications leverage declarative encodings of configurations and automated theorem proving for validation. We describe how intent is automatically derived from network architectures and verification is incorporated as prechecks for making changes, live monitoring, and for evolving legacy policies. We document how network verification, grounded in architectural constraints, can be integral to operating a reliable cloud at scale.

Supplementary Material

MP4 File (p200-jayaraman.mp4)

Download
1065.55 MB

References

[1]

Hrishikesh B. Acharya and Mohamed G. Gouda. 2009. Linear-Time Verification of Firewalls. In ICNP. 133--140.

Digital Library

[2]

Algorithmic Security Inc. 2006. Firewall Analyzer: Make your firewall really safe. www.algosec.com (Whitepaper).

[3]

Shrutarshi Basu, Nate Foster, Hossein Hojjat, Paparao Palacharla, Christian Skalka, and Xi Wang. 2017. Life on the Edge: Unraveling Policies into Configurations. In ACM/IEEE Symposium on Architectures for Networking and Communications Systems, ANCS 2017, Beijing, China, May 18-19, 2017. 178--190.

Digital Library

[4]

Ryan Beckett, Aarti Gupta, Ratul Mahajan, and David Walker. 2018. Control plane compression. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, SIGCOMM 2018, Budapest, Hungary, August 20-25, 2018. 476--489.

Digital Library

[5]

Sandeep Bhatt, Cat Okita, and Prasad Rao. 2008. Fast, Cheap, and in Control: Towards Pain-Free Security. In USENIX Systems Administration Conference. 75--90.

Digital Library

[6]

Sandeep Bhatt and Prasad Rao. 2007. Enhancements to the Vantage Firewall Analyzer. Technical Report HPL-2007-154R1. HP Laboratories.

[7]

Chiara Bodei, Pierpaolo Degano, Letterio Galletta, Riccardo Focardi, Mauro Tempesta, and Lorenzo Veronese. 2018. Language-Independent Synthesis of Firewall Policies. In 2018 IEEE European Symposium on Security and Privacy, EuroS&P 2018, London, United Kingdom, April 24-26, 2018. 92--106.

[8]

Randal E. Bryant. 1992. Symbolic Boolean Manipulation with Ordered Binary-Decision Diagrams. ACM Comput. Surv. 24, 3 (1992), 293--318.

Digital Library

[9]

Pavol Cerný, Nate Foster, Nilesh Jagnik, and Jedidiah McClurg. 2016. Optimal Consistent Network Updates in Polynomial Time. In Distributed Computing - 30th International Symposium, DISC 2016, Paris, France, September 27-29, 2016. Proceedings. 114--128.

[10]

Haoxian Chen, Anduo Wang, and Boon Thau Loo. 2018. Towards Example-Guided Network Synthesis. In Proceedings of the 2nd Asia-Pacific Workshop on Networking, APNet 2018, Beijing, China, August 02-03, 2018. 65--71.

Digital Library

[11]

L. de Moura and N. Bjørner. 2008. Z3: An Efficient SMT Solver. In TACAS 08.

Digital Library

[12]

Ahmed El-Hassany, Petar Tsankov, Laurent Vanbever, and Martin T. Vechev. 2017. Network-Wide Configuration Synthesis. In Computer Aided Verification - 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part II. 261--281.

[13]

Ari Fogel, Stanley Fung, Luis Pedrosa, Meg Walraed-Sullivan, Ramesh Govindan, Ratul Mahajan, and Todd D. Millstein. 2015. A General Approach to Network Configuration Analysis. In 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI 15, Oakland, CA, USA, May 4-6, 2015. USENIX Association, 469--483. https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/fogel

Digital Library

[14]

Andrew Gacek, John Backes, Byron Cook, Neha Rungta, Sam Bayless, Catherine Dodge, Carsten Varming, Alan Hu, Bill Kocik, Evgenii Kotelnikov, Jure Kukovec, Sean McLaughlin, Jason Reed, John Sizemore, Mark Stalzer, Preethi Srinivasan, Pavle Subotic, Blake Whaley, Yiwen Wu, and Temesghen Kahsai. 2019. Reachability Analysis for AWS-based Networks. In Computer Aided Verification - 31st International Conference, CAV.

[15]

Sergey Gorinsky and János Tapolcai (Eds.). 2018. Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, SIGCOMM 2018, Budapest, Hungary, August 20-25, 2018. ACM. http://dl.acm.org/citation.cfm?id=3230543

[16]

Mohamed G. Gouda and Alex X. Liu. 2007. Structured firewall design. Computer Networks 51, 4 (2007), 1106--1120.

Digital Library

[17]

Albert Greenberg, James R. Hamilton, Navendu Jain, Srikanth Kandula, Changhoon Kim, Parantap Lahiri, David A. Maltz, Parveen Patel, and Sudipta Sengupta. 2009. VL2: A Scalable and Flexible Data Center Network. In Proceedings of the ACM SIGCOMM 2009 Conference on Data Communication (SIGCOMM '09). ACM, New York, NY, USA, 51--62.

Digital Library

[18]

Swati Gupta, Kristen LeFevre, and Atul Prakash. 2009. SPAN: a unified framework and toolkit for querying heterogeneous access policies. In HotSec. USENIX, 5--5.

Digital Library

[19]

William T. Hallahan, Ennan Zhai, and Ruzica Piskac. 2017. Automated repair by example for firewalls. In 2017 Formal Methods in Computer Aided Design, FMCAD 2017, Vienna, Austria, October 2-6, 2017. 220--229.

Digital Library

[20]

Andrew Helwer. 2018. Z3Prover/FirewallChecker. https://github.com/Z3Prover/FirewallChecker

[21]

Alex Horn, Ali Kheradmand, and Mukul R. Prasad. 2017. Delta-net: Real-time Network Verification Using Atoms. In 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2017, Boston, MA, USA, March 27-29, 2017. 735--749.

Digital Library

[22]

Peyman Kazemian, Michael Chan, Hongyi Zeng, George Varghese, Nick McKeown, and Scott Whyte. 2013. Real Time Network Policy Checking Using Header Space Analysis. In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2013, Lombard, IL, USA, April 2-5, 2013. 99--111.

Digital Library

[23]

Ahmed Khurshid, Xuan Zou, Wenxuan Zhou, Matthew Caesar, and P. Brighten Godfrey. 2013. VeriFlow: Verifying Network-Wide Invariants in Real Time. In Presented as part of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13). USENIX, Lombard, IL, 15--27. https://www.usenix.org/conference/nsdi13/technical-sessions/presentation/khurshid

Digital Library

[24]

Petr Lapukhov, Ariff Premji, and Jon Mitchell. 2016. Use of BGP for Routing in Large-Scale Data Centers. RFC 7938.

[25]

Andrew Lerner. 2017. Intent-based Networking. https://blogs.gartner.com/andrew-lerner/2017/02/07/intent-based-networking

[26]

Alex X. Liu, Mohamed G. Gouda, Huibo H. Ma, and Anne H. H. Ngu. 2004. Firewall Queries. In International Conference On Principles Of Distributed Systems. 197--212.

Digital Library

[27]

Hongqiang Harry Liu, Yibo Zhu, Jitu Padhye, Jiaxin Cao, Sri Tallapragada, Nuno P. Lopes, Andrey Rybalchenko, Guohan Lu, and Lihua Yuan. 2017. CrystalNet: Faithfully Emulating Large Production Networks. In Proceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China, October 28-31, 2017. ACM, 599--613.

Digital Library

[28]

Jed Liu, William Hallahan, Cole Schlesinger, Milad Sharif, Jeongkeun Lee, Robert Soulé, Han Wang, Calin Cascaval, Nick McKeown, and Nate Foster. 2018. p4v: practical verification for programmable data planes, See {15}, 490--503.

Digital Library

[29]

Nuno Lopes. {n. d.}. Cloud Topology Generator. http://web.ist.utl.pt/nuno.lopes/netverif/netverif-scripts-0.2.tar.bz2

[30]

Nuno P. Lopes, Nikolaj Bjórner, Patrice Godefroid, Karthick Jayaraman, and George Varghese.2015. Checking Beliefs in Dynamic Networks. In 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI 15, Oakland, CA, USA, May 4-6, 2015. 499--512.

Digital Library

[31]

Nuno P. Lopes and Andrey Rybalchenko. 2019. Fast BGP Simulation of Large Datacenters. In Verification, Model Checking, and Abstract Interpretation - 20th International Conference, VMCAI 2019, Cascais, Portugal, January 13-15, 2019, Proceedings (Lecture Notes in Computer Science), Constantin Enea and Ruzica Piskac (Eds.), Vol. 11388. Springer, 386--408.

[32]

Haohui Mai, Ahmed Khurshid, Rachit Agarwal, Matthew Caesar, Brighten Godfrey, and Samuel Talmadge King. 2011. Debugging the data plane with anteater. In Proceedings of the ACM SIGCOMM 2011 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, Toronto, ON, Canada, August 15-19, 2011. 290--301.

Digital Library

[33]

Robert M. Marmorstein and Phil Kearns. 2005. An Open Source Solution for Testing NAT'd and Nested iptables Firewalls. In LISA. 103--112.

Digital Library

[34]

Alain J. Mayer, Avishai Wool, and Elisha Ziskind. 2000. Fang: A Firewall Analysis Engine. In IEEE Symposium on Security and Privacy. 177--187.

Digital Library

[35]

Jedidiah McClurg, Hossein Hojjat, Pavol Cerný, and Nate Foster. 2015. Efficient synthesis of network updates. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, USA, June 15-17, 2015. 196--207.

Digital Library

[36]

Sanjai Narain, Gary Levin, Sharad Malik, and Vikram Kaul. 2008. Declarative Infrastructure Configuration Synthesis and Debugging. J. Netw. Syst. Manage. 16, 3 (Sept. 2008), 235--258.

Digital Library

[37]

Sanjai Narain, Rajesh Talpade, and Gary Levin. 2009. Network Configuration Validation. In Guide to Reliable Internet Services and Application, Charles Kalmanek, Richard Yang, and Sudip Misra (Eds.).

[38]

Timothy Nelson, Christopher Barratt, Daniel J. Dougherty, Kathi Fisler, and Shriram Krishnamurthi. 2010. The margrave tool for firewall analysis. In LISA. USENIX Association, Berkeley, CA, USA, 1--8.

Digital Library

[39]

Aurojit Panda, Ori Lahav, Katerina J. Argyraki, Mooly Sagiv, and Scott Shenker. 2017. Verifying Reachability in Networks with Mutable Datapaths. In 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2017, Boston, MA, USA, March 27-29, 2017, Aditya Akella and Jon Howell (Eds.). USENIX Association, 699--718. https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/panda-mutable-datapaths

Digital Library

[40]

Ruzica Piskac. 2018. New Applications of Software Synthesis: Verification of Configuration Files and Firewall Repair. In Static Analysis - 25th International Symposium, SAS 2018, Freiburg, Germany, August 29-31, 2018, Proceedings. 71--76.

[41]

Gordon D. Plotkin, Nikolaj Bjørner, Nuno P. Lopes, Andrey Rybalchenko, and George Varghese. 2016. Scaling network verification using symmetry and surgery. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016. 69--83.

Digital Library

[42]

Mark Reitblatt, Nate Foster, Jennifer Rexford, and David Walker. 2011. Consistent updates for software-defined networks: change you can believe in!. In Tenth ACM Workshop on Hot Topics in Networks (HotNets-X), HOTNETS '11, Cambridge, MA, USA - November 14 - 15, 2011. 7.

Digital Library

[43]

Radu Stoenescu, Dragos Dumitrescu, Matei Popovici, Lorina Negreanu, and Costin Raiciu. 2018. Debugging P4 programs with vera, See {15}, 518--532.

Digital Library

[44]

George Varghese. 2015. Technical Perspective: Treating Networks Like Programs. Commun. ACM 58, 11 (Oct. 2015), 112--112.

Digital Library

[45]

Avishai Wool. 2001. Architecting the Lumeta Firewall Analyzer. In Proceedings of the 10th Conference on USENIX Security Symposium - Volume 10 (SSYM'01). USENIX Association, Berkeley, CA, USA, Article 7.

Digital Library

[46]

Geoffrey G. Xie, Jibin Zhan, David A. Maitz, Hui Zhang, Albert G. Greenberg, Gísli Hjálmtýsson, and Jennifer Rexford. 2005. On static reachability analysis of IP networks. In INFOCOM 2005. 24th Annual Joint Conference of the IEEE Computer and Communications Societies, 13-17 March 2005, Miami, FL, USA. 2170--2183.

[47]

Hongkun Yang and Simon S. Lam. 2016. Real-Time Verification of Network Properties Using Atomic Predicates. IEEE/ACM Trans. Netw. 24, 2 (2016), 887--900.

Digital Library

[48]

N. Ben Souayeh Ben Youssef and Adel Bouhoula. 2010. Automatic Conformance Verification of Distributed Firewalls to Security Requirements. In IEEE ICSC. 834--841.

Digital Library

[49]

Arseniy Zaostrovnykh, Solal Pirelli, Luis Pedrosa, Katerina J. Argyraki, and George Candea. 2017. A Formally Verified NAT. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, SIGCOMM 2017, Los Angeles, CA, USA, August 21-25, 2017. 141--154.

Digital Library

[50]

Hongyi Zeng, Shidong Zhang, Fei Ye, Vimalkumar Jeyakumar, Mickey Ju, Junda Liu, Nick McKeown, and Amin Vahdat. 2014. Libra: Divide and Conquer to Verify Forwarding Tables in Huge Networks. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). USENIX Association, Seattle, WA, 87--99. https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/zeng

Digital Library

[51]

Shuyuan Zhang, Abdurrahman Mahmoud, Sharad Malik, and Sanjai Narain. 2012. Verification and synthesis of firewalls using SAT and QBF. In ICNP. 1--6.

Digital Library

Cited By

Li RYe FYuan YYang RTian BGuo TWu HZhu XGuan ZMa QZeng XXu CCai DZhai EVanbever LZhang I(2024)Reasoning about network traffic load property at production scaleProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691884(1063-1081)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691884
Gao ZAbhashkumar ASun ZJiang WWang YVanbever LZhang I(2024)CrescentProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691883(1045-1062)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691883
Sherwood RShi JZhang YSpring NSundaresan SBagga JPeddi PKukkadapu VShrivastava RKR MPatil PGopu SVaradan VShi EMorsy HBu YYang RJönsson RZhang WArredondo JSaha DChoi SVanbever LZhang I(2024)NetcastleProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691880(993-1008)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691880
Show More Cited By

Index Terms

Validating datacenters at scale

Recommendations

Formal verification of cloud systems elasticity

Cloud computing is a new delivery model based on a simple idea, which consists of providing a set of virtualised resources as on demand services that users can acquire according to their needs in an elastic way. In the context of preserving cloud ...
Highly reliable architecture using the 80/20 rule in cloud computing datacenters

Cloud datacenters host hundreds of thousands of physical servers that offer computing resources for executing customer jobs. While the failures of these physical machines are considered normal rather than exceptional, in large-scale distributed systems ...
Formal verification approaches and standards in the cloud computing

Providing a summary of the current challenges related to the formal verification approaches in the cloud computing.Presenting a Systematic Literature Review (SLR) method for the formal verification approaches in the cloud computing topics.Discussing the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '19: Proceedings of the ACM Special Interest Group on Data Communication

August 2019

526 pages

ISBN:9781450359566

DOI:10.1145/3341302

General Chairs:
Jianping Wu
Tsinghua University, China
,
Wendy Hall
University of Southampton, UK

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCOMM '19

Sponsor:

SIGCOMM

SIGCOMM '19: ACM SIGCOMM 2019 Conference

August 19 - 23, 2019

Beijing, China

Acceptance Rates

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
2,341
Total Downloads

Downloads (Last 12 months)153
Downloads (Last 6 weeks)11

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li RYe FYuan YYang RTian BGuo TWu HZhu XGuan ZMa QZeng XXu CCai DZhai EVanbever LZhang I(2024)Reasoning about network traffic load property at production scaleProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691884(1063-1081)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691884
Gao ZAbhashkumar ASun ZJiang WWang YVanbever LZhang I(2024)CrescentProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691883(1045-1062)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691883
Sherwood RShi JZhang YSpring NSundaresan SBagga JPeddi PKukkadapu VShrivastava RKR MPatil PGopu SVaradan VShi EMorsy HBu YYang RJönsson RZhang WArredondo JSaha DChoi SVanbever LZhang I(2024)NetcastleProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691880(993-1008)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691880
Karimi PPirelli SKakarla SBeckett RSegarra SLi BNamyar PArzani B(2024)Towards Safer Heuristics With XPlainProceedings of the 23rd ACM Workshop on Hot Topics in Networks10.1145/3696348.3696884(68-76)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3696348.3696884
Wen SAbhashkumar AZhao CJiang W(2024)Scaling Data Plane Verification via ParallelizationProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663420(81-87)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663420
Kakarla SYan FBeckett R(2024)Diffy: Data-Driven Bug Finding for ConfigurationsProceedings of the ACM on Programming Languages10.1145/36563858:PLDI(199-222)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656385
Ding RLiu XYang SHuang QXie BSun RZhang ZCui BSekar VYu MSeneviratne AVeitch D(2024)RD-Probe: Scalable Monitoring With Sufficient Coverage In Complex Datacenter NetworksProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672256(258-273)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672256
Li RYuan YYe FLiu MYang RYu YGuo TMa QZeng XXu CCai DZhai ESekar VYu MSeneviratne AVeitch D(2024)A General and Efficient Approach to Verifying Traffic Load Properties under Arbitrary k FailuresProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672246(228-243)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672246
Xu XYuan YKincaid ZKrishnamurthy AMahajan RWalker DZhai ESekar VYu MSeneviratne AVeitch D(2024)Relational Network VerificationProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672238(213-227)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672238
Singh ASharma SGumaste A(2024)VERCEL: Verification and Rectification of Configuration Errors With Least SquaresIEEE/ACM Transactions on Networking10.1109/TNET.2024.342203532:6(4600-4614)Online publication date: Dec-2024
https://doi.org/10.1109/TNET.2024.3422035
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents