skip to main content
10.5555/3691938.3691973guideproceedingsArticle/Chapter ViewAbstractPublication PagesosdiConference Proceedingsconference-collections
research-article

Anvil: verifying liveness of cluster management controllers

Published: 07 January 2025 Publication History

Abstract

Modern clouds depend crucially on an extensible ecosystem of thousands of controllers, each managing critical systems (e.g., a ZooKeeper cluster). A controller continuously reconciles the current state of the system to a desired state according to a declarative description. However, controllers have bugs that make them never achieve the desired state, due to concurrency, asynchrony, and failures; there are cases where after an inopportune failure, a controller can make no further progress. Formal verification is promising for avoiding bugs in distributed systems, but most work so far focused on safety, whereas reconciliation is fundamentally not a safety property.
This paper develops the first tool to apply formal verification to the problem of controller correctness, with a general specification we call eventually stable reconciliation, written as a concise temporal logic liveness property. We present Anvil, a framework for developing controller implementations in Rust and verifying that the controllers correctly implement eventually stable reconciliation. We use Anvil to verify three Kubernetes controllers for managing ZooKeeper, RabbitMQ, and FluentBit, which can readily be deployed in Kubernetes platforms and are comparable in terms of features and performance to widely used unverified controllers.

References

[1]
etcd. https://etcd.io/.
[2]
Flink controller state machine. https://github.com/lyft/flinkk8soperator/blob/master/docs/state_machine.md, 2020.
[3]
Alloy 6. https://alloytools.org/alloy6.html, 2021.
[4]
Controllers and Reconciliation. https://cluster-api.sigs.k8s.io/developer/providers/implementersguide/controllers_and_reconciliation.html, 2021.
[5]
CustomResourceDefinition Validation Rules. https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/, 2022.
[6]
Spark controller state machine. https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/pkg/controller/sparkapplication/controller.go#L485-L520, 2022.
[7]
Annotations. https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/, 2023.
[8]
Controllers. https://kubernetes.io/docs/concepts/architecture/controller/, 2023.
[9]
FluentBit. https://fluentbit.io/, 2023.
[10]
Ironfleet liveness lemma. https://github.com/microsoft/Ironclad/blob/2fe4dcdc323b92e93f759cc3e373521366b7f691/ironfleet/src/Dafny/Distributed/Protocol/LiveSHT/LivenessProof/LivenessProof.i.dfy#L31, 2023.
[11]
kube-rs/kube: Rust Kubernetes client and controller runtime. https://github.com/kube-rs/kube, 2023.
[12]
Kubernetes. https://kubernetes.io/, 2023.
[13]
Kubernetes API Concepts: Updates to existing resources. https://kubernetes.io/docs/reference/using-api/api-concepts/#patch-and-apply, 2023.
[14]
Kubernetes ConfigMaps. https://kubernetes.io/docs/concepts/configuration/configmap/, 2023.
[15]
Kubernetes DaemonSet. https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/, 2023.
[16]
Kubernetes Garbage Collection. https://kubernetes.io/docs/concepts/architecture/garbage-collection/, 2023.
[17]
Kubernetes Service. https://kubernetes.io/docs/concepts/services-networking/service/, 2023.
[18]
Kubernetes StatefulSet. https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/, 2023.
[19]
Official FluentBit controller. https://github.com/fluent/fluent-operator, 2023.
[20]
Official RabbitMQ controller. https://github.com/rabbitmq/cluster-operator, 2023.
[21]
Owners and Dependents. https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/, 2023.
[22]
Pravega. https://cncf.pravega.io/, 2023.
[23]
Pravega ZooKeeper controller. https://github.com/pravega/zookeeper-operator, 2023.
[24]
RabbitMQ. https://www.rabbitmq.com/, 2023.
[25]
RabbitMQ: Scale down. https://github.com/rabbitmq/cluster-operator/issues/223, 2023.
[26]
Stateful set never gets updated because zk node is missing. https://github.com/pravega/zookeeperoperator/issues/569, 2023.
[27]
BAUMANN, P., MAJUMDAR, R., THINNIYAM, R. S., AND ZETZSCHE, G. Context-Bounded Verification of Liveness Properties for Multithreaded Shared-Memory Programs. In Proceedings of the 48th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL'21) (Jan. 2021).
[28]
BERNER, C. Scaling kubernetes to 2,500 nodes. https://openai.com/research/scaling-kubernetes-to-2500-nodes, Jan. 2023.
[29]
BRUNEL, J., CHEMOUIL, D., CUNHA, A., AND MACEDO, N. The Electrum Analyzer: Model Checking Relational First-Order Temporal Specifications. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE'18) (Sept. 2018).
[30]
BURNS, B., GRANT, B., OPPENHEIMER, D., BREWER, E., AND WILKES, J. Borg, Omega, and Kubernetes. Communications of the ACM 59, 5 (May 2016), 50-57.
[31]
CHAJED, T., TASSAROTTI, J., KAASHOEK, M. F., AND ZELDOVICH, N. Verifying concurrent, crash-safe systems with Perennial. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP'19) (Oct. 2019).
[32]
CHAJED, T., TASSAROTTI, J., THENG, M., JUNG, R., KAASHOEK, M. F., AND ZELDOVICH, N. GoJournal: a verified, concurrent, crash-safe journaling system. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI'21) (July 2021).
[33]
CHAJED, T., TASSAROTTI, J., THENG, M., KAASHOEK, M. F., AND ZELDOVICH, N. Verifying the DaisyNFS concurrent and crash-safe file system with sequential reasoning. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI'22) (July 2022).
[34]
CHANG, Y.-S., JUNG, R., SHARMA, U., TASSAROTTI, J., KAASHOEK, M. F., AND ZELDOVICH, N. Verifying vMVCC, a high-performance transaction library using multiversion concurrency control. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI'23) (July 2023).
[35]
CHEKRYGIN, I. Keep the Space Shuttle Flying: Writing Robust Operators. https://youtu.be/uf97lOApOv8?t=1457, May 2019.
[36]
CHEN, H., CHAJED, T., KONRADI, A., WANG, S., ILERI, A., CHLIPALA, A., KAASHOEK, M. F., AND ZELDOVICH, N. Verifying a High-Performance Crash-Safe File System Using a Tree Specification. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP'17) (Oct. 2017).
[37]
CHEN, H., ZIEGLER, D., CHAJED, T., CHLIPALA, A., KAASHOEK, M. F., AND ZELDOVICH, N. Using Crash Hoare Logic for Certifying the FSCQ File System. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP'15) (Oct. 2015).
[38]
COOK, B., PODELSKI, A., AND RYBALCHENKO, A. Proving Program Termination. Communications of the ACM 54, 5 (May 2011), 88-98.
[39]
DE MOURA, L., AND BJORNER, N. Z3: An Efficient SMT Solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'08/ETAPS'08) (Mar. 2008).
[40]
DWORK, C., LYNCH, N., AND STOCKMEYER, L. Consensus in the Presence of Partial Synchrony. Journal of the ACM 35, 2 (Apr. 1988), 288-323.
[41]
FARZAN, A., KINCAID, Z., AND PODELSKI, A. Proving Liveness of Parameterized Programs. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS'16) (July 2016).
[42]
GOEL, A., AND SAKALLAH, K. On Symmetry and Quantification: A New Approach to Verify Distributed Protocols. In Proceedings of the 13th NASA Formal Methods Symposium (NFM'21) (May 2021).
[43]
GOTSMAN, A., COOK, B., PARKINSON, M., AND VAFEIADIS, V. Proving That Non-Blocking Algorithms Don't Block. In Proceedings of the 36th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL'09) (Jan. 2009).
[44]
GU, J. T., SUN, X., ZHANG, W., JIANG, Y., WANG, C., VAZIRI, M., LEGUNSEN, O., AND XU, T. Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP'23) (Oct. 2023).
[45]
GUILLOUX, S. Writing a Kubernetes Operator: the Hard Parts. https://youtu.be/wMqzAOp15wo?t=411, Nov. 2019.
[46]
HAASE, S. How an Operator Becomes the Hero of the Edge. In OperatorCon (May 2019).
[47]
HALL, C. AWS, Google, Microsoft, Red Hat's New Registry to Act as Clearing House for Kubernetes Operators. https://www.datacenterknowledge.com/open-source/awsgoogle-microsoft-red-hats-new-registry-actclearing-house-kubernetes-operators, Mar. 2019.
[48]
HANCE, T., HEULE, M., MARTINS, R., AND PARNO, B. Finding Invariants of Distributed Systems: It's a Small (Enough) World After All. In Proceedings of the 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI'21) (Apr. 2021).
[49]
HANCE, T., LATTUADA, A., HAWBLITZEL, C., HOWELL, J., JOHNSON, R., AND PARNO, B. Storage Systems are Distributed Systems (So Verify Them That Way!). In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI'20) (Nov. 2020).
[50]
HANCE, T., ZHOU, Y., LATTUADA, A., ACHERMANN, R., CONWAY, A., STUTSMAN, R., ZELLWEGER, G., HAWBLITZEL, C., HOWELL, J., AND PARNO, B. Sharding the State Machine: Automated Modular Reasoning for Complex Concurrent Systems. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI'23) (July 2023).
[51]
HAWBLITZEL, C., HOWELL, J., KAPRITSOS, M., LORCH, J. R., PARNO, B., ROBERTS, M. L., SETTY, S., AND ZILL, B. IronFleet: Proving Practical Distributed Systems Correct. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP'15) (Oct. 2015).
[52]
HOARE, C. A. R. An axiomatic basis for computer programming. Communications of the ACM 12 (1969).
[53]
HOCKIN, T. Kubernetes: Edge vs. Level Triggered Logic. https://speakerdeck.com/thockin/edge-vslevel-triggered-logic, June 2017.
[54]
HOWARD, J. Building Better Controllers. https://www.youtube.com/watch?v=GKPBQDJ2Hjk&t=160s, Nov. 2023.
[55]
KILLIAN, C., ANDERSON, J. W., JHALA, R., AND VAHDAT, A. Life, Death, and the Critical Transition: Finding Liveness Bugs in Systems Code. In Proceedings of the 4th USENIX Symposium on Networked Systems Design and Implementation (NSDI'07) (Apr. 2007).
[56]
KLEIN, G., ELPHINSTONE, K., HEISER, G., ANDRONICK, J., COCK, D., DERRIN, P., ELKADUWE, D., ENGELHARDT, K., KOLANSKI, R., NORRISH, M., SEWELL, T., TUCH, H., AND WINWOOD, S. SeL4: Formal Verification of an OS Kernel. In Proceedings of the 22nd ACM Symposium on Operating Systems Principles (SOSP'09) (Oct. 2009).
[57]
LAGRESLE, M. Moving to Kubernetes: the Bad and the Ugly. https://youtu.be/MoIdU0J0f0E?t=263, June 2019.
[58]
LAMPORT, L. The Temporal Logic of Actions. ACM Transactions on Programming Languages and Systems 16, 3 (May 1994), 872-923.
[59]
LAMPORT, L. Specifying Systems: The TLA+ Languange and Tools for Hardware and Software Engineers. Addison-Wesley, 2002.
[60]
LANDER, R. Kubernetes Operators: Should You Use Them? https://tanzu.vmware.com/developer/blog/kubernetes-operators-should-you-use-them/, July 2021.
[61]
LATTUADA, A., HANCE, T., CHO, C., BRUN, M., SUBASINGHE, I., ZHOU, Y., HOWELL, J., PARNO, B., AND HAWBLITZEL, C. Verus: Verifying Rust Programs Using Linear Ghost Types. In Proceedings of 2023 ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA'23) (Apr. 2023).
[62]
LEINO, K. R. M. Dafny: An Automatic Program Verifier for Functional Correctness. In Proceedings of the 17th International Conference on Logic for Programming, Artificial Intelligence, and Reasoning (LPAR'10) (Oct. 2010).
[63]
LEROY, X. Formal Verification of a Realistic Compiler. Communications of the ACM 52, 7 (July 2009), 107-115.
[64]
LIU, B., KHERADMAND, A., CAESAR, M., AND GODFREY, P. B. Towards Verified Self-Driving Infrastructure. In Proceedings of the 19th ACM Workshop on Hot Topics in Networks (HotNets'20) (Nov. 2020).
[65]
LIU, B., LIM, G., BECKETT, R., AND GODFREY, P. B. Kivi: Verification for Cluster Management. In Proceedings of the 2024 USENIX Annual Technical Conference (ATC'24) (July 2024).
[66]
LORCH, J. R., CHEN, Y., KAPRITSOS, M., PARNO, B., QADEER, S., SHARMA, U., WILCOX, J. R., AND ZHAO, X. Armada: Low-Effort Verification of High-Performance Concurrent Programs. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'20) (June 2020).
[67]
LOU, C., HUANG, P., AND SMITH, S. Understanding, Detecting and Localizing Partial Failures in Large System Software. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI'20) (Feb. 2020).
[68]
MA, H., AHMAD, H., GOEL, A., GOLDWEBER, E., JEANNIN, J.-B., KAPRITSOS, M., AND KASIKCI, B. Sift: Using Refinement-guided Automation to Verify Complex Distributed Systems. In Proceedings of the 2022 USENIX Annual Technical Conference (ATC'22) (July 2022).
[69]
MA, H., GOEL, A., JEANNIN, J.-B., KAPRITSOS, M., KASIKCI, B., AND SAKALLAH, K. A. I4: Incremental Inference of Inductive Invariants for Verification of Distributed Protocols. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP'19) (Oct. 2019).
[70]
MACEDO, N., BRUNEL, J., CHEMOUIL, D., AND CUNHA, A. Pardinus: A temporal relational model finder. J. Autom. Reason. 66, 4 (2022), 861-904.
[71]
MADHU, C. Preventing Controller Sprawl From Taking Down Your Cluster. https://youtu.be/fu5GXo7jmV0?t=732, Oct. 2022.
[72]
MCMILLAN, K. L., AND PADON, O. Ivy: A Multi-Modal Verification Tool for Distributed Algorithms. In Proceedings of the 32nd International Conference on Computer Aided Verification (CAV'20) (July 2020).
[73]
MELISSARIS, T., NABAR, K., RADUT, R., REHMTULLA, S., SHI, A., CHANDRASHEKAR, S., AND PAPAPANAGIOTOU, I. Elastic Cloud Services: Scaling Snowflake's Control Plane. In Proceedings of the 13th ACM Symposium on Cloud Computing (SOCC'22) (Nov. 2022).
[74]
MOTWANI, S., AND MAHESHWARI, A. Deep Dive Into Writing a Kubernetes Operator: Let's Avoid Data Loss and Down Times. https://www.youtube.com/watch?v= 2NjMHLACvc0&t=737s, Nov. 2023.
[75]
NELSON, L., SIGURBJARNARSON, H., ZHANG, K., JOHNSON, D., BORNHOLT, J., TORLAK, E., AND WANG, X. Hyperkernel: Push-Button Verification of an OS Kernel. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP'17) (Oct. 2017).
[76]
PADON, O., HOENICKE, J., LOSA, G., PODELSKI, A., SAGIV, M., AND SHOHAM, S. Reducing Liveness to Safety in First-Order Logic. In Proceedings of the 45th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL'18) (Jan. 2018).
[77]
PADON, O., HOENICKE, J., MCMILLAN, K. L., PODELSKI, A., SAGIV, M., AND SHOHAM, S. Temporal Prophecy for Proving Temporal Properties of Infinite-State Systems. In Proceedings of the 18th Conference on Formal Methods in Computer-Aided Design (FMCAD'18) (Oct. 2018).
[78]
PADON, O.,MCMILLAN, K. L., PANDA, A., SAGIV,M., AND SHOHAM, S. Ivy: Safety Verification by Interactive Generalization. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'16) (June 2016).
[79]
PADON, O., WILCOX, J. R., KOENIG, J. R., MCMILLAN, K. L., AND AIKEN, A. Induction Duality: Primal-Dual Search for Invariants. In Proceedings of the 49th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL'22) (Jan. 2022).
[80]
PNUELI, A. The Temporal Logic of Programs. In Proceedings of the 18th Annual Symposium on Foundations of Computer Science (Oct. 1977).
[81]
RATIS, P. Lessons Learned using the Operator Pattern to build a Kubernetes Platform. In USENIX SREcon (Oct. 2021).
[82]
SHARMA, U., JUNG, R., TASSAROTTI, J., KAASHOEK, F., AND ZELDOVICH, N. Grove: A Separation-Logic Library for Verifying Distributed Systems. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP'23) (Oct. 2023).
[83]
SIGURBJARNARSON, H., BORNHOLT, J., TORLAK, E., AND WANG, X. Push-Button Verification of File Systems via Crash Refinement. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI'16) (Nov. 2016).
[84]
SOSA, C., AND BHATIA, P. Application management made easier with Kubernetes Operators on GCP Marketplace. https://cloud.google.com/blog/products/ containers-kubernetes/application-managementmade-easier-with-kubernete-operators-on-gcpmarketplace, May 2019.
[85]
SUN, X., LUO, W., GU, J. T., GANESAN, A., ALAGAPPAN, R., GASCH, M., SURESH, L., AND XU, T. Automatic Reliability Testing for Cluster Management Controllers. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI'22) (July 2022).
[86]
SUN, X., SURESH, L., GANESAN, A., ALAGAPPAN, R., GASCH, M., TANG, L., AND XU, T. Reasoning about modern datacenter infrastructures using partial histories. In Proceedings of the 18th Workshop on Hot Topics in Operating Systems (HotOS-XVIII) (May 2021).
[87]
TANG, C., YU, K., VEERARAGHAVAN, K., KALDOR, J., MICHELSON, S., KOOBURAT, T., ANBUDURAI, A., CLARK, M., GOGIA, K., CHENG, L., CHRISTENSEN, B., GARTRELL, A., KHUTORNENKO, M., KULKARNI, S., PAWLOWSKI, M., PELKONEN, T., RODRIGUES, A., TIBREWAL, R., VENKATESAN, V., AND ZHANG, P. Twine: A Unified Cluster Management System for Shared Infrastructure. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI'20) (Nov. 2020).
[88]
TAUBE, M., LOSA, G., MCMILLAN, K. L., PADON, O., SAGIV, M., SHOHAM, S., WILCOX, J. R., AND WOOS, D. Modularity for Decidability of Deductive Verification with Applications to Distributed Systems. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'18) (June 2018).
[89]
VERMA, A., PEDROSA, L., KORUPOLU, M., OPPENHEIMER, D., TUNE, E., AND WILKES, J. Large-Scale Cluster Management at Google with Borg. In Proceedings of the 10th European Conference on Computer Systems (EuroSys'15) (Apr. 2015).
[90]
WILCOX, J. R., WOOS, D., PANCHEKHA, P., TATLOCK, Z., WANG, X., ERNST, M. D., AND ANDERSON, T. Verdi: A Framework for Implementing and Formally Verifying Distributed Systems. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'15) (June 2015).
[91]
YAO, J., TAO, R., GU, R., AND NIEH, J. DuoAI: Fast, Automated Inference of Inductive Invariants for Verifying Distributed Protocols. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI'22) (July 2022).
[92]
YAO, J., TAO, R., GU, R., AND NIEH, J. Mostly Automated Verification of Liveness Properties for Distributed Protocols with Ranking Functions. In Proceedings of the 51st ACM SIGPLAN Symposium on Principles of Programming Languages (POPL'24) (Jan. 2024).
[93]
YAO, J., TAO, R., GU, R., NIEH, J., JANA, S., AND RYAN, G. DistAI: Data-Driven Automated Invariant Learning for Distributed Protocols. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI'21) (July 2021).
[94]
ZOU, M., DING, H., DU, D., FU, M., GU, R., AND CHEN, H. Using Concurrent Relational Logic with Helpers for Verifying the AtomFS File System. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP'19) (Oct. 2019).

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings
OSDI'24: Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation
July 2024
1005 pages
ISBN:978-1-939133-40-3

Sponsors

  • Amazon
  • ROBLOX
  • databricks
  • Microsoft
  • Meta

Publisher

USENIX Association

United States

Publication History

Published: 07 January 2025

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 12 Jan 2025

Other Metrics

Citations

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media