research-article

Automatically testing string solvers

Authors:

Alexandra Bugariu,

Peter MüllerAuthors Info & Claims

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Pages 1459 - 1470

https://doi.org/10.1145/3377811.3380398

Published: 01 October 2020 Publication History

Abstract

SMT solvers are at the basis of many applications, such as program verification, program synthesis, and test case generation. For all these applications to provide reliable results, SMT solvers must answer queries correctly. However, since they are complex, highly-optimized software systems, ensuring their correctness is challenging. In particular, state-of-the-art testing techniques do not reliably detect when an SMT solver is unsound.

In this paper, we present an automatic approach for generating test cases that reveal soundness errors in the implementations of string solvers, as well as potential completeness and performance issues. We synthesize input formulas that are satisfiable or unsatisfiable by construction and use this ground truth as test oracle. We automatically apply satisfiability-preserving transformations to generate increasingly-complex formulas, which allows us to detect many errors with simple inputs and, thus, facilitates debugging.

The experimental evaluation shows that our technique effectively reveals bugs in the implementation of widely-used SMT solvers and applies also to other types of solvers, such as automata-based solvers. We focus on strings here, but our approach carries over to other theories and their combinations.

References

[1]

[n.d.]. CVC4 Documentation for the String Theory. http://cvc4.cs.stanford.edu/wiki/Strings.

[2]

[n.d.]. CVC4 Regression Test Suite. https://github.com/CVC4/CVC4/tree/master/test/regress.

[3]

[n.d.]. MT-ABC Tested Version. https://github.com/vlab-cs-ucsb/ABC/commit/86b00141fddd183de7b9ae5c92c240e19dda1950.

[4]

[n.d.]. SMT-COMP. https://smt-comp.github.io.

[5]

[n.d.]. SMT-LIB Unicode Strings Theory. http://smtlib.cs.uiowa.edu/theories-UnicodeStrings.shtml/.

[6]

[n.d.]. StringFuzz Test Suite. http://stringfuzz.dmitryblotsky.com/problems/.

[7]

[n.d.]. Z3 SMT Solver. https://github.com/Z3Prover/z3/.

[8]

[n.d.]. Z3 Test Suite. https://github.com/Z3Prover/z3/tree/master/src/test.

[9]

Cyrille Artho, Armin Biere, and Martina Seidl. 2013. Model-Based Testing for Verification Back-Ends. In Tests and Proofs, Margus Veanes and Luca Viganò (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 39--55.

[10]

Abdulbaki Aydin, Lucas Bang, and Tevfik Bultan. 2015. Automata-Based Model Counting for String Constraints. In Computer Aided Verification, Daniel Kroening and Corina S. Păsăreanu (Eds.). Springer International Publishing, Cham, 255--272.

[11]

Abdulbaki Aydin, William Eiers, Lucas Bang, Tegan Brennan, Miroslav Gavrilov, Tevfik Bultan, and Fang Yu. 2018. Parameterized Model Counting for String and Numeric Constraints. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018). ACM, New York, NY, USA, 400--410.

Digital Library

[12]

Clark Barrett, Pascal Fontaine, and Cesare Tinelli. 2017. The SMT-LIB Standard: Version 2.6. Technical Report. Department of Computer Science, The University of Iowa. Available at www.SMT-LIB.org.

[13]

Murphy Berzish, Vijay Ganesh, and Yunhui Zheng. 2017. Z3str3: A String Solver with Theory-aware Heuristics. In 2017 Formal Methods in Computer Aided Design (FMCAD). 55--59.

[14]

Armin Biere, Marijn Heule, Hans van Maaren, and Toby Walsh. 2009. Handbook of Satisfiability: Volume 185 Frontiers in Artificial Intelligence and Applications. IOS Press, Amsterdam, The Netherlands, The Netherlands.

[15]

Nikolaj Bjørner, Vijay Ganesh, Raphaël Michel, and Margus Veanes. 2012. An SMT-LIB Format for Sequences and Regular Expressions. Strings (01 2012).

[16]

Dmitry Blotsky, Federico Mora, Murphy Berzish, Yunhui Zheng, Ifaz Kabir, and Vijay Ganesh. 2018. StringFuzz: A Fuzzer for String Solvers. In Computer Aided Verification, Hana Chockler and Georg Weissenbacher (Eds.). Springer International Publishing, Cham, 45--51.

[17]

Sascha Böhme and Tjark Weber. 2010. Fast LCF-Style Proof Reconstruction for Z3. In Interactive Theorem Proving, Matt Kaufmann and Lawrence C. Paulson (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 179--194.

[18]

Robert Brummayer and Armin Biere. 2009. Fuzzing and delta-debugging SMT solvers. ACM International Conference Proceeding Series (01 2009), 1--5.

Digital Library

[19]

Robert Brummayer, Florian Lonsing, and Armin Biere. 2010. Automated Testing and Debugging of SAT and QBF Solvers. In Theory and Applications of Satisfiability Testing - SAT 2010, Offer Strichman and Stefan Szeider (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 44--57.

Digital Library

[20]

Leonardo de Moura and Nikolaj Bjørner. 2007. Efficient E-Matching for SMT Solvers. In Automated Deduction - CADE-21, Frank Pfenning (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 183--198.

[21]

David Detlefs, Greg Nelson, and James B. Saxe. 2005. Simplify: A Theorem Prover for Program Checking. J. ACM 52, 3 (May 2005), 365--473.

Digital Library

[22]

Jean-Christophe Filliâtre and Andrei Paskevich. 2013. Why3---Where Programs Meet Provers. In Programming Languages and Systems (ESOP) (Lecture Notes in Computer Science), Matthias Felleisen and Philippa Gardner (Eds.), Vol. 7792. Springer, 125--128.

[23]

Jonathan Ford and Natarajan Shankar. 2002. Formal Verification of a Combination Decision Procedure. In Proceedings of the 18th International Conference on Automated Deduction (CADE-18). Springer-Verlag, Berlin, Heidelberg, 347--362. http://dl.acm.org/citation.cfm?id=648238.751562

Digital Library

[24]

Stéphane Lescuyer and Sylvain Conchon. 2008. A Reflexive Formalization of a SAT Solver in Coq. In In Proceedings of TPHOLs.

[25]

Tianyi Liang, Andrew Reynolds, Nestan Tsiskaridze, Cesare Tinelli, Clark Barrett, and Morgan Deters. 2016. An Efficient SMT Solver for String Constraints. Form. Methods Syst. Des. 48, 3 (June 2016), 206--234.

Digital Library

[26]

Loi Luu, Shweta Shinde, Prateek Saxena, and Brian Demsky. 2014. A Model Counter for Constraints over Unbounded Strings. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (Edinburgh, United Kingdom) (PLDI '14). ACM, New York, NY, USA, 565--576.

Digital Library

[27]

Filip Mari. 2010. Formal Verification of a Modern SAT Solver by Shallow Embedding into Isabelle/HOL. Theor. Comput. Sci. 411, 50 (Nov. 2010), 4333--4356.

Digital Library

[28]

William M. McKeeman. 1998. Differential Testing for Software. DIGITAL TECHNICAL JOURNAL 10, 1 (1998), 100--107.

[29]

Tukaram Muske and Alexander Serebrenik. 2016. Survey of Approaches for Handling Static Analysis Alarms. In 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM). 157--166.

[30]

Aina Niemetz, Mathias Preiner, and Armin Biere. 2017. Model-Based API Testing for SMT Solvers. In Proceedings of the 15th International Workshop on Satisfiability Modulo Theories, SMT 2017), affiliated with the 29th International Conference on Computer Aided Verification, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Martin Brain and Liana Hadarean (Eds.). 10 pages.

[31]

Andrew Reynolds, Haniel Barbosa, and Pascal Fontaine. 2018. Revisiting Enumerative Instantiation. In Tools and Algorithms for the Construction and Analysis of Systems, Dirk Beyer and Marieke Huisman (Eds.). Springer International Publishing, Cham, 112--131.

[32]

Aaron Stump, Duckki Oe, Andrew Reynolds, Liana Hadarean, and Cesare Tinelli. 2013. SMT Proof Checking Using a Logical Framework. Form. Methods Syst. Des. 42, 1 (Feb. 2013), 91--118.

Digital Library

[33]

Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and Isolating Failure-Inducing Input. IEEE Trans. Softw. Eng. 28, 2 (Feb. 2002), 183--200.

Digital Library

[34]

Lintao Zhang and Sharad Malik. 2003. Validating SAT solvers using an independent resolution-based checker: practical implementations and other applications. In 2003 Design, Automation and Test in Europe Conference and Exhibition. 880--885.

[35]

Yunhui Zheng, Xiangyu Zhang, and Vijay Ganesh. 2013. Z3-str: A Z3-based String Solver for Web Application Analysis. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (Saint Petersburg, Russia) (ESEC/FSE 2013). ACM, New York, NY, USA, 114--124.

Digital Library

Cited By

Winterer DSu Z(2024)Validating SMT Solvers for Correctness and Performance via Grammar-Based EnumerationProceedings of the ACM on Programming Languages10.1145/36897958:OOPSLA2(2378-2401)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689795
Zhang CSu Z(2024)SMT2Test: From SMT Formulas to Effective Test CasesProceedings of the ACM on Programming Languages10.1145/36897198:OOPSLA2(222-245)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689719
Jha AMonahan RWu HCombemale BWimmer MChechik MEgyed A(2024)Verifying UML Models Annotated with OCL StringsProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3687822(1106-1110)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3652620.3687822
Show More Cited By

Index Terms

Automatically testing string solvers
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Fuzzing SMT solvers via two-dimensional input space exploration
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

Satisfiability Modulo Theories (SMT) solvers serve as the core engine of many techniques, such as symbolic execution. Therefore, ensuring the robustness and correctness of SMT solvers is critical. While fuzzing is an efficient and effective method for ...
Generative type-aware mutation for testing SMT solvers

We propose Generative Type-Aware Mutation, an effective approach for testing SMT solvers. The key idea is to realize generation through the mutation of expressions rooted with parametric operators from the SMT-LIB specification. Generative Type-Aware ...
On the unusual effectiveness of type-aware operator mutations for testing SMT solvers

We propose type-aware operator mutation, a simple, but unusually effective approach for testing SMT solvers. The key idea is to mutate operators of conforming types within the seed formulas to generate well-typed mutant formulas. These mutant formulas ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

June 2020

1640 pages

ISBN:9781450371216

DOI:10.1145/3377811

General Chairs:
Gregg Rothermel
North Carolina State University
,
Doo-Hwan Bae
KAIST, South Korea

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Conference

ICSE '20

Sponsor:

SIGSOFT

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, South Korea

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
258
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)16

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Winterer DSu Z(2024)Validating SMT Solvers for Correctness and Performance via Grammar-Based EnumerationProceedings of the ACM on Programming Languages10.1145/36897958:OOPSLA2(2378-2401)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689795
Zhang CSu Z(2024)SMT2Test: From SMT Formulas to Effective Test CasesProceedings of the ACM on Programming Languages10.1145/36897198:OOPSLA2(222-245)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689719
Jha AMonahan RWu HCombemale BWimmer MChechik MEgyed A(2024)Verifying UML Models Annotated with OCL StringsProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3687822(1106-1110)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3652620.3687822
Carvalho LDegiovanni RCordy MAguirre NLe Traon YPapadakis MRoychoudhury APaiva AAbreu RStorey M(2024)SpecBCFuzz: Fuzzing LTL Solvers with Boundary ConditionsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639087(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639087
Mikek BZhang QChandra SBlincoe KTonella P(2023)Speeding up SMT Solving via Compiler OptimizationProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616357(1177-1189)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616357
Mansur MWüstholz VChristakis MJust RFraser G(2023)Dependency-Aware Metamorphic Testing of Datalog EnginesProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598052(236-247)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598052
Kim JSo SOh HGrundy JPollock LPenta M(2023)DIVER: Oracle-Guided SMT Solver Testing with Unrestricted Random MutationsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00187(2224-2236)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00187
Zhao XQu HXu JLi XLv WWang G(2023)A systematic review of fuzzingSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-023-09306-228:6(5493-5522)Online publication date: 31-Oct-2023
https://dl.acm.org/doi/10.1007/s00500-023-09306-2
Li PMeng WLu KRoychoudhury ACadar CKim M(2022)SEDiff: scope-aware differential fuzzing to test internal function models in symbolic executionProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549080(57-69)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549080
Kan SLin ARümmer PSchrader MPopescu AZdancewic S(2022)CertiStr: a certified string solverProceedings of the 11th ACM SIGPLAN International Conference on Certified Programs and Proofs10.1145/3497775.3503691(210-224)Online publication date: 17-Jan-2022
https://dl.acm.org/doi/10.1145/3497775.3503691
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents