article

SATzilla: portfolio-based algorithm selection for SAT

Authors:

Holger H. Hoos,

Kevin Leyton-BrownAuthors Info & Claims

Journal of Artificial Intelligence Research, Volume 32, Issue 1

Pages 565 - 606

Published: 01 June 2008 Publication History

Abstract

It has been widely observed that there is no single "dominant" SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.

References

[1]

Bacchus, F. (2002a). Enhancing Davis Putnam with extended binary clause reasoning. In Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI'02), pp. 613-619.

Digital Library

[2]

Bacchus, F. (2002b). Exploring the computational tradeoff of more reasoning and less searching. In Proceedings of the Fifth International Conference on Theory and Applications of Satisfiability Testing (SAT'02), pp. 7-16.

[3]

Bacchus, F., & Winter, J. (2003). Effective preprocessing with hyper-resolution and equality reduction. In Proceedings of the Sixth International Conference on Theory and Applications of Satisfiability Testing (SAT'03), pp. 341-355.

[4]

Biere, A. (2007). Picosat version 535. Solver description, SAT competition 2007.

[5]

Biere, A., Cimatti, A., Clarke, E. M., Fujita, M., & Zhu, Y. (1999). Symbolic model checking using SAT procedures instead of BDDs. In Proceedings of Design Automation Conference (DAC'99), pp. 317-320.

Digital Library

[6]

Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.

Digital Library

[7]

Bregman, D. R., & Mitchell, D. G. (2007). The SAT solver MXC, version 0.5. Solver description, SAT competition 2007.

[8]

C. M. Li, W. W., & Zhang, H. (2007). Combining adaptive noise and promising decreasing variables in local search for SAT. Solver description, SAT competition 2007.

[9]

Carchrae, T., & Beck, J. C. (2005). Applying machine learning to low-knowledge control of optimization algorithms. Computational Intelligence, 21(4), 372-387.

[10]

Crawford, J. M., & Baker, A. B. (1994). Experimental results on the application of satisfiability algorithms to scheduling problems. In Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI'94), pp. 1092-1097.

Digital Library

[11]

Davis, M., Logemann, G., & Loveland, D. (1962). A machine program for theorem proving. Communications of the ACM, 5(7), 394-397.

Digital Library

[12]

Davis, M., & Putnam, H. (1960). A computing procedure for quantification theory. Journal of the ACM, 7(1), 201-215.

Digital Library

[13]

Dechter, R., & Rish, I. (1994). Directional resolution: The Davis-Putnam procedure, revisited. In Principles of Knowledge Representation and Reasoning (KR'94), pp. 134-145.

[14]

Dequen, G., & Dubois, O. (2007). kcnfs. Solver description, SAT competition 2007.

[15]

Dubois, O., & Dequen, G. (2001). A backbone-search heuristic for efficient solving of hard 3-SAT formulae. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI'01), pp. 248-253.

Digital Library

[16]

Eén, N., & Söörensson, N. (2003). An extensible SAT-solver. In Proceedings of the Sixth International Conference on Theory and Applications of Satisfiability Testing (SAT'03), pp. 502-518.

[17]

Eén, N., & Söörensson, N. (2006). Minisat v2.0 (beta). Solver description, SAT Race 2006.

[18]

Gagliolo, M., & Schmidhuber, J. (2006a). Impact of censored sampling on the performance of restart strategies. In Twelfth Internatioal Conference on Principles and Practice of Constraint Programming (CP'06), pp. 167-181.

Digital Library

[19]

Gagliolo, M., & Schmidhuber, J. (2006b). Learning dynamic algorithm portfolios. Annals of Mathematics and Artificial Intelligence, 47(3-4), 295-328.

Digital Library

[20]

Gebruers, C., Hnich, B., Bridge, D., & Freuder, E. (2005). Using CBR to select solution strategies in constraint programming. In Proceedings of the Sixth International Conference on Case-Based Reasoning (ICCBR'05), pp. 222-236.

Digital Library

[21]

Gebruers, C., Guerri, A., Hnich, B., & Milano, M. (2004). Making choices using structure at the instance level within a case based reasoning framework. In International Conference on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems (CPAIOR-04), pp. 380-386.

[22]

Gomes, C. P., & Selman, B. (2001). Algorithm portfolios. Artificial Intelligence, 126(1-2), 43-62.

Digital Library

[23]

Guerri, A., & Milano, M. (2004). Learning techniques for automatic algorithm portfolio selection. In Proceedings of the 16th European Conference on Artificial Intelligence (ECAI-04), pp. 475-479.

Digital Library

[24]

Guo, H., & Hsu, W. H. (2004). A learning-based algorithm selection meta-reasoner for the real-time MPE problem. In Proceedings of the Seventeenth Australian Conference on Artificial Intelligence, pp. 307- 318.

Digital Library

[25]

Guyon, I., Gunn, S., Nikravesh, M., & Zadeh, L. (2006). Feature Extraction, Foundations and Applications. Springer.

Digital Library

[26]

Heule, M., & v. Maaren, H. (2007). march_ks. Solver description, SAT competition 2007.

[27]

Heule, M., Zwieten, J., Dufour, M., & Maaren, H. (2004). March_eq: implementing additional reasoning into an efficient lookahead SAT solver. In Proceedings of the Seventh International Conference on Theory and Applications of Satisfiability Testing (SAT'04), pp. 345-359.

Digital Library

[28]

Hoos, H. H. (2002). An adaptive noise mechanism for WalkSAT. In Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI'02), pp. 655-660.

Digital Library

[29]

Hoos, H. H., & Stüützle, T. (2005). Stochastic Local Search - Foundations & Applications. Morgan Kaufmann Publishers, San Francisco, CA, USA.

Digital Library

[30]

Horvitz, E., Ruan, Y., Gomes, C. P., Kautz, H., Selman, B., & Chickering, D. M. (2001). A Bayesian approach to tackling hard computational problems. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI'01), pp. 235-244.

Digital Library

[31]

Huang, J. (2007). TINISAT in SAT competition 2007. Solver description, SAT competition 2007.

[32]

Huberman, B., Lukose, R., & Hogg, T. (1997). An economics approach to hard computational problems. Science, 265, 51-54.

[33]

Hutter, F., Hamadi, Y., Hoos, H. H., & Leyton-Brown, K. (2006). Performance prediction and automated tuning of randomized and parametric algorithms. In Twelfth Internatioal Conference on Principles and Practice of Constraint Programming (CP'06), pp. 213-228.

Digital Library

[34]

Hutter, F., Tompkins, D. A. D., & Hoos, H. H. (2002). Scaling and probabilistic smoothing: Efficient dynamic local search for SAT. In Proceedings of the Eighth International Conference on Principles and Practice of Constraint Programming, pp. 233-248.

Digital Library

[35]

Ishtaiwi, A., Thornton, J., Anbulagan, Sattar, A., & Pham, D. N. (2006). Adaptive clause weight redistribution. In Twelfth Internatioal Conference on Principles and Practice of Constraint Programming (CP'06), pp. 229-243.

Digital Library

[36]

Kautz, H., & Selman, B. (1996). Pushing the envelope: Planning, propositional logic, and stochastic search. In Proceedings of the Thirteenth National Conference on Artificial Intelligence and the Eighth Innovative Applications of Artificial Intelligence Conference, pp. 1194-1201.

Digital Library

[37]

Kautz, H. A., & Selman, B. (1999). Unifying SAT-based and graph-based planning. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (IJCAI'99), pp. 318-325.

Digital Library

[38]

Knuth, D. (1975). Estimating the efficiency of backtrack programs. Mathematics of Computation, 29(129), 121-136.

[39]

Krishnapuram, B., Carin, L., Figueiredo, M., & Hartemink, A. (2005). Sparse multinomial logistic regression: Fast algorithms and generalization bounds. In IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 957-968.

Digital Library

[40]

Kullmann, O. (2002). Investigating the behaviour of a SAT solver on random formulas. http://cs- svr1.swan.ac.uk/~csoliver/Artikel/OKsolverAnalyse.html.

[41]

Lagoudakis, M. G., & Littman, M. L. (2001). Learning to select branching rules in the DPLL procedure for satisfiability. In LICS/SAT, pp. 344-359.

[42]

Le Berre, D., & Simon, L. (2004). Fifty-five solvers in Vancouver: The SAT 2004 competition. In Proceedings of the Seventh International Conference on Theory and Applications of Satisfiability Testing (SAT'04), pp. 321-344.

Digital Library

[43]

Leyton-Brown, K., Nudelman, E., Andrew, G., McFadden, J., & Shoham, Y. (2003a). Boosting as a metaphor for algorithm design. In Ninth Internatioal Conference on Principles and Practice of Constraint Programming (CP'03), pp. 899-903.

Digital Library

[44]

Leyton-Brown, K., Nudelman, E., Andrew, G., McFadden, J., & Shoham, Y. (2003b). A portfolio approach to algorithm selection. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI'03), pp. 1542-1543.

Digital Library

[45]

Leyton-Brown, K., Nudelman, E., & Shoham, Y. (2002). Learning the empirical hardness of optimization problems: The case of combinatorial auctions. In Eighth Internatioal Conference on Principles and Practice of Constraint Programming (CP'02), pp. 556-572.

Digital Library

[46]

Li, C., & Huang, W. (2005). Diversification and determinism in local search for satisfiability. In Proceedings of the Eighth International Conference on Theory and Applications of Satisfiability Testing (SAT'05), pp. 158-172.

Digital Library

[47]

Lobjois, L., & Lemaîître, M. (1998). Branch and bound algorithm selection by performance prediction. In Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI'98), pp. 353-358.

Digital Library

[48]

Mahajan, Y. S., Fu, Z., & Malik, S. (2005). Zchaff 2004: an efficient SAT solver. In Proceedings of the Eighth International Conference on Theory and Applications of Satisfiability Testing (SAT'05), pp. 360-375.

Digital Library

[49]

Murphy, K. (2001). The Bayes Net Toolbox for Matlab. In Computing Science and Statistics, Vol. 33. http://bnt.sourceforge.net/.

[50]

Nadel, A., Gordon, M., Palti, A., & Hanna, Z. (2006). Eureka-2006 SAT solver. Solver description, SAT Race 2006.

[51]

Nudelman, E., Leyton-Brown, K., Hoos, H. H., Devkar, A., & Shoham, Y. (2004a). Understanding random SAT: Beyond the clauses-to-variables ratio. In Tenth Internatioal Conference on Principles and Practice of Constraint Programming (CP'04), pp. 438-452.

Digital Library

[52]

Nudelman, E., Leyton-Brown, K., Devkar, A., Shoham, Y., & Hoos, H. (2004b). Satzilla: An algorithm portfolio for SAT. Solver description, SAT competition 2004.

[53]

Pham, D. N., & Anbulagan (2007). Resolution enhanced SLS solver: R+AdaptNovelty+. Solver description, SAT competition 2007.

[54]

Pham, D. N., & Gretton, C. (2007). gNovelty+. Solver description, SAT competition 2007.

[55]

Pipatsrisawat, K., & Darwiche, A. (2006). Rsat 1.03: SAT solver description. Tech. rep. D-152, Automated Reasoning Group, UCLA.

[56]

Pipatsrisawat, K., & Darwiche, A. (2007). Rsat 2.0: SAT solver description. Solver description, SAT competition 2007.

[57]

Rice, J. R. (1976). The algorithm selection problem. Advances in Computers, 15, 65-118.

[58]

Samulowitz, H., & Memisevic, R. (2007). Learning to solve QBF. In Proceedings of the Twentysecond National Conference on Artificial Intelligence (AAAI'07), pp. 255-260.

Digital Library

[59]

Schmee, J., & Hahn, G. J. (1979). A simple method for regression analysis with censored data. Technometrics, 21(4), 417-432.

[60]

Selman, B., Kautz, H., & Cohen, B. (1994). Noise strategies for improving local search. In Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI'94), pp. 337-343.

Digital Library

[61]

Selman, B., Levesque, H., & Mitchell, D. (1992). A new method for solving hard satisfiability problems. In Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI'92), pp. 440-446.

Digital Library

[62]

Sörensson, N., & Eéén, N. (2007). Minisat2007. http://www.cs.chalmers.se/Cs/Research/FormalMethods/MiniSat/.

[63]

Spence, I. (2007). Ternary tree solver (tts-4-0). Solver description, SAT competition 2007.

[64]

Stephan, P., Brayton, R., & Sangiovanni-Vencentelli, A. (1996). Combinational test generation using satisfiability. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 15, 1167-1176.

Digital Library

[65]

Streeter, M., Golovin, D., & Smith, S. F. (2007). Combining multiple heuristics online. In Proceedings of the Twentysecond National Conference on Artificial Intelligence (AAAI'07), pp. 1197-1203.

Digital Library

[66]

Subbarayan, S., & Pradhan, D. (2005). Niver: Non-increasing variable elimination resolution for preprocessing sat instances. Lecture Notes in Computer Science, Springer, 3542/2005, 276-291.

Digital Library

[67]

Tompkins, D. A. D., & Hoos, H. H. (2004). UBCSAT: An implementation and experimentation environment for SLS algorithms for SAT & MAX-SAT. In Proceedings of the Seventh International Conference on Theory and Applications of Satisfiability Testing (SAT'04).

Digital Library

[68]

Vallstrom, D. (2005). Vallst documentation. http://vallst.satcompetition.org/index.html.

[69]

van Gelder, A. (2002). Another look at graph coloring via propositional satisfiability. In Proceedings of Computational Symposium on Graph Coloring and Generalizations (COLOR-02), pp. 48-54.

[70]

Wei, W., Li, C. M., & Zhang, H. (2007). Deterministic and random selection of variables in local search for SAT. Solver description, SAT competition 2007.

[71]

Xu, L., Hoos, H. H., & Leyton-Brown, K. (2007a). Hierarchical hardness models for SAT. In Thirteenth International Conference on Principles and Practice of Constraint Programming (CP'07), pp. 696-711.

Digital Library

[72]

Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2007b). Satzilla-07: The design and analysis of an algorithm portfolio for SAT. In Thirteenth Internatioal Conference on Principles and Practice of Constraint Programming (CP'07), pp. 712-727.

Digital Library

[73]

Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2007c). Satzilla2007: a new & improved algorithm portfolio for SAT. Solver description, SAT competition 2007.

[74]

Zhang, L., Madigan, C. F., Moskewicz, M. W., & Malik, S. (2001). Efficient conflict driven learning in Boolean satisfiability solver. In Proceedings of the International Conference on Computer Aided Design, pp. 279-285.

Digital Library

[75]

Zhang, L. (2002). The quest for efficient Boolean satisfiability solvers. In Proceedings of 8th International Conference on Computer Aided Deduction (CADE-02), pp. 313-331.

Digital Library

Cited By

Wu XZhong YWu JJiang BTan KLarson K(2024)Large language model-enhanced algorithm selectionProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/579(5235-5244)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/579
Graham DLeyton-Brown KRoughgarden TKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Formalizing preferences over runtime distributionsProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3618876(11659-11682)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3618876
Balcan MPrasad SSandholm TVitercik EKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Structural analysis of branch-and-cut and the learnability of gomory mixed integer cutsProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3602726(33890-33903)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3602726
Show More Cited By

Recommendations

SATzilla-07: the design and analysis of an algorithm portfolio for SAT
CP'07: Proceedings of the 13th international conference on Principles and practice of constraint programming

It has been widely observed that there is no "dominant" SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate ...
On stateless multihead automata: Hierarchies and the emptiness problem

We look at stateless multihead finite automata in their two-way and one-way, deterministic and nondeterministic variations. The transition of a k-head automaton depends solely on the symbols currently scanned by its k heads, and every such transition ...
Pebble alternating tree-walking automata and their recognizing power

Pebble tree-walking automata with alternation were first investigated by Milo, Suciu and Vianu (2003), who showed that tree languages recognized by these devices are exactly the regular tree languages. We strengthen this by proving the same result for ...

Comments

Information & Contributors

Information

Published In

cover image Journal of Artificial Intelligence Research

Journal of Artificial Intelligence Research Volume 32, Issue 1

May 2008

974 pages

ISSN:1076-9757

Issue’s Table of Contents

Publisher

AI Access Foundation

El Segundo, CA, United States

Publication History

Published: 01 June 2008

Received: 01 November 2007

Published in JAIR Volume 32, Issue 1

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

210
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu XZhong YWu JJiang BTan KLarson K(2024)Large language model-enhanced algorithm selectionProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/579(5235-5244)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/579
Graham DLeyton-Brown KRoughgarden TKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Formalizing preferences over runtime distributionsProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3618876(11659-11682)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3618876
Balcan MPrasad SSandholm TVitercik EKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Structural analysis of branch-and-cut and the learnability of gomory mixed integer cutsProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3602726(33890-33903)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3602726
McElfresh DKhandagale SValverde JDickerson JWhite CKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)On the generalizability and predictability of recommender systemsProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3600589(4416-4432)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3600589
Liu TAmadini RGabbrielli MMauro J(2022)sunny-as2Journal of Artificial Intelligence Research10.1613/jair.1.1311672(329-376)Online publication date: 4-Jan-2022
https://dl.acm.org/doi/10.1613/jair.1.13116
Alyahya TMenai MMathkour H(2022)On the Structure of the Boolean Satisfiability Problem: A SurveyACM Computing Surveys10.1145/349121055:3(1-34)Online publication date: 30-Mar-2022
https://dl.acm.org/doi/10.1145/3491210
Sitaru IRaschip M(2022)Algorithm Selection for Combinatorial Packing Problems2022 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC55065.2022.9870417(1-8)Online publication date: 18-Jul-2022
https://dl.acm.org/doi/10.1109/CEC55065.2022.9870417
Janičić PNarboux J(2022)Theorem Proving as Constraint Solving with Coherent LogicJournal of Automated Reasoning10.1007/s10817-022-09629-z66:4(689-746)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10817-022-09629-z
Burgess MGretton CMilthorpe JCroak LWillingham TTiu A(2022)Dagster: Parallel Structured Search with Case StudiesPRICAI 2022: Trends in Artificial Intelligence10.1007/978-3-031-20862-1_6(75-89)Online publication date: 10-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-20862-1_6
Hůla JJakubův JJanota MKubej L(2022)Targeted Configuration of an SMT SolverIntelligent Computer Mathematics10.1007/978-3-031-16681-5_18(256-271)Online publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-16681-5_18
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents