Generating Extended Resolution Proofs with a BDD-Based SAT Solver

Robinson [41] observed that the resolution inference rule formulated by Davis and Putnam [21] could form the basis for a refutation theorem-proving technique for first-order logic. Here, we consider its specialization to propositional logic. For clauses of the form \(C\vee x\), and \(\overline {x} \vee D\), the resolution rule derives the new clause \(C \vee D\). This inference is written with a notation showing the required conditions above a horizontal line, and the resulting inference (known as the resolvent) below:Intuitively, the resolution rule is based on the property that implication is transitive. To see this, let proposition \(p\) denote \(\lnot C\) and proposition \(q\) denote \(D\). Then, \(C \vee x\) is equivalent to \(p \rightarrow x\), \(\overline {x} \vee D\) is equivalent to \(x \rightarrow q\), and \(C \vee D\) is equivalent to \(p \rightarrow q\). In other words, the resolution rule encodes the property that if \(p \rightarrow x\) and \(x \rightarrow q\), then \(p \rightarrow q\). As a special case, when \(C\) contains a literal \(\ell\) and \(D\) contains its complement \(\overline {\ell }\), then the resolvent of \(C \vee x\) and \(D \vee \overline {x}\) will be a tautology.

Resolution provides a mechanism for proving that a set of clauses is unsatisfiable. Suppose the input consists of \(m\) clauses. A resolution proof is given as a trace consisting of a series of steps \(S\), where each step \(s_i\) consists of a clause \(C_i\) and a (possibly empty) list of antecedents \(A_i\), where each antecedent is the index of one of the previous steps. The first set of steps, denoted \(S_m\), consists of the input clauses without any antecedents. Each successive step then consists of a clause and a set of antecedents such that the clause can be derived from the clauses in the antecedents by one or more resolution steps. It follows by transitivity that for each step \(s_i\), with \(i \gt m\), clause \(C_i\) is logically implied by the input clauses, written \(S_m \vDash C_i\). If, through a series of steps, we can reach a step \(s_t\) where \(C_t\) is the empty clause, then the trace provides a proof that \(S_m \vDash \bot\), that is, the set of input clauses is not satisfiable.

A typical resolution proof contains many applications of the resolution rule. These enable deriving sequences of implications that combine by transitivity. For example, consider the following implications, shown both as formulas and as clauses:

We can derive the final clause from the first three using two resolution steps:

Reverse unit propagation (RUP) provides an easily checkable way to express a linear sequence of resolution operations as a single proof step [25, 49]. It is the core rule supported by standard proof checkers [29, 51] for propositional logic. Let \(C = \ell _1 \, \ell _2 \, \cdots \, \ell _p\) be a clause to be proved and let \(D_1, D_2, \ldots , D_k\) be a sequence of supporting antecedent clauses occurring earlier in the proof. A RUP step proves that \(\bigwedge _{1\le i \le k} D_i \rightarrow C\) by showing that the combination of the antecedents plus the negation of \(C\) leads to a contradiction. The negation of \(C\) is the formula \(\overline{\ell }_1 \wedge \overline{\ell }_2 \wedge \cdots \wedge \overline{\ell }_p\) having a CNF representation consisting of \(p\) unit clauses of the form \(\overline {\ell }_i\) for \(1 \le i \le p\). A RUP check processes the clauses of the antecedent in sequence, inferring additional unit clauses. In processing clause \(D_i\), if all but one of the literals in the clause is the negation of one of the accumulated unit clauses, then we can add this literal to the accumulated set. That is, all but this literal have been falsified, and so it must be set to true for the clause to be satisfied. The final step with clause \(D_k\) must cause a contradiction, that is, all of its literals are falsified by the accumulated unit clauses.

As an example, consider a RUP step to derive \(x \rightarrow (a \rightarrow d)\) from the three clauses shown in the earlier example. A RUP proof would take the following form. Here, the target and antecedent clauses are listed along the top, while the resulting unit clauses are shown on the bottom, along with the final contradiction.

RUP is an alternative formulation of resolution. For target clause \(C\), it can be seen that applying resolution operations to the antecedent clauses from right to left will derive a clause \(C^{\prime }\) such that \(C^{\prime } \subseteq C\). By subsumption [40], we then have \(C^{\prime } \rightarrow C\). Compared with listing each resolution operation as a separate step, using RUP as the basic proof step makes the proofs more compact.

Grigori S. Tseitin [46] introduced the extended-resolution proof framework in a presentation at the Leningrad Seminar on Mathematical Logic in 1966. The key idea is to allow the addition of new extension variables to a resolution proof in a manner that preserves the soundness of the proof. In particular, in introducing variable \(e\), there must be an accompanying set of clauses that encode \(e \leftrightarrow F\), where \(F\) is a formula over variables (both original and extension) that were introduced earlier [32]. These are referred to as the defining clauses for extension variable \(e\). Variable \(e\) then provides a shorthand notation by which \(F\) can be referenced multiple times. Doing so can reduce the size of a clausal representation of a problem by an exponential factor.

An extension variable \(e\) is introduced into the proof by including its defining clauses in the list of clauses being generated. The proof checker must then ensure that the defining clauses obey the requirements for extension variables, as is discussed below. Thereafter, other clauses can include the extension variable or its complement, and they can list the defining clauses as antecedents.

Tseitin transformations are commonly used to encode a logic circuit or formula as a set of clauses without requiring the formulas to be “flattened” into a conjunctive normal form over the circuit inputs or formula variables. These introduced variables are called Tseitin variables and are considered to be part of the input formula. An extended resolution proof takes this concept one step further by introducing additional variables as part of the proof. The proof checker must ensure that the extension variables are used in a way that does not result in an unsound proof. Some problems for which the minimum resolution proof must be of exponential size can be expressed with polynomial-sized proofs in extended resolution [18].

We use a clausal proof system to validate our proofs based on the DRAT proof framework [29]. This framework supports both extended resolution and resolution operations based on a proof rule that generalizes reverse unit propagation. There are a number of fast and formally- verified checkers for these proofs [20, 33, 51]. The checker ensures that all extension variables are used properly and that each new clause can be derived via RUP from its antecedent clauses.

As in a resolution proof, a clausal proof is given as a trace, where each step \(s_i\) consists of a clause \(C_i\) and a list of antecedents \(A_i\), where the initial \(m\) clauses are the input clauses. Let \(S_m\) denote the set of input clauses, and for \(i \gt m\), define \(S_i\) inductively as \(S_i = S_{i-1} \cup \lbrace C_i \rbrace\). The proof steps \(s_{m+1}, \dots , s_t\) represent a derivation from \(S_m\) to \(S_t\). A clausal proof is a refutation if \(S_t\) contains the empty clause. Step \(s_i\) in a proof is valid if the equisatisfiability¹ of \(S_{i-1}\) and \(S_{i}\) can be checked using a polynomially decidable redundancy property. For the case in which \(C_i\) was obtained via RUP, we can simply perform a RUP check using \(C_i\) and the antecedents. In the case in which \(C_i\) is one of the defining clauses for some extension variable \(e\), the checker must ensure that the clause is blocked [32]. That is, all possible resolvents of \(C_i\) with clauses in \(S_{i-1}\) that contain \(e\) must be tautologies. The blocked clause proof system is a generalization of extended resolution and allows the addition of blocked clauses that are blocked on non-extension variables. However, we do not use such capabilities in our proofs.

Clausal proofs also allow the removal of clauses. A proof can indicate that clause \(C_j\) can be removed after step \(s_i\) if it will not be used as an antecedent in any step \(s_k\) with \(k \gt i\). With this restriction, clause deletion does not affect the integrity of the proof. As the experimental results of Section 5 demonstrate, deleting clauses that are no longer needed can substantially reduce the number of clauses the checker must track while processing a proof.

Reduced, Ordered Binary Decision Diagrams (which we refer to as simply “BDDs”) provide a canonical form for representing Boolean functions and an associated set of algorithms for constructing them and testing their properties [6]. With BDDs, functions are defined over a set of variables \(X = \lbrace x_1, x_2, \ldots , x_n\rbrace\). We let \(\mathbf {T}_0\) and \(\mathbf {T}_1\) denote the two leaf nodes, representing the constant functions \({\bf 0}\) and \({\bf 1}\), respectively.

Each nonterminal node \(\mathbf {u}\) has an associated variable \(\text {Var}(\mathbf {u})\) and children \(\text {Hi}(\mathbf {u})\), indicating the case in which the node variable has value 1, and \(\text {Lo}(\mathbf {u})\), indicating the case in which the node variable has value 0.

Two lookup tables—the unique table and the operation cache—are critical for guaranteeing the canonicity of the BDDs and for ensuring polynomial performance of the BDD construction algorithms.

A node \(\mathbf {u}\) is stored in a unique table, indexed by a key of the form \(\langle \text {Var}(\mathbf {u}), \text {Hi}(\mathbf {u}), \text {Lo}(\mathbf {u})\rangle\), so that isomorphic nodes are never created. The nodes are shared across all of the BDDs [38]. In presenting algorithms, we assume a function GetNode \((x, \mathbf {u}_1, \mathbf {u}_0)\) that checks the unique table and either returns the node stored there or it creates a new node and enters it into the table. With this table, we can guarantee that the subgraphs with root nodes \(\mathbf {u}\) and \({\mathbf {v}}\) represent the same Boolean function if and only if \(\mathbf {u}={\mathbf {v}}\). We can therefore uniquely identify Boolean functions with their BDD root nodes.

BDD packages support multiple operations for constructing and testing the properties of Boolean functions represented by BDDs. A number of these are based on the Apply algorithm [6]. Given a set of BDD roots \(\mathbf {u}_1, \mathbf {u}_2, \ldots , \mathbf {u}_k\) representing functions \(f_1, f_2, \ldots , f_k\), respectively, and a Boolean operation Op, the algorithm generates the BDD representation \({\mathbf {w}}\) of the operation applied to those functions. For example, with \(k = 2\), and Op = And, Apply (And, \(\mathbf {u}_1, \mathbf {u}_2)\) returns the root node for the BDD representation of \(f_1 \wedge f_2\).

Figure 1 shows pseudo-code describing the overall structure of the Apply algorithm. The details for a specific operation are embodied in the functions IsTerminal, TerminalValue, and ApplyRecur. The first two of these detect terminal cases and what value to return when a terminal case is encountered. The third describes how to handle the general case, in which the arguments must be expanded recursively. The algorithm makes use of memoizing, where previously computed results are stored in an operation cache, indexed by a key consisting of the operands [37]. Whenever possible, results are retrieved from this cache, avoiding the need to perform redundant calls to ApplyRecur. With this cache, the worst-case number of recursive steps required by the algorithm is bounded by the product of the sizes (in nodes) of the arguments.

The BDD representation for a clause \(C\) has a simple, linear structure. For root node \(\mathbf {u}\), it is easy to prove that \(C \vDash u\) using one RUP step. The general algorithm is described by Sinz and Biere [44]. Here, we illustrate the idea via an example.

Figure 2 shows the BDD representation of clause \(C = a\,\overline {b}\,c\). As can be seen, the BDD for a clause has a very specific structure. For each literal in the clause, there is a node labeled by the variable, with one child being leaf \(\mathbf {T}_1\) and the other being either the node for the next literal in the variable ordering or leaf \(\mathbf {T}_0\). The lower part of the figure shows a RUP justification of \(C \vDash u_a\), where \(\mathbf {u}_a\) is the root node of the BDD. The proof uses the antecedents \(\text {HU}(\mathbf {u})\) and \(\text {LU}(\mathbf {u})\) for each node \(\mathbf {u}\) in the BDD (except for the tautological case representing the final edge to \(\mathbf {T}_0\)), with the final antecedent being the clause itself. The RUP steps introduce the complements of the clause variables as unit clauses, causing a contradiction with the input clause. The order in which the two defining clauses for a node are listed in the antecedent depends on whether the variable is positive or negative in the clause. As this example demonstrates, we can generate a single proof step for \(C_i \vDash u_i\) for each input clause \(C_i\).

The key idea in generating proofs for the conjunction operation is to follow the recursive structure of the Apply algorithm. We do this by integrating proof generation into the Apply procedures, as is shown in Figure 3. This follows the standard form of the Apply algorithm (Figure 1), with the novel feature that each result includes both a BDD node \({\mathbf {w}}\) and a proof step number \(s\). For arguments \(\mathbf {u}\) and \({\mathbf {v}}\), step \(s\) lists clause \(\overline {u}\,\overline {v}\,w\) along with antecedents defining a RUP proof of the implication \(u \wedge v \rightarrow w\).

As the table of terminal cases shows, these cases all correspond to tautologies. For example, the case of \(\mathbf {u} = \mathbf {T}_1\), giving \({\mathbf {w}} = {\mathbf {v}}\) is justified by that tautology \(\top \wedge v \rightarrow v\). Failing a terminal or previously computed case, the function must recurse, branching on the variable \(x\) that is the minimum of the two root variables. The procedure accumulates a set of proof steps \(J\) to be used in the implication proof. These include the two steps (possibly tautologies) from the two recursive calls. At the end, it invokes a function JustifyAnd to generate the required proof. In returning the pair \(({\mathbf {w}}, s)\), this value will be stored in the operation cache and returned as the result of the Apply operation.

As described in Section 3, we need not track the detailed logic of the algorithm that performs existential quantification. Instead, when the quantification operation applied to node \(\mathbf {u}\) generates node \({\mathbf {v}}\), we generate a proof of implication afterwards, using the Apply algorithm adapted for implication checking, as shown in Figure 7. A failure of this implication check would indicate an error in the BDD package. Thus, its only purpose is to generate a proof that the implication holds, signaling a fatal error if the implication does not hold.

This particular operation does not generate any new nodes; thus, the returned result is simply a proof step number. The (successful) terminal cases correspond to the tautological cases \(u \rightarrow u\), \(\bot \rightarrow v\), and \(u \rightarrow \top\).

Each recursive step accumulates up to six proof steps as the set \(J\) to be used in the implication proof. Figure 8 shows the structure of these clauses for the standard case in which neither equality holds and neither recursive call returns \(\top\). The clauses consist of the two downward defining clauses for argument \(\mathbf {u}\), labeled \({\rm\small UHD}\) and \({\rm\small ULD}\), the two upward defining clauses for argument \({\mathbf {v}}\), labeled \({\rm\small VHU}\) and \({\rm\small VLU}\), and the clauses returned by the recursive calls, labeled \({\rm\small IMH}\) and \({\rm\small IML}\).

Figure 9 shows the two RUP steps required to prove the standard case. The first step proves the weaker target \(x \rightarrow (u \rightarrow v)\), having clausal representation \(\overline {x}\,\overline {u}\,v\) using the three supporting clauses containing \(\overline {x}\). The second proves the full target, having clausal representation \(\overline {u}\,v\) using the weaker result plus the supporting clauses containing \(x\).

As with the conjunction operation, there can be many special cases, but they can be handled with the same general strategy. If either recursive result \({\rm\small IMH}\) or \({\rm\small IML}\) is a tautology, a one-step proof is attempted. If that fails, or if neither recursive result is a tautology, a two-step proof is generated.

Chew and Heule [17] introduced a benchmark problem based on computing the parity of a set of Boolean values \(x_1, \ldots , x_n\) using two different orderings of the inputs and with one of the variables negated in the second computation:where \(\pi\) is a random permutation, and each \(p_i\) is either 0 or 1, with the restriction that \(p_i = 1\) for only one value of \(i\). The two sums associate from left to right. Therefore, the formula \({\it ParityA} \wedge {\it ParityB}\) is unsatisfiable, but the permutation makes this difficult for CDCL solvers to determine. The CNF has a total of \(3n - 2\) variables: \(n\) values of \(x_i\), plus the auxiliary variables encoding the intermediate terms in the two expressions.

Chew and Heule experimented with the CDCL solver CaDiCaL [3] and found it could not handle cases with \(n\) greater than 50. They devised a specialized method for directly generating proofs in the DRAT proof system, obtaining proofs that scale as \(O(n \log n),\) and gave results for up to \(n=4{,}000\). They also tried ebddres, but only in its default mode, where it performs only linear evaluation without any quantification.

Figure 10 shows the result of applying both tbsat and kissat to this problem. In this and other figures, the top graph shows how the runtime scales with the problem size, whereas the bottom graph shows how the number of proof clauses scale. Both graphs are log-log plots; thus, the values are highly compressed along both dimensions. Linear evaluation performs poorly, only handling up to \(n=24\) within the 1000-second time limit, generating a proof with over 312 million clauses. Using kissat, we found that the results were very sensitive to the choice of random permutation. Thus, we show results using three different random seeds for each value of \(n\). We were able to generate proofs for instances with \(n\) up to 46 within the time limit but also started having timeouts with \(n=42\). We can see that kissat does better than linear evaluation with tbsat, but both appear to scale exponentially.

Bucket elimination, on the other hand, displays much better scaling. We found that the best performance was achieved by randomly permuting the variables, although this strategy only yields a constant factor improvement over the ordering from the CNF file. As the graphs show, we were able to handle cases with \(n\) up to \(9{,}750\), within the time limit. This generated a proof with over 419 million clauses, but the LRAT checker was able to verify this proof in 256 seconds. Although tbsat could generate proofs for larger values of \(n\), these exceeded the capacity of the LRAT checker.

Included in the second graph are results for running Chew and Heule’s proof generator on this problem. As can be seen, the proof sizes generated by tbsat are comparable to theirs up to around \(n=100\). From there on, however, the benefit of their \(O(n\,\log n)\) algorithm becomes apparent. Even for \(n = 10{,}000\), their proof contains less than 11 million clauses. Of course, their construction relies on particular properties of the underlying problem, while ours was generated by a general-purpose SAT solver.

Urquhart [48] introduced a family of formulas that require resolution proofs of exponential size. Over the years, two families of SAT benchmarks have been labeled as “Urquhart Problems”: one developed by Chatalic and Simon [16], and the other by Li [34]. These are considered to be difficult challenge problems for SAT solvers. Here, we define their general form, describe the differences between the two families, and evaluate the performance of both kissat and tbsat on both classes.

Urquhart’s construction is based on a class of bipartite graphs with special properties. Define \(G_k\) as the set of undirected graphs, with each graph satisfying the following properties:

Furthermore, the graphs must be expanders, defined as follows [30]. For a subset of vertices \(U \subseteq L\), define \(R(U)\) to be those vertices in \(R\) adjacent to the vertices on \(U\). A graph in \(G_k\) is an expander if there is some constant \(d \gt 0\) such that for any \(U \subset L\) with \(|U| \le |L|/2\), the set \(R(U)\) satisfies \(|R(U)| \ge (d+1) |U|\). Urquhart considers expander graphs with degree bound \(k=5\) and that are parameterized by a size value \(m\) with \(|L|=|R|=m^2\).

To transform such a graph into a formula, each edge \((i, j) \in E\) has an associated variable \(x_{\lbrace i,j\rbrace }\). (We use this notation to emphasize that the order of the indices does not matter.) Each vertex is assigned a polarity \(p_i \in {0,1}\) such that the sum of the polarities is odd. The clauses then encode the formula:This is false, of course, since each edge gets counted twice in the sum and the sum of the polarities is odd.

The two families of benchmarks differ in how the graphs are constructed. Li’s benchmarks are based on the explicit construction of expander graphs due to Margulis [24, 35] that is cited by Urquhart. Thus, his graphs are fully defined by the size parameter \(m\). Simon’s benchmarks are based on randomly generated graphs. Thus, they are characterized by both the size parameter \(m\) and the initial random seed \(s\). Although random graphs satisfy the expander condition with high probability [30], it is unlikely that the particular instances generated by Simon’s benchmark generator are truly expander graphs. The widely used SAT benchmarks with names of the form Urq\(M\)_\(S\).cnf were generated by Simon’s program for size parameter \(m=M\) and initial seed \(S\). For Simon’s benchmarks, we used five different seeds for each value of \(m\).

Figure 11 shows data for running kissat as well as tbsat using bucket elimination. The data for kissat demonstrate how difficult these benchmark problems are for CDCL solvers. With a time limit of 1000 seconds, we found that kissat could handle all five instances of Simon’s benchmarks with \(m = 3\) but none for larger values of \(m\). For Li’s benchmarks, it failed for even the minimum case of \(m=3\). Running tbsat with bucket elimination with a random ordering of the variables fares much better. For Li’s benchmarks, it successfully handled instances up to \(m=38\), yielding a proof with around 373 million clauses. For Simon’s benchmarks, bucket elimination handled benchmarks for all five seeds up to \(m=60\). We can also see that Simon’s benchmarks are decidedly easier than Li’s, requiring up to an order of magnitude fewer clauses in the proofs.

Jussila et al. [31] showed benchmark results for what appear to be Simon’s Urquhart formulas up to \(m=8\) with performance (in terms of proof size) comparable to ours. Indeed, in using bucket elimination, we are replicating their approach. We know of no prior proof-generating SAT solver that can handle Urquhart formulas of this scale.

The mutilated chessboard problem considers an \(n \times n\) chessboard, with the corners on the upper left and the lower right removed. It attempts to tile the board with dominos, with each domino covering two squares. Since the two removed squares had the same color, and each domino covers one white and one black square, no tiling is possible. This problem has been well studied in the context of resolution proofs, for which it can be shown that any proof must be of exponential size [1].

A standard CNF encoding involves defining Boolean variables to represent the boundaries between adjacent squares, set to 1 when a domino spans the two squares, and set to 0 otherwise. The clauses then encode an Exactly1 constraint for each square, requiring each square to share a domino with exactly one of its neighbors. We label the variables representing a horizontal boundary between a square and the one below as \(y_{i,j}\), with \(1 \le i \lt n\) and \(1 \le j \le n\). The variables representing the vertical boundaries are labeled \(x_{i,j}\), with \(1 \le i \le n\) and \(1 \le j\lt n\). With a mutilated chessboard, we have that \(y_{1,1} = x_{1,1} = y_{n-1,n} = x_{n,n-1} = 0\).

As the plots of Figure 12 show, a straightforward application of linear conjunctions or bucket elimination by tbsat displays exponential scaling. Indeed, tbsat fares no better than kissat when operating in either of these modes, with all limited to \(n \le 20\) within the 1000-second time limit.

On the other hand, another approach, inspired by symbolic model checking [15], demonstrates far better scaling, reaching \(n=340\). It is based on the following observation: when processing the columns from left to right, the only information required to place dominos in column \(j\) is the identity of those rows \(i\) for which a domino crosses horizontally from \(j-1\) to \(j\). This information is encoded in the values of \(x_{i,j-1}\) for \(1 \le i \le n\).

In particular, group the variables into columns, with \(X_j\) denoting variables \(x_{1,j}, \ldots , x_{n,j}\), and \(Y_j\) denoting variables \(y_{1,j}, \ldots , y_{n-1,j}\). Scanning the board from left to right, consider \(X_j\) to encode the “state” of processing after completing column \(j\). As the scanning process reaches column \(j\), there is a characteristic function \(\sigma _{j-1}(X_{j-1})\) describing the set of allowed crossings of horizontally oriented dominos from column \(j-1\) into column \(j\). No other information about the configuration of the board to the left is required. The characteristic function after column \(j\) can then be computed as:where \(T_j(X_{j-1}, Y_{j}, X_{j})\) is a “transition relation” consisting of the conjunction of the Exactly1 constraints for column \(j\). From this, we can existentially quantify the variables \(Y_{j}\) to obtain a BDD encoding all compatible combinations of the variables \(X_{j-1}\) and \(X_{j}\). By conjuncting this with the characteristic function for column \(j-1\) and existentially quantifying the variables \(X_{j-1}\), we obtain the characteristic function for column \(j\). With a mutilated chessboard, we generate leaf node \(L_0\) in attempting the final conjunction. Note that Equation (3) does not represent a reformulation of the mutilated chessboard problem. It simply defines a way to schedule the conjunction and quantification operations over the input clauses.

One important rule-of-thumb in symbolic model checking is that the successive values of the next-state variables must be adjacent in the variable ordering. Furthermore, the vertical variables in \(Y_j\) must be close to their counterparts in \(X_{j-1}\) and \(X_{j}\). Both objectives can be achieved by ordering the variables row-wise, interleaving the variables \(x_{i,j}\) and \(y_{i,j}\), ordering first by row index \(i\) and then by column index \(j\). This requires the quantification operations of Equation (3) to be performed on non-root variables.

In our experiments, we found that this scanning reaches a fixed point after processing \(n/2\) columns. That is, from that column onward, the characteristic functions become identical, except for a renaming of variables. This indicates that the set of all possible horizontal configurations stabilizes halfway across the board. Moreover, the BDD representations of the states grow as \(O(n^2)\). For \(n=340\) largest has just \(29{,}239\) nodes. The problem size for the mutilated chessboard scales is \(n^2\), the number of squares in the board. Thus, an instance with \(n=340\) is 289 times larger than an instance with \(n=20\) in terms of the number of input variables and clauses. Column scanning yields a major benefit in the solver performance.

The plot labeled “No Quantification” demonstrates the importance of including existential quantification in solving this problem. These data were generated by using the same schedule as with column scanning but with all quantification operations omitted. As can be seen, this approach could not scale beyond \(n=10\).

It is interesting to reflect on how our column-scanning approach relates to SAT-based bounded model checking (BMC) [4]. This approach to verification encodes the operation of a state transition system for \(k\) steps, for some fixed value of \(k\), by instantiating the transition relation \(k\) times. It then uses a SAT solver to detect whether some condition can arise within \(k\) steps of operation. By contrast, we effectively compress the mutilated chessboard problem into a state machine that adds tiles to successive columns of the board and then perform a BDD-based reachability computation for this system, much as would a symbolic model checker [15]. Just as BDD-based model checking can outperform SAT-based BMC for some problems, we have demonstrated that a BDD-based SAT solver can sometimes outperform a search-based SAT solver.

The pigeonhole problem is one of the most studied problems in propositional reasoning. Given a set of \(n\) holes and a set of \(n+1\) pigeons, it asks whether there is an assignment of pigeons to holes such that (1) every pigeon is in some hole and (2) every hole contains at most one pigeon. The answer is no, of course, but any resolution proof for this must be of exponential length [27]. Groote and Zantema have shown that any BDD-based proof of the principle that uses only conjunction operations must be of exponential size [26]. On the other hand, Cook constructed an extended resolution proof of size \(O(n^4)\), in part to demonstrate the expressive power of extended resolution [18].

We used a representation of the problem that scales as \(O(n^2)\), using an encoding of the at-most-one constraints due to Sinz [43]. It starts with a set of variables \(p_{i,j}\) for \(1 \le i \le n\) and \(1 \le j \le n+1\), with the interpretation that pigeon \(j\) is assigned to hole \(i\). Encoding the property that each pigeon \(j\) is assigned to some hole can be expressed with a single clause:

Sinz’s method of encoding the property that each hole \(i\) contains at most one pigeon introduces auxiliary variables to effectively track which holes are occupied, starting with pigeon 1 and working upward. These variables are labeled \(s_{i,j}\) for \(1 \le i \le n\) and \(1 \le j \le n\). Informally, variables \(s_{i,1}, s_{i,2}, \ldots , s_{i,n}\) serves as a signal chain that indicates the point at which a pigeon has been assigned to hole \(i\). For each hole \(i\), there is a total of \(3n-1\) clauses:

Each of these clauses serves either to define how the next value in the chain is to be computed, or to describe the effect of the signal on the allowed assignments of pigeons to the hole. That is, for hole \(i\), the signal is generated at position \(j\) if pigeon \(j\) is assigned to that hole. Once set, the signal continues to propagate across higher values of \(j\). Once the signal is set, it suppresses further assignments of pigeons to the hole. This encoding requires \(3n-1\) clauses and \(n\) auxiliary variables per hole.

Figure 13 shows the results of running the two solvers on this problem. Once again, we see tbsat with either linear or bucket evaluation having exponential scaling, as does kissat. None can go beyond \(n=13\) within the 1000-second time limit.

On the other hand, the column scanning approach used for the mutilated checkerboard can also be applied to the pigeonhole problem when the Sinz encoding is used. Consider an array with hole \(i\) represented by row \(i\) and pigeon \(j\) represented by column \(j\). Let \(S_{j}\) represent the auxiliary variables \(s_{i,j}\) for \(1 \le i \le n\). The “state” is then encoded in these auxiliary variables. In processing pigeon \(j\), we can assume that the possible combinations of values of auxiliary variables \(S_{j-1}\) is encoded by a characteristic function \(\sigma _{j-1}(S_{j-1})\). In addition, we incorporate into this characteristic function the requirement that each pigeon \(k\), for \(1 \le k \le j-1\) is assigned to some hole. Letting \(P_j\) denote the variables \(p_{i,j}\) for \(1 \le i \le n\), the characteristic function at column \(j\) can then be expressed aswhere the “transition relation” \(T_j\) consists of the clauses associated with the auxiliary variables plus the clause encoding constraint \({\it Pigeon}_j\). As with the mutilated chessboard, having a proper variable ordering is critical to the success of a column scanning approach. We interleave the ordering of the variables \(p_{i,j}\) and \(s_{i,j}\), ordering them first by \(i\) (holes) and then by \(j\) (pigeons).

Figure 13 demonstrates the effectiveness of the column-scanning approach. We were able to handle instances up to \(n=210\). Unlike with the mutilated chessboard, the scanning does not reach a fixed point. Instead, the BDDs start very small, because they must encode the locations of only a small number of occupied holes. They reach their maximum size at pigeon \(n/2\), as the number of combinations for occupied and unoccupied holes reaches its maximum of \(C(n, n/2)\). The BDD sizes then drop off, symmetrically to the first \(n/2\) pigeons, as the encoding needs to track the positions of a decreasing number of unoccupied holes. Fortunately, all of these BDDs scale quadratically with \(n\), reaching a maximum of \(11{,}130\) nodes for \(n = 210\).

We also ran experiments using a direct encoding of the at-most-one constraints, having a clause \(\overline {p}_{i,j} \vee \overline {p}_{i,k}\) for each hole \(i\) and for \(1 \le j \lt k \le n+1\). This encoding scales as \(\Theta (n^3)\). With this encoding, we were unable to find any method that avoided exponential scaling using either tbsat or kissat.

Overall, our results demonstrate the potential for generating small proofs of unsatisfiability using BDDs. We were able to greatly outperform traditional CDCL solvers for four well-known challenge problems.

The success for the first two benchmark problems relies on the ability of BDDs to handle exclusive-or operations efficiently. Generally, the exclusive-or of \(k\) variables can be expressed as a BDD with \(2k+1\) nodes, including the leaves. These representations are also independent of the variable ordering. As we saw, however, it is critical to quantify variables whenever possible to avoid requiring the BDD to encode the parity relationships among many overlapping subsets of the variables. We found that bucket elimination works well on these problems and that randomness in the problem structure and the variable ordering did not adversely affect performance. This strategy was outlined by Jussila et al. [31]; our experimental results serve as a demonstration of the utility of their work.

The success of column scanning for the final two benchmark problems relies on finding a way to scan in one dimension, encoding the “state” of the scan in a compact form. This strategy only works when the problem is encoded in a way that it can be partitioned along two dimensions. This approach draws its inspiration from symbolic model checking, and it requires the more general capability to handle quantification that we have presented. One strength of modern SAT solvers is that they generally succeed without any special guidance from the user. It remains an open question whether column scanning can be made more general and whether a suitable schedule and variable ordering can be generated automatically. Without these capabilities, our results for column scanning show promise, but they require too much guidance from the user.

Other studies have compared BDDs to CDCL solvers on a variety of benchmark problems. Several of these observed exponential performances for BDD-based solvers for problems for which we have obtained more promising results. Uribe and Stickel [47] ran experiments with the mutilated chessboard problem, but they did not do any variable quantification. Pan and Vardi [39] applied a variety of scheduling and variable ordering strategies for the mutilated chessboard and pigeonhole problems. Although they found that they could get better performance than with a CDCL solver, their performance still scaled exponentially. Obtaining scalability requires devising more problem-specific approaches than the ones they considered. Our experiments with kissat confirm that a BDD-based SAT solver requires careful attention to the problem encoding, the variable ordering, and the use of quantification in order to outperform a state-of-the CDCL solver.

Tables 1 and 2 provide some performance data for the largest instances solved for each of the four benchmark problems. A first observation is that these problems are very large, with tens of thousands of input variables and clauses.

Looking at the BDD data, the total number of BDD nodes indicates the total number generated by the function GetNode and for which extension variables are created. These are numbered in the millions, and far exceed the number of input variables.

The entries for “Maximum live clauses” show the peak number of clauses that had been added but not yet deleted across the entire proof. As can be seen, these can vary from 7% to nearly 40% of the total clauses. The peak number of live clauses proved to be a limiting factor for the LRAT proof checker.

Figure 14 provides more insight into the nature of the proofs generated by the CDCL solver kissat and the BDD-based solver tbsat. Each point indicates one benchmark run, with the value on the Y axis indicating the runtime of the solver divided by the number of clauses generated, scaled by \(10^6\), whereas the X value is the proof size. In other words, the Y values show the average time in microseconds for each proof clause to be generated. In all, 340 points are shown, with 75 for kissat and the rest for tbsat in its various operating modes.

These data reflect a fundamental difference between how proofs are generated with a CDCL solver versus with a BDD-based solver. A CDCL solver emits a clause each time it encounters a conflict during the search. This may come after many steps involving selecting a decision variable and performing Boolean constraint propagation. Thus, there can be considerable and highly variable amounts of processing between successive clause emissions. We see average times ranging between 4 and 60 microseconds for the kissat runs, and even these averages mask the considerable variations that can occur within a single run.

With a BDD-based solver, on the other hand, the proof has the form of a log describing the recursive steps taken by the BDD algorithm, expressed within a standard proof framework. There is very little variability from one run to the next, and the different evaluation modes have minimal impact. The only trend of note is a general increase in the average time per clause as the proofs get longer. The short runs require less than 1.0 microsecond per clause, whereas the longer ones require over 2.0. This increase can be attributed to the complexity of managing long BDD computations, requiring garbage collection, table resizing, and other overhead operations.

Figure 15 shows a similar plot, but with the Y axis indicating the average time for the proof checkers to check each clause. Again, we see two important characteristics. The proof steps generated by kissat do not include lists of antecedent clauses (hints). Instead, the checking program drat-trim scans the set of clauses and constructs each hint sequence. This takes significant effort and can vary greatly across benchmarks. The proofs generated by tbsat, on the other hand, contain full hints and can therefore be readily checked at an average of around 0.7 \(\mu\) seconds per proof clause, regardless of the proof size or solution method.