DINOS: Data INspired Oligo Synthesis for DNA Data Storage

Information in conventional electronic storage media is usually stored as ordered sequence of binary digits, namely, {0, 1}². We define the bits for a file \(F\) to be the ordered set \(F_B = \lbrace b_0, b_{1}, \ldots b_{z-1} \rbrace\) where \(z\) is the length of the record in bits. Due to the high error rates constructing long strands using current synthesis technologies, we cannot make a single DNA strand long enough to hold a large file. \(F_B\) is divided into chunks of length \(n^{\prime }\) bits that are feasible to synthesize. The length \(n^{\prime }\) can be chosen arbitrarily, even if \(n^{\prime }\) does not divide \(z\) evenly, and the last strand can be padded with extra bits if fixed length payloads are needed. Now file \(F\) is a set of chunks \(F_C = \lbrace C_0, C_1, \ldots C_{\lceil z/n^{\prime } \rceil - 2}, C_{\lceil z/n^{\prime } \rceil - 1}\rbrace\). Each chunk will eventually become a DNA strand.

An important consideration in the design of these chunks is the method by which chunks are reassembled into a complete file after sequencing and decoding, since the process of synthesis and sequencing does not preserve order. There are a variety of approaches that are possible. The most prevalent and densest approach adds additional bits to each strand to represent the index of the chunk within the file [19]. The index of each strand, \(i\), represents its order in the file, and it can be represented with a fixed number of bits, \(\lbrace a_0,\ldots ,a_{log_2({\lceil z/n \rceil })-1}\rbrace\). Hence, all chunks \(C_i \in F_C\) can be represented as follows: \(C_i = \lbrace a_0, \ldots , a_{log_2({\lceil z/n^{\prime } \rceil }) - 1}, b_{i\cdot n^{\prime }}, \ldots b_{n^{\prime } \cdot (i+1)-1} \rbrace\). Note, the placement of the index vector, \(a\), can be anywhere in the strand as long as the decoder knows where to find it during decoding. It is trivial to re-assemble the correct ordering of all bits in \(F_B\) by placing all chunks in order according to their index.

It is also common to add error correction bits to a chunk or to append or insert additional chunks into a file that contain error correction codes for detecting and correcting errors. Such approaches may further increase the size of each chunk or the number of chunks that are ultimately synthesized into DNA [10, 17, 27].

From here on, we will not distinguish indexing bits and error correction bits from the data payload bits, and we will only consider a chunk as an arbitrary sequence of bits, \(C = \lbrace c_0, c_1,\ldots ,c_{n-1}\rbrace\), and each chunk has \(n\) ordered bits. With each chunk partitioned, they can be encoded as a strand of DNA through an encoding function \(E_C: \mathfrak {C} \rightarrow \mathcal {S}\) that represents binary information on the DNA nucleotide alphabet \(\Sigma = \lbrace A,G,C,T\rbrace\), where \(\mathfrak {C}\) is the set of all possible chunks and \(\mathcal {S} \subseteq \Sigma ^L\) is the set of all strands of length \(L\) base pairs that represent each chunk, and we use \(\Sigma ^L\) to denote all possible length \(L\) strands on the four-base DNA nucleotide alphabet.

We assume that the encoding process \(E_C\) adopts a code word based approach in its construction of each DNA strand \(S_i \in \mathcal {S}\). That is, the encoder maintains a set of code words \(\mathcal {C} \subseteq \Sigma ^t\), where each code word is a \(t\)-nucleotide DNA-string. For each of the code words \(E_C\) maintains a bijective mapping function \(f: \mathcal {C} \leftrightarrow \lbrace 0,1\rbrace ^k\) that maps each code word uniquely to a \(k\)-bit symbol such that all \(k\)-bit strings have a corresponding code word. This mapping is also known as a codebook. Using the codebook, the encoder \(E_C\) creates each strand \(S_i \in \mathcal {S}\) by dividing each corresponding chunk into a string of \(k\)-bit symbols defined as \(\mathcal {Z} = \lbrace s_0,s_1,\ldots ,s_{\lceil n/k \rceil -1}\rbrace \mid s_0,s_1,\ldots \in \lbrace 0,1\rbrace ^k\). Through a lookup operation in the codebook, each of the symbols \(s_i \in \mathcal {Z}\) is assigned its corresponding code word from \(\mathcal {C}\). With each symbol assigned its code word, all code words can be concatenated to create the final strand \(S_i \in \mathcal {S}\). A small example that shows the encoding process from chunks to symbols and DNA codewords is given in Figure 1. Constructions of the code word set customarily avoid problematic secondary structures such as long homopolymers that reduce sequencing accuracy and sequences used for addressing mechanisms [10, 14, 16, 17, 27, 39].

After encoding all of the strands associated with a single file, they are bookended by a pair of 20 nucleotide DNA sequences known as primers that are common across all strands in the same file. These primers act as addressing mechanisms that allow biochemical processes to implement random access in the sense that all strands pertaining to a certain file can be separated from the strands encoding other files [10, 23, 27, 39, 40].

In a computational sense, assembly of DNA fragments for the purposes of gene assembly or assembling long DNA strings for data storage can be thought of as a concatenation operation that joins together input DNA strings in a specified order. The order is specified in the bio-chemical process by overlapping regions across sub-sequences that are Watson-Crick pairs. An example illustration of how Watson-Crick pairs drive the assembly order of DNA strands is shown in Figure 2. In this example, the DNA structure consists of three distinct regions as indicated by three unique colors on each initial fragment. The inner black double strand region represents either a sub-sequence that may have been assembled in a previous reaction or a primitive DNA code word representing some symbol of k-bits as described in the previous section. On each side of the double strand are single-stranded regions, denoted as \(o_i\), which drive the ordering of assembly. Each single strand region \(o_i\), known hereafter as overhangs, will join with its Watson-Crick complement overlapping region of the same color notated as \(\overline{o_i}\). This process, known as hybridization, ultimately creates a double strand of DNA. After all complementary overhangs join, the complete final assembly is constructed. This final assembly will have two overhangs itself such that it can be used as sub-assembly in a subsequent reaction. Note that, since the black region is double stranded, it is blocked from participating in anymore hybridization, and thus the single stranded overhangs are the sole determiner of ordering. We refer to a set of all possible overlapping regions that can be used in the process of assembly as \(\mathcal {O}\) and the cardinality of the set as \({\vert {\mathcal {O}}\vert }\).

Because overhangs dictate the ordering of fragments within a strand, repeating an overhang sequence within a reaction will result in an ambiguous ordering amongst fragments and create undesired product. It follows that there is a limit on the length of a strand that can be assembled in a single pot. To construct strands of arbitrary length, hierarchical approaches have been developed to use multiple separate intermediate reactions whose outputs are combined to create longer strands [22, 36, 37]. When designing sets of overhangs, they must be different enough to ensure that there is no non-specific hybridization. If this occurs, like the case of repeating an overhang, then there will be an ambiguous ordering of information. It has been shown in Reference [30] that overhang sets can be constructed such that non-specific hybridization is negligible. Hence, we do not consider effects of non-specific hybridization in our models, but they are an important consideration in the design of such a system.

To this point, we have described a general model that just uses hybridization to concatenate smaller strings together to generate a longer output. In a biological context, one last step is needed to permanently join the fragments together to create one single long linear DNA strand. This can be implemented by designing the overlapping sequences to be four-base sequences that facilitate ligation reactions using a ligase enzyme as is done in Golden Gate Assembly approaches [22, 30, 36, 37], or by designing longer overhang sequences so they can be used as start points for polymerase chain reaction (PCR) as is done in CPEC [31]. However, in all of these approaches, the hybridization of overlapping Watson-Crick sequences is all that is needed to understand the outcome of an assembly. So, we will not cover all of the requisite biochemistry, and we direct the reader to the referenced material.

Figure 3 gives an illustration of the simplest DINOS configuration using a binary code and three overhangs. As in Section 2.2, we assume that assembly is governed solely by hybridization determined by exactly matching Watson-Crick pairs. With only three overhangs, we can at most assemble two oligos at a time to ensure that there is no ambiguity of assembly from repeated overhangs. We refer to such reactions that have no ambiguity in how overhangs should assemble as well formed reactions. Two overhangs are reserved for connecting with adjacent reactions, and one is reserved to connect the oligos together. Hence, we form a binary assembly tree, wherein each reaction takes two oligos and produces a single product. We convert a chunk of 8 bits into a data strand by assembling eight 1-bit code words. Let \(\overline{o}\) denote a complementary sequence to \(o\). Starting from the first code word, we assign overhangs to each end of the code word in a cyclic manner, ensuring that the overhang between two adjacent code words are complementary. For example, the first code word will have left and right overhangs \(o_0\) and \(\overline{o_1}\), respectively, and the second code word will have left and right overhangs \(o_1\) and \(\overline{o_2}\). When the last overhang, \(o_2\), appears as the left overhang at the third code word, the right side wraps around back to \(\overline{o_0}\).

Our arrangement of overhangs has an important key property, namely, it ensures that the entire assembly tree is well formed. Not only are the first set of reactions trivially well formed due to the cyclic assignment of overhangs, but all reactions above them are also well formed. The cyclic arrangement of overhangs ensures that the proper permutation of overhangs is used each reaction such that an overhang is never repeated after grouping two reactions for the next layer of the tree. Furthermore, since each reaction will inherit the left and right overhangs of their left and rightmost children, respectively, the overhang assignment ensures every adjacent reaction will have proper overhangs that can join their products together.

In this system, we only need six basic DNA structures to assemble strands that represent arbitrarily long permutations of symbols. We refer to these basic structures as primitive blocks. There are only three ways that overhangs can be paired, and we need all combinations of code words and these three pairs, resulting in six. For this case, the number of primitive blocks needed is equal to the product of the number of code words and number of overhangs. Next, we generalize this result beyond binary symbols and a set of three overhangs.

We extend the process in Figure 3 to arbitrary overhang sets, strand lengths, and code word sets. Recall from Section 2.1 that we assume an encoding process that breaks chunks of bits into \(k\)-bit symbols such that a chunk can be represented by a multiset of binary symbols: \(\mathcal {Z} = \lbrace s_0,s_1,\ldots ,s_{\lceil n/k \rceil -1}\rbrace \mid s_0,s_1,\ldots \in \lbrace 0,1\rbrace ^k\). Also recall that we assume that the encoder maintains a set of DNA strings \(\mathcal {C}\) such that each string maps uniquely to each \(k\)-bit symbol.

Now, we define the set of primitive blocks, \(\mathcal {U}\), that we use to construct arbitrary sequences. Because we use a cyclic overhang assignment, there are \({\vert {\mathcal {O}}\vert }\) distinct left and right overhang pairs. Therefore, we need \({\vert {\mathcal {O}}\vert }\) versions of each code word for each possible overhang pair. Formally, this set is described as \(\mathcal {U} = \lbrace \,o_j \cdot c \cdot \overline{o_{(j+1) mod {\vert {\mathcal {O}}\vert }}} \ \forall c \in \mathcal {C} \mid o_j \in \mathcal {O},0\le j\gt {\vert {\mathcal {O}}\vert }\rbrace\), where \(\cdot\) is used to explicitly identify the concatenation of the left overhang, code word, and right overhang. Specifying the primitive block like this abstracts away the underlying structure, e.g., a two-strand structure like Figure 2. However, it still conveys the information that an assembly approach must have two complementary versions of an overhang to allow construction to occur. Now, for each symbol in the set \(\mathcal {Z}\) that represents a chunk, we assign the primitive block from \(\mathcal {U}\) that has the appropriate encoding portion and overhang pairs based on the symbol’s position in the set. This in turn creates an ordered multiset of primitive blocks, \(\mathcal {Z}_u\), that has a one-to-one correspondence with \(\mathcal {Z}\), and can be expressed as \(\mathcal {Z}_{u} = \lbrace u_{x} \mid u_{x} \in \mathcal {U},0\le x \lt \lceil n/k \rceil , u_x = o_{x \mod {\vert {\mathcal {O}}\vert }} \cdot c_{x} \cdot \overline{o_{(x+1) mod {\vert {\mathcal {O}}\vert }}}\,\rbrace\) where \(c_{x} \in \mathcal {C}\) is the DNA code word of the corresponding symbol \(s_{x} \in \mathcal {Z}\) at position \(x\) in the ordered set \(\mathcal {Z}\).

Now, we construct an assembly tree of degree \({\vert {\mathcal {O}}\vert }-1\). We collect the primitive blocks at the base of the tree, from left to right, into groups of size \({\vert {\mathcal {O}}\vert }-1\). These groups serve as the inputs to the leaf reactions at the base of the tree. This left-to-right grouping is done at each layer of the tree until the root reaction is reached and a complete tree with degree \({\vert {\mathcal {O}}\vert }-1\) is created. The growth rate, the factor by which the assembled strands grow each layer, is \({\vert {\mathcal {O}}\vert }-1\). Hence, the required height is \(h=\log _{{\vert {\mathcal {O}}\vert }-1}(\lceil n/k \rceil)\), and the number of reactions at layer \(p\), \(\vert \mathcal {R}^p\vert\), can be calculated as \(\frac{{\vert {\mathcal {Z}}\vert }}{({\vert {\mathcal {O}}\vert }-1)^{p+1}}\), assuming \(p=0\) for the leaf layer.

Because reactions in the tree only require a topological ordering, all reactions at layer \(p\), \(\mathcal {R}^p\), can be performed in parallel. Therefore, the time complexity of assembling a strand will be \(O(\log _{{\vert {\mathcal {O}}\vert }-1}(n))\) compared to \(O(n)\) for sequential de novo approaches. Furthermore, the number of nodes in a \(q\)-ary tree is known to be \(\frac{q^{h}-1}{q-1}\), so it follows from our definition of height that the space complexity for synthesizing a single strand is \(O(n)\) for DINOS compared to \(O(1)\) for de novo synthesis assuming that strands can grow longer without increasing the required space.

Figure 4 illustrates the complete process of implementing four different configurations of our algorithm to assemble the same 32-bit chunk. First, the chunk is divided into symbols with the size of \(k=\log _2|C|\) bits. Each symbol is used to generate a 2-tuple where the first value represents the symbol (base-10 in Figure 4) and the second part of the tuple describes the symbol’s position calculated as \(i \ (\mathrm{mod}\ {\vert {\mathcal {O}}\vert })\) for the \(i\)th symbol in the chunk. With the tuples generated for the chunk, each tuple is used to look up the primitive block in the codebook (top left table for each configuration). These primitive blocks from the codebook are then organized into leaf reactions as indicated by the table on the right of each configuration, where \(L_j\) is the \(j\)th leaf from the left of the base of the tree. With the primitive block sets constructed for each leaf, the assembly tree is constructed with a certain height, degree, and reaction count dictated by the configuration. Clearly, each different configuration leads to a different number of primitive block sets per leaf reaction, changes the height of the reaction tree, changes the number of reactions, and leads to different DNA strand (codebook) sizes. We investigate the cost and space implications of each of these configurations along with others in Section 5.

In this section, we provide rigorous definitions for terms used to construct our assembly algorithm. Furthermore, once these definitions are described, we are able to formally define the exact problem that our assembly algorithm targets. We begin by describing the basic building block of assembly synthesis, the primitive block.

In other words, primitive blocks are the most basic building block of assembly, since they are individual code words that encode each \(\lbrace 0,1\rbrace ^k\) of the primitive \(k\)-bit symbols. Thus, these blocks serve as the starting point of assembly and are joined together by reactions that perform concatenations. The definition for such reactions aims to be as abstract as possible to allow any biochemical process that fits these descriptions. We start with a definition for a special concatenation operation, referred to as hybridization concatenation, which reflects the product produced by the hybridization of overlapping Watson-Crick pairs as shown in Figure 2.

There are several characteristics to point out about hybridization concatenations. One, the output of the operation is only defined if the right and left overhangs for the first and second strings, respectively, are Watson-Crick complements.³ This captures the fact that two overlapping regions will not join together due to a lack of Watson-Crick pairing. Second, since the overhang regions necessarily start as single-stranded DNA to enable connections, the hybridization turns these regions into double-strand segments after hybridization (Figure 2).

A major component of a reaction is being able to define what the permutation function \(\sigma\) is. This function defines the output string as it determines the positioning of each input string \(d_i\) in the series of hybridization concatenations that define the output \(R(D)\), and is driven solely by the overlapping regions at each end of each input string. Since we are interested in deterministically producing one unique product for a given reaction, there must exist one unique function \(\sigma\) for a given set of input strings \(D\). If there is, however, more than one possible \(\sigma\) that may be defined for a set \(D\), then there is more than one possible arrangement that will occur with high probability when biochemically implemented. This will also occur if there does not exist a maximal \(\sigma\) that includes all of the \(d_i \in D\) input strings in the set of hybridization concatenations. The existence of a maximal and unique \(\sigma\) leads us to definition of a well formed reaction.

The existence of a permutation function \(\sigma\) depends on the overhangs of each input string, as these determine the possible biochemical interactions that may occur. The following theorem states the overhang requirements for a set \(D\) to ensure a maximal unique \(\sigma\), along with a maximum value for the size of the input set \(D\) given a starting overhang set \(\mathcal {O}\). From this point on, it is assumed that each input string in \(D\) must have a left and right overhang.

By defining well formed reaction and how to construct a set of input strings to a reaction so it is well formed, and by observing that the output strings of reactions can be used as input strings to subsequent reactions, since each output string has a left and right overhang as determined by the corresponding overhangs of the first and last strings in the reaction’s hybridization concatenation sequence, the definition of a well formed reaction network can be constructed.

Finally, this leads us to the definition of the precise problem statement that our algorithm addresses:

Next, we show that our assembly algorithm satisfies the DNA Strand Assembly Problem.

In other words, starting from the root, each node can be treated like a leaf node. Given a node with an arbitrary left and right overhangs, \({\vert {\mathcal {O}}\vert }\) overhangs can be used to ensure the assembly of its \({\vert {\mathcal {O}}\vert }-1\) children. This process can be repeated down the tree until the overhangs of each code word are specified, thus specifying the set of primitive blocks needed to assemble a complete strand. We are now interested in knowing an overhang assignment that minimizes \({\vert {\mathcal {U}}\vert }\), the size of the primitive block set.

The size of the primitive block set for tree assembly is upper bounded by \(({\vert {\mathcal {O}}\vert }^2-{\vert {\mathcal {O}}\vert })\cdot 2^k\) and lower bounded by \({\vert {\mathcal {O}}\vert }\cdot 2^k\). The upper bound arises from a naive overhang assignment that allows for any two unique overhangs to be assigned to the left and right overhangs of a child. Thus, given the fact an overhang cannot repeat such that the left and right overhangs are equal, there are \(({\vert {\mathcal {O}}\vert }^2-{\vert {\mathcal {O}}\vert })\) possible overhang pairs for each code word. The primitive block set cannot be lower than \({\vert {\mathcal {O}}\vert }\cdot 2^k\), because there needs to be at least \({\vert {\mathcal {O}}\vert }-1\) versions of a given code word to assemble a \({\vert {\mathcal {O}}\vert }-1\) long repetition of the same code word in a leaf reaction. Furthermore, the fact that the right overhang is unique in the leaf reaction implies that there is at least one more version of the code word with this overhang as its left overhang. This other version would be required to to facilitate the assembly of a \({\vert {\mathcal {O}}\vert }\)-run of a code word, because the assembly of the first \({\vert {\mathcal {O}}\vert }-1\) would need to be subsequently assembled with \({\vert {\mathcal {O}}\vert }\)-th code word.

We identify three different optimization techniques for coalescing reactions. The ideal optimization allows for two reactions to be coalesced if they produce the same data even if they do not necessarily share the same bookend overhangs. This serves as a lower bound on the number of reactions that are needed to synthesize some data. This technique is not implementable in a real system but is useful for analysis. We also consider an optimization that searches through all of the reactions of an un-optimized assembly tree and coalesces all reactions that share the same overhangs and produce the same data. We call this base optimization base-opt. In our studies, a significant gap exists between the ideal and base-opt, and to address this, we study an alignment technique to facilitate more reaction coalescing.

To facilitate more occurrences of reactions producing the same data with matching overhangs, we develop an alignment technique that trades off storage density and primitive block set size for reaction elimination opportunities. The key idea of alignment padding is that we can leverage the cyclic assignment to adjust overhangs of an arbitrary reaction by inserting padding to its left, thereby shifting overhang assignment and creating a match that enables coalescing. We refer to such padding as a no-operation (NOP) reaction, since it represents no symbol. Figure 5 illustrates the idea by inserting NOP reactions into the assembly tree to shift a reaction’s overhangs to align with the overhangs of another reaction producing the same data. These NOP reactions can be thought of as a subtree that has reactions that only assemble NOP strands. All reactions to the right of the NOP will be shifted by the same amount as the target reaction.

To the level of the tree of which padding reactions are added, they are interpreted like any other reaction. So, if NOP reactions are added to layer \(p\), then there must be an appropriate number of reactions at layer \(p+1\) to handle the additional load. Thus, NOP padding adds reactions to the tree. However, the NOP reactions of multiple alignment transformations can be effectively bundled together, limiting the number of new reactions added at layer \(p+1\). Because reactions group \({\vert {\mathcal {O}}\vert }-1\) children, if multiple alignment transformations collectively add less NOP reactions than \({\vert {\mathcal {O}}\vert }-1\), then only one additional reaction is needed at \(p+1\). Thus, for the overhead of one additional reaction, multiple reactions can be transformed to be redundant and optimized out. Note, this bundling is only possible if the padding reactions for multiple transformations are inserted for data that are assembled on the same strand. This is because insertions on the assembly tree of one strand are not visible to the grouping process of an assembly tree for another strand of data. So, insertions must be applied appropriately so the overall number of reactions do not increase.

To avoid such situations, we propose a heuristic based on two metrics: distance and re-use count before padding. Distance is the number of NOPs that must be inserted to transform the overhangs of certain reactions. Placing an upper bound on the distance helps avoid using too many NOPs for any one redundancy operation and allows more redundancy operations to be bundled together in a reaction. Through experimentation, we found that the distance must be less than \(({\vert {\mathcal {O}}\vert }-1)/4\). This means that each padding operation at most takes up \(1/4\) of the capacity of a parent node. Given enough opportunity in a single strand, at least four padding transformations can be amortized for a single overhead parent reaction.

We also consider the number of times a reaction is re-used in the base-opt case. The reason to consider this count is that it reflects the amount of times that a single transformation will have to be applied. Applying a transformation to an already highly redundant base-opt reaction may simply add overhead. Through experimentation, we determined that if the re-use count exceeds a threshold of 10, then it is already sufficiently redundant, and we do not perform any alignment padding, since it may only increase reactions.

There are also several optimizations that can be applied to NOP trees so reaction count increases and density loss is limited. For NOP reactions, no physical reaction must be implemented. Instead, only a single NOP primitive block need be inserted at the root of the NOP sub-tree. For example, from Figure 5, the NOP sub-trees for \(NOP^p_1\) and \(NOP^p_2\) can be replaced by NOP primitives with (left,right) overhang pairs \((o_0,\overline{o_1})\) and \((o_1,\overline{o_2})\), respectively. From Corollary A.1, we show that there are two unique possible rotations through the set of overhangs at any level in the tree. So, to have enough NOP strands for each rotation, the primitive set will need to be expanded by \(2{\vert {\mathcal {O}}\vert }\) strands. Furthermore, as shown in Figure 5, consecutive NOP strands in a reaction may be compressed to one single strand. However, this requires an additional \({\vert {\mathcal {O}}\vert }^2-{\vert {\mathcal {O}}\vert }\) primitive blocks to accommodate all possible overhang permutations that may arise from this optimization.

The two parameters that drive the primitive block set size and the assembly algorithm are symbol size (k) and overhang set size. First, we consider (1 bit, 3 overhangs). This configuration requires the largest number of reactions, because it has the fewest symbols and the fewest parts are assembled in each reaction. The last two columns of Table 1 report the total reactions needed to assemble each file under the case of (1 bit, 3 overhangs). The case of uncompressed and compressed files are reported in columns 6 and 7, respectively. Note, no optimizations are included in these results. Since this configuration requires the largest number of reactions, we normalize all other configurations to it.

Table 2 shows the geometric mean of normalized reaction counts across all files with no optimizations applied, where all reaction counts for each (symbol size, overhang count) pairing are normalized to that of (1 bit, 3 overhangs). This table shows that increasing both parameters has a strong impact on the number of reactions. For example, increasing to (2,3) reduces the reaction count to half, and (1,5) reduces the reaction count by a factor of approximately 3. However, as symbol length and overhang count are increased, the effect diminishes, for example moving from (1,5) to (1,9) results in only a 2.36\(\times\) reduction. This suggests diminishing returns, where increasing these parameters leads to less and less benefit and also that overhangs have a bigger impact than scaling symbol length for the same factor increase.

Last, it should be noted that scaling \(k\) exponentially increases the primitive block size, \({\vert {\mathcal {U}}\vert }\), while scaling \({\vert {\mathcal {O}}\vert }\) only linearly scales it. Hence, \(\vert \mathcal {U}_{(1,3)}\vert =6\) while \(\vert \mathcal {U}_{(8,3)}\vert =768\). For a similar improvement in reactions as (8,3) has over (1,3), we can select (1,9) where \(\vert \mathcal {U}_{(1,9)}\vert =18\), in other words, \(42 \times\) fewer primitive blocks.

Next, we evaluate the amount of redundancy that exists while assembling strands for a file. The overhang counts are varied from 3–65, and the symbol size is kept constant at 1 bit. This ensures that we study a wide range of data granularities and that will be important when understanding the effects of optimizations. Note, hereinafter, we report relative values for the reaction counts associated with each optimization. Using the last two columns of Table 1 and the data of Table 2, raw reaction counts can be computed for each configuration. Though, for convenience, we also include raw reactions for every configuration and benchmark in our supplementary document.

Figure 6(a) shows ideal’s reaction count normalized to no optimization and averaged using geometric mean across the corpus. Recall that ideal ignores overhangs and assumes that as long as the same sequence of symbols is produced that the reactions can be coalesced. The general trend is that as the number of overhangs increases, the effectiveness of optimization decreases. This is due to the increase in the granularity of information created at each layer of the tree that makes it less likely to find redundant data. As expected, some files have more redundancy than others. For example, with 3 overhangs, the number of reactions for nci can be reduced by 99% relative to no optimization, and for 65 overhangs reactions can be reduced by 92.23%. This is in contrast to the average reduction in reactions of 97.59% and 57.6% for 3 and 65 overhangs, respectively.

Now, we compare ideal with base-opt in Figures 6(b) and 6(c). Recall that base-opt requires that the bookend overhangs of two reactions match in addition to their data sequences matching. Even when we take into account the restriction of matching overhangs for redundant data, there is a fair amount of redundancy available. On average, base-opt is able to reduce reactions by 97.23% and 28.83% for 3 and 65 overhangs, respectively. Although base-opt is effective at \({\vert {\mathcal {O}}\vert }=65\), the ideal case can remove an additional 28.77% portion of no-opt’s reactions. Figure 6(c) provides a direct comparison between the two optimizations by normalizing the reaction count of the ideal case to that of base-opt. On average, the disparity between the two grows as the number of overhangs increases. This is due to the increasing difficulty for base-opt to take advantage of redundancy due to a greater chance of mismatched overhangs.

The added requirement of matching overhangs effectively creates more information states that need to be assembled compared to that of ideal, and theoretically it can increase the number of states by a factor that is at most \({\vert {\mathcal {O}}\vert }\).

For example, when \({\vert {\mathcal {O}}\vert }=17\) for dickens, there are only 2,344 unique 16-bit information states each repeated 2,174 times on average, while sao uses all \(2^{16}\) unique 16-bit information states each repeated only 55 times on average. When considering overhangs, both dickens and sao see similar increases in unique states of 11.2\(\times\) and 12.88\(\times\), respectively. However, since sao uses all \(2^{16}\) states, the overhangs inject more total states that must be synthesized than for dickens. This leads to a larger relative reaction gap when comparing ideal and base-opt across these two benchmarks, e.g., ideal removes 71% and 6.45% of base-opt’s reactions for sao and dickens, respectively.

Figure 7(a) compares the number of reactions of ideal against the number of reactions for no optimization when compressing files with zpaq. Since compression removes redundancy, the ideal optimization is no longer able to find significant redundancy when \({\vert {\mathcal {O}}\vert }=65\), resulting in only 0.02% reduction in reactions relative to no optimization on average. However, for smaller \({\vert {\mathcal {O}}\vert }\) that have more layers with smaller data granularities, the relative reductions for the compressed files compared to the de-compressed files are similar on average. For 2 and 17 overhangs, reductions for the compressed files are still 93.27% and 87.46% as compared to 97.58% and 93.96% for de-compressed files.

When comparing base-opt and ideal for compressed files, the largest discrepancy between optimizations occurs at 17 overhangs. On average, as shown in Figure 7(b), ideal can additionally remove 78.7% of base-opt’s reactions for this \({\vert {\mathcal {O}}\vert }\). This large difference can be explained by studying the distributions of information states for the compressed files when \({\vert {\mathcal {O}}\vert }=17\). We found that the compression algorithm expands the number of information states required for synthesizing a file and the variance of the frequencies for each information state is much smaller than that of decompressed files. For example, the uncompressed dickens had 2,344 states at the first level of the assembly tree when \({\vert {\mathcal {O}}\vert }=17\) with an average frequency for each information state of 2,174 with variance across the frequencies of \(75\cdot 10^6\). With compression, it used all \(2^{16}\) possible states with an average re-use count of 16 and a variance of 17. This explains the considerable gap when compared to no compression. Also, since \({\vert {\mathcal {O}}\vert }=17\) leads to larger data granularities, thus more information states that get multiplied by a large multiplier, this leads to larger relative increases in reactions as compared to smaller \({\vert {\mathcal {O}}\vert }\) during compression. This effect is similar to why dickens behaved better previously with base-opt for decompressed data compared to sao.

Interestingly, when comparing the total number of reactions for base-opt both before and after compression, there are overhang set size configurations that result in smaller reaction counts for decompressed files. This tends to occur primarily for \({\vert {\mathcal {O}}\vert }=17\) and for decompressed files that tend to have a small number of information states at this granularity, e.g., mr. We find in this case that decompressed version of mr has 60% less reactions than its compressed counterpart, the largest of any record studied in this work. This implies that the large scale of reaction coalescing afforded by the low entropy of information states for certain files and granularities can be more effective for reducing reactions rather than increasing the entropy and reducing the absolute amount of data to be synthesized with decompression.

We also compared the number of reactions before and after optimization for two files filled with random data. One with approximately as many bytes as the average file size before compression (17 MB) and another smaller random file of size equal to the average file size of compressed data (3 MB). We found that ideal optimization reduces the reactions by nearly the same amount compared to compressed files with similar size. The reason reactions can be easily removed from random data in some cases is because for small granularities (\(\lt \!\!16\) bits), it is inevitable that they will be repeated for the file sizes we study. This also indicates the compression used here is approximate to random files in terms of entropy, which makes sense given the larger number of states with lower variance mentioned before for dickens. The data for random files can be found in the supplementary information.

To understand how the entropy of the synthesized data may influence the choice of symbol size and overhang count when considering total reactions, Table 3 was constructed by varying the symbol size (in bits) and overhang count for both the ideal and base-opt case of removing redundancy. The values represent the geometric mean across all decompressed files in the Silesia corpus of the normalized reaction counts, where normalization is done with respect to the (1 bit, 3 overhangs) case. In Section 5.1, we showed that when applying no redundancy optimization, that although there are diminishing returns with larger overhang counts and symbol sizes, the total number of reactions decreases monotonically. In contrast, Table 3 shows that when optimization is factored in, the reaction count can actually increase when increasing both overhang count and symbol size. For example, ideal experiences a 1.64\(\times\) increase and base-opt sees a 1.86\(\times\) increase in reactions when going from (1 bit, 17 overhangs) to (1 bit, 65 overhangs). Both optimizations also see a 1.37\(\times\) increase in reactions when increasing symbol size from (1 bit, 3 overhangs) to (3 bit, 3 overhangs). While increasing both parameters theoretically decreases tree size as shown in Section 5.1, the tree constructed with larger symbol/overhang set sizes will have larger data granularities at the same corresponding levels than that of smaller overhang/symbol sizes. If the smaller granularity facilitates enough redundancy elimination, then the benefits of a larger number of symbols or overhangs in terms of the number of reactions are lost due to a lack of redundancy. We find similar patterns in compressed files, but the results are omitted to save space.

Figure 8 shows the percent change in reactions relative to base-opt when applying the alignment technique (Section 4.2) with the same parameters as Section 5.2. A negative percent change indicates the additional percentage of reactions removed over base-opt (better), and percent increases indicate that reactions are added (worse). These results show that adding padding reactions can decrease the reaction count for both compressed and decompressed files. Overhang set sizes of 3 and 5 are not present, since they are not large enough to satisfy the \(|\mathcal {O}-1|/4\) distance stipulation.

On average for decompressed files, we were able to reduce reaction counts by 0.88%, 4.35%, and 15.48% relative to base-opt for 9, 17, and 65 overhangs, respectively. Large benefits were observed for files where the addition of overhangs distributes the redundancy found by ideal over many infrequently used information states in base-opt. For example, sao experiences the largest decrease (28%) for 17 overhangs due to its larger information set that had less redundancy before alignment. For decompressed files, when \({\vert {\mathcal {O}}\vert }=65\) every file except x-ray sees decreases in reactions, since redundancy is limited at this larger granularity in the ideal case, and distributing across many overhangs leads to many opportunities that fit the requirements for padding.

As discussed in Section 5.2, zpaq tends to increase the information set size for granularities like 16 bits, with the redundancy for each state being similar. Small data redundancy results in many unique states when considering overhangs, leading to many low-cost opportunities for padding. This results in a 30% decrease on average for \({\vert {\mathcal {O}}\vert }=17\).

As mentioned in Section 4.2, there is indeed a tradeoff when improving data alignment to facilitate more sharing. In our implementation, we assume a system where both rotations of overhangs in the tree are synthesized with NOP codewords. For the 1-bit symbol sizes studied here, this leads to an approximate increase of \(2\times\) in the number of primitive blocks needed. A similar set size increase would be to increase \(k\) from 1-bit to 2-bits. However, Table 3 shows us that increasing the set size this way could in fact lead to more reactions for \({\vert {\mathcal {O}}\vert }=17\). The second tradeoff is in density, and so we measure the number of codewords inserted per strand on average for each benchmark of Figure 8(a) (decompressed data). The resulting geometric mean for the number of NOP codewords added per strand across the benchmarks for the three overhang settings \({\vert {\mathcal {O}}\vert }=9,17,65\) is 2.00, 7.49, 40.74, respectively. Assuming NOP codewords are the same length as symbol codewords, this leads to just a 0.9% additional codeword overhead at most considering the worst-case average of 40.74 codewords/strand and 1-bit symbols.

Our analysis up to this point is independent of the underlying chemical implementation of the assembly process. To further motivate the benefits of DINOS, it is necessary to pick an implementation to understand the approach’s cost implications. For DINOS, there are two costs: costs of chemistries to assemble strands and costs to acquire the primitive blocks. For the former, we assume costs associated with a ligase enzyme that can be used to permanently connect together primitive blocks that have a structure like those blocks in Figure 2, and for the latter, we assume a column-based de novo synthesis approach. We compare this to using a array-based de novo synthesis approach for a whole set of data. An overview of these synthesis approaches, and why we choose these, can be found in Appendix Section A.3.1.

Figure 9(a) compares the cost of using DINOS to the cost of using de novo for the synthesis of the entire dataset. Each curve shows a relative cost for a select primitive block composition. Details about assumptions, the equations, and parameters used to generate these curves can be found in the Appendix Section A.3.2. The purpose of this cost model is not to consider detailed costs that would be incurred for developing a platform capable of large-scale data synthesis, but to rather compare the raw material cost incurred for constructing the DNA strands themselves. To be consistent with the evaluation analysis in the previous section, the strand length is assumed to be 4,096 bits. We assume de novo synthesis scales at a fixed cost/base in spite of low yields at such long lengths. Also note that we do not assume any cost reductions for coalescing reactions in this analysis, which may substantially lower the cost of DINOS even more.

There are three main costs associated with the materials required to construct DNA strands that encode information using assembly. One is that there is a cost associated with the procurement of the primitive block set. In this analysis, we assume that de novo synthesis is used to synthesize the single-stranded DNA needed to construct the single-double-single strand structure depicted in Figure 2. Therefore two unique strands must be synthesized per primitive block. The second cost of assembly is replenishing the primitive block set once all strands initially synthesized are used during assembly. We assume primitive block usage is evenly distributed across the set. Last, the third cost is that of materials such as enzymes and buffers that are needed to implement assembly. It is assumed that such materials are dependent on both the height of the assembly tree, and the number of DNA fragments that must be connected together in a single reaction.

When comparing these three costs against de novo synthesis for a volume of synthesized data, we can see from Figure 9(a), for the entire range (\(10^9\)B to \(10^{15}\)B), that DINOS based on ligation chemistry is at least five orders of magnitude less than de novo synthesis. For small dataset sizes, the larger primitive block sets incur higher costs than that of smaller sets. This is because the dataset is not large enough to leverage the larger set of primitive blocks. However, this relationship generally inverts as the dataset increases. This is mainly due to reaction costs dominating the overall cost, because we estimate that each code word pool can be utilized on the order of \(10^{12}\) times (Appendix A.3). This shows that it is cost-effective, from a consumable point of view, to decrease reactions by utilizing larger primitive block sets.

Another important trend is that, as dataset sizes increase for each configuration, the cost savings of using assembly improve relative to de novo and then asymptotically approach a minimum normalized cost. This trend occurs because at smaller dataset sizes, the cost of creating the initial set of strands is large compared to the cost of reactions and the cost of refilling the initial set. As the dataset becomes larger, this initial cost becomes negligible and the cost of assembly becomes dependent on two terms that are both proportional to the dataset size, resulting in the ratio of assembly costs to that of de novo costs to reach a constant value.

Our analysis concludes with a more detailed treatment of the choice of the composition of the primitive strand set. As mentioned previously, as the dataset to be synthesized approaches very large volumes, the larger primitive set typically results in lower costs. However, Figure 9(a) shows a contradiction to this statement. This can be seen by comparing the costs between the configurations \((k=1,{\vert {\mathcal {O}}\vert }=65)\), which has 130 primitive blocks, and \((k=4,{\vert {\mathcal {O}}\vert }=3)\), which has 48 blocks. Although the former has more than 2.7\(\times\) more primitive blocks, at large dataset sizes it is about a factor of 2 more expensive. The reason for this is that increasing overhangs mainly affects cost by reducing the amount of materials by decreasing the height of the assembly tree. Since the height of an assembly tree is rapidly reduced as overhang set size increases from the minimum of \({\vert {\mathcal {O}}\vert }=3\), cost savings also initially rapidly reduce. However, eventually increasing overhang set size has small impacts on height, thus increasing primitive block set sizes with minimal benefits. This indicates that choosing a configuration that minimizes cost for some desired primitive block set size requires carefully balancing both the number of code words and overhangs.

To better illustrate the impact of set construction on cost, Figure 9(b) shows the cost per bit against different primitive block set sizes for various combinations of code word and overhang set sizes. When keeping the symbol size constant and increasing overhangs for the curve shown by \(k=3\), we see that there are large cost decreases initially when increasing \({\vert {\mathcal {O}}\vert }\). Eventually, as the cost benefits are exhausted from increasing the overhang set size for this symbol size, a curve corresponding to \(k=4\) appears directly beneath the curve of \(k=3\). This implies that increasing overhangs to reduce cost for \(k=3\) is no longer the optimal decision for increasing the primitive block set and thus the code word set should be expanded. However, eventually there are also diminishing returns for increasing the symbol size, since the number of leaf reactions are inversely related to the symbol size. This causes the curves to group closer in terms of cost, requiring large increases in the primitive block set size to gain relatively small cost benefits. For example, between 6 and 1,000 primitive blocks, costs can be decreased by a factor of 10.7. However, from 1,000 to \(10^6\) primitive blocks, the cost is only reduced by a factor of 2.2. This implies that most of the cost benefits can be realized with a relatively small number of primitive blocks.

Since overhangs do not carry information in DINOS, they will negatively impact the density of the storage system. Table 4 provides a comparison for different symbol sizes on the density of DINOS compared to a standard short-strand that does not include overhangs. For our density analysis, we assume strands that encode 4,096 bits where each symbol is encoded to DNA using a base 3 representation[10, 27], 4 base overhangs [22, 30, 36, 37], and a total primer length of 40 bases [10, 27, 40]. For Short de novo synthesis, we keep the same encoding and primer overhead and assume a maximum strand length of 200 bases [9], but there are no overhangs. Formulas for calculating these densities are given in Section A.2. The table shows that for a binary code the overhangs have a significant effect on density, however, the gap diminishes as the symbol size is scaled up to a 26-bit code. The density of DINOS surpasses a short strand at 28 bits, since Short de novo is ultimately limited in density by the large fraction of bases devoted to its primers. However, if de novo synthesis progresses such that error rates are small in long strands, then the encoding density will be largely determined by the conversion from binary symbols to DNA. This results in Long de novo being more dense than assembly and is shown in the last row of Table 4. However, achieving strand lengths past 200 bases is difficult. Even with current de novo efficiencies being 99.5%, the yield rate is approximately just 36% for 200 base strands [9]. So, while the overhang sequences do add overhead, DINOS can have competitive density with de novo synthesis.

Now, we compare our assembly algorithms with existing techniques to better understand how efficiently overhangs are used from both a storage density and an algorithmic time and space complexity standpoint. We compare against Metclo and Short de novo oligo synthesis. We choose Metclo’s assembly algorithm [22] as a comparison point, since they provide an algorithm that will work for any \({\vert {\mathcal {O}}\vert }\), and the overhang overhead inserted into strands is identical to many other Golden Gate gene assembly methods such as Goldenbraid 2.0, Moclo, Loop Assembly, and so on [6, 29, 36, 37, 43]. Metclo is able to assemble \({\vert {\mathcal {O}}\vert }-1\) oligos in each reaction, and hence achieves identical time and space complexity to our work. However, it incurs more overhead from overhangs, requiring three overhangs between code words that are joined outside of leaf reactions. We give an in-depth treatment of Metclo in Appendix A.1.1.

To compare density of our assembly method with Metclo, we calculate the minimum symbol size so density is equivalent to the density of Short de novo synthesis. Using that same analysis, we calculate the percent increase in storage density compared to Metclo that our work provides. We do this by taking a strand designed for each system, counting the bases that contribute to overhead, and then calculating the resulting density.

The left chart of Figure 10 solves for the upper bound of overhang length that still allows for equivalent density to de novo synthesis for assembly approaches. A line is drawn at the 4 base marker. Any symbol size and overhang length combinations that fall in between the lower bound and the upper bound (inclusive) indicate that the assembly approach will have an equal or higher storage density than de novo synthesis. We see that our approach intersects the lower bound at 27 bits compared to 31 bits for Metclo configured with \({\vert {\mathcal {O}}\vert }=17\). Note, our system results in the same density regardless of the number of overhangs, but for Metclo, increasing overhang set size reduces the number of boundaries in which there are three overhangs between code words resulting in the upper bound curves shifting to the left. This implies that even if Metclo and our approach had coinciding intersection points, Metclo would require a larger primitive block set, making it less desirable.

The chart on the right of Figure 10 normalizes the storage density of our algorithm to Metclo. Since symbol size and overhang set size decrease the number of times in which 3 overhangs are inserted between code words, the gap between the density of Metclo and our approach decreases with larger numbers of symbols. At the point of the smallest primitive block set size we support (1 bit, 3 overhangs), our approach provides 80% greater storage density. Closing the gap between storage densities requires Metclo to grow their primitive block set. For example, if Metclo uses 17 overhangs and our approach uses 3, for 5.3\(\times\) more strands ((17-1)/3), Metclo can reduce the gap to 10% at 25-bit symbols. Raw densities are also included for Metclo in Table 4 for \({\vert {\mathcal {O}}\vert }=3,5\) and a subset of symbol sizes shown in Figure 10.

The number of bases inserted by overhangs for our system, \(B_{o}\), can be calculated as:where \(L_o\) is the length of overhangs in bases, \(I_{ha}\) is the length in bits of a strand that can be assembled by hierarchical assembly, and \(k\) is symbol size in bits. This expression counts the number of overhang bases on the right of each code word plus the overhang bases on the left of the first code word in the strand. For Metclo, an additional term is required to factor in the cases in which there are three overhangs between code words. This will occur at every boundary between two sub-assemblies of size \(k\cdot ({\vert {\mathcal {O}}\vert }-1)\) bits, and the total overhang bases for Metclo, \(B_{o,Metclo}\) can be expressed as the following:The first two terms calculate the need for at least one overhang in between codewords, and the third term accounts for the two extra overhangs between code words that are not assembled in the leaves. With the number of overhang bases inserted into the final strand known for both approaches, we can now calculate the density (bits/base) of a hierarchical assembly system, \(D_{ha}\). Generically, this can be described with the following equation:where \(L_p\) is the number of bases allocated in the strand for primers, \(\sigma _{ha}\) is the raw encoding density for the encoding process that converts payload information into bases for hierarchical assembly synthesis, \(B_o\) is the total number of overhand bases in a strand constructed with assembly, \(k\) is the length in bits for a symbol. Finally, we calculate the density of a de novo synthesis system to be able to calculate operating points in which assembly is equally dense. The density of de novo synthesis, \(D_{dn}\), can be calculated as:where \(\sigma _{dn}\) is the raw encoding density for the process that converts payload information into bases for de novo synthesis, and \(B_{dn}\) is the length (in bases) of a strand that can be grown by de novo synthesis. With the densities of the two synthesis approaches known, we solve the inequality of Equation (5) for \(L_o\) for both our assembly system and Metclo. This provides an upper bound, as shown in Figure 10, for overhang length that is allowable for the density of assembly synthesis to be equivalent to de novo synthesis.

As pointed out in Section 4.2, if there are only two unique rotations through the overhang set \(\mathcal {O}\), then only \(2{\vert {\mathcal {O}}\vert }\) NOP strands are required to represent the result of any NOP tree root. The following corollary proves this and provides the exact overhang rotations that exist at each level of the tree, allowing for NOP strands to be fully specified.