research-article

Lazy Lempel-Ziv Factorization Algorithms

Authors:

Juha Kärkkäinen,

Simon J. PuglisiAuthors Info & Claims

Journal of Experimental Algorithmics (JEA), Volume 21

Article No.: 2.4, Pages 1 - 19

https://doi.org/10.1145/2699876

Published: 21 October 2016 Publication History

Abstract

For decades the Lempel-Ziv (LZ77) factorization has been a cornerstone of data compression and string processing algorithms, and uses for it are still being uncovered. For example, LZ77 is central to several recent text indexing data structures designed to search highly repetitive collections. However, in many applications computation of the factorization remains a bottleneck in practice. In this article, we describe a number of simple and fast LZ77 factorization algorithms, which consistently outperform all previous methods in practice, use less memory, and still offer strong worst-case performance guarantees. A common feature of the new algorithms is that they compute longest common prefix information in a lazy fashion, with the degree of laziness in preprocessing characterizing different algorithms.

References

[1]

1000 Genomes Project Consortium. 2015. A global reference for human genetic variation. Nature 526, 7571 (2015), 68--74.

[2]

A. Abeliuk, R. Cánovas, and G. Navarro. 2013. Practical compressed suffix trees. Algorithms 6, 2 (2013), 319--351.

[3]

A. Al-Hafeedh, M. Crochemore, L. Ilie, E. Kopylova, W. F. Smyth, G. Tischler, and M. Yusufu. 2012. A comparison of index-based Lempel-Ziv LZ77 factorization algorithms. ACM Comput. Surv. 45, 1 (2012), 5:1--5:17.

Digital Library

[4]

G. Badkobeh, M. Crochemore, and C. Toopsuwan. 2012. Computing the maximal-exponent repeats of an overlap-free string in linear time. In Proceedings of the 19th International Symposium on String Processing and Information Retrieval (SPIRE’12), Lecture Notes in Computer Science, Vol. 7608. Springer, 61--72.

Digital Library

[5]

M. Burrows and D. J. Wheeler. 1994. A Block Sorting Lossless Data Compression Algorithm. Technical Report 124. Digital Equipment Corporation, Palo Alto, California.

[6]

M. Charikar, E. Lehman, D. Liu, R. Panigrhy, M. Prabhakaran, A. Sahai, and A. Shelat. 2005. The smallest grammar problem. IEEE Trans. Inf. Theory 51, 7 (2005), 2554--2576.

Digital Library

[7]

G. Chen, S. J. Puglisi, and W. F. Smyth. 2008. Lempel-Ziv factorization using less time and space. Math. Comput. Sci. 1, 4 (2008), 605--623.

[8]

M. Crochemore and L. Ilie. 2008. Computing longest previous factor in linear time and applications. Inform. Process. Lett. 106, 2 (2008), 75--80.

Digital Library

[9]

M. Crochemore, L. Ilie, C. S. Iliopoulos, M. Kubica, W. Rytter, and T. Walen. 2009. LPF computation revisited. In Proceedings of the 20th International Workshop on Combinatorial Algorithms (IWOCA’09), Lecture Notes in Computer Science, Vol. 5874. Springer, 158--169.

Digital Library

[10]

M. Crochemore, L. Ilie, and W. F. Smyth. 2008. A simple algorithm for computing the Lempel-Ziv factorization. In Proceedings of the 2008 Data Compression Conference (DCC’08). IEEE Computer Society, 482--488.

Digital Library

[11]

J.-P. Duval, R. Kolpakov, G. Kucherov, T. Lecroq, and A. Lefebvre. 2004. Linear-time computation of local periods. Theor. Comput. Sci. 326, 1--3 (2004), 229--240.

Digital Library

[12]

H. Ferrada, T. Gagie, T. Hirvola, and S. J. Puglisi. 2014. Hybrid indexes for repetitive datasets. Philos. Trans. R. Soc. A 372, 2016 (2014).

[13]

P. Ferragina and G. Manzini. 2005. Indexing compressed text. J. ACM 52, 4 (2005), 552--581.

Digital Library

[14]

P. Ferragina and G. Manzini. 2010. On compressing the textual web. In Proceedings of the 3rd ACM International Conference on Web Search and Data Mining (WSDM’10). ACM, 391--400.

Digital Library

[15]

J. Fischer and V. Heun. 2011. Space-efficient preprocessing schemes for range minimum queries on static arrays. SIAM J. Comput. 40, 2 (2011), 465--492.

Digital Library

[16]

T. Gagie, P. Gawrychowski, J. Kärkkäinen, Y. Nekrich, and S. J. Puglisi. 2012. A faster grammar-based self-index. In Proceedings of the 6th International Conference on Language and Automata Theory and Applications (LATA’12), Lecture Notes in Computer Science, Vol. 7183. Springer, 240--251.

Digital Library

[17]

T. Gagie, P. Gawrychowski, and S. J. Puglisi. 2011. Faster approximate pattern matching in compressed repetitive texts. In Proceedings of the 22nd International Symposium on Algorithms and Computation (ISAAC’11), Lecture Notes in Computer Science, Vol. 7074. Springer, 653--662.

Digital Library

[18]

K. Goto and H. Bannai. 2013. Simpler and faster Lempel Ziv factorization. In Proceedings of the 2013 Data Compression Conference (DCC’13). IEEE Computer Society, 133--142.

Digital Library

[19]

D. Gusfield and J. Stoye. 2004. Linear time algorithms for finding and representing all the tandem repeats in a string. J. Comput. Syst. Sci. 69, 4 (2004), 525--546.

Digital Library

[20]

C. Hoobin, S. J. Puglisi, and J. Zobel. 2011. Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections. PVLDB 5, 3 (2011), 265--273.

Digital Library

[21]

J. Kärkkäinen, D. Kempa, and S. J. Puglisi. 2012. Slashing the time for BWT inversion. In Proceedings of the 2012 Data Compression Conference (DCC’12). IEEE Computer Society, 99--108.

Digital Library

[22]

J. Kärkkäinen, D. Kempa, and S. J. Puglisi. 2013a. Lightweight Lempel-Ziv parsing. In Proceedings of the 12th International Symposium on Experimental Algorithms (SEA’13), Lecture Notes in Computer Science, Vol. 7933. Springer, 139--150.

[23]

J. Kärkkäinen, D. Kempa, and S. J. Puglisi. 2013b. Linear time Lempel-Ziv factorization: Simple, fast, small. In Proceedings of the 24th Annual Symposium on Combinatorial Pattern Matching (CPM’13), Lecture Notes in Computer Science, Vol. 7922. Springer, 189--200.

[24]

J. Kärkkäinen, G. Manzini, and S. J. Puglisi. 2009. Permuted longest-common-prefix array. In Proceedings of the 20th Annual Symposium on Combinatorial Pattern Matching (CPM’09), Lecture Notes in Computer Science, Vol. 5577. Springer, 181--192.

[25]

J. Kärkkäinen and S. J. Puglisi. 2010. Medium-space algorithms for inverse BWT. In Proceedings of the 18th Annual European Symposium on Algorithms (ESA’10), Lecture Notes in Computer Science, Vol. 6346. Springer, 451--462.

Digital Library

[26]

J. Kärkkäinen, P. Sanders, and S. Burkhardt. 2006. Linear work suffix array construction. J. ACM 53, 6 (2006), 918--936.

Digital Library

[27]

D. Kempa and S. J. Puglisi. 2013. Lempel-Ziv factorization: Simple, fast, practical. In Proceedings of the 2013 Workshop on Algorithm Engineering and Experiments (ALENEX’13). SIAM, 103--112.

Digital Library

[28]

T. Kociumaka, M. Kubica, J. Radoszewski, W. Rytter, and T. Walen. 2012. A linear time algorithm for seeds computation. In Proceedings of the 23th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’12). SIAM, 1095--1112.

Digital Library

[29]

R. Kolpakov, G. Bana, and G. Kucherov. 2003. mreps: Efficient and flexible detection of tandem repeats in DNA. Nucleic Acids Res. 31, 13 (2003), 3672--3678.

[30]

R. Kolpakov and G. Kucherov. 2003. Finding approximate repetitions under Hamming distance. Theor. Comput. Sci. 303, 1 (2003), 135--156.

Digital Library

[31]

S. Kreft and G. Navarro. 2010. LZ77-like compression with fast random access. In Proceedings of the 2010 Data Compression Conference (DCC’10). IEEE Computer Society, 239--248.

Digital Library

[32]

S. Kreft and G. Navarro. 2013. On compressing and indexing repetitive sequences. Theor. Comput. Sci. 483 (2013), 115--133.

Digital Library

[33]

V. Mäkinen, G. Navarro, J. Sirén, and N. Välimäki. 2010. Storage and retrieval of highly repetitive sequence collections. J. Comput. Biol. 17, 3 (2010), 281--308.

[34]

G. Manzini. 2001. An analysis of the Burrows-Wheeler transform. J. ACM 48, 3 (2001), 407--430.

Digital Library

[35]

G. Navarro. 2012. Indexing highly repetitive collections. In Proceedings of the 23rd International Workshop on Combinatorial Algorithms (IWOCA’12), Lecture Notes in Computer Science, Vol. 7643. Springer, 274--279.

[36]

G. Navarro and V. Mäkinen. 2007. Compressed full-text indexes. ACM Comput. Surv. 39, 1 Article 2 (2007).

Digital Library

[37]

E. Ohlebusch and S. Gog. 2011. Lempel-Ziv factorization revisited. In Proceedings of the 22nd Annual Symposium on Combinatorial Pattern Matching (CPM’11), Lecture Notes in Computer Science, Vol. 6661. Springer, 15--26.

Digital Library

[38]

D. Okanohara and K. Sadakane. 2008. An online algorithm for finding the longest previous factors. In Proceedings of the 16th Annual European Symposium on Algorithms (ESA’08), Lecture Notes in Computer Science, Vol. 5193. Springer, 696--707.

Digital Library

[39]

I. Pavlov. 2012. 7-zip. http://www.7-zip.org/. (2012).

[40]

J. Sirén, N. Välimäki, V. Mäkinen, and G. Navarro. 2008. Run-length compressed indexes are superior for highly repetitive sequence collections. In Proceedings of the 15th International Symposium on String Processing and Information Retrieval (SPIRE’08), Lecture Notes in Computer Science, Vol. 5280. Springer, 164--175.

Digital Library

[41]

F. Wu. 2009. Sequential file prefetching in Linux. In Advanced Operating Systems and Kernel Applications: Techniques and Technologies, Y. Wiseman and S. Jiang (Eds.). IGI Global, Chapter 11, 217--236.

[42]

J. Ziv and A. Lempel. 1977. A universal algorithm for sequential data compression. IEEE Trans. Inf. Theory 23, 3 (1977), 337--343.

Digital Library

Cited By

Nuha HRizky Putra Sailellah HAbo Absa A(2024)Huffman Tree Compression and Lempel-Ziv Coding Using Java2024 International Conference on Decision Aid Sciences and Applications (DASA)10.1109/DASA63652.2024.10836416(1-5)Online publication date: 11-Dec-2024
https://doi.org/10.1109/DASA63652.2024.10836416
Navarro G(2021)Indexing Highly Repetitive String Collections, Part IACM Computing Surveys10.1145/343439954:2(1-31)Online publication date: Mar-2021
https://doi.org/10.1145/3434399
Han LLao BNong G(2021)Succinct parallel Lempel–Ziv factorization on a multicore computerThe Journal of Supercomputing10.1007/s11227-021-04165-w78:5(7278-7303)Online publication date: 8-Nov-2021
https://doi.org/10.1007/s11227-021-04165-w
Show More Cited By

Index Terms

Lazy Lempel-Ziv Factorization Algorithms
1. Theory of computation
  1. Design and analysis of algorithms

Recommendations

A comparison of index-based lempel-Ziv LZ77 factorization algorithms

Since 1977, when Lempel and Ziv described a kind of string factorization useful for text compression, there has been a succession of algorithms proposed for computing “LZ factorization”. In particular, there have been several recent algorithms proposed ...
The effect of non-greedy parsing in Ziv-Lempel compression methods
DCC '95: Proceedings of the Conference on Data Compression

Most practical compression methods in the LZ77 and LZ78 families parse their input using a greedy heuristic. However the popular gzip compression program demonstrates that modest but significant gains in compression performance are possible if non-...
Optimized relative Lempel-Ziv compression of genomes
ACSC '11: Proceedings of the Thirty-Fourth Australasian Computer Science Conference - Volume 113

High-throughput sequencing technologies make it possible to rapidly acquire large numbers of individual genomes, which, for a given organism, vary only slightly from one to another. Such repetitive and large sequence collections are a unique challange ...

Comments

Information & Contributors

Information

Published In

cover image ACM Journal of Experimental Algorithmics

ACM Journal of Experimental Algorithmics Volume 21, Issue

Special Issue SEA 2014, Regular Papers and Special Issue ALENEX 2013

2016

404 pages

ISSN:1084-6654

EISSN:1084-6654

DOI:10.1145/2888418

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2016

Accepted: 01 October 2014

Revised: 01 August 2013

Received: 01 April 2013

Published in JEA Volume 21

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Academy of Finland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
462
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)4

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nuha HRizky Putra Sailellah HAbo Absa A(2024)Huffman Tree Compression and Lempel-Ziv Coding Using Java2024 International Conference on Decision Aid Sciences and Applications (DASA)10.1109/DASA63652.2024.10836416(1-5)Online publication date: 11-Dec-2024
https://doi.org/10.1109/DASA63652.2024.10836416
Navarro G(2021)Indexing Highly Repetitive String Collections, Part IACM Computing Surveys10.1145/343439954:2(1-31)Online publication date: Mar-2021
https://doi.org/10.1145/3434399
Han LLao BNong G(2021)Succinct parallel Lempel–Ziv factorization on a multicore computerThe Journal of Supercomputing10.1007/s11227-021-04165-w78:5(7278-7303)Online publication date: 8-Nov-2021
https://doi.org/10.1007/s11227-021-04165-w
Kempa DKociumaka T(2020)Resolution of the Burrows-Wheeler Transform Conjecture2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS)10.1109/FOCS46700.2020.00097(1002-1013)Online publication date: Nov-2020
https://doi.org/10.1109/FOCS46700.2020.00097
Kempa DChan T(2019)Optimal construction of compressed indexes for highly repetitive textsProceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms10.5555/3310435.3310517(1344-1357)Online publication date: 6-Jan-2019
https://dl.acm.org/doi/10.5555/3310435.3310517
Kärkkäinen JKempa D(2019)Better External Memory LCP Array ConstructionACM Journal of Experimental Algorithmics10.1145/329772324(1-27)Online publication date: 14-Feb-2019
https://dl.acm.org/doi/10.1145/3297723
Ohlebusch EWeber P(2019)On the Computation of Longest Previous Non-overlapping FactorsString Processing and Information Retrieval10.1007/978-3-030-32686-9_26(372-381)Online publication date: 3-Oct-2019
https://doi.org/10.1007/978-3-030-32686-9_26
Fischer JKöppl D(2017)Practical Evaluation of Lempel-Ziv-78 and Lempel-Ziv-Welch TriesString Processing and Information Retrieval10.1007/978-3-319-67428-5_16(191-207)Online publication date: 6-Sep-2017
https://doi.org/10.1007/978-3-319-67428-5_16

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents