research-article

Donag: Generating Efficient Patches and Diffs for Compressed Archives

Author:

Michael J. MayAuthors Info & Claims

ACM Transactions on Storage (TOS), Volume 18, Issue 3

Article No.: 26, Pages 1 - 41

https://doi.org/10.1145/3507919

Published: 21 September 2022 Publication History

Abstract

Differencing between compressed archives is a common task in file management and synchronization. Applications include source code distribution, application updates, and document synchronization. General purpose binary differencing tools can create and apply patches to compressed archives, but don’t consider the internal structure of the compressed archive or the file lifecycle. Therefore, they miss opportunities to save space based on the archive’s internal structure and metadata. To address the gap, we develop a content-aware, format independent theory for differencing on compressed archives and propose a canonical form and digest for compressed archives. Based on them, we present Donag, a content-aware differencing and patching algorithm that produces smaller patches than general purpose binary differencing tools on versioned archives by exploiting the compressed archives’ internal structure. Donag uses the VCDiff and BSDiff engines internally. We compare Donag’s patches to ones produced by bsdiff, xdelta3, and Delta++ on three classes of compressed archives: open-source code repositories, large and small applications, and office productivity documents (DOCX, XLSX, PPTX). Donag’s patches are typically 10% to 89% smaller than those produced by bsdiff, xdelta3, and Delta++, with reasonable memory overhead and throughput on commodity hardware. In the worst case, Donag’s patches are negligibly larger.

References

[1]

Stephen Adams. 2009. Software Updates: Courgette. Online. (Jul. 2009). Retrieved 4 Nov. 2018 from http://dev.chromium.org/developers/design-documents/software-updates-courgette.

[2]

Gioele Barabucci. 2013. A Universal Delta Model. Dissertation. Universita di Bologna, Bologna, Italy. https://core.ac.uk/download/pdf/11014284.pdf.

[3]

Gioele Barabucci, Paolo Ciancarini, Angelo Di Iorio, and Fabio Vitali. 2016. Measuring the quality of diff algorithms: A formalization. Computer Standards and Interfaces46 (2016), 52–65.

Digital Library

[4]

Reed Bittinger, Nils C. Brubaker, Barron Cornelius Housel III, and Steve Wang. 2000. Method and system for differencing container files. US Patent. (Nov. 2000). https://patents.google.com/patent/US6148340A/en. Patent No. US6148340A, Filed 30 Apr. 1998, Issued 14 Nov. 2000.

[5]

John Boyer and Glenn Marcy. 2008. Canonical XML Version 1.1. W3C Recommendation. World Wide Web Consortium.

[6]

A. Z. Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171). 21–29.

[7]

Wayne E. Carlson. 1991. A survey of computer graphics image encoding and storage formats. SIGGRAPH Comput. Graph. 25, 2 (April 1991), 67–75.

Digital Library

[8]

David James Clarke IV. 2004. CNE for NetWare 6 Study Guide (1st ed.). Novell Press.

[9]

Wayne Davison. 2020. rsync. (Aug. 2020). Retrieved 2 Sep. 2021 from https://rsync.samba.org/.

[10]

Laurent Denoue, Scott Carter, and Matthew Cooper. 2018. SlideDiff: Animating textual and media changes in slides. In Proceedings of the ACM Symposium on Document Engineering 2018, DocEng 2018, Halifax, NS, Canada, August 28-31, 2018. ACM, 37:1–37:4.

Digital Library

[11]

P. Deutsch. 1996. RFC1951: DEFLATE Compressed Data Format Specification version 1.3. IETF.

Digital Library

[12]

David Ehrmann. 2019. VCDiff-java. Online. Retrieved 22 Sep. 2020 from https://github.com/ehrmann/vcdiff-java. Version 0.1.1.

[13]

Garrick D. Evans, Liang Han, Carolyn E. Kreisel, and Tong Zhang. 2011. Dynamic manipulation of archive files. US Patent. (Sep. 2011). https://patents.google.com/patent/US8024382B2/en. Patent No. US8024382B2, Filed 20 Jan. 2009, Issued 20 Sep. 2011.

[14]

P. Ferragina, F. Luccio, G. Manzini, and S. Muthukrishnan. 2006. Compressing and searching XML data via two zips. In Proceedings of the 15th International Conference on World Wide Web (WWW ’06). ACM, New York, NY, USA, 751–760.

Digital Library

[15]

Google Code Labs. 2011. CRX Package Format. Retrieved 3 Oct. 2019 from http://www.adambarth.com/experimental/crx/docs/crx.html.

[16]

D. A. Huffman. 1952. A method for the construction of minimum-redundancy codes. Proceedings of the IRE 40, 9 (Sep. 1952), 1098–1101.

[17]

James J. Hunt, Kiem-Phong Vo, and Walter F. Tichy. 1998. Delta algorithms: An empirical analysis. ACM Trans. Softw. Eng. Methodol. 7, 2 (April 1998), 192–214.

Digital Library

[18]

Shmuel T. Klein, Tamar C. Serebro, and Dana Shapira. 2008. Modeling delta encoding of compressed files. International Journal of Foundations of Computer Science 19, 01 (2008), 137–146.

[19]

Shmuel T. Klein and Dana Shapira. 2007. Compressed delta encoding for LZSS encoded files. In 2007 Data Compression Conference (DCC’07). 113–122.

Digital Library

[20]

D. Korn, J. MacDonald, J. Mogul, and K. Vo. 2012. The VCDIFF Generic Differencing and Compression Data Format. RFC RFC 3284. IETF. https://tools.ietf.org/html/rfc3284.

[21]

David G. Korn and Kiem-Phong Vo. 2002. Engineering a differencing and compression data format. In USENIX Annual Technical Conference 2002. USENIX.

[22]

Jim Kurose and Keith Ross. 2020. Computer Networking: A Top Down Approach Powerpoint Slides. (2020). Retrieved 23 Sep. 2020 from http://gaia.cs.umass.edu/kurose_ross/ppt.htm.

[23]

Debra A. Lelewer and Daniel S. Hirschberg. 1987. Data compression. ACM Comput. Surv. 19, 3 (Sep. 1987), 261–296.

Digital Library

[24]

Eelco Lempsink and Andres Löh. 2009. gdiff: Generic diff and patch. Retrieved 6 Oct. 2019 from http://hackage.haskell.org/package/gdiff.

[25]

Xing Lin, Guanlin Lu, Fred Douglis, Philip Shilane, and Grant Wallace. 2014. Migratory compression: Coarse-grained data reordering to improve compressibility. In 12th USENIX Conference on File and Storage Technologies (FAST 14). USENIX Association, Santa Clara, CA, 256–273. https://www.usenix.org/conference/fast14/technical-sessions/presentation/lin.

[26]

Josh Macdonald. 2016. xdelta3. Retrieved 22 Sep. 2020 from http://xdelta.org/.

[27]

Matthew Malensek. 2019. jbsdiff. Retrieved 10 Dec. 2020 from https://github.com/malensek/jbsdiff.

[28]

Michael J. May, Etamar Laron, Khalid Zoabi, and Havah Gerhardt. 2019. On the lifecycle of the file. ACM Trans. Storage 15, 1, Article 1 (Feb. 2019), 45 pages.

Digital Library

[29]

Microsoft. 2020. Windows Sysinternals. Retrieved 1 Oct. 2020 from https://docs.microsoft.com/en-us/sysinternals/.

[30]

Microsoft Docs. 2019. NTFS overview. Retrieved 2 Oct. 2019 from https://docs.microsoft.com/en-us/windows-server/storage/file-server/ntfs-overview.

[31]

Stephen L. Nelson and E. C. Nelson. 2015. Excel Data Analysis For Dummies (3rd ed.). John Wiley & Sons, Inc.

Digital Library

[32]

Colin Percival. 2006. Binary diff/patch utility. (2006). Retrieved 6 Oct. 2019 from http://www.daemonology.net/bsdiff/.

[33]

PKWare Inc. 2012. APPNOTE.TXT - .ZIP File Format Specification (version 6.3.3 ed.). PKWare Inc.

[34]

OPhone Platform. 2010. The Structure of Android Package (APK) Files. (Nov. 2010). Retrieved 3 Oct. 2019 from https://web.archive.org/web/20110208193918 http://en.ophonesdn.com/article/show/354.

[35]

N. Samteladze and K. Christensen. 2014. DELTA++: Reducing the size of Android application updates. IEEE Internet Computing 18, 2 (Mar. 2014), 50–57.

[36]

Philip Shilane, Mark Huang, Grant Wallace, and Windsor Hsu. 2012. WAN Optimized replication of backup datasets using stream-informed delta compression. In 10th USENIX Conference on File and Storage Technologies (FAST 12). USENIX Association, San Jose, CA. https://www.usenix.org/conference/fast12/wan-optimized-replication-backup-datasets-using-stream-informed-delta-compression.

Digital Library

[37]

James A. Storer and Thomas G. Szymanski. 1982. Data compression via textual substitution. J. ACM 29, 4 (Oct. 1982), 928–951.

Digital Library

[38]

Torsten Sul and Nasir Memon. 2003. Lossless Compression Handbook. Academic Press, An Imprint of Elsevier Science, Chapter Algorithms for Delta Compression and Remote File Synchronization, 269–290.

[39]

Sean C. Sullivan and James Stewart. 2004. zipdiff. Online. (2004). Retrieved 22 Sep. 2020 from http://zipdiff.sourceforge.net/index.html.

[40]

Scott Watanabe. 2010. Solaris 10 ZFS Essentials (1st ed.). Prentice Hall.

Digital Library

[41]

Wen Xia, Hong Jiang, Dan Feng, Lei Tian, Min Fu, and Yukun Zhou. 2014. Ddelta: A deduplication-inspired fast delta compression approach. Performance Evaluation 79 (2014), 258–272. Special Issue: Performance 2014.

Cited By

Sun TJiang BLi BLv JGao YDong WBagchi SZhang Y(2024)SimEncProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692030(615-630)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691992.3692030
Sun TJiang BJin LZhang WGao YLi ZDong W(2024)Understanding Differencing Algorithms for Mobile Application UpdatesIEEE Transactions on Mobile Computing10.1109/TMC.2024.340786723:12(12146-12160)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TMC.2024.3407867
Jin LDong WJiang BSun TGao Y(2024)Exploiting Multiple Similarity Spaces for Efficient and Flexible Incremental Update of Mobile AppsIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621326(1541-1550)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621326
Show More Cited By

Index Terms

Donag: Generating Efficient Patches and Diffs for Compressed Archives

Recommendations

Scalability in recursively stored delta compressed collections of files
AWC '14: Proceedings of the Second Australasian Web Conference - Volume 155

The archiving and maintenance of vast quantities of data is a key challenge for the current use of information technology. When storing large repositories, possibly mirrored at multiple sites, an archiving system aims to reduce both storage and ...
Near-Optimal Compression for Compressed Sensing
DCC '15: Proceedings of the 2015 Data Compression Conference

In this note we study the under-addressed quantization stage implicit in any compressed sensing signal acquisition paradigm. We also study the problem of compressing the bit-stream resulting from the quantization. We propose using Sigma-Delta (ΣΔ) ...
MPEG-1 Super-Resolution Decoding for the Analysis of Video Still Images
SSIAI '00: Proceedings of the 4th IEEE Southwest Symposium on Image Analysis and Interpretation

A digital image sequence coded at a low bitrate using a motion-compensated video compression standard should contain little data redundancy. However, the success of a particular super-resolution enhancement algorithm is predicated on subpixel-resolution ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Storage

ACM Transactions on Storage Volume 18, Issue 3

August 2022

244 pages

ISSN:1553-3077

EISSN:1553-3093

DOI:10.1145/3555792

Editor:
Sam H. Noh
Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2022

Online AM: 27 July 2022

Accepted: 21 December 2021

Revised: 10 December 2021

Received: 08 April 2021

Published in TOS Volume 18, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
300
Total Downloads

Downloads (Last 12 months)66
Downloads (Last 6 weeks)6

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sun TJiang BLi BLv JGao YDong WBagchi SZhang Y(2024)SimEncProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692030(615-630)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691992.3692030
Sun TJiang BJin LZhang WGao YLi ZDong W(2024)Understanding Differencing Algorithms for Mobile Application UpdatesIEEE Transactions on Mobile Computing10.1109/TMC.2024.340786723:12(12146-12160)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TMC.2024.3407867
Jin LDong WJiang BSun TGao Y(2024)Exploiting Multiple Similarity Spaces for Efficient and Flexible Incremental Update of Mobile AppsIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621326(1541-1550)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621326
Qin GWang YLiang YSong J(2023)Automotive OTA Upgrade Scheme Based on Optimal Difference Algorithm2023 3rd International Conference on Robotics, Automation and Intelligent Control (ICRAIC)10.1109/ICRAIC61978.2023.00032(140-144)Online publication date: 24-Nov-2023
https://doi.org/10.1109/ICRAIC61978.2023.00032

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents