research-article

Online Additive Quantization

Authors:

Enhong ChenAuthors Info & Claims

KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

Pages 1098 - 1108

https://doi.org/10.1145/3447548.3467441

Published: 14 August 2021 Publication History

Abstract

Approximate nearest neighbor search (ANNs) plays an important role in many applications ranging from information retrieval, recommender systems to machine translation. Several ANN indexes, such as hashing and quantization, have been designed to update for the evolving database, but there exists a remarkable performance gap between them and retrained indexes on the entire database. To close the gap, we propose an online additive quantization algorithm (online AQ) to dynamically update quantization codebooks with the incoming streaming data. Then we derive the regret bound to theoretically guarantee the performance of the online AQ algorithm. Moreover, to improve the learning efficiency, we develop a randomized block beam search algorithm for assigning each data to the codewords of the codebook. Finally, we extensively evaluate the proposed online AQ algorithm on four real-world datasets, showing that it remarkably outperforms the state-of-the-art baselines.

References

[1]

Artem Babenko and Victor Lempitsky. 2014. Additive quantization for extreme vector compression. In Proceedings of CVPR'14. 931--938.

Digital Library

[2]

Artem Babenko and Victor Lempitsky. 2015. Tree quantization for large-scale similarity search and classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4240--4248.

[3]

Fatih Cakir, Sarah Adel Bargal, and Stan Sclaroff. 2017. Online supervised hashing. Computer Vision and Image Understanding 156 (2017), 162--173.

Digital Library

[4]

Fatih Cakir, Kun He, Sarah Adel Bargal, and Stan Sclaroff. 2017. Mihash: Online hashing with mutual information. In Proceedings of the IEEE International Conference on Computer Vision. 437--445.

[5]

Fatih Cakir and Stan Sclaroff. 2015. Adaptive hashing for fast similarity search. In Proceedings of the IEEE international conference on computer vision. 1044--1052.

Digital Library

[6]

Moses Charikar, Chandra Chekuri, Tomás Feder, and Rajeev Motwani. 2004. Incremental clustering and dynamic information retrieval. SIAM J. Comput. 33, 6 (2004), 1417--1440.

Digital Library

[7]

Patrick Chen, Si Si, Sanjiv Kumar, Yang Li, and Cho-Jui Hsieh. 2018. Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks. In International Conference on Learning Representations.

[8]

Xixian Chen, Irwin King, Michael R Lyu, et al. 2017. FROSH: FasteR Online Sketching Hashing. In UAI.

[9]

Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. 2014. Optimized product quantization. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 4 (2014), 744--755.

Digital Library

[10]

Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. 2013. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 35, 12 (2013).

Digital Library

[11]

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning. PMLR, 3887-- 3896.

[12]

Long-Kai Huang, Qiang Yang, and Wei-Shi Zheng. 2013. Online Hashing. In IJCAI. 1422--1428.

[13]

Long-Kai Huang, Qiang Yang, and Wei-Shi Zheng. 2017. Online hashing. IEEE transactions on neural networks and learning systems 29, 6 (2017), 2309--2322.

[14]

Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2011. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 1 (2011), 117--128.

Digital Library

[15]

Sham M Kakade, Shai Shalev-Shwartz, and Ambuj Tewari. 2012. Regularization techniques for learning with matrices. The Journal of Machine Learning Research 13, 1 (2012), 1865--1890.

Digital Library

[16]

Weihao Kong and Wu-Jun Li. 2012. Isotropic hashing. In Proceedings of NIPS'12. 1646--1654.

Digital Library

[17]

Cong Leng, Jiaxiang Wu, Jian Cheng, Xiao Bai, and Hanqing Lu. 2015. Online sketching hashing. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2503--2511.

[18]

Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. 2019. Approximate nearest neighbor search on high dimensional data-experiments, analyses, and improvement. IEEE Transactions on Knowledge and Data Engineering 32, 8 (2019), 1475--1488.

[19]

Edo Liberty. 2013. Simple and deterministic matrix sketching. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. 581--588.

Digital Library

[20]

Yury Malkov, Alexander Ponomarenko, Andrey Logvinov, and Vladimir Krylov. 2014. Approximate nearest neighbor algorithm based on navigable small world graphs. Information Systems 45 (2014), 61--68.

[21]

Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824--836.

[22]

Julieta Martinez, Joris Clement, Holger H Hoos, and James J Little. 2016. Revisiting additive quantization. In European Conference on Computer Vision. Springer, 137--153.

[23]

Yusuke Matsui, Yusuke Uchida, Hervé Jégou, and Shin'ichi Satoh. 2018. A survey of product quantization. ITE Transactions on Media Technology and Applications 6, 1 (2018), 2--10.

[24]

Stanislav Morozov and Artem Babenko. 2018. Non-metric similarity graphs for maximum inner product search. Advances in Neural Information Processing Systems 31 (2018), 4721--4730.

[25]

Stephen Mussmann and Stefano Ermon. 2016. Learning and inference via maximum inner product search. In International Conference on Machine Learning. PMLR, 2587--2596.

[26]

Behnam Neyshabur and Nathan Srebro. 2015. On Symmetric and Asymmetric LSHs for Inner Product Search. In Proceedings of ICML'15. 1926--1934.

[27]

Francesco Orabona, Koby Crammer, and Nicolo Cesa-Bianchi. 2015. A generalized online mirror descent with applications to classification and regression. Machine Learning 99, 3 (2015), 411--435.

Digital Library

[28]

Parikshit Ram and Alexander G Gray. 2012. Maximum inner-product search using cone trees. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 931--939.

Digital Library

[29]

Ruslan Salakhutdinov and Geoffrey Hinton. 2009. Semantic hashing. International Journal of Approximate Reasoning 50, 7 (2009), 969--978.

Digital Library

[30]

Hanan Samet. 2006. Foundations of multidimensional and metric data structures. Morgan Kaufmann.

[31]

Anshumali Shrivastava and Ping Li. 2014. Asymmetric LSH (ALSH) for sublinear time Maximum Inner Product Search (MIPS). In Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2. 2321--2329.

[32]

Jun Wang, Wei Liu, Sanjiv Kumar, and Shih-Fu Chang. 2015. Learning to hash for indexing big data-A survey. Proc. IEEE 104, 1 (2015), 34--57.

[33]

Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al. 2017. A survey on learning to hash. IEEE Trans. Pattern Anal. Mach. Intell. (2017).

[34]

Yair Weiss, Antonio Torralba, and Rob Fergus. 2009. Spectral hashing. In Proceedings of NIPS'09. 1753--1760.

Digital Library

[35]

Donna Xu, Ivor W Tsang, and Ying Zhang. 2018. Online product quantization. IEEE Transactions on Knowledge and Data Engineering 30, 11 (2018), 2185--2198.

Digital Library

[36]

Minjia Zhang, Xiaodong Liu, Wenhan Wang, Jianfeng Gao, and Yuxiong He. 2018. Navigating with graph representations for fast and scalable decoding of neural language models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 6311--6322.

[37]

Ting Zhang, Chao Du, and Jingdong Wang. 2014. Composite Quantization for Approximate Nearest Neighbor Search. In Proceedings of ICML'14. 838--846.

[38]

Ting Zhang, Guo-Jun Qi, Jinhui Tang, and Jingdong Wang. 2015. Sparse composite quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4548--4556.

Cited By

Feng CLian DWang XLiu ZXie XChen E(2023)Reinforcement Routing on Proximity Graph for Efficient RecommendationACM Transactions on Information Systems10.1145/351276741:1(1-27)Online publication date: 10-Jan-2023
https://dl.acm.org/doi/10.1145/3512767
Baranchuk DDouze MUpadhyay YYalniz I(2023)DeDrift: Robust Similarity Search under Content Drift2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01012(10992-11001)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01012

Index Terms

Online Additive Quantization
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information systems applications
    1. Data mining
      1. Nearest-neighbor search

Recommendations

Fast additive quantization for vector compression in nearest neighbor search

Vector quantization has been widely employed in nearest neighbor search because it can approximate the Euclidean distance of two vectors with the table look-up way that can be precomputed. Additive quantization (AQ) algorithm validated that low ...
Optimized Product Quantization for Approximate Nearest Neighbor Search
CVPR '13: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition

Product quantization is an effective vector quantization approach to compactly encode high-dimensional vectors for fast approximate nearest neighbor (ANN) search. The essence of product quantization is to decompose the original high-dimensional space ...
Confirmation Sampling for Exact Nearest Neighbor Search
Similarity Search and Applications
Abstract
Locality-sensitive hashing (LSH), introduced by Indyk and Motwani in STOC ’98, has been an extremely influential framework for nearest neighbor search in high-dimensional data sets. While theoretical work has focused on the approximate nearest ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

August 2021

4259 pages

ISBN:9781450383325

DOI:10.1145/3447548

General Chairs:
Feida Zhu
Singapore Management University
,
Beng Chin Ooi
National University of Singapore
,
Chunyan Miao
Nanyang Technology University
,
Program Chairs:
Haixun Wang,
Iryna Skrypnyk,
Wynne Hsu,
Sanjay Chawla

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

JD AI Research and the Fundamental Research Funds for the Central Universities
National Natural Science Foundation of China

Conference

KDD '21

Sponsor:

KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 14 - 18, 2021

Virtual Event, Singapore

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
372
Total Downloads

Downloads (Last 12 months)57
Downloads (Last 6 weeks)7

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Feng CLian DWang XLiu ZXie XChen E(2023)Reinforcement Routing on Proximity Graph for Efficient RecommendationACM Transactions on Information Systems10.1145/351276741:1(1-27)Online publication date: 10-Jan-2023
https://dl.acm.org/doi/10.1145/3512767
Baranchuk DDouze MUpadhyay YYalniz I(2023)DeDrift: Robust Similarity Search under Content Drift2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01012(10992-11001)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01012

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents