On adaptive distance estimation
Article No.: 938, Pages 11178 - 11190
Abstract
We provide a static data structure for distance estimation which supports adaptive queries. Concretely, given a dataset $X = \{x_i\}_{i = 1}^n$ of n points in ℝd and 0 < p ≤ 2, we construct a randomized data structure with low memory consumption and query time which, when later given any query point q ∈ ℝd, outputs a (1 + ε)-approximation of ||q—xi||p with high probability for all i ∈ [n]. The main novelty is our data structure's correctness guarantee holds even when the sequence of queries can be chosen adaptively: an adversary is allowed to choose the jth query point qj in a way that depends on the answers reported by the data structure for q1,..., qj_1. Previous randomized Monte Carlo methods do not provide error guarantees in the setting of adaptively chosen queries [JL84, Ind06, TZ12, IW18]. Our memory consumption is Õ((n + d)d/ε2), slightly more than the O(nd) required to store X in memory explicitly, but with the benefit that our time to answer queries is only Õ(ε-2(n + d)), much faster than the naive Θ(nd) time obtained from a linear scan in the case of n and d very large. Here Õ hides log(nd/ε) factors. We discuss applications to nearest neighbor search and nonparametric estimation.
Our method is simple and likely to be applicable to other domains: we describe a generic approach for transforming randomized Monte Carlo data structures which do not support adaptive queries to ones that do, and show that for the problem at hand, it can be applied to standard nonadaptive solutions to ℓp norm estimation with negligible overhead in query time and a factor d overhead in memory.
Supplementary Material
Supplemental material.
- Download
- 388.39 KB
References
[1]
Thomas Dybdahl Ahle. Optimal Las Vegas locality sensitive data structures. In Chris Umans, editor, 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017, Berkeley, CA, USA, October 15-17, 2017, pages 938–949. IEEE Computer Society, 2017.
[2]
Noga Alon and Bo'az Klartag. Optimal compression of approximate inner products and dimension reduction. In Proceedings of the 58th IEEE Annual Symposium on Foundations of Computer Science (FOCS), pages 639–650, 2017.
[3]
N. S. Altman. An introduction to kernel and nearest-neighbor nonparametric regression. Amer. Statist., 46(3):175–185, 1992.
[4]
Christopher G. Atkeson, Andrew W. Moore, and Stefan Schaal. Locally weighted learning. Artif. Intell. Rev., 11(1-5):11–73, 1997.
[5]
Noga Alon, Yossi Matias, and Mario Szegedy. The space complexity of approximating the frequency moments. J. Comput. Syst. Sci., 58(1):137–147, 1999.
[6]
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. CoRR, abs/1708.06131, 2017.
[7]
Omri Ben-Eliezer, Rajesh Jayaram, David P. Woodruff, and Eylon Yogev. A framework for adversarially robust streaming algorithms. In Proceedings of the 39th ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS), 2020.
[8]
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities. Oxford University Press, Oxford, 2013. A nonasymptotic theory of independence, With a foreword by Michel Ledoux.
[9]
Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: applications to image and text data. In Doheon Lee, Mario Schkolnick, Foster J. Provost, and Ramakrishnan Srikant, editors, Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, San Francisco, CA, USA, August 26-29, 2001, pages 245–250. ACM, 2001.
[10]
Raef Bassily, Kobbi Nissim, Adam D. Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In Daniel Wichs and Yishay Mansour, editors, Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2016, Cambridge, MA, USA, June 18-21, 2016, pages 1046–1059. ACM, 2016.
[11]
Leo Breiman. Bagging predictors. Mach. Learn., 24(2):123–140, 1996.
[12]
Leo Breiman. Random forests. Mach. Learn., 45(1):5–32, 2001.
[13]
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM Comput. Surv., 41(3):15:1–15:58, 2009.
[14]
Moses Charikar, Kevin C. Chen, and Martin Farach-Colton. Finding frequent items in data streams. Theor. Comput. Sci., 312(1):3–15, 2004.
[15]
Kenneth L. Clarkson. A randomized algorithm for closest-point queries. SIAM J. Comput., 17(4):830–847, 1988.
[16]
J. M. Chambers, C. L. Mallows, and B. W. Stuck. A method for simulating stable random variables. J. Amer. Statist. Assoc., 71(354):340–344, 1976.
[17]
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. In Proceedings of the 28th Annual Conference on Advances in Neural Information Processing Systems (NIPS), pages 2350–2358, 2015.
[18]
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. The reusable holdout: preserving validity in adaptive data analysis. Science, 349(6248):636–638, 2015.
[19]
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Leon Roth. Preserving statistical validity in adaptive data analysis. In Proceedings of the 47th Annual ACM on Symposium on Theory of Computing (STOC), pages 117–126, 2015.
[20]
Thomas G. Dietterich. Ensemble methods in machine learning. In Josef Kittler and Fabio Roli, editors, Multiple Classifier Systems, First International Workshop, MCS 2000, Cagliari, Italy, June 21-23, 2000, Proceedings, volume 1857 of Lecture Notes in Computer Science, pages 1–15. Springer, 2000.
[21]
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S. Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Jack Snoeyink and Jean-Daniel Boissonnat, editors, Proceedings of the 20th ACM Symposium on Computational Geometry, Brooklyn, New York, USA, June 8-11, 2004, pages 253–262. ACM, 2004.
[22]
Cynthia Dwork, Adam Smith, Thomas Steinke, and Jonathan Ullman. Exposed! a survey of attacks on private data. Annual Review of Statistics and Its Application, 4(1):61–84, 2017.
[23]
Anna C. Gilbert, Brett Hemenway, Atri Rudra, Martin J. Strauss, and Mary Wootters. Recovering simple signals. In 2012 Information Theory and Applications Workshop, ITA 2012, San Diego, CA, USA, February 5-10, 2012, pages 382–391. IEEE, 2012.
[24]
Anna C. Gilbert, Brett Hemenway, Martin J. Strauss, David P. Woodruff, and Mary Wootters. Reusable low-error compressive sampling schemes through privacy. In IEEE Statistical Signal Processing Workshop, SSP 2012, Ann Arbor, MI, USA, August 5-8, 2012, pages 536–539. IEEE, 2012.
[25]
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[26]
Francois Le Gall and Florent Urrutia. Improved rectangular matrix multiplication using powers of the Coppersmith-Winograd tensor. In Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1029–1046, 2018.
[27]
Moritz Hardt, Nimrod Megiddo, Christos H. Papadimitriou, and Mary Wootters. Strategic classification. In Madhu Sudan, editor, Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science, Cambridge, MA, USA, January 14-16, 2016, pages 111–122. ACM, 2016.
[28]
Thomas Hofmann, Bernhard Schölkopf, and Alexander J. Smola. Kernel methods in machine learning. Ann. Statist., 36(3):1171–1220, 2008.
[29]
Moritz Hardt and David P. Woodruff. How robust are linear sketches to adaptive inputs? In Dan Boneh, Tim Roughgarden, and Joan Feigenbaum, editors, Symposium on Theory of Computing Conference, STOC'13, Palo Alto, CA, USA, June 1-4, 2013, pages 121–130. ACM, 2013.
[30]
Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Jeffrey Scott Vitter, editor, Proceedings of the Thirtieth Annual ACM Symposium on the Theory of Computing, Dallas, Texas, USA, May 23-26, 1998, pages 604–613. ACM, 1998.
[31]
Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. ACM, 53(3):307–323, 2006.
[32]
Piotr Indyk and Tal Wagner. Approximate nearest neighbors in limited space. In Proceedings of the Conference On Learning Theory (COLT), pages 2012–2036, 2018.
[33]
William B. Johnson and Joram Lindenstrauss. Extensions of Lipschitz mappings into a Hilbert space. In Conference in modern analysis and probability (New Haven, Conn., 1982), volume 26 of Contemp. Math., pages 189–206. Amer. Math. Soc., Providence, RI, 1984.
[34]
Jon M. Kleinberg. Two algorithms for nearest-neighbor search in high dimensions. In Frank Thomson Leighton and Peter W. Shor, editors, Proceedings of the Twenty-Ninth Annual ACM Symposium on the Theory of Computing, El Paso, Texas, USA, May 4-6, 1997, pages 599–608. ACM, 1997.
[35]
Daniel M. Kane, Jelani Nelson, Ely Porat, and David P. Woodruff. Fast moment estimation in data streams in optimal space. In Proceedings of the 43rd ACM Symposium on Theory of Computing (STOC), pages 745–754, 2011.
[36]
Daniel M. Kane, Jelani Nelson, and David P. Woodruff. On the exact space complexity of sketching and streaming small norms. In Proceedings of the 21st Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1161–1178, 2010.
[37]
Eyal Kushilevitz, Rafail Ostrovsky, and Yuval Rabani. Efficient search for approximate nearest neighbor in high dimensional spaces. SIAM J. Comput., 30(2):457–474, 2000.
[38]
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[39]
Gábor Lugosi and Shahar Mendelson. Near-optimal mean estimators with respect to general norms. Probab. Theory Related Fields, 175(3-4):957–973, 2019.
[40]
Michel Ledoux and Michel Talagrand. Probability in Banach spaces. Classics in Mathematics. Springer-Verlag, Berlin, 2011. Isoperimetry and processes, Reprint of the 1991 edition.
[41]
S. Meiser. Point location in arrangements of hyperplanes. Inform. and Comput., 106(2):286–303, 1993.
[42]
Ilya Mironov, Moni Naor, and Gil Segev. Sketching in adversarial environments. SIAM J. Comput., 40(6):1845–1870, 2011.
[43]
Shahar Mendelson and Nikitz Zhivotovskiy. Robust covariance estimation under L4-L2 norm equivalence. arXiv preprint arXiv:1809.10462, 2018.
[44]
J. P. Nolan. Stable Distributions - Models for Heavy Tailed Data. Birkhauser, Boston, 2018. In progress, Chapter 1 online at http://fs2.american.edu/jpnolan/www/stable/stable.html.
[45]
Rasmus Pagh. Coveringlsh: Locality-sensitive hashing without false negatives. ACM Trans. Algorithms, 14(3):29:1–29:17, 2018.
[46]
Nicolas Papernot, Patrick D. McDaniel, and Ian J. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR, abs/1605.07277, 2016.
[47]
Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Ramesh Karri, Ozgur Sinanoglu, Ahmad-Reza Sadeghi, and Xun Yi, editors, Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017, pages 506–519. ACM, 2017.
[48]
Gregory Shakhnarovich, Trevor Darrell, and Piotr Indyk. Nearest-neighbor methods in learning and vision. IEEE Trans. Neural Networks, 19(2):377, 2008.
[49]
Jeffrey S. Simonoff. Smoothing methods in statistics. Springer Series in Statistics. Springer-Verlag, New York, 1996.
[50]
Piotr Sankowski and Piotr Wygocki. Approximate nearest neighbors search without false negatives for 2 for c > √log log n. In Yoshio Okamoto and Takeshi Tokuyama, editors, 28th International Symposium on Algorithms and Computation, ISAAC 2017, December 9-12, 2017, Phuket, Thailand, volume 92 of LIPIcs, pages 63:1–63:12. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2017.
[51]
Mikkel Thorup and Yin Zhang. Tabulation-based 5-independent hashing with applications to linear probing and second moment estimation. SIAM J. Comput., 41(2):293–331, 2012.
[52]
Roman Vershynin. High-dimensional probability, volume 47 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2018. An introduction with applications in data science, With a foreword by Sara van de Geer.
[53]
Alexander Wei. Optimal las vegas approximate near neighbors in ℓp. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1794–1813, 2019.
[54]
M. P. Wand and M. C. Jones. Kernel smoothing, volume 60 of Monographs on Statistics and Applied Probability. Chapman and Hall, Ltd., London, 1995.
[55]
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE Trans. Neural Networks Learn. Syst., 30(9):2805– 2824, 2019.
[56]
V. M. Zolotarev. One-dimensional stable distributions, volume 65 of Translations of Mathematical Monographs. American Mathematical Society, Providence, RI, 1986. Translated from the Russian by H. H. McFaden, Translation edited by Ben Silver.
Recommendations
Fast Partial Distance Estimation and Applications
PODC '15: Proceedings of the 2015 ACM Symposium on Principles of Distributed ComputingWe study approximate distributed solutions to the weighted all-pairs shortest-paths (APSP) problem in the CONGEST model. We obtain the following results. A deterministic (1+epsilon)-approximation to APSP with running time O(ε-2n log n) rounds. The best ...
Adaptive and Incremental Processing for Distance Join Queries
A spatial distance join is a relatively new type of operation introduced for spatial and multimedia database applications. Additional requirements for ranking and stopping cardinality are often combined with the spatial distance join in online query ...
Adaptive Estimation of Generalized Distance to Uniformity
2018 IEEE International Symposium on Information Theory (ISIT)In this paper, we reconsider the problem of distance to uniformity estimation of discrete distributions. As a fundamental problem in distribution property estimation, the problem with known alphabet size has been addressed in [1] [3] and is fairly well ...
Comments
Information & Contributors
Information
Published In
December 2020
22651 pages
ISBN:9781713829546
- Editors:
- H. Larochelle,
- M. Ranzato,
- R. Hadsell,
- M.F. Balcan,
- H. Lin
Copyright © 2020 Neural Information Processing Systems Foundation, Inc.
Publisher
Curran Associates Inc.
Red Hook, NY, United States
Publication History
Published: 06 December 2020
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 22Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)5
Reflects downloads up to 27 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in