skip to main content
10.5555/3310435.3310605acmotherconferencesArticle/Chapter ViewAbstractPublication PagessodaConference Proceedingsconference-collections
research-article

Efficient algorithms and lower bounds for robust linear regression

Published: 06 January 2019 Publication History

Abstract

We study the prototypical problem of high-dimensional linear regression in a robust model where an ε-fraction of the samples can be adversarially corrupted. We focus on the fundamental setting where the covariates of the uncorrupted samples are drawn from a Gaussian distribution N(0, Σ) on Rd. We give nearly tight upper bounds and computational lower bounds for this problem. Specifically our main contributions are as follows:
• For the case that the covariance matrix is known to be the identity we give a sample near-optimal and computationally efficient algorithm that draws Õ(d/ε2) labeled examples and outputs a candidate hypothesis vector [MATH HERE] that approximates the unknown regression vector β within ℓ2-norm O(ε log(1/ε)σ), where σ is the standard deviation of the random observation noise. An error of Ω(εσ) is information-theoretically necessary even with infinite sample size. Hence, the error guarantee of our algorithm is optimal, up to a logarithmic factor in 1/ε. Prior work gave an algorithm for this problem with sample complexity [MATH HERE] whose error guarantee scales with the ℓ2-norm of β.
• For the case of unknown covariance Σ, we show that we can efficiently achieve the same error guarantee of O(ε log(1/ε)σ), as in the known covariance case, using an additional Õ(d2 / ε2) unlabeled examples. On the other hand, an error of O(εσ) can be information-theoretically attained with O(d/ε2) samples. We prove a Statistical Query (SQ) lower bound providing evidence that this quadratic tradeoff in the sample size is inherent. More specifically, we show that any polynomial time SQ learning algorithm for robust linear regression (in Huber's contamination model) with estimation complexity O(d2−c), where c > 0 is an arbitrarily small constant, must incur an error of [MATH HERE].

References

[1]
{ABL14} P. Awasthi, M. F. Balcan, and P. M. Long. The power of localization for efficiently learning linear separators with noise. In STOC 2014, pages 449--458, 2014.
[2]
{BDLS17} S. Balakrishnan, S. S. Du, J. Li, and A. Singh. Computationally efficient robust sparse estimation in high dimensions. In Proceedings of the 30th Conference on Learning Theory, COLT 2017, pages 169--212, 2017.
[3]
{BFJ<sup>+</sup>94} A. Blum, M. L. Furst, J. Jackson, M. J. Kearns, Y. Mansour, and S. Rudich. Weakly learning DNF and characterizing statistical query learning using Fourier analysis. In Proc. 26th Annual ACM Symposium on Theory of Computing (STOC), pages 253--262. ACM Press, 1994.
[4]
{BJK15} K. Bhatia, P. Jain, and P. Kar. Robust regression via hard thresholding. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, pages 721--729, 2015.
[5]
{BJKK17} K. Bhatia, P. Jain, P. Kamalaruban, and P. Kar. Consistent robust regression. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 2107--2116, 2017.
[6]
{CDKS18} Y. Cheng, I. Diakonikolas, D. M. Kane, and A. Stewart. Robust learning of fixed-structure bayesian networks. In Proceedings of the 31th International Conference on Neural Information Processing Systems, NIPS'18, 2018.
[7]
{CSV17} M. Charikar, J. Steinhardt, and G. Valiant. Learning from untrusted data. In Proceedings of STOC 2017, pages 47--60, 2017.
[8]
{DKK<sup>+</sup>16} I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. Robust estimators in high dimensions without the computational intractability. In Proceedings of FOCS'16, pages 655--664, 2016.
[9]
{DKK<sup>+</sup>17} I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. Being robust (in high dimensions) can be practical. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, pages 999--1008, 2017. Full version available at https://arxiv.org/abs/1703.00893.
[10]
{DKK<sup>+</sup>18a} I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. Robustly learning a gaussian: Getting optimal error, efficiently. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2018, pages 2683--2702, 2018. Full version available at https://arxiv.org/abs/1704.03866.
[11]
{DKK<sup>+</sup>18b} I. Diakonikolas, G. Kamath, D. M Kane, J. Li, J. Steinhardt, and A. Stewart. Sever: A robust meta-algorithm for stochastic optimization. arXiv preprint arXiv:1803.02815, 2018.
[12]
{DKS17a} I. Diakonikolas, D. M. Kane, and A. Stewart. Learning geometric concepts with nasty noise. CoRR, abs/1707.01242, 2017.
[13]
{DKS17b} I. Diakonikolas, D. M. Kane, and Al. Stewart. List-decodable robust mean estimation and learning mixtures of spherical gaussians. CoRR, abs/1711.07211, 2017.
[14]
{DKS17c} I. Diakonikolas, D. M. Kane, and Al. Stewart. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017, pages 73--84, 2017. Full version available at http://arxiv.org/abs/1611.03473.
[15]
{DKS18} Ilias Diakonikolas, Weihao Kong, and Alistair Stewart. Efficient algorithms and lower bounds for robust linear regression. arXiv preprint arXiv:1806.00040, 2018.
[16]
{Fel16a} V. Feldman. A general characterization of the statistical query complexity. CoRR, abs/1608.02198, 2016.
[17]
{Fel16b} V. Feldman. Statistical query learning. In Encyclopedia of Algorithms, pages 2090--2095. 2016.
[18]
{FGR<sup>+</sup>13} V. Feldman, E. Grigorescu, L. Reyzin, S. Vempala, and Y. Xiao. Statistical algorithms and a lower bound for detecting planted cliques. In Proceedings of STOC'13, pages 655--664, 2013.
[19]
{FGV15} V. Feldman, C. Guzman, and S. Vempala. Statistical query algorithms for stochastic convex optimization. CoRR, abs/1512.09170, 2015.
[20]
{FPV15} V. Feldman, W. Perkins, and S. Vempala. On the complexity of random satisfiability problems with planted solutions. In Proceedings of the Forty-Seventh Annual ACM on Symposium on Theory of Computing, STOC, 2015, pages 77--86, 2015.
[21]
{Gao17} C. Gao. Robust regression via mutivariate regression depth. coRR, abs/1702.04656, 2017.
[22]
{HL17} S. B. Hopkins and J. Li. Mixture models, robustness, and sum of squares proofs. CoRR, abs/1711.07454, 2017.
[23]
{Hub64} P. J. Huber. Robust estimation of a location parameter. Ann. Math. Statist., 35(1):73--101, 03 1964.
[24]
{Kea98} M. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM, 45(6):983--1006, 1998.
[25]
{KKM18} A. Klivans, P. Kothari, and R. Meka. Efficient algorithms for outlier-robust regression. arXiv preprint arXiv:1803.03241, 2018.
[26]
{KLS09} A. Klivans, P. Long, and R. Servedio. Learning halfspaces with malicious noise. To appear in Proc. 17th Internat. Colloq. on Algorithms, Languages and Programming (ICALP), 2009.
[27]
{KS17a} P. K. Kothari and J. Steinhardt. Better agnostic clustering via relaxed tensor norms. CoRR, abs/1711.07465, 2017.
[28]
{KS17b} P. K. Kothari and D. Steurer. Outlier-robust moment-estimation via sum-of-squares. CoRR, abs/1711.11581, 2017.
[29]
{LRV16} K. A. Lai, A. B. Rao, and S. Vempala. Agnostic estimation of mean and covariance. In Proceedings of FOCS'16, 2016.
[30]
{PSBR18} A. Prasad, A. S. Suggala, S. Balakrishnan, and P. Ravikumar. Robust estimation via robust gradient estimation. arXiv preprint arXiv:1802.06485, 2018.
[31]
{RL87} P. J. Rousseeuw and A. M. Leroy. Robust Regression and Outlier Detection. John Wiley & Sons, Inc., New York, NY, USA, 1987.
[32]
{Val84} L. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134--1142, 1984.

Cited By

View all
  1. Efficient algorithms and lower bounds for robust linear regression

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    SODA '19: Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms
    January 2019
    2993 pages

    Sponsors

    • SIAM Activity Group on Discrete Mathematics

    In-Cooperation

    Publisher

    Society for Industrial and Applied Mathematics

    United States

    Publication History

    Published: 06 January 2019

    Check for updates

    Qualifiers

    • Research-article

    Conference

    SODA '19
    Sponsor:
    SODA '19: Symposium on Discrete Algorithms
    January 6 - 9, 2019
    California, San Diego

    Acceptance Rates

    Overall Acceptance Rate 411 of 1,322 submissions, 31%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)6
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 28 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media