Article

Learning mixtures of arbitrary gaussians

Authors:

Ravi KannanAuthors Info & Claims

STOC '01: Proceedings of the thirty-third annual ACM symposium on Theory of computing

Pages 247 - 257

https://doi.org/10.1145/380752.380808

Published: 06 July 2001 Publication History

Abstract

Mixtures of gaussian (or normal) distributions arise in a variety of application areas. Many techniques have been proposed for the task of finding the component gaussians given samples from the mixture, such as the EM algorithm, a local-search heuristic from Dempster, Laird and Rubin~(1977). However, such heuristics are known to require time exponential in the dimension (i.e., number of variables) in the worst case, even when the number of components is $2$.

This paper presents the first algorithm that provably learns the component gaussians in time that is polynomial in the dimension. The gaussians may have arbitrary shape provided they satisfy a “nondegeneracy” condition, which requires their high-probability regions to be not “too close” together.

References

[1]

{1} N. Alon, J. Spencer and P. Erdös. The probabilistic method. Wiley Interscience, 1992.

[2]

{2} J. Bourgain. Random points in isotropic convex sets. Convex geometric analysis (Berkeley, CA, 1996), 53-58, Math. Sci. Res. Inst. Publ., 34, Cambridge Univ. Press, Cambridge, 1999.

[3]

{3} M. Charikar, S. Guha, E. Tardos, and D. Shmoys. A constant-factor approximation algorithm for the k-median problem. Proc. 31st ACM STOC, 1999.

Digital Library

[4]

{4} S. DasGupta. Learning mixtures of gaussians. Proc. IEEE Foundations of Computer Science, 1999.

Digital Library

[5]

{5} S. Dasgupta and L. Schulman. Personal communication, March 2000.

[6]

{6} A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statistics Soc. Ser. B, 39:1-38, 1977.

[7]

{7} Y. Freund and Y. Mansour. Estimating a mixture of two product distributions. ACM Conference on Computational Learning Theory, 1999.

Digital Library

[8]

{8} W. J. Hoeffding. Probability inequalities for sums of bounded random variables. J. American Statistical Assoc, 58(301):13-30, March 1963.

[9]

{9} P. J. Huber. Projection pursuit. Annals of Statistics, 13(2):435-475, June 1985.

[10]

{10} W. B. Johnson and J. Lindenstrauss. Extensions of Lipshitz mapping into Hilbert space. Contemp. Math. 26:189-206, 1984.

[11]

{11} R. Kannan, G. Li, Sampling according to the multivariate normal density, in the Proceedings of the 37th Annual Symposium on the Foundations of Computer Science, IEEE (1996) pp. 204-213.

Digital Library

[12]

{12} L. Leindler. On a certain converse of Hölder's inequality II. stochastic programming. Acta Sci. Math. Szeged 33:217-223(1972).

[13]

{13} B. Lindsay. Mixture models: theory, geometry, and applications. American Statistical Association, Virginia 1995.

[14]

{14} A. Prékopa. Logarithmic concave measures with applications to stochastic programming. Acta Sci. Math. Szeged 32:301-316 (1971).

[15]

{15} A. Prékopa. On logarithmic concave measures and functions. Acta Sci. Math. Szeged 34:335-343 (1973).

[16]

{16} R. A. Redner, H. F. Walker. Mixture densities, maximum likelihood and the EM Algorithm. SIAM Review, 26(2):195-239, 1984.

Digital Library

[17]

{17} M. Rudelson. Random vectors in the isotropic position. J. Func. Anal. 164:60-72, 1999.

[18]

{18} D. M. Titterington, A. F. M. Smith, and U. E. Makov. Statistical analysis of finite mixture distributions, Wiley, 1985.

Cited By

Taveira Blomenhofer A(2025)Gaussian mixture identifiability from degree 6 momentsAlgebraic Statistics10.2140/astat.2025.16.116:1(1-28)Online publication date: 1-Jan-2025
https://doi.org/10.2140/astat.2025.16.1
Guo BNie JYang Z(2024)Diagonal Gaussian mixture models and higher order tensor decompositionsNumerical Algebra, Control and Optimization10.3934/naco.2024053(0-0)Online publication date: 2024
https://doi.org/10.3934/naco.2024053
Sangani PKashettiwar AChakraborty PReddy BSivasubramanian DRamakrishnan GIyer RDe AKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Discrete-continuous optimization framework for simultaneous clustering and training in mixture modelsProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619651(29950-29970)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619651
Show More Cited By

Index Terms

Learning mixtures of arbitrary gaussians
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis
2. Mathematics of computing
  1. Mathematical software

Recommendations

Robustly learning mixtures of k arbitrary Gaussians
STOC 2022: Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing

We give a polynomial-time algorithm for the problem of robustly estimating a mixture of k arbitrary Gaussians in ℝ^d, for any fixed k, in the presence of a constant fraction of arbitrary corruptions. This resolves the main open problem in several ...
Learning Mixtures of Gaussians in High Dimensions
STOC '15: Proceedings of the forty-seventh annual ACM symposium on Theory of Computing

Efficiently learning mixture of Gaussians is a fundamental problem in statistics and learning theory. Given samples coming from a random one out of k Gaussian distributions in Rn, the learning problem asks to estimate the means and the covariance ...
Learning mixtures of spherical gaussians: moment methods and spectral decompositions
ITCS '13: Proceedings of the 4th conference on Innovations in Theoretical Computer Science

This work provides a computationally efficient and statistically consistent moment-based estimator for mixtures of spherical Gaussians. Under the condition that component means are in general position, a simple spectral decomposition technique yields ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

STOC '01: Proceedings of the thirty-third annual ACM symposium on Theory of computing

July 2001

755 pages

ISBN:1581133499

DOI:10.1145/380752

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGACT: ACM Special Interest Group on Algorithms and Computation Theory

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 July 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

STOC01

Sponsor:

SIGACT

STOC01: 33rd ACM Symposium on Theory of Computing

Hersonissos, Greece

Acceptance Rates

STOC '01 Paper Acceptance Rate 83 of 230 submissions, 36%;

Overall Acceptance Rate 1,469 of 4,586 submissions, 32%

Upcoming Conference

STOC '25

Sponsor:
sigact

57th Annual ACM Symposium on Theory of Computing (STOC 2025)

June 23 - 27, 2025

Prague , Czech Republic

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

70
Total Citations
View Citations
165
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Taveira Blomenhofer A(2025)Gaussian mixture identifiability from degree 6 momentsAlgebraic Statistics10.2140/astat.2025.16.116:1(1-28)Online publication date: 1-Jan-2025
https://doi.org/10.2140/astat.2025.16.1
Guo BNie JYang Z(2024)Diagonal Gaussian mixture models and higher order tensor decompositionsNumerical Algebra, Control and Optimization10.3934/naco.2024053(0-0)Online publication date: 2024
https://doi.org/10.3934/naco.2024053
Sangani PKashettiwar AChakraborty PReddy BSivasubramanian DRamakrishnan GIyer RDe AKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Discrete-continuous optimization framework for simultaneous clustering and training in mixture modelsProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619651(29950-29970)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619651
Arbas JAshtiani HLiaw CKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Polynomial time and private learning of unbounded Gaussian mixture modelsProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3618450(1018-1040)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3618450
Lyu ZXia D(2023)Optimal estimation and computational limit of low-rank Gaussian mixturesThe Annals of Statistics10.1214/23-AOS226451:2Online publication date: 1-Apr-2023
https://doi.org/10.1214/23-AOS2264
Liu AMoitra A(2023)Robustly Learning General Mixtures of GaussiansJournal of the ACM10.1145/358368070:3(1-53)Online publication date: 23-May-2023
https://dl.acm.org/doi/10.1145/3583680
Spaeh FTsourakakis C(2023)Learning Mixtures of Markov Chains with Quality GuaranteesProceedings of the ACM Web Conference 202310.1145/3543507.3583524(662-672)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583524
Giesen JKahlmeyer PLaue SMitterreiter MNussbaum FStaudt C(2023)Mixed membership GaussiansJournal of Multivariate Analysis10.1016/j.jmva.2022.105141195(105141)Online publication date: May-2023
https://doi.org/10.1016/j.jmva.2022.105141
Zhu MWang SLi Y(2022)Model-Based Representation and Deinterleaving of Mixed Radar Pulse Sequences With Neural Machine Translation NetworkIEEE Transactions on Aerospace and Electronic Systems10.1109/TAES.2021.312241158:3(1733-1752)Online publication date: Jun-2022
https://doi.org/10.1109/TAES.2021.3122411
Gupte AVafa NVaikuntanathan V(2022)Continuous LWE is as Hard as LWE & Applications to Learning Gaussian Mixtures2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS)10.1109/FOCS54457.2022.00112(1162-1173)Online publication date: Oct-2022
https://doi.org/10.1109/FOCS54457.2022.00112
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents