Article

Parallel Markov chain Monte Carlo for nonparametric mixture models

Authors:

Sinead A. Williamson,

Eric P. XingAuthors Info & Claims

ICML'13: Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28

Pages I-98 - I-106

Published: 16 June 2013 Publication History

Abstract

Nonparametric mixture models based on the Dirichlet process are an elegant alternative to finite models when the number of underlying components is unknown, but inference in such models can be slow. Existing attempts to parallelize inference in such models have relied on introducing approximations, which can lead to inaccuracies in the posterior estimate. In this paper, we describe auxiliary variable representations for the Dirichlet process and the hierarchical Dirichlet process that allow us to perform MCMC using the correct equilibrium distribution, in a distributed manner. We show that our approach allows scalable inference without the deterioration in estimate quality that accompanies existing methods.

References

[1]

Ahmed, A., Ho, Q., Teo, C. H., Eisenstein, J., Smola, A. J., and Xing, E. P. Online inference for the infinite topic-cluster model: Storylines from streaming text. In AISTATS, 2011.

[2]

Aldous, D. J. Exchangeability and related topics. In École d'Été de probabilités de Saint-Flour XIII. 1985.

[3]

Antoniak, C. E. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Statist., 2(6):1152-1174, 1974.

[4]

Asuncion, A., Smyth, P., and Welling, M. Asynchronous distributed learning of topic models. In NIPS, 2008.

[5]

Blei, D. M. and Jordan, M. I. Variational methods for the Dirichlet process. In ICML, 2004.

[6]

Fearnhead, P. Particle filters for mixture models with an unknown number of components. Statistics and Computing, 14:11-21, 2004.

[7]

Ferguson, T. S. A Bayesian analysis of some nonparametric problems. Ann. Statist., 1(2):209-230, 1973.

[8]

Fox, E. B., Sudderth, E. B., Jordan, M. I., and Willsky, A. S. An HDP-HMM for systems with state persistence. In ICML, 2008.

[9]

Ghosh, J. K. and Ramamoorthi, R. V. Bayesian Nonparametrics . Springer, 2003.

[10]

Ishwaran, H. and James, L. F. Gibbs sampling methods for stick-breaking priors. JASA, 96(453):161- 173, 2001.

[11]

Kingman, J. F. C. Completely random measures. Pacific Journal of Mathematics, 21(1):59-78, 1967.

[12]

Kurihara, K., Welling, M., and Teh, Y.-W. Collapsed variational Dirichlet process mixture models. In IJCAI, 2007.

[13]

Lovell, D., Adams, R. P., and Mansingka, V. K. Parallel Markov chain Monte Carlo for Dirichlet process mixtures. In Workshop on Big Learning, NIPS, 2012.

[14]

Neal, R. M. Markov chain sampling methods for Dirichlet process mixture models. Technical Report 9815, Dept. of Statistics, University of Toronto, 1998.

[15]

Rodriguez, A. On-line learning for the infinite hidden Markov model. Communications in Statistics - Simulation and Computation, 40(6):879-893, 2011.

[16]

Sohn, K.-A. and Xing, E. P. A hierarchical Dirichlet process mixture model for haplotype reconstruction from multi-population data. Ann. Appl. Stat., 3(2): 791-821, 2009.

[17]

Sudderth, E. B., Torralba, A., Freeman, W. T., and Willsky, A. S. Describing visual scenes using transformed Dirichlet processes. In NIPS, 2005.

[18]

Teh, Y.-W., Jordan, M. I., Beal, M. J., and Blei, D. M. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476): 1566-1581, 2006.

[19]

Teh, Y.-W., Kurihara, K., and Welling, M. Collapsed variational inference for HDP. In NIPS, 2007.

[20]

Ulker, Y., Gunsel, B., and Cemgil, A. T. Sequential Monte Carlo samplers for Dirichlet process mixtures. In AISTATS, 2010.

[21]

Wang, C., Paisley, J., and Blei, D. M. Online variational inference for the hierarchical Dirichlet process. In AISTATS, 2011.

[22]

Xing, E. P., Ng, A. Y., Jordan, M. I., and Russell, S. Distance metric learning, with application to clustering with side-information. In NIPS, 2002.

Cited By

Meguelati KFontez BHilgert NMasseglia FHung CPapadopoulos G(2019)Dirichlet process mixture models made scalable and effective by means of massive distributionProceedings of the 34th ACM/SIGAPP Symposium on Applied Computing10.1145/3297280.3297327(502-509)Online publication date: 8-Apr-2019
https://dl.acm.org/doi/10.1145/3297280.3297327
Xuan JLu JZhang G(2019)A Survey on Bayesian Nonparametric LearningACM Computing Surveys10.1145/329104452:1(1-36)Online publication date: 25-Jan-2019
https://dl.acm.org/doi/10.1145/3291044
Zuanetti DMüller PZhu YYang SJi Y(2019)Bayesian nonparametric clustering for large data setsStatistics and Computing10.1007/s11222-018-9803-929:2(203-215)Online publication date: 16-Mar-2019
https://dl.acm.org/doi/10.1007/s11222-018-9803-9
Show More Cited By

Parallel Markov chain Monte Carlo for nonparametric mixture models
1. Computing methodologies

Recommendations

A Monte Carlo Markov chain algorithm for a class of mixture time series models

This article generalizes the Monte Carlo Markov Chain (MCMC) algorithm, based on the Gibbs weighted Chinese restaurant (gWCR) process algorithm, for a class of kernel mixture of time series models over the Dirichlet process. This class of models is an ...
Parallel Markov chain Monte Carlo for Pitman-Yor mixture models
UAI'14: Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence

The Pitman-Yor process provides an elegant way to cluster data that exhibit power law behavior, where the number of clusters is unknown or unbounded. Unfortunately, inference in Pitman-Yor process-based models is typically slow and does not scale well ...
Parallel algorithms for Markov chain Monte Carlo methods in latent spatial Gaussian models

Markov chain Monte Carlo (MCMC) implementations of Bayesian inference for latent spatial Gaussian models are very computationally intensive, and restrictions on storage and computation time are limiting their application to large problems. Here we ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ICML'13: Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28

June 2013

2534 pages

Publisher

JMLR.org

Publication History

Published: 16 June 2013

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Meguelati KFontez BHilgert NMasseglia FHung CPapadopoulos G(2019)Dirichlet process mixture models made scalable and effective by means of massive distributionProceedings of the 34th ACM/SIGAPP Symposium on Applied Computing10.1145/3297280.3297327(502-509)Online publication date: 8-Apr-2019
https://dl.acm.org/doi/10.1145/3297280.3297327
Xuan JLu JZhang G(2019)A Survey on Bayesian Nonparametric LearningACM Computing Surveys10.1145/329104452:1(1-36)Online publication date: 25-Jan-2019
https://dl.acm.org/doi/10.1145/3291044
Zuanetti DMüller PZhu YYang SJi Y(2019)Bayesian nonparametric clustering for large data setsStatistics and Computing10.1007/s11222-018-9803-929:2(203-215)Online publication date: 16-Mar-2019
https://dl.acm.org/doi/10.1007/s11222-018-9803-9
Wang RLin D(2017)Scalable estimation of dirichlet process mixture models on distributed dataProceedings of the 26th International Joint Conference on Artificial Intelligence10.5555/3171837.3171935(4632-4639)Online publication date: 19-Aug-2017
https://dl.acm.org/doi/10.5555/3171837.3171935
Bouchard-Côté ADoucet ARoth A(2017)Particle gibbs split-merge sampling for Bayesian inference in mixture modelsThe Journal of Machine Learning Research10.5555/3122009.312203718:1(868-906)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.5555/3122009.3122037
Cheng DLiu YMacskassy SPerlich CLeskovec JWang WGhani R(2014)Parallel gibbs sampling for hierarchical dirichlet processes via gamma processes equivalenceProceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/2623330.2623708(562-571)Online publication date: 24-Aug-2014
https://dl.acm.org/doi/10.1145/2623330.2623708

View Options

View options

Media

Figures

Other

Tables

View Table of Contents