research-article

Accounting for burstiness in topic models

Authors:

Charles ElkanAuthors Info & Claims

ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning

Pages 281 - 288

https://doi.org/10.1145/1553374.1553410

Published: 14 June 2009 Publication History

Abstract

Many different topic models have been used successfully for a variety of applications. However, even state-of-the-art topic models suffer from the important flaw that they do not capture the tendency of words to appear in bursts; it is a fundamental property of language that if a word is used once in a document, it is more likely to be used again. We introduce a topic model that uses Dirichlet compound multinomial (DCM) distributions to model this burstiness phenomenon. On both text and non-text datasets, the new model achieves better held-out likelihood than standard latent Dirichlet allocation (LDA). It is straightforward to incorporate the DCM extension into topic models that are more complex than LDA.

References

[1]

Airoldi, E. M., Fienberg, S. E., & Xing, E. P. (2007). Mixed membership analysis of genome-wide expression data. Arxiv preprint arXiv:0711.2520.

[2]

Blei, D., & Lafferty, J. (2005). Correlated topic models. Advances in Neural Information Processing Systems 18 (pp. 147--154).

[3]

Blei, D., Ng, A., & Jordan, M. (2001). Latent Dirichlet allocation. Advances in Neural Information Processing Systems 14 (pp. 601--608).

[4]

Blei, D., Ng, A., & Jordan, M. (2003). Latent Dirichlet allocation. J. of Machine Learning Research, 3, 993--1022.

Digital Library

[5]

Celeux, G., Chaveau, D., & Diebolt, J. (1996). Stochastic versions of the EM algorithm: An experimental study in the mixture case. J. of Statistical Computation and Simulation, 55, 287--314.

[6]

Church, K., & Gale, W. A. (1995). Poisson mixtures. Natural Language Engineering, 1, 163--190.

[7]

Elkan, C. (2006). Clustering documents with an exponential-family approximation of the Dirichlet compound multinomial distribution. Proceedings of the 23rd International Conference on Machine Learning (pp. 289--296).

Digital Library

[8]

Fei-Fei, L., & Perona, P. (2005). A Bayesian hierarchical model for learning natural scene categories. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 524--531).

Digital Library

[9]

Globerson, A., Chechik, G., Pereira, F., & Tishby, N. (2004). Euclidean embedding of co-occurrence data. Advances in Neural Information Processing Systems 17 (pp. 497--504).

[10]

Griffiths, T., & Steyvers, M. (2004). Finding scientific topics. Proceedings of the National Academy of Sciences, 104, 5228--5235.

[11]

Griffiths, T., Steyvers, M., Blei, D., & Tenenbaum, J. (2004). Integrating topics and syntax. Advances in Neural Information Processing Systems 17 (pp. 537--544).

[12]

Heinrich, G. (2005). Parameter estimation for text analysis. Available at http://www.arbylon.net/publications/text-est.pdf.

[13]

Li, W., & McCallum, A. (2006). Pachinko allocation: DAG-structured mixture models of topic correlations. Proceedings of the 23rd International Conference on Machine Learning (pp. 577--584).

Digital Library

[14]

Li, W., & McCallum, A. (2008). Pachinko allocation: Scalable mixture models of topic correlations. J. of Machine Learning Research. Submitted.

[15]

Madsen, R., Kauchak, D., & Elkan, C. (2005). Modeling word burstiness using the Dirichlet distribution. Proceedings of the 22nd International Conference on Machine Learning (pp. 545--552).

Digital Library

[16]

Newton, M., & Raftery, A. (1994). Approximate Bayesian inference with the weighted likelihood bootstrap. Journal of the Royal Statistical Society B, 56, 3--48.

[17]

Rennie, J. D. M., Shih, L., Teevan, J., & Karger, D. R. (2003). Tackling the poor assumptions of naive Bayes text classifiers. Proceedings of 20th International Conference on Machine Learning (pp. 616--623).

[18]

Zhu, C., Byrd, R. H., Lu, P., & Nocedal, J. (1997). Algorithm 778: L-BFGS-B: Fortran routines for large scale bound constrained optimization. ACM Transactions on Mathematical Software, 23, 550--560.

Digital Library

Cited By

Ghadimi SEnnajari HBouguila NAmayri M(2023)Adaptive Priors for Burstiness Analysis in Topic Modeling with Generalized Dirichlet Distributions2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys)10.1109/HPCC-DSS-SmartCity-DependSys60770.2023.00134(926-933)Online publication date: 17-Dec-2023
https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys60770.2023.00134
Liang Q(2023)Tree-based data filtering for online user-generated reviewsIISE Transactions10.1080/24725854.2023.222886156:8(824-840)Online publication date: Aug-2023
https://doi.org/10.1080/24725854.2023.2228861
Buntine W(2022)Understanding Hierarchical ProcessesEntropy10.3390/e2412170324:12(1703)Online publication date: 22-Nov-2022
https://doi.org/10.3390/e24121703
Show More Cited By

Index Terms

Accounting for burstiness in topic models
1. Computing methodologies
  1. Machine learning
  2. Modeling and simulation
    1. Model development and analysis
      1. Model verification and validation
      2. Modeling methodologies
2. Mathematics of computing
  1. Probability and statistics
    1. Distribution functions

Recommendations

Topic analysis for topic-focused multi-document summarization
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Topic-focused multi-document summarization has been a challenging task because the created summary is required to be biased to the given topic or query. Existing methods consider the given topic as a single coarse unit and then directly incorporate the ...
Topic sentiment change analysis
MLDM'11: Proceedings of the 7th international conference on Machine learning and data mining in pattern recognition

Public opinions on a topic may change over time. Topic Sentiment change analysis is a new research problem consisting of two main components: (a) mining opinions on a certain topic, and (b) detect significant changes of sentiment of the opinions on the ...
Topic-driven reader comments summarization
CIKM '12: Proceedings of the 21st ACM international conference on Information and knowledge management

Readers of a news article often read its comments contributed by other readers. By reading comments, readers obtain not only complementary information about this news article but also the opinions from other readers. However, the existing ranking ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning

June 2009

1331 pages

ISBN:9781605585161

DOI:10.1145/1553374

General Chair:
Andrea Danyluk
Williams College
,
Program Chairs:
Léon Bottou
NEC Laboratories America
,
Michael Littman
Rutgers University

Copyright © 2009 Copyright 2009 by the author(s)/owner(s).

Sponsors

NSF
Microsoft Research: Microsoft Research
MITACS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 June 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

National Institutes of Health

Conference

ICML '09

Sponsor:

Microsoft Research

ICML '09: The 26th Annual International Conference on Machine Learning held in conjunction with the 2007 International Conference on Inductive Logic Programming

June 14 - 18, 2009

Quebec, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
554
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ghadimi SEnnajari HBouguila NAmayri M(2023)Adaptive Priors for Burstiness Analysis in Topic Modeling with Generalized Dirichlet Distributions2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys)10.1109/HPCC-DSS-SmartCity-DependSys60770.2023.00134(926-933)Online publication date: 17-Dec-2023
https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys60770.2023.00134
Liang Q(2023)Tree-based data filtering for online user-generated reviewsIISE Transactions10.1080/24725854.2023.222886156:8(824-840)Online publication date: Aug-2023
https://doi.org/10.1080/24725854.2023.2228861
Buntine W(2022)Understanding Hierarchical ProcessesEntropy10.3390/e2412170324:12(1703)Online publication date: 22-Nov-2022
https://doi.org/10.3390/e24121703
Kaur PMalhi APannu H(2022)Hybrid SOM based cross-modal retrieval exploiting Hebbian learningKnowledge-Based Systems10.1016/j.knosys.2021.108014239:COnline publication date: 5-Mar-2022
https://dl.acm.org/doi/10.1016/j.knosys.2021.108014
Wang DJing BLu CWu JLiu GDu CZhuang F(2021)Coarse Alignment of Topic and Sentiment: A Unified Model for Cross-Lingual Sentiment ClassificationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2020.297922532:2(736-747)Online publication date: Feb-2021
https://doi.org/10.1109/TNNLS.2020.2979225
Zhang CZhang HLi QWu KJiang DSong YLin PChen L(2021)Burstiness-Aware Web Search Analysis on Different Levels of EvidencesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2021.3109304(1-1)Online publication date: 2021
https://doi.org/10.1109/TKDE.2021.3109304
Chen XHan T(2020)A Micro Perspective of Research Dynamics Through “Citations of Citations” Topic AnalysisJournal of Data and Information Science10.2478/jdis-2020-00345:4(19-34)Online publication date: 28-Jul-2020
https://doi.org/10.2478/jdis-2020-0034
Lee NHan YXong WSong M(2020)Two layer-based trajectory analysis of the research trend in automotive fuel industryScientometrics10.1007/s11192-020-03506-5Online publication date: 25-May-2020
https://doi.org/10.1007/s11192-020-03506-5
Zhao ZPei JZhu KLo ELiu C(2019)InferSpark: Statistical Inference at Scale2019 IEEE International Conference on Big Data and Smart Computing (BigComp)10.1109/BIGCOMP.2019.8679482(1-8)Online publication date: Feb-2019
https://doi.org/10.1109/BIGCOMP.2019.8679482
Trichet RO'Connor N(2019)Gaussian Normalization: Handling Burstiness in Visual Data2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)10.1109/AVSS.2019.8909857(1-8)Online publication date: Sep-2019
https://doi.org/10.1109/AVSS.2019.8909857
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten