Article

Novelty and redundancy detection in adaptive filtering

Authors:

Thomas MinkaAuthors Info & Claims

SIGIR '02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 81 - 88

https://doi.org/10.1145/564376.564393

Published: 11 August 2002 Publication History

Abstract

This paper addresses the problem of extending an adaptive information filtering system to make decisions about the novelty and redundancy of relevant documents. It argues that relevance and redundance should each be modelled explicitly and separately. A set of five redundancy measures are proposed and evaluated in experiments with and without redundancy thresholds. The experimental results demonstrate that the cosine similarity metric and a redundancy measure based on a mixture of language models are both effective for identifying redundant documents.

References

[1]

J. Allan, J. Carbonell, G. Doddington, J. Yamron, and Y. Yang. Topic detection and tracking pilot study. In Topic Detection and Tracking Workshop Report 2001.

[2]

J. Allan, V. Lavrenko, and H. Jin. First story detetion in TDT is hard. In Proc. of the 9th International Conference on Information and Knowledge Management 2000.

Digital Library

[3]

J. Allan, R. Papka, and V. Lavrenko. On- line new event detection and tracking. In Proc. of 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 1998.

Digital Library

[4]

J. Carbonell, Y. Yang, R. Brown, C. Jin, and J. Zhang. CMU TDT report 13-- 14 Nov 2001. In Topic Detection and Tracking Workshop Report 2001.

[5]

M. Franz, A. Ittycheriah, J. S. McCarley, andT. Ward. First story detection: Combining similarity and novelty based approaches. In Topic Detection and Tracking Workshop Report 2001.

[6]

W. P. Jones and G. W. Furnas. Pictures of relevance. Journal of the American Society for Information Science 1987.

[7]

W. Kraaij, R. Pohlmann, and D. Hiemstra. Twenty- one at TREC- 8: using language technology for information retrieval. In Proceedings of the Eighth Text REtrieval Conference (TREC- 8), 1999.

[8]

L. Lee. Measures of distributional similarity. In Proceedings of the 37th ACL 1999.

Digital Library

[9]

A. McCallum, R. Rosenfeld, T. Mitchell, and A. Y. Ng. Improving text classification by shrinkage in a hierarchy of classes. In Proceedings of The Eighteenth International Conference on Machine Learning 1998.

Digital Library

[10]

D. R. H. Miller, T. Leek, and R. Schwartz. A hidden markov model information retrieval system. In Proceedings of the 22th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval pages 214 --221, 2001.

Digital Library

[11]

S. Robertson. Threshold setting in adaptive filtering. Journal of Documentation 2000.

[12]

S. Robertson and D. Hull. The TREC- 9 Filtering track report. In The Ninth Text REtrieval Conference (TREC- 9), 2001.

[13]

M. Spitters and W. Kraaij. TNO at TDT2001: Language model-based topic detection. In Topic Detection and Tracking Workshop Report. 2001.

[14]

N. Stokes and J. Carthy. Combining semantic and syntactic document classifiers to improve first story detection. In Proceedings of the 24th Annual International ACM SIGIR Conferenc eon Research and Development in Information Retrieval 2001.

Digital Library

[15]

J. Yamron, S. Knecht, and P. van Mulbregt. Dragon's tracking and detection systems for the TDT2000 evaluation. In Proceedings of the Broadcast News Transcription and Understanding Workshop 1998.

[16]

C. Zhai and J. Lafferty. Model- based feedback in the language modeling approach to information retrieval. In Proceedings of Tenth International Conference on Information and Knowledge Management 2001.

Digital Library

[17]

C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proc. of the 24th Annual Int'l ACM SIGIR Conferenc eon Research and Development in Information Retrieval pages 334 --342, 2001.

Digital Library

[18]

Y. Zhang and J. Callan. Maximum likelihood estimation for lteirng thresholds. In Proc. of the 24th Annual Int'l ACM SIGIR Conference on Research and Development in Information Retrieval 2001.

Digital Library

Cited By

Nguyen KNguyen NNgo NTran NNguyen H(2024)Investigating Consumers’ Purchase Resistance Behavior to AI-Based Content Recommendations on Short-Video Platforms: A Study of Greedy And Biased RecommendationsJournal of Internet Commerce10.1080/15332861.2024.237596623:3(284-327)Online publication date: 11-Jul-2024
https://doi.org/10.1080/15332861.2024.2375966
Suzuki TGoto Y(2024)Development of a Product Recommendation Method for the Real-Time Cart Context in Supermarkets by Integrating Cooking Recipes and Purchase Co-occurrencesThe Review of Socionetwork Strategies10.1007/s12626-024-00171-518:2(279-301)Online publication date: 27-Jul-2024
https://doi.org/10.1007/s12626-024-00171-5
Delon FBédubourg GBouscarrat LMeynard JValois AQueyriaux BRamisch CTanti M(2024)Infectious risk events and their novelty in event-based surveillance: new definitions and annotated corpusLanguage Resources and Evaluation10.1007/s10579-024-09728-wOnline publication date: 5-Mar-2024
https://doi.org/10.1007/s10579-024-09728-w
Show More Cited By

Index Terms

Novelty and redundancy detection in adaptive filtering
1. Information systems
  1. Information systems applications
2. Software and its engineering

Recommendations

Adaptive Feature Redundancy Minimization
CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

Most existing feature selection methods select the top-ranked features according to certain criterion. However, without considering the redundancy among the features, the selected ones are frequently highly correlated with each other, which is ...
Link-Time Path-Sensitive Memory Redundancy Elimination
HPCA '04: Proceedings of the 10th International Symposium on High Performance Computer Architecture

Optimizations performed at link-time or directly applied to final program executables have received increased attention in recent years. This paper discusses the discovery and elimination of redundant memory operations in the context of a link-time ...
Measuring novelty and redundancy with multiple modalities in cross-lingual broadcast news

News videos from different channels, languages are broadcast everyday, which provide abundant information for users. To effectively search, retrieve, browse and track news stories, news story similarity plays a critical role in assessing the novelty and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval

August 2002

478 pages

ISBN:1581135610

DOI:10.1145/564376

General Chair:
Kalervo Järvelin
University of Tampere, Finland
,
Program Chairs:
Micheline Beaulieu
University of Sheffield, UK
,
Ricardo Baeza-Yates
University of Chile, Chile
,
Sung Hyon Myaeng
Chungnam National University, Korea

Copyright © 2002 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 August 2002

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGIR02

Sponsor:

SIGIR

SIGIR02: 25th ACM/SIGIR International Conference on Research and Development in Information Retrieval

August 11 - 15, 2002

Tampere, Finland

Acceptance Rates

SIGIR '02 Paper Acceptance Rate 44 of 219 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

285
Total Citations
View Citations
1,795
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)8

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nguyen KNguyen NNgo NTran NNguyen H(2024)Investigating Consumers’ Purchase Resistance Behavior to AI-Based Content Recommendations on Short-Video Platforms: A Study of Greedy And Biased RecommendationsJournal of Internet Commerce10.1080/15332861.2024.237596623:3(284-327)Online publication date: 11-Jul-2024
https://doi.org/10.1080/15332861.2024.2375966
Suzuki TGoto Y(2024)Development of a Product Recommendation Method for the Real-Time Cart Context in Supermarkets by Integrating Cooking Recipes and Purchase Co-occurrencesThe Review of Socionetwork Strategies10.1007/s12626-024-00171-518:2(279-301)Online publication date: 27-Jul-2024
https://doi.org/10.1007/s12626-024-00171-5
Delon FBédubourg GBouscarrat LMeynard JValois AQueyriaux BRamisch CTanti M(2024)Infectious risk events and their novelty in event-based surveillance: new definitions and annotated corpusLanguage Resources and Evaluation10.1007/s10579-024-09728-wOnline publication date: 5-Mar-2024
https://doi.org/10.1007/s10579-024-09728-w
Gupta KAhmad AGhosal TEkbal A(2024)SciND: a new triplet-based dataset for scientific novelty detection via knowledge graphsInternational Journal on Digital Libraries10.1007/s00799-023-00386-x25:4(639-659)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s00799-023-00386-x
Nair B(2023)Predicting document novelty: an unsupervised learning approachKnowledge and Information Systems10.1007/s10115-023-01989-166:3(1709-1728)Online publication date: 12-Oct-2023
https://doi.org/10.1007/s10115-023-01989-1
Ghosal TTiwary PPatton RStahl C(2022)Towards establishing a research lineage via identification of significant citationsQuantitative Science Studies10.1162/qss_a_001702:4(1511-1528)Online publication date: 4-Feb-2022
https://doi.org/10.1162/qss_a_00170
Ghosal TSaikh TBiswas TEkbal ABhattacharyya P(2022)Novelty Detection: A Perspective from Natural Language ProcessingComputational Linguistics10.1162/coli_a_0042948:1(77-117)Online publication date: 4-Apr-2022
https://doi.org/10.1162/coli_a_00429
Pavlopoulou NCurry E(2022)PoSSUM: An Entity-centric Publish/Subscribe System for Diverse Summarization in Internet of ThingsACM Transactions on Internet Technology10.1145/350791122:3(1-30)Online publication date: 24-Mar-2022
https://dl.acm.org/doi/10.1145/3507911
Jasim MCollins CSarvghad AMahyar N(2022)Supporting Serendipitous Discovery and Balanced Analysis of Online Product Reviews with Interaction-Driven Metrics and Bias-Mitigating SuggestionsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517649(1-24)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517649
Zhou YGuo JSong BChen CChang JYu F(2022)Trust-aware Multi-task Knowledge Graph for RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3221160(1-14)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3221160
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents