Article

Improving realism of topic tracking evaluation

Authors:

James AllanAuthors Info & Claims

SIGIR '02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 89 - 96

https://doi.org/10.1145/564376.564394

Published: 11 August 2002 Publication History

Abstract

Topic tracking and information filtering are models of interactive tasks, but their evaluations are generally done in a way that does not reflect likely usage. The models either force frequent judgments or disallow any at all, assume the user is always available to make a judgment, and do not allow for user fatigue. In this study we extend the evaluation framework for topic tracking to incorporate those more realistic issues. We demonstrate that tracking can be done in a realistic interactive setting with minimal impact on tracking cost and with substantial reduction in required interaction.

References

[1]

J. Allan. Relevance feedback with too much data. In Proceedings of ACM SIGIR, pages 337--343, 1995.

Digital Library

[2]

J. Allan, editor. Topic Detection and Tracking: Event-based Information Organization. Kluwer Academic Publishers, Boston, 2002.

Digital Library

[3]

J. Allan, V. Lavrenko, D. Frey, and V. Khandelwal. UMass at TDT 2000. Notebook publication for participants only, Nov. 2001.

[4]

C. Cieri, S. Strassel, D. Graff, N. Martey, K. Rennert, and M. Liberman. Corpora for topic detection and tracking. In J. Allan, editor, Topic Detection and Tracking: Event-based Information Organization, pages 33--66. Kluwer Academic Publishers, 2002.

Digital Library

[5]

J. G. Fiscus and G. R. Doddington. Topic detection and tracking evaluation overview. In J. Allan, editor, Topic Detection and Tracking: Event-based Information Organization, pages 17--31. Kluwer Academic Publishers, Boston, 2002.

Digital Library

[6]

W. Hersh and P. Over. The TREC-9 interactive track report. In Proceedings of the Text Retrieval Conference (TREC-9), pages 41--50, 2001.

[7]

G. N. Lance and W. T. Williams. A general theory of classificatory sorting strategies: 1. hierarchical systems. Computer Journal, 9:373--380, 1967.

[8]

A. Leuski. Interactive Information Organization: Techniques and Evaluation. PhD thesis, University of Massachusetts at Amherst, May 2001.

Digital Library

[9]

A. Leuski and J. Allan. Evaluating a visual navigation system for a digital library. International Journal on Digital Libraries, 3(2):170--184, 2000.

[10]

D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In Proceedings of ACM SIGIR, pages 385--404, 1994.

Digital Library

[11]

I. Mani, T. Firmin, D. House, G. Klein, B. Sundheim, and L. Hirschman. The TIPSTER SUMMAC text summarization evaluation. In Proc. of EACL'99, 1999.

Digital Library

[12]

K. McKeown and D. Radev. Generating summaries of multiple news articles. In I. Mani and M. Maybury, editors, Advances in Automatic Text Summarization. MIT Press, Cambridge, Massachusetts, 1999.

[13]

S. Robertson and D. A. Hull. The TREC-9 filtering track final report. In Proceedings of the Text Retrieval Conference (TREC-9), pages 25--40, 2001.

[14]

S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In D. Harman and E. Voorhees, editors, Third Text REtrieval Conference (TREC-3). NIST, 1995.

[15]

G. Salton. Automatic Text Processing. Addison-Wesley, 1989.

Digital Library

[16]

Proceedings of the TDT 2001 workshop. Notebook publication for participants only, 2001.

Cited By

Chen CChen YChen M(2018)An Aging Theory for Event Life-Cycle ModelingIEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans10.1109/TSMCA.2006.88637037:2(237-248)Online publication date: 25-Dec-2018
https://dl.acm.org/doi/10.1109/TSMCA.2006.886370
Khy SIshikawa YKitagawa H(2018)A Novelty-based Clustering Method for On-line DocumentsWorld Wide Web10.1007/s11280-007-0018-911:1(1-37)Online publication date: 25-Dec-2018
https://dl.acm.org/doi/10.1007/s11280-007-0018-9
Shen DWang LLi C(2016)Event energy clustering and evaluation based on shock wave modelCluster Computing10.1007/s10586-016-0627-019:4(1963-1974)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1007/s10586-016-0627-0
Show More Cited By

Index Terms

Improving realism of topic tracking evaluation
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment
    2. Retrieval tasks and goals
      1. Document filtering
      2. Information extraction

Recommendations

Topic tracking based on keywords dependency profile
AIRS'08: Proceedings of the 4th Asia information retrieval conference on Information retrieval technology

Topic tracking is an important task of Topic Detection and Tracking (TDT). Its purpose is to detect stories, from a stream of news, related to known topics. Each topic is "known" by its association with several sample stories that discuss it. In this ...
Topic tracking with time granularity reasoning

Temporal information is an important attribute of a topic, and a topic usually exists in a limited period. Therefore, many researchers have explored the utilization of temporal information in topic detection and tracking (TDT). They use either a story's ...
Incorporating topic transition in topic detection and tracking algorithms

Topics often transit among documents in a document collection. To improve the accuracy of the topic detection and tracking (TDT) algorithms in discovering topics or classifying documents, it is necessary to make full use of this kind of topic transition ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval

August 2002

478 pages

ISBN:1581135610

DOI:10.1145/564376

General Chair:
Kalervo Järvelin
University of Tampere, Finland
,
Program Chairs:
Micheline Beaulieu
University of Sheffield, UK
,
Ricardo Baeza-Yates
University of Chile, Chile
,
Sung Hyon Myaeng
Chungnam National University, Korea

Copyright © 2002 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 August 2002

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR02

Sponsor:

SIGIR

SIGIR02: 25th ACM/SIGIR International Conference on Research and Development in Information Retrieval

August 11 - 15, 2002

Tampere, Finland

Acceptance Rates

SIGIR '02 Paper Acceptance Rate 44 of 219 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
631
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen CChen YChen M(2018)An Aging Theory for Event Life-Cycle ModelingIEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans10.1109/TSMCA.2006.88637037:2(237-248)Online publication date: 25-Dec-2018
https://dl.acm.org/doi/10.1109/TSMCA.2006.886370
Khy SIshikawa YKitagawa H(2018)A Novelty-based Clustering Method for On-line DocumentsWorld Wide Web10.1007/s11280-007-0018-911:1(1-37)Online publication date: 25-Dec-2018
https://dl.acm.org/doi/10.1007/s11280-007-0018-9
Shen DWang LLi C(2016)Event energy clustering and evaluation based on shock wave modelCluster Computing10.1007/s10586-016-0627-019:4(1963-1974)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1007/s10586-016-0627-0
Chen CChen MChen M(2009)An adaptive threshold framework for event detection using HMM-based life profilesACM Transactions on Information Systems10.1145/1462198.146220127:2(1-35)Online publication date: 9-Mar-2009
https://dl.acm.org/doi/10.1145/1462198.1462201
Allan JHarding SFisher DBolivar AGuzman-Lara SAmstutz P(2005)Taking Topic Detection From Evaluation to PracticeProceedings of the Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS'05) - Track 4 - Volume 0410.1109/HICSS.2005.576Online publication date: 3-Jan-2005
https://dl.acm.org/doi/10.1109/HICSS.2005.576

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents