research-article

A general evaluation measure for document organization tasks

Authors:

Enrique Amigó,

Felisa VerdejoAuthors Info & Claims

SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Pages 643 - 652

https://doi.org/10.1145/2484028.2484081

Published: 28 July 2013 Publication History

Abstract

A number of key Information Access tasks -- Document Retrieval, Clustering, Filtering, and their combinations -- can be seen as instances of a generic {\em document organization} problem that establishes priority and relatedness relationships between documents (in other words, a problem of forming and ranking clusters). As far as we know, no analysis has been made yet on the evaluation of these tasks from a global perspective. In this paper we propose two complementary evaluation measures -- Reliability and Sensitivity -- for the generic Document Organization task which are derived from a proposed set of formal constraints (properties that any suitable measure must satisfy).

In addition to be the first measures that can be applied to any mixture of ranking, clustering and filtering tasks, Reliability and Sensitivity satisfy more formal constraints than previously existing evaluation metrics for each of the subsumed tasks. Besides their formal properties, its most salient feature from an empirical point of view is their strictness: a high score according to the harmonic mean of Reliability and Sensitivity ensures a high score with any of the most popular evaluation metrics in all the Document Retrieval, Clustering and Filtering datasets used in our experiments.

References

[1]

E. Amigó, J. Artiles, J. Gonzalo, D. Spina, B. Liu, and A. Corujo. WePS3 Evaluation Campaign: Overview of the On-line Reputation Management Task. In 2nd Web People Search Evaluation Workshop (WePS 2010), CLEF 2010 Conference, Padova Italy, 2010.

[2]

E. Amigó, J. Gonzalo, J. Artiles, and F. Verdejo. A comparison of extrinsic clustering evaluation metrics based on formal constraints. Inf. Retr., 12:461--486, August 2009.

Digital Library

[3]

E. Amigó, J. Gonzalo, and F. Verdejo. A comparison of evaluation metrics for document filtering. In Proceedings of CLEF'11, CLEF'11, pages 38--49, Berlin, Heidelberg, 2011. Springer-Verlag.

Digital Library

[4]

C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR '04, pages 25--32, New York, NY, USA, 2004. ACM.

Digital Library

[5]

B. Carterette. System effectiveness, user models, and user utility: a conceptual framework for investigation. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, SIGIR '11, pages 903--912, New York, NY, USA, 2011. ACM.

Digital Library

[6]

B. Carterette, E. Kanoulas, and E. Yilmaz. Simulating simple user behavior for system effectiveness evaluation. In CIKM, pages 611--620, 2011.

Digital Library

[7]

O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In WWW, pages 1--10, 2009.

Digital Library

[8]

J. M. Cigarrán, A. Peñas, J. Gonzalo, and F. Verdejo. Automatic selection of noun phrases as document descriptors in an fca-based information retrieval system. In ICFCA, pages 49--63, 2005.

Digital Library

[9]

C. L. A. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty and diversity in information retrieval evaluation. In SIGIR, pages 659--666, 2008.

Digital Library

[10]

J. Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37, 1960.

[11]

G. Cormack and T. Lynam. Trec 2005 spam track overview. In Proceedings of the fourteenth Text Retrieval Conference 8TREC 2005), 2005.

[12]

G. V. Cormack and T. R. Lynam. TREC 2005 Spam Track Overview. In Proceedings of the fourteenth Text REtrieval Conference (TREC-2005), 2005.

[13]

M. Halkidi, Y. Batistakis, and M. Vazirgiannis. On Clustering Validation Techniques. Journal of Intelligent Information Systems, 17(2--3):107--145, 2001.

Digital Library

[14]

M. A. Hearst and J. O. Pedersen. Reexamining the cluster hypothesis: Scatter/gather on retrieval results. pages 76--84, 1996.

Digital Library

[15]

B. Hu, Y. Zhang, W. Chen, G. Wang, and Q. Yang. Characterizing search intent diversity into click models. In Proceedings of the 20th international conference on World wide web, WWW '11, pages 17--26, New York, NY, USA, 2011. ACM.

Digital Library

[16]

K. J\"arvelin and J. Kek\"al\"ainen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20:422--446, October 2002.

Digital Library

[17]

B. Krishnamurthy, P. Gill, and M. Arlitt. A few chirps about twitter. In WOSP '08: Proceedings of the first workshop on Online social networks, pages 19--24, New York, NY, USA, 2008. ACM.

Digital Library

[18]

A. Leuski. Evaluating document clustering for interactive information retrieval. In CIKM, pages 33--40, 2001.

Digital Library

[19]

M. Meila. Comparing clusterings. In Proceedings of COLT 03, 2003.

[20]

T. M. Mitchell. Machine learning. McGraw Hill, New York, 1997.

Digital Library

[21]

A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., 27(1):2:1--2:27, Dec. 2008.

Digital Library

[22]

A. Rosenberg and J. Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of EMNLP-CoNLL 2007, pages 410--420, 2007.

[23]

M. D. Smucker and C. L. Clarke. Time-based calibration of effectiveness measures. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, SIGIR '12, pages 95--104, New York, NY, USA, 2012. ACM.

Digital Library

[24]

S. Vargas and P. Castells. Rank and relevance in novelty and diversity metrics for recommender systems. In 5th ACM Conference on Recommender Systems (RecSys 2011), pages 109--116, Chicago, Illinois, October 2011.

Digital Library

[25]

E. M. Voorhees. The trec-8 question answering track report. In In Proceedings of TREC-8, pages 77--82, 1999.

Cited By

Schumacher TLutz MSikdar SStrohmaier M(2024)Properties of Group Fairness Measures for RankingsACM Transactions on Social Computing10.1145/3674883Online publication date: 27-Aug-2024
https://dl.acm.org/doi/10.1145/3674883
Wu HZhang YMa CLyu FHe BMitra BLiu X(2024)Result Diversification in Search and Recommendation: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.338226236:10(5354-5373)Online publication date: Oct-2024
https://doi.org/10.1109/TKDE.2024.3382262
Giner F(2024)An Intrinsic Framework of Information Retrieval Evaluation MeasuresIntelligent Systems and Applications10.1007/978-3-031-47721-8_47(692-713)Online publication date: 10-Jan-2024
https://doi.org/10.1007/978-3-031-47721-8_47
Show More Cited By

Index Terms

A general evaluation measure for document organization tasks
1. Hardware
  1. Robustness

Recommendations

A new suffix tree similarity measure for document clustering
WWW '07: Proceedings of the 16th international conference on World Wide Web

In this paper, we propose a new similarity measure to compute the pairwise similarity of text-based documents based on suffix tree document model. By applying the new suffix tree similarity measure in Group-average Agglomerative Hierarchical Clustering (...
Measurement of clustering effectiveness for document collections
Abstract
Clustering of the contents of a document corpus is used to create sub-corpora with the intention that they are expected to consist of documents that are related to each other. However, while clustering is used in a variety of ways in document ...
Pairwise-adaptive dissimilarity measure for document clustering

This paper introduces a novel pairwise-adaptive dissimilarity measure for large high dimensional document datasets that improves the unsupervised clustering quality and speed compared to the original cosine dissimilarity measure. This measure ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

July 2013

1188 pages

ISBN:9781450320344

DOI:10.1145/2484028

General Chairs:
Gareth J.F. Jones
Dublin City University, Ireland
,
Páraic Sheridan
Dublin City University, Ireland
,
Program Chairs:
Diane Kelly
University of North Carolina, Chapel Hill, USA
,
Maarten de Rijke
University of Amsterdam, The Netherlands
,
Tetsuya Sakai
Microsoft Research Asia, China

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 July 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tag

effectiveness measures

Qualifiers

Research-article

Conference

SIGIR '13

Sponsor:

SIGIR

SIGIR '13: The 36th International ACM SIGIR conference on research and development in Information Retrieval

July 28 - August 1, 2013

Dublin, Ireland

Acceptance Rates

SIGIR '13 Paper Acceptance Rate 73 of 366 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

66
Total Citations
View Citations
640
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)5

Reflects downloads up to 15 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schumacher TLutz MSikdar SStrohmaier M(2024)Properties of Group Fairness Measures for RankingsACM Transactions on Social Computing10.1145/3674883Online publication date: 27-Aug-2024
https://dl.acm.org/doi/10.1145/3674883
Wu HZhang YMa CLyu FHe BMitra BLiu X(2024)Result Diversification in Search and Recommendation: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.338226236:10(5354-5373)Online publication date: Oct-2024
https://doi.org/10.1109/TKDE.2024.3382262
Giner F(2024)An Intrinsic Framework of Information Retrieval Evaluation MeasuresIntelligent Systems and Applications10.1007/978-3-031-47721-8_47(692-713)Online publication date: 10-Jan-2024
https://doi.org/10.1007/978-3-031-47721-8_47
Giner F(2023)Information Retrieval Evaluation Measures Defined on Some Axiomatic Models of PreferencesACM Transactions on Information Systems10.1145/363217142:3(1-35)Online publication date: 8-Nov-2023
https://dl.acm.org/doi/10.1145/3632171
Giner FCrestani FPasi GGaussier E(2022)On the Effect of Ranking Axioms on IR Evaluation MetricsProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3539813.3545153(13-23)Online publication date: 23-Aug-2022
https://dl.acm.org/doi/10.1145/3539813.3545153
Amigó EMizzaro SSpina DAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Ranking InterruptusProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3532051(588-598)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3532051
Völske MBondarenko AFröbe MStein BSingh JHagen MAnand AHasibi FFang YAizawa A(2021)Towards Axiomatic Explanations for Neural Ranking ModelsProceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3471158.3472256(13-22)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3471158.3472256
Parapar JRadlinski F(2021)Towards Unified Metrics for Accuracy and Diversity for Recommender SystemsProceedings of the 15th ACM Conference on Recommender Systems10.1145/3460231.3474234(75-84)Online publication date: 13-Sep-2021
https://dl.acm.org/doi/10.1145/3460231.3474234
Sharma TDyer HBashir M(2021)Enabling User-centered Privacy Controls for Mobile ApplicationsACM Transactions on Internet Technology10.1145/343477721:1(1-24)Online publication date: 30-Jan-2021
https://dl.acm.org/doi/10.1145/3434777
Chicha EBouna BNassar MChbeir RHaraty ROussalah MBenslimane DAlraja M(2021)A User-Centric Mechanism for Sequentially Releasing Graph Datasets under Blowfish PrivacyACM Transactions on Internet Technology10.1145/343150121:1(1-25)Online publication date: 17-Feb-2021
https://dl.acm.org/doi/10.1145/3431501
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents