Cited By
View all- Sakai TZeng Z(2020)Retrieval Evaluation Measures that Agree with Users’ SERP PreferencesACM Transactions on Information Systems10.1145/343181339:2(1-35)Online publication date: 31-Dec-2020
Merge-Tie-Judge: Low-Cost Preference Judgments with Ties
Authors: 
Kai Hui, 
Klaus BerberichAuthors Info & Claims
Kai Hui, Klaus BerberichAuthors Info & ClaimsPages 277 - 280
Abstract
Preference judgments have been demonstrated to yield more accurate labels than graded judgments and also forego the need to define grades upfront. These benefits, however, come at the cost of a larger number of judgments that is required. Prior research, by exploiting the transitivity of preferences, successfully reduced the overall number of preference judgments required to O(N log(N)) for N documents, which is still prohibitive in practice. In this work, we reduce the overall number of preference judgments required by allowing for ties and exploiting that ties naturally cluster documents. Our novel judgment mechanism Merge-Tie-Judge exploits this ``clustering effect'' by automatically inferring preferences between documents from different clusters. Experiments on relevance judgments from the TREC Web Track show that the proposed mechanism requires fewer judgments
References
[1]
N. Ailon and M. Mohri. An efficient reduction of ranking to classification. arXiv preprint arXiv:0710.2889, 2007.
[2]
M. Bashir, J. Anderton, J. Wu, P. B. Golbus, V. Pavlu, and J. A. Aslam. A document rating system for preference judgements. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 909--912. ACM, 2013.
[3]
B. Carterette, P. N. Bennett, D. M. Chickering, and S. T. Dumais. Here or there: Preference Judgments for Relevance. In Advances in Information Retrieval, ECIR '08, pages 16--27. Springer, 2008.
[4]
K. Hui and K. Berberich. Low-cost preference judgment via ties. In Advances in Information Retrieval, ECIR '17, pages 626--632. Springer, 2017.
[5]
K. Hui and K. Berberich. Transitivity, time consumption, and quality of preference judgments in crowdsourcing. In Advances in Information Retrieval, ECIR '17, pages 239--251. Springer, 2017.
[6]
G. Kazai, N. Craswell, E. Yilmaz, and S. Tahaghoghi. An analysis of systematic judging errors in information retrieval. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM '12, pages 105--114, New York, NY, USA, 2012. ACM.
[7]
S. Niu, J. Guo, Y. Lan, and X. Cheng. Top-k learning to rank: labeling, ranking and evaluation. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 751--760. ACM, 2012.
[8]
K. Radinsky and N. Ailon. Ranking from pairs and triplets: Information quality, evaluation methods and query complexity. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM '11, pages 105--114, New York, NY, USA, 2011. ACM.
[9]
M. E. Rorvig. The simple scalability of documents. Journal of the American Society for Information Science, 41(8):590, 1990.
[10]
R. Song, Q. Guo, R. Zhang, G. Xin, J.-R. Wen, Y. Yu, and H.-W. Hon. Select-the-best-ones: A new way to judge relative relevance. Information processing & management, 47(1):37--52, 2011.
[11]
S. Tong and D. Koller. Support vector machine active learning with applications to text classification. The Journal of Machine Learning Research, 2:45--66, 2002.
Index Terms
- Merge-Tie-Judge: Low-Cost Preference Judgments with Ties
Recommendations
Preferences on a Budget: Prioritizing Document Pairs when Crowdsourcing Relevance Judgments
WWW '22: Proceedings of the ACM Web Conference 2022In Information Retrieval (IR) evaluation, preference judgments are collected by presenting to the assessors a pair of documents and asking them to select which of the two, if any, is the most relevant. This is an alternative to the classic relevance ...
Dealing with Incomplete Judgments in Cascade Measures
ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information RetrievalCascade measures like alpha-nDCG, ERR-IA, and NRBP take into account novelty and diversity of query results and are computed using judgments provided by humans, which are costly to collect. These measures expect that all documents in the result list of ...
Using PageRank to infer user preferences
SIGIR '12: Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrievalRecently, researchers have shown interest in the use of preference judgments for evaluation in IR literature. Although preference judgments have several advantages over absolute judgment, one of the major disadvantages is that the number of judgments ...
Comments
Information & Contributors
Information
Published In
October 2017
348 pages
ISBN:9781450344906
DOI:10.1145/3121050
- General Chairs:
- Jaap Kamps,
- Evangelos Kanoulas,
- Maarten de Rijke,
- Program Chairs:
- Hui Fang,
- Emine Yilmaz
Copyright © 2017 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 October 2017
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
ICTIR '17
Sponsor:
ICTIR '17: ACM SIGIR International Conference on the Theory of Information Retrieval
October 1 - 4, 2017
Amsterdam, The Netherlands
Acceptance Rates
ICTIR '17 Paper Acceptance Rate 27 of 54 submissions, 50%;
Overall Acceptance Rate 235 of 527 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 51Total Downloads
- Downloads (Last 12 months)1
- Downloads (Last 6 weeks)0
Reflects downloads up to 03 Jan 2025
Other Metrics
Citations
Cited By
View all- Sakai TZeng Z(2020)Retrieval Evaluation Measures that Agree with Users’ SERP PreferencesACM Transactions on Information Systems10.1145/343181339:2(1-35)Online publication date: 31-Dec-2020
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in