research-article

Query-Performance Prediction Using Minimal Relevance Feedback

Authors:

David CarmelAuthors Info & Claims

ICTIR '13: Proceedings of the 2013 Conference on the Theory of Information Retrieval

Pages 14 - 21

https://doi.org/10.1145/2499178.2499201

Published: 29 September 2013 Publication History

Abstract

There has been much work on devising query-performance prediction approaches that estimate search effectiveness without relevance judgments (i.e., zero feedback). Specifically, post-retrieval predictors analyze the result list of top-retrieved documents. Departing from the zero-feedback approach, in this paper we show that relevance feedback for even very few top ranked documents can be exploited to dramatically improve prediction quality. Specifically, applying state-of-the-art zero-feedback-based predictors to only a very few relevant documents, rather than to the entire result list as originally designed, substantially improves prediction quality. This novel form of prediction is based on quantifying properties of relevant documents that can attest to query performance. We also show that integrating prediction based on relevant documents with zero-feedback-based prediction is highly effective; specifically, with respect to utilizing state-of-the-art direct estimates of retrieval effectiveness when minimal feedback is available.

References

[1]

N. Abdul-Jaleel, J. Allan, W. B. Croft, F. Diaz, L. Larkey, X. Li, M. D. Smucker, and C. Wade. UMASS at TREC 2004 --- novelty and hard. In Proc. of TREC-13, 2004.

[2]

G. Amati, C. Carpineto, and G. Romano. Query difficulty, robustness and selective application of query expansion. In Proc. of ECIR, pages 127--137, 2004.

[3]

C. Buckley. Why current IR engines fail. Information Retrieval, 12(6):652--665, 2009.

Digital Library

[4]

C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proc. of SIGIR, pages 25--32, 2004.

Digital Library

[5]

D. Carmel and E. Yom-Tov. Estimating the Query Difficulty for Information Retrieval. Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool Publishers, 2010.

Digital Library

[6]

S. Cronen-Townsend, Y. Zhou, and W. B. Croft. Predicting query performance. In Proc. of SIGIR, pages 299--306, 2002.

Digital Library

[7]

S. Cronen-Townsend, Y. Zhou, and W. B. Croft. Precision prediction based on ranked list coherence. Information Retrieval, 9(6):723--755, 2006.

Digital Library

[8]

O. Kurland, A. Shtok, D. Carmel, and S. Hummel. A unified framework for post-retrieval query-performance prediction. In Proc. of ICTIR, pages 15--26, 2011.

Digital Library

[9]

V. Lavrenko and W. B. Croft. Relevance models in information retrieval. In Language Modeling for Information Retrieval, pages 11--56. Kluwer, 2003.

[10]

M. Lease. Incorporating relevance and pseudo-relevance feedback in the markov random field model. In Proc. of TREC 2008, 2008.

[11]

Y. Lv and C. Zhai. Adaptive relevance feedback in information retrieval. In Proc. of CIKM, pages 255--264, 2009.

Digital Library

[12]

M. Sanderson. Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4(4):247--375, 2010.

[13]

A. Shtok, O. Kurland, and D. Carmel. Using statistical decision theory and relevance models for query-performance prediction. In Proc. of SIGIR, pages 259--266, 2010.

Digital Library

[14]

F. Song and W. B. Croft. A general language model for information retrieval (poster abstract). In Proc. of SIGIR, pages 279--280, 1999.

Digital Library

[15]

E. M. Voorhees. Overview of the TREC 2004 Robust Retrieval Track. In Proc. of TREC-13, 2004.

[16]

R. W. White, J. M. Jose, and I. Ruthven. An implicit feedback approach for interactive information retrieval. Inf. Process. Manage., 42:166--190, January 2006.

Digital Library

[17]

E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proc. of CIKM, pages 102--111, 2006.

Digital Library

[18]

C. Zhai and J. D. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proc. of SIGIR, pages 334--342, 2001.

Digital Library

[19]

L. Zhao, C. Liang, and J. Callan. Extending relevance model for relevance feedback. In Proc. of TREC 2008, 2008.

[20]

Y. Zhou. Retrieval Performance Prediction and Document Quality. PhD thesis, University of Massachusetts, 2007.

Digital Library

[21]

Y. Zhou and W. B. Croft. Query performance prediction in web search environments. In Proc. of SIGIR, pages 543--550, 2007.

Digital Library

Cited By

Datta SGanguly DGreene DMitra MSelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)Deep-QPPProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3498491(201-209)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3498491
Ben-Porat ORosenberg ITennenholtz MLarochelle HRanzato MHadsell RBalcan MLin H(2020)Content provider dynamics and coordination in recommendation ecosystemsProceedings of the 34th International Conference on Neural Information Processing Systems10.5555/3495724.3497313(18931-18941)Online publication date: 6-Dec-2020
https://dl.acm.org/doi/10.5555/3495724.3497313
Roitman HBalog KSetty VLioma CLiu YZhang MBerberich K(2020)ICTIR Tutorial: Modern Query Performance Prediction: Theory and PracticeProceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval10.1145/3409256.3409813(195-196)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.1145/3409256.3409813
Show More Cited By

Index Terms

Query-Performance Prediction Using Minimal Relevance Feedback
1. Information systems
  1. Information retrieval

Recommendations

Query-performance prediction: setting the expectations straight
SIGIR '14: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval

The query-performance prediction task has been described as estimating retrieval effectiveness in the absence of relevance judgments. The expectations throughout the years were that improved prediction techniques would translate to improved retrieval ...
Using statistical decision theory and relevance models for query-performance prediction
SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

We present a novel framework for the query-performance prediction task. That is, estimating the effectiveness of a search performed in response to a query in lack of relevance judgments. Our approach is based on using statistical decision theory for ...
Estimating query representativeness for query-performance prediction
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

The query-performance prediction (QPP) task is estimating retrieval effectiveness with no relevance judgments. We present a novel probabilistic framework for QPP that gives rise to an important aspect that was not addressed in previous work; namely, the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICTIR '13: Proceedings of the 2013 Conference on the Theory of Information Retrieval

September 2013

148 pages

ISBN:9781450321075

DOI:10.1145/2499178

Editors:
Oren Kurland
Technion - Israel Institute of Technology
,
Donald Metzler
Google, USA
,
Christina Lioma
University of Copenhagen, Denmark
,
Birger Larsen
University of Copenhagen, Denmark
,
Peter Ingwersen
University of Copenhagen, Denmark

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Findwise: Findwise AB
Google Inc.
Spinque: Spinque
Univ. of Copenhagen: University of Copenhagen
LARM: LARM Audio Research Archive
Royal School of Library and Information Science: Royal School of Library and Information Science
Yahoo! Labs

In-Cooperation

SIGIR: ACM Special Interest Group on Information Retrieval
British Computer Society: BCS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 September 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tag

query-performance prediction

Qualifiers

Research-article
Research
Refereed limited

Conference

ICTIR '13

Sponsor:

Findwise
Spinque
Univ. of Copenhagen
LARM
Royal School of Library and Information Science

ICTIR '13: International Conference on the Theory of Information Retrieval

September 29 - October 2, 2013

Copenhagen, Denmark

Acceptance Rates

ICTIR '13 Paper Acceptance Rate 11 of 51 submissions, 22%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
155
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Datta SGanguly DGreene DMitra MSelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)Deep-QPPProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3498491(201-209)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3498491
Ben-Porat ORosenberg ITennenholtz MLarochelle HRanzato MHadsell RBalcan MLin H(2020)Content provider dynamics and coordination in recommendation ecosystemsProceedings of the 34th International Conference on Neural Information Processing Systems10.5555/3495724.3497313(18931-18941)Online publication date: 6-Dec-2020
https://dl.acm.org/doi/10.5555/3495724.3497313
Roitman HBalog KSetty VLioma CLiu YZhang MBerberich K(2020)ICTIR Tutorial: Modern Query Performance Prediction: Theory and PracticeProceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval10.1145/3409256.3409813(195-196)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.1145/3409256.3409813
Roy DGanguly DMitra MJones G(2019)Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance predictionInformation Processing and Management: an International Journal10.1016/j.ipm.2018.10.00956:3(1026-1045)Online publication date: 1-May-2019
https://dl.acm.org/doi/10.1016/j.ipm.2018.10.009
Melucci MPaggiaro A(2019)Evaluation of information retrieval systems using structural equation modelingComputer Science Review10.1016/j.cosrev.2018.10.00131(1-18)Online publication date: Feb-2019
https://doi.org/10.1016/j.cosrev.2018.10.001
Ben Basat RTennenholtz MKurland OAllan JCroft Bde Vries AZhai C(2015)The Probability Ranking Principle is Not Optimal in Adversarial Retrieval SettingsProceedings of the 2015 International Conference on The Theory of Information Retrieval10.1145/2808194.2809456(51-60)Online publication date: 27-Sep-2015
https://dl.acm.org/doi/10.1145/2808194.2809456
Cummins R(2014)Document Score Distribution Models for Query Performance Inference and PredictionACM Transactions on Information Systems10.1145/255917032:1(1-28)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.1145/2559170
Geng BZhou FQu JZhang BCui XYin X(2014)Social Book Search with Pseudo-Relevance FeedbackNeural Information Processing10.1007/978-3-319-12640-1_25(203-211)Online publication date: 2014
https://doi.org/10.1007/978-3-319-12640-1_25

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents