research-article

Utilizing relevance feedback in fusion-based retrieval

Authors:

Ella Rabinovich,

Oren KurlandAuthors Info & Claims

SIGIR '14: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval

Pages 313 - 322

https://doi.org/10.1145/2600428.2609573

Published: 03 July 2014 Publication History

Abstract

Work on using relevance feedback for retrieval has focused on the single retrieved list setting. That is, an initial document list is retrieved in response to the query and feedback for the most highly ranked documents is used to perform a second search. We address a setting wherein the list for which feedback is provided results from fusing several intermediate retrieved lists. Accordingly, we devise methods that utilize the feedback while exploiting the special characteristics of the fusion setting. Specifically, the feedback serves two different, yet complementary, purposes. The first is to directly rank the pool of documents in the intermediate lists. The second is to estimate the effectiveness of the intermediate lists for improved re-fusion. In addition, we present a meta fusion method that uses the feedback for these two purposes simultaneously. Empirical evaluation demonstrates the merits of our approach. As a case in point, the retrieval performance is substantially better than that of using the relevance feedback as in the single list setting. The performance also substantially transcends that of a previously proposed approach to utilizing relevance feedback in fusion-based retrieval.

References

[1]

N. Abdul-Jaleel, J. Allan, W. B. Croft, F. Diaz, L. Larkey, X. Li, M. D. Smucker, and C. Wade. UMASS at TREC 2004 -- novelty and hard. In Proc. of TREC-13, pages 715--725, 2004.

[2]

C. C. V. ant Garrison W. Cottrell. Fusion via linear combination of scores. Information Retrieval, 1(3):151--173, 1999.

Digital Library

[3]

J. A. Aslam and M. Montague. Models for metasearch. In Proc. of SIGIR, pages 276--284, 2001.

Digital Library

[4]

J. A. Aslam, V. Pavlu, and R. Savell. A unified model for metasearch, pooling, and system evaluation. In Proc. of CIKM, pages 484--491, 2003.

Digital Library

[5]

J. A. Aslam, V. Pavlu, and E. Yilmaz. Measure-based metasearch. In Proc. of SIGIR, pages 571--572, 2005.

Digital Library

[6]

N. Balasubramanian and J. Allan. Learning to select rankers. In Proc. of SIGIR, pages 855--856, 2010.

Digital Library

[7]

B. T. Bartell, G. W. Cottrell, and R. K. Belew. Automatic combination of multiple ranked retrieval systems. In Proc. of SIGIR, pages 173--181, 1994.

Digital Library

[8]

S. M. Beitzel, E. C. Jensen, A. Chowdhury, O. Frieder, D. A. Grossman, and N. Goharian. Disproving the fusion hypothesis: An analysis of data fusion via effective information retrieval strategies. In Proc. of SAC, pages 823--827, 2003.

Digital Library

[9]

C. Buckley and S. Robertson. Relevance feedback track overview: TREC 2008. In Proc. of TREC-17, 2008.

[10]

A. Chowdhury, O. Frieder, D. A. Grossman, and M. C. McCabe. Analyses of multiple-evidence combinations for retrieval strategies. In Proc. of SIGIR, pages 394--395, 2001.

Digital Library

[11]

W. B. Croft, editor. Advances in Information Retrieval: Recent Research from the Center for Intelligent Information Retrieval. Number 7 in The Kluwer International Series on Information Retrieval. Kluwer, 2000.

Digital Library

[12]

W. B. Croft. Combining approaches to information retrieval. In CroftciteCroft:00a, chapter 1, pages 1--36.

[13]

W. B. Croft and J. Lafferty, editors. Language Modeling for Information Retrieval. Number 13 in Information Retrieval Book Series. Kluwer, 2003.

Digital Library

[14]

E. A. Fox and J. A. Shaw. Combination of multiple searches. In Proc. of TREC-2, 1994.

[15]

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computing Systems Science, 55(1):119--139, 1997.

Digital Library

[16]

T. Hofmann, J. Puzicha, and M. I. Jordan. Learning from dyadic data. In Proc. of NIPS, pages 466--472, 1998.

Digital Library

[17]

E. Ide. New experiments in relevance feedback. In Salton G. (Ed.), The SMART Retrieval System (pp. 337--354). Englewood Cliffs, N. J.: Prentice-Hall, Inc, 1971.

[18]

M. Karimzadehgan and C. Zhai. Improving retrieval accuracy of difficult queries through generalizing negative document language models. In Proc. of CIKM, pages 27--36, 2011.

Digital Library

[19]

A. K. Kozorovitzky and O. Kurland. Cluster-based fusion of retrieved lists. In Proc. of SIGIR, pages 893--902, 2011.

Digital Library

[20]

V. Lavrenko and W. B. Croft. Relevance models in information retrieval. In Croft and LaffertyciteCroft

[21]

Lafferty:03a, pages 11--56.

[22]

J. H. Lee. Combining multiple evidence from different properties of weighting schemes. In Proc. of SIGIR, pages 180--188, 1995.

Digital Library

[23]

J. H. Lee. Analyses of multiple evidence combination. In Proc. of SIGIR, pages 267--276, 1997.

Digital Library

[24]

D. Lillis, F. Toolan, R. W. Collier, and J. Dunnion. Probfuse: a probabilistic approach to data fusion. In Proc. of SIGIR, pages 139--146, 2006.

Digital Library

[25]

I. Markov, A. Arampatzis, and F. Crestani. Unsupervised linear score normalization revisited. In Proc. of SIGIR, pages 1161--1162, 2012.

Digital Library

[26]

G. Markovits, A. Shtok, O. Kurland, and D. Carmel. Predicting query performance for fusion-based retrieval. In Proc. of CIKM, pages 813--822, 2012.

Digital Library

[27]

M. Montague and J. A. Aslam. Condorcet fusion for improved retrieval. In Proc. of CIKM, pages 538--548, 2002.

Digital Library

[28]

M. H. Montague and J. A. Aslam. Relevance score normalization for metasearch. In Proc. of CIKM, pages 427--433, 2001.

Digital Library

[29]

K. B. Ng and P. P. Kantor. An investigation of the preconditions for effective data fusion in information retrieval: A pilot study, 1998.

[30]

J. J. Rocchio. Relevance feedback in information retrieval. In G. Salton, editor, The SMART Retrieval System: Experiments in Automatic Document Processing, pages 313--323. Prentice Hall, 1971.

[31]

I. Ruthven and M. Lalmas. A survey on the use of relevance feedback for information access systems. Knowledge Engineering Review, 18(2):95--145, 2003.

Digital Library

[32]

D. Sheldon, M. Shokouhi, M. Szummer, and N. Craswell. Lambdamerge: merging the results of query reformulations. In Proc. of WSDM, pages 795--804, 2011.

Digital Library

[33]

M. Shokouhi. Segmentation of search engine results for effective data-fusion. In Proc. of ECIR, pages 185--197, 2007.

Digital Library

[34]

A. Shtok, O. Kurland, and D. Carmel. Using statistical decision theory and relevance models for query-performance prediction. In Proc. of SIGIR, 2010.

Digital Library

[35]

I. Soboroff, C. K. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. In Proc. of SIGIR, pages 66--73, 2001.

Digital Library

[36]

T. Tsikrika and M. Lalmas. Merging techniques for performing data fusion on the web. In Proc. of CIKM, pages 127--134, 2001.

Digital Library

[37]

C. C. Vogt and G. W. Cottrell. Predicting the performance of linearly combined IR systems. In Proc. of SIGIR, pages 190--196, 1998.

Digital Library

[38]

X. Wang, H. Fang, and C. Zhai. A study of methods for negative relevance feedback. In Proc. of SIGIR, pages 219--226, 2008.

Digital Library

[39]

S. Wu. Data fusion in information retrieval. Springer, 2012.

Digital Library

[40]

S. Wu and F. Crestani. Data fusion with estimated weights. In Proc. of CIKM, pages 648--651, 2002.

Digital Library

[41]

E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proc. of CIKM, pages 102--111, 2006.

Digital Library

[42]

C. Zhai and J. D. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proc. of SIGIR, pages 334--342, 2001.

Digital Library

Cited By

Lillis D(2020)On the Evaluation of Data Fusion for Information RetrievalProceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation10.1145/3441501.3441506(54-57)Online publication date: 16-Dec-2020
https://dl.acm.org/doi/10.1145/3441501.3441506
Lu XKurland OCulpepper JCraswell NRom OFang YZhang YAllan JBalog KCarterette BGuo J(2019)Relevance Modeling with Multiple Query VariationsProceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3341981.3344224(27-34)Online publication date: 26-Sep-2019
https://dl.acm.org/doi/10.1145/3341981.3344224
Roitman HSong DLiu TSun LBruza PMelucci MSebastiani FYang G(2018)Utilizing Pseudo-Relevance Feedback in Fusion-based RetrievalProceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3234944.3234969(203-206)Online publication date: 10-Sep-2018
https://dl.acm.org/doi/10.1145/3234944.3234969
Show More Cited By

Index Terms

Utilizing relevance feedback in fusion-based retrieval
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment

Recommendations

Utilizing Pseudo-Relevance Feedback in Fusion-based Retrieval
ICTIR '18: Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval

The usage of positive relevance feedback in fusion-based retrieval was previously shown to be very useful. Yet, in many retrieval use-cases, no actual relevance feedback may be available. With the absence of relevance data, pseudo-relevance feedback ...
Interactive content-based image retrieval using relevance feedback

Database search engines are generally used in a one-shot fashion in which a user provides query information to the system and, in return, the system provides a number of database instances to the user. A relevance feedback system allows the user to ...
Image retrieval based on indexing and relevance feedback

In content based image retrieval (CBIR) system, search engine retrieves the images similar to the query image according to a similarity measure. It should be fast enough and must have a high precision of retrieval. Indexing scheme is used to achieve a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '14: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval

July 2014

1330 pages

ISBN:9781450322577

DOI:10.1145/2600428

General Chairs:
Shlomo Geva
Queensland University of Technology
,
Andrew Trotman
University of Dunedin
,
Program Chairs:
Peter Bruza
Queensland University of Technology
,
Charles L.A. Clarke
University of Waterloo
,
Kal Järvelin
University of Tampere

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 July 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '14

Sponsor:

SIGIR

SIGIR '14: The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 6 - 11, 2014

Queensland, Gold Coast, Australia

Acceptance Rates

SIGIR '14 Paper Acceptance Rate 82 of 387 submissions, 21%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
474
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lillis D(2020)On the Evaluation of Data Fusion for Information RetrievalProceedings of the 12th Annual Meeting of the Forum for Information Retrieval Evaluation10.1145/3441501.3441506(54-57)Online publication date: 16-Dec-2020
https://dl.acm.org/doi/10.1145/3441501.3441506
Lu XKurland OCulpepper JCraswell NRom OFang YZhang YAllan JBalog KCarterette BGuo J(2019)Relevance Modeling with Multiple Query VariationsProceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3341981.3344224(27-34)Online publication date: 26-Sep-2019
https://dl.acm.org/doi/10.1145/3341981.3344224
Roitman HSong DLiu TSun LBruza PMelucci MSebastiani FYang G(2018)Utilizing Pseudo-Relevance Feedback in Fusion-based RetrievalProceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3234944.3234969(203-206)Online publication date: 10-Sep-2018
https://dl.acm.org/doi/10.1145/3234944.3234969
Kurland OCulpepper JCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Fusion in Information RetrievalThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210186(1383-1386)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210186
Ntalianis KDoulamis ATsapatsoulis NMastorakis N(2018)Social Relevance Feedback Based on Multimedia Content PowerIEEE Transactions on Computational Social Systems10.1109/TCSS.2017.27662505:1(109-117)Online publication date: Mar-2018
https://doi.org/10.1109/TCSS.2017.2766250
Huang WChen JMeng LLillis D(2018)Towards an open science platform for the evaluation of data fusion2018 IEEE 3rd International Conference on Cloud Computing and Big Data Analysis (ICCCBDA)10.1109/ICCCBDA.2018.8386529(290-294)Online publication date: Apr-2018
https://doi.org/10.1109/ICCCBDA.2018.8386529
Anava YShtok AKurland ORabinovich EMukhopadhyay SZhai CBertino ECrestani FMostafa JTang JSi LZhou XChang YLi YSondhi P(2016)A Probabilistic Fusion FrameworkProceedings of the 25th ACM International on Conference on Information and Knowledge Management10.1145/2983323.2983739(1463-1472)Online publication date: 24-Oct-2016
https://dl.acm.org/doi/10.1145/2983323.2983739
Ntalianis KTsapatsoulis N(2016)Wall-Content Selection in Social Media: A Revelance Feedback Scheme Based on Explicit Crowdsourcing2016 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData)10.1109/iThings-GreenCom-CPSCom-SmartData.2016.122(534-539)Online publication date: Dec-2016
https://doi.org/10.1109/iThings-GreenCom-CPSCom-SmartData.2016.122
(2016)On interactive learning-to-rank for IRNeurocomputing10.1016/j.neucom.2016.03.084208:C(3-24)Online publication date: 5-Oct-2016
https://dl.acm.org/doi/10.1016/j.neucom.2016.03.084

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents