research-article

Learning to Reinforce Search Effectiveness

Authors:

Hui YangAuthors Info & Claims

ICTIR '15: Proceedings of the 2015 International Conference on The Theory of Information Retrieval

Pages 271 - 280

https://doi.org/10.1145/2808194.2809468

Published: 27 September 2015 Publication History

Abstract

Session search is an Information Retrieval (IR) task which handles a series of queries issued for a search task. In this paper, we propose a novel reinforcement learning style information retrieval framework and develop a new feedback learning algorithm to model user feedback, including clicks and query reformulations, as reinforcement signals and to generate rewards in the RL framework. From a new perspective, we view session search as a cooperative game played between two agents, the user and the search engine. We study the communications between the two agents; they always exchange opinions on "whether the current stage of search is relevant" and "whether we should explore now." The algorithm infers user feedback models by an EM algorithm from the query logs. We compare to several state-of-the-art session search algorithms and evaluate our algorithm on the most recent TREC 2012 to 2014 Session Tracks. The experimental results demonstrates that our approach is highly effective for improving session search accuracy.

References

[1]

Y. Artzi and L. Zettlemoyer. Bootstrapping semantic parsers from conversations. In EMNLP '11.

Digital Library

[2]

D. P. Bertsekas. Dynamic Programming: Deterministic and Stochastic Models. Prentice-Hall, 1987.

Digital Library

[3]

S. R. K. Branavan, H. Chen, L. S. Zettlemoyer, and R. Barzilay. Reinforcement learning for mapping instructions to actions. In ACL '09.

Digital Library

[4]

S. R. K. Branavan, D. Silver, and R. Barzilay. Non-linear monte-carlo search in civilization ii. In IJCAI'11.

Digital Library

[5]

S. R. K. Branavan, L. S. Zettlemoyer, and R. Barzilay. Reading between the lines: learning to map high-level instructions to commands. In ACL '10.

Digital Library

[6]

D. L. Chen and R. J. Mooney. Learning to sportscast: A test of grounded language acquisition. In ICML '08.

Digital Library

[7]

A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data via the EM Algorithm. J. Roy. Statist. Soc. Ser. B, 39(1):1--38, 1977.

[8]

A. Diriye, R. White, G. Buscher, and S. Dumais. Leaving so soon?: Understanding and predicting web search abandonment rationales. In CIKM '12, pages 1025--1034, 2012.

Digital Library

[9]

J. Eisenstein, J. Clarke, D. Goldwasser, and D. Roth. Reading to learn: constructing features from semantic abstracts. In EMNLP '09.

Digital Library

[10]

S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst., 23(2), Apr. 2005.

Digital Library

[11]

K. Georgila, C. Nelson, and D. Traum. Single-agent vs. multi-agent techniques for concurrent reinforcement learning of negotiation dialogue policies. In ACL '14.

[12]

D. Goldwasser, R. Reichart, J. Clarke, and D. Roth. Confidence driven unsupervised semantic parsing. In HLT '11.

Digital Library

[13]

D. Guan, S. Zhang, and H. Yang. Utilizing query change for session search. In SIGIR '13.

Digital Library

[14]

R. Howard. Dynamic Programming and Markov Process. MIT Press, 1960.

[15]

J. Hu and M. P. Wellman. Multiagent reinforcement learning: Theoretical framework and an algorithm. In ICML '98.

Digital Library

[16]

K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422--446, 2002.

Digital Library

[17]

L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. J Artificial Intelligence Res., 4:237--285, 1996.

Digital Library

[18]

P. Kanani, A. McCallum, and S. Hu. Resource-bounded information extraction: Acquiring missing feature values on demand. In Advances in Knowledge Discovery and Data Mining, 2010.

Digital Library

[19]

E. Kanoulas, B. Carterette, M. Hall, P. Clough, and M. Sanderson. Overview of the trec 2013 session track. In TREC'13.

[20]

E. Kanoulas, B. Carterette, M. Hall, P. Clough, and M. Sanderson. Overview of the trec 2014 session track. In TREC'14.

[21]

R. J. Kate and R. J. Mooney. Learning language semantics from ambiguous supervision. In AAAI '07.

Digital Library

[22]

M. Lauer and M. A. Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In ICML '00.

Digital Library

[23]

V. Lesser, B. Horling, F. Klassner, A. Raja, T. Wagner, and S. Zhang. BIG: An Agent for Resource-Bounded Information Gathering and Decision Making. Artificial Intelligence Journal, Special Issue on Internet Information Agents, 118(1--2):197--244, 2000.

Digital Library

[24]

P. Liang, M. I. Jordan, and D. Klein. Learning semantic correspondences with less supervision. In ACL '09.

Digital Library

[25]

M. L. Littman. markov games as a framework for multi-agent reinforcement learning. In ICML '94.

[26]

R. T. Loftin, J. MacGlashan, B. Peng, M. E. Taylor, M. L. Littman, J. Huang, and D. L. Roberts. A strategy-aware technique for learning behaviors from discrete human feedback. In AAAI '14.

[27]

J. Luo, S. Zhang, and H. Yang. Win-win search: Dual-agent stochastic game in session search. In SIGIR '14.

Digital Library

[28]

D. Meger, P.-E. Forssén, K. Lai, S. Helmer, S. McCann, T. Southey, M. Baumann, J. J. Little, and D. G. Lowe. Curious george: An attentive semantic robot. Robot. Auton. Syst., 56(6):503--511, 2008.

Digital Library

[29]

L. Peshkin, K.-E. Kim, N. Meuleau, and L. P. Kaelbling. Learning to cooperate via policy search. In UAI '00.

Digital Library

[30]

J. Schmidhuber. Sequential decision making based on direct search. In R. Sun and C. L. Giles, editors, Sequence Learning: Paradigms, Algorithms, and Applications. Springer, 2001.

Digital Library

[31]

R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.

Digital Library

[32]

M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In ICML '93.

[33]

A. Vogel and D. Jurafsky. Learning to follow navigational directions. In ACL '10.

Digital Library

[34]

H. Yang, M. Sloan, and J. Wang. Tutorial on dynamic information retrieval modeling. In SIGIR '14.

Digital Library

[35]

H. Yang, M. Sloan, and J. Wang. Tutorial on dynamic information retrieval modeling. In WSDM '15.

Digital Library

[36]

C. Zhang, S. Abdallah, and V. Lesser. Efficient multi-agent reinforcement learning through automated supervision. In AAMAS '08.

Digital Library

Cited By

Qu CYang LChen CCroft WKrishna KIyyer M(2021)Weakly-Supervised Open-Retrieval Conversational Question AnsweringAdvances in Information Retrieval10.1007/978-3-030-72113-8_35(529-543)Online publication date: 27-Mar-2021
https://doi.org/10.1007/978-3-030-72113-8_35
Li ZKiseleva Jde Rijke MGrotov AKamps JKanoulas Ede Rijke MFang HYilmaz E(2017)Towards Learning Reward Functions from User InteractionsProceedings of the ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3121050.3121098(289-292)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1145/3121050.3121098
Moraes FSantos RZiviani NKamps JKanoulas Ede Rijke MFang HYilmaz E(2017)On Effective Dynamic Search in Specialized DomainsProceedings of the ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3121050.3121065(177-184)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1145/3121050.3121065
Show More Cited By

Index Terms

Learning to Reinforce Search Effectiveness
1. Information systems
  1. Information retrieval

Recommendations

Win-win search: dual-agent stochastic game in session search
SIGIR '14: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval

Session search is a complex search task that involves multiple search iterations triggered by query reformulations. We observe a Markov chain in session search: user's judgment of retrieved documents in the previous search iteration affects user's ...
Query change as relevance feedback in session search
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Session search is the Information Retrieval (IR) task that performs document retrieval for an entire session. During a session, users often change queries to explore and investigate the information needs. In this paper, we propose to use query change as ...
Automated index management for distributed web search
CIKM '03: Proceedings of the twelfth international conference on Information and knowledge management

Distributed heterogeneous search systems are an emerging phenomenon in Web search, in which independent topic-specific search engines provide search services, and metasearchers distribute user's queries to only the most suitable search engines. Previous ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICTIR '15: Proceedings of the 2015 International Conference on The Theory of Information Retrieval

September 2015

402 pages

ISBN:9781450338332

DOI:10.1145/2808194

General Chairs:
James Allan
University of Massachusetts Amherst, USA
,
Bruce Croft
University of Massachusetts Amherst, USA
,
Program Chairs:
Arjen de Vries
CWI Amsterdam, The Netherlands
,
Chengxiang Zhai
University of Illinois at Urbana-Champaign, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ICTIR '15

Sponsor:

SIGIR

ICTIR '15: ACM SIGIR International Conference on the Theory of Information Retrieval

September 27 - 30, 2015

Massachusetts, Northampton, USA

Acceptance Rates

ICTIR '15 Paper Acceptance Rate 29 of 57 submissions, 51%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
143
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Qu CYang LChen CCroft WKrishna KIyyer M(2021)Weakly-Supervised Open-Retrieval Conversational Question AnsweringAdvances in Information Retrieval10.1007/978-3-030-72113-8_35(529-543)Online publication date: 27-Mar-2021
https://doi.org/10.1007/978-3-030-72113-8_35
Li ZKiseleva Jde Rijke MGrotov AKamps JKanoulas Ede Rijke MFang HYilmaz E(2017)Towards Learning Reward Functions from User InteractionsProceedings of the ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3121050.3121098(289-292)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1145/3121050.3121098
Moraes FSantos RZiviani NKamps JKanoulas Ede Rijke MFang HYilmaz E(2017)On Effective Dynamic Search in Specialized DomainsProceedings of the ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3121050.3121065(177-184)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1145/3121050.3121065
Yang GSloan MWang J(2016)Dynamic Information Retrieval ModelingSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S00718ED1V01Y201605ICR0498:3(1-144)Online publication date: 15-Jun-2016
https://doi.org/10.2200/S00718ED1V01Y201605ICR049

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten