research-article

An Optimization Framework for Weighting Implicit Relevance Labels for Personalized Web Search

Authors:

Yury Ustinovskiy,

Pavel SerdyukovAuthors Info & Claims

WWW '15: Proceedings of the 24th International Conference on World Wide Web

Pages 1144 - 1154

https://doi.org/10.1145/2736277.2741105

Published: 18 May 2015 Publication History

Abstract

Implicit feedback from users of a web search engine is an essential source providing consistent personal relevance labels from the actual population of users. However, previous studies on personalized search employ this source in a rather straightforward manner. Basically, documents that were clicked on get maximal gain, and the rest of the documents are assigned the zero gain. As we demonstrate in our paper, a ranking algorithm trained using these gains directly as the ground truth relevance labels leads to a suboptimal personalized ranking.

In this paper we develop a framework for automatic reweighting of these labels. Our approach is based on more subtle aspects of user interaction with the result page. We propose an efficient methodology for deriving confidence levels for relevance labels that relies directly on the objective ranking measure. All our algorithms are evaluated on a large-scale query log provided by a major commercial search engine. The results of the experiments prove that the current state-of-the-art personalization approaches could be significantly improved by enriching relevance grades with weights extracted from post-impression user behavior.

References

[1]

E. Agichtein, E. Brill, and S. Dumais. Improving web search ranking by incorporating user behavior information. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '06, pages 19--26, New York, NY, USA, 2006. ACM.

Digital Library

[2]

A. Aitken. On least squares and linear combination of observations. volume 55 of Proc. R. Soc. Edinb, pages 42--48, 1934.

[3]

P. N. Bennett, F. Radlinski, R. W. White, and E. Yilmaz. Inferring and using location metadata to personalize web search. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '11, pages 135--144, New York, NY, USA, 2011. ACM.

Digital Library

[4]

P. N. Bennett, R. W. White, W. Chu, S. T. Dumais, P. Bailey, F. Borisyuk, and X. Cui. Modeling the impact of short- and long-term behavior on search personalization. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '12, pages 185--194, New York, NY, USA, 2012. ACM.

Digital Library

[5]

K. Collins-Thompson, P. N. Bennett, R. W. White, S. de la Chica, and D. Sontag. Personalizing web search results by reading level. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM '11, pages 403--412, New York, NY, USA, 2011. ACM.

Digital Library

[6]

Z. Dou, R. Song, and J.-R. Wen. A large-scale evaluation and analysis of personalized search strategies. In Proceedings of the 16th International Conference on World Wide Web, WWW '07, pages 581--590, New York, NY, USA, 2007. ACM.

Digital Library

[7]

J. H. Friedman. Stochastic gradient boosting. Comput. Stat. Data Anal., 38(4):367--378, Feb. 2002.

Digital Library

[8]

D. Jiang, K. W.-T. Leung, and W. Ng. Context-aware search personalization with concept preference. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM '11, pages 563--572, New York, NY, USA, 2011. ACM.

Digital Library

[9]

T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, pages 133--142, New York, NY, USA, 2002. ACM.

Digital Library

[10]

C. Liu, R. W. White, and S. Dumais. Understanding web browsing behaviors through weibull analysis of dwell time. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, pages 379--386, New York, NY, USA, 2010. ACM.

Digital Library

[11]

N. Matthijs and F. Radlinski. Personalizing web search using long term browsing history. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM '11, pages 25--34, New York, NY, USA, 2011. ACM.

Digital Library

[12]

F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data re ect retrieval quality? In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM '08, pages 43--52, New York, NY, USA, 2008. ACM.

Digital Library

[13]

P. Serdyukov, G. Dupret, and N. Craswell. Log-based personalization: The 4th web search click data (wscd) workshop. WSDM '14, pages 685--686, New York, NY, USA, 2014. ACM.

Digital Library

[14]

X. Shen, B. Tan, and C. Zhai. Context-sensitive information retrieval using implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '05, pages 43--50, New York, NY, USA, 2005. ACM.

Digital Library

[15]

M. Shokouhi, R. W. White, P. Bennett, and F. Radlinski. Fighting search engine amnesia: Reranking repeated results. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '13, pages 273--282, New York, NY, USA, 2013. ACM.

Digital Library

[16]

D. Sontag, K. Collins-Thompson, P. N. Bennett, R. W. White, S. Dumais, and B. Billerbeck. Probabilistic models for personalizing web search. WSDM '12, pages 433--442, New York, NY, USA, 2012. ACM.

Digital Library

[17]

B. Tan, X. Shen, and C. Zhai. Mining long-term search history to improve search accuracy. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pages 718--723, New York, NY, USA, 2006. ACM.

Digital Library

[18]

M. Taylor, J. Guiver, S. Robertson, and T. Minka. Softrank: Optimizing non-smooth rank metrics. In Proceedings of the 2008 International Conference on Web Search and Data Mining, WSDM '08, pages 77--86, New York, NY, USA, 2008. ACM.

Digital Library

[19]

Y. Ustinovskiy and P. Serdyukov. Personalization of web-search using short-term browsing context. In Proceedings of the 22Nd ACM International Conference on Conference on Information and Knowledge Management, CIKM '13, pages 1979--1988, New York, NY, USA, 2013. ACM.

Digital Library

[20]

M. N. Volkovs and R. S. Zemel. Boltzrank: Learning to maximize expected ranking gain. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 1089--1096, New York, NY, USA, 2009. ACM.

Digital Library

[21]

R. W. White, P. N. Bennett, and S. T. Dumais. Predicting short-term interests using activity-based search context. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management, CIKM '10, pages 1009--1018, New York, NY, USA, 2010. ACM.

Digital Library

[22]

R. W. White, W. Chu, A. Hassan, X. He, Y. Song, and H. Wang. Enhancing personalized search by mining and modeling task behavior. In Proceedings of the 22Nd International Conference on World Wide Web, WWW '13, pages 1411--1420, Republic and Canton of Geneva, Switzerland, 2013. International World Wide Web Conferences Steering Committee.

Digital Library

[23]

B. Xiang, D. Jiang, J. Pei, X. Sun, E. Chen, and H. Li. Context-aware ranking in web search. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, pages 451--458, New York, NY, USA, 2010. ACM.

Digital Library

[24]

J. Xu, T.-Y. Liu, M. Lu, H. Li, and W.-Y. Ma. Directly optimizing evaluation measures in learning to rank. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '08, pages 107--114, New York, NY, USA, 2008. ACM.

Digital Library

[25]

L. Xu, M. White, and D. Schuurmans. Optimal reverse prediction: A unified perspective on supervised, unsupervised and semi-supervised learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 1137--1144, New York, NY, USA, 2009. ACM.

Digital Library

[26]

Y. Yue, Y. Gao, O. Chapelle, Y. Zhang, and T. Joachims. Learning more powerful test statistics for click-based retrieval evaluation. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, pages 507--514, New York, NY, USA, 2010. ACM.

Digital Library

Cited By

Pandey APatil KSharma SKulkarni M(2021)Personalization of Information using Graph Convolutional Network2021 International Conference on Advancements in Electrical, Electronics, Communication, Computing and Automation (ICAECA)10.1109/ICAECA52838.2021.9675543(1-6)Online publication date: 8-Oct-2021
https://doi.org/10.1109/ICAECA52838.2021.9675543
Ni YOu DLiu SLi XOu WZeng ASi LGuo YFarooq F(2018)Perceive Your Users in DepthProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3219819.3219828(596-605)Online publication date: 19-Jul-2018
https://dl.acm.org/doi/10.1145/3219819.3219828
Huo CZhao YRen WBen-Othman JGang FLiu JArai M(2017)User behavior sequence modeling to optimize ranking mechanism for e-commerce searchProceedings of the 3rd International Conference on Communication and Information Processing10.1145/3162957.3163045(164-169)Online publication date: 24-Nov-2017
https://dl.acm.org/doi/10.1145/3162957.3163045
Show More Cited By

Index Terms

An Optimization Framework for Weighting Implicit Relevance Labels for Personalized Web Search
1. Information systems
  1. Information retrieval

Recommendations

Personalized click prediction in sponsored search
WSDM '10: Proceedings of the third ACM international conference on Web search and data mining

Sponsored search is a multi-billion dollar business that generates most of the revenue for search engines. Predicting the probability that users click on ads is crucial to sponsored search because the prediction is used to influence ranking, filtering, ...
Personalized ontology for web search personalization
COMPUTE '08: Proceedings of the 1st Bangalore Annual Compute Conference

The highest problem faced by the users of web search today is the quality and the amount of the results they get back. The results often frustrate and consume precious time of the users. Most of the existing search engines perform searches that are ...
Using Online Relevance Feedback to Build Effective Personalized Metasearch Engine
WISE '01: Proceedings of the Second International Conference on Web Information Systems Engineering (WISE'01) Volume 1 - Volume 1

Metasearch Engine is popular for facilitating users' queries over multiple search engines and increasing the coverage of the WWW. How to rank the merged results becomes crucial for the success of metasearch engines. Many current metasearch engines have ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

WWW '15: Proceedings of the 24th International Conference on World Wide Web

May 2015

1460 pages

ISBN:9781450334693

General Chairs:
Aldo Gangemi
National Research Council, Italy & Paris 13 University-CNRS, France
,
Stefano Leonardi
Sapienza University of Rome, Italy
,
Alessandro Panconesi
Sapienza University of Rome, Italy

Copyright © 2015 Copyright is held by the International World Wide Web Conference Committee (IW3C2).

Sponsors

IW3C2: International World Wide Web Conference Committee

In-Cooperation

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

International World Wide Web Conferences Steering Committee

Republic and Canton of Geneva, Switzerland

Publication History

Published: 18 May 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WWW '15

Sponsor:

IW3C2

WWW '15: 24th International World Wide Web Conference

May 18 - 22, 2015

Florence, Italy

Acceptance Rates

WWW '15 Paper Acceptance Rate 131 of 929 submissions, 14%;

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
234
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pandey APatil KSharma SKulkarni M(2021)Personalization of Information using Graph Convolutional Network2021 International Conference on Advancements in Electrical, Electronics, Communication, Computing and Automation (ICAECA)10.1109/ICAECA52838.2021.9675543(1-6)Online publication date: 8-Oct-2021
https://doi.org/10.1109/ICAECA52838.2021.9675543
Ni YOu DLiu SLi XOu WZeng ASi LGuo YFarooq F(2018)Perceive Your Users in DepthProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3219819.3219828(596-605)Online publication date: 19-Jul-2018
https://dl.acm.org/doi/10.1145/3219819.3219828
Huo CZhao YRen WBen-Othman JGang FLiu JArai M(2017)User behavior sequence modeling to optimize ranking mechanism for e-commerce searchProceedings of the 3rd International Conference on Communication and Information Processing10.1145/3162957.3163045(164-169)Online publication date: 24-Nov-2017
https://dl.acm.org/doi/10.1145/3162957.3163045
Cai FWang Sde Rijke M(2017)Behavior-based personalization in web searchJournal of the Association for Information Science and Technology10.1002/asi.2373568:4(855-868)Online publication date: 1-Apr-2017
https://dl.acm.org/doi/10.1002/asi.23735
SERIZAWA KKAMEI SHAYASHI SFUJITA S(2016)Personalized Web Page Recommendation Based on Preference Footprint to Browsed PagesIEICE Transactions on Information and Systems10.1587/transinf.2016EDP7134E99.D:11(2705-2715)Online publication date: 2016
https://doi.org/10.1587/transinf.2016EDP7134
Ustinovskiy YFedorova VGusev GSerdyukov PPerego RSebastiani FAslam JRuthven IZobel J(2016)An Optimization Framework for Remapping and Reweighting Noisy Relevance LabelsProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval10.1145/2911451.2911501(105-114)Online publication date: 7-Jul-2016
https://dl.acm.org/doi/10.1145/2911451.2911501
Bennett PShokouhi MCaruana RAllan JCroft Bde Vries AZhai C(2015)Implicit Preference Labels for Learning Highly Selective Personalized RankersProceedings of the 2015 International Conference on The Theory of Information Retrieval10.1145/2808194.2809464(291-300)Online publication date: 27-Sep-2015
https://dl.acm.org/doi/10.1145/2808194.2809464

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents