poster

Vertical-Aware Click Model-Based Effectiveness Metrics

Authors:

Fabio CrestaniAuthors Info & Claims

CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

Pages 1867 - 1870

https://doi.org/10.1145/2661829.2661944

Published: 03 November 2014 Publication History

Get Access

Abstract

Today's web search systems present users with heterogeneous information coming from sources of different types, also known as verticals. Evaluating such systems is an important but complex task, which is still far from being solved. In this paper we examine the hypothesis that the use of models that capture user search behavior on heterogeneous result pages helps to improve the quality of offline metrics. We propose two vertical-aware metrics based on user click models for federated search and evaluate them using query logs of the Yandex search engine. We show that depending on the type of vertical, the proposed metrics have higher correlation with online user behavior than other state-of-the-art techniques.

References

[1]

O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In WWW '09, pages 1--10, 2009.

Digital Library

Google Scholar

[2]

O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In CIKM '09, pages 621--630, 2009.

Digital Library

Google Scholar

[3]

D. Chen, W. Chen, H. Wang, Z. Chen, and Q. Yang. Beyond ten blue links: enabling user click modeling in federated web search. In WSDM '12, pages 463--472, 2012.

Digital Library

Google Scholar

[4]

A. Chuklin, P. Serdyukov, and M. de Rijke. Click model-based information retrieval metrics. In SIGIR '13, pages 493--502, 2013.

Digital Library

Google Scholar

[5]

G. E. Dupret and B. Piwowarski. A user browsing model to predict search engine click data from past observations. In SIGIR '08, pages 331--338, 2008.

Digital Library

Google Scholar

[6]

F. Guo, C. Liu, and Y. M. Wang. Efficient multiple-click models in web search. In WSDM '09, pages 124--131, 2009.

Digital Library

Google Scholar

[7]

K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Information Systems, 20 (4): 422--446, 2002.

Digital Library

Google Scholar

[8]

A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Information Systems, 27 (1): 2:1--2:27, 2008.

Digital Library

Google Scholar

[9]

Y. Song, X. Shi, R. W. White, and A. Hassan. Context-aware web search abandonment prediction. In SIGIR '14, 2014.

Digital Library

Google Scholar

[10]

C. Wang, Y. Liu, M. Zhang, S. Ma, M. Zheng, J. Qian, and K. Zhang. Incorporating vertical results into search click models. In SIGIR '13, pages 503--512, 2013.

Digital Library

Google Scholar

[11]

E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. Expected browsing utility for web search evaluation. In CIKM '10, pages 1561--1564, 2010.

Digital Library

Google Scholar

[12]

K. Zhou, T. Sakai, M. Lalmas, Z. Dou, and J. M. Jose. Evaluating heterogeneous information access. In Proc. MUBE workshop, 2013.

Google Scholar

Cited By

View all

Choi WKim S(2022)A Novel Probabilistic Graphical Model-Based Click Model for Vertical SearchJournal of the Korean Institute of Industrial Engineers10.7232/JKIIE.2022.48.2.13848:2(138-150)Online publication date: 15-Apr-2022
https://doi.org/10.7232/JKIIE.2022.48.2.138
Arguello JChoi B(2019)The Effects of Working Memory, Perceptual Speed, and Inhibition in Aggregated SearchACM Transactions on Information Systems10.1145/332212837:3(1-34)Online publication date: 16-May-2019
https://dl.acm.org/doi/10.1145/3322128
Takanobu RZhuang THuang MFeng JTang HZheng B(2019)Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement LearningThe World Wide Web Conference10.1145/3308558.3313455(1771-1781)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313455
Show More Cited By

Index Terms

Vertical-Aware Click Model-Based Effectiveness Metrics
1. Information systems
  1. Information retrieval

Recommendations

Evaluating intuitiveness of vertical-aware click models
SIGIR '14: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval

Modeling user behavior on a search engine result page is important for understanding the users and supporting simulation experiments. As result pages become more complex, click models evolve as well in order to capture additional aspects of user ...
Time-Aware Click Model

Click-through information is considered as a valuable source of users’ implicit relevance feedback for commercial search engines. As existing studies have shown that the search result position in a search engine result page (SERP) has a very strong ...
Interest and Evaluation of Aggregated Search
WI-IAT '11: Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01

Major search engines perform what is known as Aggregated Search (AS). They integrate results coming from different vertical search engines (images, videos, news, etc.) with typical Web search results. Aggregated search is relatively new and its ...

Comments

Information & Contributors

Information

Published In

CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

November 2014

2152 pages

ISBN:9781450325981

DOI:10.1145/2661829

General Chairs:
Jianzhong Li
Harbin Inst. of Technology
,
X. Sean Wang
Fudan University
,
Program Chairs:
Minos Garofalakis
Technical University of Crete, Greece
,
Ian Soboroff
National Institute of Standards, USA
,
Torsten Suel
New York University, USA
,
Min Wang
Google Research, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Funding Sources

Seventh Framework Programme
Swiss National Science Foundation
Nederlandse Organisatie voor Wetenschappelijk Onderzoek
Center for Creation Content and Technology (CCCT)

Conference

CIKM '14

Sponsor:

CIKM '14: 2014 ACM Conference on Information and Knowledge Management

November 3 - 7, 2014

Shanghai, China

Acceptance Rates

CIKM '14 Paper Acceptance Rate 175 of 838 submissions, 21%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
122
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Choi WKim S(2022)A Novel Probabilistic Graphical Model-Based Click Model for Vertical SearchJournal of the Korean Institute of Industrial Engineers10.7232/JKIIE.2022.48.2.13848:2(138-150)Online publication date: 15-Apr-2022
https://doi.org/10.7232/JKIIE.2022.48.2.138
Arguello JChoi B(2019)The Effects of Working Memory, Perceptual Speed, and Inhibition in Aggregated SearchACM Transactions on Information Systems10.1145/332212837:3(1-34)Online publication date: 16-May-2019
https://dl.acm.org/doi/10.1145/3322128
Takanobu RZhuang THuang MFeng JTang HZheng B(2019)Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement LearningThe World Wide Web Conference10.1145/3308558.3313455(1771-1781)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313455
Thomas PMoffat ABailey PScholer FCraswell N(2018)Better Effectiveness Metrics for SERPs, Cards, and RankingsProceedings of the 23rd Australasian Document Computing Symposium10.1145/3291992.3292002(1-8)Online publication date: 11-Dec-2018
https://dl.acm.org/doi/10.1145/3291992.3292002
Arguello J(2017)Aggregated SearchFoundations and Trends in Information Retrieval10.1561/150000005210:5(365-502)Online publication date: 6-Mar-2017
https://dl.acm.org/doi/10.1561/1500000052
Markov IBorisov Ade Rijke MLim EWinslett MSanderson MFu ASun JCulpepper SLo EHo JDonato DAgrawal RZheng YCastillo CSun ATseng VLi C(2017)Online Expectation-Maximization for Click ModelsProceedings of the 2017 ACM on Conference on Information and Knowledge Management10.1145/3132847.3133053(2195-2198)Online publication date: 6-Nov-2017
https://dl.acm.org/doi/10.1145/3132847.3133053
Chen YLiu YZhang MMa S(2017)User Satisfaction Prediction with Mouse Movement Information in Heterogeneous Search EnvironmentIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2017.273915129:11(2470-2483)Online publication date: 1-Nov-2017
https://dl.acm.org/doi/10.1109/TKDE.2017.2739151
Chen YLiu YZhou KWang MZhang MMa SBailey JMoffat AAggarwal Cde Rijke MKumar RMurdock VSellis TYu J(2015)Does Vertical Bring more Satisfaction?Proceedings of the 24th ACM International on Conference on Information and Knowledge Management10.1145/2806416.2806473(1581-1590)Online publication date: 17-Oct-2015
https://dl.acm.org/doi/10.1145/2806416.2806473

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Evaluating intuitiveness of vertical-aware click models

Time-Aware Click Model

Interest and Evaluation of Aggregated Search