Article

A large-scale evaluation and analysis of personalized search strategies

Authors:

Ji-Rong WenAuthors Info & Claims

WWW '07: Proceedings of the 16th international conference on World Wide Web

Pages 581 - 590

https://doi.org/10.1145/1242572.1242651

Published: 08 May 2007 Publication History

Abstract

Although personalized search has been proposed for many years and many personalization strategies have been investigated, it is still unclear whether personalization is consistently effective on different queries for different users, and under different search contexts. In this paper, we study this problem and get some preliminary conclusions. We present a large-scale evaluation framework for personalized search based on query logs, and then evaluate five personalized search strategies (including two click-based and three profile-based ones) using 12-day MSN query logs. By analyzing the results, we reveal that personalized search has significant improvement over common web search on some queries but it also has little effect on other queries (e.g., queries with small click entropy). It even harms search accuracy under some situations. Furthermore, we show that straightforward click-based personalization strategies perform consistently and considerably well, while profile-based ones are unstable in our experiments. We also reveal that both long-term and short-term contexts are very important in improving search performance for profile-based personalized search strategies.

References

[1]

S. M. Beitzel, E. C. Jensen, A. Chowdhury, D. Grossman, and O. Frieder. Hourly analysis of a very large topically categorized web query log. In Proceedings of SIGIR '04, pages 321--328, 2004.

Digital Library

[2]

J. Boyan, D. Freitag, and T. Joachims. Evaluating retrieval performance using clickthrough data. In Proceedings of AAAI Workshop on Internet Based Information Systems, 1996.

[3]

A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3--10, 2002.

Digital Library

[4]

J. M. Carroll and M. B. Rosson. Paradox of the active user. Interfacing thought: cognitive aspects of human-computer interaction, pages 80--111, 1987.

Digital Library

[5]

P. A. Chirita, C. Firan, and W. Nejdl. Summarizing local context to personalize global web search. In Proceedings of CIKM '06, 2006.

Digital Library

[6]

P. A. Chirita, W. Nejdl, R. Paiu, and C. Kohlschutter. Using odp metadata to personalize search. In Proceedings of SIGIR '05, pages 178--185, 2005.

Digital Library

[7]

S. Cronen-Townsend and W. B. Croft. Quantifying query ambiguity. In Proceedings of HLT '02, pages 94--98, 2002.

Digital Library

[8]

C. Dwork, R. Kumar, M. Naor, and D. Sivakumar. Rank aggregation methods for the web. In Proceedings of WWW '01, pages 613--622, 2001.

Digital Library

[9]

P. Ferragina and A. Gulli. A personalized search engine based on web-snippet hierarchical clustering. In WWW '05: Special interest tracks and posters of the 14th international conference on World Wide Web, pages 801--810, 2005.

Digital Library

[10]

T. H. Haveliwala. Topic-sensitive pagerank. In Proceedings of WWW '02, 2002.

Digital Library

[11]

B. J. Jansen, A. Spink, J. Bateman, and T. Saracevic. Real life information retrieval: a study of user queries on the web. SIGIR Forum, 32(1):5--17, 1998.

Digital Library

[12]

B. J. Jansen, A. Spink, and T. Saracevic. Real life, real users, and real needs: a study and analysis of user queries on the web. Information Processing and Management, 36(2):207--227, 2000.

Digital Library

[13]

J. C. Borda. Mémoire sur les élections au scrution. Histoire de l'Académie Royal des Sciences, 1781.

[14]

G. Jeh and J. Widom. Scaling personalized web search. In Proceedings of WWW' 03, pages 271--279,2003.

Digital Library

[15]

D. H. John S. Breese and C. Kadie. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of UAI '98, pages 43--52, 1998.

Digital Library

[16]

R. Krovetz and W. B. Croft. Lexical ambiguity and information retrieval. Information Systems, 10(2):115--141, 1992.

Digital Library

[17]

U. Lee, Z. Liu, and J. Cho. Automatic identification of user goals in web search. In Proceedings of WWW '05, pages 391--400, 2005.

Digital Library

[18]

Y. Li, Z. Zheng, and H. K. Dai. Kdd cup-2005 report: facing a great challenge. SIGKDD Explor. Newsl.,7(2):91--99, 2005.

Digital Library

[19]

F. Liu, C. Yu, and W. Meng. Personalized web search by mapping user queries to categories. In Proceedings of CIKM '02, pages 558--565, 2002.

Digital Library

[20]

L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Computer Science Department, Stanford University, 1998.

[21]

J. Pitkow, H. Schutze, T. Cass, R. Cooley, D. Turnbull, A. Edmonds, E. Adar, and T. Breuel. Personalized search. Commun. ACM, 45(9):50--55, 2002.

Digital Library

[22]

A. Pretschner and S. Gauch. Ontology based personalized search. In Proceedings of ICTAI '99, pages 391--398, 1999.

Digital Library

[23]

F. Qiu and J. Cho. Automatic identification of user interest for personalized search. In Proceedings of WWW '06, pages 727--736, 2006.

Digital Library

[24]

D. Shen, R. Pan, J. -T. Sun, J. J. Pan, K. Wu, J. Yin, and Q. Yang. Q2cυust: our winning solution to query classification in kddcup 2005. SIGKDD Explor. Newsl., 7(2):100--110, 2005.

Digital Library

[25]

X. Shen, B. Tan, and C. Zhai. Context-sensitive information retrieval using implicit feedback. In Proceedings of SIGIR '05, pages 43--50, 2005.

Digital Library

[26]

X. Shen, B. Tan, and C. Zhai. Implicit user modeling for personalized search. In Proceedings of CIKM '05, pages 824--831, 2005.

Digital Library

[27]

C. Silverstein, H. Marais, M. Henzinger, and M. Moricz. Analysis of a very large web search engine query log. SIGIR Forum, 33(1):6--12, 1999.

Digital Library

[28]

M. Speretta and S. Gauch. Personalized search based on user search histories. In Proceedings of WI '05, pages 622--628, 2005.

Digital Library

[29]

K. Sugiyama, K. Hatano, and M. Yoshikawa. Adaptive web search based on user profile constructed without any effort from users. In Proceedings of WWW '04, pages 675--684, 2004.

Digital Library

[30]

J. T. Sun, H. J. Zeng, H. Liu, Y. Lu, and Z. Chen. Cubesvd: a novel approach to personalized web search. In Proceedings of WWW '05, pages 382--390, 2005.

Digital Library

[31]

B. Tan, X. Shen, and C. Zhai. Mining long-term search history to improve search accuracy. In Proceedings of KDD '06, pages 718--723, 2006.

Digital Library

[32]

F. Tanudjaja and L. Mui. Persona: A contextualized and personalized web search. In Proceedings of HICSS'02, pages volume3, pp. 53, 2002.

Digital Library

[33]

J. Teevan, E. Adar, R. Jones, and M. Potts. History repeats itself: repeat queries in yahoo's logs. In Proceedings of SIGIR '06, pages 703--704, 2006.

Digital Library

[34]

J. Teevan, S. T. Dumais, and E. Horvitz. Beyond the commons: Investigating the value of personalizing web search. In Proceedings of PIA '05, 2005.

[35]

J. Teevan, S. T. Dumais, and E. Horvitz. Personalizing search via automated analysis of interests and activities. In Proceedings of SIGIR '05, pages 449--456, 2005.

Digital Library

[36]

S. Wedig and O. Madani. A large-scale analysis of query logs for assessing personalization opportunities. In Proceedings of KDD '06, pages 742--747, 2006.

Digital Library

[37]

Y. Xie and D. R. O'Hallaron. Locality in search engine queries and its implications for caching. In INFOCOM '02, 2002.

Cited By

Deng ZDou ZSu ZWen J(2024)Multi-grained Document Modeling for Search Result DiversificationACM Transactions on Information Systems10.1145/365285242:5(1-22)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3652852
Penha GVardasbi APalumbo EDe Nadai MBouchard H(2024)Bridging Search and Recommendation in Generative Retrieval: Does One Task Help the Other?Proceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688123(340-349)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688123
Bai YZhou YDou ZWen J(2024)Intent-Oriented Dynamic Interest Modeling for Personalized Web SearchACM Transactions on Information Systems10.1145/363981742:4(1-30)Online publication date: 8-Jan-2024
https://dl.acm.org/doi/10.1145/3639817
Show More Cited By

Index Terms

A large-scale evaluation and analysis of personalized search strategies
1. Information systems
  1. Information retrieval

Recommendations

Improving search personalisation with dynamic group formation
SIGIR '14: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval

Recent research has shown that the performance of search engines can be improved by enriching a user's personal profile with information about other users with shared interests. In the existing approaches, groups of similar users are often statically ...
Improving Ranking Consistency for Web Search by Leveraging a Knowledge Base and Search Logs
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

In this paper, we propose a new idea called ranking consistency in web search. Relevance ranking is one of the biggest problems in creating an effective web search system. Given some queries with similar search intents, conventional approaches typically ...
Personalized interactive faceted search
WWW '08: Proceedings of the 17th international conference on World Wide Web

Faceted search is becoming a popular method to allow users to interactively search and navigate complex information spaces. A faceted search system presents users with key-value metadata that is used for query refinement. While popular in e-commerce and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '07: Proceedings of the 16th international conference on World Wide Web

May 2007

1382 pages

ISBN:9781595936547

DOI:10.1145/1242572

General Chairs:
Carey Williamson
University of Calgary, Canada
,
Mary Ellen Zurko
IBM, USA
,
Program Chairs:
Peter Patel-Schneider
Bell Labs Research, USA
,
Prashant Shenoy
University of Massachusetts at Amherst, USA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

WWW'07

Sponsor:

ACM

WWW'07: 16th International World Wide Web Conference

May 8 - 12, 2007

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

313
Total Citations
View Citations
2,636
Total Downloads

Downloads (Last 12 months)86
Downloads (Last 6 weeks)11

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Deng ZDou ZSu ZWen J(2024)Multi-grained Document Modeling for Search Result DiversificationACM Transactions on Information Systems10.1145/365285242:5(1-22)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3652852
Penha GVardasbi APalumbo EDe Nadai MBouchard H(2024)Bridging Search and Recommendation in Generative Retrieval: Does One Task Help the Other?Proceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688123(340-349)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688123
Bai YZhou YDou ZWen J(2024)Intent-Oriented Dynamic Interest Modeling for Personalized Web SearchACM Transactions on Information Systems10.1145/363981742:4(1-30)Online publication date: 8-Jan-2024
https://dl.acm.org/doi/10.1145/3639817
Chatur PJain NTrivedi VDhoot AMuthiah S(2024)Google Search in India: Unveiling the Geo-Personalized WebProceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD)10.1145/3632410.3632420(403-411)Online publication date: 4-Jan-2024
https://dl.acm.org/doi/10.1145/3632410.3632420
Mo FZhao LHuang KDong YHuang DNie JSerra ESpezzano F(2024)How to Leverage Personal Textual Knowledge for Personalized Conversational Information RetrievalProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679939(3954-3958)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679939
Deng ZDou ZZhu YQin XCheng PWu JWang HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)JDivPS: A Diversified Product Search DatasetProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657888(1152-1161)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657888
Zhou YZhu QJin JDou ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Cognitive Personalized Search Integrating Large Language Models with an Efficient Memory MechanismProceedings of the ACM Web Conference 202410.1145/3589334.3645482(1464-1473)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645482
Meng YLi RQin HWu XDuan HLu YWang G(2024)Encoding Group Interests With Persistent Homology for Personalized SearchIEEE Transactions on Systems, Man, and Cybernetics: Systems10.1109/TSMC.2024.341002954:9(5606-5616)Online publication date: Sep-2024
https://doi.org/10.1109/TSMC.2024.3410029
Liu JDou ZNie JWen J(2024)Integrated Personalized and Diversified Search Based on Search LogsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.329100636:2(694-707)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TKDE.2023.3291006
Skjæveland MBalog KBernard NŁajewska WLinjordet T(2024)An ecosystem for personal knowledge graphs: A survey and research roadmapAI Open10.1016/j.aiopen.2024.01.0035(55-69)Online publication date: 2024
https://doi.org/10.1016/j.aiopen.2024.01.003
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents