short-paper

A Comparative Analysis of Human and Automatic Query Variants

Authors:

J. Shane CulpepperAuthors Info & Claims

ICTIR '19: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval

Pages 47 - 50

https://doi.org/10.1145/3341981.3344223

Published: 26 September 2019 Publication History

Get Access

Abstract

We present an in-depth comparative analysis of the effectiveness distributions of sets of human-created and automatically-created query variations used to represent the same information need. The automatic variations are generated using Bing's click graph. Experiments performed with TREC datasets show that using automatic variations for retrieval can result in similar effectiveness to that of using human variations, although the two types of variations can be appreciably different in several important respects --- e.g., their similarities and corresponding retrieved lists.

References

[1]

L. Azzopardi, M. de Rijke, and K. Balog. 2007. Building simulated queries for knownitem topics: an analysis using six european languages. In Proc. SIGIR. 455--462.

Google Scholar

[2]

P. Bailey, A. Moffat, F. Scholer, and P. Thomas. 2016. UQV100: A Test Collection with Query Variability. In Proc. SIGIR. 725--728.

Google Scholar

[3]

P. Bailey, A. Moffat, F. Scholer, and P. Thomas. 2017. Retrieval Consistency in the Presence of Query Variations. In Proc. SIGIR. 395--404.

Google Scholar

[4]

R. Benham and J. S. Culpepper. 2017. Risk-Reward Trade-o's in Rank Fusion. In Proc. ADCS. 1--8.

Google Scholar

[5]

R. Benham, J. S. Culpepper, L. Gallagher, X. Lu, and J. Mackenzie. 2018. Towards effcient and effective query variant generation. In Proc. DESIRES. 62--67.

Google Scholar

[6]

N. Craswell and M. Szummer. 2007. Random walks on the click graph. In Proc. SIGIR. 239--246.

Google Scholar

[7]

R. Cummins, M. Lalmas, and C. O'Riordan. 2011. The limits of retrieval effectiveness. In Proc. ECIR. 277--282.

Digital Library

Google Scholar

[8]

D. Sheldon, M. Shokouhi, M. Szummer, and N. Craswell. 2011. LambdaMerge: Merging the Results of Query Reformulations. In Proc. WSDM. 795--804.

Google Scholar

[9]

F. Song andW. B. Croft. 1999. A general language model for information retrieval (poster abstract). In Proc. of SIGIR. 279--280.

Digital Library

Google Scholar

[10]

E.Walker and A. S. Nowacki. 2011. Understanding equivalence and noninferiority testing. J. General Internal Medicine 26, 2 (2011), 192--196.

Crossref

Google Scholar

[11]

W. Webber, A. Moffat, and J. Zobel. 2010. A Similarity Measure for Indeffnite Rankings. ACM Trans. Information Systems 28, 4 (Nov. 2010), 20.1--20.38.

Digital Library

Google Scholar

Cited By

View all

Zerhoudi SGranitzer M(2024)Generative Agents Navigating Digital LibrariesSustainability and Empowerment in the Context of Digital Libraries10.1007/978-981-96-0865-2_14(171-188)Online publication date: 4-Dec-2024
https://dl.acm.org/doi/10.1007/978-981-96-0865-2_14
Wang SScells HKoopman BPotthast MZuccon G(2023)Generating Natural Language Queries for More Effective Systematic Review Screening PrioritisationProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625322(73-83)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3625322
Mackie IOwoicho PGemmell CFischer SMacAvaney SDalton JAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)CODEC: Complex Document and Entity CollectionProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531712(3067-3077)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531712
Show More Cited By

Index Terms

A Comparative Analysis of Human and Automatic Query Variants
1. Information systems
  1. Information retrieval
    1. Information retrieval query processing
    2. Retrieval models and ranking
      1. Combination, fusion and federated search
      2. Language models

Recommendations

Evaluation of phrasal query suggestions
CIKM '07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management

This paper evaluates the uptake and efficacy of a unified approach to phrasal query suggestions in the context of a high-precision search engine. The search engine performs ranked extended-Boolean searches with the proximity operator <scp>NEAR</scp> ...
Exploiting underrepresented query aspects for automatic query expansion
KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining

Users attempt to express their search goals through web search queries. When a search goal has multiple components or aspects, documents that represent all the aspects are likely to be more relevant than those that only represent some aspects. Current ...
Diversifying Query Auto-Completion

Query auto-completion assists web search users in formulating queries with a few keystrokes, helping them to avoid spelling mistakes and to produce clear query expressions, and so on. Previous work on query auto-completion mainly centers around ...

Comments

Information & Contributors

Information

Published In

ICTIR '19: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval

September 2019

273 pages

ISBN:9781450368810

DOI:10.1145/3341981

General Chairs:
Yi Fang
Santa Clara University, USA
,
Yi Zhang
University of California, Santa Cruz, USA
,
Program Chairs:
James Allan
University of Massachusetts, Amherst, USA
,
Krisztian Balog
University of Stavanger, Norway
,
Ben Carterette
Spotify, USA
,
Jiafeng Guo
Chinese Academy of Sciences, China

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Conference

ICTIR '19

Sponsor:

SIGIR

ICTIR '19: The 2019 ACM SIGIR International Conference on the Theory of Information Retrieval

October 2 - 5, 2019

CA, Santa Clara, USA

Acceptance Rates

ICTIR '19 Paper Acceptance Rate 20 of 41 submissions, 49%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
197
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zerhoudi SGranitzer M(2024)Generative Agents Navigating Digital LibrariesSustainability and Empowerment in the Context of Digital Libraries10.1007/978-981-96-0865-2_14(171-188)Online publication date: 4-Dec-2024
https://dl.acm.org/doi/10.1007/978-981-96-0865-2_14
Wang SScells HKoopman BPotthast MZuccon G(2023)Generating Natural Language Queries for More Effective Systematic Review Screening PrioritisationProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3625322(73-83)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3625322
Mackie IOwoicho PGemmell CFischer SMacAvaney SDalton JAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)CODEC: Complex Document and Entity CollectionProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531712(3067-3077)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531712
Alaofi MGallagher LMckay DSaling LSanderson MScholer FSpina DWhite RAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Where Do Queries Come From?Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531711(2850-2862)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531711
Breuer TFuhr NSchaer P(2022)Validating Simulations of User Query VariantsAdvances in Information Retrieval10.1007/978-3-030-99736-6_6(80-94)Online publication date: 5-Apr-2022
https://doi.org/10.1007/978-3-030-99736-6_6
Di Nunzio GFaggioli G(2021)A Study of a Gain Based Approach for Query Aspects in Recall Oriented TasksApplied Sciences10.3390/app1119907511:19(9075)Online publication date: 29-Sep-2021
https://doi.org/10.3390/app11199075
Mackenzie JMoffat AHasibi FFang YAizawa A(2021)Modality Effects When Simulating User Querying TasksProceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3471158.3472244(197-201)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3471158.3472244
Otero DParapar JBarreiro ÁHung CHong JBechini ASong E(2021)The wisdom of the rankersProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3441947(672-680)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3441947
Benham RMackenzie JCulpepper JMoffat AScholer FThomas PElsweiler DJoho HKando NSmith C(2021)Different Keystrokes for Different FolksProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446054(331-335)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446054
Liu BLu XCulpepper J(2021)Strong natural language query generationInformation Retrieval Journal10.1007/s10791-021-09395-3Online publication date: 15-Jul-2021
https://doi.org/10.1007/s10791-021-09395-3
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Evaluation of phrasal query suggestions

Exploiting underrepresented query aspects for automatic query expansion

Diversifying Query Auto-Completion