short-paper

On Topic Difficulty in IR Evaluation: The Effect of Systems, Corpora, and System Components

Authors:

Stefano MizzaroAuthors Info & Claims

SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 909 - 912

https://doi.org/10.1145/3331184.3331279

Published: 18 July 2019 Publication History

Get Access

Abstract

In a test collection setting, topic difficulty can be defined as the average effectiveness of a set of systems for a topic. In this paper we study the effects on the topic difficulty of: (i) the set of retrieval systems; (ii) the underlying document corpus; and (iii) the system components. By generalizing methods recently proposed to study system component factor analysis, we perform a comprehensive analysis on topic difficulty and the relative effects of systems, corpora, and component interactions. Our findings show that corpora have the most significant effect on topic difficulty.

References

[1]

David Banks, Paul Over, and Nien-Fan Zhang. 1999. Blind men and elephants: Six approaches to TREC data. Information Retrieval, Vol. 1, 1 (1999), 7--34.

Digital Library

Google Scholar

[2]

David Carmel and Elad Yom-Tov. 2010. Estimating the query difficulty for information retrieval. Synthesis Lectures on Information Concepts, Retrieval, and Services, Vol. 2, 1 (2010), 1--89.

Crossref

Google Scholar

[3]

Nicola Ferro, Yubin Kim, and Mark Sanderson. 2019. Using Collection Shards to Study Retrieval Performance Effect Sizes. ACM TOIS, Vol. 5, 44 (2019), 59.

Digital Library

Google Scholar

[4]

Nicola Ferro and Mark Sanderson. 2017. Sub-corpora impact on system effectiveness. In Proceedings of the 40th ACM SIGIR. ACM, 901--904.

Digital Library

Google Scholar

[5]

Nicola Ferro and Gianmaria Silvello. 2016. A general linear mixed models approach to study system component effects. In 39th ACM SIGIR. 25--34.

Digital Library

Google Scholar

[6]

Nicola Ferro and Gianmaria Silvello. 2018. Toward an anatomy of IR system component performances. JASIST, Vol. 69, 2 (2018), 187--200.

Google Scholar

[7]

Donna Harman and Chris Buckley. 2009. Overview of the reliable information access workshop. Information Retrieval, Vol. 12, 6 (2009), 615--641.

Digital Library

Google Scholar

[8]

Stefano Mizzaro, Josiane Mothe, Kevin Roitero, and Md Zia Ullah. 2018. Query Performance Prediction and Effectiveness Evaluation Without Relevance Judgments: Two Sides of the Same Coin. In The 41st ACM SIGIR (SIGIR '18). 1233--1236.

Digital Library

Google Scholar

[9]

Stefano Mizzaro and Stephen Robertson. 2007. Hits Hits TREC: Exploring IR Evaluation Results with Network Analysis. In Proceedings 30th SIGIR. 479--486.

Digital Library

Google Scholar

[10]

Kevin Roitero, Eddy Maddalena, and Stefano Mizzaro. {n. d.}. Do Easy Topics Predict Effectiveness Better Than Difficult Topics?. In ECIR2017. 605--611.

Google Scholar

[11]

Mark Sanderson, Andrew Turpin, Ying Zhang, and Falk Scholer. 2012. Differences in effectiveness across sub-collections. In Proc. of the 21st ACM CIKM. 1965--1969.

Digital Library

Google Scholar

Cited By

View all

Joseph MRavana S(2024)Reliable Information Retrieval Systems Performance Evaluation: A ReviewIEEE Access10.1109/ACCESS.2024.337723912(51740-51751)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3377239
Roitero KBarbera DSoprano MDemartini GMizzaro SSakai T(2023)How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance JudgmentsACM Transactions on Information Systems10.1145/359720142:1(1-26)Online publication date: 22-May-2023
https://dl.acm.org/doi/10.1145/3597201
Ceolin DRoitero KGuo F(2023)Predicting Crowd Workers Performance: An Information Quality CaseWeb Engineering10.1007/978-3-031-34444-2_6(75-90)Online publication date: 16-Jun-2023
https://doi.org/10.1007/978-3-031-34444-2_6
Show More Cited By

Index Terms

On Topic Difficulty in IR Evaluation: The Effect of Systems, Corpora, and System Components
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Test collections

Recommendations

Effort-based information retrieval evaluation with varied evaluation depth and topic sizes
ICBIM '19: Proceedings of the 3rd International Conference on Business and Information Management

The information retrieval accessed globally is a vital productivity boost for most organization. However, the outcome of information retrieval system evaluation does not agree with the real user's satisfaction. Information retrieval systems retrieving ...
Topic analysis for topic-focused multi-document summarization
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Topic-focused multi-document summarization has been a challenging task because the created summary is required to be biased to the given topic or query. Existing methods consider the given topic as a single coarse unit and then directly incorporate the ...
Topic sentiment change analysis
MLDM'11: Proceedings of the 7th international conference on Machine learning and data mining in pattern recognition

Public opinions on a topic may change over time. Topic Sentiment change analysis is a new research problem consisting of two main components: (a) mining opinions on a certain topic, and (b) detect significant changes of sentiment of the opinions on the ...

Comments

Information & Contributors

Information

Published In

SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2019

1512 pages

ISBN:9781450361729

DOI:10.1145/3331184

General Chairs:
Benjamin Piwowarski
CNRS - Sorbonne Universite, France
,
Max Chevalier
Universite de Toulouse, CNRS, France
,
Eric Gaussier
Universite Grenoble Alpes, CNRS, France
,
Program Chairs:
Yoelle Maarek
Amazon Research, Israel
,
Jian-Yun Nie
University of Montreal, Canada
,
Falk Scholer
RMIT University, Australia

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Conference

SIGIR '19

Sponsor:

SIGIR

SIGIR '19: The 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 21 - 25, 2019

Paris, France

Acceptance Rates

SIGIR'19 Paper Acceptance Rate 84 of 426 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
192
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Joseph MRavana S(2024)Reliable Information Retrieval Systems Performance Evaluation: A ReviewIEEE Access10.1109/ACCESS.2024.337723912(51740-51751)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3377239
Roitero KBarbera DSoprano MDemartini GMizzaro SSakai T(2023)How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance JudgmentsACM Transactions on Information Systems10.1145/359720142:1(1-26)Online publication date: 22-May-2023
https://dl.acm.org/doi/10.1145/3597201
Ceolin DRoitero KGuo F(2023)Predicting Crowd Workers Performance: An Information Quality CaseWeb Engineering10.1007/978-3-031-34444-2_6(75-90)Online publication date: 16-Jun-2023
https://doi.org/10.1007/978-3-031-34444-2_6
Soprano MRoitero KLa Barbera DCeolin DSpina DMizzaro SDemartini G(2022)The many dimensions of truthfulnessInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10271058:6Online publication date: 22-Apr-2022
https://dl.acm.org/doi/10.1016/j.ipm.2021.102710
Albahem ASpina DScholer FCavedon L(2021)Component-based Analysis of Dynamic Search PerformanceACM Transactions on Information Systems10.1145/348323740:3(1-47)Online publication date: 22-Nov-2021
https://dl.acm.org/doi/10.1145/3483237
Roitero KSoprano MPortelli BDe Luise MSpina DMea VSerra GMizzaro SDemartini G(2021)Can the crowd judge truthfulness? A longitudinal study on recent misinformation about COVID-19Personal and Ubiquitous Computing10.1007/s00779-021-01604-627:1(59-89)Online publication date: 16-Sep-2021
https://doi.org/10.1007/s00779-021-01604-6
Roitero KCarterette BMehrotra RLalmas M(2020)Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender SystemsCompanion Proceedings of the Web Conference 202010.1145/3366424.3384362(694-702)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366424.3384362

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Effort-based information retrieval evaluation with varied evaluation depth and topic sizes

Topic analysis for topic-focused multi-document summarization

Topic sentiment change analysis