short-paper

On the Quality of the TREC-COVID IR Test Collections

Authors:

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 2422 - 2428

https://doi.org/10.1145/3404835.3463244

Published: 11 July 2021 Publication History

Get Access

Abstract

Shared text collections continue to be vital infrastructure for IR research. The COVID-19 pandemic offered an opportunity to create a test collection that captured the rapidly changing information space during a pandemic, and the TREC-COVID effort was created to build such a collection using the TREC framework. This paper examines the quality of the resulting TREC-COVID test collections, and in doing so, offers a critique of the state-of-the-art in building reusable IR test collections. The largest of the collections--called 'TREC-COVID Complete'--is found to be on par with previous TREC ad~hoc collections with existing quality tests uncovering no apparent problems. Yet the lack of any way to definitively demonstrate the collection's quality and its violation of previously used quality heuristics suggest much work remains to be done to understand the factors affecting collection quality.

Supplementary Material

MP4 File (SIGIR21-fp1400.mp4)

Presentation for the paper "On the Quality of the TREC-COVID IR Test Collections".

Download
33.23 MB

References

[1]

Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen Voorhees. 2007. Bias and the Limits of Pooling for Large Collections. Information Retrieval 10 (2007), 491--508.

Digital Library

Google Scholar

[2]

Chris Buckley and Ellen M. Voorhees. 2000. Evaluating Evaluation Measure Stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, N. Belkin, P. Ingwersen, and M.K. Leong (Eds.). 33--40.

Google Scholar

[3]

Gordon V. Cormack and Maura R. Grossman. 2016. Engineering Quality and Reliability in Technology-Assisted Review. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR '16). ACM, 75--84.

Google Scholar

[4]

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 Deep Learning Track. In Proceedings of Twenty-Eighth Text REtrieval Conference (TREC 2019).

Google Scholar

[5]

Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen Voorhees, Lucy Lu Wang, and William R Hersh. 2020. TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19. Journal of the American Medical Informatics Association 27, 9 (2020), 1431--1436.https://doi.org/10.1093/jamia/ocaa091

Crossref

Google Scholar

[6]

Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen Voorhees, Lucy Lu Wang, and William R Hersh. 2021. Searching for Scientific Evidence in a Pandemic: An Overview of TREC-COVID. arXiv2104.09632 (2021).

Google Scholar

[7]

Tetsuya Sakai. 2006. Evaluating Evaluation Metrics Based on the Bootstrap. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '06). 525--532. https://doi.org/10.1145/1148170.1148261

Digital Library

Google Scholar

[8]

Gerard Salton and Chris Buckley. 1997. Improving retrieval performance by relevance feedback. Journal of the Association for Information Science and Technology 41 (1997), 355--364.

Google Scholar

[9]

Mark Sanderson and Ian Soboroff. 2007. Problems with Kendall's Tau. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 839--840.

Digital Library

Google Scholar

[10]

Alan Stuart. 1983. Kendall's Tau. In Encyclopedia of Statistical Sciences, Samuel Kotz and Norman L. Johnson (Eds.). Vol. 4. John Wiley & Sons, 367--369.

Google Scholar

[11]

Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2020. TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection.ACM SIGIR Forum 54, 1 (June 2020).

Digital Library

Google Scholar

[12]

Ellen M. Voorhees. 2018. On Building Fair and Reusable Test Collections Using Bandit Techniques. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18). 407--416. https://doi.org/10.1145/3269206.3271766

Digital Library

Google Scholar

[13]

Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William. Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Bran-don Stilson, Alex D. Wade, Kuansan Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020. CORD-19: The Covid-19 Open Research Dataset. ArXivabs/2004.10706(2020).

Google Scholar

[14]

Justin Zobel. 1998. How Reliable are the Results of Large-Scale Information Retrieval Experiments?. In Proceedings of the 21st Annual International ACMSIGIR Conference on Research and Development in Information Retrieval, W. Bruce Croft, Alistair Moffat, C.J. van Rijsbergen, Ross Wilkinson, and Justin Zobel (Eds.). ACM Press, New York, Melbourne, Australia, 307--314.

Google Scholar

Cited By

View all

Roberts K(2024)Information RetrievalNatural Language Processing in Biomedicine10.1007/978-3-031-55865-8_8(195-230)Online publication date: 9-Jun-2024
https://doi.org/10.1007/978-3-031-55865-8_8
Althammer SHofstätter SVerberne SHanbury AAl Hasan MXiong L(2022)TripJudgeProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557714(3801-3805)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557714
Roberts KAlam TBedrick SDemner-Fushman DLo KSoboroff IVoorhees EWang LHersh W(2021)Searching for scientific evidence in a pandemicJournal of Biomedical Informatics10.1016/j.jbi.2021.103865121:COnline publication date: 1-Sep-2021
https://dl.acm.org/doi/10.1016/j.jbi.2021.103865

Index Terms

On the Quality of the TREC-COVID IR Test Collections
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Test collections

Recommendations

Test theory for assessing IR test collections
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

How good is an IR test collection? A series of papers in recent years has addressed the question by empirically enumerating the consistency of performance comparisons using alternate subsets of the collection. In this paper we propose using Test Theory, ...
Reusable test collections through experimental design
SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

Portable, reusable test collections are a vital part of research and development in information retrieval. Reusability is difficult to assess, however. The standard approach--simulating judgment collection when groups of systems are held out, then ...
Understanding and Predicting Characteristics of Test Collections in Information Retrieval
Information for a Better World: Shaping the Global Future
Abstract
Research community evaluations in information retrieval, such as NIST’s Text REtrieval Conference (TREC), build reusable test collections by pooling document rankings submitted by many teams. Naturally, the quality of the resulting test collection ...

Comments

Information & Contributors

Information

Published In

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2021

2998 pages

ISBN:9781450380379

DOI:10.1145/3404835

General Chairs:
Fernando Diaz
(Google)
,
Chirag Shah
University of Washington
,
Torsten Suel
New York University
,
Program Chairs:
Pablo Castells
Universidad Autónoma de Madrid, Amazon
,
Rosie Jones
Spotify
,
Tetsuya Sakai
Waseda University

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '21

Sponsor:

SIGIR

SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2021

Virtual Event, Canada

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
168
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Roberts K(2024)Information RetrievalNatural Language Processing in Biomedicine10.1007/978-3-031-55865-8_8(195-230)Online publication date: 9-Jun-2024
https://doi.org/10.1007/978-3-031-55865-8_8
Althammer SHofstätter SVerberne SHanbury AAl Hasan MXiong L(2022)TripJudgeProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557714(3801-3805)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557714
Roberts KAlam TBedrick SDemner-Fushman DLo KSoboroff IVoorhees EWang LHersh W(2021)Searching for scientific evidence in a pandemicJournal of Biomedical Informatics10.1016/j.jbi.2021.103865121:COnline publication date: 1-Sep-2021
https://dl.acm.org/doi/10.1016/j.jbi.2021.103865

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Test theory for assessing IR test collections

Reusable test collections through experimental design

Understanding and Predicting Characteristics of Test Collections in Information Retrieval