Browsing and Searching Metadata of TREC
Pages 313 - 323
Abstract
Information Retrieval (IR) research is deeply rooted in experimentation and evaluation, and the Text REtrieval Conference (TREC) has been playing a central role in making that possible since its inauguration in 1992. TREC's mission centers around providing the infrastructure and resources to make IR evaluations possible at scale. Over the years, a plethora of different retrieval problems were addressed, culminating in data artifacts that remained as valuable and useful tools for the IR community. Even though the data are largely available from TREC's website, there is currently no resource that facilitates a cohesive way to obtain metadata information about the run file - the IR community's de-facto standard data format for storing rankings of system-oriented IR experiments.
To this end, the work at hand introduces a software suite that facilitates access to metadata of experimental resources, resulting from over 30 years of IR experiments and evaluations at TREC. With a particular focus on the run files, the paper motivates the requirements for better access to TREC metadata and details the concepts, the resources, the corresponding implementations, and possible use cases. More specifically, we contribute a web interface to browse former TREC submissions. Besides, we provide the underlying metadatabase and a corresponding RESTful interface for more principled and structured queries about the TREC metadata.
References
[1]
Maristella Agosti, Giorgio Maria Di Nunzio, and Nicola Ferro. 2006. Scientific data of an evaluation campaign: do we properly deal with them? In Evaluation of Multilingual and Multi-modal Information Retrieval, 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain, September 20-22, 2006, Revised Selected Papers (Lecture Notes in Computer Science). Carol Peters, Paul D. Clough, Fredric C. Gey, Jussi Karlgren, Bernardo Magnini, Douglas W. Oard, Maarten de Rijke, and Maximilian Stempfhuber, (Eds.) Vol. 4730. Springer, 11--20.
[2]
Maristella Agosti, Giorgio Maria Di Nunzio, and Nicola Ferro. 2007. The importance of scientific data curation for evaluation campaigns. In Digital Libraries: Research and Development, First International DELOS Conference, Pisa, Italy, February 13-14, 2007, Revised Selected Papers (Lecture Notes in Computer Science). Costantino Thanos, Francesca Borri, and Leonardo Candela, (Eds.) Vol. 4877. Springer, 157--166.
[3]
Enrique Amigó, Jorge Carrillo de Albornoz, Mario Almagro-Cádiz, Julio Gonzalo, Javier Rodríguez-Vidal, and Felisa Verdejo. 2017. Evall: open access evaluation for information access systems. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017. Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, (Eds.) ACM, 1301--1304.
[4]
Marco Angelini, Vanessa Fazzini, Nicola Ferro, Giuseppe Santucci, and Gianmaria Silvello. 2018. CLAIRE: A combinatorial visual analytics system for information retrieval evaluation. Inf. Process. Manag., 54, 6, 1077--1100.
[5]
Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Evaluatir: an online tool for evaluating and comparing IR systems. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009. James Allan, Javed A. Aslam, Mark Sanderson, Cheng Xiang Zhai, and Justin Zobel, (Eds.) ACM, 833.
[6]
Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements that don't add up: ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, China, November 2-6, 2009. David Wai-Lok Cheung, Il-Yeol Song,WesleyW. Chu, Xiaohua Hu, and Jimmy Lin, (Eds.) ACM, 601--610.
[7]
Matthew Arnold et al. 2019. Factsheets: increasing trust in AI services through supplier's declarations of conformity. IBM J. Res. Dev., 63, 4/5, 6:1--6:13.
[8]
Elias Bassani. 2022. Ranx: A blazing-fast python library for ranking evaluation and comparison. In Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part II (Lecture Notes in Computer Science). Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty, (Eds.) Vol. 13186. Springer, 259--264.
[9]
Elias Bassani. 2023. Ranxhub: an online repository for information retrieval runs. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023. Hsin-Hsi Chen, Wei-Jou (Edward) Duh, Hen-Hsen Huang, Makoto P. Kato, Josiane Mothe, and Barbara Poblete, (Eds.) ACM, 3210--3214.
[10]
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: toward mitigating system bias and enabling better science. Trans. Assoc. Comput. Linguistics, 6, 587--604.
[11]
Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2018. Elastic chatnoir: search engine for the clueweb and the common crawl. In Advances in Information Retrieval - 40th European Conference on IR Research, ECIR 2018, Grenoble, France, March 26-29, 2018, Proceedings (Lecture Notes in Computer Science). Gabriella Pasi, Benjamin Piwowarski, Leif Azzopardi, and Allan Hanbury, (Eds.) Vol. 10772. Springer, 820--824. 6941-7_83.
[12]
Carl Boettiger. 2015. An introduction to docker for reproducible research. ACM SIGOPS Oper. Syst. Rev., 49, 1, 71--79.
[13]
Timo Breuer, Nicola Ferro, Maria Maistro, and Philipp Schaer. 2021. repro_eval: A python interface to reproducibility measures of system-oriented IR experiments. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II (Lecture Notes in Computer Science). Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani, (Eds.) Vol. 12657. Springer, 481--486.
[14]
Timo Breuer, Jüri Keller, and Philipp Schaer. 2022. ir_metadata: an extensible metadata schema for IR experiments. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022. Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai, (Eds.) ACM, 3078--3089.
[15]
Cyril W. Cleverdon. 1991. The significance of the cranfield tests on index languages. In Proceedings of the 14th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Chicago, Illinois, USA, October 13-16, 1991 (Special Issue of the SIGIR Forum). Abraham Bookstein, Yves Chiaramella, Gerard Salton, and Vijay V. Raghavan, (Eds.) ACM, 3--12.
[16]
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2020.Overview of the TREC 2020 deep learning track. In Proceedings of the Twenty-Ninth Text REtrieval Conference, TREC 2020, Virtual Event [Gaithersburg, Maryland, USA], November 16-20, 2020 (NIST Special Publication). Ellen M. Voorhees and Angela Ellis, (Eds.) Vol. 1266. National Institute of Standards and Technology (NIST). https://trec.nist.gov/pubs/trec29/papers/OVERVIEW.DL.pdf.
[17]
Michael Dittenbach, Bernhard Pflugfelder, Andreas Pesenhofer, Giovanna Roda, and Helmut Berger. 2009. SOIRE: a service-oriented IR evaluation architecture. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, China, November 2-6, 2009. David Wai-Lok Cheung, Il-Yeol Song,WesleyW. Chu, Xiaohua Hu, and Jimmy Lin, (Eds.) ACM, 2101--2102.
[18]
David Donoho. 2017. 50 years of data science. Journal of Computational and Graphical Statistics, 26, 4, 745--766.
[19]
Nicola Ferro, Norbert Fuhr, Kalervo Järvelin, Noriko Kando, Matthias Lippold, and Justin Zobel. 2016. Increasing reproducibility in IR: findings from the dagstuhl seminar on "reproducibility of data-oriented experiments in e-science". SIGIR Forum, 50, 1, 68--82.
[20]
Nicola Ferro and Carol Peters, (Eds.) 2019. Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF. The Information Retrieval Series. Vol. 41. Springer. isbn: 978-3-030-22947-4.
[21]
Juliana Freire, Norbert Fuhr, and Andreas Rauber. 2016. Reproducibility of data-oriented experiments in e-science (dagstuhl seminar 16041). Dagstuhl Reports, 6, 1, 108--159.
[22]
Maik Fröbe, Jan Heinrich Reimer, Sean MacAvaney, Niklas Deckers, Simon Reich, Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2023. The information retrieval experiment platform. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023. Hsin-Hsi Chen, Wei-Jou (Edward) Duh, Hen-Hsen Huang, Makoto P. Kato, Josiane Mothe, and Barbara Poblete, (Eds.) ACM, 2826--2836.
[23]
Maik Fröbe, Matti Wiegmann, Nikolay Kolyada, Bastian Grahm, Theresa Elstner, Frank Loebe, Matthias Hagen, Benno Stein, and Martin Potthast. 2023. Continuous integration for reproducible shared tasks with tira.io. In Advances in Information Retrieval - 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part III (Lecture Notes in Computer Science). Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, and Annalina Caputo, (Eds.) Vol. 13982. Springer, 236--241. 241-6_20.
[24]
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM, 64, 12, 86--92.
[25]
Daniel Gildea, Min-Yen Kan, Nitin Madnani, Christoph Teichmann, and Martín Villalba. 2018. The ACL Anthology: current state and future directions. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS). Eunjeong L. Park, Masato Hagiwara, Dmitrijs Milajevs, and Liling Tan, (Eds.) Association for Computational Linguistics, Melbourne, Australia, (July 2018), 23--28.
[26]
Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_eval: an extremely fast python interface to trec_eval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018. Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz, (Eds.) ACM, 873--876. 9978.3210065.
[27]
Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv, abs/1805.03677. http://arxiv.org/abs/1805.03677.
[28]
Frank Hopfgartner et al. 2018. Evaluation-as-a-service for the computational sciences: overview and outlook. ACM J. Data Inf. Qual., 10, 4, 15:1--15:32.
[29]
Sayash Kapoor and Arvind Narayanan. 2023. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4, 9, 100804.
[30]
Dawn J. Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, and Eugene Yang. 2022. Overview of the TREC 2022 neuclir track. In Proceedings of the Thirty-First Text REtrieval Conference, TREC 2022, online, November 15-19, 2022 (NIST Special Publication). Ian Soboroff and Angela Ellis, (Eds.) Vol. 500--338. National Institute of Standards and Technology (NIST). https://trec.nist.gov/pubs/trec31/papers/Overview_neuclir.pdf.
[31]
Jeremy Leipzig, Daniel Nüst, Charles Tapley Hoyt, Karthik Ram, and Jane Greenberg. 2021. The role of metadata in reproducible computational research. Patterns, 2, 9, 100322.
[32]
Mark Liberman. 2010. Fred Jelinek. Computational Linguistics, 36, 4, (Dec. 2010), 595--599.
[33]
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Frassetto Nogueira. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021. Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, (Eds.) ACM, 2356--2362.
[34]
Jimmy Lin, Joel M. Mackenzie, Chris Kamphuis, Craig Macdonald, Antonio Mallia, Michal Siedlaczek, Andrew Trotman, and Arjen P. de Vries. 2020. Supporting interoperability between open-source search engines with the common index file format. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020. Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, (Eds.) ACM, 2149--2152.
[35]
Aldo Lipani, Florina Piroi, Linda Andersson, and Allan Hanbury. 2014. An information retrieval ontology for information retrieval nanopublications. In Information Access Evaluation. Multilinguality, Multimodality, and Interaction - 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, UK, September 15-18, 2014. Proceedings (Lecture Notes in Computer Science). Evangelos Kanoulas, Mihai Lupu, Paul D. Clough, Mark Sanderson, Mark M. Hall, Allan Hanbury, and Elaine G. Toms, (Eds.) Vol. 8685. Springer, 44--49.
[36]
Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2022. Streamlining evaluation with ir-measures. In Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part II (Lecture Notes in Computer Science). Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty, (Eds.) Vol. 13186. Springer, 305--310.
[37]
Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian. 2021. Simplified data wrangling with ir_datasets. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021. Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, (Eds.) ACM, 2429--2436.
[38]
Craig Macdonald, Richard McCreadie, Rodrygo L. T. Santos, and Iadh Ounis. 2012. From puppy to maturity: experiences in developing terrier. In Proceedings of the SIGIR 2012 Workshop on Open Source Information Retrieval, OSIR@SIGIR 2012, Portland, Oregon, USA, 16th August 2012. Andrew Trotman, Charles L. A. Clarke, Iadh Ounis, J. Shane Culpepper, Marc-Allen Cartright, and Shlomo Geva, (Eds.) University of Otago, Dunedin, New Zealand, 60--63.
[39]
Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. 2021. Pyterrier: declarative experimentation in python from BM25 to dense retrieval. In CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021. Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong, (Eds.) ACM, 4526--4533.
[40]
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29-31, 2019. danah boyd and Jamie H. Morgenstern, (Eds.) ACM, 220--229.
[41]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016 (CEUR Workshop Proceedings). Tarek Richard Besold, Antoine Bordes, Artur S. d'Avila Garcez, and Greg Wayne, (Eds.) Vol. 1773. CEUR-WS.org. https://ceur-ws.org/Vol-1773/CoCoNIPS_2016 _paper9.pdf.
[42]
Giorgio Maria Di Nunzio and Nicola Ferro. 2005. DIRECT: A system for evaluating information access components of digital libraries. In Research and Advanced Technology for Digital Libraries, 9th European Conference, ECDL 2005, Vienna, Austria, September 18-23, 2005, Proceedings (Lecture Notes in Computer Science). Andreas Rauber, Stavros Christodoulakis, and A Min Tjoa, (Eds.) Vol. 3652. Springer, 483--484.
[43]
João R. M. Palotti, Harrisen Scells, and Guido Zuccon. 2019. Trectools: an open-source python library for information retrieval practitioners involved in trec-like campaigns. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019. Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer, (Eds.) ACM, 1325--1328.
[44]
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research(a report from the neurips 2019 reproducibility program). J. Mach. Learn. Res., 22, 164:1--164:20. http://jmlr.org/papers/v22/20-303.html.
[45]
Benjamin Piwowarski. 2020. Experimaestro and datamaestro: experiment and dataset managers (for IR). In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020. Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-RongWen, and Yiqun Liu, (Eds.) ACM, 2173--2176.
[46]
Martin F. Porter. 1980. An algorithm for suffix stripping. Program, 14, 3, 130-- 137.
[47]
Martin Potthast et al. 2021. The information retrieval anthology. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021. Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, (Eds.) ACM, 2550--2555.
[48]
Edward Raff. 2019. A step toward quantifying independently reproducible machine learning research. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, (Eds.), 5486--5496. https://proceedings.neurips.cc/paper/2019/hash/c4 29429bf1f2af051f2021dc92a8ebea-Abstract.html.
[49]
Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen M. Voorhees, Lucy Lu Wang, and William R. Hersh. 2021. Searching for scientific evidence in a pandemic: an overview of TREC-COVID. J. Biomed. Informatics, 121, 103865.
[50]
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994 (NIST Special Publication). Donna K. Harman, (Ed.) Vol. 500--225. National Institute of Standards and Technology (NIST), 109--126. http://trec.nist.gov/pu bs/trec3/papers/city.ps.gz.
[51]
Mark Sanderson. 2010. Test collection based evaluation of information retrieval systems. Found. Trends Inf. Retr., 4, 4, 247--375.
[52]
Harrisen Scells, Shengyao Zhuang, and Guido Zuccon. 2022. Reduce, reuse, recycle: green information retrieval research. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022. Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai, (Eds.) ACM, 2825--2837.
[53]
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Anna Korhonen, David R. Traum, and Lluís Màrquez, (Eds.) Association for Computational Linguistics, 3645--3650.
[54]
Noriko Kando Tetsuya Sakai Douglas W. Oard, (Ed.) 2021. Evaluating Information Retrieval and Access Tasks: NTCIR's Legacy of Research Impact. Springer Singapore. isbn: 9789811555541.
[55]
Ellen M Voorhees and Donna K Harman, (Eds.) 2005. TREC: Experiment and Evaluation in Information Retrieval. en. Digital Libraries and Electronic Publishing. MIT Press, Cambridge, Massachusetts.
[56]
Ellen M. Voorhees, Shahzad Rajput, and Ian Soboroff. 2016. Promoting repeatability through open runs. In Proceedings of the Seventh International Workshop on Evaluating Information Access, EVIA 2016, a Satellite Workshop of the NTCIR- 12 Conference, National Center of Sciences, Tokyo, Japan, june 7, 2016. Emine Yilmaz and Charles L. A. Clarke, (Eds.) National Institute of Informatics (NII). http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/pdf/evia/04- EVIA2016-VoorheesE.pdf.
[57]
Peilin Yang and Hui Fang. 2016. A reproducibility study of information retrieval models. In Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval, ICTIR 2016, Newark, DE, USA, September 12- 6, 2016. Ben Carterette, Hui Fang, Mounia Lalmas, and Jian-Yun Nie, (Eds.) ACM, 77--86.
[58]
Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017. Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, (Eds.) ACM, 1253--1256.
Recommendations
Using TREC for developing semantic information retrieval benchmark for Urdu
AbstractInformation Retrieval (IR) systems are developed to fulfill several needs of users. As a plethora of IR techniques has been developed, therefore the evaluation of these techniques has been of paramount importance. The evaluation of ...
Highlights- Large corpus of over 2,887,169 Urdu documents in the TREC defined SGML format.
- ...
Relevance for browsing, relevance for searching
The concept of relevance has received a great deal of theoretical attention. Separately, the relationship between focused search and browsing has also received extensive theoretical attention. This article aims to integrate these two literatures with a ...
Text retrieval conference (TREC) genomics pre-track workshop
JCDL '02: Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital librariesThe goal of this workshop is to allow individuals interested in the Text Retrieval Conference (TREC, trec.nist.gov) Genomics Pre-Track to come together to discuss common goals and interests for the pre-track. The workshop will be designed to generate a ...
Comments
Information & Contributors
Information
Published In
July 2024
3164 pages
ISBN:9798400704314
DOI:10.1145/3626772
- General Chairs:
- Grace Hui Yang,
- Hongning Wang,
- Sam Han,
- Program Chairs:
- Claudia Hauff,
- Guido Zuccon,
- Yi Zhang
Copyright © 2024 ACM.
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SIGIR 2024
Sponsor:
SIGIR 2024: The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
July 14 - 18, 2024
Washington DC, USA
Acceptance Rates
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 84Total Downloads
- Downloads (Last 12 months)84
- Downloads (Last 6 weeks)25
Reflects downloads up to 15 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in