Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations. Retrieval using sparse representations is provided via integration with our group's Anserini IR toolkit, which is built on Lucene. Retrieval using dense representations is provided via integration with Facebook's Faiss library.

Pyserini is primarily designed to provide effective, reproducible, and easy-to-use first-stage retrieval in a multi-stage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, prebuilt indexes, and evaluation scripts for many commonly used IR test collections. With Pyserini, it's easy to reproduce runs on a number of standard IR test collections!

For additional details, our paper in SIGIR 2021 provides a nice overview.

✨ New! Guide to working with the MS MARCO 2.1 Document Corpus for TREC 2024 RAG Track.

❗ Anserini was upgraded from JDK 11 to JDK 21 at commit 272565 (2024/04/03), which corresponds to the release of v0.35.0. Correspondingly, Pyserini was upgraded to JDK 21 at commit b2f677 (2024/04/04).

Install via PyPI:

pip install pyserini

Pyserini is built on Python 3.10 (other versions might work, but YMMV) and Java 21 (due to its dependency on Anserini). A pip installation will automatically pull in major dependencies such as PyTorch, 🤗 Transformers, and the ONNX Runtime.

The toolkit also comes with "extras":

pip install 'pyserini[extras]'

Notably, faiss-cpu, lightgbm, and nmslib are included in these "extras". Installation of these packages can be temperamental, which is why they are not included in the core dependencies. It might be a good idea to install these yourself separately.

The software ecosystem is rapidly evolving and a potential source of frustration is incompatibility among different versions of underlying dependencies. We provide additional detailed installation instructions here.

If you're planning on just using Pyserini, then the pip instruction (without "extras") should be fine. However, if you're planning on contributing to the codebase or want to work with the latest not-yet-released features, you'll need a development installation. Instructions are provided here.

Pyserini supports different types of retrieval models. See this guide for details on how to search common corpora in IR and NLP research (e.g., MS MARCO, NaturalQuestions, BEIR, etc.) using indexes that we have already built for you. Here are direct links into the guide:

• Traditional lexical models (e.g., BM25) using Lucene.
• Learned sparse retrieval models (e.g., uniCOIL, SPLADE, etc.) using Lucene.
• Learned dense retrieval models (e.g., DPR, Contriever, BGE, etc.) using Lucene or Faiss.
• Hybrid retrieval models (e.g., dense-sparse fusion).

Once you get the top-k results, you'll actually want to fetch the document text... See this guide for how.

Well, it depends on what type of retrieval model you want to search with:

• Building a BM25 Index (Direct Java Implementation)
• Building a BM25 Index (Embeddable Python Implementation)
• Building a Sparse Vector Index
• Building a Dense Vector Index

The steps are different for different classes of models: this guide (same as the links above) describes the details.

• How do I configure search? (Guide to Interactive Search)
• How do I manually download indexes? (Guide to Interactive Search)
• How do I perform dense and hybrid retrieval? (Guide to Interactive Search)
• How do I iterate over index terms and access term statistics? (Index Reader API)
• How do I traverse postings? (Index Reader API)
• How do I access and manipulate term vectors? (Index Reader API)
• How do I compute the tf-idf or BM25 score of a document? (Index Reader API)
• How do I access basic index statistics? (Index Reader API)
• How do I access underlying Lucene analyzers? (Analyzer API)
• How do I build custom Lucene queries? (Query Builder API)
• How do I iterate over raw collections? (Collection API)

With Pyserini, it's easy to reproduce runs on a number of standard IR test collections! We provide a number of prebuilt indexes that directly support reproducibility "out of the box".

In our SIGIR 2022 paper, we introduced "two-click reproductions" that allow anyone to reproduce experimental runs with only two clicks (i.e., copy and paste). Documentation is organized into reproduction matrices for different corpora that provide a summary of different experimental conditions and query sets:

• MS MARCO V1 Passage
• MS MARCO V1 Document
• MS MARCO V2 Passage
• MS MARCO V2 Document
• BEIR
• Mr.TyDi
• MIRACL
• Open-Domain Question Answering
• CIRAL

For more details, see our paper on Building a Culture of Reproducibility in Academic Research.

Additional reproduction guides below provide detailed step-by-step instructions.

• Guide to prebuilt indexes
• Guide to interactive searching
• Guide to text classification with the 20Newsgroups dataset
• Guide to working with the COVID-19 Open Research Dataset (CORD-19)
• Guide to working with entity linking
• Guide to working with spaCy
• Usage of the Analyzer API
• Usage of the Index Reader API
• Usage of the Query Builder API
• Usage of the Collection API
• Direct Interaction via Pyjnius

• v0.42.0 (w/ Anserini v0.38.0): November 8, 2024 [Release Notes]
• v0.41.0 (w/ Anserini v0.38.0): November 7, 2024 [Release Notes] [Known Issues]
• v0.40.0 (w/ Anserini v0.38.0): October 28, 2024 [Release Notes]
• v0.39.0 (w/ Anserini v0.38.0): September 27, 2024 [Release Notes]
• v0.38.0 (w/ Anserini v0.38.0): September 11, 2024 [Release Notes]
• v0.37.0 (w/ Anserini v0.37.0): August 26, 2024 [Release Notes]
• v0.36.0 (w/ Anserini v0.36.1): June 17, 2024 [Release Notes]
• v0.35.0 (w/ Anserini v0.35.0): April 4, 2024 [Release Notes]

If you use Pyserini, please cite the following paper:

@INPROCEEDINGS{Lin_etal_SIGIR2021_Pyserini,
   author = "Jimmy Lin and Xueguang Ma and Sheng-Chieh Lin and Jheng-Hong Yang and Ronak Pradeep and Rodrigo Nogueira",
   title = "{Pyserini}: A {Python} Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations",
   booktitle = "Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021)",
   year = 2021,
   pages = "2356--2362",
}

This research is primarily supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada.