Registry / data / pytrec-eval-terrier

pytrec-eval-terrier

library0.5.10pypypi✓ verified 24d ago

pytrec-eval-terrier provides Python bindings for common Information Retrieval evaluation measures, leveraging the highly optimized `trec_eval` C library. It simplifies the process of evaluating ranking performance for search systems. The current version is 0.5.10, and releases occur periodically, often tied to Python version support or minor bug fixes.

data ai-ml

pip install pytrec-eval-terrier

github ↗package ↗

INSTALL

IMPORT

SIG · PYTREC-EVAL-TERRIE

pytrec-eval-terrier

datapythonv0.5.10

Install

7.4s avg

Import

237ms

Disk

233MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.5.10 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.236s · 235.1MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 7.4s · import 0.238s · 223MB

233MB installed

● package 233MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

pytrec_eval

✓ import pytrec_eval

✗ import pytrec_eval_terrier

The PyPI package is 'pytrec-eval-terrier', but the imported module is 'pytrec_eval'.

RelevanceEvaluator

✓ from pytrec_eval import RelevanceEvaluator

This quickstart demonstrates how to use `pytrec_eval` to evaluate a set of runs against relevance judgments. It shows how to prepare QRELs and runs as dictionaries, select evaluation measures, instantiate the `RelevanceEvaluator`, and compute both per-query and aggregated results.

import pytrec_eval # Example QRELS (Query Relevance Judgments) # Format: {query_id: {doc_id: relevance_score}} qrels = { 'q1': {'d1': 1, 'd2': 0, 'd3': 1}, 'q2': {'d4': 1, 'd5': 0} } # Example RUNS (System Rankings) # Format: {query_id: {doc_id: score}} runs = { 'q1': {'d1': 0.9, 'd2': 0.8, 'd4': 0.7}, 'q2': {'d4': 0.95, 'd6': 0.85} } # Define measures to evaluate measures = pytrec_eval.supported_measures # Or a specific set: measures = {'map', 'ndcg_cut.10', 'recip_rank'} # Instantiate the evaluator evaluator = pytrec_eval.RelevanceEvaluator(qrels, measures) # Evaluate the runs results = evaluator.evaluate(runs) # Print results for a specific query and measure print(f"MAP for q1: {results['q1']['map']:.4f}") print(f"NDCG@10 for q2: {results['q2']['ndcg_cut_10']:.4f}") # Print average results across all queries agg_results = pytrec_eval.compute_aggregated_results(evaluator, results, measures) print(f"Average MAP: {agg_results['map']:.4f}")

Debug

Known issues

breakingMinimum required Python version was upgraded to 3.8. Users on Python 3.7 or older will encounter errors or build failures with versions 0.5.7 and newer.

fix

Upgrade to Python 3.8 or newer. Ensure your CI/CD pipelines use a compatible Python version.

affects: >=0.5.7

gotchaThe PyPI package name is `pytrec-eval-terrier`, but the Python module to import is `pytrec_eval`.

fix

Always use `import pytrec_eval` or `from pytrec_eval import ...` in your Python code, not `pytrec_eval_terrier`.

affects: All

gotchaWhen installing on systems without pre-built wheels (e.g., specific Linux distributions, older Python versions, or ARM architectures), compilation from source may require a C/C++ compiler (like GCC or Clang) and Python development headers (`python3-dev` or similar).

fix

Ensure `build-essential` (Debian/Ubuntu) or `Development Tools` (CentOS/RHEL) and `python3-dev` (or equivalent) packages are installed on your system if you encounter build errors during `pip install`.

affects: All

gotchaInput QRELs and runs must adhere to specific dictionary formats: `{query_id: {doc_id: score}}`. Incorrectly structured inputs will lead to evaluation errors or unexpected results.

fix

Carefully ensure your QRELs and run files are parsed into the expected nested dictionary format before passing them to `RelevanceEvaluator`. Consult the official documentation for examples.

affects: All

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'pytrec_eval_terrier'

The `pytrec-eval-terrier` package has not been installed in the current Python environment or the environment is not active.

fix

Install the package using pip: `pip install pytrec-eval-terrier`

error: command 'gcc' failed with exit status 1

The C compiler (gcc) required to build the underlying `trec_eval` C extension is not installed or not found in the system's PATH during installation.

fix

Install the necessary build tools for your operating system (e.g., `sudo apt-get install build-essential` for Debian/Ubuntu, `xcode-select --install` for macOS).

AttributeError: 'list' object has no attribute 'keys'

The `run` or `qrels` arguments passed to `pytrec_eval_terrier.evaluate` were lists or other non-dictionary types, while dictionaries of dictionaries are expected.

fix

Ensure both `run` and `qrels` are structured as dictionaries where outer keys are query IDs and inner keys are document IDs, with relevance scores or rankings as values.

ValueError: Evaluation measure 'invalid_measure' is not supported.

One or more measure names provided in the `measures` set to `pytrec_eval_terrier.evaluate` is not recognized by the underlying `trec_eval` library.

fix

Use only valid `trec_eval` measure names (e.g., 'map', 'ndcg', 'P_5', 'R_100'). Refer to `trec_eval` documentation for supported measures.

Upgrade

Version history

0.5.10latest on PyPI · released Oct 20, 2025

Audit

Dependencies

No dependency data recorded yet.

Agent activity

15 hits · last 30 days

node

OpenAI (training)

Amazon

Resources

githubhttps://github.com/terrierteam/pytrec_eval ↗

packagehttps://pypi.org/project/pytrec-eval-terrier/ ↗