Install & Compatibility
Where this runs
tested against v? · pip install
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
build_error
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
MTEB
✓ from mteb import MTEB
✗ from mteb.MTEB import MTEB
The MTEB class was part of a major refactor in v2; direct use is less common than 'mteb.evaluate' or 'mteb.get_model' now.
evaluate
✓ import mteb
results = mteb.evaluate(model, tasks=tasks)
The recommended way to run evaluations.
get_model
✓ import mteb
model = mteb.get_model('sentence-transformers/all-MiniLM-L6-v2')
Recommended for loading existing models as implemented in MTEB for reproducibility.
get_tasks
✓ import mteb
tasks = mteb.get_tasks(tasks=['Banking77Classification.v2'])
Used to select specific benchmark tasks.
SentenceTransformer
✓ from sentence_transformers import SentenceTransformer
Used to load models that are not yet directly implemented in MTEB's registry.
This quickstart demonstrates how to load a pre-trained Sentence Transformer model and evaluate it on a specific MTEB task using the `mteb.evaluate` function. It showcases how to select tasks and retrieve the evaluation results.
import mteb
from sentence_transformers import SentenceTransformer
# Select a model to evaluate
model_name = "sentence-transformers/all-MiniLM-L6-v2"
# It's recommended to use mteb.get_model for reproducibility if the model is in MTEB's registry
# Otherwise, SentenceTransformer can be used directly
model = mteb.get_model(model_name) # Will fall back to SentenceTransformer if not registered in MTEB
# Select tasks to run (e.g., a specific classification task)
tasks = mteb.get_tasks(tasks=["Banking77Classification.v2"], languages=["eng"])
# Evaluate the model on the selected tasks
print(f"Running evaluation for {model_name} on {len(tasks)} tasks...")
results = mteb.evaluate(model, tasks=tasks)
print("Evaluation complete. Results:")
for task_name, task_results in results.items():
print(f"Task: {task_name}")
print(f" Main score: {task_results['main_score']:.4f}")
# Example of accessing detailed metrics
if 'accuracy' in task_results['mteb_results']:
print(f" Accuracy: {task_results['mteb_results']['accuracy']:.4f}")
# To save results to a specific folder
# output_folder = f"./results/{model_name.replace('/', '_')}"
# results = mteb.evaluate(model, tasks=tasks, output_folder=output_folder)
# print(f"Results saved to: {output_folder}")
mteb --version
Debug
Known issues
breakingMTEB v2 introduced a large-scale refactor with breaking changes, particularly affecting direct usage of `mteb.MTEB` class and `mteb.load_results` functions. Past minor/patch releases also occasionally introduced breaking changes.fixRefer to the official MTEB documentation for the updated API, especially focusing on `mteb.evaluate`, `mteb.get_model`, and `mteb.get_tasks`. Ensure your code aligns with the new functional interface rather than directly instantiating the `MTEB` class.
affects: All versions prior to 2.x when upgrading to 2.x; potentially minor versions before 2.x.
gotchaEvaluating high-performing or large multilingual models on MTEB can be computationally very expensive, requiring significant GPU resources and time, especially for tasks with large document collections like retrieval.fixStart with smaller task subsets or mini-benchmarks to estimate resource usage. Consider using optimized models or distributed evaluation setups. MTEB also offers caching mechanisms to speed up repeated evaluations.
affects: All versions
gotchaModels excelling on the general MTEB leaderboard might underperform on domain-specific data. The benchmark datasets may not perfectly reflect unique domain, user behavior, or query patterns.fixAlways perform additional evaluations on your specific domain data to validate model suitability. MTEB can be extended with custom tasks to facilitate this.
affects: All versions
deprecatedDirectly submitting model results to the MTEB leaderboard by adding metadata to Hugging Face model cards is no longer supported.fixFollow the updated submission guidelines on the MTEB GitHub repository or documentation to ensure results are correctly associated with the model implementation.
affects: Post-v1.x (Exact version unclear, but mentioned after v2 refactor)
gotchaWhen evaluating existing models, it is recommended to use `mteb.get_model("{model_name}")` instead of directly using `SentenceTransformer("{model_name}")`. This ensures consistent and reproducible results as it loads the model as MTEB implemented it, accounting for specific normalizations, quantizations, or prompts.fixReplace `model = SentenceTransformer(model_name)` with `model = mteb.get_model(model_name)`. MTEB's function will fall back to `SentenceTransformer` if the model isn't specifically registered.
affects: All versions, particularly relevant for models already on the MTEB leaderboard.
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'mteb.types'
This error typically occurs when `mteb` or one of its core dependencies, like `datasets`, is not correctly installed, or there's an outdated version mismatch causing certain internal modules or submodules to be missing or reorganized in the installed package.
fixEnsure `mteb` and its dependencies are up-to-date and correctly installed. It's often resolved by a fresh installation or upgrade: `pip install --upgrade mteb datasets`.
TypeError: Encoding.encode() got an unexpected keyword argument 'batch_size'
This `TypeError` happens when an embedding model, or its wrapper within `mteb`, is called with a `batch_size` argument, but the model's `encode` method (or the specific implementation being used) does not support or expect this argument. This can be due to an older or incompatible model implementation or an older `mteb` version trying to pass a `batch_size` to a model that doesn't handle it directly.
fixCheck the `mteb` and `sentence-transformers` versions for compatibility. Upgrade both libraries: `pip install --upgrade mteb sentence-transformers`. If the issue persists, the specific model might require custom handling or a different `encode_kwargs` configuration.
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X MiB (GPU 0; Y GiB total capacity; Z GiB already allocated; A GiB free; B GiB reserved in total by PyTorch)
This common error in deep learning indicates that the GPU ran out of memory while trying to load a model or process a batch of data. This can be caused by using large models, large `batch_size`, or processing long sequences on a GPU with insufficient VRAM.
fixReduce the `batch_size` when calling `mteb.evaluate` or directly when encoding. For very large models, consider loading them in lower precision (e.g., `torch_dtype=torch.float16` or 4-bit/8-bit quantization if supported) or using a GPU with more VRAM.
AttributeError: 'BaseModelOutputWithPooling' object has no attribute 'norm'
This `AttributeError` typically arises from incompatibilities between `mteb` and the `transformers` library, especially when `transformers` updates its model output structures (e.g., from raw tensors to `BaseModelOutputWithPooling` objects) or changes the API for specific model types (like audio or cross-encoders). `mteb`'s internal model handling might expect a different attribute or method that no longer exists in the new `transformers` output.
fixPin the `transformers` library to a version known to be compatible with your `mteb` version, or upgrade `mteb` to its latest version, which might include fixes for newer `transformers` versions: `pip install --upgrade mteb transformers`.
ValueError: BuilderConfig 'corpus' not found. Available: ['default'].
This `ValueError` occurs when `mteb` attempts to load a dataset from the Hugging Face `datasets` library, but the specified dataset configuration (e.g., 'corpus') is not found, and only 'default' is available. This often happens with specific tasks or datasets that have unique configurations or when an old configuration is referenced after a dataset update.
fixEnsure that the dataset configuration (e.g., `corpus`, `queries`, `qrels`) is available for the specific task and language you are running. If you are customizing a task, verify the `AbsTask` implementation or use the `default` configuration if it contains the necessary data. Clearing the Hugging Face dataset cache might also help: `rm -rf ~/.cache/huggingface/datasets`.
Upgrade
Version history
2.20.4latest on PyPI · released Aug 29, 2026
Audit
Dependencies
pythonrequiredRuntime environment
sentence-transformersrequiredCommonly used for loading and evaluating many pre-trained models. MTEB also offers its own model loading mechanism.
torchrequiredUnderlying deep learning framework, implicitly required by sentence-transformers and many models.
transformersrequiredUnderlying library for many models and tokenizers.