Install & Compatibility
Where this runs
tested against v0.4.12 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.10
✕ build_error
✓ 37.7s
py 3.11
✕ build_error
✓ 35.2s
py 3.12
✕ build_error
✓ 33.3s
py 3.13
✕ build_error
✓ 32.4s
py 3.9
✕ build_error
✕ timeout
705MB installed
● package 705MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
models.get_model
✓ from lm_eval import models
✗ from lm_eval.models.huggingface import HFLM
Use `models.get_model` for a unified interface, avoids direct import of specific backend classes which might change.
tasks.get_task_dict
✓ from lm_eval import tasks
✗ from lm_eval.tasks.hellaswag import HellaSwag
Use `tasks.get_task_dict` to load tasks dynamically by name, which is more robust to internal structure changes.
evaluator.evaluate
✓ from lm_eval import evaluator
This quickstart demonstrates how to initialize a HuggingFace model, select evaluation tasks, and run the evaluation using the Python API. It uses a small `tiny-gpt2` model on CPU for fast execution, which should be replaced with a more powerful model and GPU for meaningful results.
import os
from lm_eval import models, tasks, evaluator
# NOTE: For quickstart, we use a small model and CPU.
# For real evaluations, use a GPU and a larger model.
# You might need to install 'lm-eval[hf]' or 'lm-eval[main]'
# Setup a model (e.g., HuggingFace model)
# Using a tiny model for quick execution, replace with desired model
model_name = "sshleifer/tiny-gpt2"
lm = models.get_model("hf", pretrained=model_name, device="cpu")
# Select tasks (e.g., 'hellaswag')
task_names = ["hellaswag"]
task_dict = tasks.get_task_dict(task_names)
# Evaluate the model
results = evaluator.evaluate(
lm=lm,
task_dict=task_dict,
num_fewshot=0, # Number of few-shot examples (0 for zero-shot)
batch_size=None, # Auto-batching
device="cpu", # Or "cuda:0" for GPU
limit=10 # Limit number of samples for quick testing
)
print(results)
lm-eval --version
Debug
Known issues
breakingThe base `pip install lm_eval` no longer includes model backends (e.g., HuggingFace/PyTorch stack) by default. These must now be installed explicitly.fixInstall with extras: `pip install "lm-eval[main]"` for common backends, or `pip install "lm-eval[hf]"` for HuggingFace, `"lm-eval[vllm]"` for vLLM, etc.
affects: >=0.4.10
breakingPython 3.10 or newer is now the minimum required version.fixUpgrade your Python environment to 3.10 or later.
affects: >=0.4.9.2 (Python 3.8 support was dropped in v0.4.8)
breakingChat template delimiter handling changed, particularly affecting multiple-choice tasks. This might alter how prompts are constructed for models expecting specific chat formats.fixReview your chat template configurations and model expectations, especially for tasks sensitive to prompt formatting. Test against previous versions if possible.
affects: >=0.4.6
gotchaTask versions can change between releases. Results from a previous task version may not be directly comparable with results from an updated version.fixAlways note the `lm-eval` version and specific task versions when reporting or comparing results. Consult release notes or task definitions for version changes.
affects: All versions (ongoing concern)
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'datasets'
The 'datasets' library, a required dependency for many evaluation tasks in lm-eval, is not installed in the current Python environment.
fixInstall the 'datasets' library using pip: `pip install datasets` or, if using specific lm-eval extras, `pip install lm-eval[hf]` which includes datasets and transformers.
ValueError: No tasks specified, or no tasks found. Please verify the task names.
The evaluation harness could not find or recognize the specified task(s). This often happens due to a typo in the task name, the task not being installed, or the current working directory preventing proper task registration.
fixVerify the task name(s) by running `lm_eval --tasks list`. If the task is custom, ensure its YAML configuration is correctly placed and registered, or try running the command from a different directory if it's a path-related issue.
ValueError: Attempted to load model 'hf', but no model for this name found! Supported model names: local-completions, openai-completions, etc.
The specified model backend (e.g., 'hf' for HuggingFace models) is not recognized or its necessary dependencies are not installed, particularly after changes in lm-eval's modular installation.
fixEnsure the correct model backend extra is installed. For HuggingFace models, run `pip install lm-eval[hf]`. For other backends, consult the lm-eval documentation for the correct installation command (e.g., `pip install lm-eval[vllm]`).
AttributeError: module 'lm_eval.tasks' has no attribute 'ALL_TASKS'
This error occurs when using outdated code or tutorials that reference an old API for accessing tasks. The `ALL_TASKS` attribute was removed or refactored in newer versions of lm-eval (e.g., 0.4.x).
fixInstead of directly accessing `lm_eval.tasks.ALL_TASKS`, use the `TaskManager` API. Instantiate `tm = TaskManager()` and then access tasks via `tm.all_tasks`.
Upgrade
Version history
0.4.12latest on PyPI · released May 11, 2026
Audit
Dependencies
torchoptionalRequired for HuggingFace backend models. Part of `[hf]` extra.
transformersoptionalRequired for HuggingFace backend models. Part of `[hf]` extra.
vllmoptionalRequired for vLLM backend models. Part of `[vllm]` extra.