Install & Compatibility
Where this runs
tested against v0.3.260 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.9
✕ build_error
✕ build_error
321MB installed
● package 321MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
Task
✓ from inspect_ai import Task
task
✓ from inspect_ai import task
Sample
✓ from inspect_ai.dataset import Sample
generate
✓ from inspect_ai.solver import generate
exact
✓ from inspect_ai.scorer import exact
This 'Hello World' example defines a simple evaluation task. It instructs a model to reply with 'Hello World' and uses an exact match scorer to verify the output. To run this, you need to save the code as a Python file (e.g., `hello_eval.py`), ensure the `openai` package is installed, and set your `OPENAI_API_KEY` environment variable. You then execute it via the `inspect eval` command-line interface.
import os
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.solver import generate
from inspect_ai.scorer import exact
@task
def hello_world():
return Task(
dataset=[
Sample(input="Just reply with Hello World", target="Hello World"),
],
solver=[generate()],
scorer=exact(),
)
# To run this, save it as a Python file (e.g., hello_eval.py)
# and execute from your terminal:
# export OPENAI_API_KEY=your_openai_api_key # Or set in .env file
# inspect eval hello_eval.py --model openai/gpt-4o
# Example of setting the key for programmatic use (less common for inspect eval CLI)
# os.environ['OPENAI_API_KEY'] = os.environ.get('OPENAI_API_KEY', 'sk-...')
inspect --version
Debug
Known issues
gotchaUsing the local tool environment (`--sandbox local`) without an 'outer sandbox' is explicitly warned against as it can be a security risk, as tools are executed on the client system.fixOnly use `--sandbox local` when the entire evaluation is already contained within a secure, isolated environment (e.g., Docker, Kubernetes). For sensitive operations, consider more robust sandbox options or carefully review tool definitions.
affects: All versions
gotchaTool usage (e.g., `bash()`, `python()`) is not universally supported by all LLM model providers. Tools are executed on the client machine, not within the model's environment.fixConsult the Inspect AI documentation for 'Model Providers' to verify which models support tool use before designing evaluations that rely on them. Be aware of the execution context of tools.
affects: All versions
gotchaAPI keys for model providers (e.g., OpenAI, Anthropic, Google) must be correctly configured, typically as environment variables (e.g., `OPENAI_API_KEY`). Evaluations will fail if these are missing or incorrect.fixEnsure the relevant API key is set as an environment variable before running evaluations. For example: `export OPENAI_API_KEY=your-key`. You may also use `.env` files with `python-dotenv`.
affects: All versions
gotchaBy default, raw model API request/response logs are only captured and displayed when an error occurs. This can obscure debugging for successful but unexpected model behaviors.fixTo enable comprehensive logging of all model API calls, use the `--log-model-api` command-line option when running `inspect eval`.
affects: All versions before 0.3.184 (default changed), users may still revert to old behavior.
breakingCompatibility with external LLM client libraries (e.g., `openai`, `anthropic`, `mistralai`) frequently requires specific minimum versions due to upstream breaking changes in those packages. For example, `openai` v1.104.1 became a minimum required version due to type changes and web search action renames.fixIf encountering errors related to model API calls or types, ensure your `inspect-ai` installation is up-to-date and check the `inspect-ai` changelog for notes on required versions of specific model provider packages. Upgrade those packages as necessary.
affects: Various, depending on the specific external library and its updates. Observed for `openai` around 0.3.127-0.3.128 and `mistralai` around 0.3.191.
breakingRecent versions of `inspect-ai` (0.3.10 and newer) require Python 3.10 or a more recent version. Attempting to install in an older Python environment will result in errors like 'Requires-Python >=3.10' and 'No matching distribution found for inspect-ai'.fixEnsure your Python environment is version 3.10 or newer before installing `inspect-ai`. For example, use `python:3.10-slim` or a later version in your environment configuration.
affects: 0.3.10+
Errors
Common errors & fixes
TypeError: parse_answers() missing 1 required positional argument: 'multiple_correct'
This error occurs due to a breaking change in a recent `inspect-ai` update (around commit e4a551f), where the `parse_answers()` function now requires a new `multiple_correct` argument, affecting older evaluation implementations like the Personality eval.
fixUpdate your custom scorer or evaluation code to pass the `multiple_correct` argument to `parse_answers()`, or consult the `inspect-ai` changelog for the exact API changes and recommended migration path for your version.
ModuleNotFoundError: No module named 'inspect_ai'
This error indicates that the `inspect-ai` package is not installed in your Python environment, or your Python interpreter cannot find the installed package.
fixInstall the library using pip: `pip install inspect-ai`. If using a virtual environment, ensure it is activated. In VS Code, verify the correct Python interpreter with `inspect-ai` installed is selected for your workspace.
openai.AuthenticationError: Incorrect API key provided
This common error occurs when `inspect-ai` attempts to use an OpenAI model, but the `OPENAI_API_KEY` environment variable is either missing, incorrect, or has insufficient permissions. Similar errors can occur with other model providers if their respective API keys are not properly configured.
fixEnsure your OpenAI API key is correctly set as an environment variable (e.g., `export OPENAI_API_KEY=your-api-key`) or in a `.env` file that `inspect-ai` can load. Check the official documentation for other providers you are using.
inspect_ai.limit.LimitExceededError
This error is raised by `inspect-ai` when an evaluation task or sample exceeds predefined limits, such as maximum messages, tokens, time, or cost, which are often configured to prevent runaway LLM usage.
fixAdjust the limits for your evaluation using `inspect eval` CLI options (e.g., `--max-messages`, `--max-tokens`, `--time-limit`, `--cost-limit`) or programmatically within your task definition, or consider refining your prompt/solver to be more concise.
AttributeError: module 'inspect' has no attribute 'getargspec'
This error typically arises in Python 3.11 and newer versions because the `inspect.getargspec` function has been deprecated and removed. It often indicates that `inspect-ai` or one of its dependencies is using an outdated method to inspect function arguments, or a user might mistakenly try to use the built-in `inspect` module with deprecated functions while expecting `inspect-ai` functionality.
fixUpdate `inspect-ai` and its dependencies to their latest versions, as newer versions are likely to use `inspect.signature` instead of `getargspec`. If it's your own code, refactor to use `inspect.signature` for argument inspection. `inspect-evals` officially supports Python 3.11 and 3.12, so ensure your environment is up-to-date.
Upgrade
Version history
0.3.260latest on PyPI · released Aug 21, 2026
Audit
Dependencies
python>=3.10requiredMinimum Python version required.
pydanticrequiredUsed for data validation and settings management.
httpxrequiredAsynchronous HTTP client for API interactions.
anyiorequiredAsynchronous I/O backend.
python-dotenvoptionalCommonly used for loading environment variables like API keys.
openaioptionalRequired for evaluating OpenAI models.
anthropicoptionalRequired for evaluating Anthropic models.
google-genaioptionalRequired for evaluating Google Gemini models.