Install & Compatibility
Where this runs
tested against v2.35.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
61MB installed
● package 61MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
Dataset
✓ from pydantic_evals import Dataset
Evaluator
✓ from pydantic_evals import Evaluator
LLMProvider
✓ from pydantic_evals import LLMProvider
Case
✓ from pydantic_evals import Case
Evaluation
✓ from pydantic_evals import Evaluation
This example demonstrates how to define a mock LLM provider, a simple keyword-based evaluator, create a dataset of test cases, and run an evaluation. For real LLM interactions, replace `MockLLM` with an actual `pydantic-ai` LLM client and ensure API keys are set.
import os
from pydantic_evals import Dataset, Evaluation, Evaluator, LLMProvider, Case
from typing import ClassVar
# 1. Define your LLM provider (mocked for a runnable example without API keys)
class MockLLM(LLMProvider):
name: ClassVar[str] = "mock_llm"
model_name: ClassVar[str] = "mock_model"
def get_completion(self, prompt: str) -> str:
if "capital of France" in prompt:
return "The capital of France is Paris."
elif "Python programming" in prompt:
return "Python is named after the British sketch comedy group Monty Python."
return f"Mock LLM response to: {prompt[:50]}..."
# 2. Define your evaluation logic
class SimpleKeywordEvaluator(Evaluator):
def evaluate_case(self, case: Case, actual_output: str, llm: MockLLM) -> Evaluation:
# For simplicity, check for specific keywords based on input
score = 0.0
if "capital of France" in case.input and "Paris" in actual_output:
score = 1.0
elif "Python programming" in case.input and "Python" in actual_output:
score = 1.0
return Evaluation(score=score, details={"actual_output": actual_output})
# 3. Create a Dataset with evaluation cases
dataset = Dataset(
cases=[
Case(input="What is the capital of France?"),
Case(input="Tell me a fun fact about Python programming."),
Case(input="What is 2 + 2?") # This case is designed to fail the evaluator
]
)
# 4. Run the evaluation
if __name__ == "__main__":
llm = MockLLM()
results = dataset.evaluate(
llm=llm,
evaluators=[SimpleKeywordEvaluator()],
batch_size=1,
num_workers=1,
)
print("\nEvaluation Summary:")
for result in results:
print(f" Input: '{result.case.input}'")
print(f" Output: '{result.evaluations[0].details['actual_output']}'")
print(f" Score: {result.evaluations[0].score}")
print(f" Total Case Score: {result.score}")
pydantic_evals --version
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'opentelemetry.sdk'
Pydantic Evals uses OpenTelemetry for tracing, and if `opentelemetry-sdk` is not explicitly installed, some features or internal workings (especially related to duration tracking and Logfire integration) may fail to import necessary modules, even if Logfire itself isn't directly used.
fixInstall the `opentelemetry-sdk` package: `pip install opentelemetry-sdk`.
TypeError: '<' not supported between instances of 'MagicMock' and 'float'
This error typically occurs when trying to display evaluation reports, especially when `logfire` is not installed. Pydantic Evals may use `MagicMock` objects for duration attributes in `ReportCase` when `logfire` is absent, leading to type incompatibility when comparing them with float values during report rendering.
fixInstall `logfire` if you intend to use its features or avoid this error during report display: `pip install logfire`. Alternatively, ensure your code doesn't attempt to process or display duration-related metrics if `logfire` is intentionally omitted.
PydanticUserError: A non-annotated attribute was detected: `YOUR_ATTRIBUTE_NAME`
This is a core Pydantic validation error that arises when defining models (such as `Case` input/output models or custom `Evaluator` dataclasses) where an attribute is present without a type annotation. Pydantic requires all model fields to be type-annotated, or class-level attributes that are not fields should be marked as `ClassVar`.
fixAdd a type annotation to the identified attribute, for example: `my_attribute: str` or, if it's a class-level variable not intended to be a Pydantic field, mark it with `typing.ClassVar`: `from typing import ClassVar; MY_CONSTANT: ClassVar[str] = 'value'`.
ValidationError: Input should be '...' (or similar validation failure messages)
This common Pydantic error occurs when the data provided to a `pydantic-evals` component (like `Dataset`, `Case` inputs, `expected_output`, or custom `Evaluator` parameters) does not conform to the defined Pydantic model schema, including incorrect types, missing required fields, or failures of custom validators.
fixReview the `Pydantic` model definition for the component you are interacting with (e.g., the `Case` definition, or the parameters of a custom `Evaluator`). Ensure that the input data strictly matches the expected types, formats, and constraints defined in the schema. Check `error` messages for specific `loc` fields to pinpoint the exact location of the validation failure.
Upgrade
Version history
2.35.1latest on PyPI · released Aug 27, 2026
Audit
Dependencies
pydantic>=2.0requiredCore data validation and settings management, foundational to the library's design.
pydantic-aioptionalProvides actual LLM integrations (e.g., OpenAI, Anthropic) which are commonly used with pydantic-evals. While pydantic-evals can be used with custom LLMProvider implementations, integration with existing LLMs is often done via pydantic-ai.