Install & Compatibility
Where this runs
tested against v0.0.9 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.920 runs
installs and imports cleanly · install 0.0s · import 0.000s · 110.9MB
glibcpy 3.10–3.920 runs
installs and imports cleanly · install 11.7s · import 0.000s · 119MB
121MB installed
● package 121MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
CustomAgent
✓ import agentevals
✗ from agentevals import CustomAgent
This quickstart demonstrates how to define a custom LLM agent, instantiate an evaluator, create a scenario with input data, and run an evaluation. The output shows a JSON representation of the evaluation results.
import json
from agentevals.agents import CustomAgent
from agentevals.evaluators import CodeExecutionEvaluator
from agentevals.scenarios import HumanFeedbackScenario
# 1. Define your custom agent by inheriting from CustomAgent
# and implementing the `run` method.
class MySimpleAgent(CustomAgent):
def run(self, input_data: dict) -> dict:
task = input_data.get("task", "no task specified")
# Simulate an agent processing a task and returning an output
if "math problem" in task:
return {"output": "I processed a math problem!"}
return {"output": f"Agent processed task: '{task}'"}
# 2. Instantiate your agent
agent_instance = MySimpleAgent(name="my-eval-agent")
# 3. Instantiate an evaluator, associating it with your agent.
# CodeExecutionEvaluator is one type; others exist in `agentevals.evaluators`.
evaluator = CodeExecutionEvaluator(agent=agent_instance, max_iterations=1)
# 4. Define a scenario that provides input for your agent.
# HumanFeedbackScenario is one type; others exist in `agentevals.scenarios`.
scenario_data = {
"task": "Solve a simple math problem"
}
evaluation_scenario = HumanFeedbackScenario(
scenario_id="math_scenario_1",
input_data=scenario_data,
# expected_output is optional and its usage depends on the specific evaluator.
expected_output={"result": "Solution to math problem"}
)
# 5. Run the evaluation
results = evaluator.evaluate(scenario=evaluation_scenario)
# Print the structured results
print(json.dumps(results, indent=2))
Debug
Known issues
gotchaAgentevals is explicitly noted as being in 'early development'. This means API interfaces, class names, and method signatures are subject to frequent changes without strict adherence to semantic versioning for minor releases (e.g., `0.x.x` to `0.y.x`).fixRegularly check the official GitHub repository for updates and breaking changes before upgrading. Pin exact versions in `requirements.txt` to prevent unexpected breakages.
affects: All 0.x.x versions
breakingDue to its early development stage, `0.x.x` releases (e.g., upgrading from `0.0.8` to `0.0.9`) can introduce breaking changes. This often includes method renames, argument signature changes, or class restructurings that are not always explicitly called out in patch notes.fixAlways consult the latest README or release notes on GitHub when upgrading to a new `0.x.x` version. Test thoroughly after any version bump and be prepared to adapt your code to new API patterns.
affects: All 0.x.x versions
gotchaMany evaluators and scenarios require specific arguments to be passed during instantiation or evaluation. Forgetting or providing incorrect arguments (e.g., `agent` for evaluators, `input_data` for scenarios) will lead to runtime errors, often `TypeError` or `ValueError`.fixRefer to the specific class's `__init__` and `evaluate` method signatures in the documentation or source code. Ensure all mandatory parameters are provided with the correct types and values.
affects: All 0.x.x versions
Upgrade
Version history
0.0.9latest on PyPI · released Jul 24, 2025
Audit
Dependencies
No dependency data recorded yet.