Registry / ai-ml / trulens-providers-litellm

trulens-providers-litellm

library2.8.1pypypi✓ verified 86d ago

A provider integration for TruLens that uses LiteLLM to evaluate LLM-based applications. It enables feedback functions using models served via LiteLLM (e.g., OpenAI, Anthropic, Cohere, etc.). Version 2.8.1 requires Python >=3.9, <4.0 and is actively maintained.

ai-ml testing

pip install trulens-providers-litellm

github ↗homepage ↗

INSTALL

IMPORT

SIG · TRULENS-PROVIDERS-

trulens-providers-litellm

ai-mlpythonv2.8.1

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

LiteLLM

✓ from trulens.providers.litellm import LiteLLM

✗ from trulens_providers_litellm import LiteLLM

The package name uses hyphens in PyPI, but import is dot-based.

LiteLLMCallback

✓ from trulens.providers.litellm import LiteLLMCallback

✗ from trulens.providers.litellm_callback import LiteLLMCallback

Callback class is part of the same provider module.

Minimal setup: initialize session, create LiteLLM provider with model and API key, define a feedback function, and record/evaluate an app.

import os from trulens.core import TruSession from trulens.providers.litellm import LiteLLM session = TruSession() provider = LiteLLM( model_engine="gpt-4o", api_key=os.environ.get("OPENAI_API_KEY", "your-key-here") ) # Define a feedback function from trulens.core.feedback import Feedback from trulens.providers.litellm import LiteLLM f_qa = Feedback(provider.qs_relevance).on_input_output() # Record and evaluate an app (pseudo-code, app definition omitted) # session.record(app, feedback_functions=[f_qa]).start() # session.evaluate(app_id="my-app")

Debug

Known issues

breakingTruLens 2.x has a significantly different API compared to 1.x. The provider classes have been moved from 'trulens_eval.feedback.provider' to 'trulens.providers.litellm'.

fix

Update imports: from trulens.providers.litellm import LiteLLM (instead of from trulens_eval.feedback.provider.litellm import LiteLLM).

affects: >=2.0.0

gotchaThe LiteLLM provider requires that the underlying model (e.g., gpt-4o) is available via LiteLLM. If the model is not supported, you'll get an error. Always test with a supported model first.

fix

Set model_engine to a known LiteLLM-supported model (e.g., 'gpt-4o', 'claude-3-haiku-20240307').

affects: all

deprecatedIn TruLens 2.x, 'FeedbackMode' and 'App' classes are deprecated; use 'feedback.Feedback' and app-specific wrappers.

fix

Refer to the new documentation for creating feedback functions and recording apps.

affects: >=2.0.0

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'trulens'

TruLens core not installed or wrong Python environment.

fix

pip install trulens-core trulens-providers-litellm

ImportError: cannot import name 'LiteLLM' from 'trulens.providers.litellm'

The provider package might be installed but the imports have changed; or using an older version.

fix

Ensure you have trulens-providers-litellm>=2.0.0 and import as 'from trulens.providers.litellm import LiteLLM'.

Error: The model `gpt-4o-mini` does not exist.

The model engine string is not recognized by LiteLLM or the API key is missing.

fix

Check model spelling and ensure the corresponding API key (e.g., OPENAI_API_KEY) is set.

Upgrade

Version history

2.8.1latest on PyPI · released May 14, 2026

Audit

Dependencies

trulens-corerequiredCore TruLens library required for feedback functions and app tracking.

litellmrequiredLiteLLM is used to call various LLM providers.

Agent activity

21 hits · last 30 days

node

Bingbot

OpenAI (training)

Resources

githubhttps://github.com/truera/trulens ↗

homepagehttps://trulens.org/ ↗