Install & Compatibility
Where this runs
tested against v0.4.3 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
build_error
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 40.5s · import 1.560s · 715MB
732MB installed
● package 732MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
EvaluationDataset
✓ from ragas.dataset_schema import EvaluationDataset
✗ from ragas import EvaluationDataset
Ragas v0.2+ RAG evaluation with EvaluationDataset and class-based metrics.
# pip install ragas langchain-openai
from ragas import EvaluationDataset, SingleTurnSample, evaluate
from ragas.metrics import Faithfulness, ResponseRelevancy, LLMContextRecall
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI
import os
os.environ['OPENAI_API_KEY'] = 'your-key'
llm = LangchainLLMWrapper(ChatOpenAI(model='gpt-4o-mini'))
samples = [
SingleTurnSample(
user_input='What is the capital of France?',
response='The capital of France is Paris.',
retrieved_contexts=['Paris is the capital and most populous city of France.'],
reference='Paris' # ground truth — needed for recall
)
]
dataset = EvaluationDataset(samples=samples)
result = evaluate(
dataset,
metrics=[
Faithfulness(llm=llm),
ResponseRelevancy(llm=llm),
LLMContextRecall(llm=llm)
]
)
print(result)
# {'faithfulness': 1.0, 'response_relevancy': 0.97, 'context_recall': 1.0}
Debug
Known issues
breakingv0.2 renamed all field names: question→user_input, answer→response, contexts→retrieved_contexts. Using old field names silently produces empty/wrong evaluations.fixSingleTurnSample(user_input=..., response=..., retrieved_contexts=[...])
affects: >= 0.2
breakinganswer_relevancy metric renamed to ResponseRelevancy in v0.2. 'from ragas.metrics import answer_relevancy' still works but is deprecated and will be removed in v1.0.fixfrom ragas.metrics import ResponseRelevancy; ResponseRelevancy(llm=llm)
affects: >= 0.2
breakingevaluate() now takes EvaluationDataset not a HuggingFace Dataset. Passing HuggingFace Dataset directly raises TypeError in v0.2+.fixeval_dataset = EvaluationDataset.from_hf_dataset(hf_dataset) then evaluate(eval_dataset, ...)
affects: >= 0.2
breakingMetrics must be initialized as class instances with llm= argument. Old pattern of using lowercase singleton (faithfulness, answer_relevancy) deprecated — will be removed in v1.0.fixFaithfulness(llm=llm) not faithfulness. Pass LLM explicitly to each metric.
affects: >= 0.2
gotchaAll LLM-judge metrics require an async LLM. Ragas uses async internally — synchronous LLM wrappers will cause errors. Use LangchainLLMWrapper or ragas.llms.llm_factory.fixfrom ragas.llms import LangchainLLMWrapper; llm = LangchainLLMWrapper(ChatOpenAI(...))
affects: >= 0.2
gotchaContext recall (LLMContextRecall) requires a reference (ground truth) field. Running it without reference gives a score of 0 or error.fixInclude reference='ground truth answer' in SingleTurnSample for recall metrics.
affects: all
gotchaRagas collects anonymized telemetry by default. Set RAGAS_DO_NOT_TRACK=true to opt out.fixexport RAGAS_DO_NOT_TRACK=true
affects: all
breakingWhen using `ragas` (or its dependencies like `instructor`) with Python 3.9, a `TypeError: unsupported operand type(s) for |: 'type' and 'type'` may occur during module import. This is caused by dependencies utilizing the Python 3.10+ type union syntax (`TypeA | TypeB`) without the necessary `from __future__ import annotations` or `eval_type_backport` package in a Python 3.9 environment.fixUpgrade your Python environment to 3.10 or newer. If staying on Python 3.9 is strictly required, you might be able to resolve this by installing the `eval_type_backport` package (`pip install eval_type_backport`) if the library's usage allows for it.
affects: ragas >= 0.2, Python < 3.10
breakingBuilding libraries with C/C++/Cython extensions (like scikit-network) may fail on minimal Docker images like 'alpine' due to missing build essential tools (e.g., g++, make).fixEnsure build essentials are installed in your Dockerfile (e.g., for Alpine: 'apk add build-base').
affects: all
Upgrade
Version history
0.4.3latest on PyPI · released Jan 13, 2026
Audit
Dependencies
langchain-openaioptionalRequired if using LangChain LLM wrapper for metrics. Optional — can also use ragas.llms.llm_factory with openai directly.
openairequiredRequired for LLM-as-judge metrics (Faithfulness, ResponseRelevancy, etc.). Metrics call LLM API during evaluation.