trl

library1.12.0pypypi✓ verified 26d ago

Hugging Face library for post-training LLMs: SFT, DPO, GRPO, PPO, reward modeling. Current version is 0.29.1 (Mar 2026). Requires Python >=3.10. Extremely high API churn — major parameter renames across versions. tokenizer= renamed to processing_class= in 0.12. Still pre-1.0 (Development Status: Pre-Alpha).

llm-agents ai-ml

pip install trl

github ↗package ↗

INSTALL

IMPORT

SIG · TRL

trl

llm-agentspythonv1.12.0

Install

—

Import

—

Disk

—

Pass rate

0/ 10

Env Coverage0 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.12.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✕ build_error

2/4 runs

py 3.11

✕ build_error

2/4 runs

py 3.12

✕ build_error

2/4 runs

py 3.13

✕ build_error

3/4 runs

py 3.9

✕ build_error

✕ timeout

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

SFTConfig

✓ from trl import SFTConfig

✗ from trl import SFTConfig

SFTTrainer

✓ from trl import SFTTrainer

✗ from trl import SFTTrainer

SFT then DPO pipeline. Use SFTConfig/DPOConfig for all training args.

from datasets import load_dataset from trl import SFTConfig, SFTTrainer # SFT — minimal setup trainer = SFTTrainer( model='Qwen/Qwen2.5-0.5B', args=SFTConfig(output_dir='sft_output', num_train_epochs=1), train_dataset=load_dataset('trl-lib/Capybara', split='train'), ) trainer.train() # DPO — after SFT from trl import DPOConfig, DPOTrainer from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained('sft_output') tokenizer = AutoTokenizer.from_pretrained('sft_output') trainer = DPOTrainer( model=model, args=DPOConfig(output_dir='dpo_output', beta=0.1), train_dataset=load_dataset('trl-lib/ultrafeedback_binarized', split='train'), processing_class=tokenizer, ) trainer.train()

Debug

Known issues

breakingtokenizer= parameter deprecated since TRL 0.12 and being removed. All trainers (SFTTrainer, DPOTrainer, etc.) now use processing_class= instead. Passing tokenizer= raises DeprecationWarning now, TypeError in future release.

fix

Replace tokenizer=tokenizer with processing_class=tokenizer in all trainer constructors.

affects: >= 0.12

breakingTraining args like max_seq_length, packing, dataset_text_field have moved from SFTTrainer constructor to SFTConfig. Passing them directly to SFTTrainer raises TypeError in recent versions.

fix

Use SFTConfig(max_length=512, ...) and pass as args=SFTConfig(...) to SFTTrainer.

affects: >= 0.10

breakingTRL has extremely high API churn — major parameter renames, removals, and behavioral changes in almost every minor release. Code written for 0.8 likely fails on 0.15+. Pin versions in production.

fix

Pin exact TRL version in requirements. Read the GitHub releases page before upgrading: https://github.com/huggingface/trl/releases

affects: all

gotchaDPOTrainer with PEFT does not keep a separate reference model in memory — it disables the adapter to recover reference behavior. Passing ref_model= with a PEFT model wastes memory and raises a warning about sync_ref_model incompatibility.

fix

With PEFT models, omit ref_model entirely. DPOTrainer handles reference behavior automatically via adapter disable/enable.

affects: all

gotchaGRPOTrainer reward_funcs must be a list of callables, not a single callable. Passing a single function raises TypeError.

fix

Wrap single reward function in a list: reward_funcs=[my_reward_fn].

affects: all

gotchaSFTTrainer dataset format: conversational datasets (with 'messages' column) are handled differently from text datasets (with 'text' column). Mixing formats or using wrong column name causes silent empty-loss training.

fix

For chat format datasets use a 'messages' column with OpenAI-style message dicts. For plain text use a 'text' column. Check dataset_text_field in SFTConfig if using a custom column name.

affects: all

breakingTRL's core dependency, `torch`, often lacks pre-built wheels for less common Python versions (e.g., Python 3.13) or non-glibc environments (e.g., Alpine Linux). This results in `pip` installation failures because `torch` cannot be resolved.

fix

Ensure your Python environment (version and OS distribution) has readily available pre-built `torch` wheels. Consider using a Python version officially supported by PyTorch (e.g., Python 3.10, 3.11) on a widely supported base OS (e.g., Debian/Ubuntu). If using Alpine, you may need to build `torch` from source or switch to a glibc-based image.

affects: all

Upgrade

Version history

1.12.0latest on PyPI · released Aug 26, 2026

Audit

Dependencies

transformersrequiredRequired. Must be compatible version — TRL pins minimum transformers versions per release.

acceleraterequiredRequired. Installed automatically.

peftoptionalRequired for LoRA/QLoRA training. Install separately or use trl[peft].

datasetsoptionalRequired in practice for dataset loading. Not installed automatically.

Agent activity

62 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/huggingface/trl ↗

packagehttps://pypi.org/project/trl/ ↗