Install & Compatibility
Where this runs
tested against v0.13.3 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.10
✕ build_error
✓ 79.85s
py 3.11
✕ build_error
✓ 74.35s
py 3.12
✕ build_error
✓ 66.25s
py 3.13
✕ build_error
✓ 63.4s
py 3.9
✕ build_error
✕ timeout
7578MB installed
● package 7578MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
TensorDict
✓ from tensordict import TensorDict
TensorDict is the core data structure for TorchRL, not directly in `torchrl` namespace.
GymEnv
✓ from torchrl.envs import GymEnv
MLP
✓ from torchrl.modules import MLP
QValueActor
✓ from torchrl.modules import QValueActor
PPOLoss
✓ from torchrl.objectives import PPOLoss
SyncDataCollector
✓ from torchrl.collectors import SyncDataCollector
This quickstart demonstrates how to create a simple Gym environment, define a Q-value policy using an MLP, and collect a trajectory with a specified maximum number of steps. The collected data is stored in a TensorDict.
import torch
from torchrl.envs import GymEnv
from torchrl.modules import MLP, QValueActor
from tensordict import TensorDict
# 1. Define the environment
env = GymEnv("CartPole-v1")
# 2. Create the policy (Q-value actor with an MLP backbone)
actor = QValueActor(
MLP(
in_features=env.observation_spec["observation"].shape[-1],
out_features=env.action_spec.shape[-1] if env.action_spec.shape else 2,
num_cells=[64, 64],
),
in_keys=["observation"],
spec=env.action_spec,
)
# 3. Collect a trajectory
rollout = env.rollout(max_steps=200, policy=actor)
# Print collected info
print(f"Collected {rollout.shape[0]} steps, total reward: {rollout['next', 'reward'].sum().item():.0f}")
print(f"Rollout keys: {rollout.keys()}")
print(f"Example observation shape: {rollout['observation'].shape}")
env.close()
Debug
Known issues
breakingIn TorchRL v0.11.0, the collector codebase underwent a major refactoring. Existing implementations of collectors, especially `SyncDataCollector`, `MultiSyncDataCollector`, and `aSyncDataCollector`, may require updates to align with the new modular structure.fixReview the `torchrl.collectors` package documentation and examples for the restructured API and adjust collector instantiation and usage accordingly. The 5000+ line `collectors.py` was split into focused modules.
affects: >=0.11.0
breakingTorchRL v0.11.0 removed several deprecated features, replacing previous warnings with errors. This includes `KLRewardTransform` (use `torchrl.envs.llm.KLRewardTransform`), `LogReward` and `Recorder` (use `LogScalar` and `LogValidationReward`), and `unbatched_*_spec` properties from `VmasWrapper`/`VmasEnv` (use `full_*_spec_unbatched`).fixUpdate code to use the new, recommended paths and classes for previously deprecated features. Refer to the v0.11 release notes for a complete list.
affects: >=0.11.0
gotchaUsing TorchRL with PyTorch versions older than 2.0 (e.g., PyTorch 1.12 with Python 3.7) can lead to `ImportError: undefined symbol` errors when installing the stable `torchrl` package.fixEnsure you install or upgrade to the latest stable PyTorch release *before* installing TorchRL. If using an older PyTorch is necessary, you might need to install `functorch` compatible with your PyTorch version and then install `torchrl` from source.
affects: <2.0 (PyTorch) with stable `torchrl`
gotchaIn TorchRL versions prior to 0.7.2, a critical issue existed where incorrect device settings in `ParallelEnv` could prevent tensors in buffers from being properly cloned, causing rollouts to return the same tensor instances across steps and potentially leading to incorrect behavior.fixUpgrade TorchRL to version 0.7.2 or newer. When using `ParallelEnv` or `BatchedEnv` with different devices for sub-environments and the batched environment, ensure careful device management to prevent data corruption or unexpected behavior. Data will be automatically cast to the appropriate device during collection.
affects: <0.7.2
deprecatedThe `PPOLoss` class in TorchRL v0.11.0 issues a warning regarding the use of `critic_network` directly and suggests using the `critic_coeff` argument instead for better control over the critic's contribution to the loss.fixWhen initializing `PPOLoss`, explicitly pass `critic_coeff` instead of relying on `critic_network` for its default coefficient. For example, pass `critic_coeff=1.0` if a critic network is provided.
affects: >=0.11.0
Errors
Common errors & fixes
ModuleNotFoundError: cannot import name 'TensorDictReplayBuffer' from 'torchrl.data'
The class `TensorDictReplayBuffer` (and other replay buffer classes) was moved from the top-level `torchrl.data` module to `torchrl.data.replay_buffers` in recent versions of torchrl.
fixUpdate the import statement to `from torchrl.data.replay_buffers import TensorDictReplayBuffer`.
AttributeError: 'MyPolicyModule' object has no attribute 'get_dist'
A custom policy module is missing the `get_dist` method, which is expected by torchrl components like `ActorCritic` or `ProbabilisticActor` to sample actions or compute log-probabilities.
fixEnsure the custom policy inherits from `torchrl.modules.ProbabilisticActor` or implements a `get_dist` method that returns a `torch.distributions.Distribution` object.
RuntimeError: Expected all tensors to be on the same device, but found at least two devices
Tensors used in an operation (e.g., passing input to a model, or combining with another tensor) are located on different devices (e.g., some on CPU, some on GPU), which PyTorch operations do not allow.
fixBefore performing the operation, ensure all relevant tensors and models are explicitly moved to the same device using `.to(device)`.
RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.
This error commonly occurs on Windows or macOS when using multiprocessing (e.g., with `ParallelEnv` or `AsyncDataCollector`) if the main execution logic is not guarded by `if __name__ == '__main__':`, or if the environment factory is not pickleable.
fixWrap the main script's execution code within `if __name__ == '__main__':` and ensure any environment factory functions are defined at the top-level of the module to be pickleable.
Upgrade
Version history
0.13.3latest on PyPI · released Jul 14, 2026
Audit
Dependencies
pytorchrequiredTorchRL is built on PyTorch and requires a compatible version.
tensordictrequiredTorchRL's core data structure is TensorDict, requiring the tensordict library.
gymnasiumoptionalRequired for using environments from the Gymnasium library.
hydra-coreoptionalRequired for the experimental command-line training interface (installed with `torchrl[utils]`).
omegaconfoptionalRequired for the experimental command-line training interface (installed with `torchrl[utils]`).