Install & Compatibility
Where this runs
tested against v2.28.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.10
✕ build_error
✓ 77.8s
py 3.11
✕ build_error
✓ 72.4s
py 3.12
✕ build_error
✓ 60.5s
py 3.13
✕ build_error
✓ 59.7s
py 3.9
✕ build_error
✕ timeout
4813MB installed
● package 4813MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
TransformerWrapper
✓ from x_transformers import TransformerWrapper
Primary class for building encoder-only or decoder-only models.
Decoder
✓ from x_transformers import Decoder
Component for constructing decoder (GPT-like) attention layers.
Encoder
✓ from x_transformers import Encoder
Component for constructing encoder (BERT-like) attention layers.
XTransformer
✓ from x_transformers import XTransformer
Class for building full encoder-decoder transformer models.
This quickstart demonstrates how to set up a basic decoder-only (GPT-like) transformer model using `TransformerWrapper` and `Decoder`. It initializes a model with a specified vocabulary size, sequence length, and decoder attention layer configuration, then runs a sample forward pass.
import torch
from x_transformers import TransformerWrapper, Decoder
import os
# Example for a decoder-only (GPT-like) model
# Note: .cuda() calls are for GPU usage; remove if running on CPU only.
model = TransformerWrapper(
num_tokens = 20000,
max_seq_len = 1024,
attn_layers = Decoder(
dim = 512,
depth = 12,
heads = 8
)
)
# Move model to GPU if available
if torch.cuda.is_available():
model = model.cuda()
x = torch.randint(0, 256, (1, 1024)).cuda()
else:
x = torch.randint(0, 256, (1, 1024))
output = model(x)
print(f"Output shape: {output.shape}")
Debug
Known issues
gotchaThe library is in 'Beta' development status (Development Status :: 4 - Beta) and often integrates experimental features from recent research papers. This means API stability and feature behavior may change rapidly between versions, and some features might be experimental or less thoroughly tested than in more mature libraries.fixRefer to the GitHub README and recent release notes for the latest API and feature details. Be prepared for potential breaking changes when updating.
affects: All versions
gotchaWhen configuring embedding normalization, it's recommended to use either `l2norm_embed` or `post_emb_norm`, but not both simultaneously, as they are designed to serve similar purposes and using both might lead to redundant or conflicting behavior.fixSet only one of `l2norm_embed=True` or `post_emb_norm=True` during model initialization based on your specific requirements and experimental findings.
affects: All versions
gotchaSome advanced or experimental features, such as 'Rezero Is All You Need' (as noted in an older GitHub issue), might exhibit stability or convergence issues (e.g., producing NaN values) depending on the specific use case, dataset, and hyperparameter tuning.fixExercise caution and thorough validation when employing newer or experimental features. Start with default or recommended configurations and systematically test their stability and performance on your specific task. Consult GitHub issues for community experiences.
affects: Potentially all versions incorporating experimental features.
Errors
Common errors & fixes
TypeError: __init__() got an unexpected keyword argument 'absolute_pos_emb'
The `absolute_pos_emb` argument, previously used for absolute positional embeddings, has been deprecated or removed from `XTransformer` or `TransformerWrapper` in newer versions.
fixRemove the `absolute_pos_emb` argument from the `XTransformer` or `TransformerWrapper` constructor call.
ModuleNotFoundError: No module named 'einops'
The `einops` library, a fundamental dependency for tensor manipulations used by `x-transformers`, is not installed in the current Python environment.
fixInstall the `einops` library using pip: `pip install einops`.
RuntimeError: Expected all tensors to be on the same device as the model.
Input tensors (e.g., `x`, `mask`) are on a different device (e.g., CPU) than the `x-transformers` model, which has been moved to a GPU (or vice-versa).
fixEnsure all input tensors are explicitly moved to the same device as the model. Example: `device = torch.device("cuda" if torch.cuda.is_available() else "cpu"); model.to(device); x = x.to(device); mask = mask.to(device)`. ValueError: rotary_pos_emb must be True when specifying rotary_seq_pos_interpolation_factor
The `rotary_seq_pos_interpolation_factor` parameter, which enables sequence length interpolation for Rotary Positional Embeddings, requires `rotary_pos_emb` to be explicitly set to `True`.
fixSet `rotary_pos_emb=True` in the model constructor when `rotary_seq_pos_interpolation_factor` is also provided, e.g., `XTransformer(..., rotary_pos_emb=True, rotary_seq_pos_interpolation_factor=0.5, ...)`.
Upgrade
Version history
2.28.2latest on PyPI · released Aug 28, 2026
Audit
Dependencies
No dependency data recorded yet.