Hugging Face library to run PyTorch training across any distributed configuration with minimal code changes. Current version is 1.13.0 (Mar 2026). Requires Python >=3.10. Core pattern: Accelerator() + accelerator.prepare() + accelerator.backward(). Must run accelerate config before first use.
Install & Compatibility
Where this runs
tested against v0.0.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.13
✕ build_error
4/5 runs
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
Accelerator
✓ from accelerate import Accelerator
def training_function():
# Accelerator MUST be initialized inside the training function for notebook_launcher
accelerator = Accelerator(mixed_precision='fp16')
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
for batch in dataloader:
optimizer.zero_grad()
loss = model(batch)
accelerator.backward(loss) # NOT loss.backward()
optimizer.step()
✗ # Module-level Accelerator initialization breaks notebook_launcher multi-GPU
accelerator = Accelerator() # at top of notebook cell
def training_function():
# ValueError: Accelerator should only be initialized inside your training function
For notebook_launcher (Colab/Jupyter multi-GPU), Accelerator() must be initialized INSIDE the training function, never at module/notebook level.
accelerator.backward
✓ loss = criterion(outputs, targets)
accelerator.backward(loss)
✗ loss = criterion(outputs, targets)
loss.backward() # bypasses mixed precision scaling and gradient accumulation handling
Always use accelerator.backward(loss) instead of loss.backward(). Direct loss.backward() bypasses Accelerate's mixed precision gradient scaling and gradient accumulation logic.
Core Accelerate pattern. Run with: accelerate launch train.py
from accelerate import Accelerator
import torch
import torch.nn as nn
def train():
accelerator = Accelerator(mixed_precision='bf16')
model = nn.Linear(10, 1)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
dataloader = ... # your DataLoader
# prepare() handles device placement and distributed wrapping
model, optimizer, dataloader = accelerator.prepare(
model, optimizer, dataloader
)
model.train()
for batch in dataloader:
optimizer.zero_grad()
outputs = model(batch['input'])
loss = nn.functional.mse_loss(outputs, batch['target'])
accelerator.backward(loss) # not loss.backward()
optimizer.step()
# Save on main process only
accelerator.wait_for_everyone()
if accelerator.is_main_process:
accelerator.save_model(model, 'output/')
accelerate --version
Debug
Known issues
breakingaccelerate config must be run before first use. Without a config file, Accelerate falls back to single-process CPU mode silently — multi-GPU training simply won't use multiple GPUs.fixRun accelerate config once after install, or programmatically: from accelerate.utils import write_basic_config; write_basic_config(). For CI: set ACCELERATE_CONFIG_FILE env var pointing to a pre-built config.
affects: all
breakingPython 3.9 support dropped in 1.13.0. Accelerate now requires Python >=3.10.fixUpgrade Python to 3.10+. Pin accelerate<1.13.0 for Python 3.9 environments.
affects: >= 1.13.0
breakingAccelerator() initialized outside the training function raises ValueError when using notebook_launcher for multi-GPU. Silently falls back to 1 GPU without error if no notebook_launcher is used.fixAlways initialize Accelerator() inside the training function passed to notebook_launcher. Never create it at notebook cell level or module level when using multi-GPU in notebooks.
affects: all
breakingaccelerator.load_state() fails with PyTorch 2.6+ due to torch.load weights_only=True default flip. Optimizer states with custom objects (omegaconf.ListConfig, etc.) raise UnpicklingError.fixUse torch.serialization.add_safe_globals([ListConfig]) to allowlist custom types, or pass weights_only=False to the underlying load call if the checkpoint source is trusted.
affects: >= 1.6.0 with PyTorch >= 2.6
breakingDeepSpeed integration: only one nn.Module per Accelerator instance is supported. Passing multiple models to accelerator.prepare() with DeepSpeed raises AssertionError.fixWith DeepSpeed, create a separate Accelerator instance per model, or merge models before wrapping.
affects: all
gotchaaccelerate launch ignores Python script argument ordering. Flags intended for the script must come after --, otherwise they are parsed as accelerate launch flags.fixUse: accelerate launch script.py --my-arg value. If ambiguous: accelerate launch -- script.py --my-arg value.
affects: all
gotchaloss.backward() instead of accelerator.backward(loss) silently bypasses mixed precision gradient scaling. Training proceeds but gradients are wrong under fp16/bf16 — numerical instability or NaN loss.fixReplace all loss.backward() calls with accelerator.backward(loss) throughout the training loop.
affects: all
breakingInstallation of core dependencies like numpy fails due to missing C compilers in the environment, particularly common in minimal Docker images (e.g., Alpine). This prevents packages requiring compilation from being built from source.fixEnsure that build-essential tools, including a C compiler (e.g., gcc, g++), are installed in your environment before attempting to install Python packages that require compilation. For Alpine-based images, this typically involves `apk add build-base python3-dev`.
affects: all
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'accelerate'
The 'accelerate' library is not installed or not accessible in the current Python environment.
fixInstall the library using 'pip install accelerate'.
bash: accelerate: command not found
The 'accelerate' command-line tool is not found, possibly due to installation issues or PATH misconfiguration.
fixEnsure 'accelerate' is installed and accessible by checking the installation path and verifying the PATH environment variable.
ImportError: cannot import name 'partialstate' from 'accelerate'
Attempting to import a non-existent 'partialstate' module from the 'accelerate' library.
fixVerify the correct module name and import statement; refer to the 'accelerate' documentation for accurate usage.
AttributeError: module 'openai' has no attribute 'ChatCompletion'
The 'openai' module does not have an attribute named 'ChatCompletion', possibly due to an outdated version or incorrect import.
fixUpdate the 'openai' library to the latest version and check the documentation for the correct usage of 'ChatCompletion'.
TypeError: 'NoneType' object is not iterable
An operation is attempting to iterate over a 'None' object, indicating that a variable expected to be iterable is 'None'.
fixEnsure that the variable is properly initialized and assigned an iterable value before iteration.
Audit
Dependencies
torch>=1.10.0requiredRequired. Not installed automatically — install PyTorch separately first.
huggingface-hubrequiredRequired. Installed automatically.
safetensorsrequiredRequired. Installed automatically.
deepspeedoptionalOptional. Required for DeepSpeed ZeRO integration.