auto-round

library0.13.0pypypi✓ verified 84d ago

AutoRound is an advanced weight-only quantization algorithm for large language models (LLMs), providing up to 4-bit quantization with minimal accuracy loss. Current version 0.13.0 supports various Intel and AMD GPUs, as well as CPUs. The package is under active development by Intel.

ai-ml

pip install auto-round

github ↗homepage ↗

INSTALL

IMPORT

SIG · AUTO-ROUND

auto-round

ai-mlpythonv0.13.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

AutoRound

✓ from auto_round import AutoRound

✗ from autoround import AutoRound

Package uses underscore in module name: auto_round, not autoround.

AutoRoundConfig

✓ from auto_round import AutoRoundConfig

✗ from auto_round.config import AutoRoundConfig

Config class is directly importable from auto_round.

Quantize a Hugging Face causal LM model to 4-bit using AutoRound with default settings.

from transformers import AutoModelForCausalLM, AutoTokenizer from auto_round import AutoRound model_name = "facebook/opt-125m" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name) # Quantize with AutoRound quantizer = AutoRound( model, tokenizer, bits=4, group_size=128, sym=False, iters=200, lr=0.005, minmax_lr=0.005, enable_minmax=False, is_mse=True, seqlen=2048, nsamples=512, dataset="wikitext2" ) quantized_model = quantizer.quantize() quantized_model.save_pretrained("./int4_model") print("Quantization complete.")

Debug

Known issues

breakingThe 'quantize' method now returns the quantized model directly (since v0.10.0). Previously it returned a QuantBlock or None.

fix

Update code to use returned model: model = quantizer.quantize()

affects: <0.10.0

gotchaAutoRound requires the model to be on a GPU by default. If running on CPU, you must explicitly set device='cpu' in AutoRound constructor.

fix

Initialize AutoRound(model, tokenizer, device='cpu', ...)

affects: all

deprecatedThe parameter 'enable_minmax' is deprecated as of v0.12.0 and will be removed in future versions. Use 'is_mse' instead.

fix

Replace enable_minmax with is_mse=True/False.

affects: >=0.12.0

gotchaThe dataset argument expects a string like 'wikitext2' or a list of strings. Using an unsupported dataset name may cause a silent fallback to random data.

fix

Check supported datasets in documentation or use 'wikitext2' for standard cases.

affects: all

Errors

Common errors & fixes

ImportError: cannot import name 'AutoRound' from 'auto_round'

AutoRound was renamed from AutoRoundQuantizer in v0.5.0. Older code may still use the old name.

fix

Use from auto_round import AutoRound (or AutoRoundConfig). Check your installed version.

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0

Model and tokenizer are on different devices. AutoRound expects model on GPU by default.

fix

Move model to GPU: model = model.to('cuda') before quantizer creation, or set device='cpu'.

AttributeError: 'NoneType' object has no attribute 'save_pretrained'

The 'quantize' method returned None because quantizer.quantize() was called on an already quantized model or due to internal error.

fix

Ensure the model is not already quantized and that quantizer parameters are valid. Check the return value is not None.

Upgrade

Version history

0.13.0latest on PyPI · released May 31, 2026

Audit

Dependencies

torchrequiredRequired for model quantization and inference.

transformersrequiredRequired for loading Hugging Face models.

Agent activity

25 hits · last 30 days

node

Amazon

OpenAI (training)

Resources

githubhttps://github.com/intel/auto-round ↗

homepagehttps://github.com/intel/auto-round ↗