auto-gptq

library0.7.1pypypiunverified

AutoGPTQ is an easy-to-use LLMs quantization package based on the GPTQ algorithm. It provides user-friendly APIs for quantizing and running large language models with reduced memory usage. Current version 0.7.1 supports loading sharded quantized checkpoints and Gemma models. Release cadence is irregular, with major features in point releases.

ai-ml llm-agents

pip install auto-gptq

github ↗package ↗

INSTALL

IMPORT

SIG · AUTO-GPTQ

auto-gptq

ai-mlpythonv0.7.1

Install

82.5s avg

Import

—

Disk

5427MB

Pass rate

1/ 10

Env Coverage1 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.7.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✕ build_error

✕ timeout

py 3.11

✕ build_error

✓ 82.45s

py 3.12

✕ build_error

py 3.13

✕ build_error

py 3.9

✕ build_error

✕ timeout

5427MB installed

● package 5427MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

AutoGPTQForCausalLM

✓ from auto_gptq import AutoGPTQForCausalLM

✗ from auto_gptq.modeling import AutoGPTQForCausalLM

AutoGPTQForCausalLM is exposed at package level, not in a submodule.

BaseQuantizeConfig

✓ from auto_gptq import BaseQuantizeConfig

exllama_set_max_input_length

✓ from auto_gptq import exllama_set_max_input_length

Quickstart for quantizing and running a model (quantization not shown for simplicity, but the config is set).

import torch from transformers import AutoTokenizer from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig model_id = "facebook/opt-125m" tokenizer = AutoTokenizer.from_pretrained(model_id) quantize_config = BaseQuantizeConfig( bits=4, group_size=128, desc_act=False, ) model = AutoGPTQForCausalLM.from_pretrained( model_id, quantize_config=quantize_config, ) text = "AutoGPTQ is" inputs = tokenizer(text, return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=20) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Debug

Known issues

gotchaThe exllama kernel is the default backend for inference. If you do not install the optional exllama dependency, the package may fall back to a slower backend or raise an error. Install with `pip install auto-gptq[exllama]`.

fix

Install auto-gptq with exllama extra: `pip install auto-gptq[exllama]`

affects: >=0.5.0

breakingIn v0.3.2, the `save_dir` argument was removed from `from_quantized`. Now only `model_name_or_path` is supported.

fix

Use `model_name_or_path` instead of `save_dir` when loading quantized models.

affects: 0.3.2+

deprecatedThe `use_triton` parameter in `from_quantized` is deprecated since v0.5.0. Triton backend is no longer supported; use exllama or cuda backends.

fix

Remove `use_triton` argument. The backend is automatically selected (exllama preferred).

affects: >=0.5.0

gotchaWhen using the exllama backend, the maximum input length defaults to 2048. For longer sequences, call `exllama_set_max_input_length` before generation. Otherwise, you may get an error like 'CUDA out of memory' or silent failures.

fix

Call `from auto_gptq import exllama_set_max_input_length; model = exllama_set_max_input_length(model, 4096)` after loading.

affects: 0.4.2 - 0.7.1

gotchaQuantization with `desc_act=True` and `group_size` may not be supported with all kernels. The cuda kernel before v0.3.2 had a bug when using both. Ensure you use v0.3.2+ if you need that combination.

fix

Upgrade to v0.3.2 or later.

affects: <0.3.2

Upgrade

Version history

0.7.1latest on PyPI · released Mar 1, 2024

Audit

Dependencies

torchrequiredCore dependency for tensor operations and GPU support. AutoGPTQ requires PyTorch >=2.0.

transformersrequiredHugging Face Transformers for model architectures and tokenizers.

accelerateoptionalUsed for device mapping and large model support.

datasetsoptionalUsed for loading calibration datasets during quantization.

Agent activity

25 hits · last 30 days

node

Perplexity

Resources

githubhttps://github.com/PanQiWei/AutoGPTQ ↗

packagehttps://pypi.org/project/auto-gptq/ ↗