Install & Compatibility
Where this runs
tested against v0.3.35 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
Llama
✓ from llama_cpp import Llama
This is the primary high-level class for interacting with loaded models.
LlamaGrammar
✓ from llama_cpp import LlamaGrammar
Used for constrained text generation with GBNF grammars (e.g., JSON output).
LlamaHFTokenizer
✓ from llama_cpp import LlamaHFTokenizer
Required for certain models (like Functionary v2) where HuggingFace tokenizers are needed due to discrepancies with `llama.cpp`'s default tokenizer.
This quickstart demonstrates how to load a GGUF model and generate text using the high-level `Llama` class. It highlights important parameters like `model_path`, `n_ctx` for context size, and `n_gpu_layers` for GPU offloading. Ensure your model is in the GGUF format.
import os
from llama_cpp import Llama
# Ensure you have a GGUF model downloaded, e.g., to a 'models' directory.
# Example: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf
model_path = os.environ.get('LLAMA_MODEL_PATH', './models/llama-2-7b-chat.Q4_K_M.gguf')
# Initialize the Llama model
# Set n_gpu_layers to a value > 0 for GPU acceleration (requires GPU install config)
llm = Llama(
model_path=model_path,
n_ctx=2048, # Context window size
n_gpu_layers=0, # Set to > 0 for GPU, -1 to offload all layers if GPU is available
verbose=False # Suppress llama.cpp verbose output
)
# Generate a completion
prompt = "Q: Name the planets in the solar system? A: "
output = llm(prompt, max_tokens=128, stop=["Q:", "\n"], echo=True)
print(output["choices"][0]["text"])
llama-cpp-python --version
Debug
Known issues
breakingInstallation with GPU acceleration (CUDA, Metal, ROCm) often requires setting specific `CMAKE_ARGS` environment variables or using pre-built wheels from custom index URLs, along with correctly configured C++ compilers and GPU toolkits. Default `pip install` typically provides CPU-only support.fixRefer to the official documentation or GitHub README for specific `CMAKE_ARGS` and installation instructions for your desired backend (e.g., `CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python`). Ensure all necessary C++ compilers and GPU toolkits are correctly installed and configured in your system's PATH.
affects: All versions
gotchaOn Apple Silicon (M1/M2) Macs, `llama-cpp-python` can default to building an x86 version if an ARM64 Python interpreter is not used. This results in significantly slower performance.fixEnsure you are using an ARM64 version of Python (e.g., from Miniforge). You might need to explicitly specify `CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DGGML_METAL=on"` during installation.
affects: All versions
breakingThe library closely tracks upstream `llama.cpp` development, which can introduce breaking changes to the underlying C API that may propagate to the Python bindings. Notable past changes include the transition to GGUF model format and changes in KV cache management functions.fixAlways check the changelog when upgrading `llama-cpp-python`. If upgrading, use `--upgrade --force-reinstall --no-cache-dir` to ensure a clean rebuild from source. Be prepared for potential API adjustments, particularly in low-level usage or integrations with other libraries.
affects: Frequent, especially minor version bumps
gotchaLLM models require specific chat formats (e.g., 'llama-2', 'chatml') for proper conversational interaction. Using the wrong format can lead to 'weird' or unparsable responses, especially in chat completion APIs.fixWhen initializing `Llama` for chat, explicitly set the `chat_format` parameter (e.g., `llm = Llama(..., chat_format="llama-2")`). Consult the model card or `llama-cpp-python` documentation for the correct format for your chosen model.
affects: All versions
gotchaTo generate embeddings, you must explicitly enable them by passing `embedding=True` to the `Llama` constructor when initializing the model.fixInitialize your model with `llm = Llama(model_path="...", embedding=True)`. Then use `llm.create_embedding(...)`.
affects: All versions
Errors
Common errors & fixes
ERROR: Failed building wheel for llama-cpp-python
This error typically indicates that the C++ components of `llama-cpp-python` could not be compiled successfully during `pip install`. This is often due to missing build tools (like a C++ compiler or CMake) on your system, or an incompatible environment.
fixInstall the necessary build tools for your operating system: On Windows, install 'Desktop development with C++' from Visual Studio Installer (including MSVC v14.x C++ build tools, Windows SDK, and C++ CMake tools). On Linux, install `build-essential`, `g++`, and `clang` (e.g., `sudo apt-get install build-essential g++ clang`). Ensure CMake is installed and in your PATH. After installing tools, try `pip install llama-cpp-python` again, potentially with `--no-cache-dir --force-reinstall`.
RuntimeError: Failed to load shared library '.../llama.dll': Could not find module 'llama.dll' (or one of its dependencies)
After installation, this runtime error occurs when Python cannot locate the compiled `llama.dll` (on Windows) or `libllama.so` (on Linux/macOS) shared library, or one of its dependencies. This can be due to an incomplete build, issues with environment variables, or running in an environment where the shared library's path isn't correctly resolved.
fixEnsure that all required Visual C++ Redistributables (on Windows) are installed. If using Anaconda/Conda, prefer `conda install llama-cpp-python -c conda-forge` as it often handles dependencies better. For Linux, ensure `libllama.so` is discoverable by setting `LD_LIBRARY_PATH` or `LLAMA_CPP_LIB` environment variables to the path containing the `libllama.so` file (e.g., `export LLAMA_CPP_LIB=/path/to/site-packages/llama_cpp/lib/libllama.so`).
ModuleNotFoundError: No module named 'llama_cpp'
This error means the Python interpreter cannot find the `llama_cpp` package. This typically happens if `llama-cpp-python` was not installed correctly, installed in a different Python environment, or if the current environment's `PYTHONPATH` does not include the installation location.
fixVerify that `llama-cpp-python` is installed in your active Python environment by running `pip list` or `conda list`. If it's missing, reinstall it: `pip install llama-cpp-python`. If you are using a virtual environment or an IDE like VS Code with Jupyter, ensure the correct Python interpreter (where the package is installed) is selected. If issues persist, try reactivating your virtual environment or creating a fresh one.
llama-cpp-python not using GPU / No CUDA toolset found / cuBLAS not found
This indicates that `llama-cpp-python` failed to build with GPU (CUDA) support or is not detecting/utilizing the GPU at runtime. Common reasons include an incorrectly installed CUDA Toolkit, missing `CMAKE_ARGS` during `pip install`, or not setting `n_gpu_layers` when initializing the `Llama` model.
fixEnsure NVIDIA CUDA Toolkit (and cuDNN if applicable) is correctly installed and its binaries are in your system's PATH before installation. Then, uninstall `llama-cpp-python` and reinstall it with CUDA flags: `CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install --upgrade --force-reinstall --no-cache-dir llama-cpp-python`. For specific CUDA versions, you might need to use extra index URLs (e.g., `--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121`). When loading the model, set `n_gpu_layers` to a positive integer (e.g., `n_gpu_layers=-1` to offload all layers).
error: Microsoft Visual C++ 14.0 or greater is required.
On Windows, compiling `llama-cpp-python` (or any Python package with C++ extensions) requires the Microsoft Visual C++ build tools, which are not installed by default with Python.
fixInstall the 'Desktop development with C++' workload using the Visual Studio Installer (available with free Community edition), or install the standalone Build Tools for Visual Studio 2019/2022.
Upgrade
Version history
0.3.35latest on PyPI · released Aug 17, 2026
Audit
Dependencies
C/C++ compilerrequiredRequired to build `llama.cpp` from source, which is the default installation method and often necessary for GPU acceleration. On Windows, Visual Studio with 'Desktop development with C++' is typically needed; on Linux, `gcc` and `g++`; on macOS, Xcode Command Line Tools.
CUDA ToolkitoptionalRequired for NVIDIA GPU acceleration (cuBLAS backend).
ROCmoptionalRequired for AMD GPU acceleration (hipBLAS backend).