Registry / ai-ml / flashinfer-cubin

flashinfer-cubin

library0.6.13pypypi✓ verified 23d ago

FlashInfer-cubin provides pre-compiled kernel binaries for FlashInfer, supporting a wide range of GPU architectures. This optional package for `flashinfer-python` eliminates JIT compilation and downloading overhead at runtime, leading to faster initialization and enabling offline usage. The FlashInfer project focuses on delivering high-performance LLM GPU kernels for serving and inference, maintaining an active development cycle with frequent nightly builds and regular patch releases.

ai-ml llm-agents

pip install flashinfer-python flashinfer-cubin

github ↗package ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · FLASHINFER-CUBIN

flashinfer-cubin

ai-mlpythonv0.6.13

Install

57.6s avg

Import

—

Disk

7142MB

Pass rate

4/ 10

Env Coverage4 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.6.13 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

1/2 runs

✓ 60.9s

py 3.11

1/2 runs

✓ 57.4s

py 3.12

1/2 runs

✓ 56.2s

py 3.13

1/2 runs

✓ 55.9s

py 3.9

1/2 runs

7142MB installed

● package 7142MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

flashinfer-cubin

✓ N/A

The `flashinfer-cubin` package is not designed for direct import by end-users. It provides pre-compiled CUDA kernel files (`.cubin` files) that the `flashinfer-python` library loads and utilizes internally for optimized performance. Users interact with the `flashinfer` library directly.

This quickstart demonstrates the usage of the core `flashinfer` library for a single-request decode attention operation. When `flashinfer-cubin` is installed, it transparently provides pre-compiled CUDA kernels to `flashinfer-python`, significantly speeding up operations like this by avoiding runtime compilation overhead.

import torch import flashinfer # Example of FlashInfer's single-request decode attention # (flashinfer-cubin provides the underlying kernels for optimal performance) kv_len = 2048 num_kv_heads = 32 head_dim = 128 q = torch.randn(1, head_dim, dtype=torch.float16, device='cuda') k_tensor = torch.randn(kv_len, num_kv_heads, head_dim, dtype=torch.float16, device='cuda') v_tensor = torch.randn(kv_len, num_kv_heads, head_dim, dtype=torch.float16, device='cuda') # Prepare FlashInfer attention wrapper wrapper = flashinfer.to_flashinfer_decode_wrapper( kv_len, num_kv_heads, head_dim, 0 # page_size, use 0 for single request ) # Allocate KV cache k_cache, v_cache = wrapper.alloc_kv_cache(torch.float16, device='cuda') # Append K/V to cache (simulates historical tokens) wrapper.begin_forward(k_cache, v_cache) wrapper.end_forward() # Perform decode attention output = flashinfer.single_decode_with_kv_cache( q, k_cache, v_cache, wrapper.kv_layout, wrapper.kv_indices, wrapper.kv_indptr, wrapper.last_page_len, num_kv_heads, num_kv_heads, # num_query_heads == num_kv_heads for single decode head_dim, True # casual ) print(output.shape)

Debug

Known issues

breakingFlashInfer, and by extension `flashinfer-cubin`, has strict compatibility requirements for CUDA and PyTorch versions. Incompatible versions can lead to runtime failures due to mismatches in precompiled kernels (e.g., CUDA 12 vs 13 toolkits) or Python library dependencies.

fix

Always check the official FlashInfer documentation for supported PyTorch and CUDA versions. Use `flashinfer show-config` to verify your environment post-installation. It is recommended to install `flashinfer-python` and `flashinfer-cubin` together from PyPI to ensure compatible versions.

affects: <=0.6.x

gotcha`flashinfer-cubin` might not always contain all necessary pre-compiled cubins for every kernel or newer GPU architectures, especially for specific components like TRTLLM FMHA kernels. In such cases, `flashinfer-python` may attempt to download missing cubins at runtime, which can fail in isolated network environments or lead to unexpected JIT compilation.

fix

For air-gapped or restricted environments, consider using `flashinfer-jit-cache` (if available for your specific CUDA version) or pre-downloading kernels if `flashinfer-cubin` is insufficient. Monitor GitHub issues for updates on comprehensive cubin inclusion or use FlashInfer's source build for full control over compilation.

affects: All versions

gotchaThe `FLASHINFER_CUBIN_DIR` environment variable, intended to specify a custom path for cubin files, may be ignored when `flashinfer-cubin` is installed via pip. This can lead to issues in containerized or non-root environments where explicit control over artifact paths is required.

fix

If deploying in environments where `FLASHINFER_CUBIN_DIR` is critical, verify its behavior. It may be necessary to either prevent `flashinfer-cubin` from being installed by pip and manage cubins manually, or use `flashinfer-jit-cache` with a specified index URL for specific CUDA versions.

affects: All versions

gotchaWhile FlashInfer supports a wide range of NVIDIA GPU architectures (SM 7.5 'Turing' and later, up to SM 12.1 'Blackwell'), not all advanced features (e.g., FP8/FP4 operations, certain attention types) are supported across all compute capabilities. Performance can also vary significantly.

fix

Consult the FlashInfer documentation for a detailed breakdown of feature support per GPU architecture. Test critical workloads on your target hardware to ensure expected performance and functionality.

affects: All versions

Upgrade

Version history

0.6.13latest on PyPI · released Jun 25, 2026

Audit

Dependencies

flashinfer-pythonrequiredflashinfer-cubin provides pre-compiled kernels for the core FlashInfer library. It is not functional on its own.

torchrequiredFlashInfer is built on PyTorch and requires a compatible PyTorch installation with CUDA support.

Agent activity

79 hits · last 30 days

node

OpenAI (training)

Perplexity

Resources

githubhttps://github.com/flashinfer-ai/flashinfer ↗

packagehttps://pypi.org/project/flashinfer-cubin/ ↗

homepagehttps://flashinfer.ai ↗

docshttps://github.com/flashinfer-ai/flashinfer ↗