Install & Compatibility
Where this runs
tested against v0.3.21 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.11
✕ build_error
2/4 runs
py 3.12
✕ build_error
2/4 runs
py 3.13
✕ build_error
2/4 runs
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
get_memory_info
✓ from sgl_kernel.cuda_helper import get_memory_info
sgl-kernel primarily provides low-level, internal components for `sglang`. Direct imports by end-users are rare, but utilities like `get_memory_info` from `cuda_helper` might be used for debugging or advanced scenarios. Most users interact with `sglang` directly.
This quickstart demonstrates how to use `sglang`, the main library that leverages `sgl-kernel` for its high-performance execution. It shows a simple LLM generation task. Note that `sgl-kernel` itself does not expose a high-level API for direct user interaction; its functionality is accessed through `sglang`.
import sglang as sl
import os
os.environ['SGLANG_DEV_MODE'] = 'True' # Optional: for development features
# Launch an SGLang runtime (which utilizes sgl-kernel for execution)
runtime = sl.Runtime("openai/gpt-4o-mini") # Or your preferred local model path
@sl.function
def generate_joke(s, topic):
s += f"Give me a joke about {topic}."
s += sl.gen("joke", max_tokens=64, temperature=0.7)
# Run the function
state = runtime.run(generate_joke, topic="cats")
print(f"Joke about cats: {state['joke']}")
runtime.shutdown()
Debug
Known issues
gotchaThe `sgl-kernel` PyPI package (currently 0.3.21) is often an older version than the `sgl_kernel` sub-package distributed with the main `sglang` library (currently 0.5.10.post1). For the latest features, optimizations, and compatibility, it is strongly recommended to install `sglang`.fixInstall `sglang` via `pip install sglang` to get the up-to-date kernel components.
affects: <0.5.10.post1 (sgl-kernel PyPI)
gotchasgl-kernel heavily relies on NVIDIA GPUs and CUDA. Running without a compatible GPU, sufficient VRAM, and correctly installed NVIDIA drivers (including CUDA toolkit) will lead to runtime errors or prevent `sglang` from functioning.fixEnsure you have an NVIDIA GPU, up-to-date drivers, and CUDA toolkit installed. Check `torch.cuda.is_available()`.
affects: All versions
breakingStarting with SGLang v0.5.10, piecewise CUDA graph capture is enabled by default. While generally improving throughput and reducing memory overhead, this might subtly change performance characteristics or expose new corner cases for models with highly complex control flow. Test your applications thoroughly.fixMonitor memory usage and throughput after upgrading. If issues arise, consult SGLang documentation for potential configuration options to adjust CUDA graph behavior.
affects: >=0.5.10 (of sglang)
gotchaInstallation of `flashinfer` (a key dependency for `sgl-kernel`'s optimized attention) can sometimes fail due to specific CUDA version requirements or compilation issues, especially when `torch` and `flashinfer` versions are mismatched or system CUDA is not configured correctly.fixEnsure your `torch` version is compatible with `flashinfer`'s requirements. Check `flashinfer`'s GitHub for specific CUDA/Torch version matrices. Often, upgrading `pip` and reinstalling `flashinfer` from source or pre-built wheels helps.
affects: All versions
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'sglang'
The `sglang` library, which bundles `sgl-kernel`, is not installed.
fixRun `pip install sglang` to install the main library.
RuntimeError: No CUDA device available.
Python environment cannot detect an NVIDIA GPU or CUDA drivers are not properly installed/configured.
fixVerify NVIDIA drivers are installed and up to date. Check `nvidia-smi` and `torch.cuda.is_available()` in Python. Ensure CUDA toolkit is compatible with your PyTorch installation.
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate...
The model being loaded or the batch size/sequence length exceeds the available GPU memory.
fixTry using a smaller model, reducing batch size, or decreasing `max_tokens`. Utilize quantization (e.g., 4-bit, 8-bit) if supported by the model and SGLang. Consider using a GPU with more VRAM.
Upgrade
Version history
0.3.21latest on PyPI · released Jan 15, 2026
Audit
Dependencies
torchrequiredCore deep learning framework dependency for GPU operations.
transformersrequiredUsed for model loading and tokenizer functionalities.
flashinferrequiredOptimized attention kernel for enhanced performance.