Install & Compatibility
Where this runs
tested against v1.23.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
build_error
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 11.2s · import 0.300s · 616MB
503MB installed
● package 503MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
InferenceSession
✓ from onnxruntime import InferenceSession
✗ from onnxruntime import InferenceSession
This quickstart demonstrates how to create a simple ONNX model, save it, and then load it into an `InferenceSession` configured to prioritize GPU (CUDA) execution. It includes error handling for common GPU setup issues.
import onnxruntime as ort
import numpy as np
import onnx
from onnx import helper, TensorProto
import os
# 1. Create a dummy ONNX model for demonstration
# Define the graph (input, output, and node)
X = helper.make_tensor_value_info('X', TensorProto.FLOAT, [None, 3])
Y = helper.make_tensor_value_info('Y', TensorProto.FLOAT, [None, 3])
node = helper.make_node('Relu', ['X'], ['Y'])
graph = helper.make_graph([node], 'simple_relu', [X], [Y])
model = helper.make_model(graph, producer_name='onnx-example')
# Save it to a temporary file
model_path = "simple_relu.onnx"
onnx.save(model, model_path)
# 2. Load the model with GPU provider
try:
# Prioritize CUDAExecutionProvider for NVIDIA GPUs
# Fallback to CPUExecutionProvider if CUDA is not available or fails
session = ort.InferenceSession(
model_path,
providers=["CUDAExecutionProvider", "CPUExecutionProvider"]
)
print("ONNX Runtime session created with providers:", session.get_providers())
# Prepare dummy input data
input_data = np.random.rand(1, 3).astype(np.float32)
# Run inference
output = session.run(None, {'X': input_data})
print("Inference successful. Output shape:", output[0].shape)
except Exception as e:
print(f"\nError creating ONNX Runtime session or running inference: {e}")
print("Make sure you have a compatible CUDA environment (or other GPU runtime) ")
print("and the correct onnxruntime-gpu package installed. \n")
print("If CUDA is not available, try removing 'CUDAExecutionProvider' from the providers list.")
finally:
# Clean up the dummy model file
if os.path.exists(model_path):
os.remove(model_path)
Debug
Known issues
gotchaThe `onnxruntime-gpu` package requires a specific CUDA Toolkit and cuDNN version to be installed on your system. Mismatched versions are a very common cause of `InferenceSession` initialization failures or runtime errors.fixConsult the official ONNX Runtime documentation (e.g., 'Build ONNX Runtime from source' or release notes) for the exact CUDA/cuDNN versions compatible with your `onnxruntime-gpu` version and ensure they are correctly installed and configured in your system environment (e.g., `PATH`, `LD_LIBRARY_PATH`).
affects: All `onnxruntime-gpu` versions
gotchaWhen using `onnxruntime-gpu`, you must explicitly specify execution providers like `['CUDAExecutionProvider', 'CPUExecutionProvider']` during `InferenceSession` creation to ensure GPU acceleration is attempted. If not specified, ONNX Runtime might default to CPU execution even with the GPU package installed.fixAlways pass `providers=["CUDAExecutionProvider", "CPUExecutionProvider"]` (or `ROCMExecutionProvider` for AMD GPUs) to `onnxruntime.InferenceSession()` to prioritize GPU and gracefully fall back to CPU if GPU isn't available or fails.
affects: All `onnxruntime-gpu` versions
gotchaThere are two main PyPI packages: `onnxruntime` (CPU-only) and `onnxruntime-gpu` (GPU-enabled). Installing `onnxruntime-gpu` does *not* automatically remove `onnxruntime`. If both are installed, `onnxruntime` might be used by default or cause conflicts, leading to unexpected CPU-only execution.fixBefore installing `onnxruntime-gpu`, uninstall `onnxruntime` if it was previously installed (`pip uninstall onnxruntime`). Verify with `pip freeze | grep onnxruntime` that only the desired package is present.
affects: All versions
breakingStarting with ONNX Runtime version 1.17, official support for Python 3.8 and 3.9 was dropped. Version 1.24.0 and later also dropped support for Python 3.10. The current version (1.24.4) explicitly requires Python >= 3.11.fixUpgrade your Python environment to 3.11 or newer. If you must use an older Python version, install an older compatible `onnxruntime-gpu` version (e.g., `pip install onnxruntime-gpu<1.17` for Python 3.10 compatibility, but be aware of security and feature limitations).
affects: >= 1.17.0 (Python 3.8/3.9), >= 1.24.0 (Python 3.10)
Upgrade
Version history
1.29.0latest on PyPI · released Aug 17, 2026
Audit
Dependencies
No dependency data recorded yet.