genai-perf

library0.0.16pypypi✓ verified 87d ago

GenAI-Perf is a command-line interface (CLI) tool designed for measuring the throughput and latency of generative AI models (Large Language Models, Vision Language Models, Embedding Models, Ranking Models, and LoRA Adapters) served through an inference server. It generates load, measures key performance metrics such as output token throughput, time to first token, inter-token latency, and request throughput, and reports the results to the console, CSV, and JSON files. While currently at version 0.0.16 and under rapid development, it is being actively phased out in favor of NVIDIA's new `AIPerf` tool for generative AI benchmarking.

ai-ml devops

pip install genai-perf

github ↗package ↗

INSTALL

IMPORT

SIG · GENAI-PERF

genai-perf

ai-mlpythonv0.0.16

Install

19.4s avg

Import

477ms

Disk

867MB

Pass rate

8/ 10

Env Coverage8 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.0.16 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✓ —

✓ 35.75s

py 3.11

✓ 0.05s

✓ 33.45s

py 3.12

✓ 0.03s

✓ 32.7s

py 3.13

✓ 0.05s

✓ 33.63s

py 3.9

✕ build_error

867MB installed

● package 867MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Checkpoint

✓ from genai_perf.checkpoint.checkpoint import Checkpoint

Used for programmatic access to `analyze` command results, not for direct profiling.

Results

✓ from genai_perf.config.run.results import Results

Used for programmatic access to `analyze` command results, not for direct profiling.

This quickstart demonstrates how to run a performance profile against a hypothetical GPT-2 model served by Triton Inference Server using synthetic data. Ensure your inference server and model are already running before executing this command. The output will include metrics like Time to First Token, Inter-Token Latency, and Request Latency.

# Note: A Triton Inference Server or OpenAI-compatible API endpoint with a model (e.g., GPT-2 TensorRT-LLM) must be running. # For example, to run GPT-2 on Triton, you might use 'triton import -m gpt2 --backend tensorrtllm' and then 'triton start'. genai-perf profile \ -m gpt2 \ --backend tensorrtllm \ --synthetic-input-tokens-mean 200 \ --synthetic-input-tokens-stddev 0 \ --output-tokens-mean 100 \ --output-tokens-stddev 0 \ --output-tokens-mean-deterministic \ --streaming \ --request-count 50 \ --warmup-request-count 10 # This command will generate a load against the 'gpt2' model served by Triton # (using TensorRT-LLM backend), measuring performance metrics for 50 requests # with synthetic inputs and a deterministic output length, after a 10-request warmup. # Results are printed to console and saved to files in the 'artifacts' directory.

genai-perf --version

Debug

Known issues

deprecated`genai-perf` is being phased out. NVIDIA recommends migrating to `AIPerf` for new generative AI performance benchmarking needs, as `genai-perf` will no longer receive active feature development.

fix

For new projects or existing projects requiring continued support, evaluate and transition to `AIPerf`. Consult NVIDIA's documentation for migration guides.

affects: All versions

gotchaGenAI-Perf is a CLI tool and requires an inference server (e.g., NVIDIA Triton Inference Server or an OpenAI-compatible API endpoint) with a model loaded to be already running and accessible before `genai-perf` can perform benchmarking.

fix

Ensure your inference server is configured and running, and your target model is loaded and ready to receive requests, before invoking `genai-perf` commands.

affects: All versions

gotchaThe `genai-perf` tool is in early release and under rapid development. Command-line options and functionalities are subject to change between minor versions.

fix

Refer to the latest official documentation or `genai-perf --help` for the most up-to-date command-line arguments. Pin exact versions in production environments to avoid unexpected changes.

affects: 0.0.x

gotchaWhen benchmarking models from gated repositories (e.g., some Hugging Face models like Llama 3), an `HF_TOKEN` environment variable may be required for authentication to download the tokenizer.

fix

Set the `HF_TOKEN` environment variable with a valid Hugging Face token (e.g., `export HF_TOKEN='hf_YOUR_TOKEN'`) before running `genai-perf` if using such models.

affects: All versions

Errors

Common errors & fixes

genai-perf: command not found

The `genai-perf` executable is not in your system's PATH, or the package was not installed correctly in the active environment. This is common if installed within a Docker container and attempting to run outside it, or if a Python virtual environment isn't activated.

fix

If installed via `pip install genai-perf`, ensure your Python virtual environment is activated. If using the Triton SDK container, run `genai-perf` from *inside* the container (`docker run -it ... genai-perf --help`).

Missing 'input_output_genai_perf.csv' files in artifact directories after running 'genai-perf analyze'.

This issue has been reported in forum discussions, potentially due to the `measurement-interval` being too short for the inference server to complete enough requests, or related to specific container versions.

fix

Increase the `--measurement-interval` (e.g., `-p 10000` for 10 seconds or higher) to allow sufficient time for requests to complete and data to be recorded. Ensure you are using a consistent and recommended `genai-perf` container version as per NVIDIA's documentation, or check the `--profile-export-file` for other output files.

Error during request generation or processing (e.g., connection refused, HTTP 4xx/5xx errors).

The target inference server is either not running, inaccessible at the specified URL, or the model is not loaded correctly. Alternatively, there might be an issue with the `--model` name, `--backend`, or `--endpoint-type` configuration.

fix

Verify that your Triton Inference Server (or other endpoint) is running and reachable at the `--url` provided. Confirm the model specified by `-m` or `--model` is correctly loaded and the `--backend` and `--endpoint-type` (e.g., `chat`, `completions`, `embeddings`) match the server's configuration and model's capabilities.

Upgrade

Version history

0.0.16latest on PyPI · released Aug 26, 2025

Audit

Dependencies

pythonrequiredRequired runtime environment.

tritonclientoptionalRequired for interactions with Triton Inference Server. Automatically included with Triton SDK container or installed separately.

Agent activity

51 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/triton-inference-server/perf_analyzer ↗

packagehttps://pypi.org/project/genai-perf/ ↗