torchserve

library0.12.0pypypiunverified

TorchServe is a flexible and easy-to-use tool for serving PyTorch models for inference at scale. Current version 0.12.0, released as part of the PyTorch ecosystem, with releases roughly quarterly.

ai-ml devops web-framework data

pip install torchserve torch-model-archiver

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · TORCHSERVE

torchserve

ai-mlpythonv0.12.0

Install

3.5s avg

Import

—

Disk

82MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.12.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 83.8MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 3.5s · import 0.000s · 85MB

82MB installed

● package 82MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

torchserve

✓ import torchserve

Main import for TorchServe package; typically not used directly in user code.

Basic workflow: archive a model, start TorchServe, and make a prediction.

# Create a model archive torch-model-archiver --model-name my_model --version 1.0 --model-file model.py --serialized-file model.pt --handler image_classifier # Start TorchServe torchserve --start --model-store model_store --models my_model=my_model.mar # Send inference request curl -X POST http://localhost:8080/predictions/my_model -T input.jpg

torchserve --version

Debug

Known issues

breakingDefault backend changed from synchronous to asynchronous in v0.9.0. If your code relies on immediate callback or synchronous response, you need to enable it explicitly.

fix

Use `--sync` flag or set `sync: true` in config.properties.

affects: >=0.9.0

gotchaThe 'torchserve' package does not include the CLI entry points; you must also install 'torch-model-archiver' for the `torch-model-archiver` command.

fix

Install both packages: pip install torchserve torch-model-archiver

affects: all

gotchaTorchServe uses a custom handler API. Using a standard `nn.Module.forward()` directly will not work; you must wrap it in a handler class with `preprocess`, `inference`, and `postprocess` methods.

fix

Implement a handler class inheriting from `torchserve.base_handler.BaseHandler`.

affects: all

deprecatedThe `--model-name` flag in `torch-model-archiver` is deprecated in favor of positional argument.

fix

Use `torch-model-archiver my_model --version 1.0 --model-file model.py ...` instead of `--model-name my_model`.

affects: >=0.10.0

Upgrade

Version history

0.12.0latest on PyPI · released Sep 30, 2024

Audit

Dependencies

torchrequiredPyTorch dependency for model serving

torch-model-archiveroptionalTool for archiving models for TorchServe

Agent activity

23 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/pytorch/serve.git ↗

packagehttps://pypi.org/project/torchserve/ ↗

homepagehttps://pytorch.org/serve ↗