litserve

library0.2.17pypypi✓ verified 85d ago

LitServe is a lightweight, fast AI inference server from Lightning AI. It wraps PyTorch models (and others) into a performant API endpoint with auto-batching, GPU support, and a simple API. Current version 0.2.17, requires Python >=3.10. Released under Apache 2.0, active development with frequent updates.

devops

pip install litserve

github ↗homepage ↗

INSTALL

IMPORT

SIG · LITSERVE

litserve

devopspythonv0.2.17

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

LitServer

✓ from litserve import LitServer

✗ import litserve

Direct import of Litserver is needed to use the main class.

SimpleLitAPI

✓ from litserve import SimpleLitAPI

SimpleLitAPI is the preferred base for single-model endpoints.

LitAPI

✓ from litserve import LitAPI

LitAPI is the base class; subclass to define custom predict logic.

Basic LitServe app with a SimpleLitAPI subclass and server instantiation.

import litserve as ls class SimpleLitAPI(ls.SimpleLitAPI): def setup(self, device): self.model = lambda x: x * 2 def predict(self, x): return self.model(x) if __name__ == "__main__": api = SimpleLitAPI() server = ls.LitServer(api, accelerator='auto', devices=1, workers_per_device=1) server.run(port=8000)

Debug

Known issues

breakingThe 'LitServer' class now requires the 'devices' keyword argument (defaults to 1). In earlier versions it was optional or auto-detected. Omitting it may lead to unexpected device assignment.

fix

Always specify 'devices' explicitly, e.g., devices=1 for a single GPU or CPU.

affects: >=0.2.0

gotchaThe 'predict' method must return a JSON-serializable object. Returning raw numpy arrays or tensors will cause serialization errors.

fix

Convert outputs to Python lists, dicts, or strings before returning.

affects: all

deprecatedThe old pattern of using 'LitAPI' with 'setup' and 'predict' directly is stable but 'SimpleLitAPI' is recommended for new projects as it reduces boilerplate.

fix

Use 'SimpleLitAPI' subclass instead of implementing LitAPI from scratch, unless you need custom batch logic.

affects: >=0.1.0

gotchaUsing 'accelerator="auto"' may default to CPU if no GPU is detected. Multi-GPU setups require explicit device count.

fix

For GPU-only deployment, set accelerator='gpu' and devices=num_gpus. For CPU, accelerator='cpu' and devices=1.

affects: all

gotchaRunning on Windows may cause issues with multiprocessing workers. The server uses forking by default, which is not supported on Windows.

fix

Set 'workers_per_device=0' to disable multiprocessing, or use WSL/Linux.

affects: all

Errors

Common errors & fixes

AttributeError: module 'litserve' has no attribute 'LitServer'

Forgot to import from litserve; used 'import litserve' and then 'litserve.LitServer'.

fix

Use 'from litserve import LitServer' or 'import litserve as ls' and then 'ls.LitServer'.

TypeError: 'numpy.ndarray' object is not callable

Returned a numpy array from predict() – not JSON serializable.

fix

Convert to list: result.tolist().

ValueError: The number of devices (8) must not exceed the number of available GPUs (4)

Set devices=8 but only 4 GPUs available.

fix

Set devices to the number of GPUs available (e.g., devices=4) or use accelerator='cpu'.

RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

Running on Windows without proper if __name__ == '__main__' guard.

fix

Wrap the server.run() call inside if __name__ == '__main__': block.

Upgrade

Version history

0.2.17latest on PyPI · released Dec 23, 2025

Audit

Dependencies

No dependency data recorded yet.

Agent activity

4 hits · last 30 days

node

Resources

githubhttps://github.com/Lightning-AI/litserve ↗

homepagehttps://github.com/Lightning-AI/litserve ↗