lmcache

library0.4.7pypypiunverified

lmcache is a Python library that provides an LLM serving engine extension. It aims to reduce Time To First Token (TTFT) and increase throughput, particularly in scenarios involving long contexts. The current version is 0.4.3, and it appears to have an active development cadence.

llm-agents ai-ml

pip install lmcache

github ↗package ↗homepage ↗

INSTALL

IMPORT

SIG · LMCACHE

lmcache

llm-agentspythonv0.4.7

Install

91.9s avg

Import

—

Disk

5837MB

Pass rate

3/ 10

Env Coverage3 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.4.7 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✕ build_error

✕ timeout

py 3.11

✕ build_error

✓ 101.5s

py 3.12

✕ build_error

✓ 94.1s

py 3.13

✕ build_error

✓ 80.2s

py 3.9

✕ build_error

✕ timeout

5837MB installed

● package 5837MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Client

✓ from lmcache.client import Client

✗ from lmcache.core.client import Client

The client import path was changed in versions leading up to 0.4.0, moving from a 'core' submodule.

ChatCompletionRequest

✓ from lmcache.schemas import ChatCompletionRequest

ChatCompletionMessage

✓ from lmcache.schemas import ChatCompletionMessage

This quickstart demonstrates how to use the lmcache client to interact with a running lmcache server. It sends a chat completion request similar to the OpenAI API. Please ensure that an lmcache server is running independently before executing this client code.

import os from lmcache.client import Client from lmcache.schemas import ChatCompletionRequest, ChatCompletionMessage # NOTE: An lmcache server must be running separately for this client to connect. # Default server host is 'localhost', port 13333. try: client = Client(host=os.environ.get('LMCACHE_HOST', 'localhost'), port=int(os.environ.get('LMCACHE_PORT', 13333))) request = ChatCompletionRequest( model=os.environ.get('LMCACHE_MODEL', 'gpt-3.5-turbo'), # Replace with a model supported by your lmcache server messages=[ ChatCompletionMessage(role="user", content="Hello, how are you?"), ChatCompletionMessage(role="assistant", content="I am doing well, thank you!"), ChatCompletionMessage(role="user", content="What is your purpose?") ] ) response = client.chat_completion(request) print(f"Assistant: {response.choices[0].message.content}") except Exception as e: print(f"An error occurred: {e}") print("Ensure the lmcache server is running and accessible at the specified host and port.")

Debug

Known issues

breakingThe client-side API underwent a significant refactor in version 0.4.0 to align more closely with the OpenAI API. Code written for versions prior to 0.4.0 will likely be incompatible.

fix

Update client code to use `lmcache.client.Client` and schema objects like `ChatCompletionRequest` from `lmcache.schemas`. Replace old methods like `complete` with `chat_completion`.

affects: <0.4.0

gotchalmcache is a client-server architecture. The client library cannot function without a separate lmcache server instance running. A common error is a 'Connection Refused' message if the server is not started or is inaccessible.

fix

Ensure the lmcache server (e.g., via `lmcache serve`) is running and accessible at the host and port specified by the client. Check network configurations if running remotely.

affects: All

gotchaThe lmcache server (and thus, the library's utility) often requires significant GPU memory and computational resources, especially for large language models. Insufficient resources can lead to performance issues or failures.

fix

Allocate sufficient GPU memory and CPU resources. Consult lmcache documentation for hardware recommendations and model-specific resource requirements.

affects: All

gotchaModel compatibility and configuration can be tricky. The client's `model` parameter must correspond to a model successfully loaded and served by the lmcache server, which might require specific server configurations or local model files.

fix

Verify the model name and configuration on the lmcache server. Ensure required model weights are available to the server and that the client requests a compatible model.

affects: All

Upgrade

Version history

0.4.7latest on PyPI · released Jun 13, 2026

Audit

Dependencies

torchrequiredCore deep learning library for LLM operations (especially server side).

transformersrequiredUsed for loading and interacting with various transformer models (server side).

pydanticrequiredFor defining data schemas used in client-server communication.

httpxrequiredAsynchronous HTTP client used for client-server communication.

Agent activity

9 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/LMCache/LMCache ↗

packagehttps://pypi.org/project/lmcache/ ↗

homepagehttps://docs.lmcache.ai ↗