Install & Compatibility
Where this runs
tested against v0.2.36 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
5222MB installed
● package 5222MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
get_conversation_template
✓ from fastchat.conversation import get_conversation_template
✗ from fastchat.model import get_conversation_template
Generate a conversation prompt using the Vicuna template.
from fastchat.model import get_conversation_template
from fastchat.serve.inference import generate_stream
conversation = get_conversation_template("vicuna")
conversation.append_message(conversation.roles[0], "Hello!")
conversation.append_message(conversation.roles[1], None) # assistant placeholder
prompt = conversation.get_prompt()
# Example using a local model (requires model download)
# from fastchat.serve.model_worker import ModelWorker
# worker = ModelWorker(controller_addr="http://localhost:21001", model_names=["vicuna-7b-v1.5"], worker_addr="http://localhost:21002")
print(prompt)
fastchat --version
Debug
Known issues
gotchaFastChat's controller uses HTTP requests; all components (controller, model worker, web server) must be started separately. Failure to start the controller first leads to connection errors.fixStart controller: python -m fastchat.serve.controller, then model worker, then web server.
affects: all
gotchaThe model worker requires significant GPU memory; default settings may OOM on smaller GPUs. Adjust --num-gpus or --load-8bit accordingly.fixUse --load-8bit for 8-bit quantization or --device cpu for CPU inference (slow).
affects: all
deprecatedThe old fastchat.serve.gradio_web_server is deprecated in favor of fastchat.serve.gradio_web_server_multi (for multiple models) or the new web UI variants.fixUse python -m fastchat.serve.gradio_web_server_multi for multiple models.
affects: >=0.2.25 <0.2.36
gotchaWhen using the OpenAI-compatible API, environment variable OPENAI_API_BASE must be set to the FastChat API server URL (e.g., http://localhost:8000/v1). Otherwise clients will try to reach the real OpenAI API.fixSet OPENAI_API_BASE=http://localhost:8000/v1 in your environment.
affects: all
breakingIn version 0.2.30, the default model worker registration changed; old controller may not recognize new worker without --force-reload.fixUse --force-reload flag on model worker if registration fails.
affects: >=0.2.30
Upgrade
Version history
0.2.36latest on PyPI · released Feb 11, 2024
Audit
Dependencies
torchrequiredRequired for model inference and training
transformersrequiredHugging Face Transformers for model loading
gradiooptionalUsed for the web UI (gradio option)
vllmoptionalOptional high-throughput serving backend
sglangoptionalOptional SGLang worker for vision-language models