LLMTest

nonedevelopment

LLM proxy that benchmarks AI models on real prompts and finds cheaper, faster alternatives across 340+ models.

Install

How to run this server

[ { "cmd": "npx llmtest-mcp", "imports": [] } ]

server path: llmtest

Tools

What this server exposes

status

Show proxy status and activity summary

list_flows

List all AI flows with cost and latency stats

get_suggestions

Get pending model-switch recommendations

update_suggestion

Accept or dismiss a suggestion

run_benchmark

Benchmark a flow against challenger models

optimize_prompt

Rewrite a flow's prompt and find a cheaper model that still works

seed_samples

Add test prompts for pre-launch benchmarking

list_samples

Show stored test samples per flow

list_new_models

Show new and trending models

get_account

Check credit balance and usage

get_autopilot_status

Check whether autopilot is on and whether the account is eligible

enable_autopilot

Turn on weekly auto-optimization with safety gates + drift-based auto-revert

disable_autopilot

Turn off autopilot (existing optimizations stay active)

list_active_optimizations

List auto-accepted optimizations still inside their 24h revert window

revert_optimization

Roll an auto-accepted optimization back to the previous prompt

Configuration

Environment & auth

authnone

envLLMTEST_API_KEY

envLLMTEST_BASE_URL

Resources