disable_autopilot
Turn off autopilot. Already-accepted optimizations stay active; only future runs are paused. Use enable_autopilot to turn it back on.
LABSCO SUMMARY
A 24-hour revert window on automatic changes is what makes autopilot a defensible feature rather than a reckless one.
Letting a service change which model serves your production features is a large amount of trust, and the thing that earns it is a bounded, reversible window rather than a promise. Listing what is currently inside that window, with a revert per optimisation, means the exposure is visible at any moment. The detail about disabling autopilot leaving prior changes in place is exactly the kind of thing that is normally learned the hard way, and stating it in the tool text is right. seed_samples deserves mention too: benchmarking a low-traffic flow is otherwise impossible, and being able to supply the cases removes the chicken-and-egg.
What it is
An LLM cost and quality optimisation service that watches your AI flows through a proxy, benchmarks them against alternative models, and can apply improvements automatically within a revert window.
What you get
- list_flows shows every AI feature in the project with its current model and call volume, which is the inventory most teams do not have.
- get_suggestions returns pending recommendations to switch to a cheaper or faster model, and update_suggestion accepts or dismisses one by id.
- run_benchmark tests a flow against alternatives with an optimisation target you choose, and list_new_models surfaces what is newly worth testing.
- optimize_prompt analyses a flow's prompt and generates an improved version.
- seed_samples registers test cases so a flow can be benchmarked without waiting for real traffic — the thing that otherwise blocks evaluation entirely.
- Autopilot is explicit and reversible: check whether the prerequisites are met, enable or disable it, list auto-accepted optimisations still inside their 24-hour window, and revert one within it.
- Disabling autopilot leaves already-accepted optimisations active and only stops future ones, which the tool states rather than leaving you to discover.
- get_account reports credit balance and usage.
Requirements
LLMTEST_API_KEY, and your AI calls routed through the proxy so there is activity to observe.
Setup effort
One command plus a key — npx llmtest-mcp, then supply credentials
USE CASES
01
Finding out what your AI features actually cost
List the flows with their current models and call volumes — the inventory most teams have never assembled.
02
Testing whether a cheaper model would do
Benchmark the flow against alternatives with a chosen optimisation target, seeding test samples if the flow has too little real traffic.
03
Acting on a recommendation
Read the pending suggestions and accept or dismiss each by id, rather than taking them wholesale.
04
Improving the prompt rather than the model
Have the flow's prompt analysed and an optimised version generated.
05
Letting it run itself, carefully
Check the prerequisites, enable autopilot, and keep an eye on what is inside the 24-hour window — with a revert per optimisation while it lasts.
16 tools
Counted once with only startup setup. Other versions, modes or optional credentials may expose a different list.
enable_autopilot
Turn on autopilot. Once enabled, LLMTest will automatically optimize eligible flows every 7 days, auto-accept changes that pass all safety gates, and auto-revert if drift detection later sees regressions.
Cost: each optimization run costs ~$1-2, billed to LLMTest credits. Runs are skipped silently if the account has no credits.
Prerequisites: verified email, account age >= 14 days, and >=20 real calls per flow. Use get_autopilot_status to check readiness.
get_account
Check your credit balance and usage stats.
get_autopilot_status
Show whether autopilot is enabled and whether the account meets the prerequisites.
Autopilot automatically optimizes prompts and models for eligible flows every 7 days. Checks: optimization scheduler (weekly), drift detection (daily scan, per-optimization weekly).
Prerequisites for a flow to be auto-optimized:
- Autopilot enabled on the account
- Email verified
- Account age >= 14 days (trust ramp)
- Flow has >=20 real calls in the last 7 days
- Flow not optimized by autopilot in the last 14 days (cooldown)
- Account has credits (each run costs ~$1-2)
Safety gates (all must pass to auto-accept): 95% CI lower bound > 50%, multi-judge agreement >= 80%, >=20% total savings, no length-bias warning, golden-set regression check. After auto-accept: 24h revert window. Drift detection reverts automatically if quality degrades later.
get_suggestions
Get pending recommendations to switch to a cheaper or faster model without losing quality.
leave_feedback
After using any tool in this MCP server, you may call this to share your experience as an AI agent. This helps improve the tool. This is YOUR assessment as an agent, not the user's.
list_active_optimizations
List auto-accepted optimizations still within their 24h revert window. Each includes the flow, strategy, savings, and hours remaining to revert.
list_flows
Show all AI features in this project with their current model, call volume, latency, and cost.
list_new_models
Show new and trending models that are worth benchmarking soon.
list_samples
Show how many test samples are stored for each flow.
optimize_prompt
Analyze a flow's prompt and automatically generate an optimized version.
Tests optimized prompt against baseline using real samples and a multi-judge AI (Claude Sonnet + GPT-4o, position-swapped, self-consistency). Also tests cheaper models.
REQUIRES: at least 3 samples on the flow. If the flow has none (zero traffic), BEFORE calling this tool you should: (1) ask the user to describe the feature, (2) find or ask for their system prompt, (3) generate 5-10 realistic synthetic user inputs, (4) call seed_samples with [{role:'system', content:<prompt>}, {role:'user', content:<synthetic input>}] for each, (5) THEN call optimize_prompt. Tell the user synthetic-sample results are "indicative" vs real-traffic results.
Confidence levels: HIGH (safe to accept), MEDIUM (review first), LOW (need more traffic), NEGATIVE (do not accept).
Cost: ~$1-2 per run, billed to the user's LLMTest credits.
revert_optimization
Revert an auto-accepted optimization. Only works within the 24h revert window. Restores the previous active prompt. Use list_active_optimizations to find IDs.
run_benchmark
Benchmark a flow against alternative models. Use optimize_for to focus the search: "cost" tests cheaper models, "quality" tests potentially better models, "speed" tests faster models, "balanced" (default) tests a diverse spread.
Use challengers to test specific models (e.g. ["anthropic/claude-sonnet-4", "google/gemini-2.5-flash-preview"]). If omitted, challengers are selected automatically based on optimize_for.
For pre-launch projects with no real traffic: seed samples first with seed_samples, then call this with currentModel set to the model you are currently using (or planning to use).
For post-launch projects: currentModel is detected automatically from call history.
seed_samples
Register test samples for a flow so it can be benchmarked without real traffic. Use this for pre-launch projects or when you want to benchmark a specific scenario.
Each sample must be an array of messages in OpenAI chat format: [{ "role": "system", "content": "..." }, { "role": "user", "content": "..." }]
To use: read the codebase to find the prompt for this flow, then generate 5-10 realistic and varied user inputs that represent what real users would send. Max 10 samples per call.
status
Show LLMTest proxy status and a summary of monitored AI activity.
update_suggestion
Accept or dismiss a model-switch suggestion by ID.
BEFORE YOU INSTALL
What to know first
Tools you get
16 tools when we connected. leave_feedback is undocumented. The five autopilot tools need a verified email, a 14-day-old account, a flow with 20+ recent calls and credits; enable_autopilot and optimize_prompt rewrite production prompts.
Requires
- npx on PATH
- LLMTEST_API_KEY in your environment
Where it runs
All 3 documented clients run the server on your machine.
CONTEXT COST
Light Standing cost
Rated by the size of the 16 tool definitions that ride along on every turn, not by how many there are
INSTALLATION
Connect your client
Server setup
TERMINAL
npx llmtest-mcpMCP Clients
Claude Code
CLAUDE CODE · TERMINAL
claude mcp add llmtest -- npx llmtest-mcpCursor, Windsurf
JSON
{
"mcpServers": {
"llmtest": {
"command": "npx",
"args": [
"llmtest-mcp"
],
"env": {
"LLMTEST_API_KEY": "<YOUR_LLMTEST_API_KEY>"
}
}
}
}
