Letting a service change which model serves your production features is a large amount of trust, and the thing that earns it is a bounded, reversible window rather than a promise. Listing what is currently inside that window, with a revert per optimisation, means the exposure is visible at any moment. The detail about disabling autopilot leaving prior changes in place is exactly the kind of thing that is normally learned the hard way, and stating it in the tool text is right. seed_samples deserves mention too: benchmarking a low-traffic flow is otherwise impossible, and being able to supply the cases removes the chicken-and-egg.
An LLM cost and quality optimisation service that watches your AI flows through a proxy, benchmarks them against alternative models, and can apply improvements automatically within a revert window.
- list_flows shows every AI feature in the project with its current model and call volume, which is the inventory most teams do not have.
- get_suggestions returns pending recommendations to switch to a cheaper or faster model, and update_suggestion accepts or dismisses one by id.
- run_benchmark tests a flow against alternatives with an optimisation target you choose, and list_new_models surfaces what is newly worth testing.
- optimize_prompt analyses a flow's prompt and generates an improved version.
- seed_samples registers test cases so a flow can be benchmarked without waiting for real traffic — the thing that otherwise blocks evaluation entirely.
- Autopilot is explicit and reversible: check whether the prerequisites are met, enable or disable it, list auto-accepted optimisations still inside their 24-hour window, and revert one within it.
- Disabling autopilot leaves already-accepted optimisations active and only stops future ones, which the tool states rather than leaving you to discover.
- get_account reports credit balance and usage.
LLMTEST_API_KEY, and your AI calls routed through the proxy so there is activity to observe.
One command plus a key — npx llmtest-mcp, then supply credentials