Simulation mode is useful while writing a fixture, but a fixture that never spawns the server proves nothing — the repository is explicit that live mode runs against real servers and that `output_contains` will always fail in simulation if `expected_output` isn't set. `regression_report` surfacing fixes as well as regressions is the small touch that makes it worth running on every change.
A testing harness for MCP servers and agent workflows. Test cases are plain YAML files in version control, so they diff and review like code; the runner spawns a real server over stdio and asserts against what it actually returns, then compares this run to previous ones.
- Running: `run_suite` executes every fixture and returns a pass/fail summary, `run_case` runs one by name, `list_cases` enumerates them with step counts
- Authoring: `create_test_case` writes a new fixture, `scaffold_fixture` generates boilerplate with placeholder steps and pre-filled assertion comments
- Reporting: `regression_report` compares the current state to the last run and surfaces both regressions and fixes, `compare_results` diffs two runs by id, `generate_html_report` produces a single-file report
- Operations: `evaluate_deployment_gate` fails CI when the recent pass rate drops below a threshold, `discover_fixtures` finds fixture files across directories
- Assertions combine per step: `output_contains`, `output_not_contains`, `output_equals`, `output_matches`, `schema_match` against JSON Schema, `tool_called`, and `latency_under` in milliseconds
- A fixture with a `server` block runs live; without one it runs in simulation against `expected_output`. Fixtures made by `create_test_case` have no `server` block, so they simulate until you add one
- Steps pipe into each other with `{{steps.<step_id>.output}}`, in both modes
Node.js v22.5.0 or newer, and npm. Add it to your client as `npx -y mcp-eval-runner@latest`. Fixtures load from `./evals` unless `--fixtures` points elsewhere; run history goes to a SQLite file at `~/.mcp/evals.db` by default. `--timeout` bounds a single step at 30000 ms, `--concurrency` runs cases in parallel from a default of 1, `--watch` reruns the affected fixture on file change, and `--format` selects console, json or html output. A GitHub Action is included.
One command — npx -y mcp-eval-runner@latest