LabscoMCP early access ↗
dbsectrainerMCP SERVER

MCP Eval Runner

Regression-test MCP servers from YAML fixtures — steps, expected tool calls, expected output, pass/fail with diffs.

by dbsectrainer ↗LLM Evaluation & ObservabilityCommunity

LABSCO SUMMARY

Live mode is the default for a reason.

Simulation mode is useful while writing a fixture, but a fixture that never spawns the server proves nothing — the repository is explicit that live mode runs against real servers and that output_contains will always fail in simulation if expected_output isn't set. regression_report surfacing fixes as well as regressions is the small touch that makes it worth running on every change.

What it is

A testing harness for MCP servers and agent workflows. Test cases are plain YAML files in version control, so they diff and review like code; the runner spawns a real server over stdio and asserts against what it actually returns, then compares this run to previous ones.

What you get

  • Running: run_suite executes every fixture and returns a pass/fail summary, run_case runs one by name, list_cases enumerates them with step counts
  • Authoring: create_test_case writes a new fixture, scaffold_fixture generates boilerplate with placeholder steps and pre-filled assertion comments
  • Reporting: regression_report compares the current state to the last run and surfaces both regressions and fixes, compare_results diffs two runs by id, generate_html_report produces a single-file report
  • Operations: evaluate_deployment_gate fails CI when the recent pass rate drops below a threshold, discover_fixtures finds fixture files across directories
  • Assertions combine per step: output_contains, output_not_contains, output_equals, output_matches, schema_match against JSON Schema, tool_called, and latency_under in milliseconds
  • A fixture with a server block runs live; without one it runs in simulation against expected_output. Fixtures made by create_test_case have no server block, so they simulate until you add one
  • Steps pipe into each other with {{steps.<step_id>.output}}, in both modes

Requirements

Node.js v22.5.0 or newer, and npm. Add it to your client as npx -y mcp-eval-runner@latest. Fixtures load from ./evals unless --fixtures points elsewhere; run history goes to a SQLite file at ~/.mcp/evals.db by default. --timeout bounds a single step at 30000 ms, --concurrency runs cases in parallel from a default of 1, --watch reruns the affected fixture on file change, and --format selects console, json or html output. A GitHub Action is included.

Setup effort

One command — npx -y mcp-eval-runner@latest

See installation options