Labsco
MCP SERVER

Log an agent run to a local SQLite file, score it against 13 deterministic rules for free, and reach for an LLM judge only when the question is semantic.

LLM Evaluation & Observability✓ Verified
Summary
The free scoring path is deterministic, and the one that costs money tells you exactly what it cost.

Splitting heuristics from the judge is the design decision that matters: thirteen inspectable rules answer the safety and cost questions in milliseconds without an API call, and the judge is reserved for accuracy and helpfulness — returning its rationale and the exact spend, not just a number. The author is unusually clear about the limitation too: MCP tools fire when the model decides to call them, so a trace is logged because your agent was asked to log it. If you want capture that does not depend on that, POST the same body to /api/v1/traces with no model in the loop. `--demo` seeds a separate database so you can see a real failure before wiring anything up.

What it is

An evaluation store and scorer for agent runs, held entirely on your machine. Nine tools cover the trace lifecycle, three scoring modes and a custom rule store, with a dashboard on http://localhost:6920 showing failures worst and newest first.

What you get
  • A full execution trace persisted — input, output, spans, tool calls, cost, latency and token usage — and queried back with filters, pagination and an optional dashboard summary (log_trace, get_traces).
  • Deterministic scoring against 13 built-in rules covering PII, prompt injection, hallucination markers and cost thresholds, returning a 0-to-1 score with a per-rule breakdown, in milliseconds, with no API key and no cost (evaluate_output).
  • Semantic scoring when the heuristics cannot answer: an Anthropic or OpenAI judge returning a calibrated 0-to-1 score with its rationale, a per-dimension breakdown, and the exact cost of that call (evaluate_with_llm_judge).
  • Citation grounding as its own check: extract the citations from the output, fetch the cited sources, and have a judge decide whether each source actually supports the claim, returning per-citation verdicts and an overall support ratio (verify_citations).
  • Your own rules deployed into the store, firing on every future evaluation of their category, and removable — past results that referenced a deleted rule keep their score history (deploy_rule, list_rules, delete_rule).
  • Deletion that does not destroy the record: removing a trace cascades to its spans, while eval results keep the score with the trace_id nulled (delete_trace).
Requirements

Node.js 20 or later, and nothing else — no account, no SDK, no telemetry, and the traces stay in a SQLite file under your Iris home directory. `npx @iris-eval/mcp-server --dashboard` runs the server with the dashboard; drop the flag for headless. The LLM judge and citation verification are the only parts that need an Anthropic or OpenAI key and cost money. IRIS_API_KEY, IRIS_DB_PATH and IRIS_LOG_LEVEL are optional.

Setup effort

One command — npx @iris-eval/mcp-server