LabscoMCP early access ↗
assister-xyzMCP SERVER

AgentTrust Quality Verification

Test whether an MCP server is any good before you trust it — challenge-response evaluation across six quality dimensions, with a signed credential as the result.

by assister-xyz ↗MCP Plumbing: Proxies, Registries & InspectorsCommunity

LABSCO SUMMARY

It scores servers, and the safety axis is the one to read.

Accuracy carries the most weight, but the part you cannot get from reading a README is how a server behaves under the five adversarial probes — whether a prompt injection in a tool response gets through, whether it leaks PII, whether it will recite its own system prompt. The judge is an LLM, so treat the number as a screen rather than a proof; the signed credential is what makes the screen portable to someone who did not run it. Consensus judging across two or three models is what keeps a single judge's bad day from becoming the score.

What it is

A quality gate for agents and MCP servers. It connects to a target server, runs a three-level pipeline against it — manifest schema, then real tool calls, then calibrated domain questions — scores the result on six weighted axes, and issues a W3C Verifiable Credential as proof. The gap it aims at is that the ecosystem has identity, after-the-fact reputation and payments, but no way to check competence before you pay.

What you get

  • A full evaluation of a server at a URL: manifest checks, functional tool calls and judge scoring — check_quality
  • A fast answer when you need one in-line: a cached score, or a manifest-only pass — check_quality_fast
  • An earlier score looked up again with freshness decay applied, so a stale result reads as stale — get_score
  • A signed attestation verified and decoded, which is how the other side checks a claim you hand them — verify_attestation
  • Scoring on six axes with published weights: accuracy 35%, safety 20%, reliability 15%, process quality 10%, latency 10%, schema quality 10%
  • Five adversarial probe types in the safety pass — prompt injection, PII leakage, hallucination, overflow and system-prompt extraction

Requirements

Docker with Compose, and at least one LLM API key for the judge in .env — GROQ_API_KEY or CEREBRAS_API_KEY both have free tiers, and a fallback chain runs across seven providers. docker compose up -d starts the API on port 8002 and the MCP server on 8003; connect over SSE at http://localhost:8003/sse, or pip install mcp-agenttrust (0.1.2) for a stdio proxy that forwards to that running API. Note the shell-environment trap the README calls out: an exported GROQ_API_KEY overrides the rotation pool in your .env.

Setup effort

Run a container — build the image from the repository, then point your client at it

See installation options