Accuracy carries the most weight, but the part you cannot get from reading a README is how a server behaves under the five adversarial probes — whether a prompt injection in a tool response gets through, whether it leaks PII, whether it will recite its own system prompt. The judge is an LLM, so treat the number as a screen rather than a proof; the signed credential is what makes the screen portable to someone who did not run it. Consensus judging across two or three models is what keeps a single judge's bad day from becoming the score.
A quality gate for agents and MCP servers. It connects to a target server, runs a three-level pipeline against it — manifest schema, then real tool calls, then calibrated domain questions — scores the result on six weighted axes, and issues a W3C Verifiable Credential as proof. The gap it aims at is that the ecosystem has identity, after-the-fact reputation and payments, but no way to check competence before you pay.
- A full evaluation of a server at a URL: manifest checks, functional tool calls and judge scoring — `check_quality`
- A fast answer when you need one in-line: a cached score, or a manifest-only pass — `check_quality_fast`
- An earlier score looked up again with freshness decay applied, so a stale result reads as stale — `get_score`
- A signed attestation verified and decoded, which is how the other side checks a claim you hand them — `verify_attestation`
- Scoring on six axes with published weights: accuracy 35%, safety 20%, reliability 15%, process quality 10%, latency 10%, schema quality 10%
- Five adversarial probe types in the safety pass — prompt injection, PII leakage, hallucination, overflow and system-prompt extraction
At least one LLM API key for the judge — `GROQ_API_KEY` or `CEREBRAS_API_KEY` are both listed with free tiers, and a fallback chain runs across seven providers. `MONGODB_URL` is optional and adds persistent storage. The package is `mcp-agenttrust` (0.1.2) over stdio; the full service also runs from Docker Compose, with the API on port 8002 and the MCP server on 8003, and can be reached over SSE at `http://localhost:8003/sse`. Note the shell-environment trap the README calls out: an exported `GROQ_API_KEY` overrides the rotation pool in your `.env`.
One command plus a key — pip install mcp-agenttrust, then supply credentials