consult_sofiaLABSCO SUMMARY
Creating test cases from a conversation is the workflow this unlocks.
Evaluation suites rot because writing cases is tedious and happens outside the tool where the failure was noticed. With create_test_case reachable from the same conversation where you just watched an agent get something wrong, the case gets written while the example is in front of you. The rest of the surface is the ordinary CRUD that makes that possible; consult_sofia is the odd one — a vendor assistant exposed as a tool, which is either convenient or a layer of indirection depending on your taste.
What it is
Coval's endpoint over their agent evaluation platform. The tools mirror the domain: agents under test, test sets and cases, runs against them, and the metrics that come out.
What you get
- list_agents, get_agent, create_agent, update_agent — the agents being evaluated
- list_test_sets, get_test_set, create_test_set and the matching test-case tools — build the suite from the conversation
- list_runs, get_run, create_run — start an evaluation and read what came back
- list_metrics, get_metric — the measurements over those runs
- consult_sofia — Coval's own assistant, reachable as a tool
Requirements
OAuth 2.0 with PKCE, refresh tokens and dynamic client registration through Clerk. Tokens are org-scoped.
Setup effort
Paste a URL, then authorize — add the endpoint to your client, then approve the OAuth consent screen
USE CASES
01
Turning a failure into a test
Create the test case in the set while the example is still in front of you.
02
Running an evaluation
Kick off the run against the agent and read the result rather than watching a dashboard.
03
Checking whether a change helped
Pull the metrics across runs instead of comparing by eye.
19 tools
Listed in project docs. Your client may show a different list.
list_runsget_runcreate_runlist_agentsget_agentcreate_agentupdate_agentlist_test_setsget_test_setcreate_test_setlist_test_casesget_test_casecreate_test_caseupdate_test_caselist_metricsget_metriclist_personasget_personaBEFORE YOU INSTALL
What to know first
Tools you get
19 tools named in the project's documentation.
Requires
- an MCP client that can reach a remote endpoint
- OAuth 2.0 authorization_code + refresh_token, PKCE S256, dynamic client registration (Clerk issuer https://clerk.coval.dev); org-scoped tokens
INSTALLATION
Connect your client
Add the endpoint to your client, then approve the OAuth consent screen.
Server setup
ENDPOINT URL
https://mcp.coval.dev/mcpRequirements
- OAuth 2.0 authorization_code + refresh_token, PKCE S256, dynamic client registration (Clerk issuer https://clerk.coval.dev); org-scoped tokens
