Evaluation suites rot because writing cases is tedious and happens outside the tool where the failure was noticed. With create_test_case reachable from the same conversation where you just watched an agent get something wrong, the case gets written while the example is in front of you. The rest of the surface is the ordinary CRUD that makes that possible; consult_sofia is the odd one — a vendor assistant exposed as a tool, which is either convenient or a layer of indirection depending on your taste.
Coval's endpoint over their agent evaluation platform. The tools mirror the domain: agents under test, test sets and cases, runs against them, and the metrics that come out.
- list_agents, get_agent, create_agent, update_agent — the agents being evaluated
- list_test_sets, get_test_set, create_test_set and the matching test-case tools — build the suite from the conversation
- list_runs, get_run, create_run — start an evaluation and read what came back
- list_metrics, get_metric — the measurements over those runs
- consult_sofia — Coval's own assistant, reachable as a tool
OAuth 2.0 with PKCE, refresh tokens and dynamic client registration through Clerk. Tokens are org-scoped.
Paste a URL, then authorize — add the endpoint to your client, then approve the OAuth consent screen