Running the same schema across several providers multiplies spend in a way that is easy to underestimate and impossible to undo, and most harnesses find out the number afterwards. Putting the estimate in front of the run — and the provider and Bedrock health checks alongside it — means the decision to proceed is an informed one. conkurrence_suggest is the other tool doing real work: writing an evaluation schema from scratch is where most of these projects stall, and generating a candidate from sample items turns a blank page into an edit. The comparison and trend tools are what make it a practice rather than a one-off measurement, since agreement only means something against a previous number.
An inter-rater agreement evaluation harness: define a schema, run it across configured model providers, and read the agreement statistics — with cost estimation, templates, schema suggestion and trend analysis around it.
- conkurrence_estimate returns cost and token usage before a run rather than after, and conkurrence_run executes across whichever providers are configured.
- conkurrence_compare puts two runs side by side with agreement deltas and per-field changes, and conkurrence_trend does the same across many runs.
- conkurrence_suggest analyses sample items and proposes a schema, which is the hard part of setting up an evaluation; conkurrence_validate_schema checks one before it is used.
- conkurrence_templates lists available evaluation templates and returns the full contents of one.
- conkurrence_providers reports which provider keys are configured, and conkurrence_bedrock_health probes each Bedrock model with a minimal invocation rather than assuming availability.
- conkurrence_list_runs returns the runs held in the session with their ids and schemas, and conkurrence_report generates a markdown write-up from results.
Provider API keys for whichever models you want to evaluate across — conkurrence_providers reports what it can see. A license is managed through conkurrence_license, which validates a key.
One command — npx -y conkurrence mcp