conkurrence_bedrock_health
Test Bedrock model accessibility. Probes each model with a minimal invocation to verify IAM permissions and model enablement. Reports which models are accessible in the configured region.
LABSCO SUMMARY
Cost estimation as its own tool, before the run, is the feature that makes a multi-model evaluation something you can actually authorise.
Running the same schema across several providers multiplies spend in a way that is easy to underestimate and impossible to undo, and most harnesses find out the number afterwards. Putting the estimate in front of the run — and the provider and Bedrock health checks alongside it — means the decision to proceed is an informed one. conkurrence_suggest is the other tool doing real work: writing an evaluation schema from scratch is where most of these projects stall, and generating a candidate from sample items turns a blank page into an edit. The comparison and trend tools are what make it a practice rather than a one-off measurement, since agreement only means something against a previous number.
What it is
An inter-rater agreement evaluation harness: define a schema, run it across configured model providers, and read the agreement statistics — with cost estimation, templates, schema suggestion and trend analysis around it.
What you get
- conkurrence_estimate returns cost and token usage before a run rather than after, and conkurrence_run executes across whichever providers are configured.
- conkurrence_compare puts two runs side by side with agreement deltas and per-field changes, and conkurrence_trend does the same across many runs.
- conkurrence_suggest analyses sample items and proposes a schema, which is the hard part of setting up an evaluation; conkurrence_validate_schema checks one before it is used.
- conkurrence_templates lists available evaluation templates and returns the full contents of one.
- conkurrence_providers reports which provider keys are configured, and conkurrence_bedrock_health probes each Bedrock model with a minimal invocation rather than assuming availability.
- conkurrence_list_runs returns the runs held in the session with their ids and schemas, and conkurrence_report generates a markdown write-up from results.
Requirements
Provider API keys for whichever models you want to evaluate across — conkurrence_providers reports what it can see. A license is managed through conkurrence_license, which validates a key.
Setup effort
One command — npx -y conkurrence mcp
USE CASES
01
Getting an evaluation schema off the ground
Hand it sample items and get a schema proposal back, then validate it before spending anything on a run.
02
Knowing what a run will cost before authorising it
Estimate cost and token usage, check which providers are actually configured and reachable, and only then execute.
03
Seeing whether a change helped
Compare two runs side by side with agreement deltas and per-field changes, rather than reading two headline numbers.
04
Watching agreement over time
Analyse the trend across many runs, overall and per field, and generate the markdown report from the results.
12 tools
Counted once with only startup setup. Other versions, modes or optional credentials may expose a different list.
conkurrence_compare
Compare two Conkurrence runs side-by-side. Shows κ deltas, per-field changes, classification shifts, and cost differences. Provide run_ids or inline results.
conkurrence_estimate
Estimate the cost and token usage for a Conkurrence evaluation run before executing it.
conkurrence_license
Manage your ConKurrence license. Actions: "validate" checks a license key against the API, "activate" stores a key for this MCP session, "status" shows current license state (free runs remaining or active key info).
conkurrence_list_runs
List all Conkurrence runs stored in the current session. Shows run_id, schema name, timestamp, overall kappa, item count, and cost for each run. Use run_ids with other tools (compare, trend, report).
conkurrence_providers
Check which AI provider API keys are configured. Reports availability of Bedrock (AWS credential chain), OpenAI (OPENAI_API_KEY), Gemini (GOOGLE_API_KEY or GEMINI_API_KEY), and MCP Sampling (host model). Also documents the model prefix convention.
conkurrence_report
Generate a detailed markdown report from Conkurrence results. Provide either a run_id (from a previous conkurrence_run) or inline results.
conkurrence_run
Execute a Conkurrence evaluation. If multiple model providers are configured, measures inter-rater consensus across models (Fleiss κ). If no API credentials are available, automatically uses the host model for a self-consistency check (5 runs by default). Returns a summary with run_id — full results available via conkurrence://results/{run_id} resource.
conkurrence_suggest
Analyze sample items and suggest an evaluation schema. Returns field suggestions with reliability estimates. Use before creating a schema to get AI-guided field design. Supports multi-provider models: Bedrock (default), OpenAI ("openai." prefix), Gemini ("gemini." prefix). If no API credentials are configured, automatically uses MCP Sampling transport (host model).
conkurrence_templates
List available evaluation templates or get the full contents of a specific template. Templates include schema, sample data, and config files.
conkurrence_trend
Analyze trends across multiple Conkurrence runs. Shows overall and per-field kappa progression, degradation alerts, schema evolution, and classification distribution over time. Provide run_ids or inline results. Requires 2+ runs.
conkurrence_validate_schema
Validate an evaluation schema for use with Conkurrence. Returns validation results including any errors or warnings.
BEFORE YOU INSTALL
What to know first
Tools you get
12 tools when we connected.
Requires
npx on PATH
CONTEXT COST
Light Standing cost
Rated by the size of the 12 tool definitions that ride along on every turn, not by how many there are
INSTALLATION
Connect your client
Add the block to your client, or run the command directly.
Server setup
Configuration file
JSON
{
"mcpServers": {
"conkurrence": {
"command": "npx",
"args": [
"-y",
"conkurrence",
"mcp"
]
}
}
}npx
TERMINAL
npx -y conkurrence mcp
