eval-guide

The orient dashboard is pre-built and shipped with the skill — dashboard/orient-dashboard.html. It is identical for every agent (the maturity model and "what you walk away with" are agent-agnostic), so there is no per-session JSON write and no Python launch. Don't ask for the agent name yet — Stage 0 captures it where it's actually needed for deliverable filenames.

1. Open the static dashboard in the customer's default browser. Use the OS launcher and the install-resolved path:

   Copy

   ORIENT_HTML="$(ls ~/.claude/skills/eval-guide/dashboard/orient-dashboard.html 2>/dev/null || ls ~/.claude/plugins/cache/*/eval-guide/*/skills/eval-guide/dashboard/orient-dashboard.html 2>/dev/null | head -1)"
   case "$(uname -s 2>/dev/null)" in
     Darwin) open "$ORIENT_HTML" ;;
     Linux)  xdg-open "$ORIENT_HTML" ;;
     *)      cmd.exe /C start "" "$ORIENT_HTML" ;;  # Windows / Git Bash
   esac

   The ls ... | head -1 fallback resolves the file regardless of install location — user-global skills first (~/.claude/skills/eval-guide/), plugin-cache second.

   For dev installs (skill checked out at an arbitrary path, not in ~/.claude/), the AI should know the absolute path of the SKILL.md it's reading and substitute <SKILL.md-dir>/dashboard/orient-dashboard.html.

   This is a read-only stage. There is no feedback file, no confirmation gate, and no serve.py involvement. The customer reviews the snapshot in the browser while the conversation continues in chat.

2. While the dashboard is open, narrate one sentence in chat: "This is the eval maturity model — five pillars of eval practice, five levels each. Today's session takes Pillars 1, 2, and 4 to L300 Systematic; Pillars 3 and 5 reach L200 Defined via the reference protocols you'll get at the end."

3. Proceed to Stage 0 (Discover) without waiting. The dashboard is informational.

When to rebuild the static HTML: if templates/orient.html, templates/base.html, or examples/stage-orient-data.json change, run python dashboard/build-orient.py once and check in the regenerated orient-dashboard.html. The build script reuses serve.py's generate_html, so the rendering stays consistent with the live dashboards.

Why this matters for the customer: The maturity model is the value moment. Without it, the customer sees a series of stages with no map. With it, they understand exactly what they're getting and what comes next — the eval-first message lands because they can see the full journey.

Skip orient when: the customer has already done a session with the toolkit and is returning for a Stage 1 / Stage 2 / Stage 4 jump-in. Don't re-orient someone who already has the map.

• A 1-page Agent Vision — purpose, users, knowledge sources, core capabilities, boundaries (what the agent must NOT do), success criteria, role-based access, and the eval objective + risk tier + owner (playbook Step 1). Written down, not assumed.
• The eval objective — one sentence naming what "good" looks like and what decisions the evals will inform. It anchors every later choice.
• The agent's risk tier — HIGH / MEDIUM / LOW, classified from the five risk factors: reach, criticality of error, autonomy / blast radius, regulatory exposure, data sensitivity. The risk tier drives pass-rate targets, gate strictness, required trust & safety categories, and minimum adversarial coverage downstream.
• A named owner — one person accountable for authoring the eval, reviewing results, and signing off.
• Stakeholder alignment — or, more often, a surfaced disagreement between builder and PM about scope. 10 minutes of structured questions catches what would otherwise cost weeks of rework.
• The spec every later stage depends on. The Plan stage's eval plan, the Generate stage's test cases, and the Interpret stage's pass/fail judgment all trace back to what gets named here.

• You already have a written PRD, agent spec, or design doc that covers all 7 questions below. Bring it and skip to Stage 1.
• You have eval results in hand and need triage now — go straight to Stage 4.
• Your agent is a 50-topic monster. One Stage 0 pass won't fit; run Stage 0 per top-level capability.

Don't ask Q1–Q7 in chat. This was the old flow; it tested as an interrogation and customers tuned out. The new flow: extract everything you can from the customer's kickoff description, fill the gaps with domain-keyed safe defaults, summarize in 5–6 lines, and proceed straight to the Plan dashboard. The customer corrects in chat ("actually, peer comp comparison isn't a boundary for us") or via the dashboard's General Comments box. Nothing is locked until they confirm in the dashboard.

From the customer's 1–4 sentence description, extract:

• Purpose — usually the first clause ("Personalized HR support…")
• Users — usually implied ("employees," "customers," "internal teams")
• Capabilities — usually a list ("benefits, training, policies")
• Knowledge sources — sometimes named, often categorized ("official company resources" → SharePoint TBD)
• Tone hints — sometimes explicit ("trusted HR colleague," "efficient")
• Personalization hints — words like "personalized," "your," "based on your role"

If the kickoff is too thin (one sentence with no domain hint), ask one clarifying question — "Two more sentences on what it does and who uses it would help me draft a Vision faster" — then resume.

Domain detection runs on keywords in the kickoff description. Pick the matching default set:

The default tier is a starting point keyed on domain. Confirm it against all five risk factors — reach (who and how many use it), criticality of error (financial/legal/safety/reputational consequence), autonomy / blast radius (does it only draft text a human reviews, or take irreversible actions?), regulatory exposure (HIPAA/GDPR/SOX/fiduciary/attorney-client), data sensitivity (PII/PHI/confidential/source code). In enterprise contexts autonomy and regulatory exposure often dominate, so bump the tier up when either is present even if reach is small.

Default success criteria (always include unless customer overrides):

• Most user questions answered directly (deflection / self-service rate)
• Out-of-scope questions routed clearly to the right human or resource (graceful handoff)
• Zero privacy / boundary breaches

Default knowledge sources when only categorized:

• "some SharePoint sites" / "internal docs" → flag as Multiple SharePoint sites (TBD — name in Plan dashboard) so the customer can fill names without us blocking on it.

Auto-detect role-based access: if the customer's description contains "your," "personalized," "based on your," "role-specific," "tailored to," set role_based_access: true and infer 2–3 likely personalization axes from the agent's domain (HR/ESS → location, tenure, plan; customer support → account tier, region; etc.). Customer corrects if wrong.

Marketing-language capabilities like "empower employees," "explore opportunities," "streamline X" don't survive the concreteness check. Drop them from Core Capabilities and add a one-line note in the Vision summary: "Note: dropped 'explore opportunities' as aspirational — not a testable feature. Tell me if it's actually a concrete capability and I'll add it back."

This is silent removal with a flagged note, not a question. Customer can flag if they disagree.

Display the pre-extracted Vision compactly:

Agent Vision: [Name]

Eval objective: [one sentence — what "good" means + what decision the evals inform]
Purpose:        [one sentence from kickoff]
Users:          [extracted or default]
Knowledge:      [named sources, or "TBD — confirm in Plan dashboard"]
Capabilities:   [3–5 from kickoff, aspirational dropped]
Boundaries:     [domain default set, listed]
Success:        [default 3 criteria]
Role-based:     [auto-detected: yes/no, with axes]
Risk tier:      [domain default: HIGH/MEDIUM/LOW] — driven by 5 factors (reach, criticality, autonomy, regulatory, data sensitivity)
Owner:          [named accountable owner, or "TBD — name before deploy"]

Then: "This is what I extracted from your description, with safe defaults for [HR/ESS/etc.] domain agents filling the gaps. Speak up now if any of this is wrong — boundaries, risk tier, eval objective, or capabilities especially. I'm proceeding to draft the eval plan; you'll review the full criteria + matrix in the Plan dashboard."

Don't gate on customer confirmation. Write stage-0-data.json and proceed to Stage 1 immediately. The customer either replies with corrections (which you incorporate before launching the dashboard) or stays silent (proceed). The Plan dashboard is the real review surface.

• Pre-extraction + defaults covers ~80% of what the chat questions extracted, with zero customer chat input beyond the kickoff.
• Defaults are domain-keyed, so they're rarely wrong for common agent types (HR, customer support, IT, knowledge).
• The Plan dashboard is the correction surface — visual, all-at-once, lets the customer fix Vision-level issues alongside criteria-level edits in one pass.
• Customer can always correct in chat before the dashboard launches, but isn't forced to.

• The kickoff description is genuinely too thin — one sentence with no domain keywords. Ask one clarifying question to get enough material for safe defaults.
• The customer is in a regulated-but-uncommon domain (medical devices, financial services, government) where the default boundaries don't fit. After step 2, ask: "Domain looks like [X] — your boundaries are usually [Y]. Anything specific I should add for your context?"
• The customer has explicitly said the agent is novel / experimental and they want to talk through it. Default to conversation mode for these — but they're a small minority.

• A populated copy of the Eval Suite Planning & Logging Template for stakeholder review and ongoing use.
• An interactive HTML review page that summarizes the workbook, filters eval sets, surfaces TBDs, and keeps the chat response short.
• One row per eval set in the registry: capability sets, trust & safety sets, targets/gates, intended use, cadence, human inputs, source dependencies, grader-validation notes, and reusable-asset flags.
• Baseline placeholders and reusable-asset candidates in the existing template tabs, without adding or changing workbook structure.

• You do not have the Eval Suite Template and need a final customer artifact now. Ask for the blank template first.
• You already have a populated registry and only need test cases. Skip to Generate.
• You only need to triage run results. Skip to Interpret.

1. Copy and preserve the template.

   Use the attached Eval Suite Template workbook when available. Populate a copy of it only. Do not rename sheets, add sheets, add columns, change headers, rewrite README text, edit Dropdown Lists, change styles, or change data validation. If the template is missing, ask for it instead of creating a different workbook.

2. Populate Step 1 planning.

   Fill the existing 1 . Planning input cells:

   • agent identity;
   • one-sentence eval objective;
   • five-factor risk tier: reach, criticality of error, autonomy/blast radius, regulatory/compliance exposure, data sensitivity;
   • owners and roles;
   • deployment gates / sign-off criteria.

3. Define eval sets, not scenarios.

   Populate 2 . Eval Suite Registry with one row per eval set:

   • Capability sets: accuracy/correctness, faithfulness/groundedness, relevancy, style/tone, reasoning/tool use as applicable. Hallucination stays in faithfulness/groundedness.
   • Trust & Safety sets: guardrails, out-of-scope handling, sensitive-data handling, prompt injection/jailbreak resilience, compliance-specific behavior as applicable.

   Do not generate legacy planning-artifact rows in the workbook. The registry is one row per eval set only.

4. Apply v5 Step 4 gates and improvement targets.

   Use the existing registry columns:

   • Trust & Safety sets: absolute pass-rate hard gates, usually near 100%.
   • Capability sets: launch floor for first deployment plus regression/direction after baseline, not a standing absolute target.
   • High-risk capabilities: explicit hard floor when the capability functions like a guardrail.
   • Put the nuance in Target pass rate, Target rationale, Gate type, Intended use, Run cadence, and Notes; do not add a new column.

5. Specify Step 5 human inputs.

   Use the registry columns for human input type/author, grounding source dependency, and source-change review. Use TBD - confirm before baseline where owners or sources are unknown.

6. Plan Step 6 grader validation.

   The template has no grader-validation columns. Do not add them. Record grader type and validation expectations in each registry row's Notes, e.g. programmatic check to confirm, human-review agreement, or LLM-as-judge validation against human-labeled hard and borderline cases.

7. Seed Step 7 baseline placeholders only in existing Run Log columns.

   Add optional baseline placeholder rows in 3 . Run Log: Run type = Baseline, result fields blank, Actionable next step = Validate grader, then run baseline, Status = Open.

8. Partition Step 8 regression cadence.

   Use Intended use and Run cadence in the registry. Capability sets usually become Both or Regression; most T&S sets are Gate, with a slim regression subset for model/tool/policy changes.

9. Flag Step 10 reusable assets.

   Populate 4 . Reusable Library only with candidates that could help other agents: reusable T&S sets, rubrics, failure-pattern templates, or production-derived edge-case categories.

Do not display a long eval-set summary in chat. Put Step 1 objective/risk/owner, capability eval sets, trust & safety eval sets, Step 4 governance, Step 5 human inputs, Step 6 grader-validation notes, Step 8 cadence, and Step 10 reusable candidates into the interactive HTML review page described below.

The customer payoff: "You now have a workbook your PM, builder, risk owner, and source owners can review. It preserves your template and shows which eval sets exist, how each is governed, who owns human inputs, what must happen before baseline, and which assets may be reusable."

Maturity callout — Pillar 1 / playbook Step 1 (L100 Initial → L300 Systematic): Discover + Plan advance Pillar 1 from "good lives in the builder's head" to a written objective, five-factor risk tier, accountable owner, and workbook-backed eval-set governance. Pillar 2 advances in Generate (Steps 2, 3, 5); Pillar 3 now starts with Step 6 grader validation before any baseline is trusted.

The legacy Plan dashboard is pre-v5 criteria-based and must not be used for the v5 workbook workflow. Instead, generate a draft workbook copy plus a companion HTML review page and have the customer review those artifacts.

1. Generate eval-suite-<agent-name>-<YYYY-MM-DD>.xlsx as a populated copy of the user's template.
2. Generate eval-suite-<agent-name>-<YYYY-MM-DD>-review.html next to it using skills/eval-guide/plan-review-page.md.
3. Ask the customer to review the workbook and the HTML page:
   • 1 . Planning: objective, risk tier, owners, sign-off criteria.
   • 2 . Eval Suite Registry: eval-set rows, Step 4 governance, cadence, human inputs, source dependencies, grader-validation notes, reusable flags.
   • 3 . Run Log: baseline placeholders, if added.
   • 4 . Reusable Library: reusable candidates.
4. Apply workbook feedback by editing cell values in a new copy of the workbook and regenerating the HTML review page. Do not edit structure, sheet names, headers, styles, README, or Dropdown Lists.
5. When the customer confirms the workbook, treat it as the Plan artifact and proceed to Generate from the registry.

After confirmation, the eval plan deliverable is:

Customer-ready .xlsx eval-suite planning workbook using the /xlsx skill, named eval-suite-<agent-name>-<YYYY-MM-DD>.xlsx. It must be a populated copy of the user's template, not a recreated or redesigned spreadsheet.

Interactive HTML review page named eval-suite-<agent-name>-<YYYY-MM-DD>-review.html. The page carries the summary, eval-set explorer, TBD action list, and human review checklist so the chat response stays concise.

The workbook is the review checkpoint and the primary Plan artifact. A separate .docx narrative is optional only if the user asks for it.

Tell the customer only where to open the workbook and HTML review page. Do not duplicate the HTML page content in chat.

The kit is one deliverable. CSVs go to Copilot Studio. The test-case .docx goes to your PM. The setup guide, rerun protocol, and baseline-comparison workbook go to your eval-process docs.

• You already have a test set you trust. Bring it; skip to Stage 3.
• You have production traffic. Sample real conversations directly into a test set rather than synthesizing — generated cases anchor to AI voice; real user language beats it.
• You're testing agent UX (turn-taking, error-recovery flow). That's conversation testing, not eval — different tool.

Default to Single Response. ~80% of agents are single-response Q&A. Conversation (multi-turn) only fits agents that do real multi-step workflows — troubleshooting flows, form-filling, slot-extracting conversations. If you're not sure, you don't need Conversation mode.

When to switch to conversation eval:

• The agent walks users through multi-step processes (e.g., troubleshooting, onboarding, form completion)
• Context retention matters — later answers depend on earlier ones
• The agent needs to ask clarifying questions before answering
• The criterion involves slot-filling or information gathering across turns

When to stay with single response (the default):

• Each question is independent (FAQ, policy lookup, data retrieval)
• You need Compare meaning, Text similarity, or Exact match (conversation mode doesn't support these)
• You need more than 20 test cases in a set

Explain the choice: "I'm recommending single response eval for your knowledge-lookup criteria because each question is independent — the agent doesn't need previous context to answer. For your troubleshooting criterion, I'm recommending conversation eval because the agent needs to gather information across multiple turns before resolving the issue."

Note for CSV generation: Single response test sets use the 2-column import CSV (Question, Expected response); the testing method is assigned per row in Copilot Studio's Evaluate tab after import (see the manifest note below). Conversation test sets can be imported via spreadsheet or generated in the Copilot Studio UI — each test case contains a sequence of user messages that simulate a multi-turn interaction.

If the Agent Vision has role_based_access: true (set in Discover), the test cases for personalization criteria need user profiles in Copilot Studio. Without profiles, the agent has no context to personalize from — and the test results are misleading.

Walk the customer through this BEFORE generating cases:

1. Identify which criteria need profiles. Check the criteria list for ones that test personalization (criteria mentioning "for the employee's [attribute]" — office, tenure, plan, role, etc.).

2. Draft 3 user profiles that span the personalization axes. Pick combinations that exercise different paths:

   • Profile A: one attribute combo (e.g., Boston-2yr-PPO)
   • Profile B: a contrasting combo (e.g., Seattle-7yr-HMO)
   • Profile C: an edge combo (e.g., Remote-FirstYear-HDHP)
   • Each profile has explicit attribute values and a one-line note on which criteria it exercises.

3. Tell the customer to create the profiles in Copilot Studio (Settings → Evaluation → User Profiles) before importing test sets. The CSV import won't fail without profiles, but personalization-criterion results will be misleading.

4. Flag the two known limitations:

   • Multi-profile eval doesn't work with connector-based agents. If the Vision includes any tool/connector use, multi-profile eval can't run against those criteria — fall back to standard cases without profile context.
   • Multi-profile eval is not available in GCC. Ask the customer's tenant type: standard or GCC. If GCC, drop personalization test cases or run them as standard cases (lose the personalization signal).

5. Generate one test case set per criterion per profile, OR a single set with profile-tagged expected responses (when criterion is the same question, different expected answer). Use whichever is more efficient.

If role_based_access: false, skip this branch entirely — no profile setup needed.

When generating expected responses, the AI wraps factual content it can't independently confirm in [VERIFY: ...] markers. These are the failures-in-waiting. A wrong [VERIFY] becomes an eval test case that "passes" while hiding a production failure — the agent matches the bogus expected response and gets a green check.

The dashboard highlights every [VERIFY] span in yellow. Read every one before approving. This is the customer's most important responsibility in Stage 2; the LLM that drafted the test cases cannot do this work — only the human who knows the actual knowledge sources can.

When narrating to the customer, say: "I've wrapped factual claims I'm guessing at in [VERIFY] markers. Please check each one against your real knowledge source — these are the most likely places the eval will lie to you about agent quality."

1. Generate one or more test cases per acceptance criterion from the plan. For conversation criteria, generate multi-turn test cases with realistic dialogue sequences (up to 6 Q&A pairs). A single criterion can and often should have multiple test cases exercising different phrasings, user contexts, and edge inputs.

2. Write expected responses so they satisfy the criterion's pass condition — i.e., what the agent SHOULD say according to the Agent Vision, the criterion's statement, and its pass_condition. Note: "These expected responses reflect your stated requirements. Refine them once the agent is built and you see how it actually responds."

3. Group by eval set into separate CSV files:

   • eval-capability-accuracy-correctness.csv
   • eval-capability-faithfulness-groundedness.csv
   • eval-capability-reasoning-tool-use.csv
   • eval-trust-safety-sensitive-data-handling.csv
   • eval-trust-safety-prompt-injection-jailbreak.csv
   • eval-trust-safety-compliance-specific.csv (if applicable)

   Only create files for categories that apply.

   Versioning: Name each file with a date stamp or agent version (e.g., eval-knowledge-accuracy-2026-04-22.csv) so successive sessions produce a version history rather than overwriting the baseline. Versioning is a requirement of L300 Systematic Pillar 2.

4. CSV format — Copilot Studio import format is exactly two columns:

"Question","Expected response"
"How many PTO days do LA employees get?","LA employees receive 18 PTO days per year."

The Testing method is NOT a CSV column — it is assigned per row in Copilot Studio's Evaluate tab after import (see the manifest note below). The method chosen for each criterion travels in the companion .docx manifest and the eval-setup-guide-<agent>-<date>.docx, which walk the customer through the manual per-row assignment. Valid Testing method values (assigned in the UI): General quality, Compare meaning, Text similarity, Exact match, Keyword match (core five), plus Capability use and Custom (extensions). (A 3-column -with-methods variant may be emitted as a human-readable reference only — never import it.)

5. Inherit the method from each criterion — the method was set in Stage 1 and should carry through to every test case for that criterion. If a criterion's method doesn't fit a specific test case (e.g., one particular case needs exact-keyword verification while the rest use semantic match), override per-case rather than rewriting the criterion. Refresher table:

6. Highlight the value: "You now have [X] test cases across [Y] eval sets from your workbook registry. Compare that to the 5–10 happy-path prompts most customers start with. These include adversarial attacks, hallucination traps, robustness tests, and edge cases your users will encounter in production."

Display a summary table of test cases per eval set.

The customer payoff: "You now have a test suite that imports directly into Copilot Studio, plus the .docx report your PM can sign off on, plus the Pillar 3 and Pillar 5 starter artifacts you'll keep for ongoing operations. That's the eval kit a new team member would need to evaluate this agent — questions, expected responses, methods, re-run protocol, comparison template."

Maturity callout — Pillar 2 / playbook Steps 2, 3, 5 (L100 Initial → L300 Systematic): Generate advances Pillar 2 from "no established eval set" to versioned capability eval sets and separate trust & safety eval sets, coverage mapped to risk and value, each tagged gate-only | regression | exploratory for the regression suite (Step 8). Pillar 4 advances in Interpret. Pillars 3 and 5 reach L200 Defined via the rerun-protocol-<agent>-<date>.docx and baseline-comparison-<agent>-<date>.xlsx starter artifacts generated at session close — surface these to the customer when delivering them.

Before generating final CSV and report files, launch the test cases dashboard for review:

1. Write the test cases to stage-2-data.json. Methods and governance metadata live at the eval-set (test_set) level, inherited from the workbook registry.

   Copy

   {
     "agent_name": "...",
     "test_sets": [
       {
         "eval_set_id": "CAP-ACC-001",
         "display_name": "Policy answer correctness",
         "set_type": "capability",
         "capability_dimension": "Accuracy / correctness",
         "methods": ["Compare meaning", "Keyword match"],
         "gate_type": "Hard floor + soft target",
         "target_pass_rate": "Launch floor 90%; regression/direction after baseline",
         "run_cadence": "Weekly",
         "cases": [
           {
             "id": 1,
             "question": "...",
             "expected_responses": {
               "Compare meaning": "Canonical answer, with [VERIFY: factual content to check] markers",
               "Keyword match": "PTO, Time Off Policy, accrual"
             },
             "custom_rubric": ""
           }
         ]
       }
     ]
   }

   Key requirements:

   • Group test cases by workbook eval set, with cases nested directly under each set.
   • Each test set carries a methods: [] array — the methods for this eval set's CSV. Choose one method when one fits; choose multiple only when the eval set genuinely needs them. Default to one method.
   • Each test set carries workbook governance metadata (gate_type, target_pass_rate, target_rationale, run_cadence, owner/source/grader notes where available).
   • Each case has expected_responses: { method → value } — one entry per method in the eval set's methods array that needs a per-case reference (Compare meaning, Text similarity, Exact match, Keyword match). Methods that grade against a set-level rubric (General quality, Capability use, Custom) do NOT need entries.
   • Wrap AI-generated factual content in [VERIFY: ...] markers inside the Compare meaning / Text similarity entries so the dashboard highlights them for review.
   • Custom method in the eval set: also write a custom_rubric field on each set or case — a short LLM-judge rubric drafted from the eval-set purpose and expected behavior ("Rate the response Pass / Fail. Pass = …. Fail = …. Output PASS or FAIL with a one-sentence reason."). The dashboard shows this as an editable textarea. Don't leave Custom sets without a rubric.
   • Keyword match method: the per-case expected_responses["Keyword match"] value is a comma-separated keyword list (not a reference answer). The dashboard renders this as a "Keywords" column.

2. Launch the dashboard:

   Copy

   python "$(ls ~/.claude/skills/eval-guide/dashboard/serve.py 2>/dev/null || ls ~/.claude/plugins/cache/*/eval-guide/*/skills/eval-guide/dashboard/serve.py 2>/dev/null | head -1)" --stage generate --serve --data stage-2-data.json

3. The user reviews the Eval Sets Overview at the top, then walks the stacked eval-set sections. Per eval set: edits the Test Methods to Use chips (set-level), checks gate/target/cadence metadata, edits Custom rubric callouts if Custom is used, edits per-method columns in the cases table, checks VERIFY-highlighted factual content, and adds/removes test cases.

4. When the user confirms, parse the feedback from the bash stdout between the ===EVAL_GUIDE_FEEDBACK_BEGIN=== / ===EVAL_GUIDE_FEEDBACK_END=== markers. Apply every edit it contains, faithfully and without question. The customer's choices are final — do NOT re-litigate, do NOT suggest reverting, do NOT ask for confirmation again, do NOT partially apply. (generate-feedback.json is also on disk as a backup, but stdout is the primary channel.)

   This applies to ALL edit types:

   • [VERIFY] span corrections (the customer fact-checked your draft against their real knowledge sources — their version wins). At export time (CSV + .docx), strip every remaining [VERIFY: …] wrapper: [VERIFY: <content>] → <content>. By the time the customer has confirmed, every span is either edited (already clean) or accepted (marker is now noise).
   • Question edits
   • Per-method per-case expected-response edits — keyed by method: test_sets[i].cases[k].expected_responses["Compare meaning"], test_sets[i].cases[k].expected_responses["Keyword match"], etc. Each method's value updates that method's column for that case.
   • Custom-method rubric edits (test_sets[i].custom_rubric or test_sets[i].cases[k].custom_rubric) — the customer's refined rubric is final; use it as the LLM judge prompt verbatim.
   • Eval-set-level method additions / removals (test_sets[i].methods) — adding/removing a method changes which columns and rubric blocks render for that eval set.
   • Test case additions and deletions.
   • General Comments box content.

   Then narrate the edits back so the customer sees their changes were captured — count [VERIFY] corrections, count test case additions/deletions, list significant edits, restate updated total case count. Example: "Got it — 8 [VERIFY] corrections captured, 2 new cases for CAP-ACC-001, total now 56 cases across 7 eval sets." Don't just say "applied." The narration confirms you parsed correctly; it is NOT an invitation to re-decide.

   If changes requested instead of confirmed, regenerate and re-launch.

5. After confirmation, automatically generate ALL FIVE deliverables (A through E) — do not wait for the user to ask, do not ask "should I generate the docx now?", do not generate them in stages. The CSVs, the test-case .docx report, the eval-setup-guide .docx, the rerun-protocol .docx, and the baseline-comparison .xlsx are one delivery, produced together. The customer should see the artifact list in chat ("five files generated") and find the files on disk before they say anything more.

A. CSV files — One CSV per eval set: eval-<set-type>-<set-slug>-<date>.csv. Exactly two columns:

"Question","Expected response"

No Testing method column. Copilot Studio's Evaluation tab requires the customer to set the testing method manually per row in the UI after import — it is not pre-encoded in the CSV. The companion eval-setup-guide-<agent>-<date>.docx (deliverable E below) walks the customer through that manual step in detail.

Row generation rule. One row per active case per eval set (no case × method explosion). Per row:

• Question = the case's question.
• Expected response = whichever of the case's expected_responses is most informational, picked by this priority order against the eval set's method set:
  1. Compare meaning → case.expected_responses["Compare meaning"].
  2. Text similarity → case.expected_responses["Text similarity"].
  3. Exact match → case.expected_responses["Exact match"].
  4. Keyword match → case.expected_responses["Keyword match"] (comma-separated keyword list).
  5. None of the above (signal only has reference-free methods like General quality / Custom / Capability use) → leave the cell empty.

Strip every [VERIFY: …] marker from the cell value before writing the row. Replace [VERIFY: <content>] → <content>. The markers exist only as a review aid in the dashboard — by the time the customer has clicked Approve, every span has either been confirmed or edited. The CSV is the eval set the customer is importing into Copilot Studio; it must contain clean expected responses with no review-tooling syntax. Apply the regex \[VERIFY:\s*([^\]]*)\] → $1 (or equivalent) to every Expected response cell before emitting the row.

The customer can still edit any cell in CPS or in the CSV before import — for example, switching a row from canonical-answer to keyword-list when they decide that row should use Keyword match. The eval-setup-guide.docx makes this explicit.

An eval set with 12 cases produces exactly 12 rows. (No multiplication by methods.)

Tell the customer: "One CSV per eval set — two columns: Question and Expected response. Import each into Copilot Studio's Evaluation tab. Then in the CPS UI, set the Testing method for every row — this is a manual step. The eval-setup-guide.docx walks you through which method to pick per eval set and what threshold or regression rule to use."

B. .docx report — Generate a customer-ready report using the /docx skill. The report must be:

• Concise — no filler, no walls of text. Tables over paragraphs.
• Presentable — professional formatting with color-coded headers, clean tables, visual hierarchy
• Self-contained — a customer who wasn't in the conversation can read it and understand the eval plan + test cases

Report structure:

1. Agent Vision summary (from Stage 0) — 5-6 lines max

2. Workbook registry summary — eval sets grouped by Capability vs Trust & Safety, with Step 4 governance and Step 8 cadence

3. Test cases organized by eval set, with set-level target/gate/regression metadata

4. For each test case: Question, Expected Response, and suggested test method. Strip [VERIFY: …] markers the same way as in the CSV — [VERIFY: <content>] → <content>. The dashboard's review markers don't belong in the customer-facing report.

5. Summary table: eval set, category, test case count, methods

6. "What these tests catch" callout — 3-4 bullet points on what the customer would have missed

7. Next steps — what to do with these files. Always include a pointer line: "You're also receiving three companion artifacts (generated below) — eval-setup-guide-<agent>-<date>.docx (step-by-step Copilot Studio setup), rerun-protocol-<agent>-<date>.docx (Pillar 3 L200), and baseline-comparison-<agent>-<date>.xlsx (Pillar 5 L200). They walk you through how to set up the run today and advance Pillars 3 and 5 from L100 Initial to L200 Defined."

8. Maturity snapshot — before/after table showing where the agent stands after this session:

   Pillar	Baseline	After this session	Next-session target
   1 — Define what "good" means	L100 Initial	L300 Systematic ✓	—
   2 — Build your eval sets	L100 Initial	L300 Systematic ✓	—
   3 — Run evals across the lifecycle	L100 Initial	L200 Defined ✓ (via rerun-protocol-<agent>-<date>.docx)	L300 Systematic
   4 — Improve and iterate	L100 Initial	L100 Initial	L300 Systematic (Stage 4)
   5 — Handle changes with confidence	L100 Initial	L200 Defined ✓ (via baseline-comparison-<agent>-<date>.xlsx)	L300 Systematic

C. Pillar 3 starter — rerun-protocol-<agent>-<date>.docx — Generate using the /docx skill, sourcing structure and content from skills/eval-guide/rerun-protocol.md. This is the customer's takeaway reference for Pillar 3 L200 Defined: when to re-run evals, what scope to run, how to log the result. The docx is portable, printable, and shareable with the team.

Render the markdown sections as docx sections with the same headings (Purpose, Prerequisites, When to re-run, Run order rule, Logging discipline, Interpreting re-run results, You've reached L200 Defined when…, Path to L300 Systematic, References). Format the trigger table as a styled docx table, color-code the priority column, and put the "You've reached L200 Defined when…" exit criteria in a callout box.

D. Pillar 5 starter — baseline-comparison-<agent>-<date>.xlsx — Generate using the /xlsx skill, sourcing structure and content from skills/eval-guide/baseline-comparison-template.md. This is the customer's fill-in workbook for Pillar 5 L200 Defined: a structured template they fill in each time they compare two eval runs.

Workbook structure (auto-size columns; freeze header rows; protect instruction sheets):

E. Eval setup guide — eval-setup-guide-<agent>-<date>.docx — Always generate this alongside the CSVs (A). It is not optional and not on-request. Without it, the customer is staring at CSVs with no instructions for the manual method-assignment step in CPS. Generate using the /docx skill, sourcing structure and content from skills/eval-guide/eval-setup-guide.md. This is the customer's step-by-step walkthrough for setting up and running the CSVs in Copilot Studio's Evaluate tab — the operational companion to the eval set.

Render the markdown sections as docx sections with the same headings (What you should have before you start, Step 1–8, Per-method setup table, How to choose a threshold, Common setup issues, You've finished setup successfully when…, Related artifacts, References). Format the per-method setup section as styled docx tables; pull the eval-set method decision tree into a callout box; preserve the troubleshooting symptom/cause/fix table verbatim.

Tell the customer: "Five artifacts: the CSVs go straight into Copilot Studio, the test case .docx is for sharing, the new eval-setup-guide-<agent>-<date>.docx walks you through the Evaluate tab step by step (open it the first time you set up the run), and rerun-protocol-<agent>-<date>.docx + baseline-comparison-<agent>-<date>.xlsx are your Pillar 3 and Pillar 5 starter kits — keep them with your eval set."

• eval-results-<agent>-<date>.csv — pass/fail per case, score per LLM method, judge rationale.
• eval-results-<agent>-<date>.json — same data, programmatic-friendly.
• A baseline pass rate and gate status by eval set — the number every future change is compared against.

• Your agent isn't built yet. The deliverables from Stages 0–2 are the eval jumpstart; come back when the agent is running.
• You already have eval results (prior run, internal/external testing tool). Skip to Stage 4.

First-run pass rate is usually 40–70%, not 80%+. Customers who get 50% on the first run sometimes spiral; they shouldn't. The valuable signal is which categories pass and fail, not the headline number. Stage 4 turns the failures into ranked action.

LLM-judge methods are non-deterministic — Compare meaning and General quality show ±5% variance between runs. If a result lands borderline, run it again and take the median.

node eval-runner.js --token-endpoint "<URL>" --csv-dir .

Or use /chat-with-agent for individual questions via the Copilot Studio SDK.

Scoring methods:

• Compare meaning → semantic equivalence (0.0–1.0, LLM judge)
• General quality → relevance / groundedness / completeness / abstention (0.0–1.0, LLM judge)
• Keyword match → code-based string matching (free, deterministic)
• Exact match → code-based string equality (free, deterministic)

Required: ANTHROPIC_API_KEY for LLM-judge methods. Code-based methods run free.

• Don't panic at the first-run pass rate. 40–70% is normal. Read hard gates, eval-set pass rates, and regression/direction instead of the headline.
• Export results immediately to CSV. Copilot Studio retains run results for only 89 days. You need the CSV for long-term tracking and for Stage 4 interpretation.
• Run twice if borderline. LLM-judge scoring is non-deterministic; re-run and take the median.
• Run hard-gated Trust & Safety sets and impacted capability regression sets first. If a hard gate fails, the release decision is blocked until it is fixed or explicitly waived by the accountable owner.

Results table printed to terminal + eval-results-<agent>-<date>.csv and .json written to disk.

• A gate-based SHIP / ITERATE / BLOCK verdict — driven by the hard/soft gates defined in the eval manifest, not the aggregate pass rate.
• Gate status per eval set — capability and trust & safety sets reported separately with target, actual, gate type, and PASS/MISS status.
• Regression/direction evidence as supporting context — capability trends help prioritize fixes, but they do not override hard/soft gates.
• Failure triage table — every failure classified first into the playbook Step 7 buckets (Eval Setup vs Agent Quality), then into a fix subtype such as agent configuration or platform limitation. The classification points at the fix.
• Top 3 actions in Change → Re-run → Expect format.
• A .docx triage report for your team to act from.

• You don't have eval results yet. Run Stage 3 first.
• You already know what to fix and don't need the diagnostic. Skip the full triage; just re-run after your change.

After implementing the Top 3 actions, re-run Stage 3 and re-do Stage 4 with the new results. The before/after comparison validates whether the fix worked — and that before/after evidence is what advances Pillar 4 to L300 Systematic. A single Stage 4 pass without a follow-up re-run leaves Pillar 4 at L200.

At least 20% of failures in a new eval are eval setup bugs, not agent bugs. The test case might be wrong, the expected response might be outdated, the testing method might be inappropriate, or the LLM judge might have misread the response. Don't blame the agent until you've checked the test.

Tell the customer explicitly: "Before we blame the agent — at least 20% of failures in a new eval are eval setup issues. Let me apply the 5-question eval verification before classifying failures as agent bugs."

This single discipline is what separates productive triage from churn.

The headline pass rate ("60% passed") is wrong as a verdict. The first read is gate status from the manifest:

• Hard gate — must pass before deploy. Any failed hard gate means the agent cannot ship as-is.
• Soft target — tracked and remediated, but not blocking by itself.
• Trust & safety hard gate — usually a BLOCK verdict when missed.
• Deployment-critical capability hard gate — BLOCK or ITERATE based on severity, owner risk tolerance, and the gate policy.

Apply this decision rule:

ANY hard gate missed?
    YES -> cannot SHIP.
           Trust & safety hard-gate miss -> usually BLOCK.
           Deployment-critical capability hard-gate miss -> BLOCK or ITERATE.
           Other hard-gate miss -> ITERATE until fixed.
    NO  ->
        ANY soft target missed?
            YES -> ITERATE: track the gap and fix in priority order.
            NO  -> SHIP, assuming human review agrees coverage is sufficient.

Then read eval-set trends and regression/direction as prioritization evidence. Capability misses tell you where to improve next; hard-gated Trust & Safety misses tell you what blocks release. If no workbook/manifest is present, infer gates only as a fallback and state: "No workbook or manifest provided — gate status inferred and should be reviewed by the owner."

A green-across-the-board run is rare on first iteration; expect 2–3 Plan→Interpret cycles before every hard gate passes and soft targets are acceptable. Tell the customer the gate decision so they know what blocks shipping versus what should be tracked.

Which skill to use: For a one-shot triage report from a CSV file or results summary, invoke /eval-result-interpreter. For interactive, multi-round diagnosis with detailed remediation guidance, invoke /eval-triage-and-improvement. Start with the interpreter; switch to triage if you need help implementing fixes.

1. Pre-triage check — scan for infrastructure symptoms before classifying any failure. Don't just ask "was everything working?" — that's a yes/no the customer can't answer accurately. Look for these symptoms in the results data:

   Symptom in results	Likely cause	Action
   Empty agent response on multiple cases	Auth failure or timeout, not agent error	Don't count as agent failure — flag for re-run after infra fix
   Sudden cluster of fails all citing one source	That source was unreachable during run	Verify source connectivity, re-run those cases
   Same case passes/fails inconsistently across re-runs (>10% swing)	Non-determinism beyond normal LLM variance — likely caching, latency, or auth-token expiry mid-run	Re-run 2–3 times, take median
   Cases tagged to a user profile got responses for a different profile	Profile assignment misconfigured at import	Fix profile tags, re-run those cases
   Refusal cases pass but with generic "I can't help" not the expected escalation language	Agent has the refusal but lacks the escalation routing	Real agent issue — keep counted as failure, but classify as Agent Config (incomplete refusal) not Safety failure

   Confirm with the customer: "I see [N] empty responses / [N] cases all on Source X / [other pattern]. Was [auth / connectivity / etc.] healthy during the run?" Don't blame the agent for symptoms that match infrastructure patterns until the customer confirms.

   If anything was broken during the run, the run is invalid — re-run before triaging.

2. Gate summary and verdict — Build the eval-set table from the workbook/manifest: eval set, set type, category / dimension, testing method, intended use/cadence, target, actual, gate type, and gate status. Drive SHIP / ITERATE / BLOCK from this table. Keep capability and trust & safety sets separate. Use capability trends and regression/direction only as supporting prioritization.

3. Failure triage with the 20% rule — apply 5-question eval verification to each failure before classifying it as an agent bug. ~20% will move to Eval Setup root cause.

4. Root causes: First classify each failure into the playbook Step 7 bucket:

   • Eval Setup Issue — the test, expected answer, method, rubric, or judge is wrong. Fix the eval.
   • Agent Quality Issue — the eval correctly caught a real agent problem. Then subtype it as Agent Configuration Issue (prompt/topic/tool/retrieval/config fix) or Platform Limitation (known platform behavior / connector limitation / unsupported scenario). Fix or mitigate the agent path.

5. Top 3 actions — Each: Change X → Re-run Y → Expect Z. When re-running, run the full test set, not just the failing cases, to catch regressions elsewhere. Save pre-fix pass rates and compare before/after — that before/after evidence is what distinguishes L300 Systematic Pillar 4 from L200 Defined.

6. Pattern analysis and next-run recommendation.

The dashboard's Agree / Disagree buttons per case are the central mechanism for handling LLM-judge errors — not a power-user feature. ~5–10% of "fails" are judge errors (judge misread the response, missed an implicit citation, over-penalized minor phrasing). When you disagree, click Disagree — the case flips to an Eval Setup root cause and stops counting against the agent.

Use this aggressively. A pass rate built on uncorrected judge errors is a false signal. Domain expertise wins over the judge every time.

If everything passes, your eval is too easy. Real agents in real production conditions don't pass 100% of well-designed tests. Add harder cases — adversarial inputs, paraphrase variants, boundary conditions, sensitive-data probes. A 100% pass rate without harder cases is comfort, not evidence.

The customer payoff: "You now have a ranked action list — three specific things to change, what to re-run after each, and what outcome to expect. Combined with the rerun protocol and baseline-comparison workbook from Stage 2, you can close the loop on this eval today and the next one in half the time."

Maturity callout — Pillar 4 / playbook Steps 7, 9 (L100 Initial → L300 Systematic): Interpret advances Pillar 4 from reactive fixing to structured root-cause analysis (each failure classified eval-setup vs agent-quality), before/after validation, and regression-proofing — plus designing the production optimization loop (Step 9). All three in-session pillars (1, 2, 4) are now at L300 Systematic. Pillars 3 (Run evals across the lifecycle) and 5 (Handle changes with confidence) reach L200 Defined via the rerun-protocol-<agent>-<date>.docx and baseline-comparison-<agent>-<date>.xlsx starter artifacts generated at session close.

Before generating the final triage report, launch the interpret dashboard for review:

1. Write the triage data to stage-4-data.json using eval sets as the gate source of truth and cases as the expandable detail:

   Copy

   {
     "agent_name": "...",
     "summary": {"total": 28, "passed": 19, "failed": 9},
     "eval_sets": [
       {
         "id": "accuracy",
         "name": "Accuracy",
         "set_type": "capability",
         "category": "Accuracy",
         "method": "Compare meaning",
         "regression_class": "regression",
         "target_pass_rate": 90,
         "gate": "hard",
         "deployment_critical": true
       },
       {
         "id": "prompt_injection",
         "name": "Prompt injection / Jailbreak",
         "set_type": "trust_safety",
         "category": "prompt_injection",
         "method": "Compare meaning",
         "regression_class": "gate-only",
         "target_pass_rate": 100,
         "gate": "hard"
       }
     ],
     "eval_results": [
       {"eval_set_id": "accuracy", "case_id": 1, "question": "...", "expected": "...", "actual": "...", "method": "Compare meaning", "score": 0.92, "pass": true, "explanation": "Rationale from LLM judge..."}
     ],
     "failures": [
       {"id": 1, "eval_set_id": "prompt_injection", "case_id": 2, "question": "...", "expected": "...", "actual": "...", "root_cause": "agent_config", "explanation": "..."}
     ],
     "top_actions": [...],
     "patterns": [...]
   }

   Key requirements:

   • eval_sets is the authoritative gate table. Each set must have id, name, set_type (capability or trust_safety), category / dimension, method, intended use/cadence or regression_class, target_pass_rate, and gate (hard or soft). Capability hard gates that are deployment-critical should set deployment_critical: true.
   • eval_results[*].eval_set_id must point to an eval_sets[*].id. The dashboard aggregates case results into set-level actual pass rates, compares them to targets, and computes the live SHIP / ITERATE / BLOCK verdict.
   • eval_results contains ALL test case results (not just failures) so cases can be expanded in the dashboard and human Agree / Disagree overrides can recompute gate status.
   • Each eval result includes explanation (the LLM judge rationale) for human review
   • Do not precompute a static verdict as the source of truth. The dashboard live-computes the verdict from eval_sets + criterion_metrics + human overrides, then returns computed_verdict and gate_summary in feedback for the final report.

2. Launch the dashboard:

   Copy

   python "$(ls ~/.claude/skills/eval-guide/dashboard/serve.py 2>/dev/null || ls ~/.claude/plugins/cache/*/eval-guide/*/skills/eval-guide/dashboard/serve.py 2>/dev/null | head -1)" --stage interpret --serve --data stage-4-data.json

3. The user reviews the gate verdict first, then eval-set pass rates and regression/direction evidence, expands set rows to see test case details, uses Human Judgement (Agree/Disagree) to override LLM judge assessments, and re-classifies root causes. Disagreed failed cases are treated as eval-setup issues and no longer count against the agent's gate status; the dashboard recomputes set pass rates and the verdict live.

4. When the user confirms, parse the feedback from the bash stdout between the ===EVAL_GUIDE_FEEDBACK_BEGIN=== / ===EVAL_GUIDE_FEEDBACK_END=== markers. Apply every edit it contains, faithfully and without question. The customer's choices are final — do NOT re-litigate, do NOT suggest reverting, do NOT ask for confirmation again, do NOT partially apply. (interpret-feedback.json is also on disk as a backup, but stdout is the primary channel.)

   This applies to ALL edit types:

   • human_disagrees — every Disagree is the customer overriding the LLM judge. Each disagreed failed case flips to Eval Setup Issue root cause and stops counting against the agent's gate status. The customer's domain expertise wins; do not override their override.
   • computed_verdict and gate_summary — the dashboard's post-review verdict and set-level gate table. Use these in the final report; do not recompute independently unless the feedback is missing them.
   • Root cause reclassifications per failure (Eval Setup, Agent Quality — Config, Agent Quality — Platform)
   • Top-3-action edits
   • General Comments box content

   Then narrate the edits back — count Disagrees applied, list re-classified root causes, name any Top-3-action edits. Example: "Got it — 4 Disagrees flipped to Eval Setup, root cause for failure #7 reclassified from Agent Quality — Config to Agent Quality — Platform, and Top action #2 edited to scope to the prompt-injection gate. Updated gate verdict: BLOCK -> ITERATE after Disagrees applied." Don't just say "applied." The narration confirms you parsed correctly; it is NOT an invitation to re-decide.

   If changes requested instead of confirmed, regenerate and re-launch.

5. After confirmation, generate the customer-ready .docx triage report using the /docx skill. Same principles: concise, presentable, self-contained. Structure:

   1. Gate verdict — prominent SHIP / ITERATE / BLOCK decision from computed_verdict, plus the set-level gate table from gate_summary (eval set, set type, category / dimension, method, regression class, target, actual, gate type, status). State explicitly that the verdict is gate-based, not aggregate-pass-rate-based.

   2. Eval-set performance — set summary cards/table with actual pass rate, target, gate status, intended use/cadence, and regression/direction notes

   3. Failure triage table (eval set, case, question, expected, actual, root cause) — include human-disagreed entries as "Eval Setup — Human Disagrees"

   4. Top actions (Change → Re-run → Expect)

   5. Pattern analysis — eval-set patterns highlighting systemic issues, hard-gate misses, regressions, and source/tool clusters

   6. Next steps. Always include a pointer line: "You're also keeping the companion artifacts from Stage 2 — eval-setup-guide-<agent>-<date>.docx (step-by-step Copilot Studio setup), rerun-protocol-<agent>-<date>.docx (Pillar 3 L200), and baseline-comparison-<agent>-<date>.xlsx (Pillar 5 L200). They walk you through how to set up the run, when to re-run, and how to compare runs." (If Stage 2 was skipped, generate them now using the same flow as Stage 2 deliverables C, D, and E.)

   7. Optimization loop plan (playbook Step 9) — a short forward-looking section: which production signals to collect (thumbs-down — highest signal — plus escalations, manual overrides, support tickets), how to cluster them, how to decide where each cluster gets fixed (agent config / rubric / new eval cases), and that every shipped fix is re-validated against the regression suite from Step 8. Frame it as the bridge from "this eval" to "continuous improvement once the agent is in production."

   8. Reusable-asset register (playbook Step 10) — scan the criteria, rubrics, and trust & safety sets produced this session and flag the ones that are NOT specific to this agent (e.g., a prompt-injection set, a PII-handling rubric, a tone rubric). List each as a reusable candidate with a tier — Required (org-wide gate every agent must pass), Recommended (per-category default), or Opt-in (domain-specific) — plus provenance and a suggested owner. This is the seed of the shared eval library.

   9. Maturity snapshot — same before/after table as the Stage 2 report, updated to reflect Pillar 4 now at L300 Systematic:

      Pillar	Baseline	After this session	Next-session target
      1 — Define what "good" means	L100 Initial	L300 Systematic ✓	—
      2 — Build your eval sets	L100 Initial	L300 Systematic ✓	—
      3 — Run evals across the lifecycle	L100 Initial	L200 Defined ✓ (via rerun-protocol-<agent>-<date>.docx)	L300 Systematic
      4 — Improve and iterate	L100 Initial	L300 Systematic ✓	—
      5 — Handle changes with confidence	L100 Initial	L200 Defined ✓ (via baseline-comparison-<agent>-<date>.xlsx)	L300 Systematic