eval-suite-planner

This skill produces the Plan artifact of the /eval-guide lifecycle: a populated copy of the customer's Eval Suite Planning & Logging Template plus an interactive HTML review page. The workbook is the source-of-truth artifact; do not replace it with a scenario table, quality-signal table, generic spreadsheet, default .docx report, or HTML-only plan.

The skill aligns to skills/eval-guide/playbook.md and skills/eval-guide/eval-suite-template.md. Use the 10-step playbook as the methodology spine and the XLSX template as the output shape.

Copy the blank XLSX template and populate existing cells/rows only. Do not modify the template.

Do not rename sheets, add sheets, delete sheets, add columns, change headers, rewrite README text, edit Dropdown Lists, change styles, change data validation, or convert the template into a different spreadsheet.

If a blank template workbook is available in the session, use it. If not, ask the user to provide the template; do not silently invent a new workbook.

Ask targeted questions only when a workbook field materially affects the plan and cannot be inferred safely:

1. Eval owner / named approver.
2. Lifecycle stage and target deployment decision.
3. Whether the agent is prompt-only, RAG/knowledge-grounded, or agentic with tools/connectors.
4. Regulated/compliance obligations.
5. Authoritative sources and source owners.

If the user wants speed or cannot answer, populate TBD - confirm before baseline.

When invoked as /eval-suite-planner <agent description>:

1. Extract or infer the agent's purpose, users, knowledge sources, capabilities, boundaries, architecture, lifecycle stage, and known risks.
2. Populate Step 1 — Plan the Eval Effort:
   • one-sentence eval objective;
   • five-factor risk tier: reach, criticality of error, autonomy/blast radius, regulatory/compliance exposure, data sensitivity;
   • one accountable owner.
3. Define eval sets, not scenarios:
   • Capability eval sets: one row per capability dimension that must be diagnostic, e.g. accuracy/correctness, faithfulness/groundedness, relevancy, style/tone, reasoning/tool use.
   • Trust & Safety eval sets: one row per refusal, boundary, or safety category, e.g. guardrails, out-of-scope handling, sensitive-data handling, prompt injection/jailbreak, compliance-specific behavior.
4. Apply Step 4 v5 gates/improvement-target logic:
   • T&S sets use absolute pass-rate hard gates, usually near 100%.
   • Capability sets usually use a launch floor for first deployment plus regression/direction after baseline, not a standing absolute pass-rate target.
   • High-risk capabilities that function like guardrails keep explicit hard floors.
   • Use the template's existing Target pass rate, Target rationale, Gate type, Intended use, Run cadence, and Notes columns to express this; do not add a new column.
5. Specify Step 5 human inputs:
   • grading rubric, ground truth, golden answer, or rubric + ground truth;
   • author/owner;
   • grounding source dependency;
   • whether source changes require review.
6. Plan Step 6 grader validation without changing the template:
   • record grader type and validation expectation in the registry row's Notes;
   • for LLM-as-judge / Custom rubrics, note that human-labeled hard and borderline cases must validate the judge before baseline scores are trusted;
   • for programmatic checks, note the deterministic check to confirm;
   • for human grading, note reviewer agreement expectations where relevant.
7. Seed Step 7 baseline placeholders in 3 . Run Log only when useful:
   • one placeholder row per eval set;
   • Run type = Baseline;
   • result fields blank;
   • Actionable next step = Validate grader, then run baseline;
   • Status = Open.
8. Apply Step 8 regression partitioning in existing registry fields:
   • capability sets usually Intended use = Both or Regression;
   • most T&S sets are Gate; the slim subset likely affected by model/tool/policy changes can be Both or Regression;
   • set Run cadence using existing dropdown values such as Per-change, Nightly, Weekly, or Milestone-only.
9. Flag Step 10 reusable assets in 4 . Reusable Library:
   • reusable T&S sets;
   • grading rubrics;
   • failure-pattern templates;
   • production-derived edge-case categories when applicable.

Use skills/eval-guide/eval-suite-template.md as the exact tab/column map.

Create eval-suite-<agent-name>-<YYYY-MM-DD>.xlsx as a populated copy of the template.

Then create eval-suite-<agent-name>-<YYYY-MM-DD>-review.html next to the workbook using skills/eval-guide/plan-review-page.md.

Do not paste the summary, eval-set table, or checklist into chat. The HTML page carries that content. The final chat response should be only the workbook path, the HTML review page path, and any blocker/manual action.

Include these in the HTML review page checklist instead of displaying them in chat:

• Do not generate scenario-plan tables as the Plan artifact.
• Do not generate quality-signal sheets or quality-signal grouping as the Plan artifact.
• Do not add columns to support missing concepts; use existing fields, especially Notes.
• Do not create a .docx unless the user explicitly asks for a narrative report.
• Do not produce long narrative chat output after artifact generation; use the HTML review page for the interactive summary and checkpoints.
• Be specific to the described agent, but at eval-set granularity.

• /eval-generator — Generate test cases from the populated workbook registry.
• /eval-result-interpreter — Interpret baseline / iteration results using Step 6-7 and gate status.
• /eval-triage-and-improvement — Diagnose failures and feed the Step 9 optimization loop.
• /eval-library-promoter — Promote Step 10 reusable assets.
• /eval-guide — Orchestrated workflow with dashboard review checkpoints.