Plan Copilot Studio agent evaluations, generate test cases, and diagnose failures before your next release.
eval-guide is a toolkit for evaluating Copilot Studio agents from Claude Code or GitHub Copilot. It helps you define success criteria, plan capability and trust & safety tests, generate test cases, and interpret results using Microsoft’s agent evaluation methodology. You can begin planning from an agent description before the agent is built.
Claude and Copilot can select relevant guidance when you ask about evaluation topics. You can also invoke the guided workflow with /eval-guide:eval-guide in Claude Code or /eval-guide in Copilot CLI, or use an individual skill for planning, generation, methodology questions, or diagnosis.
In Claude Code, connecting to a live agent and running tests through the guided workflow requires the separate Copilot Studio plugin. With GitHub Copilot, all skills work in description-based mode: you describe your agent rather than connecting to it. Description-based planning and test generation work without that connection.
Describe the agent’s purpose, users, knowledge sources, and boundaries. The guided workflow helps turn that description into evaluation criteria. For the planning workbook, provide a blank Eval Suite Planning & Logging Template; the planner populates it with evaluation sets, gates, owners, and required human inputs.
Give the generator your populated planning workbook or an agent description. It drafts capability and trust & safety cases as Copilot Studio import CSVs for single-response tests, or as conversation blueprints for multi-turn workflows. Review generated expected answers against your authoritative sources before approving them.
Provide exported Copilot Studio results or a results summary to the result interpreter. Include the planning workbook or test manifest when available so it can assess the intended gates and targets. Its structured report gives a SHIP, ITERATE, or BLOCK recommendation with failure classifications and priority actions.
Use the triage skill for an interactive diagnosis of failing cases. Share the input, expected answer, actual response, evaluation method, and changes already tried. It guides you through distinguishing evaluation-setup problems from agent-quality problems and choosing what to fix before another run.
Showing all 6
Answer evaluation questions about graders, datasets, test methods, and result interpretation.
Guide evaluation from an agent description through planning, test generation, and result review.
Populate your blank planning workbook with evaluation sets, gates, owners, and human inputs.
Draft single-response import CSVs or multi-turn conversation blueprints from your plan.
Analyze results for a gate-based release recommendation and prioritized failure report.
Work through failing cases interactively to identify causes and choose fixes.
No matching results.
In a terminal on your computer, add the eval-guide marketplace.
claude plugin marketplace add microsoft/eval-guideIn the same terminal, install eval-guide from that marketplace. This installs the plugin with its bundled skills.
claude plugin install eval-guide@eval-guideStart a new local Claude Code session to load the plugin. Plugins installed on your computer are not loaded in Claude Code cloud sessions.
See Claude Code’s plugin loading guidance for loading behavior.
In that session, type / and look for /eval-guide:eval-guide in the skills menu to confirm that the guided workflow is available.
This route installs the toolkit’s skills only. In GitHub Copilot they work in description-based mode: provide an agent description or exported results rather than connecting to a live Copilot Studio agent.
In a terminal, open your project folder and run the installer for GitHub Copilot. The Skills CLI installs to the current project by default. Copilot CLI discovers project skills in .github/skills, .claude/skills, or .agents/skills.
npx skills add microsoft/eval-guideStart a Copilot CLI session in that project, or enter /skills reload in an existing session to load the installed skills.
See GitHub’s skill loading guidance for discovery paths and reloading.
Enter /skills list in Copilot CLI and confirm that eval-guide and the other toolkit skills appear among the available skills.