run-eval

Tests your chatbot while it is still a draft, so you can find out whether a change worked before anyone else is exposed to it, then suggests fixes for whatever failed.

WHEN TO USE

SCOPE

USE FOR

Pick which of your saved test sets to run, and watch it work through them.
Run the tests as a signed-in user, so tools and private documents are actually used.
Find out why a generative answer or a topic match failed on a specific test case.
Get a concrete edit to make for each test that failed, before you run it again.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (7)

Run Evaluation (PPAPI)

Run evaluations against a Copilot Studio agent's draft — no publish needed.

The caller (test agent) must provide --client-id and --workspace. If you don't have the client ID, return immediately and tell the caller to run test-auth first.

All eval-api commands run in the foreground. NEVER use run_in_background.

Step 1: List test sets and let the user choose

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js list-testsets --workspace <path> --client-id <id>

No test sets found: Tell the user to create one in Copilot Studio (Evaluate tab > New evaluation). Stop.
One test set: Tell the user which one you're using and proceed.
Multiple test sets: Show them all and ask the user to pick. Do not proceed until they answer.

Step 2: Ask about authenticated execution — MANDATORY, do not skip

You MUST ask this question and wait for the user's answer before starting the run.

Ask the user:

Does your agent use authenticated knowledge sources or connector actions (tools) that require user identity? If so, you'll need to provide a connection ID — without it, the eval runs anonymously and tools and knowledge sources will not be used.

How to obtain the connection ID:

Go to https://make.powerautomate.com

Open Connections from the side menu

Select the relevant Microsoft Copilot Studio connection

Copy the connection ID from the URL (the GUID segment after /connections/)

If your agent doesn't use authenticated knowledge or tools, you can skip this.

Do not proceed to Step 3 until the user responds.

Step 3: Start the run

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js start-run --workspace <path> --client-id <id> --testset-id <id> --run-name "Draft eval <date>"

Add --connection-id <id> if the user provided a connection ID in Step 2.

Add --published only if the user explicitly asked for published-bot testing.

Step 4: Poll until complete

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js get-run --workspace <path> --client-id <id> --run-id <runId>

Poll every 15-30 seconds. Report progress: "Processing: 3/10 test cases..."

Stop when state is Completed, Failed, Abandoned, or Cancelled.

Step 5: Fetch and analyze results

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js get-results --workspace <path> --client-id <id> --run-id <runId>

Present a summary table (total, passed, failed, errors). For failures:

Metric	What to check
`GeneralQuality` Fail	Which of relevance/completeness/groundedness/abstention failed
`ExactMatch` Fail	Score 0.0–1.0
`CapabilityUse` Fail	`missingInvocationSteps`
`Error` status	`errorReason` — often a test set config issue, not a YAML issue

Step 6: Propose fixes (if failures found)

For YAML authoring failures: find the relevant topic, read it, propose specific edits. Wait for user approval before applying.

After applying: offer to push and re-run (go back to Step 3).

Local tool required

1 FOR YOU

01
Authenticate first with the test-auth skill
Run the test-auth skill to sign in and obtain an App Registration client ID and workspace path; run-eval needs both before it can list test sets or start a run.

This skill ships inside the Copilot Studio YAML Toolkit plugin. Listed package contents include 4 agents and hooks. Installed components and skill versions may vary by AI tool.

Install the whole plugin

This route installs the Copilot Studio YAML Toolkit plugin package selected for your AI tool.

Add Microsoft's marketplace, then install the plugin

Type both lines inside a Claude Code session, not at your shell prompt. They add Microsoft's marketplace, which is this repository, and the copilot-studio plugin, with its Copilot Studio skills and four sub-agents.

Install checklist

Microsoft lists Node.js 18 or later, and VS Code with the Copilot Studio extension for cloning, pushing and pulling agents.

01

/plugin marketplace add microsoft/skills-for-copilot-studio

02

/plugin install copilot-studio@skills-for-copilot-studio

Installation and loading

Claude Code installs it once you pick an install option.

Good to know

A hook that runs whenever a session starts adds Microsoft's Copilot Studio instructions; one tells Claude to lean towards reading your requests as Copilot Studio requests unless there are clear signs otherwise.
The option that installs it for you in every project brings the hook's instructions into every project; to keep them to one Copilot Studio repository, pick the option for this repo only.
On first run, and after an update that changes its dependencies, the session-start hook runs npm install in the plugin's data folder; every session it also writes ~/.copilot-studio-cli/plugin-paths.json.

Add Microsoft's marketplace, then install the plugin

Type both lines in an interactive GitHub Copilot CLI session, not at your shell prompt. The first adds Microsoft's marketplace, which is this repository; the second installs the copilot-studio plugin.

Install checklist

Microsoft lists Node.js 18 or later, and VS Code with the Copilot Studio extension for cloning, pushing and pulling agents.

01

/plugin marketplace add microsoft/skills-for-copilot-studio

02

/plugin install copilot-studio@skills-for-copilot-studio

Good to know

The plugin also carries a hooks/hooks.json file with SessionStart hooks, a path GitHub's plugin reference lists as a hooks file for plugins in this format; read the file in the repository before you install.

Open plugin page ↗

Install without the plugin (incomplete)

What's missing

Installing just this skill adds only its own folder. Its instructions also call for other plugin components, such as shared scripts, reference files or other skills, which aren't included. Installing the whole plugin avoids this gap.

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add microsoft/skills-for-copilot-studio --skill "run-eval" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

View on GitHub

The repository is licensed under MIT — use, modification, and redistribution are allowed under that license’s terms.

View the repository’s license file on GitHub →

Run Evaluation (PPAPI)

Step 1: List test sets and let the user choose

Step 2: Ask about authenticated execution — MANDATORY, do not skip

Step 3: Start the run

Step 4: Poll until complete

Step 5: Fetch and analyze results

Step 6: Propose fixes (if failures found)

Local tool required

Install the whole plugin

Add Microsoft's marketplace, then install the plugin

Add Microsoft's marketplace, then install the plugin

More skills

add-action

add-adaptive-card

add-generative-answers

add-global-variable

add-knowledge

add-node