Reads through your chatbot and works out what it ought to be tested on, then writes a spreadsheet of questions you upload straight into Copilot Studio's Evaluate tab.
WHEN TO USE
SCOPE
USE FOR
Get a starter test set of 10 to 25 questions without thinking them up yourself.
Describe what a good answer looks like, so an AI grader can judge the reply.
Cover the greetings, the off-topic questions and the things the bot should refuse.
This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.
Create Evaluation Test Set
Create a test set CSV file that can be imported into Copilot Studio's Evaluate tab for in-product agent evaluation.
Phase 1: Understand the Agent
Read the agent's YAML files to understand what it does:
Glob: **/agent.mcs.yml — find the agent
Read agent.mcs.yml — get the agent's instructions, description, and capabilities
Read settings.mcs.yml — check orchestration mode (generative vs classic)
Aim for 10–25 test cases with good coverage across categories.
Phase 3: Write the Expected Responses
The CSV import only supports two columns: question and expectedResponse. Test methods cannot be set via CSV import — they are configured in the UI after import. The default test method (General quality) is applied to all imported test cases.
Write expected responses with this in mind:
For questions where you want General quality grading: write behavioral descriptions ("The response should recommend hotels in Paris with relevant details")
For questions where you'll later switch to Compare meaning or Exact match in the UI: write realistic agent replies that the grader can compare against
Leave expectedResponse empty for questions that only need General quality (it works without expected responses)
Available test methods (configured in UI after import)
Custom grader with your own instructions and labels
Configured in UI
Phase 4: Write the CSV
Write the CSV file using the Write tool. The format must be:
"question","expectedResponse"
"User question here","Expected agent response or behavioral rubric"
"Question without expected response",
Column specification
Column
Required
Description
question
Yes
The user message to send to the agent. Max 1,000 characters.
expectedResponse
No
The expected response or behavioral rubric. Leave empty if not needed.
Important: The Testing method column is not supported on import — it is ignored. All imported test cases get the default test method (General quality). Configure other test methods in the UI after import.
Rules
Max 100 questions per test set
Max 1,000 characters per question (including spaces)
File must be .csv format
Use double quotes around all values
For questions that will use General quality: write expected responses as behavioral descriptions
For questions that will use Compare meaning or Exact match: write expected responses as realistic agent replies
Expected response examples
Behavioral rubric (for General quality):
"Find me a hotel in Paris","The response should include hotel recommendations in Paris with relevant details like names, locations, or prices."
Realistic reply (for Compare meaning — set method in UI after import):
"Hi there","Hello! How can I help you today?"
Exact expected text (for Exact match — set method in UI after import):
"What is 2+2?","4"
Phase 5: Instruct the User
After writing the CSV, tell the user:
To import into Copilot Studio:
Open your agent in Copilot Studio
Go to the Evaluate tab
Click New evaluation > Single response
Drag or browse for the CSV file
Review the imported test cases and adjust if needed
Optionally add more test methods (Capability use, Custom) in the UI
Click Evaluate to run, or Save to run later
After import, some things can only be configured in the UI:
Pass thresholds for Compare meaning and Text similarity
Keywords for Keyword match test cases
Expected capabilities for Capability use test cases
Custom grader instructions and labels
No requirements beyond installing this skill were found.
🔌 This skill ships inside the Copilot Studio YAML Toolkit plugin. Listed package contents include 4 agents and hooks. Installed components and skill versions may vary by AI tool.
Install just this skill
Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.
Install checklist
Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.
Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.
After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.
If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.
Install the whole plugin
This route installs the Copilot Studio YAML Toolkit plugin package selected for your AI tool.
Add Microsoft's marketplace, then install the plugin
Type both lines inside a Claude Code session, not at your shell prompt. They add Microsoft's marketplace, which is this repository, and the copilot-studio plugin, with its Copilot Studio skills and four sub-agents.
Install checklist
Microsoft lists Node.js 18 or later, and VS Code with the Copilot Studio extension for cloning, pushing and pulling agents.
Claude Code installs it once you pick an install option.
Good to know
A hook that runs whenever a session starts adds Microsoft's Copilot Studio instructions; one tells Claude to lean towards reading your requests as Copilot Studio requests unless there are clear signs otherwise.
The option that installs it for you in every project brings the hook's instructions into every project; to keep them to one Copilot Studio repository, pick the option for this repo only.
On first run, and after an update that changes its dependencies, the session-start hook runs npm install in the plugin's data folder; every session it also writes ~/.copilot-studio-cli/plugin-paths.json.
Add Microsoft's marketplace, then install the plugin
Type both lines in an interactive GitHub Copilot CLI session, not at your shell prompt. The first adds Microsoft's marketplace, which is this repository; the second installs the copilot-studio plugin.
Install checklist
Microsoft lists Node.js 18 or later, and VS Code with the Copilot Studio extension for cloning, pushing and pulling agents.
The plugin also carries a hooks/hooks.json file with SessionStart hooks, a path GitHub's plugin reference lists as a hooks file for plugins in this format; read the file in the repository before you install.
Guide users through adding a new connector action to a Copilot Studio agent. Connector actions require UI-based connection setup, so this skill walks users through the Copilot Studio portal steps, then delegates to edit-action for YAML modifications.Read full description
Generate and insert an Adaptive Card into a Copilot Studio topic using AdaptiveCardPrompt. Use when the user asks to add an adaptive card, rich card, form card, info card, confirmation card, or interactive card to a topic.Read full description
Add generative answer nodes (SearchAndSummarizeContent or AnswerQuestionWithAI) to a Copilot Studio topic. Use this instead of /add-node when the user asks to add grounded answers, knowledge search, generative answers, or AI-powered responses — these nodes require specific patterns (ConditionGroup follow-up, knowledge source references, autoSend, responseCaptureType) that /add-node does not cover.Read full description
Add a global variable to a Copilot Studio agent. Use when the user needs a variable that persists across topics in the same conversation and can optionally be visible to the AI orchestrator.Read full description
Add a knowledge source (public website or SharePoint) to a Copilot Studio agent. Use when the user asks to add a knowledge source, documentation URL, website, or SharePoint site for the agent to search.Read full description
Add or modify a node in an existing Copilot Studio topic. Use when the user asks to add a question, message, condition, variable, or other node to a topic. Do NOT use this for generative answers or knowledge search — use /add-generative-answers instead.Read full description
