Labsco
microsoft logo

analyze-evals

✓ Official★ 345

by microsoft · part of microsoft/skills-for-copilot-studio

Analyze exported evaluation results from Copilot Studio's Evaluate tab. The user provides a CSV file exported from the Copilot Studio UI; this skill parses it, identifies failures, and proposes YAML fixes. No API access or published agent required — just the exported CSV.

🔌 This skill ships inside the copilot-studio plugin — install the plugin and you also get 4 sub-agents, hooks.

This is the playbook your agent receives when the skill activates — you don't need to read it to use the skill, but it's here to audit before installing.

Analyze Copilot Studio Evaluation Results

Analyze evaluation results exported from the Copilot Studio UI as CSV.

Phase 1: Get Results

  1. Ask the user for the CSV file path if not already provided. The file is typically exported from Copilot Studio's Evaluate tab and named Evaluate <agent name> <date>.csv in their Downloads folder.

  2. Read the CSV file. The in-product evaluation CSV has these columns:

    ColumnMeaning
    questionThe test utterance
    expectedResponseExpected response (may be empty)
    actualResponseWhat the agent responded
    testMethodType_1Eval method (e.g., GeneralQuality)
    result_1Pass or Fail
    passingScore_1Score threshold (may be empty)
    explanation_1Why it passed/failed (e.g., "Seems relevant; Seems incomplete; Knowledge sources not cited")

    The _1 suffix indicates the first eval method. There may be additional methods (_2, _3, etc.) with the same column pattern.

Phase 2: Analyze Results

  1. Focus on failed evaluations (result_1 = Fail, or any result_N = Fail).

  2. For each failure, use the explanation column to understand the issue:

    • "Question not answered" — The agent couldn't handle the question. Check if there's a matching topic or knowledge source.
    • "Knowledge sources not cited" — The agent responded but didn't cite sources. Check knowledge source configuration and SearchAndSummarizeContent nodes.
    • "Seems incomplete" — The response was partial. Check topic flow for early exits, missing branches, or incomplete SendActivity messages.
    • Error messages in actualResponse (e.g., GenAIToolPlannerRateLimitReached) — These are runtime errors, not authoring issues. Flag them to the user as transient failures to retry.

Phase 3: Propose Fixes

  1. For each failure, identify the relevant YAML file(s):

    • Auto-discover the agent: Glob: **/agent.mcs.yml
    • Find the relevant topic by matching the test utterance against trigger phrases and model descriptions
    • Read the topic file to understand the current flow

  2. Propose specific YAML changes to fix each failure. Present them to the user as a summary:

    • Which test(s) failed and why
    • Which file(s) need changes
    • What the proposed change is (show the diff)

  3. Wait for user decision. The user can:

    • Accept all — apply all proposed changes
    • Accept partially — apply only some changes (ask which ones)
    • Reject — discard proposed changes and discuss alternative approaches

  4. Apply accepted changes using the Edit tool. After applying, remind the user to push and publish again before re-running evaluations.