eval-result-interpreter

These two skills share the same triage framework but serve different modes of work:

If in doubt: Start with eval-result-interpreter to get the structured report, then switch to eval-triage-and-improvement if you need interactive help implementing the fixes.

0. Pre-triage infrastructure check (per the Triage Playbook)

Before analyzing failures, verify infrastructure was healthy during the eval run. If any of these were unhealthy, mark affected cases as infrastructure-blocked, not agent-failed:

• Were all knowledge sources accessible and fully indexed?
• Did any API backends return errors, timeouts, or rate-limiting?
• Were authentication tokens valid throughout the run?
• Did the eval environment match the intended configuration?

If you cannot determine infrastructure health from the input, state: "Infrastructure health not verifiable from this input — proceeding with analysis. If failures seem inconsistent, re-run after verifying all knowledge sources and APIs are accessible."

1. Baseline score summary (Step 6)

Parse the results and produce:

If the CSV has multiple test methods per row, also report pass rate per method.

Then report eval sets separately by capability and trust & safety. Prefer manifest metadata for set_type, category, testing method, gate type, pass-rate target, regression class, and provenance. Use this table for each set:

Keep capability and trust & safety reporting separate. Trust & safety categories are guardrails, out_of_scope, sensitive_data, prompt_injection, and compliance. Hallucination is a faithfulness/groundedness capability failure, not a trust & safety failure.

2. Verdict — gate-based SHIP / ITERATE / BLOCK decision (D5)

Drive the verdict from the per-set gates in the manifest, not from aggregate pass rate. For every eval set, read its pass-rate target and gate type from the manifest when available:

• Hard gate — must pass before deploy. Any failed hard gate means the agent cannot SHIP.
• Soft target — tracked and remediated, but not blocking by itself.

Apply this gate-based decision rule:

ANY hard gate missed?
    YES -> cannot SHIP.
           Trust & safety hard-gate miss -> usually BLOCK.
           Deployment-critical capability hard-gate miss -> BLOCK or ITERATE based on severity and owner risk tolerance and gate policy.
           Other hard-gate miss -> ITERATE until fixed.
    NO  ->
        ANY soft target missed?
            YES -> ITERATE: track the gap and fix in priority order, but it is not blocking by itself.
            NO  -> SHIP, assuming human review agrees coverage is sufficient.

Report each set's actual pass rate vs target and hard/soft gate status. Make explicit when aggregate pass rate is misleading: a high aggregate pass rate does not earn SHIP if a hard trust & safety gate failed.

Use risk tier (agent-level: reach, criticality of error, autonomy/blast radius, regulatory exposure, data sensitivity) to interpret target strictness and severity. Use the workbook registry's eval-set category, gate type, target, intended use, cadence, and grader-validation notes as the source of truth for gate decisions.

If no workbook/manifest is present, infer set grouping, gate type, and targets from CSV filenames or question text only as a fallback. State: "No workbook or manifest provided — gate status inferred and should be reviewed by the owner."

State the verdict prominently:

• "Verdict: SHIP." — All hard gates pass and soft targets are acceptable or explicitly accepted by the owner.
• "Verdict: ITERATE." — No blocking hard-gate failure, but one or more soft targets or non-blocking hard gates require fixes before confidence is high.
• "Verdict: BLOCK." — A hard trust & safety gate or other deployment-critical hard gate failed.

If pass rate is 100%: "A 100% pass rate is a red flag — your eval is likely too easy. Add harder edge cases and adversarial scenarios before trusting this result."

3. Failure triage — per the Triage Playbook's Layer 2

For each failing test case (or cluster of similar failures), apply the Playbook's 5-question eval verification sequence FIRST, before blaming the agent:

Every failing case must land in exactly one of the 10-step playbook's two Step 7 root buckets:

• Eval-setup problem — the response is actually acceptable; the eval flagged it wrongly. Action: fix the eval.
• Agent-quality problem — the eval correctly caught a real issue. Action: log the pattern, define a fix, and track it.

Keep the finer Triage Playbook taxonomy, but explicitly map it onto those two buckets:

Maintain a failure-pattern log with case ID, eval set, set type, category/dimension, gate type, Step 7 bucket, fine taxonomy, pattern, owner, fix location, and re-run target. Group failures that share a root cause. For example: "Cases 3, 5, and 7 all fail with 'Question not answered' — this is one agent-quality pattern (missing knowledge source or scope gap), not three independent problems."

3b. Platform diagnostic tools (recommend when applicable)

Copilot Studio provides built-in tools that accelerate triage. Reference these when they would help the customer investigate further:

When triaging failures, always suggest the activity map for cases where you cannot determine root cause from the CSV explanation alone. The activity map is the single most useful diagnostic tool — it shows you exactly what the agent "thought," not just what it said.

Supplementary signal: User reactions (thumbs up/down)

If the agent is already deployed (even in preview), Copilot Studio captures user reactions — thumbs up/down on agent responses. These are not part of the eval CSV, but they complement eval results:

• Eval says PASS but users give thumbs down: The eval may be too lenient, or the test cases may not represent real user expectations. Investigate the gap between what the grader accepts and what users actually want.
• Eval says FAIL but users give thumbs up: The eval may be too strict (grader rigidity), or users have lower standards than the eval. Revisit the expected responses for these scenarios.
• Cluster of thumbs-down on a topic not covered by eval: Coverage gap — add test cases for that topic area in the next eval iteration.

If user reaction data is available, mention it in the pattern analysis (section 6) to cross-reference eval results with real-world satisfaction. Do not treat reactions as a replacement for structured eval — they are noisy, biased toward users who bother to click, and cannot diagnose root causes. They are a signal, not a verdict.

4. Explanation analysis

4a. General Quality scoring criteria

When the test method is GeneralQuality, Copilot Studio scores the response on 4 distinct criteria. A low General Quality score means one or more of these failed — the customer needs to know WHICH one to fix the right thing:

How the 4 criteria interact: A passing General Quality score means all 4 criteria passed. A failing score means at least one failed — check the explanation field to determine which. The most common failure pattern is Relevance failing alone (knowledge gap), followed by Groundedness failing alone (hallucination). When both Relevance and Groundedness fail together, the agent is likely retrieving the wrong knowledge source entirely.

When NOT to rely on General Quality alone: General Quality checks response quality holistically but cannot verify specific factual values, check tool invocation correctness, or validate structured output formats. Use it alongside targeted methods (CompareMeaning for factual accuracy, ToolUse for action verification, KeywordMatch for required terms).

4b. Explanation pattern mapping

Parse the explanation fields from the CSV. Copilot Studio’s General Quality explanations use these patterns — map each to the criteria above and the Playbook’s diagnostic questions:

For each explanation pattern found in the failures, name the diagnostic area and suggest the specific Playbook question to investigate.

4c. Conversation (multi-turn) result interpretation

When interpreting results from conversation test sets (multi-turn evaluations), the failure patterns differ from single-response tests. Apply these additional diagnostic lenses:

Turn-level diagnosis: A conversation test case fails as a whole, but the root cause is usually in a specific turn. Read the agent's responses turn by turn to locate the first turn where quality degrades. Common patterns:

Available methods are limited: Conversation tests only support General Quality, Keyword Match, Capability Use (Capabilities match), and Custom. If you see failures that would benefit from Compare Meaning or Exact Match analysis (e.g., the agent gave the right answer but phrased differently), note this limitation and recommend the customer also create a complementary single-response test set for those specific scenarios.

Critical turn identification: When reporting failures, identify and call out the critical turn — the specific turn where the conversation went wrong. Downstream turns often fail as a consequence of an earlier turn's failure, not independently. Fixing the critical turn may resolve multiple downstream failures in one change.

4d. Set-level grading interpretation

Copilot Studio’s set-level grading evaluates the test set as a whole — not just aggregating individual pass/fail counts, but assessing overall agent quality across the full set. When the customer has set-level results, interpret them alongside case-level results using this framework:

When set-level and case-level results agree: The straightforward case. A high set-level grade with a high case-level pass rate confirms the agent is performing well. A low set-level grade with many case-level failures confirms systemic problems. Use the standard triage framework above.

When set-level and case-level results diverge — this is where interpretation matters:

How to use set-level grades in the verdict: Set-level grading is supplementary — it does not override the SHIP/ITERATE/BLOCK verdict, which is based on manifest-defined hard gates and soft targets. However, a low set-level grade on an otherwise SHIP-ready result should trigger a human review checkpoint: "Gate status says SHIP, but set-level quality assessment is below expectations. Review a sample of passing responses before shipping."

5. Top 3 actions — per the Triage Playbook's Layer 3 (Remediation Mapping)

List exactly three actions in priority order. Each must follow the Playbook's remediation pattern: change X -> re-run Y -> expect Z.

Prioritize using the playbook's gate-first order:

1. Failed hard gates first, especially trust & safety (guardrails, out_of_scope, sensitive_data, prompt_injection, compliance)
2. Agent-quality patterns affecting high-value/high-risk capability sets
3. Missed soft targets, starting with the lowest-scoring eval set
4. Recurring failures or regressions (same case failing across runs)
5. Eval-setup problems that distort the verdict or hide true risk

Examples of required specificity:

• "Change: Add the product FAQ document to the agent's knowledge sources. Re-run: Cases 4 and 7 (both show 'Question not answered'). Expect: Relevance to pass for product-related queries."
• "Change: Add an escalation instruction to the system prompt: 'If you cannot resolve the request, offer to connect the user with a human agent.' Re-run: Case 3 ('speak to a representative'). Expect: Relevance to pass."
• "Change: Update the expected response in case 5 — it references an outdated process. Re-run: Case 5 only. Expect: Compare Meaning score to improve (this is an eval setup fix, not an agent fix)."

6. Pattern analysis — per the Triage Playbook's Layer 4

Check for these cross-signal patterns from the Playbook:

Also check for concentration: if most failures share a root cause type, call it out. Per the Playbook: "80%+ same root cause = systemic issue, fix the category."

7. Interpretation rationale (teach the WHY)

After presenting the triage, explain the reasoning so the customer can apply this framework independently next time. Cover these four points:

• Why the verdict landed where it did: Walk through the hard gates and soft targets with the actual per-set numbers. Example: "The guardrails trust & safety set is a hard gate and passed at 100% against a 95% target, so it did not BLOCK. The billing accuracy capability set is a soft target and passed at 72% against an 80% target, so the verdict is ITERATE — even though aggregate pass rate is 86%, the missed soft target still needs a fix."
• Why failures were classified the way they were: For each Step 7 bucket and fine taxonomy used, explain the reasoning chain. Example: "Cases 3 and 7 were classified as eval-setup problems / Eval Setup Issue because the agent's actual response is substantively correct — it answers the question accurately — but the grader rejected it due to phrasing differences. The expected response says 'Contact support at 1-800-555-0100' but the agent says 'You can reach our support team at 1-800-555-0100.' Same information, different words. This is a grader rigidity problem, not an agent-quality problem."
• Why the top 3 actions are in that priority order: Connect each action's priority to the triage framework. Example: "The knowledge source fix is #1 because it addresses 4 of 6 agent failures and they're all core business scenarios. The prompt tweak is #2 because it fixes 2 failures but they're capability scenarios, which rank lower in the Playbook's priority order. The eval fix is #3 because it doesn't improve the agent — it just corrects the measurement."
• What this triage does NOT tell you: Name the limits. Example: "This triage is based on a single eval run. It cannot detect non-determinism issues (run the eval 3 times to check for variance). It also cannot assess whether your test cases cover the right scenarios — a passing eval with poor coverage is worse than a failing eval with good coverage."

This section teaches the methodology so customers can eventually interpret results without the skill. Each bullet must reference the specific data from this eval run, not generic advice.

8. Next-run recommendation

End with one sentence naming exactly what to re-run after making changes. Per the Playbook's re-run targeting:

Tip: After re-running, use Copilot Studio's Result comparison feature to compare the new run against the previous one. It shows which cases flipped pass→fail or fail→pass, making it easy to verify your changes fixed the intended failures without introducing regressions.

8b. Version comparison interpretation (when the customer provides two runs)

If the customer provides results from two eval runs (before/after a change, or two agent configurations), produce a comparison analysis in addition to the standard triage above. Accept this as two CSV files, two pasted summaries, or a description like "Run 1 was 78%, Run 2 is 85%."

Comparison table:

Case-level delta analysis:

Categorize every test case into one of four buckets:

Interpreting deltas:

• +/-5% overall variance between runs is normal (LLM non-determinism). Do not celebrate or panic over small swings. Run the eval 3 times and take the median to distinguish signal from noise.
• A case that flips between runs (pass in one, fail in another, on the same agent version) is a reliability problem, not a quality problem. Flag it separately.
• Regressions outnumbering fixes after a change means the change had a net negative impact, consider reverting.
• All fixes in one category, all regressions in another = instruction conflict. The prompt change that fixed trust & safety responses may have over-constrained business responses. This is the most common pattern when system prompt edits have unintended side effects.

Capability, trust & safety, and regression framing: Help the customer understand what each eval run type is FOR:

• Capability eval sets measure task quality dimensions such as accuracy, faithfulness/groundedness, relevancy, style/tone, and reasoning/tool use. Hallucination belongs here as a faithfulness failure.
• Trust & safety eval sets measure refusal, scope, sensitive-data, prompt-injection, and compliance behavior. These are usually hard gates.
• Regression sets re-run previously passing or critical test cases after changes. Pass rates should be near target; any drop is a regression that must be investigated.
• Gate-only sets run at milestones such as pre-pilot, pre-production, or post-significant-change.

A healthy eval practice uses both capability and trust & safety sets, partitioned into Step 8 regression and gate-only suites. If the customer is only running one type, recommend adding the other.

9. Production optimization-loop plan (Step 9)

Close with a short forward-looking optimization-loop plan that bridges this eval to continuous post-deployment improvement:

1. Collect signals: thumbs-down (highest signal), escalations, manual overrides, support tickets, and qualitative feedback.
2. Cluster: group signals into recurring failure patterns and map each cluster to existing eval sets or coverage gaps.
3. Decide fix location: agent config (prompt/retrieval/tools/orchestration), rubric/eval setup, or new eval cases.
4. Ship: make the smallest safe change with an owner and version note.
5. Re-evaluate: run the affected cases plus the Step 8 regression suite, then update the failure-pattern log and regression suite with any new cases.

After displaying the triage report in conversation, generate a formatted report:

Eval Results Triage Report (.docx) Use the docx skill to create a formatted document containing:

• Title: "Eval Results Triage Report"
• Date and agent name (if known)
• Score summary table
• Verdict (SHIP/ITERATE/BLOCK) with gate explanation
• Per-set actual pass rate vs target and hard/soft gate status
• Capability and trust & safety results reported separately
• Failure triage details for each failing case, including Step 7 bucket and fine taxonomy
• Failure-pattern log
• Top 3 prioritized actions
• Pattern analysis
• Interpretation rationale (from section 7 — the WHY behind the verdict, classifications, and priorities)
• Human review checkpoints table (from Step 4)
• Next-run recommendation
• Production optimization-loop plan (Step 9)

After the output file and before the conversation ends, display a Human Review Required section. Eval interpretation is where bad assumptions become bad decisions — a wrong verdict can ship a broken agent or block a good one. These checkpoints flag where human judgment is essential.

Human Review Required

After the checkpoints, add:

• Mandatory reminder: "This triage report was AI-generated from your eval results. Before acting on the verdict or remediation actions, review the failing cases with your team — especially any classified as eval setup issues. The distinction between an agent problem and an eval problem requires human judgment."

Copilot Studio deletes test run results after 89 days. Always recommend that the user:

1. Export the results CSV immediately after each eval run (Test set → Export results)
2. Store alongside the agent version in SharePoint or a repo
3. If the report recommends re-running after fixes, export the current results before changes so before/after comparison is possible

Include this reminder at the end of every generated report.

• State the verdict FIRST, before any analysis.
• Prefer manifest metadata over inference for set type, category, testing method, gate type, target, regression class, and provenance.
• Drive SHIP / ITERATE / BLOCK from hard gates and soft targets, not aggregate pass rate.
• BLOCK immediately if any hard trust & safety gate fails. Trust & trust & safety failures are non-negotiable unless the owner explicitly reclassifies the gate in the manifest.
• Always check whether failures are eval-setup problems before blaming the agent. This is the most common mistake in eval interpretation.
• Every failure must be classified into exactly one Step 7 bucket: eval-setup problem or agent-quality problem. Preserve the fine taxonomy as a secondary label.
• If pass rate is 100%, treat it as a red flag and say so.
• If input is too sparse for a confident verdict, default to ITERATE and explain why.
• When you cannot determine if a failure is an agent-quality problem or eval-setup problem from the CSV alone, say so explicitly and tell the user to read the actualResponse for that row.
• Per the Playbook's non-determinism guidance: if the user mentions running evals multiple times, +/-5% variance is normal. +/-10% requires investigation.