Labsco
microsoft logo

ci-test-failures

✓ Official★ 6,148

by microsoft · part of microsoft/aspire

Guide for diagnosing GitHub Actions test failures, extracting failed tests from runs, and creating or updating failing-test issues. Use this when asked to investigate GitHub Actions test failures, download failure logs, create failing-test issues, or debug CI issues.

🔥🔥✓ VerifiedFreeQuick setup
#git
🔒 Repo-maintenance skill. It exists to help maintain microsoft/aspire itself — it's only useful if you contribute code to that project.

This is the playbook your agent receives when the skill activates — you don't need to read it to use the skill, but it's here to audit before installing.

CI Test Failure Diagnosis and Issue Filing

Recipe: Create an Issue for a Test Failure

When the user asks to create an issue for a failing test, follow these steps. Always redirect full output to a log file (not tail) so you can inspect it if the command fails.

Step 1: List failed tests (if user didn't specify one)

Omit --test to discover all failures. Redirect output to a log file:

dotnet run --project tools/CreateFailingTestIssue -- \
  --url "<the-url-the-user-gave>" \
  --output /tmp/cfti-result.json \
  > /tmp/cfti-list.log 2>&1
dotnet run --project tools/CreateFailingTestIssue -- `
  --url "<the-url-the-user-gave>" `
  --output $env:TEMP/cfti-result.json `
  > $env:TEMP/cfti-list.log 2>&1

Then read the result with jq:

jq '{ success, availableFailedTests: .diagnostics.availableFailedTests, errorMessage: .errorMessage }' /tmp/cfti-result.json
Get-Content $env:TEMP/cfti-result.json | ConvertFrom-Json | Select-Object success, errorMessage, @{N='availableFailedTests';E={$_.diagnostics.availableFailedTests}}

If success is false, inspect the full log: cat /tmp/cfti-list.log (bash) or Get-Content $env:TEMP/cfti-list.log (PowerShell).

Ask the user which test to file for, then proceed to Step 2.

Step 2: Create the issue

dotnet run --project tools/CreateFailingTestIssue -- \
  --url "<the-url-the-user-gave>" \
  --test "<test-name>" \
  --create \
  --output /tmp/cfti-result.json \
  > /tmp/cfti-create.log 2>&1
dotnet run --project tools/CreateFailingTestIssue -- `
  --url "<the-url-the-user-gave>" `
  --test "<test-name>" `
  --create `
  --output $env:TEMP/cfti-result.json `
  > $env:TEMP/cfti-create.log 2>&1

Then read the result:

jq '{ success, issue: .issue.createdIssue, errorMessage: .errorMessage }' /tmp/cfti-result.json
Get-Content $env:TEMP/cfti-result.json | ConvertFrom-Json | Select-Object success, errorMessage, @{N='issue';E={$_.issue.createdIssue}}

If success is false, inspect the full log: cat /tmp/cfti-create.log (bash) or Get-Content $env:TEMP/cfti-create.log (PowerShell).

That's it — do not add analysis comments, do not use --dry-run unless the user explicitly asks for a preview.

Rules:

  • Always use --output <file> to keep JSON clean. Do NOT try to parse JSON from stdout — it is interleaved with dotnet build progress output.
  • Use jq to extract fields from the output file. Key paths:
    • .success — whether the operation succeeded
    • .issue.createdIssue.number and .issue.createdIssue.url — the created/updated issue
    • .diagnostics.availableFailedTests[] — test names when --test is omitted
    • .errorMessage — error details when .success is false
  • Do NOT create issues manually with gh issue create. The tool handles everything: resolving the run, finding the test, generating a template-compliant body, and creating the issue.
  • Do NOT invent your own issue markdown. The tool generates content that matches .github/ISSUE_TEMPLATE/50_failing_test.yml.
  • If the tool fails, report the error from diagnostics.log and the JSON output. Do not fall back to manual issue creation.

Recipe: Investigate a Failing Run (No Issue Creation)

To download and inspect failure artifacts without creating an issue:

cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>
Set-Location tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>

Then search the downloaded logs and .trx files for errors.

Reference Documentation

Everything below is reference material for edge cases and deeper investigation.

Overview

Use this skill in two phases:

  1. Investigate the run with DownloadFailingJobLogs.cs to fetch failed job logs and artifacts.
  2. Create or update a failing-test issue with tools/CreateFailingTestIssue --create.

Tools covered

ToolPurposeLocation
DownloadFailingJobLogs.csDownload failed job logs and test artifacts from a GitHub Actions runtools/scripts/DownloadFailingJobLogs.cs
CreateFailingTestIssueResolve a failing test from PR/run/job URLs and create/update issuestools/CreateFailingTestIssue
/create-issue workflowCreate, reopen, or comment on failing-test issues from issue/PR comments.github/workflows/create-failing-test-issue.yml

What the Tool Does

  1. Finds all failed jobs in a GitHub Actions workflow run
  2. Downloads job logs for each failed job
  3. Extracts test failures and errors from logs using regex patterns
  4. Determines artifact names from job names (pattern: logs-{testShortName}-{os})
  5. Downloads test artifacts containing .trx files and test logs
  6. Extracts artifacts to local directories for inspection

Example Workflow

# 1. Check failed jobs on a PR
gh pr checks 14105 --repo microsoft/aspire 2>&1 | Where-Object { $_ -match "fail" }

# 2. Get the run ID
$runId = gh run list --repo microsoft/aspire --branch davidfowl/my-branch --limit 1 --json databaseId --jq '.[0].databaseId'

# 3. Download failure logs
cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- $runId

# 4. Search for errors in downloaded logs
Get-Content "failed_job_0_*.log" | Select-String -Pattern "error|Error:" -Context 2,3 | Select-Object -First 20

# 5. Check .trx files for test failures
Get-ChildItem -Recurse -Filter "*.trx" | ForEach-Object {
    [xml]$xml = Get-Content $_.FullName
    $xml.TestRun.Results.UnitTestResult | Where-Object { $_.outcome -eq "Failed" }
}

# 6. Create or update the failing-test issue from the PR or issue thread
/create-issue --test "Tests.Namespace.Type.Method(input: 1)" --url https://github.com/microsoft/aspire/actions/runs/$runId

Understanding Job Log Output

The tool prints a summary for each failed job:

=== Failed Job 1/1 ===
Name: Tests / Integrations macos (Hosting.Azure) / Hosting.Azure (macos-latest)
ID: 56864254427
URL: https://github.com/microsoft/aspire/actions/runs/19846215629/job/56864254427
Downloading job logs...
Saved job logs to: failed_job_0_Tests___Integrations_macos__Hosting_Azure____Hosting_Azure__macos-latest_.log

Errors found (2):
  - System.InvalidOperationException: Step 'provision-api-service' failed...

Searching Downloaded Logs

Find Errors in Job Logs

# PowerShell
Get-Content "failed_job_*.log" | Select-String -Pattern "error|Error:" -Context 2,3

# Bash
grep -i "error" failed_job_*.log | head -50

Find Build Failures

Get-Content "failed_job_*.log" | Select-String -Pattern "Build FAILED|error MSB|error CS"

Find Test Failures

Get-Content "failed_job_*.log" | Select-String -Pattern "Failed!" -Context 5,0

Check for Disk Space Issues

Get-Content "failed_job_*.log" | Select-String -Pattern "No space left|disk space"

Check for Timeout Issues

Get-Content "failed_job_*.log" | Select-String -Pattern "timeout|timed out|Timeout"

Using GitHub API for Annotations

Sometimes job logs aren't available (404). Use annotations instead:

gh api repos/microsoft/aspire/check-runs/<job-id>/annotations

This returns structured error information even when full logs aren't downloadable.

Common Failure Patterns

Disk Space Exhaustion

Symptom: No space left on device in annotations or logs

Diagnosis:

gh api repos/microsoft/aspire/check-runs/<job-id>/annotations 2>&1

Common fixes:

  • Add disk cleanup step before build
  • Use larger runner (e.g., 8-core-ubuntu-latest)
  • Skip unnecessary build steps (e.g., /p:BuildTests=false)

Command Not Found

Symptom: exit code 127 or command not found

Diagnosis:

Get-Content "failed_job_*.log" | Select-String -Pattern "command not found|exit code 127" -Context 3,1

Common fixes:

  • Ensure PATH includes required tools
  • Use full path to executables
  • Install missing dependencies

Test Timeout

Symptom: Test hangs, then fails with timeout

Diagnosis:

Get-Content "failed_job_*.log" | Select-String -Pattern "Test host process exited|Timeout|timed out"

Common fixes:

  • Increase test timeout
  • Check for deadlocks in test code
  • Review Heartbeat.cs output for resource exhaustion

Build Failure

Symptom: Build FAILED or MSBuild errors

Diagnosis:

Get-Content "failed_job_*.log" | Select-String -Pattern "error CS|error MSB|Build FAILED" -Context 0,3

Common fixes:

  • Check for missing project references
  • Verify package versions
  • Download and analyze .binlog from artifacts

Artifact Contents

Downloaded artifacts typically contain:

artifact_0_TestName_os/
├── testresults/
│   ├── TestName_net10.0_timestamp.trx    # Test results XML
│   ├── Aspire.*.Tests_*.log              # Console output
│   ├── recordings/                        # Asciinema recordings (CLI E2E tests)
│   └── workspaces/                        # Captured project workspaces (CLI E2E tests)
│       └── TestClassName.MethodName/      # Full generated project for failed tests
│           ├── apphost.ts
│           ├── aspire.config.json
│           ├── .aspire/modules/                  # Generated SDK (aspire.js) - key for debugging
│           └── ...
├── *.crash.dmp                            # Crash dump (if test crashed)
└── test.binlog                            # MSBuild binary log

CLI E2E Workspace Capture

CLI E2E tests annotated with [CaptureWorkspaceOnFailure] automatically capture the full generated project workspace when a test fails. This includes the generated SDK (.aspire/modules/aspire.js), template output, and config files — critical for debugging template generation or aspire run failures.

Look in testresults/workspaces/{TestClassName.MethodName}/ inside the downloaded artifact.

Parsing .trx Files

# Find all failed tests in .trx files
Get-ChildItem -Path "artifact_*" -Recurse -Filter "*.trx" | ForEach-Object {
    Write-Host "=== $($_.Name) ==="
    [xml]$xml = Get-Content $_.FullName
    $xml.TestRun.Results.UnitTestResult | Where-Object { $_.outcome -eq "Failed" } | ForEach-Object {
        Write-Host "FAILED: $($_.testName)"
        Write-Host $_.Output.ErrorInfo.Message
        Write-Host "---"
    }
}

Tips

Clean Up Before Running

Remove-Item *.log -Force -ErrorAction SilentlyContinue
Remove-Item *.zip -Force -ErrorAction SilentlyContinue
Remove-Item -Recurse artifact_* -Force -ErrorAction SilentlyContinue

Run From tools/scripts Directory

The tool creates files in the current directory, so run it from tools/scripts to keep things organized:

cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>
Set-Location tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>

Don't Commit Log Files

The downloaded log files can be large. Don't commit them to the repository:

# Before committing
rm tools/scripts/*.log
rm tools/scripts/*.zip
rm -rf tools/scripts/artifact_*
# Before committing
Remove-Item tools/scripts/*.log -Force -ErrorAction SilentlyContinue
Remove-Item tools/scripts/*.zip -Force -ErrorAction SilentlyContinue
Remove-Item tools/scripts/artifact_* -Recurse -Force -ErrorAction SilentlyContinue

See Also

  • tools/scripts/README.md - Full documentation
  • tools/scripts/Heartbeat.cs - System monitoring tool for diagnosing hangs
  • .agents/skills/cli-e2e-testing/SKILL.md - CLI E2E test troubleshooting