find-traces

One web request can touch a dozen different services on its way through; this follows a single request across all of them and shows which step failed or dragged.

WHEN TO USE

SCOPE

USE FOR

Look up everything that happened during one specific trace, given its trace ID.
Find the traces from the last hour that included an error.
Find the slowest traces in a service over a given time window.
See every step of a request laid out, longest first, to find what dominated the time.
Trace a failing step back to the service and code package that produced it.

NOT FOR

Skip this for aggregated performance metrics; that is a job for logs or metrics analysis, not traces.
Skip this on data that does not follow OpenTelemetry's trace and span field conventions.
Skip this once you already have the trace query you need — just run it directly.
Skip this for ongoing alerting on trace patterns; use Axiom Monitors instead.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (12)

Trace Analysis

Analyze OpenTelemetry distributed traces to identify errors, latency issues, and root causes.

Arguments

When invoked with a trace ID (e.g., /find-traces abc123...), it's available as $ARGUMENTS.

Trace Dataset Discovery

First, find trace datasets:

axiom dataset list -f json

Look for datasets containing trace data (often named *traces*, *spans*, or otel-*).

Schema Discovery

Always verify field names first:

axiom query "['<trace-dataset>'] | getschema" --start-time -1h

Common Operations

Get Trace by ID

axiom query "['<dataset>']
| where trace_id == '<TRACE_ID>'
| sort by _time asc
| limit 100" --start-time -1h -f json

Find Error Traces

axiom query "['<dataset>']
| where _time >= ago(1h)
| where error == true
| extend error = coalesce(ensure_field(\"error\", typeof(bool)), false)
| summarize
    start_time = min(_time),
    total_duration = max(duration),
    span_count = count(),
    error_count = countif(error),
    services = make_set(['service.name']),
    root_operation = arg_min(_time, name)
  by trace_id
| sort by start_time desc
| limit 20" --start-time -1h -f json

Find Slow Traces

axiom query "['<dataset>']
| where _time >= ago(1h)
| where duration >= 1000000000
| summarize
    start_time = min(_time),
    total_duration = max(duration),
    span_count = count(),
    services = make_set(['service.name'])
  by trace_id
| sort by total_duration desc
| limit 20" --start-time -1h -f json

Find Traces by Service

axiom query "['<dataset>']
| where _time >= ago(1h)
| where ['service.name'] == '<SERVICE>'
| summarize
    start_time = min(_time),
    total_duration = max(duration),
    span_count = count(),
    error_count = countif(error == true)
  by trace_id
| sort by start_time desc
| limit 20" --start-time -1h -f json

Error Spans in Trace

axiom query "['<dataset>']
| where trace_id == '<TRACE_ID>'
| where error == true
| project _time, ['service.name'], name, duration, ['status.message']" --start-time -1h -f json

Critical Path Analysis

axiom query "['<dataset>']
| where trace_id == '<TRACE_ID>'
| project span_id, parent_span_id, ['service.name'], name, duration, error
| sort by duration desc" --start-time -1h -f json

OTel Field Reference

Field	Bracket?	Description
`trace_id`	No	32-char trace identifier
`span_id`	No	16-char span identifier
`parent_span_id`	No	Parent span (empty for root)
`name`	No	Operation name
`duration`	No	Duration in nanoseconds
`kind`	No	CLIENT, SERVER, INTERNAL, PRODUCER, CONSUMER
`error`	No	Boolean error flag
`['service.name']`	Yes	Service identifier
`['status.code']`	Yes	OK, ERROR, or nil
`['status.message']`	Yes	Error description
`['scope.name']`	Yes	Instrumentation library

Duration Conversion

OTel durations are in nanoseconds:

Human	Nanoseconds	Filter
1 ms	1,000,000	`duration >= 1000000`
100 ms	100,000,000	`duration >= 100000000`
1 s	1,000,000,000	`duration >= 1000000000`

Convert for display:

| extend duration_ms = duration / 1000000.0

Custom Attributes

Non-standard span attributes are stored in attributes.custom map:

// Filter by custom attribute
| where ['attributes.custom']['user_id'] == "123"

// Aggregation requires explicit cast
| summarize count() by tostring(['attributes.custom']['tenant'])

Without tostring(), aggregations fail with "grouping by field of type unknown".

Codebase Correlation

When working in a repository that matches the traced service, correlate trace data with source code to identify root causes.

Mapping Trace Data to Code

Extract package/module path from ['scope.name']
- Contains the instrumentation library or package path
- Strip the module prefix to get the local path
- Example: github.com/org/repo/pkg/auth → pkg/auth
Find code from operation name
- The name field often contains function names or HTTP routes
- Search the codebase for matching handlers, functions, or endpoints
Trace the call chain
- Follow parent-child span relationships
- Map each span to its corresponding code location
- Identify where errors originate and propagate

Note: Codebase correlation is optional. Proceed with trace-only analysis if code is unavailable or doesn't match the traced services.

Output Format

When analyzing a trace, provide:

## Trace Summary
- **Trace ID:** <id>
- **Duration:** <human-readable>
- **Services:** <list>
- **Outcome:** success/failure

## Sequence of Events
1. <Service> - <operation> (<duration>)
2. <Service> - <operation> (<duration>) ⚠️ ERROR
...

## Error Analysis
<What failed, when, why>

## Root Cause
<Deepest error and explanation>

## Codebase Locations (if applicable)
- **Service:** <service.name>
- **Package:** <scope.name>
- **Files:** <specific files to investigate>

## Recommended Actions
1. <Specific action>
2. <What to investigate next>

When NOT to Use

Metrics analysis: Traces are for request flow; use logs/metrics skills for aggregated performance data
Non-OTel data: This skill assumes OpenTelemetry field conventions (trace_id, span_id, etc.)
Known trace structure: If you already have the query, run it directly without invoking this skill
Alerting on trace patterns: Use Axiom Monitors for continuous alerting

APL Reference

For query syntax, invoke the axiom-apl skill which provides trace analysis patterns and duration unit guidance.

This skill ships inside the Axiom CLI Skills plugin. Listed package contents include hooks. Installed components and skill versions may vary by AI tool.

Install just this skill

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add axiomhq/cli --skill "find-traces" --full-depth

After installing for Claude Code, open it in the installation project if you chose Project; Global makes the skill available across projects on this computer. Once Claude Code loads the skill, its description helps Claude decide when to use it.

If installation created a top-level skills folder that did not exist when your Claude Code session started, run /reload-skills in that session. Run it again after later changes in that folder.

Install the whole plugin

This route installs the Axiom CLI Skills plugin package selected for your AI tool.

PLUGIN MARKETPLACE

Add the marketplace, then install the plugin

Typed inside the agent's own prompt, not in a terminal. The marketplace is called axiom-cli, which is the part after the @.

01
```
/plugin marketplace add axiomhq/cli
```
02
```
/plugin install axiom-cli@axiom-cli
```

Open plugin page ↗

What's inside

Files included with this skill.

View on GitHub

Trace Analysis

Arguments

Trace Dataset Discovery

Schema Discovery

Common Operations

Get Trace by ID

Find Error Traces

Find Slow Traces

Find Traces by Service

Error Spans in Trace

Critical Path Analysis

OTel Field Reference

Duration Conversion

Custom Attributes

Codebase Correlation

Mapping Trace Data to Code

Output Format

When NOT to Use

APL Reference

Account or credentials required

Install just this skill

Install the whole plugin

Add the marketplace, then install the plugin

What's inside

More skills

axiom-apl

detect-anomalies

explore-dataset

MCP keyword matches

Sentry MCP

VictoriaMetrics MCP Server

Perplexity Comet Bridge

PostgreSQL-Ops MCP