exploring-llm-clusters

PostHog automatically bundles an app's AI requests into groups of similar ones; this reads those groups, so a wall of individual requests becomes a handful of recognisable patterns.

WHEN TO USE

SCOPE

USE FOR

Check what the latest grouping run found and how many requests it covered.
Find out which group of requests costs the most or takes the longest.
Spot a group of AI requests that fails far more often than the rest.
Open up the individual requests behind a group to see what they really are.

This is a saved copy of the skill's SKILL.md instructions. It may differ from the upstream file or the version you install.

In this document (8)

Exploring LLM clusters

Use this skill when investigating AI observability clusters — understanding what patterns exist in your AI/LLM traffic, comparing cluster behavior, and drilling into individual clusters.

Tools

Tool	Purpose
`posthog:llma-clustering-job-list`	List clustering job configurations for the team
`posthog:llma-clustering-job-get`	Get a specific clustering job by ID
`posthog:execute-sql`	Query cluster run events and compute metrics
`posthog:query-llm-traces-list`	Find traces belonging to a cluster
`posthog:query-llm-trace`	Inspect a specific trace in detail

How clustering works

PostHog clusters LLM traces, individual generations, or evaluation events by embedding similarity. A Temporal workflow runs periodically or on-demand, producing cluster events stored as $ai_trace_clusters (trace-level), $ai_generation_clusters (generation-level), or $ai_evaluation_clusters (evaluation-level).

Each cluster event contains:

$ai_clustering_run_id — unique run identifier (format: <team_id>_<level>_<YYYYMMDD>_<HHMMSS>[_<job_id>])
$ai_clustering_level — "trace", "generation", or "evaluation"
$ai_window_start / $ai_window_end — time window of the data that was analyzed
$ai_total_items_analyzed — number of traces, generations, or evaluations processed
$ai_clusters — JSON array of cluster objects
$ai_clustering_params — algorithm parameters used

The analyzed window closes when a run starts, and the cluster event lands once the run finishes. So the cluster event's own timestamp is always after $ai_window_end, by anything from seconds to hours. Use the window only to bound the traces, generations, and evaluations that were analyzed. To find the cluster event itself, filter on $ai_clustering_run_id with a plain recent-time bound.

Cluster object shape (inside `$ai_clusters`)

{
  "cluster_id": 0,
  "size": 42,
  "title": "User authentication flows",
  "description": "Traces involving login, signup, and token refresh operations",
  "traces": {
    "<trace_or_generation_id>": {
      "distance_to_centroid": 0.123,
      "rank": 0,
      "x": -2.34,
      "y": 1.56,
      "timestamp": "2026-03-28T10:00:00Z",
      "trace_id": "abc-123",
      "generation_id": "gen-456"
    }
  },
  "centroid_x": -2.1,
  "centroid_y": 1.4
}

cluster_id: -1 is the noise/outlier cluster (items that didn't fit any cluster)
Items in traces are keyed by trace ID (trace-level), generation event UUID (generation-level), or evaluation event UUID (evaluation-level)
rank orders items by proximity to centroid (0 = closest)
x, y are 2D coordinates for visualization (UMAP/PCA/t-SNE reduced)

Clustering jobs

Each team can have up to 10 clustering jobs. A job defines:

name — human-readable label
analysis_level — "trace", "generation", or "evaluation"
event_filters — property filters scoping which items are included
enabled — whether the job runs on schedule

Default jobs named "Default - traces", "Default - generations", and "Default - evaluations" are auto-created and disabled when a custom job is created for the same level.

Workflow: explore clusters

Step 1 — List recent clustering runs

posthog:execute-sql
SELECT
    toString(properties.$ai_clustering_run_id) AS run_id,
    toString(properties.$ai_clustering_level) AS level,
    toString(properties.$ai_clustering_job_id) AS job_id,
    toString(properties.$ai_clustering_job_name) AS job_name,
    toString(properties.$ai_window_start) AS window_start,
    toString(properties.$ai_window_end) AS window_end,
    toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
    timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
    AND timestamp >= now() - INTERVAL 14 DAY
ORDER BY timestamp DESC
LIMIT 10

Step 2 — Get clusters from a specific run

posthog:execute-sql
SELECT
    toString(properties.$ai_clustering_run_id) AS run_id,
    toString(properties.$ai_clustering_level) AS level,
    toString(properties.$ai_clustering_job_id) AS job_id,
    toString(properties.$ai_clustering_job_name) AS job_name,
    toString(properties.$ai_window_start) AS window_start,
    toString(properties.$ai_window_end) AS window_end,
    toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
    properties.$ai_clusters AS clusters,
    properties.$ai_clustering_params AS params,
    timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
    AND timestamp >= now() - INTERVAL 14 DAY
    AND toString(properties.$ai_clustering_run_id) = '<run_id>'
ORDER BY timestamp DESC
LIMIT 1

Keep the lookback bound wide enough to cover the timestamp Step 1 reported for the run. Never bound this query with $ai_window_start / $ai_window_end. The cluster event is emitted after the window closes, so those bounds return zero rows.

The clusters field is a JSON array. Parse it to see cluster titles, sizes, descriptions, optional metrics, and each cluster's traces map.

Important: The clusters JSON can be very large (thousands of trace, generation, or evaluation IDs with coordinates). When the result is too large for inline display, it auto-persists to a file. Use print_clusters.py from scripts/ to get a readable summary.

Step 3 — Compute metrics for clusters

For trace-level clusters, compute cost/latency/token metrics:

posthog:execute-sql
SELECT
    properties.$ai_trace_id as trace_id,
    sum(toFloat(properties.$ai_total_cost_usd)) as total_cost,
    max(toFloat(properties.$ai_latency)) as latency,
    sum(toInt(properties.$ai_input_tokens)) as input_tokens,
    sum(toInt(properties.$ai_output_tokens)) as output_tokens,
    countIf(properties.$ai_is_error = 'true') as error_count
FROM events
WHERE event IN ('$ai_generation', '$ai_embedding', '$ai_span')
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND properties.$ai_trace_id IN ('<trace_id_1>', '<trace_id_2>', ...)
GROUP BY trace_id

For generation-level clusters, match by event UUID:

posthog:execute-sql
SELECT
    toString(uuid) as generation_id,
    toFloat(properties.$ai_total_cost_usd) as cost,
    toFloat(properties.$ai_latency) as latency,
    toInt(properties.$ai_input_tokens) as input_tokens,
    toInt(properties.$ai_output_tokens) as output_tokens,
    if(properties.$ai_is_error = 'true', 1, 0) as is_error
FROM events
WHERE event = '$ai_generation'
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND uuid IN ('<gen_uuid_1>', '<gen_uuid_2>', ...)

For evaluation-level clusters, first check each cluster's metrics field from $ai_clusters (for example pass rate, N/A rate, dominant evaluator name, and average judge cost). When you need individual evaluation rows, match by event UUID:

posthog:execute-sql
SELECT
    toString(uuid) AS evaluation_id,
    toString(properties.$ai_trace_id) AS trace_id,
    toString(properties.$ai_target_event_id) AS generation_id,
    toString(properties.$ai_evaluation_name) AS evaluation_name,
    toString(properties.$ai_evaluation_result) AS evaluation_result,
    toString(properties.$ai_evaluation_reasoning) AS evaluation_reasoning,
    toFloatOrNull(toString(properties.$ai_total_cost_usd)) AS judge_cost,
    timestamp
FROM events
WHERE event = '$ai_evaluation'
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND uuid IN ('<eval_uuid_1>', '<eval_uuid_2>', ...)

Step 4 — Drill into specific traces

Once you've identified interesting clusters, use the trace tools to inspect individual traces:

posthog:query-llm-trace
{
  "traceId": "<trace_id_from_cluster>",
  "dateRange": {"date_from": "<window_start>", "date_to": "<window_end>"}
}

When you need message content

Use events for cluster events, IDs, cost/latency/token metrics, and evaluation rows. Do not query events.properties.$ai_input, $ai_output, or $ai_output_choices when you need user messages or full model inputs/outputs — those heavy fields live on posthog.ai_events.

For a few representative examples, prefer query-llm-trace; it reads posthog.ai_events for you and returns the full event tree. For batch extraction, first get the trace IDs from the cluster, then query posthog.ai_events anchored on trace_id:

posthog:execute-sql
SELECT
    trace_id,
    timestamp,
    span_id,
    event,
    model,
    input,
    output_choices
FROM posthog.ai_events
WHERE trace_id IN ('<trace_id_1>', '<trace_id_2>', ...)
ORDER BY trace_id, timestamp

posthog.ai_events has a shorter retention window than events; older clusters may still have metadata and metrics but no message content. For more detail, use the exploring LLM traces skill's event reference.

Investigation patterns

"What kinds of LLM usage do we have?"

List recent clustering runs (Step 1)
Load the latest run's clusters (Step 2)
Review cluster titles and descriptions — each represents a distinct usage pattern
Compare cluster sizes to understand traffic distribution

"Which cluster is most expensive / slowest?"

Load clusters from a run (Step 2)
Extract trace IDs from each cluster
Compute metrics per cluster (Step 3)
Aggregate: avg(cost), avg(latency), sum(cost) per cluster
Compare across clusters

"What's in this cluster?"

Load the cluster's traces (from the traces field)
Sort by rank (closest to centroid = most representative)
Inspect the top 3-5 traces via query-llm-trace to understand the pattern
Check the cluster title and description for the AI-generated summary

"Are there error-heavy clusters?"

Compute metrics (Step 3) with error_count
Calculate error rate per cluster: items_with_errors / total_items
Focus on clusters with high error rates
Drill into errored traces to find root causes

"How do clusters compare across runs?"

List multiple runs (Step 1)
Load clusters from each run
Compare cluster titles — similar titles across runs indicate stable patterns
Track cluster size changes to detect shifts in traffic patterns

Constructing UI links

Clusters overview: https://app.posthog.com/ai-observability/clusters
Specific run: https://app.posthog.com/ai-observability/clusters/<url_encoded_run_id>
Cluster detail: https://app.posthog.com/ai-observability/clusters/<url_encoded_run_id>/<cluster_id>

Always surface these links so the user can verify visually in the PostHog UI.

Tips

Always set a time range in SQL queries — cluster events without time bounds are slow
Bound a search for a cluster event by when it was emitted, not by the window it analyzed — a run's $ai_window_end is earlier than the event's own timestamp
Start with run listing to orient, then drill into specific clusters
Cluster titles and descriptions are AI-generated summaries — verify by inspecting traces
The noise cluster (cluster_id: -1) contains outliers that didn't fit any pattern
Use llma-clustering-job-list to understand what clustering configs are active
Trace IDs in clusters can be used directly with query-llm-trace for deep inspection
Message content lives on posthog.ai_events, not events.properties; use query-llm-trace unless you need custom batch SQL
For large clusters, inspect the top-ranked traces (closest to centroid) for representative examples

This skill ships inside the PostHog AI Plugin. Listed package contents include 3 slash commands, 1 agent, hooks and an MCP server. Installed components and skill versions may vary by AI tool.

Install just this skill

Run this outside your AI tool, using Node.js 22.20.0 or newer. Skills CLI 1.7.2 skips skill selection; prompts cancel installation without an interactive terminal. If choosing Project when asked about installation scope, open your project's folder before running this.

Install checklist

Check selected tools if a picker appears; Claude Code may be unchecked under Additional agents. ↑/↓ moves, Space toggles selections, Enter confirms.
Without a tool picker, check Installing to: for your intended tool. Stop if it names a different tool.
If asked, choose Symlink (Recommended) for shared files or Copy to all agents for separate copies; review the summary before confirming installation.
Install the find-skills skill? defaults to Yes, adding another skill for all projects. Choose No if you don't want that extra installation.

Skills CLI 1.7.2 sends install data to skills.sh by default, including the source, skill names, target tools and file details; GitHub sources are reported only when GitHub confirms the repository is public, while other remote sources can also be reported. Confirmed-public GitHub sources also trigger a security-audit request. Before installing, set the environment variable DO_NOT_TRACK=1 or DISABLE_TELEMETRY=1 to disable both.

npx skills add PostHog/ai-plugin --skill exploring-llm-clusters exploring-llm-traces --full-depth

It installs two skills: this one and exploring-llm-traces, which it relies on. Once these skills are installed where Claude Code loads them, their descriptions help Claude decide when to use them.

MCP server required

The skill's steps call for tools from the PostHog MCP server. This command doesn't add that connection; set it up in your AI tool if it isn't there yet: PostHog MCP setup. Installing the whole plugin instead includes the connection settings.

Requirements for the PostHog MCP server

Connecting is free; sign in with your PostHog account.
Some tools use AI internally and may incur PostHog AI charges. Those tools require AI data processing to be enabled in your organization settings.

Install the whole plugin

This route installs the posthog plugin package selected for your AI tool.

Claude Code

Add PostHog's marketplace, then install the plugin

Run both lines in a terminal. They install PostHog's posthog plugin: this skill, PostHog's other skills, and a connection to PostHog's hosted MCP server.

claude plugin marketplace add PostHog/ai-plugin

02
```
claude plugin install posthog@posthog
```

Installation and loading

The plugin installs for you in every project and loads the next time you start Claude Code, or after /reload-plugins in an open session.
To connect your PostHog account, start claude, type /mcp, select plugin:posthog:posthog and follow the browser prompts.

Good to know

The plugin includes PostHog's session-capture hook; PostHog's README says it sends sessions to LLM Analytics only when POSTHOG_LLMA_CC_ENABLED=true and POSTHOG_API_KEY are both set.
If you use the separate posthog-telemetry plugin, PostHog says to keep only one of the two, because both register the same hook.
On a self-hosted PostHog, this route's MCP connection won't reach your instance: the plugin's config is fixed to https://mcp.posthog.com/mcp, which routes to PostHog's US or EU region. POSTHOG_MCP_URL works only for the Gemini CLI extension.

What's inside

Files included with this skill.

View on GitHub

Same repository

adding-warehouse-person-properties
Sync columns from a synced data warehouse table onto PostHog person or group properties, so warehouse data becomes usable anywhere person and group properties already work: feature flag targeting, cohorts, insight filters and breakdowns, surveys, session replay filters, workflows, and the person profile. Use when the user wants to "add a person property from my warehouse", "enrich people with Stripe/Postgres/Salesforce data", "put ARR or plan tier on my persons", "target a feature flag by a warehouse column", "sync warehouse columns onto groups or organizations", or wants to inspect, backfill, disable, or debug an existing warehouse-backed person or group property.Read full description
analyzing-expensive-users
Analyze the most expensive users in AI observability and explain why they cost so much. Use when the user asks about top spenders, expensive users, per-user LLM cost, user-level cost drivers, or patterns behind high AI observability spend.Read full description
analyzing-experiment-session-replays
Analyze session replay patterns across experiment variants to understand user behavior differences. Use when the user wants to see how users interact with different experiment variants, identify usability issues, compare behavior patterns between control and test groups, or get qualitative insights to complement quantitative experiment results. Also covers pairing the observed behavior with a linked survey when the user wants qualitative feedback beyond what recordings show.Read full description
assessing-heatmaps
Assesses what a page's heatmap is telling you and recommends concrete changes. Pulls click / rageclick / scroll-depth data for a URL, names the hot elements by cross-referencing autocapture events on the same page, and can create a saved heatmap the user opens in PostHog, then summarizes the behavior and proposes improvements. TRIGGER when: user asks what a heatmap shows, why people aren't clicking something, where users rage-click, how far they scroll, what to change on a page based on heatmap/click data, or to 'analyze/assess/review the heatmap' for a URL. DO NOT TRIGGER when: the user only wants to create a saved heatmap screenshot with no analysis (use heatmaps-saved-create directly), or is asking about session replay in general (use investigating-replay).Read full description
auditing-endpoints
Audit every endpoint in a PostHog project for staleness, failed materialisations, and unused materialised versions. Use when the user asks "what endpoints can I clean up?", "are any of my endpoints broken?", "which materialised versions are still being called?", or wants a one-shot cleanup pass over the Endpoints product. Produces a prioritised report grouped by issue type, with recommended actions but does not modify anything without explicit confirmation.Read full description
auditing-experiments-flags
Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup.Read full description