signals-scout-ai-observability

Three cheap reads cold-start a run:

• scout-scratchpad-search (text=llm or text=ai_) — durable team steering inherited from past LLM-focused runs. Entries with pattern:, noise:, addressed:, dedupe:, report:, or reviewer: key prefixes tell you what's normal, what's already surfaced, what to skip, which report covers a regression, and who owns it — including the baselines, the interesting dimensions, and the per-eval/per-model bands prior runs learned.
• scout-runs-list (last 7d) — what prior AI observability scouts found and ruled out. Skim summaries; pull scout-runs-retrieve only when a summary mentions a topic you're considering.
• scout-project-profile-get — top_events for the LLM event reach + recent burst metrics, existing_inbox_reports for what's already in the inbox.
• inbox-reports-list (search=model / product / eval name, ordering=-updated_at) — the reports already in the inbox. Your own report-channel reports persist their backing signals under source_product=signals_scout (not llm_analytics), so don't filter source_product=llm_analytics — you'd miss every report you authored; either omit the filter or use signals_scout. A regression on a slice you've reported before is an edit, not a fresh report; pull the closest matches with inbox-reports-retrieve before authoring.

The lenses below are the surfaces worth watching. Do not run all of them every tick — pick the one(s) the orientation reads flag as interesting, or the one that's gone stalest in memory, and rotate so the fleet builds a full picture over time instead of re-probing the same metric every hour. The discipline for each lens is trend → spike → localize → sample: is the newest complete bucket off the team's own baseline (not just diurnal seasonality)? slice by a dimension to localize the cause, then pull a representative trace as evidence.

Discover the team's dimensions, don't guess them. Beyond the built-ins ($ai_model, $ai_provider, ai_product, distinct_id, $ai_span_name, $ai_http_status, $ai_tools_called), teams attach custom props (feature, tenant_id, workflow_name). Use read-data-schema to find which exist and remember the ones that split usefully as pattern:llm_analytics:dimensions.

references/lenses.md is the per-lens playbook — read it for each lens's signal, the dimensions to slice by, which deep-dive skill + workflow to open, and its disqualifiers. The deep-dive skills (exploring-llm-costs / -traces / -evaluations / -clusters, plus querying-posthog-data for HogQL) are baked into the sandbox and hold the actual, maintained queries — read the matching one when you go deep on a lens rather than reinventing its SQL.

When a lens flags something, don't report the top-line number — localize and sample:

• Localize. Slice the contributing $ai_generation / $ai_trace events by a dimension (model, $ai_span_name, tool, user, ai_product, a custom dim) to show which slice drove the move — that's the difference between "cost is up" and a reportable finding.
• Sample. Pull one or two representative traces via query-llm-trace (or a failing generation sampled from the raw $ai_evaluation rows) and cite concrete trace / generation / evaluation IDs in the evidence. A trace you cite links to /ai-observability/traces/<trace_id>: use the _posthogUrl on the query result. If you only have an id from raw SQL, call query-llm-trace with that id and the relevant date range, then use its _posthogUrl; if the trace cannot be fetched, cite the bare id. Never build the link by serializing the trace query into /insights/new#q=…, which opens an empty insight because TraceQuery isn't an insight source.
• Group as a pattern when a trend spans many traces: describe the shared shape (same model + same span, same tool error, same prompt version) rather than listing rows.

Memory is a continuous activity, not an end-of-run wrap-up. Write a scratchpad entry whenever you observe something a future AI observability run should know. Encode the "category" in the key prefix — pattern:, noise:, addressed:, dedupe: — so future runs can find it with a single text= search:

• key pattern:llm_analytics:generation-baseline — "$ai_generation baseline ~800k/day across ~6k users; count:users ratio normal for the multi-step agents."
• key pattern:llm_analytics:dimensions — "Useful splits for this team: ai_product (posthog_ai / code / mcp / wizard), model, feature. tenant_id not set."
• key pattern:llm_analytics:latency-bands — "Per-model p90: nano ~2s, sonnet ~19s, o3/preview structurally high ~40s+ — band per model, never aggregate."
• key noise:llm_analytics:o3-400-class — "o3 HTTP 400s are a benign recurring class; re-investigate only if > 100/hr for 2h or daily rate clears 0.05%."
• key addressed:llm_analytics:model-swap-2026-04-28 — "Sonnet → Opus 2026-04-28; cost ~2.1x baseline expected."
• key report:llm_analytics:<entity> — the report_id of a report you authored for a regression on this slice (a model, ai_product, eval, or cluster), so the next run edits it (append_evidence with the fresh window) instead of duplicating.
• key reviewer:llm_analytics:<area> — a resolved owner (bare lowercase GitHub login) for a product / model / eval area, so reports route to a human faster.

By run #5 you'll know the team's healthy baselines, which dimensions split usefully, which spikes recur, which evals deserve more or less weight, and who owns each surface.

Before you author, check whether this slice already has a report — the report:llm_analytics:<entity> scratchpad pointer is the reliable path: it holds the report_id, so inbox-reports-retrieve it directly. Only with no pointer fall back to an inbox-reports-list search (ordering=-updated_at), and search the slice's specific terms (the model, the ai_product, the eval name, the cluster id) — a broad word like latency returns hundreds of unrelated reports on a busy project and buries yours. Then, for each candidate:

• Edit the existing report via scout-edit-report when the inbox already covers the slice. A regression is rarely brand-new — a cost step that's still elevated, a latency band that hasn't recovered, an eval still failing more: use append_evidence for the fresh window's numbers (or rewrite the title/summary on a report you authored). This is the default when a match exists and it's still live in the inbox; don't mint a near-duplicate. A persistent regression is one report across runs: when a new complete window confirms the issue is ongoing, that's a re-escalation — add the fresh window to the report with append_evidence, and advance the dedupe: gate. Do not author a fresh report per tick. But check the matched report's status first: edit-report can't change status, so appending to a resolved / suppressed / failed report (one that won't surface) buries a real relapse under a closed item. When the prior report is no longer live, author a fresh report for the relapse and repoint report:llm_analytics:<entity> at the new id.
• Author a fresh report via scout-emit-report only when nothing live in the inbox covers it. New evidence on a regression an existing report already tracks is an edit, not a new report — emit-report is for a genuinely uncovered slice (or a relapse whose prior report is no longer live, per the Edit bullet). A strong finding here: confidence ≥ 0.85, the move localized to a specific slice (not an aggregate artifact), with concrete trace / generation / evaluation / cluster IDs and query results in the evidence. A cost / latency / eval regression is chart-shaped — attach the slice's daily series via charts so the move against its own baseline is visible in the report. A cost / latency / eval regression is an investigation, not a one-line code fix, so set actionability=requires_human_input and leave priority and repository unset — they're PR-autostart fields, and supplying priority + suggested_reviewers with no repository signals PR intent that spins up a repo-selection sandbox only to no-op (autostart needs immediately_actionable). Always set suggested_reviewers regardless — resolve the owning person via scout-members-list and pass their resolved github_login (or a {user_uuid}) as an object, since suggested_reviewers is a list of objects, not bare strings (cache the login under a reviewer:llm_analytics:<area> key). It's how the report reaches a human; left empty, the report is assigned to nobody and is likely missed. After authoring, write a report:llm_analytics:<entity> scratchpad entry with the report_id so the next run edits it instead of duplicating.
• Remember if it's below the bar but worth carrying forward, or to record what you ruled out and why.
• Skip with a one-line note in your final summary if a scratchpad entry with a noise: / addressed: / dedupe: key prefix, or an existing inbox report, already covers it.

If a prior run already covered the topic, default to edit-or-skip + memory refresh rather than authoring a near-duplicate. The same fact twice in the inbox degrades signal-to-noise more than missing one finding for one tick.

Summarize the run — one paragraph: which lens(es) you looked at, which reports you authored or edited, what you remembered, what you ruled out and why. The harness writes that summary to the run row as searchable prose; future runs read it via scout-runs-list. Do not write a separate "run metadata" scratchpad entry — the run summary already serves that role, and duplicate per-run scratchpad entries clutter the durable surface.