signals-scout-logs

A few cheap reads cold-start a run:

• scout-scratchpad-search (text=logs or text=service) — durable team steering from past logs-focused runs. Entries with pattern:, noise:, addressed:, dedupe:, report:, or reviewer: key prefixes tell you what's normal, what's already reported, what to skip, which report covers a service, and who owns it.

• scout-runs-list (last 7d) — what prior logs scouts found and ruled out.

• inbox-reports-list (filter by search=service/message, source_product, ordering=-updated_at) — the reports already in the inbox. A logs shift on a service you've reported before is an edit, not a fresh report; pull the closest matches with inbox-reports-retrieve before authoring.

• The delta read (primary) — logs-patterns-diff with query.dateRange = the recent window (last 1–3h for a routine run, or -1d for a periodic sweep) and baselineDateRange omitted (defaults to the same window one week earlier, absorbing daily/weekly cycles) or set to the window just before a known deploy. Read the new entries first (novel templates — a new entry at error/fatal severity is the highest-signal thing this scout surfaces, regardless of its volume share), then the top rate_shift entries (≥2x per-second rate change, magnitude in rate_ratio); those are your suspects. Check baseline.total_count before trusting a wall of new — a tiny or empty baseline (logging started recently, service didn't exist last week) makes everything look new. Scope by serviceNames/severityLevels to focus the sample budget. Pivot a suspect to its raw lines with query-logs via the entry's match_regex/match_literal + its services/severities.

• The cheap tripwire set (runs in seconds, no firehose) — secondary localizers that size what the delta read surfaced, plus an is-anything-loud check the diff's sampling can undercount; not an unfiltered baseline diff:

  1. logs-services-create over -1h (read the services list, ignore the sparkline; -1h/-24h are valid, -Nm is months) — the all-severity volume + per-service share in one call, vs the team's lines/hour + busiest-services baseline. This is what catches an info/warn flood (e.g. a stuck retry loop logging at info) that the severity-filtered probes below would miss, and it names the hot service for localization.
  2. logs-count severityLevels=["fatal"] over 24h (add a searchTerm for a specific crash signature) — fatal is rare, so this is cheap and catches crash loops.
  3. logs-count severityLevels=["error","fatal"] over the last 1h vs the team's error+fatal/hr baseline — a severity-shift proxy.
  4. logs-alerts-list — only a new firing alert beyond known-noise ones is interesting.

  Cold start (no pattern: baseline yet): the delta read is unaffected — it computes its own baseline window — so lead with it on a first run. The comparison tripwires #1 (all-severity volume / per-service share) and #3 (error+fatal/hr), by contrast, have nothing stored to diff against; derive each baseline from the same clock hour 24h (or 7d) ago via explicit ISO date_from/date_to before judging, and don't assume the current window is normal.

  If all are at baseline, close out empty. To localize a spike, scope logs-count-ranges to the hot service from step 1 — a severity-only range still buckets the whole stream and can 500 — then query-logs.

Patterns to watch — these are starting points, not a checklist.

A bounded logs-count (severity- or service-filtered) is materially above its baseline (≥ 2x). Localize by re-running logs-count (or logs-count-ranges for the time-bucketed shape) filtered by severity and by service — these tools count a filter, they don't group, so narrow with the filter and compare. Never widen to an unfiltered count to "see everything" — that 500s. Common causes: a stuck retry loop logging at info, a feature deploy that bumped log verbosity, a misconfigured logger emitting at debug in prod.

Cross-source convergence: if top_events shows $exception flat over the same window, this is logs-exclusive — handled-but-real failures the application catches and logs but doesn't re-raise. Distinct from anything error tracking will surface.

Total volume flat but error / fatal proportion rising. Captures the kind of failure error tracking misses: caught-and-logged exceptions, retry-with-eventual-success patterns, degraded-but-functional dependencies (slow DB, cold cache, partial third-party outage).

Validate in one call with logs-services-create (read-only despite the name) over the recent window — it returns the top-25 services with error_count, error_rate, and volume_share_pct, so you see which service carries the rise without walking per-service counts. Read only the services list and ignore the bundled sparkline — the sparkline is hundreds of KB and overflows the budget to a file; the services list itself is tiny. Call it without a severity filter to get each service's error_rate, or with severityLevels=["error","fatal"] to rank services by error volume. A single service accounting for the rise is high-confidence; a uniform rise across services suggests an upstream platform issue. Drop to query-logs only for module-level detail within the culprit service.

A service that normally accounts for a meaningful share of total log volume drops to near-zero. Different shape from error tracking entirely — there's no exception, the service is just gone.

Validate: logs-services-create (read-only; read the services list, ignore the sparkline) ranks active services by volume_share_pct in one call — a service that held meaningful share before and is now absent from the list is the signal. Confirm with logs-count-ranges for that service over today vs 7d-prior (use logs-count-ranges, not logs-sparkline-query — the sparkline endpoint 500s on busy services over multi-hour windows). Cross-check top_events for the service's expected user-facing events — if those also dropped, the service is genuinely down.

logs-patterns-diff is the tool here — it labels each template new / rate_shift / gone across two windows, so you never hand-match first_seen across two query-logs pulls. A new template firing at scale (or any new error/fatal template, even at a tiny share) is a fresh code path; a rate_shift with a high rate_ratio is an existing path whose frequency jumped. Trust the labels — the tool's thresholds already handle sampling honesty (a novelty floor, ≥2x rate change, minimum raw samples on both sides). Template identity is fingerprint-based, so a value rendered User <*> not found in one window and User <num> not found in the other compares as one pattern rather than a false new+gone pair — a template that reads new is genuinely novel, not a re-render.

Pivot a suspect to its underlying lines with query-logs (the entry's match_regex/match_literal + its services/severities), then logs-attributes-list to see what structured fields the record carries (error_code, module, stack-frame fields).

If the message references an exception, cross-check query-error-tracking-issues-list first — if an issue already covers it, error tracking owns the finding.

Log records carrying trace_id correlating to slow or failing traces. When a query-llm-traces-list failure spike, an query-error-tracking-issues-list burst, and a query-logs burst all share the same trace ids — that's the cleanest cross-source convergence pattern logs enables.

logs-alerts-list exposes the team's configured alerts. An alert with state = firing whose underlying condition isn't already in inbox-reports-list is a high-confidence finding — the team has the alert plumbing but not the inbox surface.

Before trusting a firing state, check the alert's history with logs-alerts-events-list (id = the alert's UUID) — it returns fires/resolves/flaps/threshold changes. A fresh fire (a new fire event in the recent window) is real; an alert that has sat firing indefinitely is usually a misconfigured always-on threshold (record it under a noise: key), not a new signal. (This endpoint rejects personal API keys with a 403; the scout's internal token should reach it — if it 403s for you too, read the alert's filter with logs-alerts-retrieve (logs-alerts-list returns only id/name/state/threshold, not filters), then run a bounded logs-count over that filter to gauge whether it's genuinely firing.)

Memory is a continuous activity. Write a scratchpad entry whenever you observe something a future logs run should know. Encode the "category" in the key prefix — pattern:, noise:, addressed:, dedupe:, report:, reviewer: — so future runs can find it with a single text= search:

• key pattern:logs:temporal-worker — "Service temporal-worker typical log volume: ~12k/hour with ~3% error severity. Anything > 10% error in the recent window is fresh degradation."
• key noise:logs:rabbitmq-deploy-window — "Log message connection refused: rabbitmq:5672 is recurring noise during deploy windows (Mon/Wed 14:00 UTC) — auto-recovers within 5 min."
• key pattern:logs:alert-47 — "Logs alert db-connection-pool-saturated (id 47) auto-mutes 02:00–04:00 UTC for nightly batch — firing outside that window is real."
• key addressed:logs:cdp-worker-2026-04-30 — "Service cdp-worker migrated to a new runtime on 2026-04-30 — log volume baseline shifted from 8k/hour to 14k/hour, treat new baseline as normal."
• key report:logs:temporal-worker — "Authored report 019f0a96-… for the temporal-worker error-rate burst on 2026-06-30. Edit it (append_evidence) if the burst persists or worsens rather than filing a new one."
• key reviewer:logs:temporal-worker — "temporal-worker logs owned by alice (GitHub login) — route its reports there."

By run #5 you'll know per-service volume and severity baselines, which alerts are intentional outliers, which open report covers a service, who owns it, and only file fresh shifts.

Search the inbox before you author — a report covering this service / message / shift may already exist (inbox-reports-list with ordering=-updated_at, then inbox-reports-retrieve the closest matches). Then, for each candidate finding:

• Edit the existing report via scout-edit-report when the inbox already covers the service or pattern. A logs shift is rarely brand-new — a service that's still degrading, an alert that's flapping again, a burst that's worsening: use append_evidence for the fresh numbers and time range (or rewrite the title/summary on a report you authored). This is the default when a match exists; don't mint a near-duplicate. The dedupe pull is real here — the same service moving twice in two days is one report, not two.
• Author a fresh report via scout-emit-report when nothing in the inbox covers it (or a known issue has new evidence that changes the verdict). The natural fit is a single, localized, validated shift — one service's volume burst, one severity step, one silent service, one fresh message firing at scale — with concrete service / message / time-range evidence (the bar is confidence ≥ 0.85). A rate or severity shift is chart-shaped — attach the service's series via charts so the step and its onset are visible in the report. Most logs reports are an investigation, not a one-line code fix, so default to requires_human_input. Always set suggested_reviewers — resolve the owning person with scout-members-list (each member carries a resolved github_login; cache it under a reviewer:logs:<service> key). It's how the report reaches a human; left empty, the report is assigned to nobody and is likely missed. After authoring, write a report:logs:<service> scratchpad entry with the report_id so the next run edits it instead of duplicating. The harness prompt carries the full report-channel contract (field schema, safety × actionability status mapping, reviewer routing, the non-idempotency caveat, and the edit rules) — this section only adds the logs-specific framing.
• Remember via scout-scratchpad-remember if it's below the bar but worth carrying forward, or to record what you ruled out and why.
• Skip with a one-line note if a scratchpad entry with a noise: or addressed: key prefix, or an existing inbox report, already covers it.

If a prior run already covered the topic, default to edit-or-skip + scratchpad refresh rather than a fresh report. The same fact twice in the inbox degrades signal-to-noise more than missing one finding for one tick.

Summarize the run — one paragraph: looked at what, authored or edited which reports, remembered what, ruled out what. The harness writes this to the run row as searchable prose; future runs read it via scout-runs-list. Do not write a separate "run metadata" scratchpad entry — the run summary already serves that role.