signals-scout-surveys

Four cheap reads cold-start a run:

• scout-scratchpad-search (text=survey or text=nps) — durable team steering. Entries with pattern:, noise:, addressed:, dedupe:, report:, or reviewer: key prefixes, plus the team's known active survey IDs, primary NPS / CSAT survey, healthy response baselines, known themes already raised, which report covers a theme, and who owns it.
• scout-runs-list (last 7d) — what prior surveys runs found and ruled out.
• inbox-reports-list (filter by search=survey name/theme, source_product, ordering=-updated_at) — the reports already in the inbox. A theme or regression you've reported before is an edit, not a fresh report; pull the closest matches with inbox-reports-retrieve before authoring.
• scout-project-profile-get — top_events for survey shown / survey dismissed / survey sent reach (the survey product isn't yet surfaced in the profile inventory; see "When you hit a gap" below).

Then orient on surveys specifically. Order matters — busy projects can have 100+ active surveys, and surveys-get-all is never the right cold-start move there. Each survey object is 30–50 KB (questions, internal targeting flag, appearance theme, creator metadata) and even limit: 5 returns ~30 KB. Listing the lot blows the token budget before you've made a single decision.

Right order:

1. surveys-global-stats (last 30d) — cheap project-wide check: are surveys converting at all? If survey sent total is zero, close out empty.

2. Rank candidates by recent activity, not by config. Use execute-sql to find the top survey ids by survey sent volume in the last 30d:

   Copy

   SELECT
       JSONExtractString(properties, '$survey_id') AS survey_id,
       count() AS sent_count,
       max(timestamp) AS last_sent
   FROM events
   WHERE event = 'survey sent'
     AND timestamp > now() - INTERVAL 30 DAY
   GROUP BY survey_id
   ORDER BY sent_count DESC
   LIMIT 20

3. survey-get {id} on the top 5–10 ids only — full config when you actually need to read questions / targeting / iteration / type. Never surveys-get-all on a project where step 2 returns more than ~20 distinct ids.

4. survey-stats {id} per candidate for shown / dismissed / sent counts.

Use surveys-get-all {"limit": 5} only as a last resort when discovering a survey by name, and prefer surveys-get-all {"search": "..."} over a blind page walk.

Patterns to watch — starting points, not a checklist.

Surveys with rating questions (NPS 0–10, CSAT 1–5, single rating) are the cleanest quantitative signal. For each rating-style active survey, pull the last 30 days of survey sent events and compute the score trend.

Two mechanical traps make response SQL non-obvious — read references/response-querying.md before writing any. Answers land under two property key schemes (id-based $survey_response_<question_id> and legacy index-based $survey_response / $survey_response_<n>) that must be coalesced — querying the id-based key alone reads as "no responses" on legacy surveys — and newer clients can emit multiple survey sent events per submission, so every count needs the $survey_submission_id dedupe. The reference has the copy-ready rating-trend SQL with both handled.

What counts as "enough responses" depends on the survey's normal volume. Flagship NPS surveys can hit 100+/week; a feature-specific widget survey running at 15–25 responses/month is also normal. Use a tiered bar:

• High-volume surveys (baseline ≥ 30 responses/week): require ≥ 30 in the recent week, score drop ≥ 10% of scale (1 point NPS, 0.5 CSAT), holds across the most recent 7 days vs the prior trailing 21 days.
• Low-volume surveys (baseline 5–30/week): require ≥ 8 in the recent 14 days, score drop ≥ 15% of scale, comparing against the survey's own trailing 60-day baseline rather than week-over-week. Smaller samples need a larger effect to outrun noise.
• Very low-volume surveys (< 5/week): rating trends are too noisy to act on. Treat as theme-aggregation only; memory entry, not emit.

In all tiers, anchor on the survey's own trailing baseline before any global rule of thumb. A widget survey with a 6.0 trailing average that drops to 5.2 on N=12 is more interesting than a popover at NPS 32 → 31 on N=400 — and the scout's job is to spot the meaningful one.

survey-stats returns shown and sent counts. A survey that converted at 8% last month and 0.5% this week is broken — usually because the question wording changed, the target audience changed, or the survey is being shown in a different context (a flag flipped, a page was redesigned). Pair the stats with survey-get to check the updated_at and questions; if the survey config was edited near the inflection, that's the cause. If not, suspect upstream.

Disqualifier: a survey at the end of its scheduled window naturally tails off. Check schedule.end_date before treating low recent response rate as a regression.

survey shown events are impressions; survey dismissed are explicit close-outs; survey sent are completions. Their meaning depends on the survey's type, and the scout has to read type from survey-get before interpreting any ratio:

• popover — survey shown fires when the popover auto-renders. A high dismiss rate is genuine signal: users are seeing it and immediately killing it.
• widget — survey shown only fires when the user clicks the widget trigger. A high dismiss rate means users opened the widget and changed their mind, not that the team is spamming them. Baseline dismiss rates are naturally higher (50–70% is common; the Logs Feedback widget on PostHog itself runs at 64% with healthy NPS) and shouldn't be flagged as fatigue.
• api — survey shown fires from SDK calls. Semantics depend on the integrating product; check survey-get to see how it's wired before interpreting trends.

If the dismiss rate jumps sharply on a popover survey (e.g. baseline 30%, recent 70%), users are seeing it and immediately killing it. Common causes: the survey now appears at a worse moment in the user journey, or fatigue from displaying too often.

For widget and api surveys, treat dismiss-rate shifts as low signal unless they're paired with a response-volume drop — that's when something upstream of the click changed.

SELECT
    toDate(timestamp) AS day,
    countIf(event = 'survey shown') AS shown,
    countIf(event = 'survey dismissed') AS dismissed,
    countIf(event = 'survey sent') AS sent,
    dismissed / nullIf(shown, 0) AS dismiss_rate
FROM events
WHERE event IN ('survey shown', 'survey dismissed', 'survey sent')
  AND JSONExtractString(properties, '$survey_id') = '<survey_id>'
  AND timestamp > now() - INTERVAL 30 DAY
GROUP BY day
ORDER BY day

Memory note when a dismiss rate is structurally high (e.g. an exit-intent survey naturally has high dismiss); don't re-flag every run.

This is the highest-value pattern — and the one with the highest false-positive risk. For each survey with at least one open-text question, pull recent responses (the open-text pull SQL — key coalesce and submission dedupe included — is in references/response-querying.md) and look for clustering.

Read the responses. Look for:

• Convergence on a noun phrase or feature name — five users mentioning "checkout", "the new editor", "API key page" within 14 days is a real theme.
• Sentiment polarity — separate complaints from praise from feature requests. Don't combine them into a single "users said things" finding.
• Specificity — "it's slow" is too generic; "the dashboard list page is slow when I have > 10 dashboards" is concrete. The latter is report-worthy.

Theme is report-worthy when:

• ≥ 5 distinct respondents converge on the same theme within 14 days, OR
• ≥ 3 distinct respondents converge AND the theme matches a recent activity-log entry (deploy, flag flip, new feature) within the same window — strong qualitative confirmation of an impact.

When you file a report, quote 2–3 representative responses verbatim in the evidence (no PII; truncate at sentence level if a response is long). Name the theme as a concrete claim ("Users report the dashboard list is slow with > 10 dashboards"), not a vague summary ("Users have feedback about dashboards").

Don't file a report when:

• Responses are mostly NPS rating-only with no text — there's no theme to find.
• Themes are evenly split (some users complaining, others praising the same feature) — the signal cancels itself; memory entry instead.
• A memory entry tagged addressed already covers the same theme.

survey shown count diverging sharply from baseline (up 5x or down 5x) usually means an upstream targeting condition changed. Four sources to check via survey-get:

• linked_flag_id — survey shows only when this flag evaluates true. A flag rollout change directly resizes the audience.
• targeting_flag_id — user-configured cohort / property targeting. Same effect; also subject to cohort recomputation lag.
• linked_insight_id — survey gates on viewing a specific insight. If the insight is deleted or its query is broken, the survey goes dead. Cross-check with insight-get and inbox-reports-list for any insight-side issues.
• conditions — URL pattern, event-trigger, or repeatedActivation — config changes here directly resize the trigger surface.

If the upstream changed near the inflection, flag it as targeting drift, not a survey regression. (Note: the auto-managed internal_targeting_flag is a separate construct that suppresses already-responded / already-dismissed users — not a targeting source the team controls, and changes to it are usually expected.)

Memory-worthy unless the survey is load-bearing (e.g. NPS the team reports on publicly) — then file a report so the team knows the sample frame changed.

A survey created > 90 days ago with steadily declining response volume and no updated_at activity is probably forgotten. P3 recommendation, not an anomaly: suggest the team retire it, refresh the question, or rotate the audience. Don't re-file if a memory entry already flagged it.

When a theme emerges, cross-check advanced-activity-logs-list for the period around the inflection. If a deploy / flag flip / feature change in the same week matches the theme content, the finding lands much harder ("4 users complained about checkout slowness on $date; deploy of checkout-rewrite-v2 flag rolled to 100% on $date-1"). Timing is hint, not proof — say "matches" rather than "caused by".

Recurring surveys (schedule: recurring, iteration_count > 1, iteration_frequency_days > 0) cycle iterations every N days, and each iteration's responses are tagged with $survey_iteration. Comparing themes across iterations on the same survey is itself a signal:

• Theme volume rising in iteration N+1 vs N on the same survey = the issue is growing, not new.
• New theme appearing in iteration N+1 that wasn't in earlier iterations = recent product change introduced something.
• Score baseline shifting between iterations = sustainable change in user perception, more interesting than within-iteration noise.

Filter open-text and rating queries by $survey_iteration to compare cleanly:

AND JSONExtractString(properties, '$survey_iteration') = '<n>'

When filing a report on a recurring survey, name the iteration explicitly in the evidence ("iteration 3 of nps-q1-2026, last 14d") so the team reads it against the right baseline.

Memory is a continuous activity. Write a scratchpad entry whenever you observe something a future surveys run should know. Encode the "category" in the key prefix — pattern:, noise:, addressed:, dedupe:, report:, reviewer: — so future runs find it with a single text= search:

• key pattern:surveys:active-inventory — "Active surveys: nps-q1-2026 (id abc, NPS 0–10), feedback-modal (id def, open text), csat-after-purchase (id ghi, 1–5 rating)."
• key pattern:surveys:nps-q1-2026 — "Primary NPS survey is nps-q1-2026; healthy baseline 32 ± 5 over last 90 days, ~120 responses/week. Score < 25 or responses < 60/week is the alert bar."
• key noise:surveys:feedback-modal — "feedback-modal exit-intent survey naturally has 70% dismiss rate — that's expected behavior for this trigger, not a regression."
• key addressed:surveys:theme-checkout-step-2-2026-05-04 — "Theme checkout-step-2-confusion raised in run on 2026-04-30; team acknowledged, fix shipped 2026-05-04. Don't re-file unless theme reappears post-2026-05-04."
• key addressed:surveys:csat-old-stale — "Survey csat-old last got responses 2026-02; appears abandoned but the team still has it active. P3 recommendation already filed; don't re-recommend."
• key report:surveys:theme-checkout-step-2 — "Authored report 019f0a96-… for the checkout-step-2 confusion theme on 2026-06-30. Edit it (append_evidence) if the theme grows or recurs rather than filing a new one."
• key reviewer:surveys:nps-q1-2026 — "nps-q1-2026 owned by alice (GitHub login) — route its reports there."

By run #5 you'll know the team's active surveys, healthy response volumes, score baselines, which dismiss rates are structural, which themes have already been raised, which report covers a theme, and who owns it — so when a real theme or regression appears, the report lands with the right context already attached.

Search the inbox before you author — a report covering this theme / survey / regression may already exist (inbox-reports-list with ordering=-updated_at, then inbox-reports-retrieve the closest matches). Then, for each candidate finding:

• Edit the existing report via scout-edit-report when the inbox already covers the theme or survey. A theme that's growing, a regression that's deepening, a later iteration's responses confirming an earlier read: use append_evidence for the fresh response counts, score deltas, and time range (or rewrite the title/summary on a report you authored). This is the default when a match exists; don't mint a near-duplicate.
• Author a fresh report via scout-emit-report when nothing in the inbox covers it. The natural fits are a single validated theme (≥ 5 converging respondents, with 2–3 verbatim quotes — no PII) or one survey's score / response-rate / abandonment regression that clears the tiered bar, with concrete survey ids, question ids, response counts, and score deltas as evidence (the bar is confidence ≥ 0.85; sample-size matters more here than other domains — a report on 10 responses needs to be tighter than one on 200). A score or response-rate regression is chart-shaped — attach the survey's trend via charts so the drop against its own baseline is visible; a theme lives in its quotes, not a chart. A survey finding is an investigation, not a one-line code fix, so default to requires_human_input. Always set suggested_reviewers — resolve the owning person with scout-members-list (each member carries a resolved github_login; cache it under a reviewer:surveys:<survey> key). It's how the report reaches a human; left empty, the report is assigned to nobody and is likely missed. After authoring, write a report:surveys:<theme-or-survey> scratchpad entry with the report_id so the next run edits it instead of duplicating. The harness prompt carries the full report-channel contract (field schema, safety × actionability status mapping, reviewer routing, the non-idempotency caveat, and the edit rules) — this section only adds the surveys-specific framing.
• Remember via scout-scratchpad-remember if below the bar but worth carrying forward (a theme with only 3 respondents that might grow, a score wobble that didn't yet hold for two weeks), or to record what you ruled out and why.
• Skip with a one-line note if a scratchpad entry with a noise: or addressed: key prefix, or an existing inbox report, already covers it.

If a prior run already covered the theme, default to edit-or-skip + scratchpad refresh rather than a fresh report. The same theme twice in the inbox degrades signal-to-noise more than missing one finding for one tick.

Summarize the run — one paragraph: which surveys, what themes / anomalies you found, what reports you authored or edited, what you remembered, what you ruled out. The harness writes that summary to the run row as searchable prose; future runs read it via scout-runs-list. Do not write a separate "run metadata" scratchpad entry — the run summary already serves that role.

Two MCP gaps are known and may be worth flagging in a separate PR rather than working around in-skill:

• Project profile doesn't include surveys. Cold-start orientation has to call surveys-get-all directly. Adding a _surveys builder to products/signals/backend/scout_harness/profile/builders.py (a few rows: active count, top surveys by recent volume, primary NPS / CSAT survey if any) would let every scout — not just this one — see surveys at orientation time. Worth a P3.
• Survey summarization isn't MCP-callable. The product has a summarization pipeline at products/surveys/backend/summarization/ but it's not exposed as an MCP tool. If it were, this scout could lean on cached summaries instead of re-aggregating themes from scratch each run. Worth a P2 for accuracy and cost.

If you notice a third gap during a run that would meaningfully unlock this scout, write a scratchpad entry with key mcp-gap:surveys:<short-name> so the gap surfaces in the next review via text=mcp-gap.