analyzing-experiment-session-replays

First, retrieve the experiment information and the feature flag variants (source of truth).

Step 1a: Get experiment metadata

You can either:

• Option A: Use the experiment-get tool if you already have the experiment ID from context
• Option B: Query the experiments table via HogQL:

SELECT
    e.id,
    e.name,
    f.key AS feature_flag_key,
    e.start_date,
    e.end_date
FROM system.experiments e
JOIN system.feature_flags f ON f.id = e.feature_flag_id
WHERE e.id = <experiment_id>

From the experiment data, extract:

• feature_flag_key: The feature flag controlling the experiment
• start_date and end_date: The experiment's time range

Step 1b: Get variants from the feature flag

IMPORTANT: Always get variants from the feature flag, NOT from experiment.parameters.feature_flag_variants. The parameters can be out of sync or deprecated. The feature flag is the source of truth.

Query the feature flag to get the current variants:

SELECT filters.multivariate.variants AS variants
FROM system.feature_flags
WHERE key = '<feature_flag_key>'

Select the variants path directly — selecting the whole filters object gets truncated in results for flags with large targeting configs. Example structure: [{"key": "control", "name": "Control", "rollout_percentage": 50}, {"key": "test", ...}]

The variant key values (e.g., "control", "test", "variant_a") are what you'll use to filter session recordings.

For each variant in the experiment, construct recording filters that match users exposed to that variant.

Filter structure for a variant (input to query-session-recordings-list):

{
  "date_from": "<experiment.start_date>",
  "date_to": "<experiment.end_date or current time>",
  "filter_test_accounts": true,
  "properties": [
    {
      "type": "event",
      "key": "$feature/<feature_flag_key>",
      "operator": "exact",
      "value": ["<variant_key>"]
    }
  ]
}

Key points:

• The $feature/<flag_key> event property records the flag's value on each event — filtering on it matches recordings where the flag was active with that variant. This is an approximation of exposure, broader than the experiment's exposure event ($feature_flag_called, or $experiment_exposure on the new rollout — both deduped per identity): right for browsing behavior across variants, but not an exact mirror of the analysis population — the scanning-experiments-with-replay-vision skill derives that exact filter when you need it
• value is an array of variant key strings (e.g. ["control"]); for boolean flags use ["true"] or ["false"]
• Avoid the type: "flag" / flag_evaluates_to property filter for variant scoping — the recordings query accepts it but silently ignores it, returning unfiltered results (last verified 2026-06-10). If you want to try it anyway, verify it actually filters first: a query with a nonexistent flag key should return zero recordings
• Set the date range to the experiment's start and end dates
• Enable filter_test_accounts: true to exclude test users

Use the query-session-recordings-list tool with the filters constructed in step 2.

Call the tool once per variant to get recordings for each group:

• Variant "control" → recordings for control group
• Variant "test" → recordings for test variant
• Additional variants if the experiment has more than 2

The tool returns a list of recordings with metadata including:

• distinct_id — the person's distinct ID
• recording_duration, active_seconds, inactive_seconds
• click_count, keypress_count, mouse_activity_count
• console_log_count, console_warn_count, console_error_count
• start_url — first page URL visited
• start_time / end_time, activity_score

Compare the recordings between variants by looking for:

Quantitative patterns:

• Session duration differences
• Activity levels (clicks, keypresses)
• Console error rates
• Bounce rates

Qualitative insights:

• User confusion or frustration indicators
• Different navigation paths
• Feature discovery patterns
• Error recovery behavior

Summarize the behavioral differences between variants, highlighting:

• Total recordings per variant
• Notable behavior patterns unique to each variant
• Usability issues or friction points observed
• Recommendations based on the qualitative data

Watching sessions and asking users are different instruments, not substitutes. Recordings show what people did with the change; a short survey, shown when they finish the experimented flow, captures what they thought of it — a rating and an optional comment, readable per variant. For a user-facing change of real size, the two together make a fuller qualitative read than either alone, so mention the option when the behavioral comparison in step 4 leaves opinion unaccounted for, or when a pattern in the recordings is a hypothesis worth checking with the people who produced it. Once per conversation at most; drop it if declined.

Default to asking every exposed user rather than one variant: a popover shown to only one arm is itself a difference between the arms, and the response event carries the variant anyway, so the split survives.

→ See references/qualitative-feedback.md in [[diagnosing-experiment-results]]